SlowGuess/NAACL_2022_OCR huggingface dataset:
We encoded the extracted paragraphs  $\left([p_{1}, p_{2} \ldots p_{n}]\right)$  using the retriever module, medRoBERTa, to get a vector representation of 768 dimensions for each of the paragraphs  $\left([v_{1}, v_{2} \ldots v_{n}]\right)$ . Training the retriever module on auxiliary tasks of predicting SA and SI helped align these paragraph representations for SA and SI prediction. Then, we added a prediction vector  $(v_{0})$  along with all the vector representations of the paragraphs to get  $\mathcal{V} = [v_{0}, v_{1}, v_{2} \ldots v_{n}]$ . We passed  $\mathcal{V}$  through our multi-head attention model to get the hidden representations  $\mathcal{H} = [h_{0}, h_{1} \ldots h_{n}]$ . We then passed  $h_{0}$  through a SA inference layer and SI inference layer to predict the labels. During the whole training process, the weights of the retriever module were frozen whereas  $v_{0}$  was a learnable vector initialised as an embedding in the multi-head attention model. We used a separate  $v_{0}$  prediction vector so that it could retain the information from all the other paragraph representations for hospital-stay level prediction similar to how [CLS] is utilized in different transformer-based models for sequence prediction (Devlin et al., 2018; Liu et al., 2019). We tuned the number of layers and number of attention heads of our prediction module as hyper-parameters and achieved the best performance using a 2-layer and 3-attention head model. Our complete ScANER model is illustrated in Fig 4.
Since the labels for both the retriever and prediction task are imbalanced, we used macro-weighted precision, recall, and F1-score to evaluate the overall performance of our models. Macro-weighted metrics provide better model insights across all labels.
Evidence Retriever Performance Our multitask learning model achieved a F1-score of 0.83 for extracting positive evidence paragraphs and an F1-score of 0.88 overall. The retriever model has higher recall than precision for the positive evidence paragraphs (0.87 > 0.79), SA (0.74 > 0.71), and SI (0.62 > 0.46) events, as shown in Table 4. In healthcare, there is an incentive to maximize recall over precision (Watson and McKinstry, 2009). As mentioned in §4.2, ScANER was trained with added noisy paragraphs and is therefore robust to the extracted evidence paragraphs if they contain some false positives.
The retriever module achieves an overall F1-score of 0.63 for SA prediction and 0.64 for SI prediction at paragraph-level. The performance for positive SA and SI evidence is much higher than the performance for neg_unsure SA and negative SI. We looked at the confusion matrices for SA and SI paragraph-level prediction and found that largely ScANER made mistakes between positive and neg_unsure labels for SA prediction and between positive and negative labels for SI prediction (refer Appendix C). The poor performance in SA for neg_unsure evidence prediction is mainly due to data sparsity where the neg_unsure cases are only 1743; in contrast, the positive cases are 4-fold higher. Similarly, for SI the positive cases are 1.4 times higher than the negative cases.
Hospital-stay level Prediction Performance Our multi-head attention model is able to achieve an overall macro F1-score of 0.78 for SA prediction and 0.60 for SI prediction, as shown in Table 5. For SA, the prediction module achieves a recall of 0.93 for the positive label. After analysing the confusion matrix, the model largely predicts a positive label for the visits with neg_unsure label, as shown in Table 6. The poor performance for neg_unsure is largely because of its small representation in the training set of ScAN, 54 negative cases as compared to 377 positive and 1,381 neutral instances. In our future work, we plan to expand ScAN with more instances of negative SA events.
For SI, the prediction module achieves an overall F1-score of 0.60 with a precision of 0.63 and recall of 0.66. The model has a high recall for neutral-SI and positive but the positive label has a low precision of 0.49. After analysing the test set, we observed that a lot of patient hospital-stays with negative labels are getting wrongly predicted as positive, as shown in Table 6. After doing error
Paragraph Evidence Prediction	Paragraph SA Prediction	Paragraph SI Prediction
Evidence	P	R	F	Labels	P	R	F	Labels	P	R	F
Yes	0.79	0.87	0.83	Positive	0.71	0.74	0.73	Positive	0.46	0.62	0.53
No	0.95	0.91	0.93	Neg_Unsure	0.19	0.26	0.22	Negative	0.38	0.46	0.42
-	-	-	-	Neutral-SA	0.95	0.92	0.93	Neutral-SI	0.98	0.99	0.98
Overall	0.87	0.89	0.88	Overall	0.62	0.64	0.63	Overall	0.61	0.69	0.64
P: Precision, R: Recall and F: F1-score.
Table 4: Paragraph level performance of the evidence retriever module. The overall evaluation metrics (precision, recall and F1-score) are macro-weighted. Evidence prediction is the main task whereas SA and SI prediction are auxiliary tasks and help the model align the vector representations of the paragraphs for the hospital-stay level suicidal behavior prediction.
Hospital-stay SA Prediction
Labels	Precision	Recall	F1-score
Positive	0.81	0.93	0.87
Neg_Unsure	0.48	0.58	0.52
Neutral-SA	0.98	0.93	0.96
Overall	0.76	0.81	0.78
Hospital-stay SI Prediction
Labels	Precision	Recall	F1-score
Positive	0.49	0.93	0.65
Negative	0.40	0.11	0.18
Neutral-SI	0.99	0.95	0.97
Overall	0.63	0.66	0.60
Table 5: Hospital-stay level SA and SI prediction performance of ScANER.
Hospital-stay SA Prediction
Positive	Neg_Unsure	Neutral-SA
Positive	85	4	2
Neg_Unsure	5	11	3
Neutral-SA	15	8	303
Hospital-stay SI Prediction
Positive	Negative	Neutral-SI
Positive	41	2	1
Negative	27	4	4
Neutral-SI	15	4	338
Table 6: Confusion matrices for SA and SI prediction at hospital-stay level.
analysis for hospital-stays with negative labels, we observed that a lot of extracted evidence paragraphs contain information that suggests that the patient had SI before the SA but does not have SI anymore during the hospital-stay. As shown in the example in Fig 5, the past SI is an explanation for the SA but then the patient does not have any further SI
during the hospital-stay. This suggests that period assertions for these annotations are quite important and we aim to add period assertion property in our future work by further annotating ScAN.
Depression: On further questioning, the pt reported that the cocaine ingestion was a suicide attempt. He stated that he was having financial difficulty over the past few months and thought that if he attempted suicide, his family would be able to receive his life insurance. A 1:1 sitter was ordered and psychiatry consult was called. He denied any futher suicidal ideations while in the hospital.#. Fever: Pt developed fever to 101 with leukocytosis on  $[{}^{**}2189-7-20^{**}]$ .
1. On further ...... insurance: Positive SA
2. He denied .... hospital: Negative SI
Figure 5: An instance for which ScANER incorrectly predicted a negative hospital-level SI as positive.
In this paper, we introduce ScAN: a publicly available suicide attempt (SA) and ideation (SI) events dataset that consists of 12,759 EHR notes with 19,960 unique evidence annotations for suicidal behavior. To our knowledge, this is the largest and publicly available dataset for SA and SI, an important resource for suicidal behaviors research. We also provide a strong RoBERTa baseline model for the dataset: ScANER (SA and SI retriever) which consists of two sub-modules: (a.) an evidence retriever module that extracts all the relevant evidence paragraphs from the patient's notes and (b.) a prediction module that evaluates the extracted evidence paragraphs and predicts the SA and SI event label for the patient's stay at the hospital. ScAN and ScANER could help extract suicidal behavior
in patients for suicide surveillance and predictions, leading to potentially early intervention and prevention efforts by medical professionals.
Emily Alsentzer, John R Murphy, Willie Boag, WeiHung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323.
Heather D Anderson, Wilson D Pace, Elias Brandt, Rodney D Nielsen, Richard R Allen, Anne M Libby, David R West, and Robert J Valuck. 2015. Monitoring suicidal patients in primary care using electronic health records. The Journal of the American Board of Family Medicine, 28(1):65-71.
Harish S Bhat and Sidra J Goldman-Mellor. 2017. Predicting adolescent suicide attempts with neural networks. arXiv preprint arXiv:1711.10057.
Leo Breiman. 2001. Random forests. Machine learning, 45(1):5-32.
Ran Cao, Leyuan Fang, Ting Lu, and Nanjun He. 2020. Self-attention-based deep feature fusion for remote sensing scene classification. IEEE Geoscience and Remote Sensing Letters, 18(1):43-47.
Center for Behavioral Health Statistics Quality CBHSQ. 2020. 2019 national survey on drug use and health (nsduh): Methodological summary and definitions. Substance Abuse and Mental Health Services Administration.
Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning, 20(3):273-297.
Alex Crosby, LaVonne Ortega, and Cindi Melanson. 2011. Self-directed violence surveillance; uniform definitions and recommended data elements.
Marika Cusick, Prakash Adekkanattu, Thomas R Campion Jr, Evan T Sholle, Annie Myers, Samprit Banerjee, George Alexopoulos, Yanshan Wang, and Jyotishman Pathak. 2021. Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation. Journal of psychiatric research, 136:95-102.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Johnny Downs, Sumithra Velupillai, Gkotsis George, Rachel Holden, Maxim Kikoler, Harry Dean, Andrea Fernandes, and Rina Dutta. 2017. Detection of suicidality in adolescents with autism spectrum disorders: developing a natural language processing approach for use in electronic health records. In AMIA annual symposium proceedings, volume 2017, page 641. American Medical Informatics Association.
Andrea C Fernandes, Rina Dutta, Sumithra Velupillai, Jyoti Sanyal, Robert Stewart, and David Chandran. 2018. Identifying suicide ideation and suicidal attempts in a psychiatric clinical research database using natural language processing. Scientific reports, 8(1):1-10.
Krystl Haerian, Hojjat Salmasian, and Carol Friedman. 2012. Methods for identifying suicide or suicidal ideation in ehrs. In AMIA annual symposium proceedings, volume 2012, page 1244. American Medical Informatics Association.
Holly Hedegaard, Sally C Curtin, and Margaret Warner. 2020. Increase in suicide mortality in the united states, 1999-2018.
Assaf Hoogi, Brian Wilcox, Yachee Gupta, and Daniel L Rubin. 2019. Self-attention capsule networks for image classification. arXiv preprint arXiv:1904.12483.
Peter B Jensen, Lars J Jensen, and Søren Brunak. 2012. Mining electronic health records: towards better research applications and clinical care. Nature Reviews Genetics, 13(6):395-405.
Shaoxiong Ji, Shirui Pan, Xue Li, Erik Cambria, Guodong Long, and Zi Huang. 2020. Suicidal ideation detection: A review of machine learning methods and applications. IEEE Transactions on Computational Social Systems.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic iii, a freely accessible critical care database. Scientific data, 3(1):1-9.
Ronald C Kessler, Robert M Bossarte, Alex Luedtke, Alan M Zaslavsky, and Jose R Zubizarreta. 2020. Suicide prediction models: a critical review of recent research with recommendations for the way forward. Molecular psychiatry, 25(1):168-179.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Marie-Hélène Metzger, Nastassia Tvardik, Quentin Gicquel, Côte Bouvry, Emmanuel Poulet, and Véronique Potinet-Pagliaroli. 2017. Use of emergency department electronic medical records for automated epidemiological surveillance of suicide attempts: a french pilot study. International journal of methods in psychiatric research, 26(2):e1522.
Bhanu Pratap Singh Rawat, Fei Li, and Hong Yu. 2019. Naranjo question answering using end-to-end multi-task learning model. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2547-2555.
Truyen Tran, Dinh Phung, Wei Luo, Richard Harvey, Michael Berk, and Svetha Venkatesh. 2013. An integrated framework for suicide risk prediction. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1410-1418.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Philip WB Watson and Brian McKinstry. 2009. A systematic review of interventions to improve recall of medical advice in healthcare consultations. Journal of the Royal Society of Medicine, 102(6):235-243.
WHO. 2021. Suicide fact sheet: https://www.who.int/news-room/factsheets/detail/suicide.
Baosong Yang, Longyue Wang, Derek F Wong, Shuming Shi, and Zhaopeng Tu. 2021. Context-aware self-attention networks for natural language processing. Neurocomputing, 458:157-169.
Abdou Youssef. 1999. Image downsampling and upsampling methods. National Institute of Standards and Technology.
A Selected Clinical Sections
The sections selected for annotations after using clinical_sectionizer are enumerated below:
1. Allergy
2. Case Management
3. Consult
4. Discharge Summary
5. Family history
6. General
7. HIV Screening
8. Labs and Studies
9. Medication
10. Nursing
11. Nursing/other
12. Nutrition
13. Observation and Plan
14. Past Medical History
15. Patient Instructions
16. Physical Exam
17. Physician
18. Present Illness
19. Problem List
20. Radiology
21. Rehab Services
22. Respiratory
23. Sexual and Social History
24. Social Work
All the hyper-parameter settings for both modules of ScANER are provided in Table 7.
Evidence Retriever Module
Learning Rate 2e-5	Warmup steps 2,000	Optimizer Adam	Adam ε 1e-8
γ	α	β
2.5	1.1	1.5
Hospital Stay SA-SI Prediction Module
Attention Heads 3	Attention Layers 2	Learning Rate 1e-3	Warmup steps 1,200
Optimizer Adam	Adam ε 1e-8
C Confusion matrices
The confusion matrices for SA and SI prediction at paragraph level is provided in Table 8.
Table 7: Hyper-parameter setting for both retriever and prediction module of ScANER.
Paragraph SA Prediction
Positive	Neg_Unsure	Neutral-SA
Positive	1,804	285	344
Neg_Unsure	253	118	80
Neutral-SA	472	204	7,314
Paragraph SI Prediction
Positive	Negative	Neutral-SI
Positive	206	69	56
Negative	71	87	31
Neutral-SI	170	73	10,111
Table 8: Confusion matrices for the predictions on the test set of evidence retriever.