SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 2: Performance of baseline and literature-augmented outcome prediction models on all clinical outcomes. We note that LOS is a multiclass target; we observe substantial gains in 2/4 of the classes (Table 10 in the Appendix).
Model	No PMV	PMV
Prec@10	Rec@10	Prec@10	Rec@10
BLUEBERT	52.86	9.95	55.71	11.61
+Avg	64.29	12.1	60.0	12.5
+SVote	61.43	11.56	64.29	13.39
+WVote	62.86	11.83	52.86	11.01
+WAvg	58.57	11.02	52.86	11.01
UMLSBERT	58.57	11.02	57.14	11.90
+Avg	67.14	12.63	64.29	13.39
+SVote	61.43	11.56	62.86	13.1
+WVote	64.29	12.1	64.29	13.39
+WAvg	68.57	12.9	62.86	13.1
(a) For PMV
Model	No MOR	MOR
Prec@10	Rec@10	Prec@10	Rec@10
BLUEBERT	99.8	11.15	46.39	23.62
+Avg	99.59	11.13	68.91	17.81
+SVote	99.69	11.14	73.39	16.55
+WVote	99.59	11.13	68.36	16.94
+WAvg	99.8	11.15	69.46	16.07
UMLSBERT	99.8	11.15	42.06	39.21
+Avg	99.59	11.13	69.07	15.78
+SVote	99.8	11.15	40.69	38.72
+WVote	99.49	11.12	68.44	19.94
+WAvg	100.0	11.17	68.92	14.81
(b) For MOR
Table 3: Precision and recall scores for top  $10\%$  high-confidence predictions per class.
clinical outcomes, PMV and mortality. On LOS prediction, results are more mixed, with minor improvements on micro F1 but no improvements on other metrics. Comparing BLUEBERT and UMLSBERT, variants that use UMLSBERT do slightly better on PMV and mortality, while results on LOS are more mixed. Comparing across literature aggregation strategies, there is no clear winner, though voting-based strategies seem to have a slight advantage, especially on UMLSBERT.
Evaluating High-Confidence Predictions. In addition to standard evaluation, we evaluate the
<sup>6</sup>We also experiment with CORe but observe consistently lower scores (Table 8 in Appendix F).
top  $10\%$  high-confidence predictions per class for all models (precision/recall@TOP-K), informative for two key reasons. First, when using automated outcome prediction systems in a clinical setting, it is reasonable to only consider raising alarms for high-confidence positive predictions to avoid alarm fatigue (Sendelbach and Funk, 2013). Second, high-confidence predictions for both positive and negative classes can be used to reliably assist with hospital resource management (e.g., predicting future ventilation and hospital bed needs).
Tables 3a and 10 show the precision/recall@TOP-K scores for all models on prolonged mechanical ventilation, mortality, and length of stay prediction. In Table 3a, we see that our literature-augmented models achieve much higher precision scores than the baseline ( $\sim$ 9-12 points higher in most cases) for the PMV negative class. We also see higher precision scores than the baseline for the positive class ( $\sim$ 5-9 points higher in most cases). This is a strong indicator that our literature-augmented pipeline might offer more utility for PMV detection in a clinical setting than using EHR notes only. Table 3b shows similarly encouraging trends for mortality prediction. The mortality prediction dataset is the most skewed of the three datasets, and therefore we do not see much performance difference across models on the negative class. However, on the positive class, our literature-augmented models show dramatic increase in precision. In particular, BLUEBERT-based literature models show an increase in precision of  $\sim$ 22-27 points, at the expense of only  $\sim$ 6-7 point drop in recall relatively to non-literature models. $^{7}$  This also indicates that literature-augmented mortality prediction might be more precise and reliable in a clinical setting than using clinical notes alone. From Table 10 (Appendix H), we can see that for LOS prediction, our models show clear gains ( $\sim$ 2-5 points) on classes 1 and 2 (i.e., 3-7 days and 1-2 weeks), and minor gains for some variants on class 3 ( $>2$  weeks). We also perform an alternate evaluation in which we only score predictions from our literature-augmented models that show a relative confidence increase of at least  $10\%$  over the baseline prediction, presented in Appendix H.
Learning To Retrieve Using Outcomes. BEEP trains separate models for literature retrieval and outcome prediction. Inspired by Lee et al. (2019),
we develop a learning-to-retrieve (L2R) formulation that trains both jointly to ensure that the retriever can learn from outcome feedback. However, our L2R model does not improve performance over BEEP (results in Table 7 in Appendix E). We provide discussion for potential reasons in Appendix E. This is an interesting direction for future work.
Given BEEP's improved performance, we further assess the utility of retrieved literature and cases where adding literature is particularly helpful.
Diversity of retrieved literature. As a preliminary analysis, we evaluate the diversity of the abstracts retrieved for admission notes in our datasets, as a proxy for the degree to which literature is personalized to specific patient cases. For the 100 most frequently retrieved abstracts for each clinical outcome, Figures 4a, 4b, and 4c in Appendix H show proportions of patient notes for which these abstracts are judged as relevant by our retrieverank pipeline. From these histograms, we see a stark difference for LOS which is much less diverse than both PMV and MOR, indicating that the literature retrieved for length of stay prediction may be less personalized to patient cases than the literature retrieved for other outcomes. We leave to future work exploration of diversifying retrieved papers across patients and examining the effect on outcome prediction performance. $^{8}$
Qualitative examination of retrieved literature. We qualitatively examine literature retrieved for cases in which our model shows large confidence increases over the baseline to determine its utility in making the right prediction. We study increases in both directions, i.e. cases in which adding literature resulted in a confidence increase in either the correct outcome label (good) or incorrect outcome label (bad). For each clinical outcome, a bio-NLP expert looked at the top 5 cases from each category based on the magnitude of confidence increase (total 10 cases per outcome). For each case, the expert looks at the top 5 abstracts retrieved for the case (total 50 abstracts per outcome) and assigns each abstract to one of 8 categories we define for categorizing degree of relevance and type of evidence provided, including retrievals considered helpful
<sup>8</sup>We perform an ablation in which we use only the retrieved literature for prediction, showing quantitative evidence for the utility of retrieved literature (see Appendix G).
Note that since the MOR class is rare, a larger recall drop could still translate to a small number of incorrect cases only
Patient EHR	Retrieved Abstract	Evidence Type	Outcome
CHIEF COMPLAINT: liver transplant
PRESENT ILLNESS: ...s/p liver tran-plant...Dx: ESLD secondary to alcoholic cirrhosis.
MEDICAL HISTORY: EtOH Cirrhosis	Retrospective review of data of 73 consecutive patients with cirrhosis requiring MV...majority of patients, 51/64 (79.7%), dying in the first 48 hours of intubation...	Patient condition and outcome directly related	No PMV
CHIEF COMPLAINT: Aortic dissection
PRESENT ILLNESS: ...72-year-old woman...chest pain...had type A aortic dissection...an intramural hematoma...proceed with surgery...
MEDICAL HISTORY: HTN Renal failure	Acute type A aortic dissection presents a formidable challenge...the most important variables associated with in-hospital mortality in patients undergoing surgery for this condition...suggests that CPB time, diabetes mellitus and postoperative bleeding are the main determinants of in-hospital death.	Known outcome indicators not present in patient	No MOR
CHIEF COMPLAINT: Dyspnea, fever
PRESENT ILLNESS: 58F w/ HCV cirrhosis...requiring BiPAP, ultimately urgent intubation... extubated ... short of breath...
MEDICAL HISTORY: HCV cirrhosis	...study identifies specific predictors of increased mortality and resource utilization in cirrhotic patients...Increased LOS in the MICU was associated with mechanical ventilation...	Ongoing treatment and outcome related	LOS >2 weeks
Table 4: Qualitative examples of retrieved literature that is helpful for increasing prediction confidence of the correct outcome. Case 1 shows an example of retrieved literature that strongly matches patient condition and provides direct evidence linking it to the outcome of interest. Case 2 shows an example with indirect evidence, in which retrieved literature lists outcome indicators not present in the patient. Case 3 shows an example of retrieved literature describing a link between patient's ongoing treatment and outcome of interest. green: patient characteristics; blue: outcome of interest; red: known indicators of the outcome measure not present in the patient.
Figure 3: Literature categorization for both correct and incorrect outcome cases. For PMV and MOR, retrieved literature for correct cases is more often categorized as helpful, and unhelpful literature dominates for incorrect cases. For LOS, literature for both categories is more often categorized as unhelpful.
and unhelpful. For example, see Table 4 (evidence type column; more in Appendix).
As seen in Table 4, for helpful categories, retrieved literature matches patient characteristics (especially current condition) and includes evidential links between outcome of interest and patient conditions/treatment. In the first case, the retrieved abstract provides evidence that patients with cirrhosis have high mortality in the first 48 hours of intubation, entails the patient might not undergo prolonged ventilation. In the second case, the abstract
lists comorbidities associated with in-hospital mortality (outcome of interest), but none are present in the patient under consideration, which can be taken as weak indication that the patient may survive. Similarly, for the third case, the retrieved abstract mentions that cirrhotic patients may have longer hospital stays if they are on mechanical ventilation. This matches our patient's treatment history since she has cirrhosis and was briefly intubated and extubated, before experiencing shortness of breath again. Given this, the patient might have a longer length of stay. Conversely, unhelpful retrieved literature often does not match patient characteristics or may not contain evidence relevant to the outcome. See more example explanations in Appendix I.
Figure 3 presents the distribution of helpful and unhelpful categories for both kinds of cases for all outcomes. We can see that for correct outcome cases from both PMV and mortality, retrieved literature is more frequently assigned to one of the helpful categories, while for incorrect outcome cases, retrieved literature is more frequently assigned to one of the unhelpful categories. For LOS, unhelpful categories dominate both types of cases, especially prevalent in incorrect outcomes.
In this paper, we introduced BEEP, a system that automatically retrieves patient-specific literature based on intensive care (ICU) EHR notes and uses the literature to enhance clinical outcome prediction. On three challenging tasks, we obtain substantial improvements over strong recent baselines, seeing dramatic gains in top-  $10\%$  precision for mortality prediction with a boost of over  $25\%$ .
Our hope is that this work will open new research directions into bridging the gap between AI-based clinical models and the Evidence Based Medicine (EBM) paradigm in which medical decisions are based on explicit evidence from the literature. An interesting direction is to incorporate evidence identification and inference (Wallace, 2019; DeYoung et al., 2020) directly into our retrieval and predictive models. Another important question to explore relates to the implications our approach has on increasing the interpretability of clinical AI models.
This work was supported in part by NSF Convergence Accelerator Award #2132318. The authors would like to thank the members of the Semantic Scholar team, and the anonymous reviewers for their helpful feedback on this work.
Incorporating outcome prediction models into a medical decision-making pipeline effectively will require these technologies to adhere to standards set by the core principles of medical ethics: beneficence, non-maleficence, autonomy, and justice (Beauchamp et al., 2001). These requirements may raise the following concerns when deploying outcome prediction models in clinical settings:
- Out-of-Cohort Generalization: The extent to which outcome prediction models generalize to patient cohorts that may not have been present in their training data is unclear. If model accuracy is significantly lower on "out-of-cohort" patients, using inaccurate/uncertain predictions during decision making may violate the requirement that any application of technology must be beneficial and non-maleficent to individual patients. Our proposed technique can partly mitigate the generalization issue by identifying additional supporting evidence from literature, which may be better tailored to individual patient char
acteristics, instead of using only cohort-level evidence. However, biomedical literature can also have blind spots, with certain cohorts and disease combinations being under-studied, and even literature-augmented prediction may not be sufficiently accurate.
- Algorithmic Biases: Since outcome prediction models are trained on historical health data, existing inequities in healthcare access may translate into models continuing to perpetuate unintentional discrimination against patients from under-served demographics. For example, models might predict poorer outcomes (e.g., high mortality, poor response to treatment, etc.) for specific demographics that have historically had worse outcomes due to poor access to care. Such issues are a clear violation of the justice requirement, and must be tackled before deployment.
- Informed Consent: Lastly, if outcome prediction models are used in clinical settings, patients and their caregivers must be made aware of their use, since the principle of autonomy emphasizes that patients must be provided all relevant medical information to support autonomous decision making. The black-box nature of these models raises another issue: how can we help patients/caregivers understand and interpret outcome predictions to further support their autonomy in decision making? We hope that literature-augmented prediction techniques can partly ease this by using evidence snippets from literature that contributed to the model's prediction as explanations.
Emily Alsentzer, John Murphy, William Boag, WeiHung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Tom L Beauchamp, James F Childress, et al. 2001. Principles of biomedical ethics. Oxford University Press, USA.
Willie Boag, Dustin Doss, Tristan Naumann, and Peter Szolovits. 2018. What's in a note? unpacking predictive value in clinical note representations. AMIA Summits on Translational Science Proceedings, 2018:26.
Wendy W Chapman, Will Bridewell, Paul Hanbury, Gregory F Cooper, and Bruce G Buchanan. 2001.
Evaluation of negation phrases in narrative clinical reports. In Proceedings of the AMIA Symposium, page 105. American Medical Informatics Association.
Van Dang, Michael Bendersky, and W Bruce Croft. 2013. Two-stage learning to rank for information retrieval. In European Conference on Information Retrieval, pages 423-434. Springer.
Manirupa Das, Juanxi Li, Eric Fosler-Lussier, Simon Lin, Steve Rust, Yungui Huang, and Rajiv Ramnath. 2020. Sequence-to-set semantic tagging for complex query reformulation and automated text categorization in biomedical IR using self-attention. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pages 14–27, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Wang. 2021. MS $^{2}$ : Multi-document summarization of medical studies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Jay DeYoung, Eric Lehman, Benjamin Nye, Iain Marshall, and Byron C Wallace. 2020. Evidence inference 2.0: More data, better models. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pages 123-132.
Hannah Eyre, Alec B Chapman, Kelly S Peterson, Jianlin Shi, Patrick R Alba, Makoto M Jones, Tamara L Box, Scott L DuVall, and Olga V Patterson. 2021. Launching into clinical space with medspacy: a new clinical text processing toolkit in python. In AMIA Annual Symposium Proceedings 2021.
Jinyue Feng, Chantal Shaib, and Frank Rudzicz. 2020. Explainable clinical decision support from text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1478-1489, Online. Association for Computational Linguistics.
Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2021. Complement lexical retrieval model with semantic residual embeddings. In European Conference on Information Retrieval, pages 146-160. Springer.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domainspecific language model pretraining for biomedical
natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1-23.
Henk Harkema, John N Dowling, Tyler Thornblade, and Wendy W Chapman. 2009. Context: an algorithm for determining negation, experiencer, and temporal status from clinical reports. Journal of biomedical informatics, 42(5):839-851.
Mohammad Hashir and Rapinder Sawhney. 2020. Towards unstructured mortality prediction with freetext clinical notes. Journal of Biomedical Informatics, 108:103489.
Kexin Huang, Abhishek Singh, Sitong Chen, Edward Moseley, Chih-Ying Deng, Naomi George, and Charolotta Lindvall. 2020. Clinical XLNet: Modeling sequential clinical notes and predicting prolonged mechanical ventilation. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 94-100, Online. Association for Computational Linguistics.
Sarthak Jain, Ramin Mohammadi, and Byron C Wallace. 2019. An analysis of attention over clinical notes for predictive tasks. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 15-21.
Mengqi Jin, Mohammad Taha Bahadori, Aaron Colak, Parminder Bhatia, Busra Celikkaya, Ram Bhakta, Selvan Senthivel, Mohammed Khalilia, Daniel Navarro, Borui Zhang, et al. 2018. Improving hospital mortality prediction with medical named entities and multimodal learning. arXiv preprint arXiv:1811.12276.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimiciii, a freely accessible critical care database. Scientific data, 3(1):1-9.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C Wallace. 2019. Inferring which medical treatments work from reports of clinical trials. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3705-3717.
Matthew D Li, Nishanth Thumbavanam Arun, Mishka Gidwani, Ken Chang, Francis Deng, Brent P Little, Dexter P Mendoza, Min Lang, Susanna I Lee, Aileen O'Shea, et al. 2020. Automated assessment and tracking of Covid-19 pulmonary disease severity on chest radiographs using convolutional siamese neural networks. Radiology: Artificial Intelligence, 2(4):e200079.
George Michalopoulos, Yuanxin Wang, Hussam Kaka, Helen Chen, and Alexander Wong. 2021. Umlsbert: Clinical domain knowledge augmentation of contextual embeddings using the Unified Medical Language System Metathesaurus. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1744-1753, Online. Association for Computational Linguistics.
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 319-327, Florence, Italy. Association for Computational Linguistics.
Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 58-65.
Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, et al. 2018. Scalable and accurate deep learning with electronic health records. NPJ Digital Medicine, 1(1):1-10.
Kirk Roberts, Dina Demner-Fushman, Ellen M. Voorhees, and William R. Hersh. 2016. Overview of the TREC 2016 clinical decision support track. In Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016, Gaithersburg, Maryland, USA, November 15-18, 2016, volume 500-321 of NIST Special Publication. National Institute of Standards and Technology (NIST).
Kirk Roberts, Matthew S Simpson, Ellen M Voorhees, and William R Hersh. 2015. Overview of the trec 2015 clinical decision support track. In TREC.
David L Sackett, William MC Rosenberg, JA Muir Gray, R Brian Haynes, and W Scott Richardson. 1996. Evidence based medicine: what it is and what it isn't.
Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute. 2010. Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications. Journal of the American Medical Informatics Association, 17(5):507-513.