SlowGuess/NAACL_2022_OCR huggingface dataset:
Peri L Schuyler, William T Hole, Mark S Tuttle, and David D Sherertz. 1993. The umls metathe-saurus: representing different views of biomedical concepts. Bulletin of the Medical Library Association, 81(2):217.
Sue Sendelbach and Marjorie Funk. 2013. Alarm fatigue: a patient safety concern. AACN advanced critical care, 24(4):378-386.
Matthew S Simpson, Ellen M Voorhees, and William Hersh. 2014. Overview of the trec 2014 clinical decision support track. Technical report, LISTER HILL NATIONAL CENTER FOR BIOMEDICAL COMMUNICATIONS BETHESDA MD.
Alberto Ueda, Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2021. Structured fine-tuning of contextual embeddings for effective biomedical retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 2031-2035, New York, NY, USA. Association for Computing Machinery.
Ozlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552-556.
Betty van Aken, Jens-Michalis Papaioannou, Manuel Mayrdorfer, Klemens Budde, Felix Gers, and Alexander Loeser. 2021. Clinical outcome prediction from admission notes using self-supervised knowledge integration. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 881-893, Online. Association for Computational Linguistics.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing a pandemic information retrieval test collection. In ACM SIGIR Forum, volume 54, pages 1-12. ACM New York, NY, USA.
Byron C Wallace. 2019. What does the evidence say? models to help make sense of the biomedical literature. In *IJCAI: proceedings of the conference*, volume 2019, page 6416. NIH Public Access.
Yu Wang, Jinchao Li, Tristan Naumann, Chenyan Xiong, Hao Cheng, Robert Tinn, Cliff Wong, Naoto Usuyama, Richard Rogahn, Zhihong Shen, et al. 2021. Domain-specific pretraining for vertical search: Case study on biomedical literature. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3717-3725.
A PubMed Queries Per Outcome
Following are the MeSH terms that we use to retrieve literature from PubMed to construct the
outcome-specific index for each clinical outcome under consideration:
- Prolonged Mechanical Ventilation (PMV): "Respiration, Artificial". We also query using the terms "Ventilation, Mechanical" and "Ventilator Weaning" but do not find any new results.
- In-Hospital Mortality (MOR): "Hospital Mortality", "Mortality+Humans+Risk Factors". Note that the "+" operator is interpreted as AND by PubMed search.
- Length of Stay (LOS): "Length of Stay". All other MeSH terms from the tagger are aliases of this term.
Entity Extraction. First, we extract entities from clinical notes using a model trained on the i2b2 2010 concept extraction dataset (Uzuner et al., 2011). This dataset consists of clinical notes annotated with three types of entities: problems, tests, and treatments. These entity types cover the pertinent medical information that can be used to retrieve abstracts relevant to a clinical note. Moreover, the i2b2 guidelines require annotators to include all qualifiers within an entity span, so training a model on these annotations should bias it towards including pertinent entity qualifiers. Our entity extraction model uses a BERT-based language model to compute token representations, followed by a linear layer to predict entity labels.
We use ClinicalBERT (Alsentzer et al., 2019) as the the language model to train our i2b2 entity extractor. Table 6 shows the performance of our model on the i2b2 2010 test set. These numbers are close to the exact F1 scores reported by Alsentzer et al. (2019) on i2b2 2010 (87.8).
Entity Filtering. After extracting entities, we filter out all negated entities. Negated entities are detected using the ConText algorithm for negation detection from clinical text (Harkema et al., 2009). We use the implementation of ConText negated entity detection algorithm provided by medspaCy (Eyre et al., 2021).
MeSH Linking. Finally, the set of filtered entities is linked to MeSH terms using scispaCy. Entities not linked to MeSH terms are discarded. MeSH terms linked in clinical notes and abstracts are used
to compute TF-IDF vectors for the sparse retrieval model.
Bi-Encoder Given a query clinical note  $Q$  and an abstract document  $D_{i}$ , a BERT-based encoder is used to compute dense embedding representations  $E_{Q}$  and  $E_{D_{i}}$ . A scoring function  $S$  is defined as the Euclidean distance between query and document embeddings:
$$
S (Q, D _ {i}) = \left\| E _ {Q} - E _ {D _ {i}} \right\| _ {2} \tag {5}
$$
Documents closest to the query vector in the embedding space are returned as top-ranked results. The bi-encoder is trained using a triplet loss function defined as follows:
$$
\begin{array}{l} L \left(Q, D _ {i} ^ {+}, D _ {i} ^ {-}\right) = \\ \max  \left(S \left(Q, D _ {i} ^ {+}\right) - S \left(Q, D _ {i} ^ {-}\right) + m, 0\right) \tag {6} \\ \end{array}
$$
Here  $D_{i}^{+}$  is an abstract more relevant to the clinical note  $Q$  than  $D_{i}^{-}$  and  $m$  is a margin value. We use PubmedBERT (Gu et al., 2021) as the encoder and train our bi-encoder using the dataset from the TREC 2016 clinical decision support task (Roberts et al., 2016). This dataset consists of 30 de-identified EHR notes, along with  $\sim 1000$  PubMed abstracts per note marked for relevance. We select relevant abstracts per note as positive candidates  $(D_{i}^{+})$ , and irrelevant abstracts for the same note as negative candidates  $(D_{i}^{-})$ .
Outcome prediction module training. We use a weighted cross-entropy loss function to handle class imbalance. Given a dataset with  $N$  total examples,  $c$  classes and  $n_i$  examples in class  $i$ , class weights are computed as follows:
$$
w _ {i} = \frac {N}{c \cdot n _ {i}} \tag {7}
$$
We use Adam optimizer, treating initial learning rate as a hyperparameter. All models are implemented in PyTorch, and we use Huggingface implementations for all pretrained language models.
C Hyperparameter Tuning
We do a grid search over the following hyperparameter values for each aggregation:
Learning Rate (LR): [5e-4, 1e-5, 5e-5, 1e-6, 5e-6] Number of top abstracts (k): [1, 5, 10]
<sup>9</sup>We do not use data from TREC 2014 and 2015 since they use idealized case reports instead of actual EHR notes. Combining all three datasets degraded performance, likely due to differences in language between case reports and EHRs.
LM	PMV	MOR	LOS
AUROC	Micro F1	AUROC	Micro F1	AUROC	Micro F1
ClinicalBERT (Full)	54.66	53.93	81.78	86.34	70.94	40.00
ClinicalBERT (Disc.)	54.91	54.21	81.78	86.34	71.44	40.36
CORE	54.98	54.35	81.58	84.85	69.15	37.94
BLUEBERT	56.60	55.34	82.40	84.75	71.87	41.93
UMLSBERT	57.42	55.48	83.31	87.29	71.60	41.84
Table 5: Performance of various language models trained on clinical and biomedical text on all clinical outcomes. For ClinicalBERT, Disc. and Full refer respectively to variants trained on discharge summaries only and both discharge summaries and nursing notes.
Category	Exact F1
Overall	86.66
Test	87.48
Problem	86.53
Treatment	86.03
Table 6: Entity extraction model performance on i2b2 2010 test set
Gradient accumulation steps (GA): [10, 20] This hyperparameter grid stays consistent across all outcome prediction experiments. For all experiments, we currently report the outcome of a single run.
D Computing Infrastructure
Our experiments were carried out on 2 AWS p3.16xlarge instances, which are 8-GPU machines with 16 GB RAM per GPU. All our experiments can be run on a single 16 GB GPU.
E Results from Learning To Retrieve Model
Given a note  $Q$ , we first obtain a set of top 100 relevant abstracts  $(Docs(Q) = \{D_1, \dots, D_{100}\})$  from the BEEP retrieve-erank pipeline. The retriever component is then defined as follows:
$$
E _ {Q} = B E R T _ {Q} (Q) \tag {8}
$$
$$
E _ {D _ {i}} = B E R T _ {D} \left(D _ {i}\right) \tag {9}
$$
$$
S _ {r e t r} (Q, D _ {i}) = \operatorname {c o s i n e} \left(E _ {Q}, E _ {D _ {i}}\right) \tag {10}
$$
$BERT_{Q}(X)$  and  $BERT_{D}(X)$  are the query and document encoder functions. Based on retriever scores  $S_{\text{retr}}$ , we select the top  $k$  abstracts and perform outcome prediction using the same structure as the BEEP outcome prediction module. We also add the following early update loss term to the
outcome loss for the retriever component:
$$
P _ {\text {e a r l y}} (D _ {i} | Q) = \frac {\exp \left(S _ {\text {r e t r}} (Q , D _ {i})\right)}{\sum_ {D _ {j} \in D o c s (Q)} \exp \left(S _ {\text {r e t r}} (Q , D _ {j})\right)} \tag {11}
$$
$$
L _ {\text {e a r l y}} = - \log \sum_ {D _ {j} \in D o c s (Q)} y _ {j} P _ {\text {e a r l y}} \left(D _ {j} \mid Q\right) \tag {12}
$$
where  $y_{j}$  is set to 1 if using document  $D_{j}$  alongside  $Q$  results in a confidence increase in the correct outcome (as per BEEP) and 0 otherwise. Our L2R model does not improve performance over BEEP (results in Table 7). We speculate that this may partly be due to the fact that the heuristic we use to assign  $y_{j}$  values in early update loss is not as accurate as the one used by Lee et al. (2019) (directly checking for presence of the answer in a document, for the reading comprehension task).
Table 7 presents results for the learning-to-retrieve model on all clinical outcomes using UMLSBERT as the encoder. From the table, we can see that while L2R improves performance over a notes-only baseline, its performance is comparable to BEEP. As mentioned earlier, we speculate that this may partly be attributed to the fact that the heuristic we use to assign  $y_{j}$  values in early update loss is not as accurate as the one used by Lee et al. (2019) (directly checking for presence of answer in document, for the reading comprehension task). We believe that experimenting with other sources of supervision to generate  $y_{j}$  values and weighting mechanisms to better combine outcome and early update losses might lead to larger improvements, but we leave those to future work.
F Literature-Augmented Outcome Prediction with CORe
Table 8 shows the overall performance of our literature-augmented outcome prediction system
Model	PMV	MOR	LOS
AUROC	Micro F1	Macro F1	AUROC	Micro F1	Macro F1	AUROC	Micro F1	Macro F1
UMLSBERT	56.44	56.07	54.97	83.34	87.93	66.93	72.19	43.12	42.20
+Avg	54.17	53.53	41.51	84.54	90.47	60.53	71.90	44.88	41.26
+SVote	54.29	52.82	39.93	84.50	90.51	61.10	72.17	45.56	41.68
+WVote	57.60	56.50	55.93	83.92	90.54	61.20	72.72	46.46	42.17
+WAvg	58.65	55.79	53.68	84.68	90.59	62.78	72.16	45.04	40.87
Table 7: Performance of learning to retrieve (L2R) model on all clinical outcomes using the UMLSBERT language model
Model	PMV	MOR	LOS
AUROC	Micro F1	Macro F1	AUROC	Micro F1	Macro F1	AUROC	Micro F1	Macro F1
CORe	55.91	53.96	53.71	79.96	78.92	62.46	71.52	42.59	42.33
+Avg	58.76	55.51	55.43	82.41	84.67	66.06	71.99	40.54	40.39
+SVote	58.40	58.62	55.23	81.90	89.90	55.76	71.35	45.07	40.16
+WVote	58.03	56.92	53.14	82.81	89.87	53.16	70.96	44.74	39.73
+WAvg	57.53	55.51	55.49	81.98	81.86	64.63	71.17	39.48	39.67
Table 8: Performance of baseline and literature-augmented outcome prediction models on all clinical outcomes using the CORe language model
Model	PMV	MOR	LOS
AUROC	Micro F1	Macro F1	AUROC	Micro F1	Macro F1	AUROC	Micro F1	Macro F1
BLUEBERT	-	-	-	-	-	-	-	-	-
+Avg	55.72	54.38	46.95	68.72	89.49	47.23	63.40	39.40	29.15
+SVote	57.11	56.50	52.21	71.04	89.49	48.73	63.46	39.41	28.90
+WVote	55.83	53.25	43.43	71.00	89.50	48.73	63.40	39.56	27.52
+WAvg	56.99	55.65	47.97	71.39	89.48	49.26	63.46	39.34	27.99
UMLSBERT	-	-	-	-	-	-	-	-	-
+Avg	59.15	55.37	50.79	71.22	89.49	48.54	63.84	39.49	30.30
+SVote	56.53	55.09	51.76	69.31	89.50	47.71	63.14	38.95	27.12
+WVote	57.06	54.38	53.77	70.54	89.46	49.34	63.46	39.40	27.55
+WAvg	56.99	54.94	54.29	70.04	89.46	49.16	63.51	39.51	28.32
Table 9: Performance of models that only use retrieved literature for outcome prediction on all clinical outcomes
Model	<3 days	>=3 and <=7 days	>7 and <=14 days	>14 days
Prec@10	Rec@10	Prec@10	Rec@10	Prec@10	Rec@10	Prec@10	Rec@10
BLUEBERT	47.6	37.11	61.09	16.14	44.98	14.15	50.74	26.93
+Avg	54.23	24.0	60.64	16.02	45.45	14.49	49.48	25.66
+SVote	54.48	27.12	62.12	16.41	46.38	14.97	51.33	26.87
+WVote	55.73	21.68	61.66	16.29	46.68	12.78	47.99	25.18
+WAvg	52.48	28.28	60.75	16.05	47.33	15.12	51.03	26.99
UMLSBERT	47.33	37.11	59.95	15.84	44.83	13.04	48.92	25.97
+Avg	53.08	26.14	60.41	15.96	48.3	15.27	49.6	26.03
+SVote	52.37	28.55	59.5	15.72	44.38	14.38	49.36	25.72
+WVote	57.22	27.21	64.28	16.98	45.43	14.78	50.4	26.33
+WAvg	52.86	20.61	59.84	15.81	44.9	14.38	48.44	25.24
Table 10: Precision and recall scores for top  ${10}\%$  high-confidence predictions per class (precision/recall@TOP-K) for LOS.
Model	No PMV	PMV
BLUEBERT+Avg	55.47	57.48
BLUEBERT+SVote	56.82	55.56
BLUEBERT+WVote	62.50	62.67
BLUEBERT+WAvg	56.34	61.29
UMLSBERT+Avg	63.71	60.71
UMLSBERT+SVote	50.39	65.62
UMLSBERT+WVote	61.83	59.09
UMLSBERT+WAvg	57.80	63.33
(a) Precision on PMV, when considering cases for which literature-augmented models achieve  $>10\%$  increase in prediction confidence over baseline.
Model	No MOR	MOR
BLUEBERT+Avg	87.91	69.77
BLUEBERT+SVote	87.49	75.00
BLUEBERT+WVote	86.99	76.09
BLUEBERT+WAvg	87.29	77.68
UMLSBERT+Avg	85.33	83.33
UMLSBERT+SVote	90.33	31.01
UMLSBERT+WVote	86.66	52.17
UMLSBERT+WAvg	85.29	60.00
(b) Precision on MOR, when considering cases for which literature-augmented models achieve  $>10\%$  increase in prediction confidence over baseline.
on all three clinical outcomes when the CORe language model is used as an encoder. From this table, we can see that adding literature improves performance in this setting as well (with the exception of macro F1 on length of stay). However the overall scores are lower than the settings in which UMLSBERT and BLUEBERT are used as encoders (Table 2).
G Literature-Only Outcome Prediction
To quantitatively test the quality of the retrieved literature, we run an ablation study in which we predict the clinical outcome using only the literature retrieved for a specific patient case, without incorporating any information from the patient clinical note. Table 9 shows the results for this ablation study, using both BLUEBERT and UMLSBERT encoders. From this table, we can see that while removing the clinical note leads to performance drops, especially on mortality and length of stay, the retrieved literature does have some predictive ability. We take this as indication that the retrieved literature contains some clinical indicators associated with the outcome, that are also present in the patient's clinical note.
H Analyzing High Confidence Increases Over Baseline
Finally, we also examine an alternate way of using high-confidence predictions made by our mod
els. We run both baseline and literature-augmented systems, and only consider predictions from the literature-augmented system that show a high increase in confidence, such as  $>10\%$  increase relative to the baseline predictions for the same cases. Tables 11a and 11b show the precision scores of all models on prolonged mechanical ventilation and mortality in this setting. We can see that precision scores in this setting are fairly high, especially for the negative class in mortality prediction. Most averaging variants also do well on the positive class in mortality prediction.
I Examples of Literature For Incorrect Outcome Cases
We categorize examples into the following:
1. Patient condition and outcome directly related
2. Patient history and outcome related
3. Known outcome indicators not present in patient
4. Ongoing treatment and outcome related
5. No cohort match
6. No/weak condition match
7. Condition-outcome pair not studied
8. No evidence for outcome/Weak evidence for direct relationship between patient condition and outcome
From table 12, we can see that retrieved literature from unhelpful categories often does not match patient characteristics. The first case discusses a patient who has had an ICD firing incident, but the retrieved literature discusses ICD implantation therapy. While related, there is no discussion of the impact of ICD firing on various clinical outcomes.
For the second case, we see that the retrieved article discusses strokes in general, without matching any of the patient's indications or demographic characteristics. Moreover, the outcome of interest (mortality) is mentioned briefly, but links between the outcome and patient conditions are not studied. Finally, the third case provides an example of a common phenomenon we observe. There are a fair number of review articles retrieved that do not have strong evidential statements in the abstract. For the third case, the retrieved abstract discusses the need for early triage/transfer (which could lead to low length of stay), but then do not provide any conclusive evidence.
Patient EHR	Retrieved Abstract	Evidence Type	Outcome
CHIEF COMPLAINT: ICD firing
PRESENT ILLNESS: 57 yo M pre-
senting s/p ICD discharges...shocks pre-
ceded by prodrome of dizziness,...and
was shocked once...Has not had ICD fir-
ing prior to these events since implant
MEDICAL HISTORY: Heart failure...	...assess if selected clinical markers
of organ dysfunction were associ-
ated with increased 1-year mortality
despite ICD therapy...Clinical mark-
ers of liver dysfunction, recent me-
chanical ventilation, and renal im-
pairment were independently asso-
ciated with increased 1 year morta-
lity...	Weak condition match,
condition-outcome pair
not studied	PMV
CHIEF COMPLAINT: acute onset
right hemiplegia and aphasia
PRESENT ILLNESS: 84yo M...acute
onset of inability to speak and right
hemiplegia...head CT showed dense L
MCA and hypodensities in left inferior
frontal lobe and left corona radiata.
MEDICAL HISTORY: HTN Afib, off
coumadin...	Stroke is indicated by an abrupt
manifestation of neurologic deficits
secondary to an ischemic or hem-
orrhagic insult to a region of the
brain...ranked as the third lead-
ing cause of death in the United
States...report shows that despite
the use of antithrombotic and/or an-
tiplatelet aggregating drugs, the key
to stroke management is primary
prevention.	No cohort match,
condition-outcome pair
not studied	MOR
CHIEF COMPLAINT: Substernal
chest pain
PRESENT ILLNESS: ...62 yo M...
no prior cardiac history... substernal
CP... mild SOB, nausea, diaphoresis and
numbness in left arm...
MEDICAL HISTORY: foot surgery 2
weeks ago ?COPD ?gastritis?	...rising health care costs have cre-
ated pressures to increase efficiency
of coronary care units. Possible
strategies seek to decrease resource
use by identifying low-risk patients
for initial triage or early transfer to
lower levels of care...	No cohort match, no evi-
dence for outcome	LOS <3
days
Table 12: Qualitative examples of retrieved literature that is categorized as unhelpful for cases where adding literature increases confidence in incorrect outcome. Case 1 shows an example of retrieved literature that has a weak match with patient condition, but no evidence linking condition to outcome. Case 2 shows an example in which retrieved literature does not match patient case or contain evidence for outcome. Case 3 shows an example of a review article that again does not match patient case or provide outcome evidence.
(a) PMV
(b) MOR
Figure 4: Proportion of admission notes associated with the 100 most highly retrieved abstracts for each clinical outcome. From these graphs, we can see that frequently-retrieved abstracts for LOS are associated with a larger proportion of cases from the dataset, than frequently retrieved abstracts for PMV and MOR (indicative of lower literature diversity in LOS).
(c) LOS