SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Emma Chen $^{1,4*}$  Aman Kansal $^{2*}$  Julie Chen $^{2}$  Boyang Tom Jin $^{2}$  Julia Rachel Reisler $^{2}$  David A Kim $^{3\dagger}$  Pranav Rajpurkar $^{1\dagger}$
$^{1}$ Department of Biomedical Informatics, Harvard Medical School
$^{2}$ Department of Computer Science, Stanford University
$^{3}$ Department of Emergency Medicine, Stanford University School of Medicine
<sup>4</sup>Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University yingchen@g.harvard.edu
{amkansal, jchen80, tomjin, jreisler, davidak}@stanford.edu
We propose the Multimodal Clinical Benchmark for Emergency Care (MC-BEC), a comprehensive benchmark for evaluating foundation models in Emergency Medicine using a dataset of  $100\mathrm{K}+$  continuously monitored Emergency Department visits from 2020-2022. MC-BEC focuses on clinically relevant prediction tasks at timescales from minutes to days, including predicting patient decompensation, disposition, and emergency department (ED) revisit, and includes a standardized evaluation framework with train-test splits and evaluation metrics. The multimodal dataset includes a wide range of detailed clinical data, including triage information, prior diagnoses and medications, continuously measured vital signs, electrocardiogram and photoplethysmograph waveforms, orders placed and medications administered throughout the visit, free-text reports of imaging studies, and information on ED diagnosis, disposition, and subsequent revisits. We provide performance baselines for each prediction task to enable the evaluation of multimodal, multitask models. We believe that MC-BEC will encourage researchers to develop more effective, generalizable, and accessible foundation models for multimodal clinical data.
Emergency Medicine is a critical area of healthcare in which timely and accurate decisions, drawing appropriately on a wide variety of data sources, have a significant impact on patient outcomes [1]. However, developing effective foundation models for electronic health record (EHR) data in Emergency Medicine requires addressing several challenges. EHR data in Emergency Medicine is heterogeneous, including clinical notes, orders, lab results, imaging studies, and physiological waveforms. This heterogeneity can make it difficult to extract meaningful features from the data and integrate them into a single model. Data quality and missingness can also be a significant issue, due to the fast-paced and high-pressure nature of emergency care. Inaccurate or incomplete data can limit the reliability of model results. Clinical interpretability is also critical. Clinicians must be able to understand the model's predictions to make informed decisions. Therefore, models developed for EHR analysis in Emergency Medicine must be transparent and explainable. Finally, the limited scope and standardization of existing datasets are significant challenges to developing foundation models for Emergency Medicine. Many datasets focus on specific patient populations, such as trauma
arXiv:2311.04937v1 [cs.LG] 7 Nov 2023
patients, or specific tasks, such as predicting mortality. This orientation to specific subgroups and tasks makes it difficult to compare and evaluate models across tasks and patient populations.
To address these challenges, and to promote the development of robust and clinically useful foundation models for Emergency Medicine, we propose the Multimodal Clinical Benchmark for Emergency Care (MC-BEC), a comprehensive benchmark for evaluating foundation models in Emergency Medicine. MC-BEC is built on a dataset  $^{1}$  of 102,731 monitored visits made by 63,389 unique patients between 2020 and 2022, and provides a unique opportunity to study acute care in the COVID-19 era. It is the only multimodal medical dataset that exclusively covers patients during this period, while also capturing a wide range of non-COVID pathology. This dataset covers a wide range of information for emergency department (ED) patients, including triage information, prior diagnoses and medications, orders placed in the ED, medication administrations, lab results, continuously monitored vital signs and physiologic waveforms, and free-text reports for radiology studies. With its emphasis on multiple modalities, including continuous waveforms and vital signs providing physiologic context for heterogeneous and often rapidly evolving patients, MC-BEC presents opportunities to study the uniquely dynamic and complex nature of emergency care.
MC-BEC emphasizes clinically relevant downstream tasks at multiple timescales, specifically predicting patient decompensation (within minutes), disposition (within hours), and ED revisit (within days), and provides a standardized evaluation framework with train-test splits and evaluation metrics. We also provide baselines for each task to enable model comparison and evaluation. With MC-BEC, we hope to encourage researchers and clinicians to develop more effective, generalizable, and accessible foundation models for EHR analysis in Emergency Medicine, ultimately improving patient outcomes and advancing the analysis of real-world EHR data.
2.1 Current ED benchmarks are not multimodal
Existing EHR datasets for ED or critically ill patients are often limited to structured EHR data and intermittent vital sign recordings. These datasets fail to capture the comprehensive multimodal information obtained from the intensive evaluation and monitoring of ED patients. To our knowledge, only two ED-specific benchmarks exist. Xie et al. (2022) [2] introduced an ED benchmark using the MIMIC-IV-ED dataset [3]. While this dataset represents the only publicly available general-purpose ED dataset, it contains only tabular EHR data for all patients, with radiology reports for a subset. EHRShot [4] is the other recent ED benchmark, but its underlying dataset includes only coded data such as ICD diagnosis codes, and is not publicly available.
Due to the lack of robust ED benchmarks, we also reviewed existing critical care benchmarks, since patient monitoring practices in the ED and intensive care unit (ICU) are similar. A comparison in Table 1 shows most ICU datasets also focus on structured EHR data and intermittent vital signs, lacking free text or waveforms. HiRID[5] provides high-resolution physiological data but no text/reports. Strikingly, neither ICU nor ED datasets include electrocardiogram (ECG) and photoplethysmograph (PPG) waveforms, which represent essential data on the physiology of critically ill patients.
To address the lack of multimodal ED benchmarks, we propose MC-BEC, using a novel multimodal dataset including vital signs, continuous physiologic waveforms (ECG, PPG, respiration), radiology reports, and diverse structured EHR data. This supports improved evaluation of model performance on a wider range of salient patient information.
2.2 Current ED benchmarks are not suitable for generalist medical AI
Existing medical AI benchmarks fall short in comprehensively evaluating generalist medical AI (GMAI) and clinical foundation models. GMAI was recently proposed as a goal for foundation models in healthcare and medicine [6]. GMAI would perform a wide range of medical tasks and flexibly interpret different combinations of data modalities, enabled by foundation models' capability to learn broadly useful data representations from massive pretraining [7]. However, as discussed in a recent survey [8], current benchmarks focus narrowly on accuracy for predefined tasks, inadequate for evaluating key GMAI capabilities. A major limitation is the lack of assessment for multimodal
$^{1}$ Multimodal Clinical Monitoring in the Emergency Department (MC-MED-v0)
integration (Table 1). While GMAI is designed to integrate diverse data modalities, performance can decline with more modalities [9]. Yet accuracy metrics alone will not expose these nuances. Current benchmarks also rarely evaluate how models handle missing modalities, though incomplete data is pervasive in real-world EHRs.
To address these gaps, benchmarks must move beyond accuracy to rigorously assess multimodal performance, tolerance to missing data, and other facets critical for GMAI. The proposed MC-BEC benchmark exemplifies this comprehensive approach through evaluating multitask learning, multimodal performance, and fairness analyses beyond predictive accuracy alone. Rethinking evaluation is imperative as GMAI diverges from narrow benchmarks of the past toward more expansive and integrative capabilities. Robust benchmarks will be central to steering progress in this promising new direction.
Table 1: Comparison of critical and emergency care EHR benchmarks. MT Learning stands for multitask learning; MM Analysis stands for multimodal analysis; the number of patients and visits of MIMIC-III excludes neonatal patients.
Benchmark	Eval. Beyond Acc.	Source	Data Modalities
MT Learning	MM Analysis	Fairness	Dataset	ICU ED	Num. patients	Num. visits	EHR Codes	Free-Text Notes	Vital Signs	ECG/PPG Waveform
MIMIC-Extract[10]	MIMIC-III[11]	x	39K*	53K*	x	x	x
Purushotham 2018[12]
Harutyunyan 2019[13]	x
Xie 2022[2]	MIMIC-IV-ED (v1.0)[14]	x	217K	449K	x	x	x
Gupta 2022[15]	x	MIMIC-IV (v1.0) [16, 17]	x	x	257K	524k	x	x	x
eICU[18]	eICU[19]	x	139K	201K	x	x
EHR PT[20]	x	MIMIC-III[11] / eICU[18]
HiRID-ICU[21]	HiRID (v1.1.1)[5]	x	34K	56K	x	x
EHRSHOT[4]	Stanford Med.	x	x	7k	894k	x
MC-BEC	x	x	x	Stanford EM	x	63K	102K	x	x	x	x
2.3 GMAI for multimodal EHR data
Our baseline model is inspired by related work in the field. A common approach to foundation models is to first pretrain them on tasks with abundant data to generate robust representations, then to fine-tune on downstream tasks. ClinicalBERT [22] is a widely used pretrained model specifically designed for clinical text embeddings. Categorical EHR data also has pretraining options like BEHRT[23], CLMBR[24] and MOTOR [25], which predict future EHR codes based on past codes. However, these representations of EHR codes depend on the code formats used during training, limiting transferability and robustness to heterogeneous EHR formats. Instead, we employed CodeEmb's [26] approach that uses pretrained text embeddings associated with EHR code descriptions regardless of the format or code system used by the EHR. For fine-tuning on multimodal data, we adopt the HAIM framework [27], using a modular ML pipeline that integrates modality-specific pretrained embeddings.
2.4 Multitask learning
Our multitask training approach draws inspiration from prompting techniques in language models. Rather than using task-specific prediction heads, we employ a unified task representation combined with task-specific queries. This allows creating broadly capable models without architectural changes for new tasks.
Conventional multitask learning often designates separate prediction heads for each task, as in MOTOR which has heads for distinct medical predictions[25]. However, the promising avenue of using unified task representations combined with task-specific queries has gained traction, as showcased in studies like OFS [28] and Perceiver IO [29]. We enhance our baseline LightGBM model [30] by incorporating contextual embeddings from the BERT model. For each task, we concatenate BERT embeddings of the task-specific queries with other input features. This integrates BERT's rich semantic understanding, allowing the model to adeptly differentiate tasks within the shared
Figure 1: MC-BEC evaluates foundation model performance on predictions of ED patient decompensation, disposition, and revisit, using a unique multimodal dataset of 102,731 monitored ED visits.
embedding space. This provides flexibility to expand to new prediction tasks without architectural changes.
MC-BEC is a benchmark for evaluating clinical foundation models/GMAI on multimodal EHR data in the ED setting. It provides three clinically relevant prediction tasks spanning different stages of an ED visit: prediction of near-term decompensation early in the visit, prediction of disposition at the end of a visit, and prediction of revisit after the visit. MC-BEC includes evaluation metrics beyond prediction performance, capturing modality interaction, fairness, and consistency of predictions with respect to time and modalities.
3.1 Data source
The MC-BEC dataset consists of 102,731 ED visits made by 63,389 patients between September, 2020 and September, 2022. The dataset is IRB approved, with a waiver of informed consent for retrospective research on de-identified data. The de-identified data for each ED visit spans the entire visit from department arrival to departure, including triage after ED arrival, rooming and initiation of monitoring, and all interventions and results up to the point of departure from the ED. Patients in the dataset have multiple ED visits, but MC-BEC ensures no patient overlap (i.e., visits from the same patient) across the training, testing and validation cohorts. Unlike previous ED datasets, MC-BEC contains both categorical and unstructured clinical data. These modalities and data structures are described below.
Categorical data with single observation: chief complaint, triage acuity level, gender, race, ethnicity, means of arrival, disposition of most recent visit, disposition of current visit, and payor class.
Categorical data with repeated observations: current and previous ICD-10 diagnosis codes with timestamps and corresponding text descriptions; home medications and medications administered during visit with IDs, names, and timestamps of administration; orders placed during visit with IDs, names, and timestamps; lab results with IDs, names, result values, and timestamps.
Numeric data: age, hours since previous visit, and vital signs including heart rate, respiratory rate, blood pressure, oxygen saturation, and temperature.
Time-series data: continuously recorded vital signs (heart rate, respiratory rate, oxygen saturation) and secondary features (heart rate variability, pulse transit time, perfusion index) throughout the patient's visit. These measurements were averaged over 1-minute intervals.
Waveform data: electrocardiogram (ECG), photoplethysmogram (PPG) and respiratory waveforms.
Free-text data: radiology reports for imaging studies ordered during the ED visit.
3.2 Benchmark tasks
We chose decompensation, disposition, and revisit as prediction tasks for MC-BEC. These tasks have high clinical and operational relevance in the ED setting: early identification of patients at risk of decompensation allows healthcare providers to allocate appropriate resources; accurate disposition prediction helps optimize resource allocation, bed management, and patient flow within the hospital system; and understanding the probability of revisit allows healthcare providers to identify patients who may require closer follow-up or additional interventions to prevent complications or ensure proper continuity of care. These prediction tasks collectively encompass the entire timeframe of an ED visit, focusing on different aspects in the patient's journey through the ED encounter (Figure 1). We provide brief descriptions of each task as defined for our benchmark:
1. Decompassion: The goal of this binary classification task is to predict which patients are likely to experience clinical decompensation, defined as new onset of tachycardia (heart rate [HR] > 110), hypotension (mean arterial pressure [MAP] < 65mmHg), or hypoxia (oxygen saturation by pulse oximetry [SpO2] < 90%) in patients with initially normal vital signs. The task uses the first 15 minutes of data acquired after the patient is roomed (the assessment period) to predict decompensation in a 60, 90 or 120 minute-window (evaluation period) following the assessment period. Patients with abnormal vital signs at triage or at any point during the assessment period are excluded from this task, in order to prevent trivial predictions (e.g., the prediction of future hypotension in an already hypotensive patient).
2. Disposition: The objective is to predict the binary outcome of a patient's disposition from the ED: whether they will be discharged home or admitted to the hospital. This is a summative clinical decision that reflects the patient's overall clinical stability and risk as determined by the totality of evidence accrued during the visit. To avoid data leakage, any information that could reveal the disposition decision (such as a consult order to an admitting service) is excluded for this task.
3. Revisit: The goal is to predict whether a patient will revisit the ED within a 3, 7, or 14-day period after being discharged. ED revisits are a common quality metric, because return visits can sometimes suggest incomplete workup or inappropriate disposition (i.e., a patient sent home who should have been admitted). All data from a visit can be used to make the revisit prediction, and no patients or visits from the dataset need to be excluded for this task.
3.3 Evaluation framework
Prediction performance. We propose to evaluate prediction performance with area under the precision-recall curve (AUPRC) as a threshold-independent metric. AUPRC is a clinically meaningful metric because it reflects the model's performance in correctly identifying positive instances while minimizing false positives. In medical scenarios, correctly identifying positive cases is crucial, as it ensures accurate diagnosis or prediction of a particular condition. AURPC is also particularly important when dealing with unbalanced data, which is common among medical datasets, including ours.
Monotonicity in modalities. We propose to assess the performance change of a model when increasing the data modalities available for training. We hypothesize that a well-designed multimodal model should not perform worse when more data modalities are used for training. However, such monotonicity of performance in modalities is not always observed. For example, [9] found that a unimodal network performed better than the multimodal network obtained through joint training. A reduction in performance caused by the inclusion of more modalities may be attributed to modality competition during training, wherein the model learns to rely on only a subset of modalities to make predictions [31]. The evaluation of monotonicity in modalities can help guide the development of more effective and robust multimodal models.
We evaluate monotonicity in modalities  $m_{\text{modalities}}$  by calculating the concordance index  $C$  between the actual performance  $p_m(i)$  and the theoretical ground truth which would expect better performance  $y_m(i)$  for increasing numbers  $n$  of modalities  $m$ :
$$
y _ {m 1} \leq y _ {m 2} \leq \dots \leq y _ {m (n - 1)} \leq y _ {m (n)} \tag {1}
$$
$$
m _ {\text {m o d a l i t i e s}} = C = \frac {\sum_ {i = 1} ^ {n} \sum_ {j = i + 1} ^ {n} f \left(y _ {i} , y _ {j} , p _ {i} , p _ {j}\right)}{n (n - 1) / 2} \tag {2}
$$
$$
f \left(y _ {i}, y _ {j}, p _ {i}, p _ {j}\right) = \left\{ \begin{array}{l l} 1 & \text {i f} \left(y _ {i} <   y _ {j}\right) \cdot \left(p _ {i} <   p _ {j}\right) + \left(y _ {i} > y _ {j}\right) \cdot \left(p _ {i} > p _ {j}\right) \\ 0. 5 & p _ {i} = p _ {j} \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {3}
$$
Robustness against missing modalities. We evaluate the model's robustness and generalization capabilities when one or more modalities of data used in training are unavailable in testing. This metric encourages the development of models with more reliable performance in real-world applications where not all training modalities will be present in all cases.
Bias. We evaluate bias as a crucial metric to detect systematic errors or prejudices in the model's predictions or outputs. MC-BEC reflects the ED, where false negative prediction of adverse events can result in severe negative health outcomes. Incorrect disposition decisions might exacerbate health issues; misjudged revisit predictions can overlook preventive care needs; and unforeseen clinical decompensation, like hypotension or hypoxia, can be life-threatening. Hence, we chose the true positive rate (TPR) difference as our primary bias metric. Using a sensitivity threshold of 0.85 on the validation set, we assess TPR differences between different demographic groups such as age, gender, race, and ethnicity. Our approach for evaluating model fairness seeks to identify underdiagnosis bias, which is arguably most relevant and consequential in healthcare settings [32].
We developed and evaluated baseline models to demonstrate the use of MC-BEC. Our models used pretrained text and waveform embeddings and employed a modular approach for modality fusion. We trained multimodal representations using LightGBM [30], as gradient boosting ensembles are often highly performant in clinical tasks [33, 34, 35, 36]. We also proposed a novel multitask training schema with LightGBM and compared its performance against models trained on individual tasks. Our aim is to provide an example of MC-BEC evaluation, and a reference for future models evaluated with this benchmark.
4.1 Baseline models
**Featurization of multimodal data.** In our baseline model, we employ a modular approach in multimodal fusion that combines different embeddings to create a multimodal feature representation (Figure 2). The data featurization module is highly specialized for a given modality. Singly observed categorical and numeric data are fed directly into the fusion and prediction modules. All other data modalities undergo an additional step of featurization. For continuous vital sign monitoring, we calculate minimum and maximum values and linear trends. We manually engineer features such as heart rate variability, pulse transit time, and perfusion index from the ECG and PPG waveforms, and produce ECG embeddings from a pre-trained transformer [37]. To encode EHR codes with text descriptions such as diagnoses, medications, and lab results, we leverage ClinicalBERT embeddings to capture the richer and more general information provided by the text description, rather than relying on one-hot embeddings for the codes themselves. For orders, we employ Word2Vec-like embeddings, which are trained based on the co-occurrence of order pairs within patient visits. Lastly, from radiology reports, we produce text embedding features using the pre-trained RadBERT transformer as outlined in Yan et al.'s work [38]. These diverse information sources—ClinicalBERT, Word2Vec for orders, and RadBERT for radiology reports—are then concatenated to form a comprehensive multimodal embedding representation.
Multitask learning. We propose a novel multitask training schema for our LightGBM models inspired by prompting in language models, whereby a model's predictions are guided by simply providing specific task descriptions. Depending on the training schema, whether multitask or single-task, we further concatenate the multimodal representation with a task description encoded by BERT-Tiny [39, 40] embeddings. The task descriptions used are "predict disposition" for disposition prediction; "predict revisit within [D] days" for revisit prediction, where D is 3, 7 or 14 depending on the revisit horizon; and "predict decompensation within [M] minutes" for decompensation prediction, where M is 60, 90 or 120 depending on the decompensation window of interest. We chose to encode the task as a text embedding because this approach will have the capability to generalize for predictions on a non-discrete set of tasks. Finally, we train a LightGBM model to make predictions based on these enriched multimodal representations.
Figure 2: Summary of data modalities represented in MC-BEC and modality-specific featurization strategies.
Table 2: AUPRC of task-specific and multitask models (point estimate and  $95\%$  confidence interval).
Task\Model	Task-specific model	Multitask model	Class Prevalence
Decompensation	60 min	0.33 (0.27 - 0.40)	0.25 (0.20 - 0.31)	0.11
90 min	0.35 (0.30 - 0.41)	0.32 (0.27 - 0.38)	0.15
120 min	0.40 (0.35 - 0.46)	0.35 (0.30 - 0.41)	0.17
Disposition	0.87 (0.85 - 0.87)	0.82 (0.81 - 0.83)	0.37
Revisit	3 days	0.11 (0.08 - 0.14)	0.10 (0.08 - 0.13)	0.05
7 days	0.16 (0.14 - 0.19)	0.16 (0.14 - 0.19)	0.08
14 days	0.24 (0.21 - 0.27)	0.25 (0.22 - 0.28)	0.12