Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine
Authors: Emma Chen
Aman Kansal
Julie Chen
Boyang Tom Jin
Julia Rachel Reisler
David A Kim
Pranav Rajpurkar
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Benchmark
Multimodal Clinical Data
Medical AI
Deep Learning
Foundation model
Electronic health records (EHR)

Abstract: We propose the Multimodal Clinical Benchmark for Emergency Care (MC-BEC), a comprehensive benchmark for evaluating foundation models in Emergency Medicine using a dataset of 100K+ continuously monitored Emergency Department visits from 2020-2022. MC-BEC focuses on clinically relevant prediction tasks at timescales from minutes to days, including predicting patient decompensation, disposition, and emergency department (ED) revisit, and includes a standardized evaluation framework with train-test splits and evaluation metrics. The multimodal dataset includes a wide range of detailed clinical data, including triage information, prior diagnoses and medications, continuously measured vital signs, electrocardiogram and photoplethysmograph waveforms, orders placed and medications administered throughout the visit, free-text reports of imaging studies, and information on ED diagnosis, disposition, and subsequent revisits. We provide performance baselines for each prediction task to enable the evaluation of multimodal, multitask models. We believe that MC-BEC will encourage researchers to develop more effective, generalizable, and accessible foundation models for multimodal clinical data.

Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine
Emma Chen 1.4+
Aman 2 Julie Chen² Boyang Tom Jin
arXiv:231 [cs.LG] 7 Nov 2023
Julia Rachel Reisler² David A Kim Pranay Rajpurkar"
Department of Biomedical Informaties, Harvard Medical School
Department of Computer Science, Stanford University Department Emergency Medicine, Stanford University School of Medicine
Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University yingchengg. www.aaa..adeeeee (amkansal, jchen80, tomjin, jreisler, davidak) @ataakkkstaaordd..oo edu
Abstract
We propose the Multimodal Clinical Benchmark for Emergency Care (MC-BEC), comprehensive benchmark for evaluating foundation models in Emergency Medicine using a dataset of 100K+ continuously monitored Emergency Department visits from 2020-2022. MC-BEC focuses on clinically relevant prediction tasks at timescales from minutes to days, including predicting patient decompensation, disposition, and emergency department (ED) revisit, and includes a standardized evaluation framework with train-test splits and evaluation metrics. The multimodal dataset includes wide range of detailed clinical data, including triage information, prior diagnoses and medications, continuously measured vital signs, electrocar- diogram and photoplethysmoorrphh waveforms, orders placed and medications administered throughout the visit, free-text reports of imaging studies, and informa- tion on ED diagnosis, disposition, and subsequent revisits. We provide performance baselines for each prediction task to enable the evaluation of multimodal, multitask models. We believe that MC-BEC will encourage researchers to develop more effective, generalizable, and accessible foundation models for multimodal clinical data.
Introduction
Emergency Medicine a critical area of healthcare in which timely and accurate decisions, drawing appropriately on wide variety of data sources, have significant impact on patient outcomes 1. However, developing effective foundation models for electronic health record (EHR) data in Emergency Medicine requires addressing several challenges. EHR data in Emergency Medicine heterogeneous, including clinical notes, orders, lab results, imaging studics, and physiological waveforms. This heterogeneity can make it difffcult to extract meaningful features from the data and integrate them into a single model. Data quality and missingness can also be a significant issue, due to the fast-paced and gressureseure nature of emergency care. Inaccurate incomplete data limit the reliability of model results. Clinical interpretability is also critical. Clinicians must be able to understand the model's 's predictions make informed decisions. Therefore, models developed for EHR analysis in Emergency Medicine must be transparent and explainable. Finally, the limited scope and standardization of existing datasets significant challenges to developing foundation models for Emergency Medicine. Many datasets focus on specific patient populations, such as trauma
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

patients, or specific tasks, such as predicting mortality. This orientation to specific subgroups and tasks makes difficult compare and evaluate models across tasks and patient populations.
To address these challenges, and to promote the development of robust and clinically useful foundation models for Emergency Medicine, we propose the Multimodal Clinical Benchmark for Emergency Care (MC-BEC), a comprehensive benchmark for evaluating foundation models in Emergency Medicine. MC-BEC is built on dataset [ of 102,731 monitored visits made by 63,389 unique patients between 2020 and 2022, and provides a unique opportunity to study acute care in the COVID-19 era. It is the only multimodal medical dataset that exclusively covers patients during this period, while also capturing a wide range of non- non-COVID pathology. This dataset covers a wide range of information for emergency department (ED) patients, including triage information, prior diagnoses and medications, orders placed in the ED, medication administrations, lab results, continuously monitored vital signs and physiologic waveforms, and free-text reports for radiology studies. With its emphasis on multiple modalities, including continuous waveforms and vital signs providing physiologic context for heterogeneous and often rapidly evolving patients, MC-BEC presents opportunities to study the uniquely dynamic and complex nature of emergency care.
MC-BEC emphasizes clinically relevant downstream tasks at multiple timescales, specifically predict- ing patient decompensation (within minutes), disposition (within hours), and ED revisit (within days), and provides a standardized evaluation framework with train-test splits and evaluation metrics. We also provide baselines for each task to enable model comparison and evaluation. With MC-BEC, we hope to encourage researchers and clinicians to develop more effective, generalizable, and accessible foundation models for EHR analysis in Emergency Medicine, ultimately improving patient outcomes and advancing the analysis of real-world EHR data.
Related Work
2.1 Current ED benchmarks are not multimodal
Existing EHR datasets for ED or critically ill patients are often limited to structured EHR data and intermittent vital sign recordings. These datasets fail to capture the comprehensive multimodal information obtained from the intensive evaluation and monitoring of ED patients. To our knowledge, only two ED-specific benchmarks exist. Xie et al. (2022) [2] introduced an ED benchmark using the MIMIC-IV-ED dataset (3]. While this dataset represents the only publicly available general-purpose ED dataset, contains only tabular EHR data for all patients, with radiology reports for subset. EHRShot 4 is the other recent ED benchmark, but its underlying dataset includes only coded data such as ICD diagnosis codes, and is not publicly available.
Due to the lack of robust ED benchmarks, we also reviewed existing critical care benchmarks, since patient monitoring practices in the ED and intensive care unit (ICU) are similar. A comparison in Table ssoos most ICU datasets also focus on structured EHR data and intermittent vital signs, lacking free text waveforms. HiRID(5) 55 provides high-resolution physiological data but text/reports. Strikingly, neither ICU nor ED datasets include electrocardiogram (ECG) and sooogpp (PPG) waveforms, which represent essential data on the physiology of critically ill patients.
To address the lack of multimodal ED benchmarks, we propose MC-BEC, using a novel multimodal dataset including vital signs, continuous physiologic waveforms (ECG, PPG, respiration), radiology reports, and diverse structured EHR data. This supports improved evaluation of model performance on wider range of salient patient information.
2.2 Current ED benchmarks are not suitable for generalist medical AI
Existing medical AI benchmarks fall short in comprehensively evaluating generalist medical AI (GMAl) and clinical foundation models. GMAI was recently proposed as goal for foundation models in healthcare and medicine (6) GMAI would perform a wide range of medical tasks and flexibly interpret different combinations of data modalities, enabled by foundation models' capability to learn broadly useful data representations from massive pretraining (7]. However, as discussed in a recent survey (8 current benchmarks focus narrowly on accuracy for predefined tasks, inadequate for evaluating key GMAI capabilities. A major limitation is the lack of assessment for multimodal
Multimodal Clinical Monitoring in the Emergency Department (MC-MED-v0)

---

integration [ While GMAI is designed to integrate diverse data modalities, performance can decline with more modalities ( Yet aceuracy metries alone will not expose these nuances. Current benchmarks also rarely evaluate how models handle missing modalities, though incomplete data is pervasive real-world EHRs.
To address these gaps, benchmarks must move beyond accuracy to rigorously assess multimodal performance, tolerance missing data, and other facets critical for GMAL The proposed MC- BEC benchmark exemplifies this comprehensive approach through evaluating multitask learning, multimodal performance, and fairness analyses beyond predictive accuracy alone. Rethinking evaluation is imperative as GMAI diverges from narrow benchmarks of the past toward more expansive and integrative capabilities. Robust benchmarks will be central to steering progress in this promising new direction.
Table 1: Comparison of critical and emergency care EHR benchmarks. MT Learning stands for multitask learning; MM Analysis stands for multimodal analysis; the number of patients and visits of MIMIC-III III excludes neonatal patients.
Benchmark
Eval. Beyond Acc.
Source
Data Modalities
MT
MM
Faimess
Dataset
Num.
Num.
EHR
Free Text Vital ECG/PPG
Learning Analysis
patients
visals
Codes Notes
Signs Waveform
MIMIC-Fxtraet IM
Purusbotham 2018 12
MIMIC-II
39K
53K"
X
X
Hanutyunyan
Xie 2022 X
217K
449K
Gupta
|MMIC--V (v1.0)
[T
|eCC [9
139K
201K
EHR 2T
(MIMCC-I( [8
HIRID-ICU 21
HiRID (V1.1.1
34K
56K
x
EHRSHOT
|Stnnoord Med.
7k
894k
MC-BEC
|tannoord EM
102K
2.3 GMAI for multimodal EHR data
Our baseline model is inspired by related work in the field. A common approach to foundation models is to first pretrain them on tasks with abundant data to generate robust representations, then to fine- sunettune on downstream tasks. ClinicalBERT [22] is widely used pretrained model specifically designed for clinical text embeddings. Categorial EHR data also has pretraining options like BEHRT[23), CLMBR 24 and MOTOR 25), which predict future EHR codes based on past codes. However, these representations of EHR codes depend on the code formats used during training, limiting transferability and robustness to heterogeneous EHR formats. Instead, we employed CodeEmb's 's (26) approach that uses pretrained text embeddings associated with EHR code descriptions regardless of the format or
code system used by the EHR. For fine-tuning on multimodal data, we adopt the HAIM framework using a modular ML pipeline that integrates modality-specific pretrained embeddings.
27
2.4 Multitask learning
Our multitask training approach draws inspiration from prompting techniques in language models. Rather than using task-specific prediction heads, we employ unified task representation combined with task-specific queries. This allows creating broadly capable models without architectural changes for new tasks.
Conventional multitask learning often designates separate prediction heads for each task, as in MOTOR which has heads for distinct medical predictions(25) However, the promising avenue of using unified task representations combined with task-specifie queries has gained traction, as showcased in studies like OFS 23 and Perceiver 10 29]. We enhance our baseline LightGBM model 30 by incorporating contextual embeddings from the BERT model. For each task, we concatenate BERT embeddings of the task-specific queries with other input features. This integrates BERT's rich semantic understanding, allowing the model to adeptly differentiate tasks within the shared

---

Decompensation
Disposition Discharge v.s. admit?
Revisit
tachycardia, hypotension, hypoxia?
How licely to return to
triage
Patient roomed
interventions
Prediction window 3,7,14 days
Monitoring
window
Within ED visit
After ED visit
End of ED visit
Figure 1: MC-BEC evaluates foundation model performance on predictions of ED patient decompen- sation, disposition, and revisit, using a unique multimodal dataset of 102,731 monitored ED visits.
embedding space. This provides flexibility to expand to new prediction tasks without architectural changes.
3 Benchmark
MC-BEC a benchmark for evaluating clinical foundation models/GMAl on multimodal EHR data in the ED setting. provides three clinically relevant prediction tasks spanning different stages of an ED visit: prediction of near-term decompensation early in the visit, prediction of disposition at the end of visit, and prediction of revisit after the visit. MC-BEC includes evaluation metrics beyond prediction performance, capturing modality interaction, faireess, and consistency of predictions with respect time and modalities.
3.1 Data source
The MC-BEC dataset consists of 102,731 ED visits made by 63,389 patients between September, 2020 and September, 2022. The dataset is IRB approved, with waiver of informed consent for retrospective research on de-identified data. The de-identified data for each ED visit spans the entire visit from department arrival to departure, including triage after ED arrival, rooming and initiation of monitoring, and all interventions and results up to the point of departure from the ED. Patients the dataset have multiple ED visits, but MC-BEC ensures no patient overlap (i.e., visits from the same patient) across the training, testing and validation cohorts. Unlike previous ED datasets, MC-BEC contains both categorical and unstructured clinical data. These modalities and data structures described below.
Categorical data with single observation: chief complaint, triage acuity level, gender, race, ethnicity, means of arrival, disposition of most recent visit, disposition of current visit, and payor class.
Categorical data with repeated observations: current and previous ICD-10 diagnosis codes with timestamps and corresponding text descriptions; home medications and medications administered during visit with IDs, names, and timestamps of administration; orders placed during visit with IDs, names, and timestamps; lab results with IDs, names, result values, and timestamps.
Numeric data: age, hours since previous visit, and vital signs including heart rate, respiratory rate, blood pressure, oxygen saturation, and temperature.
Time-series data: continuously recorded vital signs (heart rate, respiratory rate, oxygen saturation) and secondary features (heart rate variability, pulse transit time, perfusion index) throughout the patient's visit. These measurements were averaged over 1 1-minute intervals.
Waveform data: electrocardiogram (ECG), photoplethysmogram (PPG) and respiratory waveforms.
Free-text data: radiology reports for imaging studies ordered during the ED visit.

---

3.2 Benchmark tasks
We chose decompensation, disposition, and revisit as prediction tasks for MC-BEC. These tasks have high clinical and operational relevance in the ED setting: early identification of patients at risk of decompensation allows healthcare providers to allocate appropriate resources; accurate disposition prediction helps optimize resource allocation, bed management, and patient flow within the hospital system; and understanding the probability of revisit allows healthcare providers to identify patients who may require closer follow-up or additional interventions to prevent complications proper continuity of care. These prediction tasks collectively encompass the entire timeframe of an ED visit, focusing on different aspects in the patient's 's journey through the ED encounter (Figure 1). We provide brief descriptions each task as defined for our benchmark:
ensure
1. Decompensation: The goal of this binary classification task is to predict which patients are likely to experience clinical decompensation, defined as new onset of tachycardia (heart rate [HR] > 110), hypotension (mean arterial pressure [MAP] 65mmHg), or hypoxia (oxygen saturation by pulse oximetry [SpO2] < 90%) in patients with initially normal vital signs. The task uses the first 15 minutes of data acquired after the patient is roomed (the assessment period) to predict decompensation in a 60, 90 or 120 minute-window (evaluation period) following the assessment period. Patients with abnormal vital signs at triage or at any point during the assessment period are excluded from this task, in order to prevent trivial predictions (e.g., the prediction of future hypotension in an already hypotensive patient).
2. Disposition: The objective is to predict the binary outcome of patient's 's disposition from the ED: whether they will be discharged home or admitted to the hospital. This is a summative clinical decision that reflects the patient's 's overall clinical stability and risk as determined by the totality of evidence accrued during the visit. To avoid data leakage, any information that could reveal the disposition decision (such as a consult order to an admitting service) is excluded for this task.
3. Revisit: The goal is to predict whether patient will revisit the ED within a 3, 7, or 14-day period after being discharged. ED revisits are a common quality metric, because return visits can sometimes suggest incomplete workup or inappropriate disposition (i.e., a patient sent home who should have been admitted). All data from visit can be used to make the revisit prediction, and no patients or visits from the dataset need to be excluded for this task.
3.3 Evaluation framework
Prediction performance. We propose evaluate prediction performance with area under the precision-recall curve (AUPRC) as a beesholddindepeedentt metric. AUPRC is a clinically meaningful metric because it reflects the model's performance in correctly identifying positive instances while minimizing false positives. In medical scenarios, correctly identifying positive cases is crucial, as it ensures accurate diagnosis or prediction of a particular condition. AURPC is also particularly important when dealing with unbalanced data, which is common among medical datasets, including ours.
Monotonicity in modalities. We propose to assess the performance change of a model when increasing the data modalities available for training. We hypothesize that well-designed multimodal model should not perform worse when more data modalities are used for training. However, such monotonicity of performance in modalities is not always observed. For example, [9] found that unimodal network performed better than the multimodal network obtained through joint training. A reduction in performance caused by the inclusion of more modalities may be attributed to modality competition during training, wherein the model learns to rely on only a subset of modalities to make predictions (3 The evaluation of monotonicity in modalities can help guide the development of more effective and robust multimodal models.
We evaluate monotonicity in modalities Momddtitica by calculating the concordance index C' between the actual performance Pm (i) and the theoretical ground truth which would expect better performance Vm (i) for increasing numbers of modalities m:
Dml S ym2 S... S 1miin-1 S Vm((n)
ΣL D 1(),, Ui. Pÿ)
M Mmodalities C
(2)
n(nn - 1)/2
5

---

(f: < <z) (pr
< < + (vs > fj) (p. > pp)
f(y,, My. Pt,
0.5 P: P
(3)
otherwise
Robustness against missing modalities. We evaluate the model's robustness and generalization capabilities when one more modalities of data used in training are unavailable in testing. This metric encourages the development of models with more reliable performance in real-world applications where not all training modalities will be present in all cases.
Bias. We evaluate bias as crucial metric detect systematic errors or prejudices in the model's predictions or outputs. MC-BEC reflects the ED, where false negative prediction of adverse events can result in severe negative health outcomes. Incorrect disposition decisions might exacerbate health issues; misjudged revisit predictions can overlook preventive care needs; and unforeseen clinical decompensation. like hypotension or hypoxia, can be rifetthreatennng Hence, we chose the true positive rate (TPR) difference as our primary bias metric. Using a sensitivity threshold of 0.85 on the validation set, we assess TPR differences between different demographic groups such as age, gender, race, and ethnicity. Our approach for evaluating model fairness seeks to identify underdiagnosis bias, which is arguably most relevant and consequential in healthcare settings [32
4 Experiments
We developed and evaluated baseline models to demonstrate the use of MC-BEC. Our models used pretrained text and waveform embeddings and employed modular approach for modality fusion. We trained multimodal representations using LightGBM
30, as gradient boosting ensembles are We also proposed a novel multitask training schema with LightGBM and compared its performance against models trained on individual tasks. Our aim to provide an example of MC-BEC evaluation, and a reference for future models evaluated with this benchmark.
often highly performant in clinical tasks 33 34
35 36
4.1 Baseline models
Featurization of multimodal data. In our baseline model, we employ a modular approach in multimodal fusion that combines different embeddings create multimodal feature representation (Figure 2). The data featurization module is highly specialized for a given modality. Singly observed categorical and numeric data fed directly into the fusion and prediction modules. All other data modalities undergo an additional step of featurization. For continuous vital sign monitoring, we calculate minimum and maximum values and linear trends. We manually engineer features such as heart rate variability, pulse transit time, and perfusion index from the ECG and PPG waveforms, and produce ECG embeddings from pre-trained transformer (37). To encode EHR codes with text descriptions such as diagnoses, medications, and lab results, we leverage ClinicalBERT embeddings capture the richer and more general information provided by the text description, rather than relying on one-hot embeddings for the codes themselves. For orders, we employ Word2Vec-libe embeddings, which are trained based on the co-occurrence of order pairs within patient visits. Lastly, from radiology reports, we produce text embedding features using the pre-trained RadBERT transformer as outlined Yan al.'s 's work (38]. These diverse information sources ClinicalBERT, Word2 for orders, and RadBERT for radiology reports-are are then concatenated to form a comprehensive multimodal embedding representation.
Multitask learning, We propose a novel multitask training schema for our LightGBM models inspired by prompting in language models, whereby model's predictions are guided by simply providing specific task descriptions. Depending on the training schema, whether multitask or single- task, we further concatenate the multimodal representation with a task description encoded by BERT-Tiny [39 40) embeddings. The task descriptions used are "predict disposition" for disposition prediction; "predict revisit within [D] days" for revisit prediction, where
7 or 14 depending on the revisit horizon: and "predict decompensation within [M] minutes" for decompensation prediction, where M is 60, 90 or 120 depending on the decompensation window of interest. We chose to encode the task predictions on non-discrete set tasks. Finally, we train LightGBM model to make predictions based on these enriched multimodal representations.
text embedding because this approach will have the capability to generalize for

---

BERFTiyy transforme
Multimedal fusion + prediction LGBM
amowthnng
Visa uig Pwnce
FT, HRC Partsios
Ted eriondinge
Lineat inarida,, mEls maa
Eng feored
wardived featurication
Irattformee
Etbricyy
Age Iuu sace
Time,
Antol
Race
Spr00
Tima, Hamryy
oparts
Gondor
[0/0-1
CC
5002
20066
Luc rusdit
BC Thme, Nome, Bevel]
Figure 2: Summary of data modalities represented in MC-BEC and modality-specific featurization strategies.
Table 2: AUPRC of task-specific and multitask models (point estimate and 95% confidence interval).
Model
Task-specific model
Multitask model
Class Prevalence
Task
60 min
0.33 (0.27 0.40)
0.25 (0.20 0.31)
0.11
Decompensation
90 min
0.35 (0.30 0.41)
0.32 (0.27 0.38)
0.15
120 min
0.40 (0.35 0.46)
0.35 (0.30 - 0.41)
0.17
Disposition
0.87 (0.85 0.87)
0.82 (0.81 0.83)
0.37
3 days
0.11 (0.08 0.14)
0.10 (0.08 0.13)
0.05
Revisit
7 days
0.16 (0.14 0.19)
0.16 10.14-0.11
0,08
14 days
0.24 (0.21 0.27)
0.25 (0.22 0.28)
0.12
4.2 Benchmark results
Prediction performance. We compare the performance of models trained on a single task (de- compensation, disposition, or revisit) with model trained simultaneously on all three prediction tasks. Table 2 presents the model performance as assessed by AUPRC using all available data modalities. Overall, task-specific models perform better than the multitask model in decompensation and disposition predictions, but similarly in revisit predictions. For decompensation prediction, the task-specific model has AUPRCs of 0.33, 0.35 and 0.40 (for 60-, 90- and 120-minute prediction windows), while AUPRCs for the multitask model are 0.25, 0.32, and 0.35. Disposition is predicted with AUPRC 0.87 for the task-specific model and 0.82 for the multitask model. When predicting revisit at 3, 7 and 14-day time windows, we report AUPRC scores of 0.11, 0. 0.16, and 0.24 for the spsk-sp model, and 0.10, 0.16, 0.25 for the multitask prediction model. The prevalence of the positive class in the dataset is provided as baseline AUPRC [41]. Our results indicate slightly worse performance with multitask prediction model which conforms with the assumption that the single-task prediction models will be better specialized for their specific task. However, the overall performance gap between single and multitask approaches is small. This suggests potential for more advanced multitask modeling to match or surpass the single-task results by better leveraging commonalities and differences between tasks. Recent work has shown promise for sophisticated multitask architectures surpassing single-task models in clinical predictions. For instance, multitask channel-wise LSTM exceeded single-task models predicting in in-hospital mortality 13] research can MC-BEC as avenue to improve upon multitask models.
Future
Monotonicity modalities. To assess monotonicity in modalities, we train the multitask prediction model on an increasing subset of data modalities. We start with model trained only on categorical and numeric features, then incrementally add features from diagnoses, medications, orders, lab results, radiology reports, continuous monitoring, and waveforms. Table 3 shows the AUPRC of

---

Table 3: AUPRC of multitask models with incrementally increasing testing and training modalities (point estimate and 95% confidence interval). The modality concordance index measures monotonicity of prediction performance in modalities.
Midel
Decompensation
Dicposition
Revisit day
60 unn
90 min
120 min
day
14 day
Modality added
Cattgoniallmmmeenc
0.25 (0.15 0.29)
0.30 05.25
0.25)
(0.2
0.29)
0.2] (0.58
0.72)
0.10
(0:08
0.13)
0.15
(0.13
0.18)
0.25 (0.22 0.28)
Diagnoses
0.25 (0.15- 0.28)
60:2 0.35)
00.2
0.39)
(.20
0.72)
0.09 (0:07
0.111
0.15 00.13
0.18)
0.24
(0:21 -0.2))
Medicarions
0.22 (0.15-1
0.29:0:24
0:344
0..2
0.38)
0.71 00.00-000000 0.72)
0.09 (0:07
-.112
0.15 -
0.24
(0.22-0---0 0.22)
Deders
0.24 (8:21 00.40
(.27
0.32)
0.79 (0:78-8))))) 0.30)
0.09 (0:08- 12)
0.15 -0-13-111111
0.24 (0.32-00-002
Lahe
0.25
-.29)
0.79 (0.78-88))) 8.81
0.09
0.15 00-18.10--011
0.24 00.32-0-200
Radislogy
(..19)
0.21 18.18- 0..55
(8.21
-1.29)
0.79
8.300
0.16 00.00044-011.
0.255
Mennering
00.23
0.39)
(0.79 0.82)
10.14
00.22 0.25)
0.32 (0.27
0/413
0.83)
0.16 (0.14 0.19)
(0.22 0.25)
Modality concordance
0.45
0.71
Table 4: AUPRC for multitask models with one modality missing in test data, demonstrating robustness to missing modalities (point estimate and 95% confidence interval).
Model
Decemspenaaion
Disposition
Revist
50 min 0.25 (0.20 0.31)
90 min
120 min
day
day
[4 day
All modalities
0.32 (0.27
0.38)
(0.30 0.41)
0.82
-0.81-0.89 0.83)
0.10 (0.08- - 0.13)
0.16
(8.14 0.19)
0.25
(0.22 0.225
Missing mudality
Calegniicllmmmiis
0.26 (0.21 0.32)
0.32 (0:27
0.38)
(0.30
0.41)
0.81 (0.80 0.83)
0.06 (0:05 0.07)
0.10
0.11)
0.17 (0.16 0.19)
Diagnnses
0.25 (0.20-0.31)
0.331
0.32 (0.27
0.38)
0.413
0.82 (0.8)
0.83)
0.10 (0.08-
0..33
0.16 1.1610.14 6..19
(0.22-0.23)
00.25)
Medications
0.25 -0331
0.32 (0:27
00.77)
0.35 (0.30 0.40)
0.82 (0.81 00833
0.88 (0.07
0..11)
0.13
(0.100000-0111
00.15)
0.23
(0.21 00.25)
Orders
0.26 (0.21 0.32)
0.34 (0:25 00.00
0.57 (0.32
0.43)
0.73 (0.73 0.75)
0.07 (0.05 0..9%
0.11 (0.10
-0113)
0.23
(0.21 0.25)
Labs
0.25 (0.21 0.33)
0.33 (0.25 0.39)
0.35 (0.31
0.42)
0.31 (0.80 0.82)
0.08 (0.06
0.30)
0.13 (0.11
0.15)
0.23
(0.21 0.25)
Radiology
0.25 -0.20-0.31)
0.32 (0:27 (0.37)
0.35
(0.30) 0.40)
0.50 (0. (0.81
0.87 (0 -.:30
0.12 -0.100000 0.14)
0.23
(0.21 -8.25)
Monilosing
0.14 -0112
0.20 (0.18 00.24)
0.25 (0.20
0.27)
0.52 (0.81 0.883
0.08 (0.06- 01.30)
0.13
(0.11 -0.11-0.15
0.23
(0.21 -0.25
Wiieefome
0.24 (0.19-0.29)
0.31 (0.26 0.366
0.34 (0.29
0.319)
0.83 (0.8) 0.883
0.08 (0.06- - 0.100
0.13
10.1110.155
0.23
(0.21 00.26
Max AUPRC difference
0.12
0.13
0.34
0.09
0.04
0.06
0.05
models trained with increasing numbers of modalities, along with the modality concordance index. Disposition prediction demonstrates strong monotonicity in modalities with a modality concordance index of 0.86. This indicates that disposition prediction benefits from incorporating more data modalities. Revisit predictions demonstrate moderate monotonicity, with concordance indices of 0.71, 0.79, and 0.82 for the 7, and 14-day windows. Decompensation predictions show the worst monotonicity among all tasks, with notably decreased performance when adding orders, labs, and radiology reports data. This poor monotonicity in modalities is reflected in low concordance indices of 0.50, 0.46, and 0.43 for the 60, 90, and 120-minute windows. However, the performance does increase when continuous monitoring and waveforms are added, which aligns with the expectation that continuous monitoring and waveforms of vital signs are predictive of impending decompensation events.
Robustness against missing modalities. We assess the robustness of the multitask benchmark model against missing modalities during inference by dropping individual modalities and comparing the difference in AUPRC between using all modalities versus missing modalities (Table 4 Although revisit predictions do not have the largest maximum AUPRC difference. with values of 0.04, 0.06, and 0.08 for the 3, 7, and 30-day windows, they are more likely to see a performance decrease when missing modalities. Specifically, revisit predictions show notable performance drops when all modalities except diagnoses are missing, compared to the substantial drops for decompensation predictions only when continuous monitoring data is missing and for disposition predictions only when orders data is missing. Furthermore, the performance gap widens in the worst case of missing modalities over the longer time windows for both revisit and decompensation predictions, with maximum AUPRC differences of 0.12, 0313, and O. 14 for the latter. Qualitatively, the impact on model performance when omitting modalities varies significantly depending on the prediction task. Overall, our results suggest that the benchmark predictions are reasonably robust to missing modalities during inference.
Bias. Lastly, in table 5 we report mean TPR differences for the multitask prediction model when applied to different demographic groups, specifically based on patient age, gender, race, and ethnicity. The greatest absolute disparities in mean TPR emerged when comparing different racial groups, with a mean TPR difference of 0. I for 3-day revisit prediction, followed by a mean TPR difference of 0.09 for 7-day revisit prediction. Notable model bias was also observed between gender groups for 90 and 120-minute decompensation predictions, with TPR differences of 0.08 and 0.09. Additionally, there was sizable TPR difference of 0.07 between age groups for 60-minute decompensation predictions.
8

---

Table Evaluation of model bias across demographic groups. True positive rate (TPR) and absolute TPR difference between different groups are reported below.
Decompensation
Disposition
Revisit
Demographic Group
60 min 90 min 120 min
day 7 day
14 day
Age
55
0.87
0.86
0.88
0.93
0.69
0.75
0.88
>= 55
0.94
0.91
0.92
0.97
0.65
0.79
0.92
TPR difference
0.07
0.05
0.05
0.04
0.04
0.03
0.04
Gender
Male
0.92
0.93
0.94
0.96
0.68
0.79
0.91
Female
0.88
0.85
0.85
0.95
0.67
0.74
0.87
TPR difference
0,05
0.08
0.09
0.01
0.01
0.04
0.04
Race
White
0.88
0.88
0.88
0.96
0.63
0.77
0.88
Black/African American
1.00
0.90
0.95
0.95
0.82
0.90
0.94
Asian
0.88
0.87
0,88
0.96
0.64
0.73
0.85
Native Hawaiian/Other Pacific
1.00
0.86
1.00
0.93
0.80
0.86
0.81
Islander
Other
0.90
0.90
0.91
0.95
0.70
0.74
0.90
TPR difference
0.07
0.02
0.06
0.01
0.11
0.09
0.06
Ethnicity Non-HispaniccNon-LLtiin
0.89
0.88
0.89
0.96
0.68
0.77
0.88
Hispanic/Latinn
0.92
0.91
0.92
0.94
0.66
0.75
0.91
TPR difference
0.03
0.03
0.03
0.01
0.02
0.02
0.03
We also present supplementary fairness metrics like Statistical Parity Difference (SPD). Disparate Impact (DI), and Equal Opportunity Difference (EOD) between minority and majority cohorts in Table 15 in the Appendix. Models applied the MC-BEC benchmark should be attentive to potential biases assessed by framework, and any identified biases must be carefully considered before deployment to ensure fairness and equity.
Limitations
While the benchmark provides valuable insights, MC-BEC has limitations. The benchmark does not assess the potential of foundation models in automating novel clinical tasks, such as summarizing all data obtained throughout visit into easy-to- soyy--toonddrtaadd report for the provider patient. Future iterations of the benchmark should include novel tasks specific to the ED environment to further highlight the capabilities of clinical foundation models. Although the benchmark evaluates bias measured patient characteristics, it does not account directly for socioeconomic status. Due to privacy and legal concerns, socioeconomic information, such as income education level, are not included in the underlying dataset. However, the dataset does include visit payor information, which is correlated with sociocconomic status (for instance, patients whose visits are paid by Medicaid are more likely to have lower socioeconomic status). Finally, we used a relatively simple baseline model to demonstrate the benchmark. Although this choice may limit the model's performance, the primary contribution of this work is the proposal of a new benchmark using a novel multimodal dataset. LightGBM, despite its simplicity, remains a commonly used and highly performant approach for many clinical tasks. Therefore, the baseline model serves not only as demonstration of the benchmark but also as reasonable reference for future models.
Conclusion and Future Work
Our work introduces novel benchmark (MC-BEC), enabling robust multitask evaluation of pre- diction models for Emergency Medicine, using a unique multimodal clinical dataset. We evaluate multiple baseline models, and compare the performance of multitask and task-specific models. We present an evaluation schema suited to the demands multitask clinical predictions on complex multimodal data, which assesses model's ability to make full use of multimodal data, to make robust

---

predictions despite missing data, and to make fair and unbiased predictions across demographic groups.
The benchmark models introduced in this work employed an explicit data featurization step, which highlights the challenges of working with diverse, complex and data-rich modalities. We expect future models applied to MC-BEC to leverage alternative approaches, such as with deep end-to-end trainable networks, and integration of a temporal dimension both within and across visits. MC-BEC is centered on predictions of decompensation, disposition, and revisit because these encompass the entire time scale of ED visit prediction tasks. However, future work in benchmarks for generalizable ED prediction models should also aim to evaluate prediction over dincrieeee set of tasks, such as prediction of therapeutic complications or need for post-dischand follow-up. This would enable more granular and comprehensive predictions that could help providers to target and prioritize interventions.
References
[1] Nathan Walzl, Jessica Jameson, John Kinsella, and David Lowe. Ceilings of treatment: qualitative study in the emergency department. BMC Emergency Medicine, 19(1):1-8, 2019.
[2] Feng Xie, Jun Zhou, Jin Wee Lee, Mingrui Tan, Siqi Li, Logasan S/O Rajnthern, Marcel Lucas Chee, Bibhas Chakraborty, An-Kwok lan Wong, Alon Dagan, et al. Benchmarking emergency department prediction models with machine learning and public electronic health records. Scientific Data, 9(1 (1)x688,, 2022.
[3] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng Leo Anthony Celi, and Roger Mark. MIMIC-IV (version 0.4). https://doi..rgg 10. 13026/ 3005n 2020. Phys- ioNet.
[4] Michael Wornow, Rahul Thapa, Ethan Steinberg, Jason Fries, and Nigam Shah. Ehrshot: An ehr benchmark for few-shot evaluation of foundation models. arXiv preprint Xiv222007.00008 2023.
[5] Martin Faltys, Moritz Zimmermann, Xin Lyu, Marco Hüser, Stephanie Hyland, Gunnar Rätsch, and Tobias Merz. Hirid, a high time-resolution icu dataset (version 1.1.1). PhysioNet, 2021.
[6] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):299-255, 2023.
[7] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint
2021.
[8] Michael Wornow Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg. Scott Fleming, Michael Pfeffer, Jason Fries, and Nigam Shah. The shaky foundations of clinical foundation models: A survey of large language models and foundation models for emrs. arXiv preprint arXiv:2303.1 /296/ 2023.
[9] Weiyao Wang. Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22695-12705 2020.
[10] Shirly Wang, Matthew BA McDermott, Geeticka Chauhan, Marzych Ghassemi, Michael C Hughes, and Tristan Naumann. Mimic-extract: A data extraction, preprocessing, and represen- tation pipeline for mimic-iii. In Proceedings of the ACM conference on health, inference, and learning, pages 222 225, 2020.
[11] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[12] Sanjay Purushotham, Chuizheng Meng, Zhengping Che, and Yan Liu. Benchmarking deep learning models on large healthcare datasets. Journal of biomedical informatics, 13:112-134, 2018.
[13] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):66, 2019.
10

---

[14] A. Johnson, L. Bulgarelli, T. Pollard, L. A. Celi, R. Mark, and S. Horng. Mimic-iv-ed (version 1.0). httpss///oi. org/ 13026/772666222 9w59, 2021. PhysioNet.
[15] Mehak Gupta, Brennan Gallamoza, Nicolas Cutrona, Pranjal Dhakal, Raphael Poulain, and Rahmatollah Beheshti, An extensive data processing pipeline for mimic-iv. In Machine Learning for Health, pages 3-253 PMLR, 2022.
[16] A. Johnson, L. Bulgarelli, T. Pollard, S. Horng, L. A. Celi, and R. Mark. Mimic-iv (version 1.0). https://doi. org/ 10. 13026/s6n6-22 0tt0//6666xxx0 xd98, 2021.
[17] Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
[18] Seyedmostafa Sheikhalishahi, Vevake Balaraman, and Venet Osmani. Benehmarking machine learning models on multi-centre eicu critical care dataset. Plos one, 15(7):e0235424, 2020.
[19] Tom J Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eicu collaborative research database, a freely available multi-center database for critical care research. Scientific data, 5(1):1-13, 2018.
[20] Matthew McDermott, Bret Nestor, Evan Kim, Wancong Zhang, Anna Goldenberg, Peter Szolovits, and Marzych Ghassemi. A comprehensive chr timeseries pre-training benchmark. In Proceedings of the Conference Health, Inference, and Learning, pages 257-278, 2021.
[21] Hugo Yèche, Rita Kuznetsova, Marc Zümmermann, Matthias Hüser, Xinrui Lyu, Martin Faltys, and Gunnar Rätsch. sefidd-cu-benchmarke comprehensive machine learning benchmark on high-resolution ieu data. arXiv preprini arXiv:2111.08530 2021.
[22] Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint ar:iv: 2019.
[23] Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Kalimi-Khorshiin Behrt: transformer for electronic health records. Scientific reports, 10(1):1-12, 2020.
[24] Ethan Steinberg. Ken Jung, Jason A Fries, Conor K Corbin, Stephen R Pfohl, and Nigam H Shah. Language models are an effective representation learning technique for electronic health record data. Journal of biomedical informatics, 1 113: 10:367 2021.
[25] Ethan Steinberg, Yizhe Xu, Jason Fries, and Nigam Shah. Self-supervised time-to-event modeling with structured medical records. arXiv preprint arX1v:2301.03/500 2023.
[26] Kyunghoon Hur, Jiyoung Lee, Jungwoo Oh, Wesley Price, Younghak Kim, and Edward Choi. Unifying heterogeneous electronic health records systems via text-based code embedding. In Conference on Health, Inference, and Learning, pages 183-203. PMLR, 2022.
[27] Luis R Soenksen, Yu Ma, Cynthia Zeng, Leonard Boussioux. Kimberly Villalobos Carballo, Liangyuan Na, Holly M Wiberg, Michael L Li, Ignacio Fuentes, and Dimitris Bertsimas. Integrated multimodal artificial intelligence framework for healthcare applications. NPJ Digital Medicine, 5(1):149, 2022.
[28] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple saueennctto-sequenee learning framework. In International Conference on Machine Learning, pages
PMLR, 2022.
[29] Andrew Jaegle, Sebastian Borgeaud, Jean- -Battiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint 217.992107.14795 2021.
[30] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
[31] Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. Modality competi- tion: What makes joint training of multi-modal network fail in deep learning learning? (provabyy. In International Conference Machine Learning, pages 9226-9259, PMLR, 2022.
11

---

[32] Laleh Seyyed- Kalantari, Haoran Zhang, Matthew B. A. McDermott, Irene Y. Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine, 27(12):116-2118 Dec 2021.
[33] Sameer Sundrani, Julie Chen, Boyang Tom Jin, Zahra Shakeri Hossein Abad, Pranav Rajpurkar, and David Kim. Predicting patient decompensation from continuous physiologic monitoring in the emergency department. npj Digital Medicine, 6(1):60, Apr 2023.
[34] Woo Suk Hong, Adrian Daniel Haimovich, and R. Andrew Taylor. Predicting hospital admission emergency department triage using machine learning. PLOS ONE, 13(7):1-13, 07 2018.
[35] Yuval Barak-Corren, Pradip Chaudhari, Jessica Perniciaro, Mark Waltzman, Andrew M. Fine, and Ben Reis. Prediction across healthcare settings: a case study in predicting emergency department disposition. npj Digital Medicine, 4(1):169, Dec 2021.
[36] Woo Suk Hong, Adrian Daniel Haimovich, and Richard Andrew Taylor. Predicting 72-hour and 9-day return to the emergency department using machine learning. JAMIA Open, 2(3):346-352, 07 2019.
[37] Annamalai Natarajan, Yale Chang, Sara Mariani, Asif Rahman, Gregory Boverman, Shruti Vij, and Jonathan Rubin. A wide and deep transformer neural network for 12-lead eog classification.
In 2020 Computing Cardiology, pages 1 1-4, 2020.
[38] An Yan, Julian McAuley, Xing Lu, Jiang Du, Eric Y Chang, Amilcare Gentili, and Chun-Nan Hsu. Radbert: Adapting transformer-basee language models to radiology. Radiology: Artificial Intelligence, 4(4):e210258, 2022.
[39] Prajjwal Bhargava, Aleksandr Drozd, and Anna Rogers. Generalization in nli: Ways (not) to go beyond simple heuristics, 2021.
[40] Iulia Turc, Ming- Bei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization knowledge distillation. CoRR, fass/1908.080 2019.
[41] Takaya Saito and Mare Rehmsmeier. The precision-recall plot is more informative than the plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3):e01 18432, 2015.
[42] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna:
A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.
[43] Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, al. Xgboost: extreme gradient boosting. package version 0.4-2, 1(4):1-4, 2015.
12

---

Appendix
1
Training Details
The dataset was divided into training, testing, and validation sets, as shown in Table 6 During training, we applied sample weights to visits based on the frequency of the label classes. Specifically, the weight for each visit was directly correlated with the frequency of the predominant class and inversely correlated with the frequency of its own class. The distribution labels is presented in It is worth noting that the prevalence of revisits was calculated only among visits with
Table 7 discharge disposition, since revisits are only relevant for patients discharged from the ED.
Table 6: Distribution of Dataset into Train, Validation, and Test Sets
Dataset
Number of Samples
Percentage (%)
Train
82,438
80.25
Validation
10,235
9.96
Test
10,058
9.79
Total
102,731
100
Table 7: Label distribution for different tasks
Category
Task
Class Prevalence
Disposition n/a
0.37
days
0.05
Revisit
days
0.08
14 Days
0.12
60 mins
0.11
Decompensation
90 mins
0.15
120 mins
0.17
Hyperparameter Tuning
We employed the Tree-structured Parzen Estimator (TPE) for hyperparameter tuning, implemented in the Optuna optimization framework [2]. TPE, Bayesian optimization approach, leverages outcomes from preceding trials inform the selection of subsequent hyperparameter values. this framework, Optuna adeptly pinpoints areas within the hyperparameter space anticipated to enhance the objective function's 's value. For each model, we executed 20 trials using the hyperparameter spaces delineated in Table g
13

---

Table 8: Search space used for hyperparameter tuning and the optimal values for the multitask model trained on all data modalities
Hyperparameters
Search Space
Optimal Values
LightGBM
lambda_ll _l
[1e-8, 10]
1.84
lambda_12
[1e-8, 10]
0.07
num leaves
[2, 306]
134
feature_fraction
feature fraction
[0.4, 1.0]
0.58
bagging_fraction
[0.4, 1.0]
0.79
bagging_freq
[1.
max_depth
[-1, 15]
learning_rate
[1e-4, 1]
0.09
XGBoost
o_estimators
[100, 10,000]
734
learning rate
[0.001, 0.5]
0.04
max_depth
[1, 10]
subsample
[0.25, 0.75]
0.46
colsample_bytre bytree
[0.05, 0.5]
0.30
colsample_byy bylevel
[0.05, 0.5]
0.31
Random Forest
estimators
[100, 10,000]
9864
max_ depth
[1. 10]
10
min hiin_samles__ split
[2, 11]
min paa__samlle_____ leaf
[1, 10]
10
14

---

Additional Experiments
2.1 Random Forest and XGBoost
In addition to LightGBM, we trained Random Forest and XGBoost (43) models for the same clinical prediction tasks, comparing single-task and multitask performance. All modalities were used for training. Results for Random Forest are shown in Table 9 XGBoost in Table TO, and LightGBM again Table for comparison.
LightGBM outperforms both Random Forest and XGBoost in most tasks. Therefore, we focused the remaining experiments (monotonicity in modalities, robustness missing modalities, and bias evaluation) on only the LightGBM models.
2.2 CLMBR
We conducted additional experiments to explore integrating past records of ED visits for patient into their EHR representations. We trained a clinical language model-based representation (CLMBR) 24 model on the structured portion of the EHR data. CLMBR treats a patient's 's longitudinal EHR as a "document" comprising sequences of diagnosis, procedure, medication, and laboratory codes. It learns representations of patient EHRs by predicting future EHR codes based only on past EHR codes. We used the EHR representations learned by CLMBB as input to LightGBM models to predict the clinical tasks (Table T2). As CLMBR is designed for structured EHR data, this experiment could only utilize the structured portion of the dataset, preventing direct comparison to other experiments using all modalities. However, the model's performance is not too far off from others in predicting revisits and disposition. Its performance is lower in predicting decompensation, but that is because doesn't 't have access the continuous monitoring and waveform modalities, which are important in predicting decompensation. Overall, this experiment demonstrates the potential in integrating patient's 's longitudinal history into learned EHR representations for improved predictive modeling, an area for future work.
Table 9: AUPRC performance of task-specific and multitask models (point estimate and 95% confidence interval) for Random Forest
Model
Task-specific model
Multitask model
Class Prevalence
Task
60 min
0.21 (0.17 0.26)
0.17 (0.14 0.21)
0.11
Decompensation
90 min
0.28 (0.24 0.34)
0.22 (0.19 0.27)
0.15
120 min
0.31 (0.27 0.37)
0.25 (0.21 0.30)
0.17
Disposition
0.83 (0.82 0.84)
0.80 (0.79 0.81)
0.37
3 days
0.09 (0.07 0.12)
0.08 (0.07 0.11)
0.05
Revisit
7 days
0.14 (0.12 0.16)
0.13 (0.12 0.16)
0.08
days
0.21 (0.19 0.23)
0.20 (0.18 0.23)
0.12
15

---

Table 10: AUPRC performance of task-specific and multitask models (point estimate and 95%
confidence interval) for XGBoost
Model
Task-specific model
Multitask model
Class Prevalence
Task
60 min
0.26 (0.21 -0333
0.23 (0.19 0.28)
0.11
Decompensation
90 min
0.31 (0.26 0.37)
0.30 (0.25 0.35)
0.15
120 min
0.28 (0.24 0.34)
0.33 (0.28 0.38)
0.17
Disposition
0.88 (0.87 0.89)
0.85 (0.84 0.86)
0.37
3 days
0.08 (0.06 - 0.11)
0.08 (0.06 0.10)
0,05
Revisit
7 days
0.14 -0.12-0.16)
0.14 (0.12 0.16)
0.08
14 days
0.21
0.22 (0.19 0.24)
0.12
Table 11: AUPRC performance of task-specific and multitask models (point estimate and 95%
confidence interval) for LightGBM
Model
Task-specific model
Multitask model
Class Prevalence
Task
60 min
0.33 (0.27 0.40)
0.25 (0.20 0.31)
0.11
Decompensation
90 min
0.35 (0.30 0.41)
0.32 (0.27 0.38)
0.15
120 min
0.40 (0.35 0.46)
0.35 (0.30 0.41)
0.17
Disposition
0.87 (0.85 0.87)
0.82 (0.81 0.83)
0.37
3 days
0.11 (0.08 0.14)
0.10 (0.08 0.13)
0,05
Revisit
7 days
0.16 (0.14 0.19)
0.16 (0.14 0.19)
0.08
14 days
0.24 (0.21 0.27)
0.25 (0.22 0.28)
0.12
Table 12: AUPRC performance for LightGBM with CLMBR embeddings
Model
CLMBR LightGBM
Class Prevalence
Task
60 min
0.19 (0.15 0.24)
0.11
Decompensation
90 min
0.26 (0.22 0.32)
0.15
120 min
0.28 (0.24 0.33)
0.17
Disposition
0.81 (0.80 0.82)
0.37
days
0.08 (0.07 0.10)
0.05
Revisit
days
0.13 -0.11-0.15)
0.08
14 days
0.22 (0.19-0.24)
0.12
16

---

Dataset Details
The distribution of demographies within the dataset can be found in table 13 For glimpse the primary data modalities, refer to sample datapoint presented in table 14
Table 13: Distribution of Demographic Groups with Counts and Relative Percentages
Demographic
Number of Visits
Percentage of Total Visits
Age
18-30
16,159
15.73%
30-40
15,684
15.27%
40-50
13,501
13.14%
50-60
15,355
14.95%
60-70
15,815
15.39%
70-80
13,272
12.92%
80-90
12,361
12.03%
Unknown
584
0.57%
Total
102,731
100.00%
Gender
Female
55,929
54.44%
Male
46,774
45.53%
Unknown
28
0.03%
Total
102,731
100.00%
Race
American Indian Alaska Native
276
0.27%
Asian
16,971
16.52%
Black African American
6,590
6.41%
Declines to State
475
0.46%
Native Hawaiian or Other Pacific Islander
2,147
2.09%
Other
34,536
33.62%
Unknown
444
0.43%
White
41,292
40.19%
Total
102,731
100.00%
Ethnicity
Declines to State
557
0.54%
Hispanic/Latino
28,181
27.4 27.43%
Non- Non-HHspaaii/NoonLLaiioo
73,404
71.45%
Unknown
589
0.57%
Total
102,731
100.00%
17

---

Table 14: Sample of Visits Data including Triage Details, Medications, Prior Diagnosis, Lab Tests,
Orders and Radiology Report
Category
Field
Value
Age
Triage_HD HR
95.0
Triage RR
12.0
Triage_SpO2
99.0
Triage Temp
36.4
Triage SBP
142.0
Visits
Triage DBP
81.0
Gender
F
Triage acuity
3-Urgent
CC Means of arrival cat
Self
Race
Asian
Ethnicity
Non-Hiipaaic/Noon-Latioo
Med_class
DIETARY SUPPLEMENT, MISCELLA NEOUS
Meds
Med sud_sshelass
Alternative Therapy Unclassified
Dx (Diagnosis)
DescCCS
HIV infection
Procedure
CBC W/O DIFF
Orders
Med
MAGNESIUM SULFATE IN WATER
GRAM/50 ML (4 %) IV PGBK
Component name Component aaboomaal mod
CALCIUM
Labs
Normal
Rads
Impression
The appendix is inflamed, but is favored to be reactive process close proximity to the dom- inant cecal/terminal iletis. Findings were dis- cussed by over the phone with on 19:27.
Table 15: Evaluation of model fairness across demographic groups Three metrics SPD, DI, and EOD averaged across the 7 experiments have been shown. In the ideal equitable scenario, we expect and aim for DI=1 and SPD, BOD (Adults are defined as individuals between 18-65 years age. Individuals older than 65 have been categorized as Seniors)
Minority
Majority
SPD
DI
EOD
Seniors
Adults
0.09
1.14
0.04
Hispanic/Latino
Non-HHispani/Non-Latiioo
0.00
1.01
0.01
Other
White
-0.02
0.98
0.02
Asian
White
-0.02
0.96
-0.01
Asian
Other
0.00
1.00
-0.03
Black or African American
White
0.09
1.15
0.08
Black or African American
Other
0.11
1.18
0.07
Black or African American
Asian
0.12
1.20
0.09
Native Hawaiian Other Pacific Islander
White
-0.02
0.97
0.05
Native Hawaiian or Other Pacific Islander
Other
-0.01
0.99
0.04
Native Hawaiian or Other Pacific Islander
Asian
0.00
1.01
0.06
Native Hawaiian or Other Pacific Islander
Black or African American
-0.12
0.84
-0.03
Male
Female
0.03
1.04
0.05
18


Reviews:
[{"review_id": "YLKjP5r70l", "reviewer": "Reviewer_XBLz", "summary": "", "questions": "", "limitations": "The authors do a good job of putting up a benchmark. But the authors fail to convince the reviewer that they provide a good benchmark. ", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The first useful Multimodal Clinical Benchmark for Emergency Care. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "nZ91xt1qTd", "reviewer": "Reviewer_2TbH", "summary": "", "questions": "", "limitations": "The authors have adequately addressed the limitations of their work and potential negative societal impact. They acknowledge the limitations of the current benchmark including the socioeconomic status bias issue and relatively simple LightGBM model used in the pipeline.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- A large-scale multi-modal dataset consisting of time-series, text, waveform, .. data. Compared to MIMIC-IV, this dataset has more modalities (more waveform data).\n- \"current ED benchmarks are not suitable for generalist medical AI\" The benchmark proposed more evaluation metrics beyond prediction performance, considered the multi-modality characteristics and the fairness issue.\n- Comprehensive benchmarks on all proposed metrics and at timescales from minutes to days. (Table 1,2,3,4)\n- Inspired by the development of NLP field, the prompting strategy for multitask learning is somehow novel for healthcare data modeling.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "8FBR1S007x", "reviewer": "Reviewer_RbWs", "summary": "", "questions": "", "limitations": "+ It appears that the dataset used for the study lacks demographic information, including factors such as the percentage of different ethnicities, age groups, and genders. This omission of crucial demographic variables could potentially introduce bias in the observations, particularly when attempting to make generalizations from the overall dataset.\n+ In the experimental section, important details are missing, such as information regarding the learning rate, weight decay, and number of epochs used. It is possible that the subpar results obtained by the baseline models could be attributed to inadequate hyperparameter tuning. It would be beneficial for the authors to provide further elaboration on this aspect to enhance the understanding of the experimental setup.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ The Multimodal Clinical Benchmark for Emergency Care (MC-BEC) is introduced as a comprehensive evaluation framework for foundation models in Emergency Medicine. With a dataset comprising 72,982 patients and their multiple visits, MC-BEC offers a substantial amount of data for analysis.\n+ Each data entry in MC-BEC includes various field information such as prior diagnoses, medications, and lab results. This incorporation of multiple modalities enriches the dataset, further enhancing its comprehensiveness and breadth.\n+ In addition to its rich data collection, MC-BEC provides results of downstream tasks encompassing multiple timescales. This feature facilitates the training of models in predicting and analyzing real-world Electronic Health Records (EHR) data, enabling more accurate insights into clinical scenarios.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "rOjMpP8D11", "reviewer": "Reviewer_PuQz", "summary": "", "questions": "", "limitations": "Updated review:  While the dataset offers the opportunity to investigate multiple timescale, the prediction tasks are limited to patient decompensation (within minutes), disposition (within hours), and ED revisit (within days).\n\nin my original review I raised a concern that the dataset was collected over two years in which COVID prevalence in emergency rooms was quite high (2020-2022), and thus, it is unclear whether it will be a useful benchmark for non-COVID emergency care.  Interactively the authors have clarified that that ~6.3% of cases were COVID-related. \n\nConfidence intervals for model performance are given, but are so large as to call into question the model's performance.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The size of the dataset is a strength, as well as the comprehensive, multi-modal nature of the data (triage information, prior diagnoses and medications, continuously measured vital signs, electrocardiogram and photoplethysmograph waveforms, orders placed and medications administered throughout the visit, free-text reports of imaging studies, and information on ED diagnosis, disposition, and subsequent revisits.)\n \n\nGood definitions of the three predicted metrics (decompensation, disposition, and revisit) are provided.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors propose a Multimodal Clinical Benchmark for Emergency Care (MC-BEC). The authors provide performance baselines for each prediction task to enable the evaluation of multimodal, multi-task models.
Overall the reviewer's evaluation is positive, although some limitations in the demographic information and details about the evaluation.
Decision: Accept (Poster)