SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Prediction performance. We compare the performance of models trained on a single task (decompensation, disposition, or revisit) with a model trained simultaneously on all three prediction tasks. Table 2 presents the model performance as assessed by AUPRC using all available data modalities. Overall, task-specific models perform better than the multitask model in decompensation and disposition predictions, but similarly in revisit predictions. For decompensation prediction, the task-specific model has AUPRCs of 0.33, 0.35 and 0.40 (for 60-, 90- and 120-minute prediction windows), while AUPRCs for the multitask model are 0.25, 0.32, and 0.35. Disposition is predicted with AUPRC 0.87 for the task-specific model and 0.82 for the multitask model. When predicting revisit at 3, 7 and 14-day time windows, we report AUPRC scores of 0.11, 0.16, and 0.24 for the task-specific model, and 0.10, 0.16, 0.25 for the multitask prediction model. The prevalence of the positive class in the dataset is provided as baseline AUPRC [41]. Our results indicate slightly worse performance with a multitask prediction model which conforms with the assumption that the single-task prediction models will be better specialized for their specific task. However, the overall performance gap between single and multitask approaches is small. This suggests potential for more advanced multitask modeling to match or surpass the single-task results by better leveraging commonalities and differences between tasks. Recent work has shown promise for sophisticated multitask architectures surpassing single-task models in clinical predictions. For instance, a multitask channel-wise LSTM exceeded single-task models in predicting in-hospital mortality[13]. Future research can use MC-BEC as an avenue to improve upon multitask models.
Monotonicity in modalities. To assess monotonicity in modalities, we train the multitask prediction model on an increasing subset of data modalities. We start with a model trained only on categorical and numeric features, then incrementally add features from diagnoses, medications, orders, lab results, radiology reports, continuous monitoring, and waveforms. Table 3 shows the AUPRC of
Table 3: AUPRC of multitask models with incrementally increasing testing and training modalities (point estimate and  $95\%$  confidence interval). The modality concordance index measures monotonicity of prediction performance in modalities.
Model	Decompensation	Disposition	Revisit
60 min	90 min	120 min	3 day	7 day	14 day
Modality added
+ Categorical/numeric	0.23 (0.18 - 0.29)	0.30 (0.25 - 0.35)	0.33 (0.28 - 0.39)	0.71 (0.69 - 0.72)	0.10 (0.08 - 0.13)	0.15 (0.13 - 0.18)	0.25 (0.22 - 0.28)
+ Diagnoses	0.23 (0.18 - 0.29)	0.30 (0.25 - 0.35)	0.33 (0.28 - 0.39)	0.70 (0.68 - 0.72)	0.09 (0.07 - 0.11)	0.15 (0.13 - 0.18)	0.24 (0.21 - 0.27)
+ Medications	0.22 (0.18 - 0.28)	0.29 (0.24 - 0.34)	0.32 (0.28 - 0.38)	0.71 (0.69 - 0.72)	0.09 (0.07 - 0.12)	0.15 (0.13 - 0.18)	0.24 (0.22 - 0.27)
+ Orders	0.18 (0.15 - 0.24)	0.24 (0.21 - 0.30)	0.27 (0.23 - 0.32)	0.79 (0.78 - 0.80)	0.09 (0.08 - 0.12)	0.15 (0.13 - 0.17)	0.24 (0.22 - 0.27)
+ Labs	0.16 (0.13 - 0.20)	0.22 (0.19 - 0.26)	0.25 (0.21 - 0.29)	0.79 (0.78 - 0.81)	0.09 (0.08 - 0.12)	0.15 (0.13 - 0.18)	0.24 (0.22 - 0.27)
+ Radiology	0.15 (0.13 - 0.19)	0.21 (0.18 - 0.25)	0.24 (0.21 - 0.29)	0.79 (0.78 - 0.80)	0.10 (0.08 - 0.12)	0.16 (0.14 - 0.18)	0.25 (0.22 - 0.28)
+ Monitoring	0.24 (0.19 - 0.29)	0.30 (0.25 - 0.35)	0.33 (0.28 - 0.39)	0.80 (0.79 - 0.82)	0.10 (0.08 - 0.12)	0.16 (0.14 - 0.18)	0.25 (0.22 - 0.28)
+ Waveforms	0.25 (0.20 - 0.31)	0.32 (0.27 - 0.38)	0.35 (0.30 - 0.41)	0.82 (0.81 - 0.83)	0.10 (0.08 - 0.13)	0.16 (0.14 - 0.19)	0.25 (0.22 - 0.28)
Modality concordance index	0.50	0.46	0.43	0.89	0.71	0.79	0.82
Table 4: AUPRC for multitask models with one modality missing in test data, demonstrating robustness to missing modalities (point estimate and  $95\%$  confidence interval).
Model	Decompensation	Disposition	Revisit
60 min	90 min	120 min	3 day	7 day	14 day
All modalities	0.25 (0.20 - 0.31)	0.32 (0.27 - 0.38)	0.35 (0.30 - 0.41)	0.82 (0.81 - 0.83)	0.10 (0.08 - 0.13)	0.16 (0.14 - 0.19)	0.25 (0.22 - 0.28)
Missing modality
- Categorical/numeric	0.26 (0.21 - 0.32)	0.32 (0.27 - 0.38)	0.35 (0.30 - 0.41)	0.81 (0.80 - 0.83)	0.06 (0.05 - 0.07)	0.10 (0.09 - 0.11)	0.17 (0.16 - 0.19)
- Diagnoses	0.25 (0.20 - 0.31)	0.32 (0.27 - 0.38)	0.35 (0.30 - 0.41)	0.82 (0.81 - 0.83)	0.10 (0.08 - 0.13)	0.16 (0.14 - 0.09)	0.25 (0.22 - 0.28)
- Medications	0.25 (0.20 - 0.31)	0.32 (0.27 - 0.37)	0.35 (0.30 - 0.40)	0.82 (0.81 - 0.83)	0.08 (0.07 - 0.11)	0.13 (0.11 - 0.15)	0.23 (0.21 - 0.26)
- Orders	0.26 (0.21 - 0.32)	0.34 (0.28 - 0.40)	0.37 (0.32 - 0.43)	0.73 (0.72 - 0.75)	0.07 (0.05 - 0.09)	0.11 (0.10 - 0.13)	0.23 (0.21 - 0.25)
- Labs	0.26 (0.21 - 0.33)	0.33 (0.28 - 0.39)	0.36 (0.31 - 0.42)	0.81 (0.80 - 0.82)	0.08 (0.06 - 0.10)	0.13 (0.11 - 0.15)	0.23 (0.21 - 0.25)
- Radiology	0.25 (0.20 - 0.31)	0.32 (0.27 - 0.37)	0.35 (0.30 - 0.40)	0.80 (0.79 - 0.81)	0.07 (0.06 - 0.10)	0.12 (0.11 - 0.14)	0.23 (0.21 - 0.25)
- Monitoring	0.14 (0.12 - 0.17)	0.20 (0.18 - 0.24)	0.23 (0.20 - 0.27)	0.82 (0.81 - 0.83)	0.08 (0.06 - 0.10)	0.13 (0.11 - 0.15)	0.23 (0.21 - 0.25)
- Waveforms	0.24 (0.19 - 0.29)	0.31 (0.26 - 0.36)	0.34 (0.29 - 0.39)	0.82 (0.81 - 0.83)	0.08 (0.06 - 0.10)	0.13 (0.11 - 0.15)	0.23 (0.21 - 0.26)
Max AUPRC difference	0.12	0.13	0.14	0.09	0.04	0.06	0.08
models trained with increasing numbers of modalities, along with the modality concordance index. Disposition prediction demonstrates strong monotonicity in modalities with a modality concordance index of 0.86. This indicates that disposition prediction benefits from incorporating more data modalities. Revisit predictions demonstrate moderate monotonicity, with concordance indices of 0.71, 0.79, and 0.82 for the 3, 7, and 14-day windows. Decompassion predictions show the worst monotonicity among all tasks, with notably decreased performance when adding orders, labs, and radiology reports data. This poor monotonicity in modalities is reflected in low concordance indices of 0.50, 0.46, and 0.43 for the 60, 90, and 120-minute windows. However, the performance does increase when continuous monitoring and waveforms are added, which aligns with the expectation that continuous monitoring and waveforms of vital signs are predictive of impending decompensation events.
Robustness against missing modalities. We assess the robustness of the multitask benchmark model against missing modalities during inference by dropping individual modalities and comparing the difference in AUPRC between using all modalities versus missing modalities (Table 4). Although revisit predictions do not have the largest maximum AUPRC difference, with values of 0.04, 0.06, and 0.08 for the 3, 7, and 30-day windows, they are more likely to see a performance decrease when missing modalities. Specifically, revisit predictions show notable performance drops when all modalities except diagnoses are missing, compared to the substantial drops for decompensation predictions only when continuous monitoring data is missing and for disposition predictions only when orders data is missing. Furthermore, the performance gap widens in the worst case of missing modalities over the longer time windows for both revisit and decompensation predictions, with maximum AUPRC differences of 0.12, 0.13, and 0.14 for the latter. Qualitatively, the impact on model performance when omitting modalities varies significantly depending on the prediction task. Overall, our results suggest that the benchmark predictions are reasonably robust to missing modalities during inference.
Bias. Lastly, in table 5, we report mean TPR differences for the multitask prediction model when applied to different demographic groups, specifically based on patient age, gender, race, and ethnicity. The greatest absolute disparities in mean TPR emerged when comparing different racial groups, with a mean TPR difference of 0.11 for 3-day revisit prediction, followed by a mean TPR difference of 0.09 for 7-day revisit prediction. Notable model bias was also observed between gender groups for 90 and 120-minute decompensation predictions, with TPR differences of 0.08 and 0.09. Additionally, there was a sizable TPR difference of 0.07 between age groups for 60-minute decompensation predictions.
Table 5: Evaluation of model bias across demographic groups. True positive rate (TPR) and absolute TPR difference between different groups are reported below.
Demographic Group	Decompensation	Disposition	Revisit
60 min	90 min	120 min	3 day	7 day	14 day
Age	< 55	0.87	0.86	0.88	0.93	0.69	0.75	0.88
>= 55	0.94	0.91	0.92	0.97	0.65	0.79	0.92
TPR difference	0.07	0.05	0.05	0.04	0.04	0.03	0.04
Gender	Male	0.92	0.93	0.94	0.96	0.68	0.79	0.91
Female	0.88	0.85	0.85	0.95	0.67	0.74	0.87
TPR difference	0.05	0.08	0.09	0.01	0.01	0.04	0.04
Race	White	0.88	0.88	0.88	0.96	0.63	0.77	0.88
Black/African American	1.00	0.90	0.95	0.95	0.82	0.90	0.94
Asian	0.88	0.87	0.88	0.96	0.64	0.73	0.85
Native Hawaiian/Other Pacific Islander	1.00	0.86	1.00	0.93	0.80	0.86	0.81
Other	0.90	0.90	0.91	0.95	0.70	0.74	0.90
TPR difference	0.07	0.02	0.06	0.01	0.11	0.09	0.06
Ethnicity	Non-Hispanic/Non-Latino	0.89	0.88	0.89	0.96	0.68	0.77	0.88
Hispanic/Latino	0.92	0.91	0.92	0.94	0.66	0.75	0.91
TPR difference	0.03	0.03	0.03	0.01	0.02	0.02	0.03
We also present supplementary fairness metrics like Statistical Parity Difference (SPD), Disparate Impact (DI), and Equal Opportunity Difference (EOD) between minority and majority cohorts in Table 15 in the Appendix. Models applied to the MC-BEC benchmark should be attentive to potential biases as assessed by our framework, and any identified biases must be carefully considered before deployment to ensure fairness and equity.
While the benchmark provides valuable insights, MC-BEC has limitations. The benchmark does not assess the potential of foundation models in automating novel clinical tasks, such as summarizing all data obtained throughout a visit into an easy-to-understand report for the provider or patient. Future iterations of the benchmark should include novel tasks specific to the ED environment to further highlight the capabilities of clinical foundation models. Although the benchmark evaluates bias on measured patient characteristics, it does not account directly for socioeconomic status. Due to privacy and legal concerns, socioeconomic information, such as income or education level, are not included in the underlying dataset. However, the dataset does include visit payor information, which is correlated with socioeconomic status (for instance, patients whose visits are paid by Medicaid are more likely to have lower socioeconomic status). Finally, we used a relatively simple baseline model to demonstrate the benchmark. Although this choice may limit the model's performance, the primary contribution of this work is the proposal of a new benchmark using a novel multimodal dataset. LightGBM, despite its simplicity, remains a commonly used and highly performant approach for many clinical tasks. Therefore, the baseline model serves not only as a demonstration of the benchmark but also as a reasonable reference for future models.
Our work introduces a novel benchmark (MC-BEC), enabling robust multitask evaluation of prediction models for Emergency Medicine, using a unique multimodal clinical dataset. We evaluate multiple baseline models, and compare the performance of multitask and task-specific models. We present an evaluation schema suited to the demands of multitask clinical predictions on complex multimodal data, which assesses a model's ability to make full use of multimodal data, to make robust
predictions despite missing data, and to make fair and unbiased predictions across demographic groups.
The benchmark models introduced in this work employed an explicit data featurization step, which highlights the challenges of working with diverse, complex and data-rich modalities. We expect future models applied to MC-BEC to leverage alternative approaches, such as with deep end-to-end trainable networks, and integration of a temporal dimension both within and across visits. MC-BEC is centered on predictions of decompensation, disposition, and revisit because these encompass the entire time scale of ED visit prediction tasks. However, future work in benchmarks for generalizable ED prediction models should also aim to evaluate prediction over a non-discrete set of tasks, such as prediction of therapeutic complications or need for post-discharge follow-up. This would enable more granular and comprehensive predictions that could help providers to target and prioritize interventions.
[1] Nathan Walzl, Jessica Jameson, John Kinsella, and David J Lowe. Ceilings of treatment: a qualitative study in the emergency department. BMC Emergency Medicine, 19(1):1-8, 2019.
[2] Feng Xie, Jun Zhou, Jin Wee Lee, Mingrui Tan, Siqi Li, Logasan S/O Rajnthern, Marcel Lucas Chee, Bibhas Chakraborty, An-Kwok Ian Wong, Alon Dagan, et al. Benchmarking emergency department prediction models with machine learning and public electronic health records. Scientific Data, 9(1):658, 2022.
[3] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV (version 0.4). https://doi.org/10.13026/a3wn-hq05, 2020. PhysioNet.
[4] Michael Wornow, Rahul Thapa, Ethan Steinberg, Jason Fries, and Nigam Shah. Ehrshot: An ehr benchmark for few-shot evaluation of foundation models. arXiv preprint arXiv:2307.02028, 2023.
[5] Martin Faltys, Moritz Zimmermann, Xin Lyu, Marco Hüser, Stephanie Hyland, Gunnar Ratsch, and Tobias Merz. Hirid, a high time-resolution icu dataset (version 1.1.1). PhysioNet, 2021.
[6] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259-265, 2023.
[7] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselet, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[8] Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A Pfeffer, Jason Fries, and Nigam H Shah. The shaky foundations of clinical foundation models: A survey of large language models and foundation models for emrs. arXiv preprint arXiv:2303.12961, 2023.
[9] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12695-12705, 2020.
[10] Shirly Wang, Matthew BA McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C Hughes, and Tristan Naumann. Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-iii. In Proceedings of the ACM conference on health, inference, and learning, pages 222–235, 2020.
[11] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[12] Sanjay Purushotham, Chuizheng Meng, Zhengping Che, and Yan Liu. Benchmarking deep learning models on large healthcare datasets. Journal of biomedical informatics, 83:112-134, 2018.
[13] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):96, 2019.
[32] Laleh Seyyed-Kalantari, Haoran Zhang, Matthew B. A. McDermott, Irene Y. Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine, 27(12):2176-2182, Dec 2021.
[33] Sameer Sundrani, Julie Chen, Boyang Tom Jin, Zahra Shakeri Hossein Abad, Pranav Rajpurkar, and David Kim. Predicting patient decompensation from continuous physiologic monitoring in the emergency department. npj Digital Medicine, 6(1):60, Apr 2023.
[34] Woo Suk Hong, Adrian Daniel Haimovich, and R. Andrew Taylor. Predicting hospital admission at emergency department triage using machine learning. PLOS ONE, 13(7):1-13, 07 2018.
[35] Yuval Barak-Corren, Pradip Chaudhari, Jessica Perniciaro, Mark Waltzman, Andrew M. Fine, and Ben Y. Reis. Prediction across healthcare settings: a case study in predicting emergency department disposition. npj Digital Medicine, 4(1):169, Dec 2021.
[36] Woo Suk Hong, Adrian Daniel Haimovich, and Richard Andrew Taylor. Predicting 72-hour and 9-day return to the emergency department using machine learning. JAMIA Open, 2(3):346-352, 07 2019.
[37] Annamalai Natarajan, Yale Chang, Sara Mariani, Asif Rahman, Gregory Boverman, Shruti Vij, and Jonathan Rubin. A wide and deep transformer neural network for 12-lead ecg classification. In 2020 Computing in Cardiology, pages 1-4, 2020.
[38] An Yan, Julian McAuley, Xing Lu, Jiang Du, Eric Y Chang, Amilcare Gentili, and Chun-Nan Hsu. Radbert: Adapting transformer-based language models to radiology. *Radiology: Artificial Intelligence*, 4(4):e210258, 2022.
[39] Prajjwal Bhargava, Aleksandr Drozd, and Anna Rogers. Generalization in nli: Ways (not) to go beyond simple heuristics, 2021.
[40] Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR, abs/1908.08962, 2019.
[41] Takaya Saito and Marc Rehmsmeier. The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. *PloS one*, 10(3):e0118432, 2015.
[42] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.
[43] Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, et al. Xgboost: extreme gradient boosting. R package version 0.4-2, 1(4):1-4, 2015.
1 Training Details
The dataset was divided into training, testing, and validation sets, as shown in Table 6. During training, we applied sample weights to visits based on the frequency of the label classes. Specifically, the weight for each visit was directly correlated with the frequency of the predominant class and inversely correlated with the frequency of its own class. The distribution of labels is presented in Table 7. It is worth noting that the prevalence of revisits was calculated only among visits with a discharge disposition, since revisits are only relevant for patients discharged from the ED.
Table 6: Distribution of Dataset into Train, Validation, and Test Sets
Dataset	Number of Samples	Percentage (%)
Train	82,438	80.25
Validation	10,235	9.96
Test	10,058	9.79
Total	102,731	100
Table 7: Label distribution for different tasks
Category	Task	Class Prevalence
Disposition	n/a	0.37
Revisit	3 days	0.05
7 days	0.08
14 Days	0.12
Decompensation	60 mins	0.11
90 mins	0.15
120 mins	0.17
Hyperparameter Tuning
We employed the Tree-structured Parzen Estimator (TPE) for hyperparameter tuning, as implemented in the Optuna optimization framework [42]. TPE, a Bayesian optimization approach, leverages outcomes from preceding trials to inform the selection of subsequent hyperparameter values. In this framework, Optuna adeptly pinpoints areas within the hyperparameter space anticipated to enhance the objective function's value. For each model, we executed 20 trials using the hyperparameter spaces delineated in Table 8.
Table 8: Search space used for hyperparameter tuning and the optimal values for the multitask model trained on all data modalities
Hyperparameters	Search Space	Optimal Values
LightGBM
lambda_11	[1e-8, 10]	1.84
lambda_12	[1e-8, 10]	0.07
num_leaves	[2, 306]	134
feature_fraction	[0.4, 1.0]	0.58
bagging_fraction	[0.4, 1.0]	0.79
bagging_freq	[1, 7]	4
max_depth	[-1, 15]	6
learning_rate	[1e-4, 1]	0.09
XGBoost
n_estimators	[100, 10,000]	734
learning_rate	[0.001, 0.5]	0.04
max_depth	[1, 10]	7
subsample	[0.25, 0.75]	0.46
colsample_bytree	[0.05, 0.5]	0.30
colsample_bylevel	[0.05, 0.5]	0.31
Random Forest
n_estimators	[100, 10,000]	9864
max_depth	[1, 10]	10
min_samples_split	[2, 11]	2
min_samples_leaf	[1, 10]	10
2 Additional Experiments
2.1 Random Forest and XGBoost
In addition to LightGBM, we trained Random Forest and XGBoost[43] models for the same clinical prediction tasks, comparing single-task and multitask performance. All modalities were used for training. Results for Random Forest are shown in Table 9, XGBoost in Table 10, and LightGBM again in Table 11 for comparison.
LightGBM outperforms both Random Forest and XGBoost in most tasks. Therefore, we focused the remaining experiments (monotonicity in modalities, robustness to missing modalities, and bias evaluation) on only the LightGBM models.
2.2 CLMBR
We conducted additional experiments to explore integrating past records of ED visits for a patient into their EHR representations. We trained a clinical language model-based representation (CLMBR) [24] model on the structured portion of the EHR data. CLMBR treats a patient's longitudinal EHR as a "document" comprising sequences of diagnosis, procedure, medication, and laboratory codes. It learns representations of patient EHRs by predicting future EHR codes based only on past EHR codes. We used the EHR representations learned by CLMBR as input to LightGBM models to predict the clinical tasks (Table 12). As CLMBR is designed for structured EHR data, this experiment could only utilize the structured portion of the dataset, preventing direct comparison to other experiments using all modalities. However, the model's performance is not too far off from others in predicting revisits and disposition. Its performance is lower in predicting decompensation, but that is because it doesn't have access to the continuous monitoring and waveform modalities, which are important in predicting decompensation. Overall, this experiment demonstrates the potential in integrating a patient's longitudinal history into learned EHR representations for improved predictive modeling, an area for future work.
Table 9: AUPRC performance of task-specific and multitask models (point estimate and  $95\%$  confidence interval) for Random Forest
Task\Model	Task-specific model	Multitask model	Class Prevalence
Decompensation	60 min	0.21 (0.17 - 0.26)	0.17 (0.14 - 0.21)	0.11
90 min	0.28 (0.24 - 0.34)	0.22 (0.19 - 0.27)	0.15
120 min	0.31 (0.27 - 0.37)	0.25 (0.21 - 0.30)	0.17
Disposition	0.83 (0.82 - 0.84)	0.80 (0.79 - 0.81)	0.37
Revisit	3 days	0.09 (0.07 - 0.12)	0.08 (0.07 - 0.11)	0.05
7 days	0.14 (0.12 - 0.16)	0.13 (0.12 - 0.16)	0.08
14 days	0.21 (0.19 - 0.23)	0.20 (0.18 - 0.23)	0.12
Table 10: AUPRC performance of task-specific and multitask models (point estimate and  $95\%$  confidence interval) for XGBoost
Task\Model	Task-specific model	Multitask model	Class Prevalence
Decompensation	60 min	0.26 (0.21 - 0.33)	0.23 (0.19 - 0.28)	0.11
90 min	0.31 (0.26 - 0.37)	0.30 (0.25 - 0.35)	0.15
120 min	0.28 (0.24 - 0.34)	0.33 (0.28 - 0.38)	0.17
Disposition	0.88 (0.87 - 0.89)	0.85 (0.84 - 0.86)	0.37
Revisit	3 days	0.08 (0.06 - 0.11)	0.08 (0.06 - 0.10)	0.05
7 days	0.14 (0.12 - 0.16)	0.14 (0.12 - 0.16)	0.08
14 days	0.21 (0.19 - 0.24)	0.22 (0.19 - 0.24)	0.12
Table 11: AUPRC performance of task-specific and multitask models (point estimate and  $95\%$  confidence interval) for LightGBM
Task\Model	Task-specific model	Multitask model	Class Prevalence
Decompensation	60 min	0.33 (0.27 - 0.40)	0.25 (0.20 - 0.31)	0.11
90 min	0.35 (0.30 - 0.41)	0.32 (0.27 - 0.38)	0.15
120 min	0.40 (0.35 - 0.46)	0.35 (0.30 - 0.41)	0.17
Disposition	0.87 (0.85 - 0.87)	0.82 (0.81 - 0.83)	0.37
Revisit	3 days	0.11 (0.08 - 0.14)	0.10 (0.08 - 0.13)	0.05
7 days	0.16 (0.14 - 0.19)	0.16 (0.14 - 0.19)	0.08
14 days	0.24 (0.21 - 0.27)	0.25 (0.22 - 0.28)	0.12