Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML
Authors: Robin van de Water
Hendrik Nils Aurel Schmidt
Paul Elbers
Patrick Thoral
Bert Arnrich
Patrick Rockenschaub
Venue: ICLR 2024 poster (2024)
Keywords: ICU
Intensive Care Unit
EHR
ML
Time Series
Patient Monitoring
Clinical ML
Benchmark
Multi-Center
MIMIC
eICU
HiRID
AmsterdamUMCdb
Primary area: datasets and benchmarks
TL;DR: We introduce Yet Another ICU Benchmark: a flexible, holistic framework for the standardization of clinical prediction model experiments.

Abstract: Medical applications of machine learning (ML) have experienced a surge in popularity in recent years. Given the abundance of available data from electronic health records, the intensive care unit (ICU) is a natural habitat for ML. Models have been proposed to address numerous ICU prediction tasks like the early detection of complications. While authors frequently report state-of-the-art performance, it is challenging to verify claims of superiority. Datasets and code are not always published, and cohort definitions, preprocessing pipelines, and training setups are difficult to reproduce. This work introduces Yet Another ICU Benchmark (YAIB), a modular framework that allows researchers to define reproducible and comparable clinical ML experiments; we offer an end-to-end solution from cohort definition to model evaluation. The framework natively supports most open-access ICU datasets (MIMIC III/IV, eICU, HiRID, AUMCdb) and is easily adaptable to future ICU datasets. Combined with a transparent preprocessing pipeline and extensible training code for multiple ML and deep learning models, YAIB enables unified model development, transfer, and evaluation. Our benchmark comes with five predefined established prediction tasks (mortality, acute kidney injury, sepsis, kidney function, and length of stay) developed in collaboration with clinicians. Adding further tasks is straightforward by design. Using YAIB, we demonstrate that the choice of dataset, cohort definition, and preprocessing have a major impact on the prediction performance — often more so than model class — indicating an urgent need for YAIB as a holistic benchmarking tool. We provide our work to the clinical ML community to accelerate method development and enable real-world clinical implementations.

Published as a conference paper at ICLR 2024
YET ANOTHER ICU BENCHMARK: A FLEXIBLE MULTI-CENTER FRAMEWORK FOR CLINICAL ML
Robin de Water @
Hendrik Schmidt Paul Elbers
Patrick Thoral @
Bert Arnrich Patrick Rockenschaub Hasso Plattner Institute, University Potsdam, Germany
Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands Lab for AI in Medicine, Charité Universifätsmediiin Berlin, Germany
ABSTRACT
Medical applications of machine learning (ML) have experienced a surge in pop- ularity in recent years. The intensive care unit (ICU) is a natural habitat for ML given the abundance of available data from electronic health records. Models have been proposed to address numerous ICU prediction tasks like the early detection of complications. While authors frequently report state-of-the-art performance, it is challenging verify claims of superiority. Datasets and code are often not published, and cohort definitions, preprocessing pipelines, and training setups are difficult to reproduce. This work introduces Yet Another ICU Benchmark (YAIB), modular framework that allows researchers to define reproducible and comparable clinical ML experiments; we offer an end-to-end solution from cohort definition to model evaluation. The framework natively supports most open-access ICU datasets (MIMIC III/V, eICU, HiRID, AUMCdb) and is easily adaptable to future and custom ICU datasets. Combined with transparent preprocessing pipeline and extensible training code for multiple ML and deep learning models, YAIB enables unified model development, transfer, and evaluation. Our benchmark comes with five predefined established prediction tasks (mortality, acute kidney injury, sepsis, kidney function, and length of stay) developed in collaboration with clinicians. Adding further tasks is straightforward by design. Using YAIB, we demonstrate that the choice of dataset, cohort definition, and preprocessing have a major impact on the prediction performance, often more so than model class, indicating an urgent need for YAIB holistic benchmarking tool. We provide our work to the clinical ML community to accelerate method development and enable real-world implementations.
Software Repository: https: http:::///gghhhbbcco/oeoo/eeee wtt..b..com/rrandeeattt///1E
INTRODUCTION
The intensive care unit (ICU) has long been a focus for research into data-driven decision support, owing to the impact of medical decisions well as the breadth and depth of data collected in this setting (Johnson et al. 2017). The COVID-19 pandemic confirmed the need for reliable machine learning bbsed clinical decision support that can alert healthcare professionals to worsening patient states, help them make a clinical diagnosis, or recommend treatment (Medic et al. 2019).
Despite steep increase in the number of published ICU prediction models (Shillan al. 2019) hardly any have made their way into clinical practice (Eini-Porat et al.) 2022 Fleuren et al. 2020b). A major obstacle to translation is an ongoing lack of comparability and reproducibility (Johnson et al.) 2017). By using custom datasets and definitions, preprocessing pipelines, and evaluation schemes, the benefits of novel models are conflated with differences between patient case mix, task definitions, and cohort selection (Sarwar et al. 2023 Kelly et al.) (201)). Reviewing models for early prediction of sepsis, for example. Moor et al. (221b) found that the definition of sepsis, the time of prediction,
and the available features differed substantially between the 22 included studies; similar results
were found in an earlier review (Fleuren et al., 2020a). Even among studies from the same research
"Corresponding author email: robin. wabbn.aanddeattr@@pp..d.

---

Published as a conference paper at ICLR 2024
group (Hyland 2020:
Yēche al., 2022), cohort definitions may vary substantially, precluding meaningful comparison. Inconsistencies in imputation and feature extraction further complicate an objective evaluation of research progress.
The increasing availability of open-access ICU datasets a first, important step towards urgently needed model comparability (Sauer et al.) 2022a). However, models derived from the same dataset may still vary considerably in their analytical setup. Earlier work has therefore created benchmarks that establish single pipeline for preprocessing and modeling (Yèche et al.) 2022; Harutyunyan et al., 2019). These benchmarks are hard-coded for a given dataset, following proprietary formats and supporting a limited, fixed set of tasks. Extending an existing benchmark to include new datasets or tasks requires changes to the benchmark's 's often lightly documented source code. Despite the existence of multiple benchmarks, new models are therefore rarely evaluated on more than one dataset or do not use any benchmark (Shillan et al.) 2019).
We address this gap by providing Yet Another ICU Benchmark (YAIB) as modular multi-dataset framework specifically designed for extensibility. Building recent work to harmonize ICU data (Bennett et al., (023) (i.e., match time-scale, clinical definitions, and units across datasets), we standardize the entire modeling workflow from the definition of clinical concepts (a medical abstraction to facilitate patient care) and data extraction to model fitting and evaluation across several established open-source ICU datasets (Sauer et al. 2022a). We provide a predefined set of common prediction tasks, developed in collaboration with clinical intensivists, that can be easily extended to fit user needs. Our benchmark, by default, provides endpoint prediction for ICU mortality, sepsis (Singer et al. 2016), acute kidney injury (AKI) (KDIGO) (012)) kidney function (KF), and length of stay (LoS). With this work, we aim (1) dramatically reduce the overhead of developing new ICU prediction methods, (2) provide a transparent, open-source, and reproducible definition of experiments, and (3) unify ML workflows for ICU prediction modeling.
2 RELATED WORK
Our work builds upon several previous efforts to harmonize the definition, development, and evalua- tion of ICU prediction models. YAIB combines these existing works in a novel, end-to-end fashion to enable quick, reproducible, and comparable model development.
Publicly available ICU datasets Our benchmark currently supports four established ICU datasets (Sauer al.] (202b):: the Medical Information Mart for Intensive Care (MIMIC) version III (Johnson 2016) and IV (Johnson et al. 2023), the eICU Collaborative Research Database (eICU) (Pollard al. 2018), the High Time Resolution ICU Dataset (HiRID) (Hyland) 2020), and the Amsterda- mUMCdb (AUMCdb) (Thoral et al. 2021). These datasets contain similar data items but differ in size and scope [Table 13). Together, they cover 334,812 ICU stays. We plan to integrate two recently released ICU datasets in the future (Rodemund et al. 2023 Jin et al. 2023).
et al.
Benchmarks To improve comparability between models trained on these ICU datasets, sev- eral benchmarks benchmark- like applications have been developed (Table T). These solutions mainly differ in the tasks and models they support. Notably, existing benchmarks heavily focus on benchmarking results, often hardcoding key steps like data extraction, task definition, preprocessing, feature generation, and sometimes model training. While they may reduce implementation overhead when evaluating new ML approaches. present benchmarks are difficult adapt user requirements. Core code base changes are often necessary if the users' problems do not fit into the provided task definitions. Even advanced modeling frameworks such as Jarrett et al. (202T) and (Saveliev & van der Schaar (203)) share this weakness, as they do not support reproducible data extraction or task definitions; thus, they do not provide an end-to-end solution like YAIB.
Multi-dataset support Due to considerable heterogeneity in data structure, existing benchmarks tend to focus on single datasset, most frequently MIMIC-III As MIMIC-III also has a large existing user base (Syed et al., 2021). it thus often becomes the default choice (Shillan et al. 2019). This has potentially resulted in self-enforeing bias towards the MIMIC-III datasets, which represent a single-center US population. Even frameworks that work with its successor MIMIC-IV lack backward compatibility (Mandyam et al. |021 Gupta et al. 2022) ). Among the few multi multi-dataset solutions, (Tang et al. 2020) operates both eICU and MIMIC-III, but lacks many of the model architectures found in others works and does no longer appear to be in active development. Oliver et al. (2023)
provides hardcoded pipeline to combine several datasets without providing cohort definitions,
2

---

Published as a conference paper at ICLR 2024
TABLE 1: Comparison existing benchmarks and YAIB ICU data, ordered by publication date.
Sheikhalishahi et
Harutyunyan et
Mandyam ct al.
Punishotham
Barbieri et
Wang ct al.
Jarrett et
Tang et
Yêche at
Gupta et aL.
Yang et al.
Saveliev
Oliver at
YAIB (ours)
Johnson
MIMIC-III MIMIC-IV
Datasets
XXXX
eICU
HiRID
AUMCdb
Mortality risk
Circulatory failure
Prediction tasks
Kidney function (KF)
Respiratory failure
X
X
Sepsis
X
Acute kidney injury
Phenotyping
Interventions
Length of stay (LoS) Readmission
Model architectures Preproc.
Feature engineering
Temporal imputation
Temporal resampling
Modular pipeline
LR
ML Random forest
Gradient boost
RNN
LSTM
DL
GRU
Temporal CNN
Transformer
Code available
Extensible
Dataset interoperability'
*; These tasks
included by default but may be casily added through cobort definition pipeline. 8: Duc to lack of recorded database information, these tasks can only be defined for MIMIC III and IV. Interface extensive instructions
interuperable modules following provided abstraction (datasets, prediction tasks, mudels) adjust existing modules without extensive rewriting refactoring.
Provides uncoupled interopernble dataset definition, allowing a.o. transfer learning domain adaption.
benchmarking, or an end-to-end pipeline. Finally, Yang et (023) recently proposed PyHealth as comprehensive deep learning toolkit for both ML researchers and healthcare practitioners; is perhaps most closely related to our work. Unfortunately, PyHealth only supports subsets of the full datasets, and tasks must be defined anew for each dataset. It also does not currently include time series or ways to deal with missing data, limiting its use for novel clinical or MI developments.
3 BENCHMARK DESIGN
YAIB addresses the issues identified above and provides unified interface develop clinical prediction models for the ICU. An experiment in YAIB consists of four steps: 1) define clinical concepts from the raw data; 2) extract the patient cohort and specify the prediction task; 3) preprocess the data and generate features; and 4) train and evaluate the ML models (Figure I).
3.1 DESIGN PHILOSOPHY
We strongly believe that medical research is inherently complex and that rather than providing a rigid benchmark there lies most value in providing modular setup where the user exchange
3

---

Published as a conference paper at ICLR 2024
Yet Another ICU Benchmark
Conta
Harmanizo
Clinical Expert
Define
Researcher
(ML)
Pio Confouuation
Modal Betting Popaparanctes
HIRDD
Salect
AUMoub
AKI
Mes
AUROC
Masping
Mortsilyy
Sapex
Celbertor
Acturnny
Hammnizztion
Caciing
Opndmitationn
Compatible
Variable
[Dataent XI
Hamonizatiio
Data
Selection
FIGURE 1: Schematic overview of benchmark pipeline. On the left side, the creation of harmonized ICU cohorts shown. Note that the domain expertise of clinicians often necessary for defining clinically useful tasks. The schematic overview the benchmark stages can found on right. Note that dotted line indicates that this component can be easily extended, as it follows an abstracted interface.
any part with something that better suits their needs and, importantly, do reproducibly. For example, users frequently want to highlight particular aspect of their model, prompting them to adapt the default tasks. Changes, however minor, can render results incomparable. We, therefore, prioritized extensibility across the entire experiment lifecycle. This high level of extensibility may increase the complexity of our benchmark. We mitigate this by providing range of default experiments for users with limited access to medical expertise or who are content with a fixed set of medical tasks. The experiments were designed to be directly comparable and provide common benchmark. This allows for a standardized evaluation of models similar to existing benchmarks but still henefits from out-of-the-box support for multiple datasets and easy adaptability if need be. While did our best to ensure extensibility, YAIB cannot currently support all possible use cases. Specialized use cases like federated learning reinforcement learning currently require custom code. However, we keep adding functionality to YAIB, and users may nevertheless benefit from using parts of our framework. We provide detailed documentation on how to implement any extensions (Appendix F). We strongly request users of YAIB to provide their code and detailed list of the changes they have made to the repository to accurately and transparently provide results for their experiments.
3.2 CLINICAL CONCEPTS
We ensured that our benchmark supports existing and future ICU datasets. Working with multi- ple datasets requires careful data harmonization, as datasets are collected in different locations, with different clinical recording, and may have completely different data structures. We use the ricu [Bennett et al. 2023) R package to bring datasets into a common, semantically interoperable format. This barmonization relies two things: 1) common temporal reference point and 2) dataset-independent definition clinical concepts. r cu by default distinguishes measurements recorded for patient, hospital admission, these levels of measurement. Through definition of reference points, facilitates temporal compa- rability between datasets. ri cu also allows defining clinical concepts such heart rate or SOFA score independently of any particular dataset, specifying their meaning, plausible min/max ranges, and units of measurement. concept can be enabled for dataset by specifying how it should be extracted from the data, for example, by selecting an entire column or subsetting a table based on an item identifier. ricu thus acts as an interface to the raw data (stored in fast, compressed column format), on command returning the data for concept in a table of ID-time-value pairs. This is still no panacea to make ICU datasets immediately interoperable, but it provides helpful framework for harmonization. For users unfamiliar with R, we provide an interface to access r cu concepts directly from Python. PYICU, a native Python implementation of r ricu, is in development.
ICU admission, and supports conversion between
3.3 PATIENT COHORT AND TASK DEFINITION
Once in common format, the same task definition can be applied across datasets. This facilitates code reuse and eliminates opportunities for error. Even so, care must be taken to combine clinical concepts, define meaningful prediction targets, and apply appropriate exclusion criteria. We provide default workflows and helper functions to support this process, including a transparent pipeline for applying exclusion criteria and reporting patient attrition. We supplied this functionality in
4

---

Published as a conference paper at ICLR 2024
TABLE 2: Prediction task overview. Note that the related work is non-exhaustive.
No
Task
Frequency Once per stay
Type
Related work
Mortality
Baker
Lu et al
Medic et al. 2019
Sharma et
2017
Syed
2021
2
AKI
Hourly
Huang et
2121
Nikkinen al.
2022
Pan et
2019
Kank ct
2020
Shamout et al.
2021
Wang
J0AUn glaynner
Sepsis
Hourly
et
270
Lauritson
2030
Merath
2020
Fleuron
202h
Moor
2019
Maraliiaran ct
Key ct
(Stamoss et
Wang al
4
KF
Once per stay" R
Tomalev et al. 2019 2019
Futoru al.
2016
Perotle al.
2015
Cheng al.
2018
LoS
Hourly
Shillam et al 2019
Goo el all 2020
Classification, Regression, Using data from 0-24 bours.
standalone repositoryto facilitate its use with other modeling frameworks such as Clairvoyance (Jarret)
et al.
2021 The specification of our adaptive and re-definable pipeline is found in Appendix D
3.4 PREPROCESSING AND FEATURE EXTRACTION
Further preprocessing is often required at runtime, including data normalization, generation of missingness indicators, and imputation. We provide a transparent, flexible way for users define their preprocessing pipeline (also available as a standalone package), including default implementations of historical aggregation (e.g., mean variance), resampling of the time resolution, imputation methods, and wrapper for any Scikit-learn (Pedregosa et al., 2011)) preprocessing step. Custom steps can be added by subtyping an abstracted step interface or providing a callable object to a generic step.
3.5 TRAINING AND EVALUATION
single YAIB experiment creates and optimizes a model for a given task and preprocessing pipeline. Experiments are defined using the gin-config library (Dan Holtmann-Rinn et al.) (018)) in simple Python-like text files. The model configuration defines the model architecture and contains information on hyperparameters and optimizers. Every aspect of model is fully configurable. The task configuration defines the target, the data source, the features, and the preprocessing. Additionally, one define the validdaiio splits and the number of iterations. By defining the model and task separately, they can be mixed and matched, training the same architecture for multiple tasks or training multiple models for single task. We provide details for adding new datasets, preprocessing, models, and an example of sepsis prediction in (Appendix E] Training is supervised by Py Torch Lightnnng (Falcon & team 2023), which uses standardized training and logging, GPU parallelism, and advanced debugging. Users can configure hyperparameter ranges and sampling methods for model optimization. A Gaussian Process is fit the hyperparameters using seikit-opp imize (Head) 2021 as a robust alternative to random search (Snoek et al. 2012).
et al.
Result tracking Results are automatically aggregated and written to a JSON file, in addition to optional Tensorboard (Abadi et al. (016), Py PyTorch Lighting (Falcon & team 2023). and WandB (Biewald 2020) logging for easy experiment tracking. Performance evaluation records widely-used metrics out of the box (AUROC, AUPRC, calibration curve, accuracy, loss) and sup- ports multiple evaluation libraries: TorchMetrics (Nicki Skafte Detlefsen et al., 2022), Pytorch- Ignite (Fomin al. 2020), and Scikit-Learn (Pedregosa et al. 2011)) metrics. New metrics, either developed by the user or from existing libraries, can be easily added (see Appendix[F6)
4 EXPERIMENTS
We ran experiments for five common prediction tasks: ICU mortality, onset of acute kidney injury (AKI), onset of sepsis, kidney function (KF) on day 2, and remaining length of stay (LoS) (Talle 2). Mortality and KF used data from 0-24 hours. All other task used all available data until the event or discharge. We ensured adequate data quality by excluding: 1) patients younger than 18 years, 2) stays with missing discharge times; 3) stays with less than six hours in the ICU; 4) stays with measurements in less than four time bins; and 5) stays with no measurement for more than 12 consecutive hours in the ICU. We also applied sask-spe exclusion criteria. For example, we excluded stays of less than 30 hours for the ICU mortality task, as this could introduce causal leakage from patients already dead or about to die at the time of prediction. For each task, we included 52 features, of which 4 were static and 48 were time series. Various additional features, including prescriptions and diagnoses, can be not
directly used in YAIB by adjusting the cohort generation module WAIB-cohorts), if features available, their implementation is straightforward (Appendix Fj. Information on the datasets, features, and individual cohort definitions can be found in Appendix Cland D The code define these cohorts
5

---

Published as a conference paper at ICLR 2024
is publicly available. In addition to the baseline performance for each task, dataset, and model, we used YA/B to investigate the effects of small variations in task definitions on predictive performance common obstacle to model comparability (Moor et al. 202 Fleuren et al. 2020b). Specifically, we i) only excluded stays of less than 24 hours to assess the effects of causal leakage by aligning our mortality task with Yēche et al. (202), ii) omitted static and dynamic historical features (i.e., mi n, max, count, mean) to simulate access to fewer input data, and iii) compared alternative definitions for sepsis We, additionally, evaluated transfer learning with the harmonized datasets (iv).
The data was scaled to zero mean and unit variance. 2. Imputation: After adding missing indicators, we forward-filled all columns for the dynamic data, replacing missing values with the last known values of the same stay. Missing values without a prior measurement were filled with the sample mean. To prevent data leakage, we used the mean of the train split as the sample mean for all splits. 3. Feature generation: We generated the min, max, mean, and count of measurements for each feature the dynamic data. We only applied this step for the conventional ML models, e.g., Light Gradient Boosting Machine (LGBM), as they cannot capture sequential information natively.
Preprocessing I. Scaling:
4.1 MODELS AND EXPERIMENTAL SETUP
We considered a range of algorithms used previous benchmarks (Table Iand applied work (Hyland 2020; Pirracchio et al. 2015 Silva et al. 2012 Syed et al. (0021). including regularized logistic regression (LR) and elastic net (EN) (used for classification and regression, respectively (Pedregosa) al. 2011)) LGBM (ee et al. 2017). and four variations of neural networks: Gated Recurrent Unit (GRU) (Cho et al., 2014), Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber 1997).
Temporal Convolutional Network (TCN) (Bai et al.) 2018) and transtormer (TF) Vaaswani et al. 2017). LR, EN, and IGBM were used with the feature generation described above, as they are unable to utilize time series. The implementation of neural networks was adapted from Yèche et al. (2022).
For our experiments, unless stated otherwise, we used 5 iterations of 5-fold cross-validation. Hyper- parameters were tuned on the training set using 30/50 (DL/ML, respectively) iterations of Bayesian hyperparameter optimization (Snoek et al. 2012). For computational reasons. hyperparameter tuning used only the first 2/3 folds, respectively (see (see|Apper Appendix Hfror definition of all searched and selected hyperparameters). The final validation of the best hyperparameters used all 5 folds. Each model was optimized for a maximum of 1000 epochs. Training was stopped early if performance on the validation set did not improve for 10 epochs. The epoch with the best performance on the validation set was retained and evaluated on the test set. This process was repeated for 5 iterations, after which the results were averaged, and the standard deviation was calculated.
4.2 BENCHMARKING BASELINE MODELS ON MAJOR ICU DATASETS
Baseline results for all tasks can be found in Table 3 and 4
Note that we have also benchmarked
our tasks for two openly available demo datasets from MIMIC-III and eICU; these can be directly
accessed without completing a credentialing procedure (see Table 11 and 12
ICU mortality The performance of traditional ML and DL models was highly comparable among each other and across datasets when predicting mortality based on data from the first 24 hours. Notably, AUPRC was higher in AUMCdb due to a higher outcome prevalence (Table 13).
Acute kidney injury (AKI) Maximum achievable performance was also similar across datasets when predicting the hourly onset of AKI, with the notable exception of HiRID, which had both lower AUROC and AUPRC for all models. GRU models consistently achieved the best performance.
Sepsis The performance of baseline models was worst for the hourly onset of sepsis, both for AUROC and especially AUPRC. This may be explained by the particularly low prevalence of ~ 1% hourly bins classified as septic and the relative difficulty of predicting sepsis in general (Moor et al. 2021b).
Kidney function (KF) Classical ML models achieved relatively good performance for this task, which may reflect the dependence of KF on a limited number of features (Grinsztajn et al.) 2022).
Remaining length of stay (LoS) The performance of ML and DL models also comparable across datasets. Nevertheless, predicting the length of stay seems difficult, given that the average MAE is almost two days. Transformers consistently outperformed most other model types.

---

Published as a conference paper at ICLR 2024
TABLE 3: Baseline performance the classification tasks. We embolden the best mean AUROC 100 (t. ie.. higher better) AUPRC x 100 (t) per dataset and those within a standard deviation (#).
AUMCdb
HiRID
eCCU
MIMIC-IV
Algorithm
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
Mortality
LR
83.7=0.6
52.9+1.2
84.0#0.3
36,9+111
84.8#0.2
33.0#0.7
86.140.1
39.7+0.6
LGBM
84.5+0.5
53.7+1.2
84.420.3
40.640.8
85.7+0.2
36.0#0.6
87.740.2
44.2+0.7
GRU
83,9+0.3
53.820.7
84.8+0.2
39.420.4
86.0#0.1
35.620.1
87.640.1
42.8+0.3
LSTM
83.7+0.7
53.6+14
84.0=0.7
37.8+100
85.5+0.2
35.740.8
86.740.4
41.0+0.7
TCN
84.0=0.6
54.2#1.4
84.6#0.7
39.2#1.3
05,4
34.3+0.6
87.1
41.40.8
TF
84.1+0.2
54.4#1.1
84.9#0.7
39.3#1.5
85.9#0.2
34.7#0.8
86,9+0.3
42.240.3
AKI
LR
85.5+0.3
45.1
79.6#0.1
31.8
72.8+0.1
32.2+0.2
77.1+0.2
37.7+0.3
LGBM
85.820.3
48.420.6
80.2+0.2
32.8=0.4
84.620.1
50.8+0.2
83.820.1
53.3+0.2
GRU
90.6+0.3
52.8+0.7
82.2=0.2
33.9=0.4
90.9=0.0
72.2+0.1
90.7=0.1
69.6+0.2
LSTM
86.5+0.4
40.6#0.6
81.0=0.4
31.8=0.4
90.2=0.1
69.9+0.2
89.7+0.1
66.5+0.2
TCN
89.6#0.2
50.0=0.9
81.2=0.2
32.3=0.4
90.4+0.0
70.4+0.2
89.8+0.1
66.8+0.2
TF
88.2+0.2
48.2+0.7
81,5+0.2
33.4=0.5
89.9+0.1
68.0+0.3
89.6+0.1
65.6+0.2
Sepsis
LR
74.7+1.0
4.0=0.4
76.5#0.6
8,420.3
71.8+0.3
2.940.1
77.140.4
4.6+0.1
LGBM
74.0+0.8
5.2+0.7
76.1=0.4
10.4#0.5
19.3
3.3+0.1
77.5+0.3
5.940.2
GRU
79.7+0.9
7.7=0.7
80.620.5
12.620.5
77.420.2
5.140.1
83.620.3
9.140.3
LSTM
77.1=0.8
6.420.5
78.8+0.4
11.1=0.5
74.0=0.2
4.0+0.1
82.0+0.3
8.0+0.2
TCN
78.7+0.7
7.1=0.6
80.8+0.5
13.0=0.4
76.7=0.1
4.9+0.1
82.7+0.3
8.8+0.2
TF
80.7+0.9
8.620.8
80.820.3
12.620.6
76.2+0.1
4.640.1
80.0+0.8
6.640.2
TABLE 4: Baseline performance on the regression tasks. Results are reported in Mean Absolute Eror (L)
Kidney function in mg/dl.
Length of Stay in hours
Algo. AUMCdb
HiRID
eICU
MIMIC-IV
AUMCdb
HiRID
eICU
MIMIC-IV
EN
0.24=0.00
0.28=0.00
0.31=0.00
0.25=0.00
54.9=0.0
47.2+0.1
43.6+0.0
46.5+0.0
LGBM0.32+0.00
0.34=0.00
0.2920.00
0.24#0.00
44.7+0.0
39.2+0.1
39.3+0.0
40.1
GRU
0.29=0.00
0.32+0.01
0.34+0.01
0.30+0.01
42.9+0.1
39.6+0.1
38.940.1
39.940.1
LSTM
0.29=0.00
0.33+0.00
0.28+0.01
0.28=0.01
44.8+0.1
39.8+0.1
39.2+0.1
40.6+0.1
TCN
0.28=0.01
0.23+0.01
0.31
0.28=0.01
43.720.1
39.920.1
38.9=0.0
40.4+0.1
TF
0.2620.00
0.3120.01
0.3320.01
0.32+0.01
41.8+0.1
39.1+0.1
38.220.1
39.020.1
We provide the average and Interquartile range for Kidncy Function and Length of Stay in Table 14
4.3 USING YAIB AS AN EXPERIMENTAL ML FRAMEWORK
Changing exclusion criteria for mortality cohorts As hypothesized, the choice of exclusion criteria could majorly impact achievable prediction performance (Table 5). Compared the peak performance achieved with the for the mortality task was noticeably lower. Aligning the exclusion criteria accounted for half of the performance difference. The remaining difference was likely due to the inclusion of additional predictors - most notably drug usage in the HiRID-benchmark. This highlights the difficulties of comparing works that ostensibly address the same task, even using the same dataset and model implementation.
(Yèche et al. 2022). our baseline performance
Restricting input features We observed that dynamic feature generation consistently outperformed task definitions that did not include them (Table and (). LR on MIMIC-IV showed considerable performance gap, whereas AUMCdb remained stable. We noted performance decrease that ranges between 4.0% and 19.1% for LR and between 5.2% and 13.1% % for LGBM. Omitting static features led minor drops in performance (Tblle 9 and I0): averaged across datasets, observe performance differences ranging between 0.5% and 0.2% for the transformer model.
Comparing sepsis definitions Label definitions also had considerable impact AUROC and/or AUPRC (Talle 6). which was not always apparent from the definition alone. Sepsis has been defined in several ways (Fleuren et al. 2020b), mainly because clinical gold standard that can be transferred
7

---

Published as a conference paper at ICLR 2024
TABLE 5: ICU mortality prediction HiRID with (>24h) and without (>:0h) possibility of causal leakage.
Cohort definition
w/o leakage
w/ leakage
Yèche et al. (2022)
Algorithm
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
LR
84.0+0.3
36.9#1.1
87.2+0.4
13.1+1.3
89.0=0.0
58.100.0
LGBM
84.5=0.3
40.6#0.9
87.9+0.5
47.7+1.2
88.8=0.2
54.6#0.8
GRU
84,820.2
39.4+0.4
88.2+0.3
46.1212
60.3+1.6 90.020.4
TCN
84.620.7
39.2+1.3
87.8+0.2
45.2+1.0
60.2+1.1 90.8+0.2
89.7=0.4
TF
84.920.7
39.4+1.5
88.240.3
47.1+1.2
61.020.8
TABLE 6: Sepsis prediction MIMIC-IV for different definitions of sepsis.
Sepsis definition
Seymour et al. 2016)*
Moor et al. (2021a)
Calvert et al. 2016)
Algorithm
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
LGBM
75.9=0.2
4.3+0.0 6.1+0.0
72.4+0.0
10.5+0.0
62.2+0.2 89.2+0.0
1.8=0,0
GRU
79.2+0.1
80.9.0.0
17.7.0.0
9.3+0.2
Our detinition: adapted be more clinically actionable,
Appendix D
to ML models is currently lacking. Our sepsis definition (adapted from Seymour et al. (2016),
Appendix D) can be considered closely related to that used by Moor et al. (2021a), who implement a variant of Sepsis-3 (Singer al. (0166) However,
required that antibiotics were administered continuously for 2 3 days (Reyna et al. 2019). We judged that this would increase the clinical usability of the task but found that it also severely reduced the achievable AUPRC likely due to much lower prevalence (TTblle 17). The definition used by Calvert et al. (2016) on the other hand adapted Sepsis-2 (Levy et al. 2003). which differs fundamentally from Sepsis-3 and resulted in a notably higher AUROC (Engoren et al.) 2020). This highlights the importance of precise cohort definitions, as some definitions may, by design, be more difficult to predict.
4.4 TRANSFER LEARNING
External validation YAIB's 's common dataset format allowed us evaluate model trained on equal sample of one dataset on data from all other datasets. We additionally trained a model on pooled (d-1) data from three datasets and evaluated the fourth, held-out dataset. For the ICU mortality task (Figure 2), models, as expected, performed best on independent test data from their training dataset (diagonal). Performance could drop considerably when models were evaluated in another database (off-diagonal). Notably, AUPRC performance could increase in the evaluation dataset (rows) but always remained lower than the highest achievable performance for that dataset (columns). We found that MIMIC-IV and eICU transferred well among each other. The pooled model usually performed well the best single-dataset model. Notably, AUMCdb AUPRC results demonstrate decidedly
AUMDeb
83.9
79.6
77.8
80.4
AUMMCdb
53.8
27.5
8
28.9
33.4
-86
-50
Training Dataset
HIRID
80.5
84,8
77.3
80.5
Training Dataset
HIRVD
46.1
39.4
24.3
32.3
84
efoU
82.1
79.4
86.0
86.0
AUROC (Nghar
elou
49.2
30.1
35.6
39.9
AUPRO (higher
MIMIC-V
81.6
78.5
81.7
87.6
MIMICAN
50.9
28.6
28.0
42.8
33
Pooled (6-1)
Pooled (8-1)
82.5
79.6
82.4
83.9
49.9
31.4
30.6
37.9
25
AUMCab
HIRDD Evaluation
eICU Dataset
MIMIC-N
AUMCdb
HRRID
eICU
MIMIC-IV
Evaluation
Dataset
FIGURE 2: Performance of prediction models when trained on one dataset (row) and evaluated on all others (columns). Left: Performance in AUROC of the GRU model on ICU mortality. Right: Performance in AUPRC for the same models, Pooled (d-1) refers to training model on every dataset except the evaluation dataset.
8

---

Published as a conference paper at ICLR 2024
higher performance than evaluation on other datasets, which could be the result of patient case mix and outcome prevalence (see Table 14).
Fine-tuning In Figure 2.
saw that cICU resulted in the most generalizable model for ICU mortality, which may serve as strong pre-training for transfer learning. Since it worked worst for HiRID, we further fine-tuned the eICU GRU model (source) for HiRID (target) by retraining it using an increasing number of samples from the HiRID dataset. We compared the results to a model trained from scratch on the same amount of HiRID samples (Figure 3).
85
AUROC (haher better)
76
700
HiRDD only
Train on HIRID
86
eICU only
Fine-line elCU HIRDD
Fine-tuning was profitable for any number of additional samples and especially for <4,000 samples.
2000
4000
Samples
6000
8000
10000
FIGURE 3: Fine-tuning an eICU model for ICU mortality prediction HiRID.
5 DISCUSSION
We provide extensive ML and DL baselines for five clinical prediction tasks trained across four major open-source ICU datasets. While we frequently obtained comparable results across model architectures, seemingly small differences in cohort definition could substantially impact the achieved accuracy. Our findings highlight not only the need for standardized training pipelines but also for harmonized cohort definitions to allow for meaningful comparison clinical prediction models. Our work provides the first international, multi-center ICU benchmark, including the first-ever benchmark for the AmsterdamUMCdd dataset. It naturally facilitates sorely needed external validation of model performances and allows fine-tuning of pre-trained models for new datasets. This makes YAIB relevant to a wide range of research areas beyond classical supervised learning, including domain adaption and generalization. We hope this broad reach encourages ICU data providers ensure compatibility with YAIB, as they can expect a larger overall research impact. This simplifies the use of novel datasets by the clinical and ML community.
YAIB aids researchers in training baseline models by providing them with ready-to-use imple- mentations of srate-of-thhean model architectures; new model implementations can therefore be easily compared. While most existing benchmarking studies are hard-coded, we utilize flexible, dataset-imdependent cohort definitions and configurable preprocessing facilities linked via common, shareable syntax. This setup ack acknowledges that task definitions inevitably involve arbitrary decisions, without one "size" that fits all. our work, we embrace this idea and aim to equip researchers both applied and theoretical the use of custom proprictary data) while maintaining reproducibility and reusability across studies, Models can thus be compared across multiple, slightly different task definitions and datasets, still ensuring an apples-to-apples comparison. We hope this lowers the bar for researchers to test their approaches across range of configurations and datasets.
with the tools to quickly adapt task to their individual needs (including
YAIB is currently limited to ICU settings, where several datasets are publicly available. A similar setup could be beneficial for data from other medical settings, such inpatient wards. Although created for critical care, YAIB is not specific to the ICU and can be readily extended to other settings, provided a suitable configuration is defined. Features included in YAIB, at the time of writing, mainly relate to vital signs, lab tests, and data relevant to outcome definitions. Further clinician-assister harmonization efforts will be necessary to increase the breadth of features, most notably medications and comorbidities. If YAIB is adapted to general EHR, including clinical notes and medical imaging a logical next step. We also note that we compared these models on the basis of commonly used MI. metrics; we leave the comparison with respect to clinical fairness and bias as an easy future extension to our framework (see (sce|Appente Appendix F). Finally, we advise users of our benchmark to carefully consider the compromises made to allow for cohort harmonization; we strongly recommend clinical validation before making practical decisions based on the developed models.
6 CONCLUSION
Routine medical data is highly complex. Without clear ground truth, researchers are inevitably forced to make arbitrary design choices when defining outcomes and populations of interest. To promote comparable and reproducible models in this setting, we believe that further tools are needed that allow researchers to define clinical prediction tasks transparently, share experimental setups easily, and validate results against various data sources. As a flexible and extensible framework for clinical modeling on ICU data, YAIB is meant to be a step towards that goal.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
Robin van de Water is funded by the "Gemeinsamer Bundesausschuss (G-BA) Imnoaationsausccuss" in the framework of "CASSANDRA Clinical ASSist AND aleRt Algorithms" (project number 0IVSF20015). We would like to acknowledge the work of Alisher Turubayev, Anna Shopova, Fabian Lange, Mahmut Kamalak, Paul Mattes, and Victoria Ayvasky for adding Pytorch Lightning. Weights and Biases compatibility, and several optional imputation methods to a later version of the benchmark repository.
ETHICS STATEMENT
We do not manage access and do not provide access any of the full medical datasets included in this work, and we adhere to the usage licenses for each dataset. Users can follow the credentialing procedures outlined Appendix C However, we provide two preprocessed demo datasets out of the box for reproducibility and experimentation. The demo task cohorts for MIMIC-III and eICU mentioned in that section are derived from the official demo datasets published on PhysioNet by the original authors of the respective databases. Each demo dataset represents small, curated subset of data that is freely accessible without any need for human subject training. Both demo datasets published under an Open Data Commons Open Database License v1.0, which explicitly permits the adoption and sharing the data. The original demo data, as well as further information, can be found at the MIMIC-III demo and eICU demo Physionet pages.
REPRODUCIBILITY STATEMENT
We include the source code of YAIE (main benchmark), YAIB-cchort.d (adaptable cohort extraction) and ReciPys (extensible preprocessing package) in our submission. Models for each task and architecture are publicly available" In the included source code, file called PAPER. mo describes the reproducibility steps of the experiments in this paper. Specifically, one requires the standalone codebase of YAIB-cohorts to first create the cohorts from the acquired data, once you have completed the required credentialing (see/Appendix Corr details). As mentioned, we include demo cohort data for each task (results for these cohorts are shown in Appendix B). Appendix D describes the data processing and task creation. The usage of YAIB is detailed in Appendix E Appendix Fjshows how YAIB can be extended with new datasets, clinical concepts, tasks, models, and evaluation metrics. Additionally, we refer to the README I and the wik[ the usage of YAIB. Appendix G dettail the experiment design and chosen hyperparameters, respectively. Finally, Appendix coontains the machine learning reproducibility checklist for work.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg. Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: system for large-scale machine learning. In / /2th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265-283, 2016.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling, April 2018.
Stephanie Baker, Wei Xiang, and Ian Atkinson. Continuous and automatic mortality risk prediction using vital signs in the intensive care unit: A hybrid neural network approach. Scientific Reports, 10(1):21282, December 2020. ISSN 2045-2322. doi: 10.1038/s41 00008/44199-00--7777 188777
Sebastiano Barbieri, James Kemp, Osear Perez-Concha, Sradha Kotwal, Martin Gallagher, Angus Ritchie, and Louisa Jorm. Benchmarking Deep Learning Architectures for Predicting Readmission to the ICU and Describing Patients-at-Riss Risk. Scientific Reports, 10(1): January 2020. ISSN 2045-2322. doi: 10. 00044441-8-000055555
Nicolas Bennett, Drago Plecko, Ida-Fong Ukor, Nicolai Meinshausen, and Peter Bühlmann. Ricu: R's interface to intensive care data. GiguScience, 12, June 2023. doi: 10. 109//gigaccience/giall1..
Lukas Biewald, Experiment tracking with weights and biases, 2020.
Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. Building Classifiers with Independency Constraints. In 2009 IEEE International Conference Data Mining Workshops, pp. 13-18, December 2009. doi: 10.1 D0.109//CDMW22009..88
Jacob S. Calvert, Daniel A. Price, Uli K. Chettipally, Christopher W. Barton, Mitchell D. Feldman, Jana I.. Hoffman, Melissa Jay, and Ritankar Das. A computational approach early sepsis detection. Computers in Biology and Medicine, 74:69-73, July 2016. ISSN 00104825. doi: 1...166j.compiiomed211....00
Peng Cheng, Lemuel R. Waitman, Yong Hu, and Mei Liu. Predicting Inpatient Acute Kidney Injury over Different Time Horizons: How Early and Accurate? AMIA Annual Symposium Proceedings, 2017:565 -74,, April 2018. ISSN 1942-597X.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the Properties of Neural Machine Translation: Incoder-Decodee Approaches, October 2014.
Dan Holtmann-Rice, Sergio Guadarrama, and Nathan Silberman. Gin-config. Google, 2018.
Wenjie Du. PyPOTS: A Python Toolbox for Data Mining on Partially-Observed Time Series, May 2023.
Bar Eini-Porat, Ofra Amir, Danny Eytan, and Uri Shalit. Tell me something interesting: Clinical utility of machine learning prediction models in the ICU. Journal Biomedical Informatics, 132: 104107, August 2022. ISSN 15320464. doi: 10. 10.1016/j 10166/jjbii..... (it.jjbb.22.2144444 104107.
Milo Engoren, Troy Seelhammer, Robert E. Freundlich, Michael D. Maile, Matthew J. G. Sigakis, and Thomas A. Schwann. A Comparison of Sepsis-2 (Systemic Inflammatory Response Syndrome Based) to Sepsis-3 (Sequential Organ Failure Assessment Based) Definitions-2 -A Multicenter Retrospective Study*. Critical Care Medicine, 4858): 3493. doi: 10.
September 2020. ISSN 0090)
William Falcon and The PyTorch Lightning team. PyTorch lightning. Zenodo, April 2023.
Lucas M. Fleuren, Thomas L. T. Klausch, Charlotte L. Zwager, Linda J. Schoonmade, Tingjie Guo, Luca F. Roggeveen, Eleonora L. Swart, Armand J. Girbes, Patrick Thoral, Ari Ercole, Mark Hoogendoom, and Paul W. G. Elbers. Machine learning for the prediction of sepsis: A systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Medicine, 46(3):383 400, 2020a. ISSN 0342-4642, doi: 141077/00013--019-55555
11

---

Published as a conference paper at ICLR 2024
Lucas M. Fleuren, Patrick Thoral, Duncan Shillan, Ari Ercole, Paul W. G. Elbers, Mark Hoogendoom, Ben Gibbison, Thomas L. T. Klausch, Tingjie Guo, Luca F. Roggeveen, Eleonora L. Swart, Armand J. Girbes, and Right Data Right Now Collaborators. Machine learning in intensive care medicine: Ready for take-off? Intensive Care Medicine, doi: 1007//0/0 10.11077/001134--00-6660000 34-020-06045-y.
111111886-4888 July 2020b. ISSN 1432-1238.
V. Fomin, J. Anmol, S. Desroziers, Kriss, and A. Tejani, High-level library to help with training neural networks in PyTorch, 2020.
Joseph Futoma, Mark Sendak, Blake Cameron, and Katherine Heller. Predicting Disease Progression with a Model for Multivariate Longitudinal Clinical Data. In Proceedings of the 1st Machine Learning for Healthcare Conference, pp. 42-54. PMLR, December 2016.
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, September 2022.
Chonghui Guo, Menglin Lu, and Jingfeng Chen. An evaluation of time series summary statistics as features for clinical prediction tasks. BMC Medical Informatics and Decision Making, 2081):48, December 2020. ISSN 1472-6947. doi: 10.1 11--22911000--0666
Mehak Gupta, Brennan Gallamoza, Nicolas Cutrona, Pranjal Dhakal, Raphael Poulain, and Rahma- tollah Beheshti. An Extensive Data Processing Pipeline for MIMIC-IV, September 2022.
Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1);96, December 2019. ISSN 2052-4463. doi: 10. (038//415977001--00033
Tim Head, Manoj Kumar, Holger Nahrstaedt, Gilles Louppe, and laroslav Shcherbatyi. Scikit- optimizae/cikki-optiinzz. Zenodo, October 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-688 6851. Curran Associates, Inc., 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Technical Report 9, TU Munich, 1997.
Wei Huang, Yuwen Chen, Peng Wang. Xiang Liu, and Shuguang Liu. An Interpretable Temporal Convolutional Network Model for Acute Kidney Injury Prediction in the Intensive Care Unit. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 3021-3028. December 2021. doi: 10. 10.1109/BBBM52215..00 2,2222 2.20 19669653.
Stephanie I. Hyland. Early prediction of circulatory failure in the intensive care unit using machine learning. Nature Medlclne, 26:28, 2020.
Daniel Jarrett, Ioana Bica, Ari Ercole, Jinsung Yoon, and Zhaozhi Qian. CLAIRVOYANCE; A PIPELINE TOOLKIT FOR MEDICAL TIME SERIES. International Conference on Learning Representations, pp. 32, 2021.
Daniel Jarrett, Bogdan Cebere, Tennison Liu, Alicia Curth, and Mihaela van der Schaar. Hyperlmpute: Generalized Iterative Imputation with Automatic Model Selection, June 2022.
Senjun Jin, Lin Chen, Kun Chen, Chaozhou Hu, Sheng an Hu, and Zhongheng Zhang. Establishment of Chinese critical care database from electronic healthcare records tertiary care medical cen- ter. Scientific Data, 10(1):49, January 2023, ISSN 2052-4463. doi: 10. 10.1033/441597-02--0955555
Alistair E. W. Johnson, Tom J. Pollard, and Roger G. Mark. Reproducibility in critical care: A mortality prediction case study. In Proceedings of the 2nd Machine Learning for Healthcare Conference, pp. 361-376. PMLR, November 2017.
12

---

Published as a conference paper at ICLR 2024
Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard. Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. MIMIC-IV, freely accessible electronic health record dataset. Scientific Data, 10(1):1, January
2023. ISSN 2052-4463. doi: 10. 1038/s41 0188//41557-0222-11 -97-022-01889-x.
Alistair W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. MIMIC-III, freely accessible critical care database. Scientific Data, 3(1):160035, December 2016. ISSN
2052-4463. doi: 10.1038/sdata.200 2016.35.
KDIGO. Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group: KDIGO clinical practice guideline for acute kidney injury. Kidney International Supplements, (I), March 2012. ISSN 21571716.
doi: 10. 10.1038/kisup.2012.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017.
Christopher J. Kelly, Alan Karthikesalingam, Mustafa Suleyman, Greg Corrado, and Dominic King. Key challenges for delivering clinical impact with artificial intelligence. BMC Medicine, 17(1): 195, December 2019. ISSN 1741-7015. doi: 10. 1 186/s 2916-019-111 1426-2.
Christopher Kok, V. Jahmunah, Shu Lih Oh, Xujuan Zhou, Raj Gururajan, Xiaohui Tao, Kang Hao Cheong, Rashmi Gururajan, Filippo Molinari, and U.Rajendra Acharya. Automated prediction of sepsis using temporal convolutional network. Computers in Biology and Medicine, 127:103957,
December 2020. ISSN 00104825, doi: 1110///.ccppbiooma 2020. 103957.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Wave A Versatile Diffusion Model for Audio Synthesis, March 2021.
Jay L. Koyner, Kyle A. Carey, Dana P. Edelson, and Matthew M. Churpek. The Development of Machine Learning Inpatient Acute Kidney Injury Prediction Model*:. Critical Care Medicine, 46 (7):1070-1077, July 2018. ISSN 0090-3493. doi: 10. 10.1099/CCM.00000 0000000000003122
Simon Meyer Lauritsen, Mads Ellersgaard Kalør, Emil Lund Kongsgaard, Katrine Meyer Lauritsen, Marianne Johansson Jørgensen, Jeppe Lange, and Bo Thiesson. Early detection of sepsis utilizing deep learning electronic health record event sequences. Artificial Intelligence in Medicine, 104: 101820, April 2020. ISSN 09333657. doi: 11100//..rtmme.2200 101820.
Mitchell M. Levy, Mitchell P. Fink, John C. Marshall, Edward Abraham, Derek Angus, Deborah Cook, Jonathan Cohen, Steven M. Opal, Jean-Louis Vincent, Graham Ramsay, and International Sepsis Definitions Conference. 2001 SCCW/ESCC//CCC//TTSSS Conference. Intensive Care Medicine, 29(4):530-538, April 2003. ISSN 0342-4642. doi: 00000//000134-00-16666.
International Sepsis Definitions
Bryan Lim, Sercan Ö. Ank, Nicolas Loeff, and Tomas Pfister. Temporal Fusion Transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37()): 1748- 1764, October 2021. ISSN 0169-2070. doi: 10. -f006ff-ifoorecast2202 202...2..1111101
Sheng-Chieh Lu, Cai Xu, Chandler H. Nguyen, Yimin Geng, André Pfob, and Chris Sidey-Gibbons. Machine Learning-Baed Short Term Mortality Prediction Models for Patients With Cancer Using Electronic Health Record Data: Systematic Review and Critical Appraisal. JMIR Medical Informatics, (0(3))333182 March 2022.
doi: 10.2196/33182.
Aishwarya Mandyam, Elizabeth C. Yoo, Jeff Soules, Krzysztof Laudanski, and Barbara E. Engelhardt. COP-E-CAT: Cleaning and organization pipeline for EHR computational and analytic tasks. In Proceedings of the 12th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics, pp. 1-9, Gainesville Florida, August 2021. ACM. ISBN 118-1-45003-4500 doi: 0001145/3459993334493333
Goran Medic, Melodi Kosaner Kließ, Louis Atallah, Jochen Weichert, Saswat Panda, Maarten Postma, and Amer El-Kerdi. Evidence- based Clinical Decision Support Systems for the prediction and detection of three disease states in critical care: A systematic literature review, F1000Research, 1728, 2019. ISSN 2046- 1402. doi:
13

---

Published as a conference paper at ICLR 2024
Katiuscha Merath, J. Madison Hyer, Rittal Mchta, Ayesha Farooq, Fabio Bagante, Kota Sahara, Diamantis I. Tsilimigras, Eliza Beal, Anghela Z. Paredes, Lu Wu, Aslam Ejaz, and Timothy M. Pawlik. Use of Machine Learning for Prediction of Patient Risk of Postoperative Complications After Liver, Pancreatic, and Colorectal Surgery. Journal of Gastrointestinal Surgery, 24(8): 1843-1851 August 2020. ISSN 1091-255X, 1873-4626. doi: 10. 10.1007/s1 (605-01--04338-2
Michael Moor, Max Horn, Bastian Rieck, Damian Roqueiro, and Karsten Borgwardt. Early Recog- nition of Sepsis with Gaussian Process Temporal Convolutional Networks and Dynamic Time Warping. In Proceedings of the 4th Machine Learning for Healthcare Conference, pp. 2-26. PMLR, October 2019.
Michael Moor, Nicolas Bennet, Drago Plecko, Max Horn, Bastian Rieck, Nicolai Meinshausen, Peter Bühlmann, and Karsten Borgwardt. Predicting sepsis in multi-site, multi-national intensive care cohorts using deep learning, July 2021a.
Michael Moor, Bastian Rieck, Max Horn, Catherine R. Jutzeler, and Karsten Borgwardt. Early Prediction of Sepsis in the ICU Using Machine Learning: A Systematic Review. Frontiers in Medicine, 8, 2021b. ISSN 2296-858X.
Sankavi Muralitharan, Walter Nelson, Shuang Di, Michael McGillion, Pj Devereaux, Neil Grant Barr, and Jeremy Petch. Machine searningBBased Early Warning Systems for Clinical Deterioration: Systematic Scoping Review. Journal of Medical Internet Research, 23(2)=e25187, February 2021.
ISSN 1438-8871. doi: 10.2196/25177.
Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh, Teddy Koker, Luca Di Liello, Daniel Stanel, Changsheng Quan, Maxim Grechkin, and William Falcon. TorchMetrics measuring reproducibility in PyTorch, February 2022.
Okke Nikkinen, Timo Kolehmainen, Toni Aaltonen, Elias Jämsä, Seppo Alahuhta, and Merja Vakkala. Developing a supervised machine learning model for predicting perioperative acute kidney injury in arthroplasty patients. Computers in Biology and Medicine, 144:10535,, May 2022. ISSN 00104825. doi: 10.1016ff.compppo wwtp:.commbommee.000 2022. 105351.
Matthieu Oliver, Jérôme Allyn, Rémi Carencotte, Nicolas Allou, and Cyril Ferdynus. Introducing the BlendedICU dataset, the first harmonized, international intensive care dataset. Journal of Biomedical Informatics, 146:20 October 2023. ISSN 1532-0464. doi: 10. j0..... 104502.
Ziyuan Pan, Hao Du, Kee Yuan Ngiam, Fei Wang, Ping Shum, and Mengling Feng. A Self Correcting Deep Learning Approach to Predict Acute Conditions in Critical Care. arXiv: 1at::1991..445 90.4444444444 [cs, stat], January 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 2011.
Adler Perotte, Rajesh Ranganath, Jamie S Hirsch, David Blei, and Noémie Elhadad. Risk prediction for chronic kidney disease progression using heterogeneous electronic health record data and time series analysis. Journal of the American Medical Informatics Association JAMIA, 22(4):872-880
July 2015. ISSN 1067-5027. doi: 10. 0..1993/amiaaocv024..
Romain Pirracchio, Maya L Petersen, Marco Carone, Matthieu Resche Rigon, Sylvie Chevret, and Mark J van der Laan. Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): A population- study. The Lancel Respiratory Medicine, 3(1):42-52, January 2015. ISSN 22132600.
doi: 100016//2211-22001470000
Tom J. Pollard, Alistair E. W. Johnson, Jesse D. Raffa, Leo A. Celi, Roger G. Mark, and Omar Badawi. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Scientific Data, 51):1881 December 2018. ISSN 2052-4463, doi: 10. 20188.
14

---

Published as a conference paper at ICLR 2024
Sanjay Purushotham, Chuizheng Meng, Zhengping Che, and Yan Liu. Benchmarking deep learning models on large healthcare datasets. Journal of Biomedical Informatics, 83:112- 134, July 2018.
ISSN 15320464. doi: 10.1016/1,bi..218.04444 20..0000...
Nina Rank, Boris Pfahringer, Jörg Kempfert, Christof Stamm, Titus Kühne, Felix Schoenrath, Volkmar Falk, Carsten Eickhoff, and Alexander Meyer. Deep-leaming- based real-time prediction of acute kidney injury outperforms human predictive performance, upj Digital Medicine, 3(1):139,
December 2020. ISSN 2398-6352. doi: 00.1038/441746-02--
Matthew A Reyna, Chris Josef, Salman Seyedi, Russell Jeter, Supreeth P Shashikumar, M Bran- don Westover, Ashish Sharma, Shamim Nemati, and Gari D Clifford. Early Prediction of Sepsis from Clinical Data: The PhysioNet/Computng in Cardiology Challenge 2019. In 2019 Computing in Cardiology (CinC), pp. Page 1-Page 4, September 2019. doi: (0.23919/CinC498433 9005736.
Niklas Rodemund, Bernhard Wernly, Christian Jung, Crispiana Cozowicz, and Andreas Koköfer. The Salzburg Intensive Care database (SICdb): An openly available critical care dataset. Intensive Care Medicine, April 2023. ISSN 1432-1238. doi: 10. 10.100776001334-02-000000
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Nassir Navab, Joachim Homngger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Computer-Assisted Intervention MICCA/ 2015, Lecture Notes in Computer Science, pp. 234-241, Cham, 2015. Springer International Publishing. doi: 10. 00.1007/97--33319-255444
ISBN 978-3-319-2457444
Tabinda Sarwar, Sattar Seifollahi, Jeffrey Chan, Xiuzhen Zhang, Vural Aksakalli, Irene Hudson, Karin Verspoor, and Lawrence Cavedon. The Secondary Use of Electronic Health Records for Data Mining: Data Characteristics and Challenges, ACM Computing Surveys, 55(2):1-40, March 2023. ISSN 0360-0300, 1557-7341. doi: 10.1 00.1145/3490234
Christopher M Sauer, Li-Ching Chen, Stephanie Hyland, Armand Girbes, Paul Elbers, and Leo Celi. Leveraging electronic health records for data science: Common pitfalls and how to avoid them. The Lancer Digital Health, 4(12):e893-e83) December 2022a. ISSN 25897500. doi:
1011066/22500-730022200015544445
Christopher M. Sauer, Tariq A. Dam, Leo A. Celi, Martin Faltys, Miguel A. A. de la Hoz, Lasith Adhikari, Kirsten A. Ziesemer, Armand Girbes, Patrick J. Thoral, and Paul Elbers. Systematic Review and Comparison of Publicly Available ICU Data Sets Decision Guide for Clinicians and Data Scientists. Critical Care Medicine, 1ebbeee811-8888 June 2022b. ISSN 0090-3493. doi:
Evgeny S. Saveliev and Mihaela van der Schaar. TemporAl: Facilitating Machine Learning Innovation in Time Domain Tasks for Medicine, January 2023.
Christopher W. Seymour, Vincent Liu, Theodore Iwashyna, Frank M. Brunkhorst, Thomas D. Rea, André Scherag, Gordon Rubenfeld, Jeremy M. Kahn, Manu Shankar-Hari, Mervyn Singer, Clifford S. Deutschman, Gabriel J. Escobar, and Derek C. Angus. Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis- 3). JAMA, 76668):762, February 2016. ISSN 0098-7484 doi: 10. 10.1000//ma..00111
Farah Shamout, Tingting Zhu, and David A. Clifton. Machine Learning for Clinical Outcome Prediction. IEEE Reviews Biomedical Engineering, 14:116-126, 2021. ISSN 1937-3333, 1941-1189. doi: 10. 1009/RBM..20200000777
Alok Sharma, Anupam Shukla, Ritu Tiwari, and Apoorva Mishra. Mortality Prediction of ICU patients using Machine Leaning: A survey. In Proceedings of the International Conference on Compute and Data Analysis ICCDA '17, pp. 49-53, Lakeland, FL, USA, 2017. ACM Press. doi: 10.1 10.31445/93324.33000
ISBN 978-1-4503-555
Seyedmostafa Sheikhalishahi, Vevake Balaraman, and Venet Osmani. Benchmarking machine learning models on multi-centre eICU critical care dataset. PLOS ONE, 15(7):e0234424, July 2020. ISSN 1932-6203. doi: http////aal...oe.00222...
15

---

Published as a conference paper at ICLR 2024
Duncan Shillan, Jonathan A. C. Sterne, Alan Champneys, and Ben Gibbison. Use of machine learning to analyse routinely collected intensive care unit data: A systematic review. Critical Care, 23(1): 284, December 2019. ISSN 1364-8535, doi: 10.1 18/4
Ikaro Silva, George Moody, Daniel J. Scott, Leo A. Celi, and Roger G. Mark. Predicting In-Hospital Mortality of ICU Patients: The PhysioNer/Computin in Cardiology Challenge 2012. Computing
in Cardiology, 39:245 248, 2012. ISSN 2325-8861.
Mervyn Singer, Clifford S. Deutschman, Christopher Warren Seymour, Manu Shankar-Hari, Djillali Annane, Michael Bauer, Rinaldo Bellomo, Gordon R. Bernard, Jean- Caan- Chiche, Craig M. Coopersmith, Richard S. Hotchkiss, Mitchell M. Levy, John C. Marshall, Greg S. Martin, Steven M. Opal, Gordon D. Rubenfeld, Tom van der Poll, Jean-Louis Vincent, and Derek C. Angus. The
Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA, 315(8): 801-810, February 2016. ISSN 0098-7484
doi: 10.1001/jama.2016.000
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical Bayesian Optimization of Machine Learning Algorithms, August 2012.
Mahanazuddin Syed, Shorabuddin Syed, Kevin Sexton, Hafsa Bareen Syeda, Maryam Garza, Meredith Zozus, Farhanuddin Syed, Salma Begum, Abdullah Usama Syed, Joseph Sanford, and Fred Prior. Application of Machine Learning in Intensive Care Unit (ICU) Settings Using MIMIC Dataset: Systematic Review. Informatics, 8(1):16, March 2021. ISSN 2227-9709. doi: 10.3390informaaicc:0000000
Shengpu Tang, Parmida Davarmanesh, Yanmeng Song. Danai Koutra, Michael W Sjoding, and Jenna Wiens. Democratizing EHR analyses with FIDDLE: flexible data-driven preprocessing pipeline for structured clinical data. Journal of the American Medical Informatics Association, 27(12):
1921-1934, December 2020. ISSN 1527-974X. doi: 10. 10.0099jamia/ccaal33
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI:
Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation. In Advances in Neural Information Processing Systems, volume 34, pp. 24804-24816 Curran Associates, Inc., 2021.
Patrick J. Thoral, Jan M. Peppink, Ronald H. Driessen, Eric G. Sijbrands, Erwin J. O. Kompanje, Lewis Kaplan, Heatherlee Bailey, Jozef Kesecioglu, Maurizio Cecconi, Matthew Churpek, Gilles Clermont, Mihaela van der Schaar, Ari Ercole, Armand R. J. Girbes, and Paul W. G. Elbers. Sharing ICU Patient Data Responsibly Under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: The Amsterdam University Medical Centers Database (AmsterdamUMCabb Example* Critical Care Medicine, 4966))e563-657
June 2021. ISSN 0090-3493. doi: 10.
Nenad Tomašev, Xavier Glorot, Jack W. Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri, Ivan Protsyuk, Alistair Connell, Cían O. Hughes, Alan Karthikesalingam, Julien Cornebise, Hugh Montgomery. Geraint Rees, Chris Laing, Clifton R. Baker, Kelly Peterson, Ruth Reeves, Demis Hassabis, Dominic King, Mustafa Suleyman, Trevor Back, Christopher Nielson, Joseph R. Ledsam, and Shakir Mohamed. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature, 572(7767):116-111 August 2019. ISSN 1476-4687.
doi: 10. 01938//41566-099-11333
Robin van de Water and Bert Amrich. Closing Gaps: An Imputation Analysis of ICU Vital Signs. In Deep Generative Models for Health Workshop NewrIPS 2023, October 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. urXiv:/706.000 03762 [cs), December 2017.
Pauli Virtanen, Ralf Gommers, Travis F. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, J Carey, ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy L.0: Fundamental algorithms for scientific computing in python. Nature Methods, 17:261-277 2020. doi: 10. 192-019-0666222
16

---

Published as a conference paper at ICLR 2024
Deepak Vohra. Apache Parquet. In Deepak Vohra (ed.), Practical Hadoop Ecosystem: A Definitive Guide to Maldoop-Relatee Frameworks and Tools, pp. 325-335, Apress, Berkeley, CA, 2016. ISBN 978-1-4842-2199-0. doi:
8.
Hui Wang, Fuxing Deng, Buyao Zhang, and Shuangping Zhao. Real-Time Prediction of AKI Among Middle-Aged and Older in ICU: A Retrospective and Machine Learning Study. Preprint, In Review, August 2020a.
Shirly Wang, Matthew B. McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C. Hughes, and Tristan Naumann. MIMIC-Extract: A data extraction, preprocessing, and representation pipeline for MIMIC-III In Proceedings of the ACM Conference on Health, Inference, and Learning. pp. 222 235, Toronto Ontario Canada, April 2020b. ACM. ISBN 978- 1-4503-70066-2 doi: 10.1145/36655553384466
Yuqing Wang, Yun Zhao, Rachael Callcut, and Linda Petzold. Integrating Physiological Time Series and Clinical Notes with Transformer for Early Prediction of Sepsis, March 2022.
Chaoqi Yang, Zhenbang Wu, Patrick Jiang. Zhen Lin, Junyi Gao, Benjamin P. Danek, and Jimeng
Sun. PyHealth: A Deep Learning Toolkit for Healthcare Applications. In Proceedings of the 29th
ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, pp. 5788-5789, New York, NY, USA, August 2023, Association for Computing Machinery. ISBN 9798400701030.
doi: 10.114/3580305.3599777
Hugo Yèche, Rita Kuznetsova, Mare Zimmermann, Matthias Hüser, Xinrui Lyu, Martin Faltys, and Gunnar Rätsch. HiRID-ICU-Benchmaaf A Comprehensive Machine Learning Benchmark High-resolution ICU Data. arXiv:2111.00 111///11.00800 1.08536 [cs], January 2022.
Andy B. Yoo, Morris A. Jette, and Mark Grondona. SLURM: Simple Linux Utility for Resource Management. In Dror Feitelson, Larry Rudolph, and Uwe Schwiegelshohn (eds.), Job Schedul- ing Strategies for Parallel Processing, Lecture Notes in Computer Science, pp. 44-60, Berlin, Heidelberg, 2003. Springer. ISBN 978-3-540-39777--4 doi: 10.69977.
17

---

Published as a conference paper at ICLR 2024
APPENDICES: TABLE OF CONTENTS
APPENDIX YAIB'S CONTRIBUTION IN CONTEXT
APPENDIX EXTENDED RESULTS
APPENDIX C
DATASETS
APPENDIX D
OUTCOME DEFINITIONS
APPENDIX E
YAIB'S USAGE AND IMPLEMENTATION
APPENDIX F.
EXTENDING YAIB
APPENDIX G
EXPERIMENTAL SETUP AND REPRODUCIBILIIY
APPENDIX HJ
HYPERPARAMETER,
APPENDIX I MACHINE LEARNING REPRODUCIBILITY CHECKLIST
A APPENDIX: YAIB'S CONTRIBUTION IN CONTEXT
This Appendix provides an extensive description for the positioning of YAIB in the contemporary clinical ML research landscape. We particularly recommend it to those that are looking into creating their own solutions for clinical MI..
A.1 EXTENSIBILITY AND REPRODUCIBILITY
We designed YAIB to be as extensible as possible while retaining full reproducibility. This means easy support of new databases, clinical concepts, tasks, experiment configurations, preprocessing pipelines, imputation methods, models, and evaluation metrics. If changes necessary, they need to be reproducible and easily shareable across research teams. the user only requires a few default ICU tasks from single i.i.d. dataset to test their new method, any existing ICU benchmarks could be sufficient. Users do not need to apply for access to multiple datasets and do not have to deal with the intricacies of the clinical task definition. As long the integration of simple frameworks are fit-for-purpose and abstract much of the complexity, allowing the user to only worry about one thing: their model. If multiple papers used the exact same benchmark, results also directly comparable between papers (an "apples-to-appe to-appees" comparison).
model is seamless, such
However, we found that this setup tends to be too restrictive and thus unrealistic. Users often want to highlight a particular aspect of their model, prompting them to adapt to the default task. At other times, they want to show clinical impact and need to adapt the default task to make it more realistic. Given the lack of successful translation of prediction models into clinical practice, reviewers are also increasingly requesting external validation sometimes with multiple endpoints which is difficult to shoehorn into most existing solutions. YAIB embraces the need to tweak experimental setups. Results will no longer be directly comparable between papers, but we argue that true apples-to-apples comparisons were inherently rare. Instead of forcing users into a rigid framework, it allows for adaptations but requires them to be done in a transparent manner. Absolute performance should be compared only within the same paper or among papers with the same task setup (see our examples in Tables 5 and 6).
To facilitate the transparency of adaptations, we rely on sophisticated framework to define clinical concepts across multiple datasets (ricu). We have adapted and extended ricu to provide a standard workflow for YAIB to integrate new databases and define new clinical concepts. To date, it has been successfully used to bring 4/5 +1 ICU datasets into a common format (including our addition of the Salzburg Intensive Care Database, which currently quality control). This approach is flexible enough that we have not yet encountered significant restrictions in mapping admissions, demographics, vital signs, laboratory values, medication (including rates and durations), clinical scores, and outcomes at different time scales across datasets. The main restriction of ricu is that it is the R language only, but we provide guidance on how to access it via rpy2,
currently implemented and we are in the process of porting it to Python; this will make our pipeline even more accessible. especially to clinical researchers. Our cohort definition functionality provides helper functions to criteria top of ricu and report step-by-stey attrition numbers. The cohorts can be used modular fashion with custom preprocessing steps, imputations, prediction models, and evaluation metrics, all using the exact same code across multiple datasets.
apply inclusion/exclusee
18

---

Published as a conference paper at ICLR 2024
Even so, there will likely be situations where the user may be better off with custom solution. We expect this to occur once their use case diverges significantly from standard supervised learning. For example, federated learning reinforcement learning setups may require significantly different training and evaluation loops. These are not currently supported, but we consider this as future work. In any case, the user still use our data processing, cohort generation, and possibly other parts of YAIB (e.g., by exchanging the default training module with a custom module). Authors using YAIB should, therefore, provide their code and detailed list of the changes they have made the repository; modern version control allows us to verify this against the original YAIB repository easily,
The YAIB pipeline has helped us produce reproducible results quickly and provides the required extensibility for our purposes. We are in touch with some researchers who have used YAIB to date and provided feedback, although mainly an informal way. We refer to van de Water et all (2023) as example of the usability of YAIB. This work used YAIB as bedrock for implementing imputation methods and are in the process of extending this to more methods and downstream tasks. For concrete examples and guidance for how to extend YAIB, we refer to Appendix D and the wiki documentation.
A.2 THE CHOICE OF FEATURES
We chose the 52 most common clinical features shared by all datasets. They were readily available in all benchmarked datasets, demonstrating YAIB's adaptability. This is done because our work focuses on the interoperability of datasets and the opportunity for experiments with a.o. transfer learning and domain adaption. We believe there is the most value in providing a modular setup where the user can add or remove features to suit their needs better and, most importantly, do so reproducibly.
Nevertheless, several medications for eICU and MIMIC-IV are readily available; the rieu package maintains a full list of the currently available native concepts which are available") Complex concepts, dependent on several native concepts, such as SOFA scores, are additionally available. Each concept that is available in ricu can be readily used in YAIB. Some medications that already implemented, such as antibiotics and vasopressors, are used in the definition of the complex Sepsis endpoint. Therefore, we decided leave those
have the same features for each task.
We note, additionally, that it is straightforward to implement new concepts in our pipeline; Appendix E.2 describes the addition of Potassium Chloride the ICU harmonization package ricu. similar process can be followed for adding new medications, which immediately improves the usability of YAIB. Moreover, we are actively working on integrating more features, including comorbidities and medications. We would like to note that many features are not available across all datasets; this does not mean they can not be valuable in clinical prediction tasks.
Finally, we would like to point out that YAIB's ' end-to-end pipeline is designed as a solid starting point for 1) clinicians looking for external validation to employ ML in practice, 2) dataset creators looking for a solid platform to facilitate widespread use, and 3) the ML community to contribute novel prediction models. They can use a mature and externally developed framework, which adds to the credibility of any experiment results. Adding new feature concepts for their datasets can also increase the adoption of their datasets. They are likely domain experts for their respective datasets, meaning fewer errors are made in this process. This process will improve the usability of YAIB an end-to-end benchmarking tool and improve the confidence of health experts in clinical ML.
A.3 USING YAIB IN NOVEL SCIENTIFIC WORK
We acknowledge the importance of reproducible MI. experiments. In this section, we describe how future work can transparently use YAIB as a platform for comparing their contributions. The authors ideally provide one or more open GitHub repositories so it is straightforward to check versioning; this includes:
com
concept-din dict. json
19

---

Published as a conference paper at ICLR 2024
1. The concept dictionary in JSON format if they add new concepts. The main repository contains the current version of the concept dictionary of the vanilla rict
2. The repository that used generate cohorts if they introduce new task. Ideally, this is forked from the YAIB-cohoo -chhorts repository.
3. The preprocessing py file case this has been changed.
4. The model py and dataset. .y file that contains the definition for the model and dataset and dataloader (if adjusted).
5. model. gin file that specificies the used hyperparameters and hyperparameter ranges.
6. wandb, yml if Weights and Biases is used for running experiments with this model.
7. Provide versions of ri ric YAIB-cohore and YAIB they have used as a base.
If authors cover these aspects when presenting work; easily reproduce their experiments even though they might not have used a "vanilla" implementation of YAIB. An additional benefit of providing these materials is that authors of future work hereby participate in making YAIB more comprehensive.
A.4 EXTENDED RELATED WORK
Comparison to existing frameworks We thank the reviewer for bringing up the preprint of TemporAI, which is still in early development at the time of writing. While we included an earlier work by the same group, Clairvoyance, in our related work, we have now updated the manuscript by adding this work in the related work section and to Table 1. We note that Pyhealth is already included in the related work section of the original manuscript. However, we elaborate on the differences between YAIB and both works below.
A.4.1 CLAIRVOYANCE
Clairvoyance (Jarrett et al., 2021 is "a Unified, End-to-End AutoML Pipeline for Medical Time Series". As such, it does not focus on ICUs benchmarking but instead standardizes model learning (imputation and training). model evaluation, and model selection, focusing on the computational aspects of developing model. Clairvoyance comes with some code to define task for treatment effects estimation on MIMIC III data. However, this task is hard coded and lightly documented, primarily serving as a demo of Clairvoyance. The exemplary nature of this task further exemplified by the fact that, at no point the authors mention possible confounders/colliders of the treatment effect and whether they are conceivably adjusted for by the covariates, rendering any causal interpretation moot. It is unclear how this task can be easily adapted or extended to other databases without significant amounts of custom code.
Advantages of YAIB compared to Clairvoyance: YAIB puts ICU data and tasks front and center. YAIB supports the whole workflow, from raw data to clinical concepts to well-defined cohorts. This approach greatly facilitates the transparent and reproducible preprocessing of (often messy) ICU data, which Clairvoyance does not cover. We strongly believe that unless tasks can be adapted easily and reproducibly. it will lead inevitable ad-hoc adaptations of the task that often end up irreproducible. YAIB, therefore, improves on existing modeling frameworks by putting an equal emphasis on standardized data processing for meaningful model development.
A.4.2 TEMPORAI
TemporAl (Saveliev & van der Schaar 2023) is package that is currently in early development without a peer-reviewed publication associated with it. While promises to provide: "prediction, causal inference, and time-to-event analysis, well common preprocessing utilitics and model interpretability methods," " is unclear from current documentation how to use established datasets with this package or how use relevant medical prediction tasks.
The advantages of YAIB compared to TemporAl are similar to those between YAIB and Clair- voyance: YAIB puts ICU data and tasks front and center for both ML scientists and clinicians.
com wow/eth-md//riiuueee http:://ww.....oooa////////////a
concept-dien dictt. json
20

---

Published as a conference paper at ICLR 2024
YAIB supports the whole workflow, from raw data to clinical concepts to well-defined cohorts. This approach greatly facilitates the transparent and reproducible preprocessing of (often messy) ICU data, which TemporAI, similarly to Clairvoyance, does not cover. However, we would like to note that using TemporAl (or Clairvoyance) with the YA/B pipeline to create a different end-to-end pipeline is possible it allows for "swapping out" components. We provide the functionality in CoIB-cohorts repository to convert any cohort to a format compatible with Clairvoyance and TemporAI.
A.4.3 PYHEALTH
PyHealth YYng et al. 2023 is "a comprehensive deep learning toolkit designed for both ML researchers and healtheare practitioners. PyHealth aims to support all EHR databases. It is thus similar in scope to proposed framework. Unfortunately, upon closer inspection, PyHealth only supports a small subset of the information in MIMIC and eICU, While diagnoses and prescriptions are, in theory, included, they processed as simple bag of diagnosis codes drug codes without information on strength/duration or semantic interpretation of what they represent (c.g., what is vasopressor needed in calculating the SOFA score). Vital signs are not supported all, presumably because PyHealth reads information from raw .csv files and may struggle to process large quantities of vital sign data. PyHealth further states that the datasets are independent of task definitions. This, unfortunately, appears to mean that they have to be implemented anew for each database, with custom dataset-specific code for the same task. Furthermore, all currently available ICU tasks in PyHealth use static data only and do not include any time series.
Advantages of YAIB compared to PyHealth: YAIB supports all databases within a common, principled interface (see the response data harmonization above). Moreover, YAIB enables single task definition that one can directly use for any included dataset. As far as they can work with time series data, YAIB can incorporate any model defined in PyHealth.
B APPENDIX: EXTENDED RESULTS
This Appendix contains results that were left out of the main text.
TABLE 7: Comparing the use of dynamic generation (FG) to the buseline of ICU mortality prediction. AUROC (1). Note that an otherwise identical experiment setup was used to obtain results for the "without feature generation" results,
AUMCdb
HiRID
eCCU
MIMIC-IV
Preprocessing
w/ FG
w/o FG
w/ FG
w/o FG
w/ FG
w/o FG
w/ F/
w/o FG
LR
83.720.6
82.2+0.5
84.0s0.3
81.8
84.8+0.2
81.140.1
86. 140.1
80.2
LGBM
84.5+0.6
83,520.4
84.5+0.3
82.540.6
85.720.2
83.540.3
87.720.2
85.940.1
TABLE Comparing the use of dynamic feature generation (FG) to the baseline of ICU mortality prediction. AUPRC (t). Note that an otherwise identical experiment setup was used to obtain results for the "without feature generation" results.
AUMCdb
HiRID
eICU
MIMIC-IV
Preprocessing
w/ FG
w/o FG
w/ FG
FG
w/ FG
w/o FG
w/ FG
w/o FG
LR
12.9=1.2
10.
36.9+1.1
33.1
33.0+0.7
28.3+0.4
39.7+0.6
32.16 44.2+0.7
LGBM
53.7+1.2
50.9+1.0
40.6+0.8
35.3+0.9
36.0#0.6
32.5+0.9
40.1
Feature generation We compare the use of feature generation for classical MI models. The RECIPYS package provides the functionality of assembling different preprocessing steps to be supplied by the user (i.e., recipe). We show the results in Table and g
The impact of static features We leveraged this customizable preprocessing to perform training prediction models without static features (i.e., age, sex, height, and weight). The AUROC results
21

---

Published as a conference paper at ICLR 2024
TABLE 9: AUROC (1) performance comparison of including static data for ICU mortality prediction.
AUMCdb
HiRID
eICU
MIMIC-IV
Inclusion
w/ static
w/o static
static
w/o static
w/ static
w/o static
w/ static
w/o static
LR
83.720.6
82.940.6
84.020.3
82.8+0.7
84.8+0.2
84.3+0.2
86. 16.11
84,3+0.3
LGBM
84.5+0.6
83.4+0.4
84,4+0.3
83.9+0.7
85.7.0.2
84.7+0.2
84.7+0.2
86.240.2
GRU
83.7=0.7
83,4+0.6
84.3+0.7
84.0=0.8
85.9+0.2
85.7+0.2
87.4+0.2
86.9+0.3
LSTM
83.7+0.7
82,9+0.6
84.0+0.7
83,4+0.7
85.5+0.2
85.1+0.2
86.7+0.4
86.1+0.3
TCN
84.0+0.6
83,5+0.6
84.620.7
83,9+0.8
85.420.2
86.4+0.3
87.140.3
86.440.3
84.1+0.2
83.720.4
84.920.7
84.4+0.77
85.940.2
85.740.2
86.940.3
86.540.3
TABLE 10: AUPRC (1) performance comparison of including static data for ICU Mortality Prediction.
AUMCdb
HiRID
eICU
MIMIC-IV
Inclusion
w/ static
w/o static
w/ static
w/o static
w/ static
w/o static
w/ static
w/o static
LR
52.9+1.2
51.7=112
36.9+1.1
34.0#1.1
33.0=0.7
32.2+0.6
39.7+0.6
36.9=0.6
LGBM
53.7*1.2
51.1#1.1
40.620.8
38.9+1.5
36.0=0.6
34.1+0.7
44.2+0.7
41.0=0.7
GRU
53.1+1.5
52,9=1.2
37.6+1.2
37.3+1.1
36.1=0.9
35.3+0.8
42.4+0.6
41.5=0.7
LSTM
53.6+1.4
51.0=1.0
37.8+1.0
36.2+1.4
35.7=0.8
34.6+0.7
41.0+0.7
40.2+0.8
TCN
54.2+1.4
52.7+1.0
39.2+1.3
37.5+1.5
34.3#0.6
35.4+0.8
41.4+0.8
40.8=0.7
TF
54.4+1.1
52.7+0.9
39.3+1.5
38.4+1.5
34.7=0.8
34.5+0.7
42.240.3
41.3+0.8
found in Table 9 (the AUPRC in Table 10). From these results, we can see that including the static data seems result in better performance across all models and datasets.
TABLE 11: Baxeline AUROC (1), AUPRC (1) performance of the included ML algorithms on the demo cohorts.
eCCU Demo
MIMIC-III Demo
Algorithm
AUROC
AUPRC
AUROC
AUPRC
ICU Mortality
LR
67.7+0.9
15.4
52.9+3.2
37.6+2.3
LGBM
72.9+1.1
21.4+1.7
59.0+2.1
33.0#2.4
GRU
71.5+1.
20.8
50.5+3.6
33.943.5
LSTM
11.3+1.6
18.9&1.9
55.5+2.3
34.2+3.3
TCN
69.2+1.7
18.0#1.5
15.
38,8+3.6
TF
71.6+1.3
18.7+1.7
$3.3+3.5
36,8+3.1
AKI
LR
61.3+0.6
16.820.4
52.9+2.3
16.8+2.0
LGBM
72.620.3
23.8+0.4
60.7+1.7
22.4+18
GRU
63.0#0.8
17.3+0.6
50,5+3.3
17.6
LSTM
61.9+0.9
16,2+0.6
53.542.5
15.41.
TCN
64.5+1.0
17.6+0.6
53.7+2.8
19.6+2.0
TF
70.5
21.7+0.7
53.9+2.0
15.3+1.0
Sepsis
63.8+0.9
3.7+0.3
LGBM
53.5+1.5
2,8=0.2
GRU
64.7+1.3
4.140.3
4
8
LSTM
65.3+1.3
4.3+0.5
4
TCN
66.4+1.1
4,240.3
TF
68.4+1.1
5.8+0.6
Our sepsis definition resulted in just one sepsis case for the MIMIC-III demo dataset. As result, we could not use the 5-fold cross-vat vols-vvaiiation approach to train model reliably.
Demo datasets We offer out-of-the--or (i.e., executable straight after downloading the repository) experiment definitions with five tasks defined on two demo datasets: MIMIC-II demo and eICU The traditional ml models perform better,
demo. The results can be seen in Table 11 and T2
22

---

Published as a conference paper at ICLR 2024
TABLE 12: Baseline performance the regression tasks. Results reported in Mean Absolute Error (L).
Kidney function
Length of Stay
cICU Demo
MIMIC- III Demo
eCCU Demo
MIMIC-III Demo
EN
0.30=0.00
0.33+0.03
38.5+0.2
52.1+1.3
LGBM
0.87+0.01
0.86#0.04
37.7+0.2
50.4+1.1
GRU
0.54+0.02
3,23=0.19
39.7+0.6
54.7+1.0
LSTM
0.51=0.02
2.95#0.18
39.1+0.5
54.8+1.2
TCN
0.46#0.02
2.98#0.11 19
38,4+0.6
56.8+1.2
TF
0.60#0.03
3,22+0.18
38.9+1.2
57.1
52
AUMOeb
42.9
46.5
44.1
46.2
-50
Training Dataset
HIRID
52.1 1
39.6
41.6
44.7
MAE better)
e/C
51.0
47.2
38.9
41.8
MIMIC-N
50.3
49.7
40.8
39.9
42
Pooled (d-1)
47.1
4.5.7
38.8
42.5
440
AUMCdb
HRRID Evaluation Dataset
eICU
MIMIC-V
FIGURE 4: Performance in MAE of the Transformer model length of stay (LoS).
most likely explained by the low number of samples. The kidney function task highlights the large difference in performance especially.
External validation (extended) be found in Figure 4
The length of stay (LoS) results for ICU mortality prediction can
Fine-tuning (extended) The AUPRC results for experiment with transfer learning can be found below and show a similar trend Figure
AUPRC (higher is better)
30
25
Full HiRID only
Train on HIRID
Full eICU only
Fine-tune eICU on HiRID
15
4000
8000
10000
Samples
FIGURE AUPRC fine-tutunnig an eICU GRU model for ICU mortality prediction on HiRID.
23

---

Published as a conference paper at ICLR 2024
C
APPENDIX: DATASETS
This Appendix contains detailed description of the datasets and the preprocessing methodology.
C.I DATABASE CHARACTERISTICS
The Medical Information Mart for Intensive Care (MIMIC)-III dataset is the most commonly used dataset used for ML in ICU settings: Syed al. (021) found 61 eligible studies that used a form of the MIMIC dataset. It was collected in the USA at the Beth Isreal Deaconess Medical Center JJhnson al. 2016). The newer MIMIC-IV includes several improvements, among which newer patient records and revised structure including regular hospital information (Johnson et al.) 2023). The eICU Collaborative Research Database (elCU) (Pollard et al. 2018) an effort to collect the first sizable (200,000 admissions) multi-center dataset. It was collected using Philips ICU monitoring systems in the USA at 208 participating hospitals. The High Time Resolution ICU Dataset (HiRID) dataset was collected at Bern University Hospital, Switzerland, and has incorporated more observations than the aforementioned datasets (Hyland 2020). The Amsterdam UMCdb (AUMCdb) is the most recently released ICU dataset (Thoral al. 2021). Collected in the Netherlands, has a temporal resolution of up to 1 minute and has prioritized patient de-identification. Note that there is no benchmark software for this dataset yet. Each dataset we are using has undergone de-identification procedures, and we have not tried to re-identify the people involved, as per the user agreement for each dataset. Table 13 shows some key characteristics of each dataset. more comprehensive overview of ICU datasets can be found in the work of /auer et al. (2022b).
TABLE 13: Supplemental details of openly accessible ICU datasets. Note that accessing each dataset requires
completing a credentialing procedure.
Dataset
MIMIC-II/
eICU
HiRID
AUMCdb
Stays
40k (0. 10)=K)== 773k 73k
201k (2k)
34k
23k
Version
v1.47 / v2.2
v2.0
vI.1.1
v1.0.2
Frequency (time-series)
hour
minutes
2/5 minutes
up to minute Netherlands
Origin
USA
USA
Switzerland
2015 (Johnson et al. 2016
2017 PPollrdd
2020 (Hyland
2019 Thoral
Originally published
2020 (Johnson al.
ct al.
2018)
2020)
al 2021
2023)
License
(C)7A
A (C)
Repository link
[
[(
Note that accessing each full dataset requires completing credentialing procedure.
*; Stays were taken and rounded from the latest available versions of the databases of the time of writing.
**; The brackets () indicate characteristics of the demo (freely accessible) version the dataset
A: PhysioNet Contributor Review Health Data License 1.5.0)
B: Access Request Form and End User License Agreement for AmsterdamUMCddb 1.6
C: Open Data Commons Open Database License v1.0
p: Physionet
a: Amsterdam Medical Data Science
The authors of MIMIC-III and eICU have made small selected datasets available for the purpose of experimentation. These datasets are also publicly available on Physionet. We support the publicly accessible "demo" datasets provided for eicu 10 and MIMIC-II In accordance with the demo dataset license (Open Data Commons Open Database License v v1.0, see Table 13] License C). it is permitted to adapt and share the data. Still, we recommend the user to complete human subject research training to make sure the usage of the dataset does not violate the usage proposal. They contain respectively 2,500 (eICU) and 100 stays (MIMIC-III) before exclusion. For the purposes testing and validating YAIB, we have created demo-cohorts, extracted solely from these datasets, for each of our supported tasks. Usage of the task cohorts and dataset is only permitted in accordance with the above license.
16 "ftrass//lnassnntt..cccooooooood httpss///pyysinnerr..oo//oootttm
24

---

Published as a conference paper at ICLR 2024
TABLE 14: Characteristics the included datasets (above) and 2) the task cohorts (below).
General characteristics
AUMCdb
HiRID
eCCU
MIMIC-IV
Version
1.02
1.1.1
2.0
2.0
Number of patients
19,790
160,816
53,090
Number of ICU stays
22,636
32,338
182,774
75,652
Age at admission (years)
65 [55, 75]² 7,699 (35)
65 [55, 75]
65 [53, 76] 83,940 (46)
65 [53. 76] 33,499 (44)
Female
11,542 (36)
Race
Asian
3,008 (3)
2,225 (3)
Black
19,867 (11)
8,223 (12)
White
140,938 (78)
51,575 (76)
Other
16,978 (9)
5,514 (8)
Unknown
1,983
8,115
Admission type
Medical
4,131 (21)
134,532 (79)
49.217 (65)
Surgical
14,007 (72)
31,909 (19)
25,674 (34)
Other
1,225 (6)
4,702 (3)
761 (1)
Unknown
1,069
11,631
Hospital length of stay (days)
[3. 10]
7 [4, 13]
Task cohorts
AUMCdb
HiRID
eCCU
MIMIC-IV
ICU mortality
Number of included stays
10,535 1,660 (15.8)
12,859
113,382
52,045
Died
1,097 (8.2)
6,253 (5.5)
3,779 (7.3)
Onset of acute kidney injury
Number of included stays
20,290
31,772
164,791
66,032
KDIGO > I
3,776 (18.6)
7,383 (23.2)
62,535 (37.9)
27,509 (41.7)
Onset of Sepsis
Number of included stays
18,184
29.894
123,864
67,056
Sepsis-3 criteria
764 (4.2)
1,986 (6.6)
5,835 (4.7)
3,730 (5.6)
Kidney function (creatinine)
Number of included stays
8,003
7,499
69,117
35,657
Creatinine value
0.97 [0.70, 1.61]
0.92 [0.67, 1.50]
1.00 [0.71, 1.68]
1.00 [0.70, 1.60]
ICU remaining length of stay Number of included stays
22,636
32,338
182,774
75,652
ICU length of stay (hours)
24 119, 771
24 [19, 50]
42 123, 761
48 [26, 89]
1
HiRID only provides stay-level identifiers.
Since AUMCdb only includes age groups, calculated the median of the group midpoints.
* KDIGO, Kidney Discase Improving Global Outcomes (KDIGO 2012).
Numeric variables are summarized by median [IQR].
Categorical variables are summarized by incidence (%).
C.2 EXCLUSION CRITERIA
We included all available ICU stays of adult patients in our analysis. For each stay, we applied the following exclusion criteria ensure sufficient data volume and quality: remove any stays with 1) an invalid admission or discharge time defined as a missing value or negative calculated length of stay, 2) less than six hours spent the ICU, less than four separate hours across the entire stay where at least one feature was measured, 4) any time interval of > 12 consecutive hours throughout the stay during which no feature was measured. FFigure 6 detaiis the number of stays overall and by dataset excluded this way.
Additional exclusion criteria were applied based on the individual tasks; the details can be found schematically in Figure [ For ICU mortality, we excluded all patients with a length of stay of fewer than 30 hours (either due to death or discharge). A minimum length of 30 hours was chosen to exclude any patients that were about to die (the sickest patients) or be discharged (the healthiest patients) at the time of prediction at 24 hours. For creatinine (kidney function), we excluded all patients with length of stay of fewer than 48 hours without creatinine measurement between 24 and 48 hours (which was the outcome of interest). For AKI and sepsis, we excluded any stays where disease onset was outside the ICU within the first six hours of the ICU stay. To account
25

---

Published as a conference paper at ICLR 2024
All ICU stays
N 334,812 (A: 23, 106: H: 33,904; E: 200,859; M: 76,943)
Invalid admission or discharge times
(A: 0: 2
Less than 6 hours the ICU = 18,322 (A: 349; H: 507; E: 16,412; M: 1,054)
than 4 hourly bins with clinical measurements
532
19; 430; M:
More than 12 consecutive hours without any clinical measurements
N 2,101
(A: 102; H: 1,040; E: 786; M: 173)
ICU stays with sufficient data
N 313,855 (A: 23; H: 32,338;
E: 183,229: M: 75,652)
Aged less than years
admission
(A: O: E: 455; M: 0)
Base cohort
= 313,400 (A: 22, H: 32,338; E: 182.774; M: 75,652)
FIGURE 6: Exclusion criteria applied to the base cohort. N: Total amount of cases. A: AUMCdb. H: HiRID. E: eICU, M: MIMIC-IV
26

---

Published as a conference paper at ICLR 2024
Hospital without, single
(A: 0; H: 0: E: 44,444.444 494 M: O)
Sepsis onset before the hour in the ICU
N 29, 29,908 (A: 4,45 4,452: H: 2,44; E: 14,416,6 M: 8, 8,596)
sepsis case
44, 494
N = 238,998 (A: 18, 184; H: 29, 99, 123 864; Mt 056)
Sepsis cohort
FIGURE 7: Additional exclusion criteria applied for the classification tasks.
Baseline creatinine mg/dL
Hospital without single AKI case
(A: H: 0; 20; M: O)
AKI onset before the hour the ICU
(A: AA11,80 1,930; H: 6; E: 9,877; M: 8,759)
= 11 1,929 (A: 416; H: 580; 08,092 1,861)
N 18,588 3,666
20
N = 282 885 (A: 20, 20,2900 H: 772; E: 164,791 M: 66,032)
AKI cohort
(A: 22.6 83,63; H: 32, 338; E: 182.77 774
M: 75, 75,862)
Base cohort
= 313,400 40000
Discharged alive within 30 hours of ICU admission
M: 1,513)
Died within 30 hours of ICU admission = 5,842
N 118.737 (A: 11 6,6 H: 722: E: 66,315; M: 22,094)
(A: 495 H: 757; E: 3,077,
mortality cohort
= 188,821 (A: 10 00,535; H: 12,855 859 11; M: 52, 045)
27

---

Published as a conference paper at ICLR 2024
cohort 313.400
(A: 22-22.00 606 92.358 182.774. 75.652)
Length of stay less than 48 hours
N 162,672
(A: 14,48 438; Ft 23 804: 105,542; M 58, 800))
creatinine measured
N 10,452 (A:197) 1,005; 8,115, M: 1,105)
Length - 913.400 (A: 22,838; H 338; E: 182,774, M: 75,662)
Kidney function cohort
N 120,276 (AL & 00000 H: 7,40; E: 60,117, 35,657)
FIGURE 8: Additional exclusion criteria applied for the regression tasks.
for differences in data recording across hospitals in eICU, we further excluded hospitals that did not have single patient with AKI or sepsis to exclude hospitals with an insufficient recording of features necessary to define the outcome. Finally, for the AKI task, we excluded stays where the baseline creatinine, defined the last creatinine measurement prior ICU (if exists) the earliest measurement in the ICU, was >4 mg/dl. to exclude patients with preexisting renal insufficiency. For numerical overview, please consult Table 14
C.3 PREPROCESSING
A total of 52 features were used for model training (Table 15), of which were static and 48 that were dynamic. These features were selected as they are available across all datasets for most patients. Dynamic features primarily include vital signs (7 variables) and laboratory tests (39 variables), with two more variables that measure input (fraction of inspired oxygen) and output (urine). All variables were extracted via the ricul R package (version 0.5.3). The ricu name for each package is shown in Table 15 The exact definition for each feature and how was extracted from the individual databases be found in the concept configuration file of the package's GitHub repository (commit 885bd0c). We also provided cohort definition code for this work, which can be run in both R and Python, github repository.
TABLE 15: Clinical concepts used us input to the prediction models.
Feature
ricu
unit
Static
Age at hospital admission
age
Years
Female sex
sex
Patient height
hei
cm kg
Patient weight
weight
Time-varying
Blood pressure (systolic)
sbp
mmHg
28

---

Published as a conference paper at ICLR 2024
TABLE 15: Clinical concepts used input to the prediction models (continued)
Feature
ricu
unit
Blood pressure (diastolic)
dbp
mmHg
Heart rate
hr
beats/minute
Mean arterial pressure
map
mmmg
Oxygen saturation
o2sat
Respiratory rate
resp
breaths/minute
Temperature
temp "C
Albumin
alb g/dL
Alkaline phosphatase
alp
IU/L
Alanine aminotransferase
alt
IU/L
Aspartate aminotransferase
ast
IU/L
Base excess
be
mmol/L
Bicarbonate
bicar
mmol/L
Bilirubin (total)
bili mg/dL
Bilirubin (direct)
bili_dii
mg/dL
Band form neutrophils
bnd
Blood urea nitrogen
bun
mg/dL
Calcium
mg/dL
Calcium ionized
cai
mmol/L
Creatinine
crea
mg/dL
Creatinine kinase
ck
IU/L
Creatinine kinase MB
ckmb
ng/ml.
Chloride
cl
mmol/L
CO2 partial pressure
pco2
mmilg
C-reactive protein
crp
mg/L
Fibrinogen
fgn
mg/dL
Glucose
glu
my/dl
Haemoglobin
hgb
g/dL
International normalised ratio (INR)
inr_pt
Lactate
lact
mmol/L
Lymphocytes
lymph
%
Mean cell haemoglobin
mch pg
Mean corpuscular haemoglobin concentration
mche
Mean corpuscular volume
fL
Methaemoglobin
methb
Magnesium
mg/dL
Neutrophils
neut
O partial pressure
po2
mmlig
Partial thromboplastin time
ptt
pH of blood
ph
Phosphate
phos
mg/dL
Platelets
plt
1,000 / ul.
Potassium
mmol/L
Sodium
na
mmol/l.
Troponin
tnt
ng/mL
White blood cells
wbs
1,000 / pl.
Fraction of inspired oxygen
fio2
%
Urine output
urine
ml.
Additional features Furthermore, consulted clinical experts identify which features might be missing from our prediction setup. Several clinical features are currently missing from this setup, which could potentially improve prediction performance: Glasgow coma scale score, Intubation, Ventilator settings. Renal replacement therapy, and Vasopressors. We expect to be able to integrate more concepts collaborate with authors of datasets make them available.
29

---

Published as a conference paper at ICLR 2024
D APPENDIX: OUTCOME DEFINITIONS
The outcome definitions per task for each dataset are detailed in this Appendix.
D.1 ICU MORTALITY
ICU mortality was defined as death while in the ICU. This was generally ascertained via the recorded discharge status or discharge destination. Note that our definition of ICU mortality differs from the definition of death in the ricu package, which describes hospital mortality that is unavailable for some included datasets.
AUMCdb Death was inferred from the destination column of the admi ssions table. A destination of "Overleden" (Dutch for "passed away") treated death in the ICU. Since the date of death was recorded outside the ICU and may therefore be imprecise, the recorded ICU discharge date used
precise proxy for the time of death.
HiRID Death was inferred from the column dis cischarge_stateeee tus in table general. The status of "dead" treated death in the ICU, Time of death was inferred as the last measurement of IDs 110 (mean arterial blood pressure) or 200 (heart rate) in column variableid of table observations.
eICU Death was inferred from the column unit status of "Expired" was treated as death in the ICU. The recorded ICU discharge date was used as proxy for the time of death.
in table pat patient. The
in table admi s ssssiooi iioss. Since MIMIC IV only records joint ICU/hospital expiration flag, ward trans- fers were analyzed ascertain the location of death. the last ward was the ICU, the death was considered ICU mortality.
MIMIC IV Death was inferred from the column hospital_expiee
TABLE 16: Staging of AKI according to KDIGO KDIGO 2012
Stage
Serum creatinine
Urine output
1.5-1.9 times baseline
OR
<0. ml/kg/h for 6-12 hours
20.3 mg/dl (226.5 jmol/I) increase within hours
2.0-2.9 times baseline
<0.5 ml/kg/h for 2 12 hours
3.0 times baseline (prior 7 days)
OR
<0.3 ml/kg/h for 224 hours
Increase in serum creatinine 24.0 mg/dl (2353.6 μmol/I) within 48 hours
OR
Anuria for 2 hours
OR
Initiation of renal replacement therapy
AKI, acute kidney injury; KDIGO, Kidney Disease Improving Global Outcomes.
D.2 ACUTE KIDNEY INJURY
AKI was defined KDIGO stage >> either due increase in serum creatinine or low urine output (Table 16) (KDIGO) 2012). Baseline creatinine was defined as the lowest creatinine measurement the last 7 days. Urine rate calculated the amount of urine output in ml divided by the
30

---

Published as a conference paper at ICLR 2024
number of hours since the last urine output measurement (for max gap of 24h), except for HiRID, in which urine rate was recorded directly. The earliest urine output was divided by 1. The rate per kg was calculated based on the admission weight. If weight missing, weight 75 kg was assumed instead.
AUMCdb Creatinine defined via the standard ricu concept of serum creatinine IDs 6836, 9941, or 14216 6 in column i itemic of table numer seriiiitems.... tmms. Urine output was defined as IDs 8794, 8796, 8798, 8800, 8803 in column itemid of table numerica ems (note that this includes more items than those included in the standard r cu concept of urine output).
HiRID Creatinine was defined via the standard ricu concept of serum creatinine IDD 20000600
in column variable of table ioservations. Urine rate was defined as ID 10 10020000 in column variableid of table observations.
eICU Creatinine was defined via the standard r ricu concept of serum creatinine as IDs "creatinine" in column labname of table lab. Urine output was defined via the standard ricu concept of urine output as IDs "Urine" and "URINE CATHETER" in column celllabel of table intakeout
MIMIC IV Creatinine was defined via the standard ri cu concept of serum creatinine as ID 50912 in column i temi d of table 1 labevents. s. Urine output was defined via the standard ricu concept of urine output as IDs 226557, 226558, 22 6559, 226560, 226561, 22 2563,3, 226564, 226556, 6566,, 226567, 226584, 227510 0 in column itemid of table outputevents.
TABLE 17: Comparing MIMIC-IV sepsis cohorts according to three different definitions.
Cohort
Seymour et al.
2016)**
Moor et al. (021a))
Calvert et al. 2016
Stays
67,056
53,642
65,901 2,406 (3.7%)
Prevalence
3,730 (5.6%)
8,919 (16.7%)
*Our default definition.
D.3 SEPSIS
The onset of sepsis was defined using the Sepsis-3 criteria (Singer et al. 2016). which defines sepsis as organ dysfunction due to infection. Following guidance from the original authors of Sepsis-3 (Seymour et al.] 2016), organ dysfunction was defined as an increase in SOFA score 22 points compared to the lowest value over the last 24 hours. Suspicion of infection was defined as the simultaneous use of antibiotics and culture of body fluids. The time of sepsis onset was defined as the first time of organ dysfunction within 48 hours before and 24 hours after suspicion of infection. Time of suspicion was defined as the earlier antibiotic initiation or culture request. Antibiotics and culture were considered concomitant if the culture was requested <24 hours after antibiotic initiation or if antibiotics were started <22 hours after the culture sent the lab. Where available, antibiotic treatment was inferred from administration records; otherwise, we used prescription data. To exclude prophylactic antibiotics, we required that antibiotics were administered continuously for 23 days (Reyna et al., 2019). Antibiotic treatment was considered continuous if an antibiotic was administered once every 24 hours for 3 days (or until death) or was prescribed for the entire time spent in the ICU. HiRID and eICU did not contain microbiological information. For these datasets, we followed Moor et al. (202 la) and defined suspicion of infection through antibiotics alone. Note, however, that the sepsis prevalence in our study was considerably lower than theirs, which was as high as 37% in HiRID. We suspect this is because they did not require treatment for 23 days. For comparison and to contextualize the results of our experiments, have benchmarked other sepsis definitions. See Table 17 for the number of stays and incidence for cach cohort in this experiment performed MIMIC-IV.
AUMCdb The SOFA score, microbiological cultures, and antibiotic treatment were defined via the standard r ricu concepts sofa, abx, and samp (see the ricuu package for more details).
HiRID The SOFA score and antibiotic treatment were defined via the standard ricu concept sofa and abx (see the ricu package for more details). No microbiology data were available in HiRID.
31

---

Published as a conference paper at ICLR 2024
eICU The SOFA score and antibiotic treatment were defined via the standard r ricu concept sofa and abx (see the ricu package for more details). Microbiology data in eICU was not reliable (Moor et al. 2021a) and therefore omitted.
MIMIC-IV The SOFA score and microbiological cultures were defined via the standard ricu concepts sofa and samp (see rrii package for more details). Antibiotics were defined based on table inputevents. This differs from the standard r ricu abx concept, which also considers the prescription table.
D.4 KIDNEY FUNCTION
The median creatinine level over the course of the second day of patient's ICU stay was defined as the target of interest. Given the available datasets, this was chosen as the most suitable proxy of kidney function.
AUMCab Creatinine was defined via the standard ricu concept of serum creatinine as IDs 6836, 9941, or 14216 6 in column itemid of table numeric e tems.
HiRID Creatinine was defined via the standard ricu concept of serum creatinine as 20000600 in column variableid of table observations.
eICU Creatinine was defined via standard ricu concept serum creatinine as IDs "creatinine" in column labname of table lab.
MIMIC IV Creatinine was defined via the standard r cu concept of serum creatinine ID) 50912
in column emid of table labevents.
D.5 REMAINING LENGTH OF STAY
At each hour, the remaining length of stay in the ICU was calculated in hours until discharge. A maximum forecasting window of 7 days was chosen, forecasts beyond this interval were judged extremely difficult and of lesser clinical relevance. Arguably, an even shorter window 3 days could be chosen support short-term capacity planning. This is left for future investigation.
AUMCdb Length of stay was calculated via the standard ricu concept using the columns admittedat and dischargedat of table admissions.
HiRID Length of stay was calculated via the standard ricu concept using the column Admissiontine ime of table general well the last observation table observations.
eICU Length of stay was calculated via the standard ricu concept using the columsn unitadmitoffset and mnitdischaargote
of table patient.
MIMIC IV Length of stay was calculated via the standard ricu concept using the columsn int ime and cut of table icustays.
E APPENDIX: YAIB'S USAGE AND IMPLEMENTATION
We describe the most important practical aspects of using YAIB in research in this Appendix. Please note that there wiki, dedicated to usage and development, available at: https: /taas://iiee githhbb com/ hamm/raanddwatee//A2B
E.1
USE OF EXISTING CODE REPOSITORIES
As mentioned in the main text, we used parts of the HiRID-Benchmark code and heavily modified and extended the code to support our extra features and extensibility, mentioned Table T HiRID- Benchmark is available at GitHub and makes use of an MIT license does code repository.
E.2 ADDING A DATA SOURCE
Adding a new dataset type, to use within YAIB, can be easily done by providing it in a gin task definition file, see Code Listing 1 Note, however, that any datasets formatted in the default way do
32

---

Published as a conference paper at ICLR 2024
not require any changes to be used by YAIB. By default, we have chosen work with the Apache parquet (Vohra 2016) file format, which is a modern, open-source polumn-orientee format that does not require lot of storage due to efficient data compression" | separate files: DYNAMIC, STATIC, and OUTCOME; this is defined for dynamic variables (that change during the stay), constant parameters, and the prediction task label respectively. Our cohort definition code produces the files in exactly this format. Furthermore, we see the concept of roles with the definition of the vars dictionary. These roles are assigned defined in RECIPYS, the preprocessing package developed alongside YAIB. The GROUP variable defines which internal dataset variable should be used to "group by" for, e.g,, aggregating patient vital signs. The SEQUENCE variable defines the sequential dimension of the dataset (in the common case, this would be time). The other keys in this dictionary define the feature columns and outcome variables for prediction.
We separate the data into three
CODE LISTING 1: Example preprocessng pipeline structure.
8gin,conf conf DefaultCiassid ficat conPio Soonrrerroeeeee [Pregroceszor]
gurable i Pl classi Fle fon preprocessor")
ded init (self, generate peprrate_teette teatures: bool Tree, scaling: bool True, use ase_static teatuuresstt bool True):
Azge:
generate teatures:
Generate teatures dynemic data, scell scellngg: Sowli of dyranic and st data. ieeesstati__eee teaturess Usa static reatures.
Returne:
Preprocessed
10
11
12
13
apply (self, data, vare):
15
A2gs:
16
Trein, validation and test dictionary. Further divided static, cynamic, Variables for statie, cynanic, gutcome,
Reterns:
19
Preprocessed data.
20
21
return data
23
24
def __ppeets__tttiileelle data, vars!:
25
return data
27
28
def wwwpppoossssoom pp.prresssddnanccieellle data, vare):
29
30
return dcta def
31
32
datc, dyasnic_vars):
return data
E.3 CREATING A PREPROCESSING PIPELINE
Our preprocessing pipeline is set up to be as general as possible and allows for custom implementa- tions, defined as subclass from the Preprocessor class and passed command-lin argument. For our tasks, we have defined a default preprocessing pipeline for both classification and regression tasks. Code Listing 2 shows the class structure of the default classification preprocessor. In the private methods of this class, RECIPyS is used to apply feature generation steps (which differ for ml and dl models). The abstract P Preprocessor has two functions that need to be implemented: init__ () (which initializes the preprocessor and configures the settings) and apply (data) (which returns the preprocessed data dictionary of features and labels for each of the train, validate, and test splits)
E.4 EXAMPLE: TRAINING A MORTALITY PREDICTION LSTM MODEL
We demonstrate the complete process of training a Long Short-Term Memory (LSTM) model to predict sepsis on the MIMIC-III demo dataset with YAIB. This is shown schematically in Figure 9 Code Listing 2 the basic task setup for mortality prediction after 24 hours is shown. We define the dataset files. by default split into parquet files with the corresponding names. The listing describes three dataset components: dynamic data, outcome definitions, and static data. Below, one sees the variables and different "roles" assigned to concrete strings (see Appendix E.2 for detail). In this
htttps:///aara hwwppaaruuetaaaaae.e.ooee apache.org
33

---

Published as a conference paper at ICLR 2024
LR
HIRID
LGBM
Configuration
Configuration
AKI
GRU
AUMCdb
Sepais
LSTM
Task Definiton
TCN
Lo5
MIMIC-IIVV
Transformer
Model Settings
Hyperparameters
Datasets Preprocessing
Prediction Tasks
Models Hyperparameters
FIGURE 9: Experiment definition schematic. The fundamental experiment configuration of the benchmark contains three hasic elements, 1) the dataset, 2) the prediction task, and 3) the model and (list of) hyperparameters. Each element be combined in different ways. Additionally, we provide interface for extending each element (Datasets Preprocessing. Prediction Tasks, Models Hyperparameters) an example of an experiment configuration: predicting Sepsis on MIMIC (Thoral et al. 2021) with an LSTM (Hochreiter & Schmidhuber 1997) model.
process. Provided
CODE LISTING 2: Example configuration for task definition configuration. We identify 4 main categories: the prediction lype (classification regression), the deep learning loss, the dataset variables, and preprocessing settings, and the cross validatii settings. Note that the dataset is fully configurable here.
inciude confiosn/ /task////////////////aaa http////////////////////////o/c/ comon// Emports.ginn gin"
MODE SETTING
mune "clesoizicscion"
MUN_CLASSE CLASSES Binary clorssticatiin
B0RI2CN 24
reight "balanced"
DBEP LEARNING
10
b.Predditt.oo::appor.. wew.eppper,.oeee @crors_caciopy
11
DATASET PRSPROCESSINS
pzopzocasa.co catbcoasa.ille_aaeee.
file__anes
"DYREMI"* "dyn.parquot",
*OUTCOME*; "eute.parquels
*STATIC": "sta.parqust",
vars "GROUP": "stay_id", "LABEL"; "Isbol", "SEQURML""; "Line", "DYRAMIC" t"all", talp", "alt", "ast"l, "bet) "bicaet, "bils"y "Bi14_die", "bod", "bun", feat, healf, Tek", "okmbi", "ol", *crea", "erp", "dbe", "rge", "r102", Polut, thob", thet, "inr_pe", "k7, "lass" "Iyaph" "oap", Toat", nahe", "nav",
19
22
"muihls, "ng"" Poa", feeuth, fu2sal "pou25, fpii, "plost, tplcn, "poRt, "auuh, "reap"l "sbp", "Loup" "Ent", *urine", "Mbc"l, "STATIO": Paay", feax", "hiight", "veight"),
24
27
28
SBLECTING EREPROCESSOR
prepsocass, prapzocassee plso__aiaattlaatoopppcccoaas
prepracess, vars Ivars
31
preprocess.coo static Tzum
32
SELECTING DATAGET
Predie: Cediiesionneteoo aundetasee..vars Svars
35
36
VALIDACTON
Seaoute_regwatee execute regested wtd.dedd..ee.... cov. foll folids
listing, we also pass the vars to the preprocessing and dataset class. Finally, we see the definition of the cross-vall validatio folds and iterations.
In Code Listing 3 we see the configuration for the LSTM. We first define the generating features from dynamic data (relevant for traditional ml). Then we bind the LSTM model with gin flag. After this,
34

---

Published as a conference paper at ICLR 2024
CODE LISTING 3: Example configuration for lyperparameterrs of LSTM. Tunable hyperparameter ranges be floating points, integers, and categorical values.
import gin.torch. exttennd ernal coon gurabl
import iou bendlin ka modela. atappeс imgort benchmer r mode encoders
defaull preppooc ooesaae www.eaaaeeeenoo. Cea.uema
Palse
Train parans
amdell apper
biWk apper sarper.ennonn eenodder E1:STINe () Dikragper cetperr..ptitteee.
10
6Adam
11
ntäragper.t srrper.trann.. ttpahn.epos apper--eerrr bww.n,
Dikrapper, www.pperrrrran....ommm pra sence = 10
14
ntdrrapper traiinn dellta 1e-4
15
optinizez: parame
opt mi ner/ weweeepperreeeeeeeeeeeeee ropprpprrnnn www.rrpeereeeeeem raleter.ceeee tune gAdam opt Letimizer wwtpee//hppppppprreeeeeee
18
decay 16-6 1le-b, 36-41
Encoder gorans
22
model hhtperpppeeeeeeeeeereeeee classeeer..sss...............sse
8L.TTMiet
23
/typerpaaeeeeeeeeeeeeeeeeeeee remeter.
24
model model/hype:per /type:paremteer ceammtterhht sdon Gin (32, 256, "lop-aritorn', 21
25
mode / /typyhypppppp remeber, leyer leyer_dis 11, 3)
26
Hyperperanter tuning
tune iyperperameters s. scogee
*Moce "optinizer")
29
zune www.h//pppraccttts.ooo wttpoorrnncctssn..iae wtw.ss/.nnintaapipppeeee points -
30
hyperperaneters
CODE LISTING 4: Running YA/B. We train the LSTM model on the MIMIC-III demo dataset for the Mortality24 task.
banchnarks train
-4 dano sano_datz/non
nini oudeo
Sinazyliaaiiieaciie
-ta Mortality24
-lug-dir ..//giiilogaa/
-n 187H
-go
-4 2222
the hyperparameters specified. The optimizer and encoder parameters then specified. Note that we can specify ranges of hyperparameters to be tuned by the hyperparameter optimizer. Settings for this can be found in the bottom cluster of code. Code Listing 4 shows how train our LSTM model on the mimmic_demt dataset (included in our repository).
E.5 OPTIONS
TABLE 18: Options for the tr praiiw command.
Flag
Required
Description
-reproducible
No
Make torch reproducible. (default: Truc)
-hp, -hyperparans
No
Hyperparameters for model.
-tune
No
Find best hyperparameters. (default: False)
-checkpoint
No
Use previous checkpoint.
We specify the command line options that YAIB provides for benchmarking prediction tasks. One use the ccu-benchmate rks command with either ta i or eva luate. TTabl shows the
35

---

Published as a conference paper at ICLR 2024
TABLE 19: Options for the evaluate command.
Flag
Required
Description
-Source-name
Yes
Name of the source dataset.
-source-dir
Yes
Directory containing gin and model weights.
TABLE 20: General arguments for the use of YAIB.
Flag
Required
Description
-d, -data-dir
Yes
Path to the parquet data directory.
-D, -name
Yes
Name of the (target) dataset.
-t, -task
Yes
Name of the task gin.
-m, -model
No
Name the model gin Defaaul..
-tn, -task-name
No
Name the task, used naming experiments.
-,, -experiment
No
Name of the experiment gin.
-1, -log-dir
No
T og directory with model weights.
-seed
No
Random seed for processing. tuning, and training.
-V, -verbose
No
Whether
verbose logging. (default: True)
-cpu
No
Set
CPU. (default: False)
db, -debug
No
Sct to load less data, (default: False)
-1c, -load_ caccee
No
Set to load generated data cache. (detault: False)
-ge,
No
Set to generate data cache. (delault: False)
-generate_cache
-P, -preprocessor
No
Load custom preprocessor from file.
-p1, -plot
No
Generate common plots. (default: False)
wwd, -wend
No
Activates Weights and Biases hyper parameter sweep.
(default: False)
-im,, pretraained-mmptttioo
No
Path to pre trained imputation model.
arguments that can be used with train. Table 19 contains the options to use with eva evaluate (i.e., for evaluation with a pre-trained model). In Table 20 the general options for YAIB are shown.
36

---

Published as a conference paper at ICLR 2024
F APPENDIX: EXTENDING YAIB
YAIB is built adapt your needs with as little effort as possible. YAIB allows you to change any part of your pipeline with ease: add new dataset, define additional clinical concepts, adapt a task, implement imputation algorithms, use additional models, evaluate custom metrics. This section provides examples on YAIB may be extended with respect to each of the above.
CODE LISTING 5: ID and table configuration for the Salzburg Intensive Care Database Database in JSON.
*rane"; "sic"r
"id ofig::
"patiient isnt":
"id"; "petientid", "position";
"etact"; "Elestadiissiion,
*end": "ottoetordestn",
*table"; "cssex"
"Icustay";
11
rid": "csseld",
12
"position"; 2,
13
"stact"; "attsetateerrisssteeeeeeio
fend": "cineofatay", *table"; "casoo"
16
"tablos":
19
"caees";
20
*eslas"; "azses.cav.pe",
21
"setaulto";
"index vert: roffeesiterriirettmmliiiie
23
"tinc_voro": Perfestattertisraaallooortt "ofteesatdaata")
24
"cocoid":
27
*nome": "CaceID",
28
"spea": "coi_integes"
29
"satientia":
31
"nama": "FationtID",
32
"spe=": "col_inteper"
1,
adniesionyear":
35
*nome": "adniccionYear',
35
speat: integer"
39
60
43
64
07
F.1 ADD NEW DATASET: SALZBURG INTENSIVE CARE DATABASE (SICDB)
SICdb is recently published open source ICU dataset (Rodemund et al. 2023 The dataset includes 27,000 admissions to the ICU at University Hospital Salzburg (Austria) from 2013 to 2021. SICdb provides highly granular data with up to minute level resolution. To make SICdb available within YAIB, must be interfaced to r cu. This ensures that r cu knows how to access and extract data from the dataset. Interfacing new dataset requires three key steps:
1. Define the table structure and possible ID types as a JSON configuration.
2. Define how all ID systems and their origin times relate to each other.
Define ID types and table structure First, ricu needs to know what tables and columns exist within the data source. This is specified via the JSON configuration file partially shown in Code Listing 5) Tables are defined under the tables element. The definition of each table contains the name its source file, usually provided in CSv or csv. gz format, and a list of all columns and their data types. In addition, default roles can be defined for certain columns in the table. These
37

---

Published as a conference paper at ICLR 2024
usually include the time index (if present), all other time columns, and column that is considered to contain the value of interest.
In addition to the available tables, ricu also expects information about the main ID types used in the dataset. Each piece of information in ICU datasets is usually linked to a certain unit of observation, most commonly the patient (patient) the hospital admission (hospadm), stay (icustay). By knowing what IDs a piece of information measured for, ricu is able to temporally relate all information within the dataset. For example, labevent s in MIMIC IV are recorded for hospital admissions, whereas chartevente these two ID systems relate each other allows them to be mapped a common time scale (e.g.. time since ICU admission). At the same time, knowing that an ICU stay is detailed in icustays in MIMIC and in cases in SICdb allows to define the same semantic reference point in both databases.
the specific ICU
are recorded for ICU stays. Defining how
The two ID types available in SICdb are the pat ient and the icstaay. There is no separate demarcation of the hospadm. The observation time for a patient ranges from their first observed admission to their death (if it occurred). icustays range from the current admission to the ICU until the end of the stay.
Calculate origin times for each ID type After ricu has been told which ID systems exist in the dataset, it also needs to know when they start and end. Much of this process is automated. For SICdb, all origin times are already provided in format suitable for use with ricu, and thus the default behavior is appropriate for them. However, minor adjustments are often necessary. For example, discharge times in SICdb are not provided as absolute times but in seconds since the start of admission. Such adjustments can be made on case-by-case basis by subtyping the respective functions (in this case id_win wwwnnhehe h_e per) and overwriting the default behavior (Code Listing 6). Since SICdb provides time in seconds since admission, ricu must further be told to work with relative times in seconds. This can be conveniently achieved through existing helper functions (Code Listing 6)
Following the steps above makes SICdb available and fully usable within ricu. While some further helper functions may be necessary to enable optional funetionality such as automatic determination of measurement units (SICdb stores units in a separate reference table that needs to be merged at runtime), these are not essential to the main functionality of r Note that the above only interfaces SICdb. It does not automatically map all existing clinical concepts for SICdb. Defining a clinical concept for SICdb still requires manual mapping the concept to SICdb data items, for example via the appropriate measurement IDs (see also the next session on adding a clinical concept). We do not expect that this process can ever be fully avoided (unless it was already performed prior, for example by mapping to and providing the data in OMOP format). However, we found that the framework and helper functions that ri cu provides greatly simplify this process. Additional information on ricu and its design principles can be found in Bennett et al. (2023).
F.2 DEFINE CLINICAL CONCEPT: POTASSIUM CHLORIDE
YAIB comes with wide range of pre-defined clinical concepts. However, it is likely that new tasks and applications require additional variables. An area of particular interest in this respect are medications. Using the exemplar of Potassium Chloride (a fluid frequently administered in the ICU), we demonstrate how new drugs can be added to YAIB with minimal effort. For many concepts, all that is required is a JSON dict that describes the correct measurement IDs within each dataset (Code Listing 7. The JSON snippet can then be appended to the existing r ricu concept file (for additions that should become part of the main package) or added to the search path of load__iii diiccici ionnrry via the cfg_ s__dirrrs parameter. The definition of complicated transformations such as calculation of hourly rates is also supported by existing helper functions. If custom calculations necessary, they can be provided as user-defined functions via the ca 1. lbaakk element.
F.3 ADAPT A TASK: KDIGO STAGE
Cohorts created in YAIB are built to be easily adaptable. For example, predicting ordinal KDIGO stage rather than binary presence/absence of AKI is as simple changing the outcome variable from aki to kdigo. More substantial changes to the task are also supported. If the general setup (i.e., covariates, exclusion criteria, etc.) remains the same but novel outcome should be predicted,
38

---

Published as a conference paper at ICLR 2024
CODE LISTING 6: Helper functions that calculate the origin and temporal relation between events in the Salzburg Intensive Care Database.
vi pei ale any <- functionts) (s)
return nepping between two eystens TDU stay and petient ID
incloding
cig x sost laa Wi (a), deczmasing TRDE)
ids < Sield(ofg, (idr)
Silld(ofg, "stazt")
end Sieldoffg, "endf)
10
tb1 sxc_rttbbi, miqueitielddiet. "table"!))
11
nix satdi (eInanes[ttbl)
14
<- load aro(thl, cols olies, intersect lste, colnanes (tb)),, andi)
15
assect that llenghh 110
17
resi, Firatedm asiion :- 01]
18
19
lapply (.65, minsl, -Dools Glste, endi] res tetsolorderrres, glide, endi)
res rosl, elsta, resf, tinesfatay :- offaccatterfireteee sstinn timenfatayl convert to absalute di scherge time
21
22
23
(res, efide, pastab "Latars"), paste01155, "_ond"1>, by_ret TRUE)
24
25
26
as 1dc[25), by_ref (RUE)
29
Load_ditrttne..o sic <- function zows, cols colnames(N))
30
hint
31
....
t Load time differences in sIcdb by creating each time variable as t the relative tine since adnistion in seconds saad_as_relatvv___eee
33
34
2 id_hint,
a_as_nins)
35
the outcome should be created clinical concept in ricu which then simply be used as the outcome variable. If covariates, exclusion criteria, or prediction times change, existing helper functions can be utilized. See the existing task definitions for further examples.
F.4 ADDING NEW PREDICTION MODEL: RANDOM FOREST, RNN, AND TEMPORAL FUSION TRANSFORMER
We allow prediction models to be easily added and integrated into a Pytorch-lighting (PL) (Falcon team 2023) module. This incorporates advanced logging and debugging capabilities, as well as built-in parallelism. Our interface derives from the PI. Ba seModu
For standard Seikit-Learn type ML models (e.g., Light Gradient Boosting Machine (LGBM) (Ke et al. (017)), one can implement the MIWrapper, incorporating the model steps in the process. Note that this class is also derived from the PLBaseMocis
this leads to minimal code overhead. See Code Listing 8 for details for the implementation of random forest model. The only needed code here is the hyperparameter configuration and the initialization of the superclass.
The definition of DL models can be done by creating subclass from the DLP A.PPredict iwwwappper:rrppprr: this inherits the standard methods needed for training DL learning models. Again, our implementation using PL. significantly reduces the code overhead and complexity. See Code Listing for the example of a simple RNN model. We can then create a gin configuration file for this model such as that in Code Listing 11 specify default parameters and hyperparameter ranges for hp-tuning.
More advanced, or state-of-the-art, models are also easily implemented. One of YAIB's users has implemented Temporal Fusion Transformer architecture (Lim al., 202Tp. This model provides good performance on multi-horizon time series forecasting, as well as interpretable insights. Specifically, [im al. (202I) describe the TFT follows: "(1) examining the importance of each input variable in prediction, (2) visualizing persistent temporal patterns, and (3) identifying any regimes or events that lead to significant changes in temporal dynamics". See Code Listing for
"httpss://igghtninngggn http:h/ss//iiiggggg////////////t wtwp://mwwoonnnnninniniiion witp:/gggitiiiiiii
39

---

Published as a conference paper at ICLR 2024
CODE LISTING 7: JSON definition for rate of porassium chloride across included datasets.
"desonigt
poneenitiions ton't: "potessiun chil oridd donation",
*catlegory": "medicscionn"
"aggregate"; "nax",
*eaundes";
*aunc": 1
*ida"ts 9001,
"table": "drugitens",
10
"sub_var"; "itemia",
11
*stop vart; "stop", "arp_rar"; "crderid", "ballback"; "aurc_doc""
12
14
15
16
rescut
18
>egex"; ""pstassiun chloride",
19
ttbbee:; "infoeiondrug",
sub_rart; "drugname",
sollback"; "eicu (ecouddtatioorppp_lemee Soure (55) 1"
22
*lass" "rgx_ itm"
23
24
"hrrdd: [
25
27
"ide": 1003396,
28
"table": "ptarna",
"su"_var": "pharmsid",
30
varp; "intusionid", "ailbbck": "biris_duratton"
31
34
Siiv"
35
"dda"s 1225166, 2275221,
ttable"; "inguteveats", "sub_rar"; "izomis",
38
39
vato" "andsime", "arp_var" "linkordorid",
sollbback"; "nimic dur_
62
43
CODE LISTING 8: Example ML model definition. We create Random Forest classifier that implements the default YAIB prediction model interface
Pein, con' Liqurable
class RFCI wse if Fier (N (raager))
aad mon_eattee [RunMode, classi Leation)
i (mmlé, sarge, <=kwarge):
self mnoddl self, mod. Laag (enswallle, Bar.ile...addoooo Claa sarga, / vskwagys) supee1) init Isargs, ekkuagg))
implementation details. This implementation is based upon the NVIDIA PyTorch implementatior 14 To train and optimize this model from a choice of hyperparameters, we need to specify a GIN file to bind the parameters, see Code Listing 11 Note that we can use modifiers for the optimizer (e.g, Adam optimizer) and ranges that we can specify in rounded brackets "0"" Square brackets, "["". result in random choice where the variable is uniformly sampled.
F.5 ADDING NEW IMPUTATION METHOD: CSDI
We have added a range imputation methods to YAIB, including interfaces to existing imputation li- braries (Du 2023 Jarrett et al. (2022). Here, we describe the addition of recently introduced method that uses conditional bcore-based diffusion models conditioned on observed data: the Conditional basee Diffusion models for Imputation CSDD)Tashine al. 2021). To make the process of
implementing these models easier, we have created the Imputation class that extends
httppp//hhhhanlllleeeeeseeeee/er porecasting/iie
40

---

Published as a conference paper at ICLR 2024
CODE LISTING 9: Example DL model definition. We use the inbuilt Torch RNN layers to built Recurrent Neural Network.
Bein, conf gurabbl
class Rottee wtt.eedde
Torchrch standard min model**n **o
- suppprt Lapporrtddaaarnnee
[RunKoder
del Init init.. (sele, ingut_xxie,, hidden,din,, lagerudin, nun_cleases, sstvargs): supec1) اnst
Input size-lnput_ aize, hrdden din-hi dim, layer dinelayer din, non claesee-nun classes, eegge, kwange
dim
self, _ayer_diin layer_dim
self, inn inn RNUR cinput. sizel2? hidden_dim, layer_dim, risst_iisst self, logit on, near oden din, classes]
13
14
def init hidsen (self, X1:
16
hB nen.w zeros (self. Jager,din x.size(31, aif.hhideen_dim
17
18
Forwardjself, xi:
h0 Sollfinit_hi init__isdenmn) sele.ronte pred logit lout)
21
22
return pred
24
the pre-existing DLärapper (itself a subclass of the LightningModule of Pytorch-lightning) with extra functionality.
The CSDI model a diffusion model that follows the general architecture of conditional diffusion models(Ho et al.) 2020); It introduces noise into subset of time series data used conditional observations to later denoise the data and predict accurate values for the imputation targets. CSDI is based on U-Net auchitecture(Romneergeer al. 2015) including residual connections.
Tashiro et al. (20TT included two additional features into their model, which are inspired by Dif- (Wave (Kong et al. 2021): attention mechanism and the ability to input side information. The attention mechanism uses transformer layers, as shown in Figure 10 An input with features, length, and channels is reshaped first apply temporal attention and later reshaped again apply feature attention. The second additional feature allows side information to be used as input to the model by a categorical feature embedding (Tashiro et al. 202I).
Temporal Transformer
Feature Transformer layer
L.
(1.4.C) XXf
(K,
(K.4,C
Concat /split
xt
Input
Split
Concat
Output
FIGURE 10: The attention mechanism of CSDI adapted from Tashiro et al. 2021
See Code Listing 12 12 for the most important implementation code: the model initialization. We note that of this code, very little has been adapted from the original code repository 15 included in the original publication (Tashiro al. 2021
F.6 ADDING EVALUATION METRIC: JENSEN SHANNON DIVERGENCE AND BINARY FAIRNESS
support adding multiple types of evaluation metrics for benchmarking DL or ML models. We additionally support three common metric libraries: TorchMetrics (Nicki Skafte Detlefsen et al. 2022). Ignite (Fomin et al., 2020), and Scikit-Lear (Pedregosa et al. al, (2011). Adding a metric is a straightforward procedure. We added the Jensen Shannon Divergence (JSD) with the help of the SeiPy library (Virtanen et al. 2020). See Code Listing 13 for details.
One then add the metric to be evaluated for particular, see Code Listing 15
41

---

Published as a conference paper at ICLR 2024
CODE LISTING 10: Temporal Fusion Transformer model definition. Note that this implementation is similar existing code and can use existing methods and the original dataloader of YAIB which avoids code duplication.
class 797 PriP tomtrapper)
Tng iementationn Tergoral Posson Transforner,
htspss/ http:///aaaa//1.999.9999
support re roodes [RunNodee alase Fication, ion, Barnodd..cepp regreasloon insi_
self,
nun_plasses, + Classes for nuls muls ielasa class fiostion
10
encoder__eagta, Determines interval
prediction
11
hieden layers
drapout, Dropout layers
n_heade, Attencion heade
14
dropout
15
exanpie_iength,
predict
muantilec- [0,1, 0.5, 0.9), I quanclles to prodace
17
statie categerical inp.sizeeff 9:7e-12], Nomber of categories
18
cemparal_aaom_eer jaon__cteeprriol..oooo
19
cempora Lmporral_ccervee Lacerrved wat.eerieelll..ngg........ itp__niee-(_8]
of categorionl observed variables
statin cont munus irp. sise3, Nomber of ic cone Inooue varlable
21
semparal_koom__e cottooooontmmms_nnnnoooo itp_s__pp_____e_d_
22
csserved continoous imp_size=48,
23
tempora) sarget_aizees **gaggg,
target variables
24
25
11
26
Iserived varisbles nun static vare lenistatic categorical inp sizel static centinusus inp size nun futuuuuuu___accee vars lenttamparal kampprral_koom_eee catoommcttrrrioollinge. hun_historic_vart cun ceruunnuturr_uueee temporai_ wtt.rrao_eareereeeeetettio categorical Lirp size),
29
samporo_itooc_cooniiooo wawe-inocus_inn_aaee inp_siio
30
terporoi_earge__e
(terporl
32
superi) (oteit,,_nn init Inun nww.easssssnnncddi classes, encoder, Lengthhennoooer Jenghh tideen,
33
'_headpea_hoade sxpest_attsdrorotttooo
canntlles-uantile.
nun_otatic_vvare sare-ann_statie ron_roturo_yare tageennot_tttrr_eeoeeo sur_historic_vare
num www_hietoric_vee vars, wtt.rettttccccorrttttttee inp inp___nn__s___________ size=1, terporai known trtorro__tttr__taeeprrrroo caterporrgegg sipesize ittparvee__oooprrraaiiieeee
"0, sampers_knom_tootinnne **kvargs!
temporsi_targes
35
36
self. Seff.ennoderrraae lonohh encoder Doternines from hav past we
self enbodding LazyEnoeddinn httpaaattttttttttooooooo.oooaee
htwperaa_ioonnstttttoooottttt aasii__oottinoous__nootttt temporsi_ irt_a___nnnnootottttoooee con.rnooontttuuee..oom iir__iir__ hiddes) caboddinos for all variablas
40
temporsi attrrsss__ssrreeetttsttrreee
aegpors_oooerrrss_cattiio inp_size, temporal_eargs__de
61
salf.static_eneee
ttateeoo wrm_static_vvre aidden, dropout) encoding static
variables
eFir. IFIBack (cacoder, jeneddrrr_eethhh, nun_ilstoric_vare hidsen, dropoct, run_feture_veree a_herds, dcopout_ati,, exanple Jangth, quant iles)
63
self.iegit Lnnar (qusn nun_classas) Linear layer output the number of classes and allow nedizication by prodiecionvrapeer.
In order to asses fairness within ML prediction, a common metric check is group fairness. This computed through the ratio between positivity rates and true positives rates for different groups. Two types of these metrics are demographic parity (Calders et al. 22009) and equal opportunity ratio (Hardi let al. 2016). The TorchMetries (Nicki Skafte Detlefsen et al. 2022 library includes the group fairness module interface which we can adapt for use in YAIB. We use a wrapper, that extends the TorchMetries implementation and extracts "group tensor" that indicates to which group the sample belongs. See See (oode Listing 14 to achieve the desired result. After this, we pass the data and feature names in the training step, as the function requires information about the assigned groups of the dataset; here, we track demographic parity by default, i.c., we use "sex" as the group name. In our pipeline, this has been one-hot encoded to 0 1. Then, adding this to the constants file, as seen in Code Listing 15] automatically calculates this metric during the training process.
42

---

Published as a conference paper at ICLR 2024
CODE LISTING 11: Temporal Fusion Transformer parameter configuration.
Hygerperaneters For model.
Conmon settings for bi models
inelude "conf coonfigg/reedion hoon_nodelll oummen/ modell////ummmen//oooeeennn
Opt iniiinieer params.
stwin_ me__ooooonnnnee
87FT
opt hnssse//ggerppraneeeeeee Glass.
opt aptimizcr//hgn ht..iiece//pperrrranteeee wwwggh___sssssseeseseses 16-6 opt opg.mmieer//ngerrrrnnneee.ss (1e-b, 36-41
10
11
12
Enccode porans
model/hug htpp::ppeeeeeeeeeeeeeoe..
tune 8IFT
14
model /hyperge someter.enooeee lergch 24
15
model htype:pacemeter
256
16
hnyperpeemmeeer
model/bypee sespeepaacmttsr.droaa.
10.0, 0.4)
18
model http///pppaaammeeeerr wttprtttrr..ooo........
(0.0, 0.
19
model http://hhpppeeeeeeeeee wwtpperrrrameer.... heads
nodel//ypccsa:cmette... htt.m//mmc....coem...o..........
CODE LISTING 12: Implementing the CSDI architecture in YAIB. Note that implementation is very similar the original github repository, which demonstrates the flcxibility of implementing new models YAIB.
def init self, size, time enbedding size feature enbedding size, uncondit oonal, target strtegyy nun sun_distesicnetere dictusion_sess sot__sss__saddinngg.. auattention_ nem cum_rociddaa__ppee earge, **kkorgs,
ceise_schedullo beta_stort, seta_end, n_samples,
11
init_l....)
self, salf.target_onn "esput_sizelp self self.n_stmpies f_samples
self, time din time_ ewweoding size self.enb_fedde cnt_feoture__iin foatore_emb:ddinn_ce self,is_ wtw.nconddiionns.t..........oo unzonditionsi selfitarge strategy -tarett strategy
11
12
self. ammo_totalddis self..nb_timme.dt
Sdff.cm__taatur_.de
not self. is uncond:tional.
self. htt.eontttttttaeeeeeoeeeee + 1 for conditioral mack
15
self. iayes_ on.Enbesding wew_eeereeddegeeeeee cartget bibedsing_dii--eee umb_reature_ddet
input 1 self.1s, unn.ciiinnncaiticcnll else self. delfodi:tmodel = sitt_CsDI(
19
20
21
nun_diifusion_tteg
cittucion_stepp step_enbeeddrggccm.
23
self.emb_ est crt__ottt__dd,,
24
fusttontion_eades
25
nun_reoiddai_tayye. input_din,
27
28
29
parenaters diffusion medels
self num_stops cun_dittusico_toog
31
if nolaa_achadule feuad"; self beta scidele "linwar":
32
0, bess_ond=-0.5, selt.nun_stops)
self.beta = ap. ap. (bsta_scart, bata_end, self.aum_steps)
35
35
aisaaaa self.bela ap.curprod selfsshsha_terch torch.tensor (sphf.alphe) .flatt (),, (usquecre cnsqueeze
40
G APPENDIX: EXPERIMENTAL SETUP AND REPRODUCIBILITY
To reproduce the results obtained in this paper, we have detailed our methodology in this Appendix.
G.1 INFRASTRUCTURE AND HARDWARE
We used high- segehperformance computing cluster to perform our experimentsNo data was transferred to any external parties in this process. This computing cluster is with renewable energy and can be
43

---

Published as a conference paper at ICLR 2024
CODE LISTING 13: Implementing JSD using SciPy in YA/B. In this case used the Ignite interface, but users also choose extend from the TerchMetric SK-Learn interface.
den (PpochMet
def iini (selE, Ltannfootn Gallable lambda at i, check computa faz bool Falsw) >> Noow: super IJSD, sele) inst (lambda JSD_ moheck_.ommuute_.r
outgot transtorneeotgut transforn, check congute
deł J50 En preda: torch, Tensor, targets: torch.Tensori, retuen jensensharmon ty-eddd)) - Flatbent) i abx(y.targeta)) flatten
CODE LISTING 14: Adding a wrapper for Group fairness metric.
class BinecyFaia mww. (SinecyFal
This class la a urapper: For the nerfyffa rnees necric Fron Tordhieet
gcme Xone
init grop group__tam
group_rame "xex", sarge, rskwarge)) Yones
superi) init Isargs, *skaarge) dež update (sele, preds,
date, feature_neeee Xones
vuex Standard meliie updale FunsLion
groups datals, ty festure nanes. fndex indes. (salf. group. name] proup_per. id = groupe (Fy C]
12
return supez uptlaa wwwp:ppppppdd..coooooooo
13
sargetatarges.quu
grospeegrmu. per, Long.. eepc
considered neutra. The cluster is running the SLURM (Yoo et al. 2003) management tool on Ubuntu 20.04.6 6 LTS. with a number of Nvidia A100, A40, RTX 208011. and RTX Titan GPUs. The traditional machine learning algorithms were trained with Intel Xeon Platinum 8 8160 and AMD EPYC 7643 CPU resources.
TABLE 21: Average estimated duration of truining tasks
Once per stay classification
Hourly classification
Hourly regression
Machine Learning Models Deep Learning Models
minutes
10 minutes
10 minutes
30 minutes
60 minutes
90 minutes
G.2 LIBRARIES
full list of libraries is available in the YAIB repository; please use the Conda environment manager to install these. We have aimed to use the most recent library versions (while main- taining compatibility) to improve efficiency and reduce errors. At time of writing, the most recent version of Python that supports the used libraries was used: Python 3.10. The most important libraries are: Gin-config 0.5.0, Pytorch 2.0 0 (in combination with Cuda 11.8.0), Pytorch-lightnin 2. C, Scikit-learn 1.2.2 2, Lightgbm 3.3.5, Pandas 2.0.0.
G.3 COMPLEXITY
As mentioned in Yēche et al. (2022), the transformer memory complexity concerning the sequence length is quadratic. With our hardware, training deep learning models overall takes less than hours. Training ml models takes less than hour. See TTbble 21 for average training durations. Note that there was large variability between the cohorts due to the size difference datasets (i.e., eICU Collaborative Research Database contains almost eight times the amount of stays as AmsterdamUUMCa)) The algorithmic complexity is specified in the implementation details of the Scikit-learn (Pedregosa al. (001T) (logistic regression (LR) and elastic net (EN)), LightGBM (Ke) et al. 2017) (Light Gradient Boosting Machine (LGBM)), and PyTorch library (Falcon & team
2023 (Long Short-Term Memory (LSTTM), Recurrent Neural Network (RNN), Gated Recurrent Unit
44

---

Published as a conference paper at ICLR 2024
CODE LISTING 15: Merrics recorded for binary classification. Adding the metries to this dictionary results in automatic logging. Sce the whole file for details.
DiMetriie ristt
BINARY CIASSIPICATION
"AU0": AUROC "ataaaann""n"""""""
AveragePrecisii (sast="binerr"""""
*11" F18care "bunary", murcclasaes=))
*Cal iibratiion Srronts Cellbrat Gel:brationfrron iontrror (task="biaarr"" binns=110
*Cal iibratooo Cor_Cerve" Calebrat ionCurve,
"B8_Cuxvw"s PrwcisionBeeall ICaeeee,,
Corve"; Rootorvey
3spp" J55,
groupe-2, "Bnaryy Fairnees"; BinaryFairne weeryyyaseeesseeeeee wecpeee propppe, taske dencgraptio. group_names sex"!
Repelition
Repelition
Repettion
Hyperperamelie Tuning
Evalusaion
Fod
Fold
Text
Complets
Model Training
Fold
Vel dolitation
Fold
Train
Fold
Spit 2
Spitt 9
Spitt 5
Spii
Spir
FIGURE 11: Schematic overview of times repeated S-fold cross-validation. This example only the first three splits of the first repctition for hyperparameter tuning. The repetitions and amount of folds and the folds tune on can be casily adjusted (see Code Listing
(GRU).. Furthermore, Bai et al. (2018) (Temporal Convolutional Network (TCN)) and (Yéche et al. 2022) (transformer) implementations also detailed in their respective works. |Figure 11 shows our repeated cross-validation training method. Note that in order to obtain our final results, we do repetitions of 5 fold cross-validation with excluded test set.
G.4 REPRODUCIBLITY
For detailed description to reproduce our experiments, we refer to the Paper reproducibility file (included the repository). We have followed the standards specified by official MI reproducibility guidelines by Papers with Code
45

---

Published as a conference paper at ICLR 2024
G.5 EXTERNAL VALIDATION
All external validation validation models have been trained with an 80/20 train/val split to use as much of the dataset possible. The test splits are the same used for the same dataset experiments (diagonal in Figure 2) For the external validation pooled results (d-1), we used subsets of 10,000 stays for each dataset simulate a setting where the datasets have similar sample size. We wanted to ensure the size of the datasets, which differs significantly between datasets, had no undue influence on the training.
G.6 PINE-TUNING
For fine-tuning experiment, shown in Figure 3 learning rate 0.00001 and an exponential learning rate scheduler to reduce learning rates gradually. The rest of the hyperparameters are exactly in the original source models.
we used an ADAM optimizer with starting
46

---

Published as a conference paper at ICLR 2024
H HYPERPARAMETERS
Here, we detail the tuning setup and hyperparameters used in our experiments.
H.1 TUNING APPROACH
In Table 22 and 23 specify the hyperparameters used for hyperparamette--uniin for the bascline experiments for deep learning and machine learning models, respectively. We incorporated different sampling methods for hyperparameter selection. Hyperparameters were chosen he mostly identical to Yêche et al. (2022), to improve comparability and for reproducibility reasons. However, we have chosen allow for continuous ranges hyprraarameters some cases, improve the performance and functionality of YAIB. Log-uniform means that the parameters are sampled according to the
TABLE Model hyperparameters, default values for all models (above), and the distributions for the models (below), considered during Bayesian hyperparameter optimization.
Category
Parameter
Value
Initial points
5
Hyperparameter search
Calls
30
Folds to tune
2
Epochs
1000
Min delta
1E-4
General Parameters
Patience
10
Batch size
64
Weight
Balanced
Regression task
Mean Squared Error Cross-entropy
Loss
Classification task
Optimizer
Adam
Optimizer Parameters
Weight decay
Ic-6
Learning rate
Uniform([1E::) 3E-4])
Model
Parameter
Value
Hidden dimension
Log-uniform(132, 128))
LSTM
Hidden dimension
RandomInt(1, 3)
Dropout probability
Uniform[[O.0, 0.4])
Hidden dimension
Log-uniform (32, 128))
GRU
Number of layers
RandomInt(1,5)
Hidden dimension
Log-uniform( (32, 128))
Number of layers
Log-uniform(( (332, 256))
TCN
Kernel size
Log-uniform(E ((2, 32])
Horizon
24
Hidden dimension
Log- Unifoonnfrmm3,, 128])
Number layers
RandomInt(1. (1, 10)
Number heads
Log-uniform(1. 8)
Depth
Uniform( (0moomm((1331]))) [1,3))
Transformer
Kernel size
Log-uniform( [2, 32))
Dropout probability
Uniform([0.0 0.4])
Dropout attention
Uniform(10, 0.4])
L1 regularization
0.0
Hidden multiplication
2
reciprocal distribution:
[xx a, b) x [log, (b) - (a)
for S and a >
(1)
Uniform means that the parameters are sampled according to the uniform distribution:
for b,
f(x) for < aror>b
(2)
47

---

Published as a conference paper at ICLR 2024
TABLE 23: Model hyperparameters. defaul values for all models (above), and distributions for the ML models (below), considered during Bayesian hyperparameter optimization.
Category
Parameter
Value
Initial points
10
Hyperparameter search
Calls
50
Folds to tune on
3
Patience
10
General Parameters
Jobs
Regression task
Logloss Cross-entropy
Loss
Classification task
Model
Parameter
Value
C
[og-uniform(1133 IEI])
Penalty
Choice(11, 12, elasticnet)
LR
L1 Ratio
Uniform([0.0, 1.0]) saga
Solver
Max iterations
100000
Alpha
Log-uniform( 1E1))
Tol
Log-uniform(1I: (1E-5, IE-1])
Hidden dimension
RandomInt(1,3)
EN
Dropout prohability
Uniform([0.0) ([0.0, 0.4[)
LI Ratio
Uniform ([0.0, 1.0J)
Solver
saga
Max iterations
10000
Column sample
Uniform([0.333 1.0])
Sub sample
Uniform([0.333 1.0])
Leaves
Log-uniform((8, 128))
LGBM
Max depth
RandomInt(3, 7)
Hidden dimension
RandomInt(1. 3)
Estimators
10000
Min child samples
1000
Subsample frequency
H.2
DEEP LEARNING MODELS
We detail the hyperparameters that have been chosen using our Bayesian hyperparameter optimization approach.
48

---

Published as a conference paper at ICLR 2024
Gated Recurrent Unit (GRU) The range of hyperparameters considered for the GRU model are found in Table 24
TABLE 24: Chosen hyperparameters for Gated Recurrent Unit (GRU).
Dataset
Learning Rate
Layer Dimension
Hidden Dimension
Mortality
AUMC
3.00F-04
3
48
HiRID
2.37F-044
2
52
eICU
3.00E-04
I
135
MIMIC-IV
1.43B-04
2
77
AKI
AUMC
2.81E-04
3
256
HiRID
2.06E-04
3
115
eCCU
1.43E-04
3
240
MIMIC-IV
2.82E-04
3
139
Sepsis
AUMC
2.28E-04
HiRID
3.00E-04
eCCU
8.51E-05
2
77
MIMIC-IV
2.39E-04
3
52
KF
AUMC
3.00F-04
3
HiRID
1.11E-04
3
196
eCCU
6.35E-05
196
MIMIC-IV
1.11E-04
148
LoS
AUMC
5.45B-05
3
158
HiRID
1.00E-05
1
117
ICU
1.03E-05
3
254
MIMIC-IV
1.57B-05
2
237
49

---

Published as a conference paper at ICLR 2024
Long Short-Term Memory (LSTM) The range of hyperparameters considered for the LSTM model are found Table 25
TABLE 25: Chosen hyperparameterss for Long Short- Term Memory (LSTM).
Dataset
Learning Rate
Layer Dimension
Hidden Dimension
Mortality
AUMC
1.87E-04
145
HiRID
1.54F-004
2
256
eICU
3.00E-04
3
149
MIMIC-IV
3.00B-04
2
185
AKI
AUMC
2.62E-04
3
57
HiRID
3.00E-04
3
54
eCCU
3.00E-04 04
3
70
MIMIC-IV
3.00E-04
3
256
Sepsis
AUMC
2.100-04
HiRID
2.48E-04
139
eCCU
1.12E-04
2
40
MIMIC-IV
2.46E-04
161
KF
AUMC
2.79F-04
2
81
HiRID
3.00E-04
256
eCCU
1.75E-04
3
MIMIC-IV
2.49E-04
256
LoS
AUMC
3.24E-05
62
HiRID
6.65E-05
3
255
ICU
2.86E-05
3
215
MIMIC-IV
1.80F-05
3
253
50

---

Published as a conference paper at ICLR 2024
Temporal Convolutional Network (TCN) model are found Table 26
The range of hyperparameters considered for the TCN
TABLE 26: Chosen hyperparameters for Temporal Convolutional Nenwork (TCN).
Dataset
Learning Rate
Dropout
Kernel
Number of Channels
Mortality
AUMC
5.84F-05
2.71F-01 1.12F-01
6
92
HiRID
2.14F:-04
6
80
eICU
2.35E-05
1.15E-02
23
100
MIMIC-IV
1.81-005
3,50E-01
3
130
AKI
AUMC
3.00E-04
4.00E-01
3
144
HiRID
2.56E-04
2.56E-01
12
168
eCCU
3,00E-04
1,23E-01
3
81
MIMIC-IV
2.98E-04
7.61E-02
3
249
Sepsis
AUMC
3.00E-04
0.00E+00
HiRID
3.00E-04
0.00E+00
256
eCCU
1.93E-04
3.98E-01
2
61
MIMIC-IV
2.23E-04
1.06E-01
78
KF
AUMC
2.56E-04
2.78F-01
169
HiRID
1.75E-04
2.33E-01
34
eCCU
2.15E-04
1.92E-01
MIMIC-IV
1.88-00
2.15E-01
LoS
AUMC
3.00E-04
1.92E-01
2
32
HiRID
2.74E-04
1.71E-01
29
43
ICU
3,00E-04
2.91E-01
10
32
MIMIC-IV
1.57E-04
1.67E-01
12
44
51

---

Published as a conference paper at ICLR 2024
Transformer Table 27
The range of hyperparameters considered for the transformer model are found in
TABLE 27: Chosen hyperparameters for transformer.
Dataset
Learning Rate
Dropout
Heads
Hidden Dimension
Depth
Mortality
AUMC
1.29E-04
1.32E-01
2
95
2
HiRID
1.58E-04
0.00E+00
247
I
eICU
1.00E-05
4.00E-01
3
256
MIMIC-IV
6.18E-05
1.65B-01
3
AKI
AUMC
1.18E-04
9.75E-03
8
52
3
HiRID
3.00E-04
1.50E-01
154
2
eICU
1.28E-04
1.33E-01
2
96
2
MIMIC-IV
1.22E-04
4.13B-02
2
72
3
Sepsis
AUMC
2.61E-04
4.39E-02
32
3
HiRID
3.00E-04
0,00E+00
32
1
eICU
2.76E-05
1.05E-02
2
211
2
MIMIC-IV
3,53E-05
3.67E-01
98
3
KF
AUMC
1.95E-04
4.56F-02
HiRID
2.48E-04
9.34B-02
160
3
eCCU
2.62E-04
2.82E-02
MIMIC-IV
1,53E-04
8.62E-02
5
160
2
LoS
AUMC
1.13E-04
4.11B-02
3
76
2
HiRID
1.88E-05
1.88E-05
102
1
eICU
3.96E-05
5.67B-02
3
172
2
MIMIC-IV
1.13E-04
4.11B-02
2
52

---

Published as a conference paper at ICLR 2024
H.3 MACHINE LEARNING MODELS
Logistic regression (LR)
The range of hyperparameters considered for the LR model are found in
Table 28
TABLE 28: Chosen hyperparameters for logistic regression (LR).
Dataset
C
Penalty
I.1 Ratio
Mortality
AUMC
3.63E-02
elasticnet
1000++00
HiRID
3.45E-02
12
6,63E-02
eICU
2.78E-02
elasticnet elasticnet
1,00E+00
MIMIC-IV
2.05E-01
1,00E+00
AKT
AUMC
1.77B-02
1.00E+00
HiRID
1.00E+01
4.16E-01
eCCU
2.52E-02
5,99E-01
MIMIC-IV
1.28E-01
2.98E-01
Sepsis
AUMC
4.87B-02
11
2.21E-01
HiRID
2.59E-03
12
3.74E-01
eCCU
1.98F-03
elasticnet 11
6.20E-01
MIMIC-IV
2.20B-03
1.81E-02
Elastic net (EN)
The range of hyperparameters considered for the EN model are found in Table 29
TABLE 29: Chosen hyperparamnttess for elastic net (EN).
Dataset
Alpha
Tol
L.1 Ratio
AUMC
1.04E-02
3. 3.60E-02
1 1.22E-03
HiRID
1.04--02
3.60E-02
1 1.22E-03
eCCU
1.0SE-02
9.57F-04
8.09E-03
MIMIC-IV
1.00E-02
2.45E-03
0.00E+00
LoS
AUMC
1.00B-02
1.67E-05
7.66E-02
HiRID
1.00E-02
1.02E-05
2.40E-02
cICU
1.00E-02
4.82E-02
0.00E+00
MIMIC-IV
1.00E-02
7,42E-02
0.00E+00
53

---

Published as a conference paper at ICLR 2024
Light Gradient Boosting Machine (LGBM)
The range of hyperparameters considered for the
LGBM model are found in Table 30
TABLE 30: Chosen hyperparameters for Light Gradient Boosting Machine (LGBM).
Dataset
Depth
Column Sample
Leaves
Subsample
Mortality
AUMC
6
9.89E-01
117
9.90F-01
HiRID
5
1.00F+00
8
1.00F+00
eICU
7
5,41E-01
128
1,00E+00
MIMIC-IV
7
1.00E+00
28
1.00E+00
AKI
AUMC
7
1.00=+00
110
8.69E-01
HiRID
7
1.00E+00
79
8.69E-01
eCCU
7
9.97E-01
117
8.35E-01
MIMIC-IV
7
1.00E+00
128
1.00E+00
Sepsis
AUMC
1.00E+00
7.83E-01
HiRID
6. 6.18F-01
8.22F--01
eCCU
7
8.97E-01
52
6,04E-01
MIMIC-IV
7
1.00E+00
128
5.84E-01
KF
AUMC
7
7.01
15
9.99F-01
HiRID
3
9.58E-01
16
9.95E-01
eCCU
1.00E+00
47
1.00E+00
MIMIC-IV
7
9.30E-01
33
9.96E-01
LoS
AUMC
7
3.30E-01
83
1,00E+00
HiRID
6
3.30E-01
49
5.01E-01
eCU
7
3.30E-01
51
3.30E-01
MIMIC-IV
7
3.30E-01
128
3.30E-01
54

---

Published as a conference paper at ICLR 2024
1 MACHINE LEARNING REPRODUCIBILITY CHECKLIST
This checklist (version 2.0) was found to be the most recent version at the time of writing. It can be found www. CS mwgii ca/ For all models and algorithms presented, check if you include:
p.
A clear description the mathematical setting, algorithm, and/or model. [Yes] See Appendix Gor an overview of the used infrastructure, libraries, complexity, reproducibility. See Appendix H for the hyperparameter luning setup.
A clear explanation of any assumptions. [Yes] We make limited assumptions about the data use. Mainly, we assume stays are i...d. as usual in clinical machine learning.
An analysis of the complexity (time, space, sample size) of any algorithm. [Yes] See Appendix G
For any theoretical claim, check you include:
A clear statement of the claim. [N/A] We did not make theoretical claims.
A complete proof of the claim.[N/A]
For all datasets used, check if you include:
The relevant statistics, such as number of examples. [Yes] See Appendix C for details of the data.
and Appendix D
The details of train validation / test splits. [Yes] See subssecton 4./
An explanation of any data that were excluded, and all pre-processing step. [Yes] See
subsection 4.7 Appendix C and Appendix D
A link to a downloadable version of the dataset or simulation environment. [Yes] See the included yaib cohorts code in order generate the datasets (additionally, the YAIB repository contains preprocessed demo datasets)
For new data collected, a complete description of the data collection process, such as instructions annotators and methods for quality control. [N/A] We did not collect new data.
For all shared code related to this work, check if you include:
Specification of dependencies. [Yes] This can be found in Appendix and at our GilHub repository (below).
Training code. [Yes] This can be found in the main YAIB code (cross-validae ion ion.py and train. py. Details for the used parameters can be found in Imppenddxx G
Evaluation code. [Yes] See answer above.
(Pre- (trannt model(s). [Yes] Pretrained models for each task are publicly available and can be provided upon request.
README file includes table of results accompanied by precise command to run to produce those results. [Yes] We include the results in the main text and Appendix B Additionally, we have reproducibility document (PAPER. md) in our repository that follows common guidelines.
For all reported experimental results, check if you include:
The range of syper-parameters considered, method to select the best hyper-parameter configuration, and specification all hyper- parerr-rrrmteerss used to generate results. [ Yes) For the method see subsection 3.5) for the hyperparameter configuration, see: Appendix Ii For the individual hyperparameter configurations per model, dataset, and task, see subsection H.2 and subsection H.3 for ml and dl models, respectively.
The exact number of training and evaluation runs. [Yes] See subsection 4.1.
55

---

Published as a conference paper at ICLR 2024
A clear definition of the specific measure statistics used to report results. Yes]] See subsection 4.7
A description of results with central tendency (e.g. mean) & variation (e.g. eroor bars). [Yes] We report the standard deviation of all results, to accurately reflect the precision of our results.
The average runtime for each result, estimated energy cost. Yes] See Appendix G
A description of the computing infrastructure used.[Yes) See Appendix G
56


Reviews:
[{"review_id": "buvQ39WSx6", "reviewer": "Reviewer_VWQN", "summary": "This paper introduces an ICU benchmark that incorporates several ICU datasets, including MIIMC, eICU, HiRID, and AUMCdb. It covers multiple tasks such as mortality risk, kidney function, sepsis, acute kidney injury, and length of stay predictions. Furthermore, the paper offers an end-to-end pipeline for data preprocessing, model construction, training, and evaluation.", "questions": "Refer to weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses issues of comparability and reproducibility, both of which are crucial in the field of machine learning for healthcare.\n- The benchmark unifies four commonly used datasets and allows transfer learning.\n- The experiments on adapting task definitions clearly demonstrate the impact of cohort definitions, preprocessing strategies, and training protocols.", "weaknesses": "- I acknolwedge the motivation of this paper and appreciate considerable effort invested in establishing such a benchmark dataset. However, while the paper's central claim is the adaptability of the YAIB benchmark to other datasets, tasks, and models, this assertion necessitates a comprehensive experience with the benchmark, which is challenging to evaluate during the brief review period.\n\n- Datasets like MIMIC and eICU are highly complex and heterogeneous. The proposed YAIB benchmark only supports 52 clinical features (mainly vital signs and lab tests). A vast amount of other available features are ignored (like diagnosis, prescriptions, clinical notes, x-rays). This limited feature support might hinder future model advancements (which can be seen in Tables 3 & 4 where the performance between classic ML and DL models are very similar).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ZTVpeyvU4", "reviewer": "Reviewer_oWMD", "summary": "This paper introduces Yet Another ICU Benchmark (YAIB), a modular framework which allows researchers to define and conduct clinical machine learning (ML) experiments on multiple open-source medical datasets (MIMIC III/IV, eICU, HiRID, AUMCdb). The YAIB framework includes modules for 1) clinical concept definition on harmonized datasets 2) task specification and cohort selection 3) data preprocessing and feature extraction 4) ML model training and evaluation (both traditional ML and deep learning models). Through experiments on five pre-defined clinical tasks, this work demonstrated comparable results of benchmarking baseline models on different datasets, analyzed the impact of varying task/cohort definition and showed the utility of YAIB in transfer learning via a pre-train and fine-tune paradigm.", "questions": "1. In Section 4, you investigated the effects of small variations in task definitions. For the three variations you tried, the first and second are variations to the cohort and the feature selection but not to the clinical task definition, only the third is varying the definition of sepsis - which I understand is the only one directly relevant to clinical task definition. In this case, you may need to change how you describe the variations or redefine what you mean by \"task\" to avoid the confusion.\n\n2.  In Section 4 Preprocessing, for the aggregated features, what are the time windows for aggregation and the frequencies that you update those features? Is there any specific consideration in the choice for them?\n\n3. In Table 4, it would be helpful to also include the range or the units of KF/LoS, or additionally show the MAEs as % relative to the valid ranges of KF/LoS (absolute MAEs do not provide much information).\n\n4. A minor typo, a space is missing between \"repository\" and \"to\" near the bottom on Page 4.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: In previous literature, the ICU data benchmarks mostly focused on a single dataset or only supported subsets of multiple datasets, and often required modifications to core codebase for adding new clinical tasks. The originality of this paper lies in that 1) it utilizes the full datasets of five open-source ICU datasets, and for the first time introduced the AmsterdamUMCdb (AUMCdb) dataset into a benchmark 2) it is built on a flexible framework which decouples each individual module and allows users to easily add new tasks or adjust current modules.\n\n**Quality**: This paper specifically accounted for the common issues that researchers encounter when using ICU benchmarks, such as the lack of comparisons across multiple datasets, different task/cohort definitions. Via extensive experiments, this paper demonstrated the utilities of the proposed framework and analyzed the potential impact of small perturbations to the task/cohort definitions by ablation studies. The experiment results supported their claims that 1) comparable results can be achieved when different datasets are harmonized and ML pipelines are unified 2) small changes in definitions can lead to different results.\n\n**Clarity**: This paper is clear and well-written. The tables and figures are informative and easy to understand.\n\n**Significance**: Data processing and clinical task definition have always been a burden for researchers in machine learning for healthcare. Once made public, this work has the potential to help reduce the overhead in benchmarking and developing new ICU prediction methods. This work will also allow researchers to validate their findings by evaluating and comparing the results on multiple real-world ICU datasets. Thanks to the modular implementation, researchers working on other tasks involving ICU data, e.g. reinforcement learning or representation learning, may also benefit from some modules in the proposed framework. Thus, this work has both high technical significance and clinical relevance.", "weaknesses": "1. The definition of several terms used in the paper is not clear, including \"harmonization\", \"clinical concepts\". Though it can be inferred from later text what they may refer to, I think it would still benefit the readers if you can clearly define them the first time you use them in the paper. \n\n2. Based on the experiments, it seems that the time series features are extracted from numeric data rather than waveform data (please correct me if I am wrong). Currently, most ICU benchmarks used numeric data for prediction tasks and only very few used waveform data, but waveform data are very informative and may greatly help improve the predictive performance. Thus I think the contribution would be more solid if waveform data can be utilized for a new ICU benchmark. \n\n3. Basic statistics of the datasets, e.g. number of patients, ICU stays, and the prevalence of classes or the range of regression targets, are missing in the main paper (found them in appendix). Inclusion of such information will help readers get a general idea of the datasets and understand some of the experiment results, e.g. the discrepancy in the performances shown in Figure 2 may be due to the difference in prevalence in mortality across the datasets. Also, for the pooled dataset, is there any possibility that one dataset is dominating the training set for any task?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sB5oz0N073", "reviewer": "Reviewer_o57d", "summary": "In this study, the authors present an ICU-benchmark for clinical machine learning. They employ this framework to benchmark four ICU datasets. The toolkit offered encompasses data preprocessing, model training, and evaluation.", "questions": "Please see the weaknesses above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The design of the benchmark framework is clear. The paper offers an exhaustive detailing of preprocessing, modeling, and experimental results. The code's structure is clear, and the authors also provide comprehensive guidelines.", "weaknesses": "My primary concerns are:\n\nWhile the effort in constructing a comprehensive benchmark is great, I am uncertain about its alignment with the primary focus of ICLR. The paper appears to lean heavily towards a benchmarking contribution rather than a technical innovation. It might be more fitting for this work to be submitted to NeurIPS's benchmark track or journals such as Scientific Data.\n\nMy other comment is about the comparative scope of the study. The authors posit that their framework offers flexibility and the potential for application to other datasets, suggesting its closeness to a clinical ML toolkit. I recommend contrasting this work with existing clinical ML toolkit contributions, such as references [1,2] for a more comprehensive perspective.\n\n[1] Saveliev, E. S., & van der Schaar, M. (2023). TemporAI: Facilitating Machine Learning Innovation in Time Domain Tasks for Medicine. arXiv preprint arXiv:2301.12260.\n\n[2] Yang, C., Wu, Z., Jiang, P., Lin, Z., Gao, J., Danek, B. P., & Sun, J. (2023, August). PyHealth: A Deep Learning Toolkit for Healthcare Applications. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 5788-5789).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JHBoA5bx9H", "reviewer": "Reviewer_Q5t5", "summary": "The authors present an innovative framework that facilitates reproducible and comparable machine learning experiments using publicly available ICU datasets. This framework offers: i) thorough and standardized definitions of cohorts and feature extractions, ii) Transparent feature engineering and preprocessing, iii) commonly compared deep learning baselines (including RNNs, TCN, Transformer) with flexibility in the network architecture and hyper-parameters optimization, and iv) performance metrics. This work addresses the growing demand for a standardized framework for comparing and reproducing widely used machine learning and deep learning benchmarks. Moreover, this framework can be further utilized to provide a standardized way of evaluating newly developed DL methods. This work will be extremely helpful in opening the door for wider adoption of recent deep learning methods in real clinical practices.", "questions": "1.\tThere exists a harmonized ICU dataset called BlendedICU [A] that incorporates AmsterdamUMCdb, eICU, HiRID, and MIMIC-IV. While the reviewer acknowledges that [A] has not been available online when the proposed work was submitted, what is the distinction and contribution of the proposed work from [A]?\n2.\tRegarding Weakness #1: As far as the reviewer is aware, variations in units and features can occur both within and across datasets due to differences in observation circumstances (e.g., using different medical devices). How did the authors handle such issues? Additionally, what is the reason for the extracted datasets having missing therapeutic interventions and comorbidities (while some of them are mentioned in Appendix C), which are often extremely important for predicting clinical outcomes?\n\n[A] M. Oliver et al., \"Introducing the BlendedICU dataset, the first harmonized, international intensive care dataset,\" Journal of Biomedical Informatics, 2023.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.\tThe paper is well-written.\n2.\tThe authors provide a very solid framework, covering from defining clinically relevant cohorts in ICU admissions to training and evaluating ML/DL methods. Moreover, this framework has the flexibility to incorporate user-specific DL networks which will be extremely useful in providing comparable and reproducible evaluations.\n3.\tThe well-defined cohorts and harmonized datasets hold significant potential for various applications, including domain adaptation, time-series generation, and more.\n4.\tThe experiments that emphasize the impact of incorporating different features and employing different data assembly processes (e.g., cohort/label definition) are thoroughly investigated.", "weaknesses": "1.\tAlthough the authors incorporate a wide range of measurements from the ICU datasets, many therapeutic interventions and comorbidities are missing, which can be crucial for predicting clinical outcomes of interest. Moreover, the time-varying features are mostly continuous while there exist many binary and categorical features in these ICU datasets. \n\n2. Minor comments: typo in p4 “repositoryto” $\\rightarrow$ “repository to”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "40UsZDdoE7", "reviewer": "Reviewer_5BSM", "summary": "The paper presents a multi-dataset framework that consists of preprocessing and processing techniques to aid reproducible research with consistent data and experiment methods. The authors demonstrate that the experiment choices can have a substantial impact on model prediction performance.", "questions": "1. It is possible that thresholds used to define outcomes from time series data may update over time based on new knowledge, thus impacting outcome labels. Does the repo have support to enable users to define their own outcome definitions across datasets? For example, a patient AKI outcome may occur when patient creatinine falls below a specific threshold. Or are outcome labels strictly based on definitions included in the datasets?\n\n2. Are models other than those in scikit-learn hardcoded into the repo or are there wrappers for these as well? Is there support for users to include and evaluate new/customized model architectures in the pipeline?\n\n3. Is there a reason why this framework is limited to ICU/healthcare settings and not time series in general. I could see a pipeline like this being applicable to time series from many domains. It may be worth elaborating on why the framework is ICU-specific. It may also be useful to compare YAIB to similar non-healthcare related frameworks.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "I commend the authors for their hard work to create an open source code repository, which is not a trivial task. Based on the description in the paper, this repository could have substantial impact for streamlining research with ICU datasets. \n\nThe paper is very easy to follow and organized well. Thorough descriptions of datasets and code are included. \n\nPaper includes example scripts in the appendix to demonstrate code simplicity and example use cases. \n\nTable 1 presents an excellent overview and comparison of the proposed methods and prior work. \n\nThe paper demonstrates use of the proposed framework with four prominent ICU datasets in ML literature. Additionally the datasets cover a range of populations including those in US and Europe, which can promote research that is applicable across different populations.", "weaknesses": "1. The baseline table is useful for future work to compare against. However, this table does not seem relevant to motivating the proposed framework and could be included in the appendix. To better motivate the value of the proposed unifying framework, it would be useful to show more experiments regarding significant differences in model performance when different experiment methods are used (outcome labels, cohort selection, etc.). For example, you could consider an ablation study to show that model performance significantly improves or decreases based on various outcome definitions for datasets with consistent preprocessing, model choice, hyperparameters. The experiment mentioned in the paper regarding exclusion criteria would also be a good example to elaborate on and include table results comparing impact of different exclusion criteria.\n\n2. Be specific about performance gains and losses (example: improved by XX%) and their statistical significance. A significant difference in model performance due to unaligned data preprocessing and processing pipeline would help motivate the proposed unifying framework. Stating that “differences were bigger for some datasets than others” is not a robust explanation. \n\n3. Novel ML research often proposes a new model that is evaluated across multiple datasets. Enabling support for new models to be easily integrated into this framework is important. The appendix provides example code for training a specific model, such as those in scikit-learn or implemented in pytorch. Including more thorough explanation and/or code examples for how to train a novel/custom model, such as pytorch-based model implemented in a separate python file, would be useful. \n\n4. The author mentions flexibility of the repo in allowing users to configure data preprocessing in a streamlined fashion across multiple datasets. However, does this somewhat deflate the original purpose to enable reproducible research through consistent dataset preprocessing and processing. Researchers may still conduct research with various preprocessing steps that differ across literature. The value of this framework hinges on if research exactly states the detailed use of YAIB in their paper/methods AND if future work follows these exact same steps. It would be useful if the author's address this point and potentially expand their unifying framework to include recommended approaches for how to reference/cite use of YAIB in future work that could promote reproducible research. \n\nThe low score is mostly because of points 1-3. If these points are addressed I would be happy to raise the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)