## 4 Experiment Settings
### Data Structure for Evaluation  
A robust evaluation requires not only splitting the data into training and test sets but also reserving a dedicated portion for calibration and UQ. In this study, we adopt a Bootstrapping approach to generate multiple training and out-of-bag (OOB) samples, ensuring that our performance estimates are less sensitive to a particular data partition. Specifically, we perform the following steps for each bootstrap iteration \(b\in\{1,\ldots,B\}\), where \(B=200\):  
1. _Bootstrap Sampling._ Let the preprocessed dataset be \[D=\{(x_{i},y_{i})\}_{i=1}^{N},\quad N=995,\] (41) where \(y_{i}\in\{0,1\}\) represents the binary target (e.g., ICU admission in the sepsis domain). For each iteration \(b\), we draw \(N\) samples with replacement from \(D\) to create \[D_{\text{train}}^{b}=\big{\{}\left(x_{j},y_{j}\right)\big{|}\,j\sim\mathcal{U} \{1,N\}\big{\}}_{j=1}^{N}.\] (42) The points not selected in this bootstrap sample form the out-of-bag (OOB) set, denoted by \(\text{OOB}^{b}\subseteq D\). Typically, \(\big{|}\text{OOB}^{b}\big{|}\) ranges between 343 and 394 instances (approximately 35% to 40% of the data).
2. _Stratified Splitting of OOB._ We further split OOB\({}^{b}\) into a calibration set \(D_{\text{cal}}^{b}\) and a test set \(D_{\text{test}}^{b}\). The calibration set is a stratified sample of fixed size (171 instances), preserving the class ratio of \(\text{OOB}^{b}\). Formally: \[D_{\text{cal}}^{b}=\big{\{}(x_{k},y_{k})\in\text{OOB}^{b}:k\in S_{\text{cal} }^{b}\big{\}},\quad D_{\text{test}}^{b}=\text{OOB}^{b}\setminus D_{\text{cal}} ^{b}.\] (43) With this partition, the minority class proportion in \(D_{\text{cal}}^{b}\) remains representative of OOB\({}^{b}\).
3. _Pipeline Execution._ For each iteration \(b\): 1. _Model Training:_ Train a new model \(M_{b}\) (e.g., XGBoost, CatBoost, Decision Tree, Random Forest) on \(D_{\text{train}}^{b}\). 2. _Calibration and UQ:_ Use \(D_{\text{cal}}^{b}\) to apply post-hoc calibration (Section 4.3) and to estimate uncertainty (Section 4.4). 3. _Model Evaluation:_ Report metrics on \(D_{\text{test}}^{b}\).  
This OOB bootstrapping approach ensures an unbiased estimate of each model's performance and calibration quality. By repeatedly sampling and training on different subsets, we reduce the dependence on a single train-test split and gain a more robust understanding of how well models generalize to new data.  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Model** & **Hyperparameters (Search Interval)** \\ \hline \multirow{2}{*}{XGBoost} & max\_depth: (3, 20), gamma: (0, 1), learning\_rate: (0.01, 0.6), subsample: (0.5, 1), colsample\_bytree: (0.5, 1), reg\_alpha: (0.1, 20), reg\_lambda: (0.1, 20), n\_estimators: (100, 1000) \\ \hline \multirow{2}{*}{CatBoost} & max\_depth: (3, 10), learning\_rate: (0.001, 0.4), subsample: (0.4, 1), colsample\_bylevel: (0.4, 1), 12\_leaf\_reg: (0.1, 20), n\_estimators: (100, 500) \\ \hline \multirow{2}{*}{Decision Tree} & max\_depth: (3, 15), min\_samples\_split: (2, 10), min\_samples\_leaf: (1, 10), max\_features: (0.1, 1.0) \\ \hline \multirow{2}{*}{Random Forest} & n\_estimators: (100, 500), max\_depth: (3, 20), min\_samples\_split: (2, 10), min\_samples\_leaf: (1, 10), max\_features: (0.4, 1) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Hyperparameter optimization settings for the ML models. Intervals denote the continuous ranges explored via Bayesian Optimization.