Confidence Interval Estimation of Predictive Performance in the Context of AutoML

Authors: konstantinos  paraschakis
andrea  castellani
giorgos  borboudakis
ioannis  tsamardinos
Venue: automl_cc_AutoML_2024_Conference
Published: 2024-02-26 17:21:25

Abstract: Any supervised machine learning analysis is required to provide an estimate of the out-of-sample predictive performance. However, it is imperative to also provide a quantification of the uncertainty of this performance in the form of a confidence or credible interval (CI) and not just a point estimate. In an AutoML setting, estimating the CI is challenging due to the ``winner's curse", i.e., the bias of estimation due to cross-validating several machine learning pipelines and selecting the winning one. In this work, we perform a comparative evaluation of 9 state-of-the-art methods and variants in CI estimation in an AutoML setting on a corpus of real and simulated datasets. The methods are compared in terms of inclusion percentage (does a 95\% CI interval include the true performance at least \%95 of the time), CI tightness (tighter CIs are preferable as being more informative), and execution time. The evaluation is the first one that covers most, if not all, such methods and extends previous work to multi-class, imbalanced, and small-sample tasks. In addition, we present a variant, called BBC-F, of an existing method (the Bootstrap Bias Correction, or BBC) that maintains the statistical properties of the BBC but is more computationally efficient. The results support that BBC-F and BBC dominate the other methods in all metrics measured. However, the results also point to open problems and challenges in producing accurate CIs of performance, particularly in the case of multi-class tasks.

Main paper: 
 introduction: 
 In any practical application of supervised machine learning, one needs to provide an estimate of the out-of-sample (hereafter, oos) performance (i.e., an estimate of generalization performance) of the final model. However, it is also important to quantify the uncertainty of this performance estimate. This quantification is often presented in the form of a confidence interval, or CI hereafter for short 1 . A 𝑎-CI is defined as an interval that includes the true performance of the model with (at least, if being conservative) probability 𝑎 in identical repetitions of the analysis with new datasets from the same data distribution. In the rest of the paper, by default 𝑎 = 0.95 and the term CI refers to a 95% CI, unless otherwise stated. Following the literature [19] of CIs of predictive performance, we focus on one-sided CI. In one-sided intervals the upper bound of the interval is the maximum possible performance; the lower bound of the interval is adjusted so that the the true performance falls above that level (at least) 95% of the time. In contrast, two-sided intervals "allow" some of these 5% failing cases to fall higher than the upper bound of the interval. The reason for this preference is that in practice it is important to find the tighter lower bound of performance. The importance of quantifying uncertainty is shown in this simple example: a model with 0.70 AUC could be as good as random guessing if the CI is the interval [0.3, 1.0]. CIs can also facilitate comparison of models: two models with respective AUC performances of 0.85 and 0.90, may actually be on par if their CIs are [0.70, 1.0] and [0.75, 1.0]. Important properties for CI estimation are the inclusion percentage and the interval lower bound tightness. For a 95% CI estimate the inclusion percentage should ideally also be 95%. If it is higher the estimate is unnecessarily wide and conservative, but still acceptable. If the inclusion percentage is lower than 95%, the estimate is optimistic and, arguably, misleading. Tightness is defined as the difference between the true performance and the lower bound of the interval. Between two CI estimates with an inclusion percentage of at least 95%, the tighter interval is the most informative (smaller positive tightness is better). Ideally, a CI estimate should cover exactly 95% of probability with the smallest tightness possible. For example, the interval [0.00, 1.00] is a 95% CI of the AUC for any binary classification task, but so conservative that is completely useless.Accurately estimating CIs is challenging, particularly in the context of AutoML. In most AutoML systems, numerous machine learning pipelines (a.k.a., configurations) are tried and the winning one is employed to construct the final model [18] . Returning the cross-validated performance estimate and the CI of the winning configuration exhibits the "winner's curse" bias [27, 28] , which can be up to 0.2 AUC [29] . Intuitively, the phenomenon of "winner's curse" can be explained as an increasing probability to overfit the test set or sets (in cross-validation) as many models are tried [17, 28] .In this paper, we examine and benchmark the state-of-the-art methods proposed for CI estimation applicable to an AutoML context. We employ JADBio [30] as our AutoML platform of choice to generate, fit, and cross-validate numerous configurations on a corpus of real binary classification datasets covering imbalanced classes and small-sample scenarios. The configurations include the application of feature selection algorithms and linear and non-linear classifiers with different values for their hyper-parameters. Experiments on real datasets are complemented with ones on synthetic datasets under controlled conditions. The oos predictions of these configurations on each sample in the cross-validation test folds are employed to select the winning configuration. The final predictive model is built using the winning configuration. The 95% CI of its performance is estimated from the matrix of oos predictions using 9 different state-of-the-art methods and variants. All algorithms employed are model agnostic; they only require the prediction matrix to provide estimates and do not depend on the inner workings of the configurations. The methods are compared and evaluated with respect to their inclusion percentage and tightness. The methods benchmarked include a new variant called BBC-F standing for Bootstrap Bias Correction on Folds. BBC-F extends the previous BBC method [27] that was shown to remove the bias due to the winner's curse and return accurate estimations of performance. However, an evaluation of BBC w.r.t. providing CI estimates was lacking.The results demonstrate that BBC and BBC-F dominate all other methods in all metrics measured. BBC-F is on par with BBC in terms of inclusion percentage or tightness, but it is computationally more efficient. Hence, within the scope of our experiments, we would suggest BBC-F as the method of choice for CI estimation. The contributions of the paper are to extend previous evaluations to (a) all available methodologies for CI estimation, (b) extend previous evaluations to imbalanced and small-sample tasks, and (c) propose the BBC-F variant and conclude with a clear winning methodology. While informative, the evaluation still has numerous limitations. The limitations, open problems, future directions, and the impact of this work are discussed in separate sections that conclude the paper. 
bootstrap bias correction for performance and ci estimation: 
 We now present in detail the main ideas of Bootstrap Bias Correction (BBC) and its new variant BBC Fold (BBC-F). The input to these methodologies is an oos prediction matrix produced during model training and cross-validation. Hence, we start by describing this process first. In the context of AutoML, numerous configurations, denoted by 𝑚 𝑖 , are trained. The configurations may include several types of algorithms (preprocessing, imputation, feature selection, and modeling) and their hyper-parameters. The choice of the algorithm for each step can also be thought of as another hyper-parameter so that all choices of the configuration are captured in a single vector 𝜃 𝑖 . Instead of considering as having available numerous learning pipelines, we can equivalently consider that we have available only a single learning function 𝑓 parameterized by 𝜃 as its hyper-parameters. A static search strategy (i.e., a strategy where the set of configurations is pre-determined) in the space of all possible configurations can be encoded by a set of vectors 𝜃 : Θ = {𝜃 𝑖 , 𝑖 = 1, . . . , 𝑛}. Such search strategies include grid search and random search, both of which have been employed in an AutoML context [33] . During execution, the configurations in Θ are 𝐾-fold cross-validated. The cross-validation procedure identifies the winning configuration according to some performance criterion (AUC, accuracy, 𝑅 2 , c-index) which is employed to train the final model on all available data. We denote this procedure as Cross-Validation with Tuning or 𝐶𝑉𝑇 (𝑓 , 𝐷, Θ), where 𝑓 is the learning method, 𝐷 the training data (partitioned to folds), and Θ the set of configurations to execute. The pseudo-code is in [27] , omitted due to space limitations. The function call returns ⟨𝑀, Π⟩, where 𝑀 is the final model, and Π is the matrix with the predictions of each configuration on each test sample. Specifically, Π 𝑖,𝑗 contains the prediction of a model produced by configuration 𝑚 𝑗 on sample with index 𝑖, when 𝑖 was in the test fold of cross-validation. Hence, Π 𝑖,𝑗 contains only oos predictions.To produce an unbiased estimate of the performance of the final model returned by 𝐶𝑉𝑇 (𝑓 , 𝐷, Θ) one needs to account for the fact that the winning configuration is selected among many candidates. Hence, one needs to create an estimation procedure that produces untainted test sets that have not been used to select the winning model. One option is to cross-validate CVT itself, i.e., cross-validate a procedure that already uses cross-validation to select the best model. This leads to the nested cross-validation protocol [21] . Each outer fold of the nested cross-validation is used only once to estimate the performance of the winning model for the corresponding training set, avoiding the winner's curse bias.Another option is to bootstrap the CVT procedure [11] ; we will call this the direct bootstrap approach to distinguish it from the BBC. In the direct approach, the samples of dataset 𝐷 are selected with replacement creating datasets {𝐷 1 , . . . , 𝐷 𝐵 } where 𝐵 is the number of bootstraps. 𝐷 𝑏 are called the in-bag samples, while the samples not selected 𝐷 \𝑏 ≡ 𝐷 \ 𝐷 𝑏 are called the out-of-bag samples. The performance 𝐿 𝑏 of the winning model returned by CVT on 𝐷 𝑏 is estimated on 𝐷 \𝑏 . Notice that the samples in 𝐷 \𝑏 are not used in training of the final model for dataset 𝐷 𝑏 , but also importantly, they have not been used to select the winning configuration either. The average performance over all bootstraps is returned as a point estimate of performance, while the interval that covers 95% of the performance values {𝐿 1 , . . . , 𝐿 𝐵 } is returned as the CI. Both methodologies are computationally expensive: each of them reruns CVT and retrains |Θ| models on 𝐾 folds for each outer fold of the nested cross-validation or bootstrap to perform. However, notice that neither procedure loses samples to estimation: the final model is trained on all available data by applying CVT on the original dataset [28] . This is why nested cross-validation was employed in one of the first AutoML tools [23] that targeted low-sample, bioinformatics analyses.The key enabling idea of BBC is to bootstrap the oos prediction matrix Π, instead of the original dataset 𝐷. Equivalently, instead of retraining models on each 𝐷 𝑏 and selecting the winner in each bootstrap, BBC only selects the winner for each Π 𝑏 . In other words, it approximates directly bootstrapping the complete CVT procedure with bootstrapping only the step that selects the winning model, the step that creates the estimation bias. More specifically, BBC creates a bootstrap population of prediction matrices {Π 1 , . . . , Π 𝐵 }. It then identifies the winning model 𝑀 𝑏 in each one of them. Assuming that the index of 𝑀 𝑏 is 𝐽 𝑏 , an estimate of the oos performance of 𝑀 𝑏 is the average performance in the out-of-bag predictions 𝐿 𝑏 = Π \𝑏 (:, 𝐽 𝑏 ). In the latter equation, we make use of the notation Π \𝑏 (:, 𝐽 𝑏 ) to denote the 𝐽 𝑏 column of the matrix. From the distribution of {𝐿 1 , . . . , 𝐿 𝐵 } BBC can produce a point estimate of performance. The two sided 𝑎-CI is computed by taking the interval [𝐿 (𝑎/2) , 𝐿 ( (1-𝑎)/2) ] (in the paper higher 𝐿 is considered better performance), where 𝐿 (𝑖 ) denotes the 𝑖-th quantile of the distribution of 𝐿. For 𝑎 = 0.95 this corresponds to [𝐿 (0.025) , 𝐿 (0.975) ]. The one-sided 𝑎-CI is computed as [𝐿 (𝑎) , 𝐿 𝑚𝑎𝑥 ].BBC does not require training any new models, just the operations of bootstrapping a matrix and identifying the column with the maximum performance. Intuitively, we expect BBC to provide correct estimations because the winning configurations are identified from predictions on samples not used for training models in each Π 𝑏 , and the performance of the winner is estimated from samples not used to select the winner in Π \𝑏 . The time complexity of BBC is as follows: Given the prediction matrix Π, BBC will repeat 𝐵 times (for each bootstrap) the calculation of 𝐶 average performances, where 𝐶 is the number of the configurations. For simplicity, let us assume that each performance is computed in time linear to the sample size 𝑁 (this is true for accuracy, but AUC requires sorting the values so it has complexity 𝑂 (𝑁 log 𝑁 ) instead). To identify the winner is a linear operation in 𝐶 time; checking the average performance of the winner in the out-of-bag data takes time at most 𝑁 . The total complexity of BBC is thus O(𝐵•𝐶 • 𝑁 + 𝐵 •𝐶), or simply O(𝐵 •𝐶 • 𝑁 ).BBC-F is shown schematically in Figure 1 ; the pseudo-code is in Algorithm 1. The BBC-F algorithm is the same as BBC with one difference: the bootstrapping procedure of Π does not resample over samples (rows) of the matrix but over cross-validation folds, i.e., groups of samples. To obtain BBC-F from BBC we convert the Π 𝑁 ×𝐶 matrix to Π ′ 𝐾 ×𝐶 , where 𝐾 is the number of folds of cross-validation. Each element Π ′ 𝑖,𝑗 contains the average performance of a model produced by configuration 𝑚 𝑗 on samples of fold 𝑓 𝑖 . The time complexity of the conversion of Π to Π ′ is O(𝐶 • 𝐹 • 𝑁 /𝐹 ) (computation of 𝐹 average performances, each including 𝑁 /𝐹 samples for each configuration 𝐶). Hence, the total complexity of the algorithm is found by substituting 𝐹 for 𝑁 in the complexity of BBC and adding the complexity of the conversion step: O(𝐵𝐶𝐹 + 𝐶𝑁 ). For a large number of bootstraps the first term dominates; the ratio of the complexity of BBC to BBC-F is in the order of 𝑁 /𝐹 , which equals 10 for a dataset with 1000 samples being 10-fold cross-validated. BBC-F follows the same principles as BBC to ensure unbiased estimations while being more computationally efficient. On the other hand, a bootstrapping of a matrix with 1000 rows, one for each sample, will turn into bootstrapping a matrix with 10 rows, one for each fold, potentially losing information. The comparative evaluation examines the trade-off between the quality of estimation and computational complexity. 
related work: 
 The topic of providing a point estimate of the predictive performance of machine learning models has been studied in the context of supervised machine learning. Typical methodologies include the hold-out, repeated hold-out, and 𝑘-Fold Cross Validation protocols [28] . Some recent works include [14, 32] and we do not attempt a full review of the literature.Arguably, the first method for providing point estimates of predictive performance in the context of model selection or AutoML where numerous configurations are tried is the nested-cross validation protocol [21, 1, 22] . The first method to try to remove the bias of estimation due to the winner's curse is the Tibshirani and Tibshirani method (TT) [26] . The TT method did not require nesting the cross-validation step, but it did not provide accurate estimations either [29] . Nevertheless, it did inspire future work in bias removal of the winner's curse.One of the first works for providing confidence intervals, instead of just point estimations, of performance in the context of AutoML was the Bootstrap Bias Correction Cross-validation (BBC) method [27] . The MABT [19] was recently proposed for the same problem. In addition, several other methodologies that implicitly deal with CI estimation in the presence of multiple models have been proposed or are easily adapted. We include these works as they have also been compared against MABT: these are the Tilted Bootstrapping (BT) [7] , Hanley-McNeil (HM) [9] , and DeLong (DL) [24] , in both the standard and 10p variation, where an auxiliary two-step selection process is employed [19] . While all methods take the same input (the oos prediction matrix, labels, and fold indices), the MABT, DL, HM and BT methods follow a different model selection process than BBC and BBC-F. We note that they have several drawbacks compared to BBC: (a) they don't return performance point estimates, (b) apply only to binary classification tasks, while BBC is metric agnostic and can be computed for other tasks, such as multi-class classification, regression or time-to-event analysis, and (c) the 10p variation doesn't perform well on problems with low sample size or imbalanced classes, due to the two-step configuration selection process. They also require the user to set the proportions to split the dataset to validation/evaluation sets. The only hyper-parameter of BBC and BBC-F is the number of bootstraps to run which is intuitively easy to decide. 
experimental set up: 
 In the experiments, we compare BBC-F to several algorithms for CI estimation, in terms of the quality of the estimated 95% confidence intervals of the area under the ROC curve (AUC). Furthermore, we also compare the running times and scaling behavior of BBC-F and BBC. The algorithms were compared on simulated and real-world data. We only focus on low-sample settings, as (a) it has been shown that the bias of the uncorrected estimate by CVT approaches zero regardless of the number of configurations [27] (i.e., it's less than 1% for 1000 samples), and (b) CI estimates are mainly useful in low-sample settings where performance estimates have high variance (i.e., in the limit one would expect the CI range to converge to zero). We use two metrics to compare the algorithms: inclusion percentage and tightness. For real data, we use the performance on the holdout set as an estimate of true performance, which is large enough to ensure an accurate estimate. Next, we provide additional details about the algorithms, data and protocols used in the experiments. Algorithms and Implementations. We compare BBC-F against two state-of-the-art algorithms for confidence interval estimation, the original BBC algorithm [27] , and MABT [19] . We also compare them to standard approaches: Tilted Bootstrapping (BT) [7] , Hanley-McNeil (HM) [9] and DeLong (DL) [24] , both with the standard and the 10p variation [19] . For MABT, BT, HM and DL we used the implementation of [19] available at https://github.com/pascalrink/mabt-experiments. As a baseline, we also directly bootstrapped the performance estimates of the selected model only, ignoring the "winner's curse" problem, referred to as Naive Bootstrapping (NB). For the comparisons on the real-world data we used JADBio [30] as the CVT method to generate and execute several configurations to run and compute the prediction matrices Π. JADBio is a commercial software-asa-service for AutoML; it uses various feature selection methods, such as SES [13] and LASSO [25] , and modeling algorithms, such as logistic regression, support vector machines, decision trees and random forests. Implementations of BBC-F and BBC, as well as all JADBio results (fold indices, configuration predictions, and outcome labels) and code for reproducing the results are available at https://github.com/kparaschakis/BBC_algorithm. Real Data. We use datasets from OpenML [31] , shown in Table 1 . The datasets were selected to contain at least 1000 samples, to have enough samples for accurate performance estimation. from a 𝐵𝑒𝑡𝑎(𝛼, 𝛽). In our experiments, we used (𝛼, 𝛽) ∈ {(9, 6), (24, 6)}, which correspond to mean performances of 0.6 and 0.8 with variances of 0.015 and 0.0052 respectively. Given an outcome 𝑌 and the AUC of a configuration, a prediction 𝑋 0 is drawn from N (0, 1) when 𝑌 = 0, and a prediction 𝑋 1 from N (𝜇, 1) when 𝑌 = 1. Then:𝑋 0 ∼ 𝑁 (0, 1), 𝑋 1 ∼ 𝑁 (𝜇, 1) ⇒ 𝑧 := 𝑋 1 -𝑋 0 -𝜇 √ 2∼ 𝑁 (0, 1).𝐴𝑈𝐶 = 𝑃 (𝑋 1 > 𝑋 0 ) = 𝑃 𝑋 1 -𝑋 0 -𝜇 √ 2 > - 𝜇 √ 2 = 𝑃 𝑧 < 𝜇 √ 2 = Φ 𝜇 √ 2 ⇒ 𝜇 = √ 2Φ -1 (𝐴𝑈 𝐶),where Φ(•) is the CDF of the standard normal distribution, which determines 𝜇 unequivocally. Evaluation Protocol. For the simulated data, each combination of settings was repeated 200 times to generate prediction matrices and compute CIs using all methods. For the real data, we performed 100 repetitions of a train/hold-out split. For small datasets (see Table 1 ), 50 samples were randomly sampled as the training set, while 500 were used for the larger datasets. For each split, JADBio was used (with 10-fold CV and a grid search of 766 configurations) to get the oos predictions and their fold index. Finally, each method is applied on these results, and the winning configuration is then applied on the hold-out set to get an estimate of its theoretical performance. Note, that a potentially different "winner" configuration can be selected by each of the three groups of methods: {BBC, BBC-F}, {DL, MH, BT}, and {DL10p, MH10p, BT10p, MABT}. 
experimental results: 
 In this section, we present the results of the experiments. We compare BBC-F to all other algorithms on simulated and real data in terms of inclusion percentage and tightness, as well as the running times of BBC-F and BBC. We also include results showing the hold-out performance of models selected by each method, as they use different model selection methods. Exact binomial tests are performed to test whether the inclusion percentage matches the theoretical CI range. Furthermore, we note that some methods failed to execute on some datasets, particularly in cases with a lowfrequency minority class or extra validation/evaluation partitions. .93 (0.22) 0.91 (0.20) 0.87 (0.27) 0.87 (0.27) 0.37 (-0.08) 0.59 (0.09) 0.41 (-0.06) 0.54 (0.00) 0.52 (-0.02) 0.28 (-0.04) phishing 0.96 (0.13) 0.90 (0.11) 0.64 (0.15) 0.64 (0.12) 0.89 (0.20) 0.24 (0.00) 0.22 (-0.03) 1.00 (0.28) 0.94 (0.26) 0.44 (-0.01) elec 0.99 (0.05) 0.97 (0.05) 0.97 (0.07) 0.97 (0.06) 0.99 (0.07) 0.99 (0.10) 0.94 (0.05) 1.00 (0.13) 0.99 (0.08) 0.90 (0.02) mozilla4 0.97 (0.03) 0.94 (0.03) 0.85 (0.04) 0.82 (0.03) 0.91 (0.06) 0.84 (0.06) 0.71 (0.02) 0.98 (0.10) 0.91 (0.06) 0.76 (0.01) nomao 0.97 (0.02) 0.97 (0.02) 0.85 (0.02) 0.89 (0.02) 0.95 (0.03) 0.88 (0.02) 0.78 (0.01) 1.00 (0.05) 1.00 (0.04) 0.70 (0.00) adult 1.00 (0.04) 0.99 (0.04) 0.91 (0.05) 0.94 (0.06) 0.91 (0.06) 0.95 (0.07) 0.92 (0.04) 0.98 (0.10) 0.95 (0.06) 0.66 (0.01) bank 0.93 (0.06) 0.93 (0.06) 0.82 (0.07) 0.92 (0.09) 0.87 (0.08) 0.89 (0.08) 0.83 (0.06) 0.94 (0.11) 0.91 (0.11) 0.59 (0.00) eeg-eye 0.92 (0.05) 0.93 (0.06) 0.87 (0.07) 0.88 (0.06) 0.91 (0.08) 0.99 (0.11) 0.79 (0.04) 0.98 (0.13) 0.97 (0.10) 0.83 (0.03) Avg Rnk 1.61.8 5.9 5.3 5.6 6.9 8.0 5.8 4.9 9.0 Comparison of CI estimation. Tables 2 and 3 show the results for the simulated data and real data respectively. The results on the real data are split into two groups, low sample size datasets (𝑁 = 50) and high sample size datasets (𝑁 = 500). The tables show the inclusion percentages and the tightness (in parentheses) for each method and settings. Bold numbers indicate values that were not statistically significantly lower than 95% at a 5% statistical level according to an exact binomial test. The average rank of each method is shown at the bottom of the tables. For a fair comparison, only rows where all methods have returned results were used. To rank methods, the following rules are applied: (a) non-rejected (bold) methods get ranked higher than rejected ones, (b) tightness is used as a tie breaker for non-rejected methods, with lower values being ranked higher, (c) rejected methods are ranked according to their distance of their inclusion percentage to 95%, and (d) all tied methods get assigned the highest rank.In addition to the tables, we also visually summarize the results on the swarm plots in Figure 2 . Blue dots correspond to cases where the hypothesis test is not rejected, while orange dots correspond to the rejected ones. Again, only rows where all methods returned results were considered. The y-axis shows the ratio of the average tightness of each method relative to BBC, that is, a value > 1 means that the method is worse than BBC and vice versa. We chose BBC as the baseline as it was the best performing method, which also happens to always return non-rejected estimates.The results show that BBC and BBC-F clearly dominate all competing methods, producing accurate and tight CIs. As expected, NB fails to produce accurate CIs, as it ignores the bias introduced due to the "winner's curse". The other methods often produce statistically significantly different CIs or even fail to run, especially on smaller and unbalanced datasets and, when they don't, their average tightness is worse than that of BBC and BBC-F.Comparison of selected model performances. Next, we computed the average true and hold-out performance of the selected models on the simulated and real data respectively. On the simulated data, {BBC, BBC-F} had a performance of 0.9081, followed by {DL, HM, BT} with 0.9022, and {DL10p, HM10p, BT10p, MABT} with an AUC of 0.8948. Similarly, on the real data {BBC, BBC-F} had a performance of 0.8570, followed by {DL10p, HM10p, BT10p, MABT} with 0.8499, and {DL, HM, BT} with an AUC of 0.8496. While the differences are small, BBC and BBC-F again consistently outperform all competitors.Comparison of BBC-F and BBC running times. Finally, we performed an empirical evaluation to compare the running time of BBC-F and BBC. By default, sample size was set to 500, the number of configurations to 5 and the number of folds to 3. We performed 3 experiments, one for each of the above parameters, varying one of them and keeping the rest fixed, to investigate how they scale with sample size, number of configurations and number of folds.The results are summarized in Figure 3 . The y-axis shows the median running time on logarithmic scale based on 100 repetitions of the experiment. We observe that BBC-F consistently outperforms BBC by 1-2 orders of magnitude. Additionally, both algorithms exhibit similar scaling behavior w.r.t. the number of configurations, while the running time of BBC-F and BBC is not affected by sample size and number of folds respectively, as expected (see discussion in Section 2). It is important to note that, in practical applications, the number of folds is typically no higher than 10. Overall, BBC-F performs almost identically to BBC, while being computationally faster. 
impact, limitations, and conclusions: 
 After careful reflection, the authors have determined that this work presents no notable negative impacts on society or the environment. We hope the work to have a positive scientific impact raising awareness regarding the information to provide users of machine learning and facilitating decision-making based on ML results. There are numerous limitations in the study. First, the approximation of BBC and BBC-F to direct bootstrapping is only valid for static HPO strategies. While they can be applied to dynamic strategies in principle (as they only require the prediction matrices as input), it is unclear if and how the dynamic hyper-parameter search introduces any bias, as the input dataset is used to guide its search. Other metrics of classification performance (accuracy, F1, balanced accuracy) need to be considered. BBC-F runs only when a sufficient number of folds have been cross-validated. Other supervised tasks, such as multi-class classification, regression and censored time-to-event analyses need to be considered.The paper presents the first extensive evaluation of CI estimation methods on binary classification tasks in the context of AutoML. It introduces a new variant, BBC-F that achieves a speedup of 𝑁 /𝐹 vs. BBC, where 𝑁 is the number of samples, and 𝐹 is the number of folds, with a minimal drop in estimation quality. BBC and BBC-F dominate all other methods w.r.t. probability coverage and interval tightness. The problem of computing CIs when providing performance estimates in ML has not been sufficiently studied, in our opinion. The results and limitations point to new research directions and future work in the field. In Table 4 are reported the ROC-AUC performance on the simulation setting. In Table 5 are reported the AUC-ROC lower bound on the real-world benchmarks datasets.  
 
 Appendix: In Bayesian statistics, uncertainty is quantified with credible intervals instead. For all practical purposes, both CIs and credible intervals are employed in the same way for decisions. In the rest of the paper, we will focus on CIs. 


Decision: This paper investigates the estimation of confidence intervals for predictive performance in AutoML. Throughout the rebuttal period, the authors diligently addressed reviewers' concerns. After thoroughly considering the feedback from all reviewers and the authors' responses, I recommend acceptance and strongly advise the authors to integrate the reviewers' feedback into the final version of the paper.

Reviews: review: paper_summary: paper_summary: The contributions of this paper are:
1. Comparison of all state-of-the-art CI estimation methods.
2. Extending CI benchmarking to multi-class, imbalanced and small-sample datasets.
3. Proposing a computationally efficient extension to a previous method for CI estimation.
main_review: main_review: This paper tackles the "winner's curse" problem, which is especially prominent in AutoML due to the large quantity of model comparisons. They perform a large scale evaluation of 9 different CI estimation methods on multiple real data scenarios, and they introduce a new variant of a CI estimation method.

It is not clear if the computational cost of BBC really is a problem for real-world use. It is discussed how its time complexity is higher for larger datasets but it is not shown empirically.

The results from the evaluation shows that the BBC-F performs almost as well as BBC, indicating that the loss of information in BBC-F is often not too damaging. This means it can be a useful method if the practical benefits of its computational speed-up are shown.
review_summary: review_summary: This paper tackles the "winner's curse" problem, which is especially prominent in AutoML due to the large quantity of model comparisons. They perform a large scale evaluation of 9 different CI estimation methods on multiple real data scenarios, and they introduce a new variant of a CI estimation method. I think this is a good paper that could be improved further by showiing the empirical benefits of their faster method.
paper_summary: paper_summary: The paper evaluates the estimation of confidence intervals for predictive performance in AutoML, focusing on the challenge posed by the "winner's curse" - a bias introduced by selecting the best-performing model among several candidates based on cross-validation. It comprehensively assesses nine state-of-the-art CI estimation methods, including a novel variant called Bootstrap Bias Correction-Folds (BBC-F), across various datasets. BBC-F enhances the computational efficiency of the Bootstrap Bias Correction method while maintaining its statistical properties.
main_review: main_review: Pros:
- The approach is comprehensively evaluated across various CI-computation methods and datasets with varying degrees of imbalance.
- The BBC-F variant aims to improve computational efficiency without compromising accuracy.
- A thorough discussion of the implications for AutoML, highlighting the study's practical significance, is given.
- The approach's limitations are discussed thoroughly, including further steps and challenges.

Cons:
- Some multi-class challenges could be further elaborated, especially regarding the methods' limitations and the observed performance discrepancies.
- While the authors address this already, the static HPO focus of the paper is quite limited.
review_summary: review_summary: The paper introduces BBC-F, a variant to enhance BBC’s computational efficiency and work around the “winner’s curse.” The methods are applied across a spectrum of datasets to encompass multi-class, imbalanced, and small-sample scenarios, thus evaluating the robustness of each technique. The proposed approach benefits energy-efficient evaluation for uncertainty quantification. However, it is (currently) limited to static HPO scenarios and thus not applicable to Bayesian Optimization, commonly used in AutoML. In summary, the authors provide a beneficial and interesting AutoML approach. I therefore recommend accepting the paper.
paper_summary: paper_summary: The main contributions of the paper are: 
(1) introducing a variation of BBC, called BBC Fold, which works like BBC but on fold scores (averages) instead of per sample/instance;
(2) a comparison of nine methods to estimate the one-sided 95% confidence intervals for ROC AUC for 15 real-world datasets and 16 synthetic datasets;
(3) evidence that BBC outperforms all other compared methods w.r.t. inclusion percentages.
main_review: main_review: ## Positive 
* The research topic is important as the current ecosystem of the AutoML field lacks support for conformal predictions.
* The proposed method BBC-F is valuable because it is a straightforward and reasonable extension of BBC. 
* The paper is well-written and clear in most parts. 

## Negative 
* The work is missing very important related work and baselines, as highlighted above. 
* The evaluation might be unfair for the compared methods, and it's unclear if confounding factors such as overfitting or search space are the main reasons for the final results. 
* The evaluation might not hold up for the broader application in AutoML because it focuses on a too narrow set of experiments. 
* The reproducibility of the work based on the provided code is questionable.
review_summary: review_summary: I recommend the rejection of the submission at this time. 

While I find the work important and would like to see more like this, this submission's approach to studying confidence interval estimation for AutoML has several flaws. 

With the current state of the submission, I am not convinced that the proposed approach and the baseline methods are compared fairly or even more useful in the context of AutoML than existing conformal prediction frameworks.