Confidence Intervals and Simultaneous Confidence Bands Based on Deep Learning
Authors: Asaf Ben Arie
Malka Gorfine
Venue: Accepted by TMLR ()

Abstract: Deep learning models have significantly improved prediction accuracy in various fields, gaining recognition across numerous disciplines. Yet, an aspect of deep learning that remains insufficiently addressed is the assessment of prediction uncertainty. Producing reliable uncertainty estimators could be crucial in practical terms. For instance, predictions associated with a high degree of uncertainty could be sent for further evaluation. Recent works in uncertainty quantification of deep learning predictions, including Bayesian posterior credible intervals and a frequentist confidence-interval estimation,  have proven to yield either invalid  or overly conservative intervals. Furthermore, there is currently no method for quantifying uncertainty that can accommodate deep neural networks for survival (time-to-event) data that involves right-censored outcomes.  In this work, we provide a non-parametric bootstrap method that disentangles data uncertainty from the noise inherent in the adopted optimization algorithm. %, ensuring that based on deep learning estimators with small bias, the resulting point-wise confidence intervals or the  simultaneous confidence bands are accurate (i.e., valid and not overly conservative).  The validity of the proposed approach is demonstrated through an extensive simulation study, which shows that the method is accurate (i.e., valid and not overly conservative) as long as the network is sufficiently deep to ensure that the estimators provided by the deep neural network exhibit minimal bias. Otherwise, undercoverage of up to 8\% is observed.  The proposed ad-hoc method can be easily integrated into any deep neural network without interfering with the training process. The utility of the proposed approach is demonstrated through two applications: constructing simultaneous confidence bands for survival curves generated by deep neural networks dealing with right-censored survival data, and constructing a confidence interval for classification probabilities in the context of binary classification regression. Code for the data analysis and reported simulation is available at Githubsite: \url{https://github.com/Asafba123/Survival_bootstrap}.

Published in Transactions on Machine Learning Research (11/2024)
Confidence Intervals and Simultaneous Confidence Bands Based on Deep Learning
Asaf Ben Arie Department of Statistics and Operations Research Tel Aviv Universily, Isruel
aaffienarie@malltauucc..
Malka Gorfine
gorfinem Glauer. ac..
Deportment Statistics and Operations Research Tel Aviv University, Ismael
Reviewed on OpenReview: https: // openreveew. net/ pet//frum? id= XXXX
Abstract
Deep learning models have significantly improved prediction accuracy in various fields, gain- ing recognition across numerous disciplines. Yet, aspect of deep learning that remains insufficiently addressed is the assessment of prediction uncertainty. Producing reliable un- certainty estimators could be crucial in practical terms. For instance, predictions associated with high degree of uncertainty could be sent for further evaluation. Recent works in un- certainty quantification of deep learning predictions, including Bayesian posterior credible intervals and a frequentist confidence-inters or overly conservative intervals. Furthermore, there is currently no method for quantifying uncertainty that accommodate deep neural networks for survival (time-to-event) data. that involves right-censored outcomes. In this work, provide a non-parametric bootstrap method that disentangles data, uncertainty from the noise inherent in the adopted optimiza- tion algorithm. The validity of the proposed approach is demonstrated through an extensive simulation study, which shows that the method is accurate (i.c., valid and not overly con- servative) as long as the network sufficiently deep to ensure that the estimators provided by the deep neural network exhibit minimal bias. Otherwise, undercoverage of up 8% is observed. The proposed ad-hoc method can be casily integrated into any deep neural network without interfering with the training process. The utility of the proposed approach is demonstrated through two applications: constructing simultaneous confidence bands for survival curves generated by deep neural networks dealing with right-censored survival data, and constructing a confidence interval for classification probabilities in the context of binary classification regression. Code for the data analysis and reported simulation is available at Githubsite: hattps///githen com/ Asafba123 Surviivvvaa _aottalllboottrpppppppp
estimation, have proven to yield either invalid
1 Introduction
Deep neural networks (DNNs) have gained popularity due several key factors: (i) High performance DNNNs have demonstrated superior performance in various tasks, such image recognition, speech processing, and natural language understanding, often surpassing traditional statistical machine learning methods. (ii) Ability to learn complex patterns- with multiple layers of neurons, DNNs can learn intricate and hicrarchical pallerns in data, making them effective for tasks that involve complex data structures. (iii) Scalability DNNs can handle large-scale data and benefit from the availability big data and powerful computational resources. (iv) Flexibility and adaptability DNNs can be adapted to wide range of applications and domains. Ongoing research and development in neural network architectures, optimization techniques, and hardware accelerators, such GPUs and TPUs, inntin enhance the capabilities and efficiency of DNNs.

---

Published in Transactions on Machine Learning Research (11/2024)
These factors, among others, have led to the widespread use and popularity of deep neural networks in both academia and industry (LeCun et al. 2015
However, a recent survey on uncertainty in DNN (Gawlikowski et al. 2023 suggests that the real-world applications of DNNs are still restricted, primarily because of their failure to offer reliable estimates of uncertainty. Accurate uncertainty estimates are of high practical importance. For instance, predictions made with a high degree of uncertainty can be either disregarded or referred to human specialists for further evaluation [Gal & Ghahramani] 2016 Gawlikowski et al., 2023 and references therein). In this work, we present a resampling- based estimator of DNN prediction uncertainty that can be integrated with any DNN algorithm without interfering with its operation or compromising its accuracy.
1.1 Quantifying Uncertainty in Deep Learning- Related Works
Recently, there has been incrcasing attention to assossing uncortainty within DNNs. Gawlikowski et al. (2023)
provided a comprehensive overview of uncertainly estimation in neural networks, reviewing recent advances in the field and categorizing existing methods into four groups based on the number (single or multiple) and the nature (deterministic or stochastic) of the neural networks used. [Ala & Van Der Schaar (2020) noted that existing methods for uncertainty quantification predominantly rely on Bayesian neural nerwsrks (BNNs), while Bayesian credible intervals do not guarantee frequentist coverage, and approximate posterior inference undermines discriminative accuracy. They also mentioned that BNNs often require major modifications to the training procodure, and exact Bayesian inference is computationally prohibitive in practice. Moreover, Kuleshov et al. (2018) and Alaa & Van Der Schaar (2020) demonst that Bayesian uncertainly estimates often fail to capture the true data distribution. For instance, a 95% posterior credible interval generally contains the true value much less than 95% of the time. Kuleshov et al. (2018) suggested an additional calibration step, showing that it can guarantee the desired coverage rate for a variety of BNN regression algorithms.
Alaa & Van Der Schaar (2020) introduced a discriminative jackknise jakkriknnntiveejckkkniee (DJ) procedure for point-wise predictive confidence intervals, frequentist method based on Barber et al. (2021). Their approach is applicable to various DNN models, easy to implement, and can be applied in an ad-hoc fashion without interfering with model training or compromising its accuracy. The experiments in their study show that DJ is often overly conservative, with coverage rates exceeding the desired nominal level. For example, achieving a 100% coverage rate instead of the intended 90%. This conservatism, while valid, suggests that narrower intervals could be used while still maintaining the desired nominal coverage level. Obviously, using narrower confidence interval at the same confidence level is more informative and thus advantageous.
The resampling method presented in the current work will be demonstrated to provide accurate coverage rate, as long as the estimators provided by the DNN have only small bias. In such cases, our proposed approach effectively achieves the desired target coverage rate without being overly conservative. While the point-wise predictive confidence interval by Alaa & Van Der Schaar (2020) is suitable for various deep learning models, is not applicable to deep learning for survival analysis for two reasons. First, their approach relies on residuals-the difference between the actual and predicted outcomes-bie in censored survival data, the actual outcome is often unobservable. Second, they focus on confidence intervals for the outcome of a new observation, whereas in some deep learning applications, it is more relevant to estimate function of a new observation, such as the probability of new patient surviving the next len years. This limitation will be elaborated further in the following sections. Our proposed approach is suitable for various deep learning methods, including those used in survival analysis.
1.2 Survival Prediction with Deep Learning
Survival analysis involves modeling the time until a predefined event occurs. This type of analysis is common in various fields such as medicine, public health, epidemiology, engineering, and finance. Survival prediction methods are often used at some "baseline" time, such as the time of diagnosis or treatment initiation, to address questions such as, "What the probability that this individual will be alive in ten years, given their bascline characteristics"" Clinicians often use these predicted probabilities to make critical decisions about patient care and the implementation of specific therapies. In the context of credit. risk, credit scoring model
2

---

Published in Transactions on Machine Learning Research (11/2024)
involves predicting the probability that an account will default over a future time period based on number of observed features that characterize account holders or applicants.
Training datsets of survival data typically include censored or truncated observations. Censored data arise when the exact time of the event is unknown but falls within a certain period. There are several types of censoring. Right censoring: The individual known to be free of the event at a given time. Left censoring: The individual experienced the event before the study began. Interval censoring: The event occurs within specified interval. Truncation involves different schemes. Left truncation: Only individuals with event time beyond certain time are included in the sample. Right truncation: Ouly individuals who have experienced the event by specific time are included. Each type of censoring and truncation requires a specialized estimation procedure to obtain reliable risk predictions (Klein & Moeschberger 2006 Kalbfleisch & Prentice 2011). In this work, we mainly focus on right-censored data since this is the most common setting. However, our approach is general and can be adopted to any type of censoring and truncation.
The popularity of survival DNNs has been on the rise in recent years. Nearly every new development in DNN is swiftly accompanied by adaptation for survival analysis. Recent reviews on deep learning for survival analysis can be found in Hao et al. (2021); Zhong et al. [2022): Deepa & Gunavathi (2022); ; Wiegrebe ct al. 2024). To summarize, Faraggi & Simon (1995) replaced the linear component of the well-known Cox
proport ional hazards model Cox (972)) with a one-hiiden-llaer neural network. However, their approach
did not produce significant improvements over the traditional Cox model in terms of the concordance index Harrell et al. (1982). Later, Katzman et al. (2016) expanded on Faraggi & Simon (1995) 's approach by using a multilayer neural network, named DeepSurv, which achieved remarkable results in study on breast cancer. Various adaptations of the Cox model using more complex architectures have been developed for specific applications, such genomics data, clinical research, and medical imaging (Yousefi et al., 2017
Ching et al. 2018: Matsuo et al.) 2019 Haarburger et al. 2019 Li et al.) 2019). Additional DNNs methods for survival data include Ranganath et al. (2016); Chapfuwa et al. (2018): Giunchiglia et al. (2018); Lee ( Ren ct al. (2019), among others. In the comprehensive systomatic review by Wiegrobe et al.
ct al. (2024), ; it is observed that of the 61 reviewed methods, 26 are based on exlended versions of the Cox model. This includes DeepSurv by [Ktzmaa et al. (2016) and CaxTime by Kvamme et al. (2019), an efficient and flexible extension of DecpSurv that incorporates time-dependent effeets. Another popular approach is the DeepHit of Lee et al. (2018), discrete-tine deep learning survival method that avoids assumptions about the underlying stochastic process. However, all current DNN approaches for survival analysis overlook estimating the uncertainty of predictions.
1.3 Contributions
In this work, we introduce a novel resampling approach for estimating prediction uncertainty, applicable to a wide range of deep neural networks and machine learning methods. The approach is presented in a general context, with its utility demonstrated through two applications: survival analysis and binary classification regression.
Point- confidence intervals and simultaneous confidence bands are used to express the uncertainty around istimated parameters or functions, but they differ in their scope and interpretation. In survival analysis, for example, a point-wise confidence interval for the survival probability at a certain time point is only valid for that single time point. A simultancous confidence band provides an estimate for the value of a function over a range of points. Confidence bands are generally more complex to construct due to the need to account for the correlation between points over the range. A common incorrect practice is to plot point- wise confidence intervals for multiple time points and to interpret the resulting curves as a confidence band. This misinterpretation suggests, for example, 95% confidence that the area between the curves covers the true survival curve. However, it is well known that the bands obtained this way are too narrow for such an inference to be correct. To the best of our knowledge, no published works exist on confidence bands based on neural network or DNN analysis. Moreover, the currently existing methods for prediction uncertainly based on DNNs are not applicable for survival data with censoring.
In this work, we present a bootstrap approach that can be applied ad hoc to any deep neural network (DNN) for estimating uncertainty. Unlike existing methods, our approach provides: (1) Valid point-wise confidence
3

---

Published in Transactions on Machine Learning Research (11/2024)
intervals that are not overly conservative, provided the estimators generated by the DNN exhibit minimal bias. Otherwise, undercoverage may occur. (2) The first simultaneous confidence bands specifically for DNNs. (3) The first point-wise confidence intervals and confidence bands tailored for survival DNNs.
2 Bootstrap with DNN and the Proposed Bootstrap Approach
2.1 Setup
Optimization of any DNN usually involves multiple random steps, such as initial values, batch partitions, and vraining-validation data splitting (the training set is used for computing the loss function, and the validation set is used for stopping criteria in the optimization procedure). Running the same DNN on the same dataset multiple times provides highly similar results but not necessarily identical oncs. Some of the random steps, such as the step of training-validationn data splitting, can be easily controlled by the users without compromising the optimization quality, but others, such as batch partitions, cannot.
A naive application of non-parametric bootstrap for prediction uncertainty cstimation involves training multiple DNNs on different bootstrap training samples, which are random samples with replacement from the training dataset. One of the fundamental requirements for successfully applying the bootstrap procedure is to apply the exact same procedure on each bootstrap sample as done on the original training data. However, in DNN procedures, bootstrap samples will often be based on nonidentical random steps that are intrinsic parts of the DNN and cannot easily be controlled by the user or are not recommended to be fixed. Hence, the results of the bootstrap samples include not only variability due to the data but also arbitrary variability due to the use of different randomness in various steps within the optimization procedure. Therefore, such naive approach is expected to overestimate uncertainty, as will be shown below.
Let On be the DNN estimator based on n observations (training and validation), to be used for prodiction or estimation at a new test point. It is desirable for a prediction or estimation to be accompanied by a measure of error or uncertainty. In particular, we consider an interval or band around the prediction or estimation, as defined in the following examples.
Example 1: Consider a standard supervised learning setting where denotes the features, and y E y denotes the outcome. A prediction model f(;;0) is trained to predict y using the data D, {(x;, 34), i I.....n] and we get f(x: 8,). For a new test point with features x E X the outcome y is predicted by f(x; 0,). A point-wise confidence interval (also known as a prediction interval) (T(x), Uxx) of level (1 - 100%,, a E (0, 1), is defined by
X' C Rd
Pryy E T (x))) =1
(1)
where T(xx) and Uxx are functions of f(x; 0.)) and the probability is taken with respect to a new test point yx and Dn, An estimator of uncertainty in the prediction of y is expressed through the pointwise confidence interval U[x)),((x)) For any given a, it is desired to find the pair L(x) and U(x) that will be close together in some sense. For example, if Ux) - I(x), which represents the length of the confidence interval, is random, the goal is to select the pair ((x) and (x) that makes the average length of the interval smallest.
Alaa & Van Der Schaar 2020 considered this example but with a conservative definition of the point-wise confidence interval, such that the equality sign in Eq. ① is replacod with 2. Therefore, thoir proposed approach is often provides a confidence level much higher than 1- at the cost of a wider interval. Clearly, wider intervals imply less confidence, and vice versa.
In time-to-event data with right censoring, the true event times are typically unknown for all individuals. The training and validation data consists of yı = (44, di) and x; for each observation i, where & = min(tf.ca) to and & are the event and right-censoring times, respectively, and d, = 1(t, = tt)) is the event indicator. Hence, the dataset (training and validation) consists of D, = [(x,, a new test point x, the goal in survival analysis usually not to predict the event time t" based on x, but rather to provide estimator for the conditional survival function, given x, within pre-specified interval,
Example 2:
d.) (x,,,,,d))
i = 1....,n! n... For
4

---

Published in Transactions on Machine Learning Research (11/2024)
e.g. [0, T> 0. Specifically, the conditional survival function is defined by
S(s)x) Pr(t" s|x; 0) x E X , E [0, 00).
Hence, in this setting, (a(s)x, B.)) is estimated survival curve produced by the DNN (c.g., DecpHit and CoxTime) for vector of features x and E [0, Usually, t is defined not only by the desired upper value but also by the data. For example, the survival function cannot be reliably estimated beyond the last observed failure time (i.c., the maximum value of t, such that d, =1..
A point-wise confidence interval for a new test point x provides a confidence interval for the survival function at fixed time so [0, 7] and given by
Pr(S(>n|x) E (T(s0|x), (80o (+o]]))]) = a
(2)
where [(sn(x)) and (s are functions of S, (sp 2n) and the probability is taken with respect to new test point x and Dn. In many applications (Sachs et al. 2022, and references therein), it is of interest to find upper and lower curves which guarantee, with given confidence level, that the conditional survival function of a new test is covered within the curves for all in a desired interval B, B C [0, r]. Namely, we wish to find two random functions L(s(x) and (ss)x), which are functions of (s|s 0n), such that
Pr (5) E [s]xx, (s] for all s E B} 11-a. a,
(3)
and the probability is taken with respect to new test point x and D.. We call such a pair of curves L(slx) and Us)xx), s E B, a (1 a) 100% simultaneous confidence band for S(s(x), s E B.
In the next section, we present a very general resampling-based approach for constructing confidence intervals and simultancous confidence bands.
2.2 Main Idea- Ensemble- Based Bootstrap Procedure
In the following, we develop a bootstrap approach as a method of estimating the distribution of a function of DNN statistic and a new data point, denoted by [x, 0,). In examples 1 and 2 above, (x, 0,,) = f(x 0,) and (x. 0x) = Sn (5u x, 0x), respectively. To generate valid confidence intervals or simultaneous confidence bands based on DNNs, it is required to assume that the DNNs under consideration provide consistent estimators Carncy al.) 1999 Alaa & Van Der Schaar 2020). Although these assumptions do not hold in general, it is often accepted that the bias of various DNNs estimators is small and much smaller than the variance (Geman et al.. the assumptions regarding CoxTime (Kvamme et al. 2019 and classification regression with deep networks, under certain smoothness constraints, given a sufficienlly large training dataset and sufficiently deep network.
1999 The simulation study in Sections 3 and 4 supports 1992. Carney et al.
Let μ denote the true value of the desired quantity and assume that the distribution of the difference, pn - nh, contains all the information needed for assessing the precision of p.. We start by decomposing Pr into sum of two random variables Z1, Z2, such that
Zi ー n v1) and 7 ~ F2(0,
(4)
where P1 and Fg are unknown distributions with means 0 and variances us and n2, respectively, so the desired variance is
var(fn)) = 0 + 02 + 2012
where V12 co, Zı is constant given the dataset D.,, while Zu captures the added variability due to the inherited randomness of the DNN optimization procedure. Thus, Zz is not constant given Dn, and as long as 02 > 0, running the same DNN on the same dataset multiple times provides highly similar results but not necessarily identical oncs. A naive bootstrap procedure based on B bootstrap samples of the training dataset provides g,0)) and then T = Th ) 0 = 1,..., B. Similarly, each pa is decomposed into 2(0) z where the conditional distributions, given D,, are
-000 17 D,, Fi(0, vz) and ID, DD, 12)
(5)
5

---

Published in Transactions on Machine Learning Research (11/2024)
Therefore,
var 1 + _ Zi Zz)D,, = + 202 + 2012
(6)
since (2( 0, j 1,2, b ,,...,, B. a naive bootstrap variance estimator, overestimates the variance. However, even the naive bootstrap procedure provides conservative confidence intervals or simultancous confidence bands, it should not generally be considered valid approach, as includes arbitrary noise that is not part of the true uncertainty we wish to estimate. This will be demonstrated in the simulation study in Sections 3 and 4.
The extra variance, U2 in Eq. is the reason that
The above decomposition motivates the following ensemble-based bootstrap procedure for climinating the exlra variance U2:
Generate an ensemble estimator T. = M- Pn,m. where An,m = Pn (x, 6nm) and 0,,1...., Bn are M repetitions of the DNN applied the original dataset without interfering with the model training. M should be large enough such that U /M = 0. In our simulation and data analysis we found that M = 100 is sufficient.
(B) Evaluate the bootstrap distribution based on pil ..... . around P. (instead of pn)-
This approach requires B + M runs of the DNN.
A DNN training process is based on an iterative process with a convergence rule. To avoid over fitting, during training, the model is evaluated on a holdout validation dataset after each epoch, while the gradient descent procedure is applied only on the training dataset. If the performance of the model, evaluated on the validation datasct, not improving for a pre-specified number of consocutive runs (known as patience), that is, the validation loss function increases, the training process is stopped. The final model is the best performing model in terms of validation loss. The patience is often set somewhere between 10 and 100 (10 or 20 is more common, we used 15), but it depends on the dataset and network. To avoid overlap between the training and holdout validation dataset within each bootstrap sample, we first do cianing-validationn splitting and then bootstrap the training dataset.
Consider, for example, a binary classification regression where y E (0, probability p(x) Pr(y = 1|x) using a DNN estimator, P.. (x), along with the corresponding confidence interval. Specifically, we aim to satisfy
1). The objective is to estimate the
Pr (x) - p(x)) S d(x)) a
The proposed ensemble-based bootstrap variance estimator of 7hn is given by B- (ph) P. = PM (x) = -M EM=1 (x) The corresponding ensemblo-based confidence interval [L(x), Uxx] is constructed through the following steps:
7 )2 where
For each bootstrap sample b, b= 1,..., B, get dx) = Pn (x) P. (x)).
Get the 1 - a percentile of (x), b= b=1 1,.... B, denoted as dioote
3. Define I((x) = max (Px (x) - d 0} and U(x) = min (f, + (x)ott(()) 1}.
In the next subscction we demonstrate the proposed ensemble-bootstray for constructing simultancous con- fidence band for S(s(x) of Example 2.
2.3 Simultaneous Confidence Bands
We start with a simultaneous confidence bands for the survival function based on the well-known Kolmogorov- Smirnov test statistics (Mood et al. 1963). For a new x, we look for constant d(x) (constant with respect to s) such that
Pr sup <E(0rr]
(s]x; O..) - S(s)x) s d(x)) = -1aa 1aa,

---

Published in Transactions on Machine Learning Research (11/2024)
and d(x) is computed based on the ensemble-bootstrap procedure. The following is the complete algorithm for generating simultaneous confidence band of S(11x) at level of (1 1000%
1. Generate ensemble estimator, for example, S (s)x) = M Em Sn,m (8) 0 mn,m); [0, 7)
2. For any bootstrap sample b, b = 1,...; , B, get Sh (s 0% -ih (x, 0n [0, and
(x) max
((b) (s| x; 0,11) - S (s|x)
+E(0,r]
3. Get the 1 - a percentile of d
(x) (x)..... ((B)(x)) (x), denoted by de (x).
4. For any 8 E [0, we define
L (s(x) max (n(s) 0m) - (x), 0]
and
(s(x) min ( (s)xx)) + de 1}
The above KS and UKS provide fixed-width band. The following weight-based based bands have varied widths function of E B, and are expected to provide narrower bands. Specifically, for new x, now look for (x)) such that
(s|x; 0.)) - S(s)x)) S <d((x) =1- a,
Pr
sup .E(0.]] W (s
where
(=);; -x) (x((x;; 0m))]
and (s) 0x) is defined similarly to (s): x; 0x) but with values truncated to the range of 0.01 to 0.99. Hence, the width of the confidence band of (s) based on the above, is proportional to the "approximated" (x)x;;0))) The use of truncated survival estimator
standard error of its estimator, (s|x; (1) S (s|x; 0%) in the denominator in order to avoid instabilities in cases where (s(:xx0)) is too close to either or 0. The following is the modified algorithm:
1. Generate an ensemble estimator, for example, 5 (8) = M- Em (s|x; On, [0,
2. For any bootstrap sample b, b = 1,..., B, get
((() (s|x; 0 [0, T0, and
2 (s|x: 0
S. (s|x)
(x) = max
sE(urr]
:
(b) (s x; ;
Get the 1 - a percentile of dr:(1) (x). d' (B) denoted by df boote (x).
4. For any 8 E [), we define
[s)op)) max (s/x: (s|x: on Japotts (xx) 0}
and
(spx) = min (s(x: 0..) (s|x: Dn (x)x 13
Through a comprehensive sinulation study and benchmark datasets, we show in the next sections that confidence bands based on the naive bootstrap approach are overly conservative, resulting in wide confidence bands. In contrast, with deep networks and sufficiently large training datasets, the proposed approach provides the desired confidence level with narrower confidence widths compared the naive bootstrap approach. Moreover, the weight-based bands, prrop and Uprop, are narrower than those of 7, T,KS and UKS
7

---

Published in Transactions on Machine Learning Research (11/2024)
3 Simulation Study Survival Analysis
We illustrate the performance of the proposed ensemble-based approach by constructing confidence bands for survival curves, specifically focusing on the CoxTime DNN (Kvamme et al. 2019). When comparing CoxTime with DeepSurv (Katzman et al. 2016), the two leading DNNs methods for survival analysis, we observe that DeepSurv often shows substantial bias estimating S(: x) for new observations, whereas CoxTime shows substantially less bias.
CoxTimm is highly efficient in terms of runtime. One of its effective shortcuts cleverly adopts concepts from nested casc-control designs, utilizing small random subsample of the risk set, referred to controls, for each loss function evaluation at each observed failure time, instead of the entire risk set. Moreover, the controls are resampled at each epoch. Users can specify the number of controls. While the authors suggested that even a very small number of controls, as few as one, is sufficient, it will be demonstrated here that the number of controls can substantially inffuence the performance of confidence bands.
3.1 Data Generation and Measures of Performances
We evaluated five different settings, each with multiple repetitions. The extensive number of repetitions for each setting generated a significant amount of data. To save space, the results were stored and summarized using a grid of points T defined for each sctting. The hazard functions considered in the simulation study are a general form
h(t[x) = ho(t) ext])(x x)},
with the functions ho and g based on Kvamune et al. (2019) (Settings 1 3) and Zhong el al. 2022 (Settings 4 4-:):
Setting 1 Linear proportional hazards (PH) model: ho(t) = 0.1, g(t,x) g(x) = -8x with B = (0.44, 0.66, 0.88). Each covariate 2j, j = 1,2,3, was uniformly sampled from U[-1, Consor- ing times were generated from an exponential distribution with parameter 1/30, and individuals still at risk time t = 30 were censored that time, resulting approximately a 30% censoring rate. = [0. with jumps of 0.1.
Setting 2 Non linear PH model: here
g(x) 5"x x2x32/3 + = 21 + FiTs + T243)
and the rest follows Setting This resulted in approximately 20% censoring rate.
Setting 3 Non linear and non PH model: (t) = 0.02, x(t,)) = a(x) b(x)t, a(x) x1 + 2 + x243) + I3, b(x) = {0.2(zro + +1) + 0.5x0z1 and the rest follows Setting 1. This resulted in approximately 30% censoring rate.
+ az
Sctting& 4 Non linear PH model: ho(t) 0.1f and
g(x) = 27 log(æ" + 1) + VayTs exp(xs/2) 8.2
were the covariates generated from a Gaussian copula on [0, 2] and correlation parameter 0.5. Right- censoring times were generated from an exponential distribution with parameter 1/28, and individ- uals still al risk at lime & = 34 were censored at that time. This resulted approximately a 50% censoring rate. T [2, 34] with jumps of 0.1.
Setting 5 Non linear PH model: here g(x) is replaced by
g(x) و log(xs 1) VzTTs + + (2/m///)/)//0 6.0,
and the censoring parameter equals 1/45. The rest follows Setting This resulted in approximately 60% censoring rate.
8

---

Published in Transactions on Machine Learning Research (11/2024)
Some of the above settings differ from the source references because CoxTime requires smoothness in g and its derivation with respect to x. In Settings 4 and 5, the time grid starts at 2 because these settings sometimes produce extreme samples with event times as small as 10-4. In these unrealistic cases, the survival probability drops dramatically from 1 to 0, and the time grid was truncated to exclude such cases.
The studied sample size of the training plus validation data range from n 000 to 10,000, with an 80%-20% split between training and validation. The number of controls varies, being 1, 2, 4, and 8. Each configuration, with M 100 and B 200, is repeated R 100 times. For each repetition r, survival estimates were evaluated curves and confidence bands for cach test point X,, i = 1,..., 1000. Subsequently, for each test point i, we assessed the proportion of times the 100 confidence bands covered the entire true survival curve, based on the adopted grid T. The final reported empirical coverage rate is the average proportion across the Reest = 1000 individuals. Additionally, to compare different methods, we also examine (half of) the confidence band width, defined by
1,.... R, test set of size Reasst 1000. This resulted in R = 100 estimated
[(U());)) - L((x)))
2nheed
i=1 TET
where [T] is the cardinality of T.
3.2 Additional Computational Aspects
The analysis was conducted using the coxtime package in Python with Kaiming initialization. We employed dropout rate of 0.1 and a batch size of 1000. The learning rate was dynamically determined using the Irfinder method, and the Adam optimizer was used for optimization. The networks were standard multi- layer DNNs with ReLU activation and batch normalization between layers. In coxtime, transformers is utilized, and observed failure times were standardized to have zero mean and a variance of one, which is necessary due to the inclusion of I in function We implemented batch normalization with respect to x. The training was conducted for 1500 epochs. To ensure the effectiveness of the proposed ensemble- bootstrap method, it essential to minimize estimator bias (a standard requirement). Accordingly, for complex settings, we explored various layer configurations, demonstrating that deeper networks with wider layers are necessary in these cases.
For the proposed ensemble-based bootstrap method, each dataset with training and validation observations had xt), i == 1....,,, estimated M times, resulting in (t, m = 1....,M, We then computed 9h (t, 3 >m mn (t, (4) to be used in the Breslow estimator of Ho(() = f ho(s)ds. In particular, define the cumulative hazard H([xx) = - ho(s) exp(g(s, dd,,s, and its estimator is given by
In,
(t(x)) Σ exp((t) (t,,))
4Stt
where
AHo(t.) = Σj I(4 2 4,) ( (li, xú)}
Subsequently, the ensemble-based survival function estimator is given by 3.M (1[x) = -xx)(-MMM
3.3
Results
The simulation results are summarized in Figures | and include empirical coverage rates and widths of the simultancous confidence bands of the naive bootstrap method, as well as for the two ensemble-based approaches, one based on Kolmogorov-Smirnov (KS) statistics, LKS UKS and its modification Lrrop, Uprop,
Figures I prddpreeent the empirical coverage rates and widths for different sizes of controls with n = 10, 000. The results indicate that the naive bootstrap approach consistently exhibits significant over-coverage across all settings. In contrast, our ensemble approach achieves coverage rates that are reasonably close to the nominal level. Additionally, as the number of controls increases, the coverage rates of the ensemble-based
9

---

Published in Transactions on Machine Learning Research (11/2024)
method improve and their confidence bands become narrower. While larger control sizes might yield better results, they would also be more computationally expensive. Setting 3 demonstrates high sensitivity to the number of layers, with 6 layers and 8 controls providing good performance in terms of coverage rates. Coverage rates in Settings and also improve with an increased number of controls. Our findings suggest that more layers might. be required to achieve the "small bias" claim of DNNs survival estimators based on CoxTime of Kvamme et al. (2019). Even though the naive bootstrap procedure can yield over-coverage cases with highly biased DNN estimates (e.g., Setting 3 with only 4 layers), it should not be regarded as valid approach. The over-coverage simply indicates that the additional arbitrary noise in the naive bootstrap was large enough to offset the bias.
Figures 3 and 1 present the empirical coverage rates and widths for different sizes of controls with n 1000, 1500 and 2000. Figure 3 reveals that even under small sample sizes, the naive bootstrap approach remains overly conservative, whereas the proposed aisemble-basod bootstrap methods perform well in terms of coverage rates. Figure f shows that as n increases, the width decreases as expectod, and the conservative nature of the naive bootstrap leads to substantially wider confidence bands. Notably, the proportional- KS method consistently produces narrower simultaneous confidence bands while maintaining coverage rates similar to those of the KS method.
Clearly, the ensemble estimator 3: exhibits reduced bias and variance compared to that of S, (results not shown). However, the computational impracticality of constructing a confidence band for the ensemble survival estimator 5M persists using the proposed ensemble-based method.
4 Simulation Study Binary Classification
Here present the results of smaller simulation study focused on binary classification regression. The outcome, Vi E 1), follows the probability model
exp(g(x)}
Pr(y= = 1[x) 1 exp(g(x))
where
g(x) -5 4x, + 4x0 + 201 + 2zz + 2sin(T)44ss) + Σ (215 0.25(j 1))
The variables (21, were sampled from a bivariate normal distribution with expectation vector (0, (0,0), unit variances, and a correlation of 0.65. 2z was uniform sampled from U[-1, 1], I4 and 23 were uniformly sampled from (0, 1}, and .e....,211 were sampled independently from the standard normal distribution. The objective is to estimate the probability p(x) = Pr(y = 1|x) using a DNN estimator along with the corresponding confidence interval based on Eq. 7 and the subsequent algorithm.
We conducted the simulations with sample sizes of n 1000, 2000 and 5000, splitting the data into 80% for training and 20% for validation. Each configuration, with M = 100 and B = 500, is repeated R = 100 times. The test set comprised 1000 observations. The results, shown in Figure 5 were obtained using DNN with 5, and 9 fully connected hidden layers, each consisting of 128 nodes and utilizing ReLU activation. The output layer consisted of a single neuron with sigmoid activation, and the model was trained using logistic loss. The implementation was done in Keras.
It is apparent that the naive bootstrap approach yields overly conservative coverage rates with unnecessarily wide intervals and the coverage rates are not improved the number of layers increases. In contrast, the coverage rates of the proposed method are close to the nominal rates and the reduction in confidence interval width achieved by our approach is substantial. Although the proposed method with n = 1000 and five layers shows slight undercoverage (e.g., 0.867 instead of 0.90), the coverage rate improves as the number of layers increases (c.g., 0.892 with nine layers). These results align with those observed Sctting 3 of Scction 3.1, where the complex function g appears sensitive to the number of layers, and the bias of the DNN estimates tend to decrease substantially the number of layers increases. Once again, the over-coverage observed in
10

---

Published in Transactions on Machine Learning Research (11/2024)
Swtting
Sellin
Saling 3 (4 lapwa)
cc Prrmis
cantibs
cortul onnea
cantob
Excrtza
3r0p
Setting 31 mynrg
Seeting
Settin 5
1.00
commrll
carnd
L.00
OUT 2.57
2.000
0.50
013
cracnbic
meted
nebic
metol
(a) Empirical Coverage Rates 90% Simultancous Confidence Band
Sctting
Setting 2
Sotting 3 layors)
LO2
Rizesoi
L.00
of hpss
0:203
Locyr
0.55
0230uu (100)
0803 gs
0.55
0.91
1000
0122
0,502
0,00
010
0.59
0.00
0.05
0.50
Ansertle
ACOAESAA popar
1.80
AYAATER prevetions
0.00
Pesvedte wwpp
(b) Empirical Coverage Rates of 95% Simultaneous Confidence Band
Figure 1: Simulation results of empirical coverage rates, Settings with n = 10, 000.
11

---

Published in Transactions on Machine Learning Research (11/2024)
Setting
Setimy??
Saling 3 lapea)
cc Pormms
cantub
cortul coneas
contob
caped Preems
bob
Excrtza
0..0
0.00
0.0
0.20
0.2
0381
0009
0.79
0965 LEA U00600
Serting 31 myyrg
Seeting
Settin 5
commni
carrd
of
8.15
2.00
0.12
0121
376,000
0.00
230
01.79
L000
201
cracntic
meted
metot
rafed
(a) Empirical Width 90% Simultancous Confidence Band
Sctting
Setting
Sotting 3 layors)
Rizesok
ralnd carrdds
Vodh Cr
0.15
11/2
125
L.15
115
0.104
0.00
0.12
0:10
5,5
0.h6
0.25
0:5
0.75
0.20
Whit Cenitro Dard
012
0331
0352
0.1A
0.175
BEW
0.00
aesertle
ACOAESA popar
1.00
INVARTE prrpattonn
0.00
popprross
rechud
(b) Empirical Width of 95% Simultaneous Confidence Band
Figure& 2: Simulation results of empirical width, Settings 5 with n 10, 000.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Switing 1000
Seriig 1500
Seling 2010 sumples
ccnital Prrmms
cancob
cond
Fremas
0100
COC4
0533
0110
0,08
01415
0,00
3000
Senting
1000 somples
Seriig 1500 somppess
Serting 2 2000 camples
1.00
L.00
6:900.00
0.9000.00
0S
(164)
LUTE
crocmblc
metol
(a) Empirical Coverage Rates 90% Simultaneous Confidence Band
Seding
1000 samples
Scrang 1 1500 sampies
Setting 1 2000 sample
LO2
co
cared
1.00
0222
0.29
0,00
0,902
D1ST
0008
US07
0.05
1100
0230140
1295 2000 p050
0000
5.9
2:91
0.55
0.5
0.05
0.00
ANAAEAA 3 yopar
2.00
AYATEE pryvations
0.00
reoveato www.trtooooooooo..
(b) Empirical Coverage Rates of 95% Simultancous Confidence Band
Figure 3: Simulation results of empirical coverage rates, Settings 1 5 with varied size of training and vali- dation dataset, n.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Switing
1000
Sering 1500
Seling
2010 samples
cccitall Prrmms
contool coneva
cancob
carnd
Eccrtza
Fremas
0.39
0.25
022
0.55
Veche Collirec
Emn
1000
+177
0172
L154
030
C127
0.5
020
005
Senting
1000 somples
Seriin 1900 somppes
Serting 2 2000 samples
0.99
carrd
0.30
5:30
0000
Vidi of Of
L1354
0.50
2.2
122
0333
0144
0112
0.50
0.32
0.10
cractiic metidd
cracnbic
mettt
rafed
(a) Empirical Mean Width of 90% Simultaneous Confidence Band
Seding
1000 samples
Scrang 1 1500 sampies
Setting 1 2000 sample
0.99
coatrol
calee carrdds
0.50
0:30
0.00
[12%
0222
0,211
Vhdta Chekkerre
0135
D1:0
9.50
0.00
0.22
0:00
6.709
0.25
0.20
5:20
0.20
122
11:45
0.22
0.0
1.00
AYAATED prypotions
0.00
poqerros
(b) Empirical Moan Width of 95% Simultancous Confidence Baud
Figure 4: Simulation results of empirical mean width, Settings 1 5 with varied size of training and validation dataset, n.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Card dence balene
200 2000 500
0.0
Coordi: Flym
Plyccs
Stuttr
(a) Empirical Coverage Rates of 90% (left) and 95% (right) Confidence Intervals
Iatarol
9hl Card dence Ielendl
200
2000
0.00
3.00
0,222
C112
2200
0.00
130
Deneel
(b) Empirical Mean Width of 90% (left.) and 95% (right) Confidence Intervals
Figure 5: Simulation results of binary classification with varied size of training and validation dataset, n, and varied number of layers.
the naive bootstrap approach suggests that the added arbitrary noise is sufficient to offset the bias; however, this does not indicate the validity of the approach.
5 Experiments
In this section, we analyze four commonly used survival datasets to demonstrate the utility of our proposed approach. These datasets were introduced and used by Katzman et al. (2016) and Kvamme et al. (2019) among others, and are available through PyCox. Here are some details of the datasets:
SUPPORT: Study to Understand Prognoses Preferences Outcomes and Risks of Treatment. This dataset. includes 8,873 observations, 14 covariates, and a censoring rate of 0.32.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: Hyperparameters search space for SUPPORT, METABRIC, Rot. GBSG and FLCHAIN dataset
Hyperparameter
Values
Learning rate
10-3
Patience
10
Controls
8
Hidden layers
{1, 2, 4]
Layer width
{32, 64, 128, 256)
Dropout
{0, 0.1, 0.2) (256, 1024)
Batch size
Table 2: Mean width of simultaneous confidence bands for the naive and ensemble-based boolstrap methods.
Naive
Ensemble- M 100, B = 200
KS
Prop KS
Dalaset
90%
95%
90%
95%
90%
95%
SUPPORT
0.146
0.172
0.137
0.161
0.134
0.161
METABRIC
0.167
0.191
0.147
0.171
0.140
0.167
Rot& GBSG
0.135
0.158
0.121
0.144
0.115
0.150
FLCHAIN
0.075
0.082
0.044
0.055
0.036
0.044
METABRIC: Molecular Taxonomy of Breast Cancer International Consortium. This dataset con- tains 1,904 observations, 9 covariates, and censoring rate 0.42.
Rot. & GBSG: Rotterdam tumor bank and German Breast Cancer Study Group. This dataset consists of 2,323 obscrvations with 7 covariates and censoring rate of 0.43.
FLCHAIN: Assay of Serun Free Light Chain. This dataset includes 6,524 observations with 8 covariates and censoring rate of 0.70.
Table 2 in Kvamme et al. (2019) shows that CoxTime and DeepSurv are the top two methods based on the concordance measure when analyzing these datasets. Tn the current analysis, pre-encoded categorical variables were used as-is, while numerical variables were standardized following Kvamme et al. (2019) 's recommendations. Hyperparameters luning was conducted using voossvaliiattion. A grid search over the hyperparameters search space, as detailed in Table [ was performed by splitting the data into 10 folds for cach configuration and scoring the C-index on the held-out set. The set of hyperparamcters with the highest average C-index was selected. The following are the selected hyperparameters for each dataset: SUPPORT hidden layer, layer width 256, dropout 0.3 and batch size of 256; METABRIC: 2 hidden layer, layer width of 256, dropout 0.3 and batch size of 1024; RGBS&GBBG- hidden layer, layer width of 128, dropout 0.1 and batch size of 256; and FLCHAIN- hidden layer, layer width of 256, dropout 0.1 and batch size of 256.
The results of the confidence bands analysis, summarized in Table 2 are based on 10-fold cross-validation. The held-out fold serves as the test set, while the remaining data were split 80%-20% into training and validation sets. The total computation time required for each dataset, including the selection of hyperpa- rameters, as follows: SUPPORT 210 minutes, METABRIC 66 minutes, Rot.& GBSG 52 minutes, and FLCHAIN 98 minutes. Examples of survival curves along with their confidence bands can be found in Figure As anticipated, the widths of the ensemble-be based methods are shorter than those of the naive bootstrap, with the proportional-KS method generally having the narrowest widths.
6 Theoretical Aspects
The following discussion divided into two parts: (i) The motivation for the proposed bootstrap method for estimating the uncertainty of fon based on the assumption that 7h is consistent estimator of μºt and
16

---

Published in Transactions on Machine Learning Research (11/2024)
1.00
0.75
SUPPORT
0.50
0.25
0.00
250
500
750
250
500
750
250
500
750
1.00
METABRIC
0.50
0.25
0.00
150
200
100
150
100
200
1.00
0.75
Rot. & GBSG
0.50
0.25
0.00
40
60
20
20
1.00
0.75
FLCHAIN
0.50
0.25
0.00
0
1000
2000
3000
4000
1000
2000
3000
4000
1000
2000
3000
4000
Estimated Survival Curve
Ensemble-Based
Ensemble-Based Proportional
Naive
Figure 6: Examples of survival curves and simultaneous confidence bands produced by the naive and ensemble-based bootstrap methods.
17

---

Published in Transactions on Machine Learning Research (11/2024)
[b]] the decomposition of Ph and In as defined in (4) and
(5) hold true. (ii) A discussion on existing deep
neural networks that produce consistent estimators.
Assume that M is sufficiently large such that the uncertainty of the ensemble estimator due to the inherent randomness of the DNN optimization process, is zero; that is, var (f**) = on and var (TM Da)) = 0. For simplicity, we focus on the asymptotic variance, and similar arguments can be used for other statistical functionals. We write,
var(pn)
= E ( - -pM + t =1)²}}
= E (m - -x:a)) )² + FF| - + +2E ( ((in - ) (T - [10}}
-100)
= E' { (Z1 + Z2 - =1)²} + Ei(Z1 -1'))
+ 2E ((2 + U2 - 21)(21 - =1²)}
= var (Zz) + var (Z1) + 2cor (Zz,
vi + 02 + 2012
The third equality holds based on the main concept in Section 2.2, which decomposes Ph into Z1 + Zz. where Zi is fixed given Dn, and Ze represents the additional variability introduced by the DNN optimization process. While the variance of Z2 is not zero given Da, the ensemble estimator, with sufficiently large M, ensures that Z₃ =. 0.
In scenario with unlimited computational resources, one would idcally base predictions or estimations T instead of T.. The corresponding bootstrap samples, denoted by (b) b = 1,..., B, could then be (2M(() (D.) U1. However, since ph, is used in practice, we now
used for uncertainty estimation, with var (Im
turn our attention to the recommended approach for variance estimation, namely,
E (m!h) pb)) [Dn] E
(nl Pe M())
+ (MM
)
PM)) IDn,
E (mll) pMM (Dn
{ -pal) IDn
E
+2E { (mit) mmn p((l(i))
(nib((t)
(in
14 |Dm]]
E (2-0) + 2 z 2 IDm²
} E -21" - [Dn] } 22ov(21)) 22 (D,)
= 01 + U22 2012
The third equality follows from similar reasoning as before, specifically by decomposing ph into - and _ For (b)(()) (which does not need to be computed in practice), -,5² 0. The last equality assumes that var(Zı) var (M U1, var(Zz) assumptions are based on a common principle (or proven result) in bootstrap methods, often referred to
( 22 and cov(Z1, Z2) cov(z("), -!a²) U12 These
as bootstrap distribution consistency. In our setting, this principle asserts that the (asymptotic) distri- bution of a(n) (Tmn - -4)) is equal to the (asymptotic) conditional distribution of a(n) (pl T given Day where a(n) is typically vū. It is important to note that, due to the DNN optimization error, we use a(n) Tun - 7M instead of a(n) (nl - -m.)
Under mild regularity conditions, and using a DNN whose complexity grows with the data, often used in Saw.ointt.sSsssh-i...i /Schmidt-Hicber (2020) established consistency and convergence rate of the nonpara- metric function estimator in the regression model Y = g(x) + e where g is an unknown function and e is a random noise. Recently, Zhong et al. (2022) extended these results to the partially linear Cox mode of the form (t) cxT z + g(x)). study, but do not cover Setting 3 with g(t, x), although our numerical results are excellent when the number of layers is large enough.
practical applications,
The work of [Zhong et al. (2022) covers Settings 1,2,4 and of our numerical
18

---

Published in Transactions on Machine Learning Research (11/2024)
7 Concluding Remarks
In this paper, we introduced bootstrap methodology to estimate uncertainty in predictions estimations based on DNN. The core concept involves an ensemble approach to separate data uncertainty from the noise inherent in the optimization algorithm. Our method is general and can be applied to any DNN analysis without interfering with or compromising the DNN predictions themselves.
We demonstrated the utility of our general approach in the context of survival analysis, where the primary focus is estimating the survival function based on new set of features and in the context of binary classification. Our comprehensive simulation study indicates that the proposed method is valid and not overly conservative, as long the est.imators provided by the DNN have only small bias. Otherwise, undercoverage of up to 8% is observed. Often, bias reduction in DNN can be achieved by increasing the number of layers and nodes. Future research should aim to refine the proposed approach to reduce its computational complexity and perform a comprehensive theoretical analysis to rigorously establish the (asymptotic) accuracy of the coverage probability across diverse conditions. Code for the data analysis and reported simulation is avalable at Githubsite:
Acknowledgments
We thank the anonymous reviewers for helpful discussions and feedback. The work was supported by the Israel Science Foundation (ISF) grant number 767/21 and by a grant from the Tel-Aviv University Center for AT and Data Science (TAD).
References
Ahmed Alaa and Mihaela Van Der Schaar. Discriminative jackknife: Quantifying uncertainty deep learning via higher-order influence functions. In International Conference on Machine Learning, pp. 165-174, 2020.
Rina Foygel Barber, Emmanuel J Candès, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife+t. The Annals of Statistics, 49(1), 2021.
John G Carney, Pádraig Cunningham, and Umesh Bhagwan. Confidence and prediction intervals for neural network ensembles. In IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cot. No. 99CH36939), volume 2, pp. 1215-1218. IEEE, 1999.
Paidamoyo Chapfuwa, Chenyang Tao, Chunyuan Li, Courtney Page, Benjamin Goldstein, Lawrence Carin Duke, and Ricardo Henao. Adversarial time-to-event modeling. In International Conference on Machine Learning, pp. 735 744. PMLR, 2018.
Travers Ching, Xun Zhu, and Lana X Garmire. Cox-nnet: artificial neural network method for prognosis prediction of high-throughput omics data. PLoS computational biology, 14(4)=e1000066, 2018.
David Cox. Regression models and lables. Journal of the Royal Statistical Society: Series (Method- ological), 34(2):187 202, 1972.
P Deepa and C Gunavathi. A systematic review machine learning and deep learning techniques in cancer survival prediction. Progress in Biophysics and Moleculur Biology, 174:62-77 71, 2022.
David Faraggi and Richard Simon. neural network model for survival data. Statistics in medicine, 14(1): 73-82, 1995.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Jakob Gawlikowski, Cedrique Rovile Njicutchou Tassi, Mohsin Ali, Jongscok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, al. A survey of uncertainty in deep neural networks. Artificial Intelligence Reviem, 56(Suppl 1):1513-1588 2023.
19

---

Published in Transactions on Machine Learning Research (11/2024)
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58 1992.
Eleonora Ghunchiglia, Anton Nemchenko, and Mihaela van der Schaar. Run-surv: A deep recurrent model for survival analysis. In Artificial Neural Networks and Machine Learning- ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27, pp. 23 32. Springer, 2018.
Christoph Haarburger, Philippe Weitz, Oliver Rippel, and Dorit Merhof. image-based survival prediction for lung cancer patients using cmns. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 1197-1201. IEEE, 2019.
Lin Hao, Juncheol Kim, Sookhee Kwon, and II Do Ha. Deep learning-bee based survival analysis for high- dimensional survival data. Mathematics, 9(1)::1244 2021.
Frank E Harrell, Robert M Califf, David B Pryor, Kerry L Lee, and Robert A Rosati. Evaluating the yield of medical tests. Jama, 247(18):2543 2546, 1982.
John D Kalbfleisch and Ross L Prentice. The statistical analysis of faiture time data. John Wiley & Sons, 2011.
Jared L Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deep survival: A deep cox proportional hazards network. stat, 1050(2):1- 10, 2016.
John P Klein and Melvin Moeschberger. Survival onalysis: techniques for censored and truncated data. Springer Science Business Media, 2006.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning, pp. 2796 2804. PMLR, 2018.
Hávard Kvamme, Ørmulf Borgan, and Ida Scheel. Time-to-event prediction with neural networks and cox regression. Journal of machine learning research, 20(129):1-30, 2019.
Yann LeCun, Yoshua Bengio, and Geolfrey Hinton. Deep learning, nature, 521((553)::38 444, 2015.
Changhee Lee, William Zame, Jinsung Yoon, and Mihaela Van Der Schaar. Doephit: A deep learning approach to survival analysis with competing risks. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Hongming Li, Pamela Boimel, James Sanopaul-Naylen Haoyu Zhong, Ying Xiao, Edgar Ben-Josef, and Yong Fan. Deep convolutional neural networks for imaging data based survival analysis of rectal cancer. In 2019 IEEE 16th International Symposium on Biomedical Imoging (ISBI 2019), pp. 846-86 849. IEEE, 2019.
Koji Matsuo, Sanjay Purushotham, Bo Jiang, Rachel S Mandelbaum, Tsuyashi Takiuchi, Yan Liu, and Lynda Roman. Survival outcome prediction in cervical cancer: Cox models vs deep-learning model. American journal of obstetrics and gynecology, 220(4):381-c,, 2019.
AM Mood, FA Graybill, and DC BOES. Introduction to the theory of statistics. me-graw hill book company. Inc., New York, 1963.
Rajesh Ranganath, Adler Perotte, Noémie Elhadad, and David Blei. Deep survival analysis. In Machine Learning for Healthcare Conference, pp. 101 114. PMLR, 2016.
Kan Ren, Jiarui Qin, Lei Zheng, Zhengyu Yang, Weinan Zhang, Lin Qiu, and Yong Yu. Deep recurrent survival analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4798 4805, 2019.
Michael C Sachs, Adam Brand, and Erin E Gabriel. Confidence bands in survival analysis. British Journal of Cancer, 127(9):1636 16111611,1 2022.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with ReLU activation function. The Annals of Stalisties, 48(4):1875 1897, 2020.
Simon Wiegrebe, Philipp Kopper, Raphael Sonabend, Bernd Bischl, and Andreas Bender. Deep learning for survival analysis: review. Artificial Intelligence Review, 57(3):65, 2024.
Safoora Yousefi, Fatemeh Amrollahi, Mohamed Amgad, Chengliang Dong, Joshua Lewis, Congzheng Song, David Gutman, Sameer Halani, Jose Enrique Velazquez Vega, Damiel Brat, et al. Predicting clinical outcomes from large scale cancer genomic profiles with deep survival models. Scientific reports, 7(1):1-11, 2017.
Qixian Zhong, Jonas Mueller, and Jane-Ling Wang. Deep learning for the partially linear cox model. The Annals of Statistics, 50(3):1348 1375, 2022.
21


Reviews:
[{"review_id": "l5SrZ5Puho", "reviewer": "Reviewer_R19s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The approach is well motivated and the problem of variability due to randomness in training as a cause of overly conservative confidence intervals does not appear to have been considered before.\n\n2. The approach empirically outperforms the baselines in terms of more precise coverage with narrower confidence intervals for a range of simulation settings and real-world datasets.\n\nWeaknesses:\n\n1. The proposed approach increases the computational cost of inference since it requires at least M extra DNN runs (due to ensembling). However, the authors have acknowledged this and recommended that future work should look into it and so I don't think it is a major concern at this stage.\n\n2. While the authors claim to provide exactly 1-$\\alpha$ coverage in Section 2.3 there is no proof for this. Moreover, results in Section 3 show that the coverage is never exactly 1 -$\\alpha$ (corresponding to 90% in Fig 1.a and 95% in Fig 1.b). Thus, I feel that it should be clearly stated in section 2.3 that such exact coverage cannot be guaranteed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new approach for uncertainty quantification in deep learning using a non-parametric bootstrap approach where the bootstrap distribution is evaluated around an ensemble estimator of the prediction at a point to eliminate the uncertainty due to the training procedure. The estimated confidence intervals can also be augmented with a variable width component based on an estimate of the standard error of the predictor which. The authors evaluate the approach specifically for the case of survival analysis where the data is typically censored which adds an additional challenge to the task of uncertainty quantification. The proposed estimators generally give good coverage rates with narrower intervals than the baseline bootstrap estimators.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Why do you say that current methods for prediction uncertainty are not applicable for survival data with censoring (last line of second paragraph in Section 1.3)?\n\n2. Is it wrong to use the same initialization, hyperparameters and validation set for each bootstrap sample? If not, then wouldn't that eliminate the variability due to the training process?\n\n3. Keeping aside the concern around computational cost, why not apply M repetitions i.e. ensembling to the bootstrap estimates also? Would that not further reduce the uncertainty?\n\n4. Why are the plots for settings 3-5 missing in Fig 3 and 4?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "abPaYpaINZ", "reviewer": "Reviewer_m1E5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1.\tThe paper introduces a novel bootstrap method that effectively separates data uncertainty from algorithmic noise, offering a significant advancement in the field of uncertainty quantification for deep learning models.\n\n2.\tThe proposed method is designed to be easily integrated into existing DNN frameworks without requiring changes to the training process, making it highly practical.\n\n3.\tThe method ensures that the produced confidence intervals and bands are accurate and not overly conservative, providing more reliable uncertainty estimates compared to existing approaches.\n\n4.\tThe authors provide extensive validation through simulation studies and application to real-world survival data, demonstrating the robustness and effectiveness of their approach.\n\n**Weaknesses**\n\n1.\tThe ensemble-based bootstrap method requires running multiple DNNs, which can be computationally expensive and time-consuming, potentially limiting its scalability to extremely large datasets.\n\n2.\tThe validity of the proposed method relies on certain assumptions regarding DNN estimators, which may not always hold in practice, affecting the generalizability of the results.\n\n3.\tAlthough the method is theoretically applicable to various DNNs, the paper predominantly focuses on survival analysis, with limited demonstration across other types of deep learning applications.\n\n4.\tThe empirical evaluation, though extensive, should be expanded by including a wider range of DNN architectures to further validate the method's versatility and effectiveness across different scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper  focuses on addressing the challenge of predicting uncertainty in deep learning models, particularly for survival data. The authors propose a non-parametric bootstrap method to construct valid, non-overly conservative point-wise confidence intervals along with simultaneous confidence bands. This method can be integrated into any deep neural network (DNN) without affecting the training process. The paper demonstrates the practicality of this approach using survival curves derived from DNNs handling right-censored survival data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "First, the evaluation should be expanded across diverse DNN architectures to demonstrate the method's generalizability beyond survival analysis, which is crucial for broader applicability. \n\nSecond, the discussion on the underlying assumptions of the proposed method needs to be strengthened to clarify how these assumptions impact the validity of the results, ensuring that readers understand the method's limitations. \n\nLast, optimizing the computational complexity of the method is essential, as the current high computational cost may limit its practical use, especially in large-scale applications.", "broader_impact_concerns": "The paper's work on uncertainty quantification in deep learning, particularly for survival analysis, raises numerous ethical concerns. These include potential misuse in critical fields such as medicine, where inaccuracies could affect patient care, as well as the risk of biased predictions unfairly impacting certain groups.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kahvTVbQf4", "reviewer": "Reviewer_AGt3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n+ Well-written paper.\n+ The advantage of the proposed methods is supported by experiments.\n+ Experiments highlight the usefulness of the proposed methods in practical settings.\n\n**Weaknesses**\n\n- My main concern is the lack of theoretical analysis; see the requested changes.\n- Numerical experiments show cases where the lower bound is too liberal.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a bootstrap approach for constructing prediction intervals, specifically focusing on survival analysis.\n\nThe authors offer two new bootstrap variants, which are shown empirically to be less conservative than the naive one. In particular, the truncated variant is designed to avoid instabilities involved in the normalization term.\n\nNumerical experiments support the approximate validity of the proposed methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I strongly encourage the authors to support their methods with theoretical analysis for the coverage guarantee. This is of great importance given the high-risk nature of the problems of interest. Notably, the theoretical analysis can even be done for special cases and under simplifying assumptions. The goal here is to shed light on the correctness of the proposed algorithms, as a way to further support the empirical results.", "broader_impact_concerns": "I don't see an issue here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel non-parametric bootstrap method for computing confidence intervals and simultaneous confidence bands for deep learning models, focusing on survival analysis.

The reviewers agree that the paper is well-written, engaging, and well-motivated. However, the proposed method lacks theoretical guarantees, and the empirical results show that the method's proposed confidence intervals undercover the target variable. 

Thus, to make the paper fully evidenced for the camera-ready, the authors should clarify in the abstract and throughout the paper that they focus on empirical results, acknowledge that the empirical undercovers the target variable (up to 8% points in some settings?), and only provide a theoretical motivation for the method. The statement that the method provides valid and accurate confidence intervals (as stated in the abstract) must be fixed and qualified.

Usually, uncertainty quantification methods are conservative (like the used baseline), as it is better to overcover than undercover in many settings, such as when there are safety risks or in medical settings.

Overall, this is an acceptance with minor revisions, and I will closely follow up with the authors to ensure that the changes are incorporated.
Decision: Accept (poster)