Toward Conditional Distribution Calibration in Survival Prediction
Authors: Shi-ang Qi
Yakun Yu
Russell Greiner
Venue: NeurIPS 2024 poster (2024)
Keywords: survival analysis; calibration; conformal prediction; censorship; discrimination
Primary area: machine_learning_for_healthcare
TL;DR: This paper underscores the critical role of marginal and conditional calibration in survival analysis and introduces a method that enhances both marginal and conditional calibration without sacrificing discrimination performance in survival models.

Abstract: Survival prediction often involves estimating the time-to-event distribution from censored datasets. Previous approaches have focused on enhancing discrimination and marginal calibration. In this paper, we highlight the significance of *conditional calibration* for real-world applications – especially its role in individual decision-making. We propose a method based on conformal prediction that uses the model’s predicted individual survival probability at that instance’s observed time. This method effectively improves the model’s marginal and conditional calibration, without compromising discrimination. We provide asymptotic theoretical guarantees for both marginal and conditional calibration and test it extensively across 15 diverse real-world datasets, demonstrating the method’s practical effectiveness and
versatility in various settings.

Toward Conditional Distribution Calibration in Survival Prediction
Shi-ang Qi Yakun Yu 2 Russell Greiner
*Computing Science, University of Alberta, Edmonton, Canada
Electrical Computer Engineering, University of Alberta, Edmonton, Canada Allberta Machine Intelligence Institute, Edmonton, Canada {shiang, yakun2, rgreiner)ouallerta.cc
Abstract
Survival prediction often involves estimating the time-to-event distribution from censored datasets. Previous approaches have focused on enhancing discrimination and marginal calibration. In this paper, highlight the significance of condi- tional calibration for real-world applications - especially its role in individual We propose method based on conformal prediction that uses the model's predicted individual survival probability at that instance's s observed time. This method effectively improves the model's 's marginal and conditional calibration, without compromising discrimination. We provide asymptotic theoretical guaran- tees for both marginal and conditional calibration and test it extensively across 15 diverse real-world datasets, demonstrating the method's practical effectiveness and versatility in various settings.
Introduction
Individual survival distribution (ISD), or time-to-event distribution, is probability distribution that describes the times until the occurrence of specific event of interest for an instance, based on information about that individual. Accurately estimating ISD is essential for effective decision- making and clinical resource allocation. However, challenge in learning such survival prediction models training on datasets that include censored instances, where we only know lower bound of their time-to-event.
Survival models typieally focus two important but distinet properties during optimization and evaluation: (i) discrimination measures how well a model's relative predictions between individuals align with the observed order
which is useful for pairwise decisions such as prioritizing treatments; (ii) calibration assesses how well the predicted survival probabilities match the actual distribution of observations 3 4 supporting both individual-level (e.g., determining high-risk treatments based on the probability) and group-level (e.g.. allocating clinical resources) decisions. Some prior research has sought to improve calibration by integrating a calibration-specifie loss during optimization [5] 6 4. However, these often produce models with poor discrimination 7 g limiting
their utility in scenarios where precise pairwise decisions are critical.
Furthermore, previous studies have typically addressed calibration a marginal sense - i.e., assessing whether probabilities align with the actual distribution across the entire population. However, for many applications, marginal calibration may be inadequate we often require that predictions are correctly calibrated, conditional on any combination of features. This can be helpful for making more precise clinical decisions for individuals and groups. For example, when treating an overweight male, a doctor might decide on cardiovascular surgery using a model calibrated for both overweight and male. Note this might lead to a different decision that one based on model that was calibrated for all patients. Similarly, a hospice institution may want to allocate nursing care based model
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

that generates calibrated predictions for elderly individuals. This also aligns with the faimess perspective (.] where clinical decision systems should guarantee equalized calibration performance across any protected groups.
Contributions To overcome these challenges, we introduce the CiPOT framework, a post- processing approach built upon conformal prediction [10 [8] that uses the Individual survival Probability at Observed Time (iPOT) as conformity scores and generates conformalized survival distributions. The method has 3 important properties: (i) this conformity score naturally conforms to the definition in distribution calibration survival analysis [3] (ii) it also captures the distribution variance of the ISD, therefore is adaptive to the features; and (iii) the method is computationally friendly for survival analysis models. Our key contributions are:
Motivating the use of conditional distribution calibration in survival analysis, and proposing a metric (Calws. defined in Section 4) to evaluate this property.
Developing the CiPOT framework, to accommodate censorship. The method effectively solves some issues of previous conformal methods wrt inaccurate Kaplan-Meier estimation;
Theoretically proving that CiPOT asymptotically guarantees marginal and conditional distri- bution calibration under some specified assumptions;
Conducting extensive experiments across 15 datasets, showing that CiPOT improves both marginal and conditional distribution calibration without sacrificing discriminative ability;
Demonstrating that CiPOT is computationally more efficient than prior conformal method on survival analysis.
2 Problem statement and Related Work
2.1 Notation
A survival dataset = [(x;, 6;)) contains tuples, each containing covariates E R an observed time I IR,, and an event indicator & {0, 1). For each subject, there are two potential times of interest: the event time e; and the censoring time q. However, only the earlier of the two observable. We assign happened by f; (right-censored) and 8, 1 indicates the event occurred at 4 (uncensored). Let I denote the set of indices dataset 2, then we can use i Z to represent (æ;, (,, 5,) DD.
min (e and 5, [ee S G... so 6; = O means the event has not
Our objective is to estimate the Individualized Survival Distribution (ISD), S(t æ.) (Pe > 1 X 2;), which represents the survival probabilities of the i-th subject for any time
2.2 Notions of calibration in survival analysis
Calibration measures the alignment between the predictions against observations. Consider dis- tribution calibration at the individual level: if an oracle knows the true ISD S(t æ;), and draws realizations of e; x; (call them e, e (,,)) ....). then the survival probability at observed time
(S(e(T)) am))m. should be distributed across a standard uniform distribution Uo.1] (probability integral theorem [33.. However, in practice, for each unique I;, there is only one realization of e Rt. meaning we cannot check the calibration in this individual manner.
To solve this, Haider et al. 3 proposed marginal calibration, which holds if the predicted survival probabilities at event times e; over the æ; in the dataset, (S(e; | (x)) matches Ujo.1]-
Definition 2.1. For uncensored dataset, a model has perfect marginal calibration iff [21. [0,
P (S((c mx)) e [21, i 6 1) E [ [S(c]] ax)) c [pL. 6 =1
P2 P1-
(1)
We can "blur" each censored subject uniformly over the probability intervals after the survival probability at censored time $(ca ax) 3 (see the derivation in Appendix A):
IP (Š(E; x+)+ [pI,p2] 8, 0) ($(..le:)=p))((((A))C Je( (onp::/p)))))((tsetceeeeee))er)) pp])) ( ()))))))))))))))))
(2)
S(t./z,)

---

Predictions
Histograms
P-P Plots
Survival Prob.
Survival Prob.
Observed Prob.
Marginal
Marginel
w
0
2
0
1/3
2/3
Days
(b)
Counts
(c)
Predicted Prob.
Survival Prob.
Survival Prob.
Prob..
Observed Prob.
=
Conditional
Marginii
2
0
1/3
2/3
(4)
Days
Counts
Predicted Prob.
Figure I: Two notions of distribution calibration: marginal and conditional, illustrated using 3 bins separated at 3 and 5. The curves (a, d) represent the predicted ISDs. The colors of the stars distinguish the six subjects, with horizontal coordinates indicating the true event time (consistent across all panels) and vertical coordinates representing predicted survival probability event time. Note the two groups (orange for 0 and blue for 1) correspond to the colors of the curves and histograms (a, b, d, e). Note that all three P-P lines in the conditional case (f) coincide.
Figure aa) illustrates how marginal calibration is assessed using 6 uncensored subjects. Figure Tb. c) show the histograms and P-P plots, showing the predictions are marginally calibrated, over the predefined 3 bins, as we can see there are 6/3 = 2 instances in each of the 3 bins. However, if we divide the datasets into two groups (orange vs. blue think men vs. women), we can see that this not the case, there is orange instance in the [ bin, and 2 orange instances in the [3. 1] bin.
In summary, individual calibration ideal but impractical. Conversely, marginal calibration is more feasible but fails to assess calibration relative to certain subsets of the population by features. This discrepancy motivates us to explore a middle ground conditional calibration. A conditionally calibrated prediction, which ensures that the predicted survival probabilities are uniformly distributed in each of these groups, as shown in Figure dd. e. f), is more effective in real-world scenarios. Consider predicting employee attrition within company: while a marginal calibration using Kaplan- Meiaer (KM) [14] curve might reflect overall population trends, fails account for variations such as the tendency of lower-salaried employees to leave earlier. A model that is calibrated for both high and low salary levels would more helpful predicting the actual quitting times and facilitate planning. Similarly, when predicting the timing of death from cardiovascular diseases, models calibrated for older populations, who exhibit more predictable and less varied outcomes (15), may not apply to younger individuals with higher outcome variability. Using age-inappropriate models could lead to inaccurate predictions, adversely affecting treatment plans.
2.3 Maintaining discriminative performance while ensuring good calibration
Methods based on the objective function 5 have been developed to enhance the marginal calibration of ISDs, involving the addition of calibration loss to the model's original objective function (e.g., likelihood loss). However, while those methods are effective in improving the marginal calibration performance of the model, their model often significantly harms the discrimination performance 66 7 a phenomenon known as the peterriiinnaiinccaaiibian
trade-off (
Post-processee methods 12 g have been proposed to solve this trade-off by disentangling calibration from discrimination in the optimization process. Candès et al. [12] uses the individual censoring probability as the weighting addition to the regular Conformalized Quantile Regression (CQR) [T]] method. However, their weighting method is only applicable to Type-I censoring settings where each subject must have known censored time [6 datasets.
which is not applicable most the right-censoring

---

Qi et al. 8 developed Conformalized Survival Distribution (CSD) by first discretizing the ISD curves into percentile times (via predefined percentile levels), and then applying CQR [ for each percentile level (see the visual illustration of CSD in Figure 6 in Appendix B). Their method handles right-censoring using KM-sampling, which employs a conditional KM curve to simulate multiple event times for censored subject, offering calibrated approximation for the ISD based on the subject's censored time. However, their method struggles with some inherent problems of KM 14] e.g., KM can be inaccurate when the dataset contains a high proportion of censoring (17]. Furthermore, we also observed that the KM estimation often concludes at high probabilities (as seen in datasets like HFCR, FLCHAIN, and Employee in Figure 9 This poses challenge in extrapolating beyond the last KM time point, which hinders the accuracy of KM-sampling, thereby constraining the efficacy of CSD (see results in Figure
Our work inspired by CSD [ and can be seen as percentile-base refinement of their regression- based approach. Specifically, our CiPOT effectively addresses and resolves issues prevalent in the KM-sampling, significantly outperforming existing methods in terms of improving the marginal distribution calibration performance. Furthermore, to our best knowledge, this is the first approach that optimizes conditional calibration within the survival analysis that can deal with censorship.
However, achieving conditional calibration (also known as conditional coverage in some litera- ture [TO) is challenging because it cannot be attained in a distribution free manner for non-trivite predictions. In fact, guarantees of finite sample for conditional calibration are impossible to achieve even for standard regression datasets without censorship [O
This limitation is an important topic in the statistical learning and conformal prediction literature 1T0
18
19
I8 19. Therefore, our paper does not attempt to provide finite sample guarantees. Instead, following the approach of many other researchers [1] 24), we provide only asymptotic guarantees as the sample size approaches infinity, The key idea behind this asymptotic conditional guarantee that the construction of post-processing predictions relies on the quality of the original predictions. Thus, we aim for conditional calibration only within the class of predictions that can be learned well that is, consistent estimators.
We acknowledge that this assumption may not hold in practice; however, (i) reliance on consistent estimators is a standard (albeit strong) assumption in the field of conformal prediction (21 23). (ii) to the best of our knowledge, no previous results have proven conditional calibration under more relaxed conditions, and (iii) we provide empirical evidence of conditional calibration using extensive experiments (see Section
3 Methods
This section describes our proposed method: Conformalized survival distribution using Individual survival Probability at Observed Time (CiPOT), which motivated by the definition of distribution calibration [3] and consists of three components: the estimation of continuous ISD prediction, the computation of suitable conformity scores (especially for censored subjects), and their conformal calibration.
3.1 Estimating survival distributions
For simplicity, our method is motivated by the split conformal prediction 225 We start the process by splitting the instances of the training data into proper training set Dirain and a conformal set Deon Then, we can use any survival algorithm or quantile regression algorithm (with the capability of handling censorship) to train model M using Dirain that can make ISD predictions for Dcon see Figure aa).
With little loss of generality, we assume that the ISD predicted by the model, Syn(t | x;),, are right-continuous and have unbounded range, i.e., (M xx)) > for all / 2 O. For survival algorithms that can only generate piecewise constant survival probabilities (e.g., Cox-based methods [26] 27 discrete-time methods (8 29], etc.), the continuous issue can be fixed by applying some interpolation algorithms (e.g., linear or spline).

---

Predictions
Histograms
P-P Plots
1
Survival Prob. ue
Survica Prob.
Observed Frob.
Before CIPTT
1 Marginal
2
1/3
2/3
Days
Counts
Prodicted Prob.
Surwiaa Prob.
Survivl Prob.
Observed Prob. wih
Madgnal
GPOT
Survival
3
2/3 Prodicted Prob
Days
Days
Counts
Figure 2: A visual example of using CiPOT make the prediction "ronditionallyy-calibraac. (a) Ini- tialize ISD predictions from an arbitrary survival algorithm with associated (b) histograms and (c) P-P plots. (d) Calculate Percentile I'M) (grey lines) for all ps, and find the intersections (hollow points) of the ISD curves and the Percentile(p; I'M) lines; (e) Generate new ISD by vertically shifting the hollow points the p's level, with associated histogram and (g) P-P plots. Figure provide side-by-side byde visual comparison between CSD and our method.
3.2 Compute conformal scores and calibrate predicting distributions
start by sketching how CiPOT deals with only uncensored subjects. Within the conformal set, for each subject E I0em we define a distributional conformity score, wrt the model M, termed the predicted Individual survival Probability at Observed Time (iPOT):
Mi.M == (e, a.))
Here, for uncensored subjects, the observed time corresponds to the event time, eq. Recall from Section 2.2tiionn221uee 2.2 that predictions from model M are marginally calibrated if the iPOT values follow Ujo.1] i.e., if we collect the distributional conformity scores for every subject in the conformal set I'M ( Jizwe, the p-th percentile value in this set should be equal to exactly p. If so, post processing adjustments are necessary.
In general, of course, the estimated Individualized Survival Distributions (ISDs) S(t ax)) may not perfectly align with the true distributions S(t | æ;) from the oracle. Therefore, for a testing subject with index n 1, we can simply apply the following adjustment its estimated ISD:
S (n-1) 551 Percentilee(;; MM) 1a+1), Vpe (0, 1).
Here, Percentile(p: p; M)) calculates the [e(2)*++)] -th empirical percentile of I M. This adjustment [D
aims re-calibrate the estimated ISD based on the empirical distribution of the conformity scores.
Visually, this adjustment involves three procedures:
(i) It first identifies the empirical percentiles of the conformity scores Percentile ; IM) and Percentile I M,. illustrated by the two grey lines 0.28 and 0.47 in Figure d d). respectively which uniformly divide the stars according to their vertical locations;
(ii) It then determines the corresponding times on the predicted ISDs that match these empirical percentiles (the hollow circles, where each ISD crosses the horizontal line);
(iii) Finally, the procedure shifts the empirical percentiles (grey lines) to the appropriate height of desired percentiles ( and } along with all the circles. This operation indicated by the vertical shifts of the hollow points, depicted with curved red arrows in Figure d).
5

---

This adjustment results in the post-pre processed curves depicted in Figure ee.. It shifts the vertical position of the green star * from the interval [ 5 to - 11. and the pink star * from [0, 5] to [ 1:33.. 3]. These shifts ensure that the calibration histograms and P-P plots achieve a uniform distribution (both marginally the whole population and conditionally each group) across the defined intervals.
After generating the post-processed curves, we can apply final step, which involves transforming the inverse ISD function back into the ISD function for the testing subject:
5M (t Øn+1) inf p Syx (pl Øx+1) st}.
(5)
The simple visual example in Figure 2 shows only two percentiles created at and 3.2 In practical applications, the user provides predefined set of percentiles, P, adjust the ISDs. The choice of P can slightly affect the resulting survival distributions, each capable of achieving provable distribution calibration; see ablation study #2 in Appendix E.6 for how P affects the performance.
3.3 Extension to censorship
is challenging to incorporate censored instances into the analysis as we do not observe their true event times, 6;, which means cannot directly apply conformity score in (3) and the subsequent conformal steps. Instead, we only observe the censoring times, which serve as lower bounds of the event times.
Given the monotonic decreasing property the ISD curves, the iPOT value for a censored subject, i.e., (t, æ;) = SM(c; wz), now serves as the upper bound of (M æ;). Therefore, given the prior knowledge that SM (ex æ;;) ~ [[..1]. the observation of the censoring time updates the possible range of this distribution. Given that (ez xx)) must be less than or equal to S,M (c: w;), the updated posterior distribution follows SM(e;; m:) ~ (0. (=(=x()
Following the calibration calculation 3 where censored patients are evenly "blurred" across subsequent bins of ((c æx)) our approach uses the above posterior distribution to uniformly draw R potential conformity scores for censored subject, for some constant R E Z", Specifically, for censored subject, we calculate the conformity scores as:
TiM (c (z,) uR. where un [0/R. 1/R....,R/AA
Here, uR is a pseudo-uniform vector to mimic the uniform sampling operation, significantly reducing computational overhead compared to actual uniform distribution sampling. For uncensored subjects, we also need to apply a similar sampling strategy to maintain a balanced censoring rate within the conformal set. Because the exact iPOT value is known and deterministic for uncensored subjects, sampling involves directly drawing from degenerate distribution centered at Sm(e;; (e: ax:) i.e., just drawing (e; xx) R times. The pseudo-code for implementing the CiPOT process with censoring outlined in Algorithm| iin Appendix B
Note that the primary computational demand of this method stems from the optional interpolation and extrapolation of the piecewise constant ISD predictions. Calculating the conformity scores and estimating their percentiles incur negligible costs in terms of both time and space, once the right- continuous survival distributions are established. We provide computational analysis in Appendix E.5
3.4 Theoretical analysis
Here we discuss the theoretical properties of CiPOT. Unlike CSD S which adjusts the ISD curves horizontally (changing the times, for a fixed percentile), our refined version scales the ISD curves vertically. This vertical adjustment leads to several advantageous properties. In particular, we highlight why our method is expected yield superior performance in terms of marginal and conditional calibration compared to CSD [ Table I summarizes the properties of the two methods.
Calibration CiPOT differs from CSD in two major ways: CiPOT (i) essentially samples the event time from (1 L> a,, ax) for a censored subject, and (ii) subsequently converts these times into corresponding survival probability values on the curve.

---

Table 1: Properties of CSD and CiPOT Note that the calibration guarantees refer to asymptotic
calibration guarantees. 'ee Appendix E.5
Marginal calibration guarantee
Conditional calibration guarantee
Harrell's discrimination guarantee
Antolini's discrimination guarantee
Space complexity
Methods
Monotonic
CSD 8 CiPOT
O(N IPI- RR)
X
O R)
The first difference contrasts with the CSD method, which samples from a conditional KM distri- bution, SKM( & c₂ assuming homoskedastic survival distribution across subjects (where the conditional KM curves have the same shape and the random disturbance of e; is independent of the features æ.). However, CiPOT differs by considering the heteroskedastic nature of survival distributions SM (t I > Ct2 x..). For instance, consider the symptom onset times following exposure to the COVID-19 virus. Older adults, who may exhibit more variable immunc responses, could experience a broader range of onset times compared younger adults, whose symptom onset times are generally more consistent [3). By integrating this cature-dependen variability, CiPOT captures the inherent heteroskedasticity of survival distributions and adjusts the survival estimates accordingly. which helps with conditional calibration.
Furthermore, by transforming the times into the survival probability values the predicted ISD) curves (the second difference), we mitigate the trouble of inaccurate interpolation and extrapolation of the distribution. This approach particularly useful when the conditional distribution terminates at relatively high probability, where extrapolating beyond the observed range is problematic due to the lack of data for estimating the tail behavior. Different extrapolation methods, whether parametric or spline-basee based, can yield widely varying behaviors in the tails of the distribution, potentially leading to significant inaccuracies in survival estimates. However, by converting event times into survival percentiles, CiPOT circumvents these issues. This method capitalizes on the probability integral transform 1.), which ensures that regardless of the specific tail behavior of survival function, its inverse probability values will follow a uniform distribution.
The results state that the output of method has asymptotic marginal calibration, with necessary assumptions (exchangeability, conditional independent censoring, and continuity). We also prove the asymptotic conditional calibrated guarantee for CiPOT. The proof's of these two results are inspired by the standard conformal prediction literature (11,2)) with adequate modifications to accommodate our method. We refer the reader Appendix C.I for the complete proof.
Theorem 3.1 (Asymptotic marginal calibration). If the instances D are exchangeable, and follow the conditional independent censoring assumption, then for new instance 1, p1 P2 [0. 1].
P1 S
(t xm+1) ) (p1,P2)) < P2 P1- Deun +1
Theorem 3.2 (Asymptotic conditional calibration). In addition to the assumptions in Theorem 3.1 if
(i) the non-processed prediction (t (t] xx) is a consistent survival estimator; (ūi its inverse function differentiable; and (iii) the 1st derivation the inverse function is bounded by a constant, then the CGPOT process will achieve asymptotic conditional distribution calibration.
Monotonicity Unlike CSD, CiPOT does not face any non-monotonic issues for the post-processed curves as long as the original ISD predictions are monotonic; see proof in Appendix C.2
Theorem 3.3. CiPOT process preserves the monotonic decreasing property of the ISD.
CSD, built on the Conformalized Quantile Regression (CQR) framework. struggles with the common issue of non-mon monotonoo quantile curves (refer to Appendix D.2 in 8 and our Appendix C.2). While some methods, like the one proposed by Chernozhukov al. [3], address this issue by rearranging quantiles, they can be computationally intensive and risk (slightly) recalibrating and distorting discrimination the rearranged curves. By inherently maintaining monotonicity, CiPOT not only enhances computational efficiency but also avoids these risks.
7

---

Discrimination Qi al. demonstrated that CSD theoretically guarantees the preservation of the original model's discrimination performance in terms of Harrell's concordance index (C-index) M
However, CiPOT lacks this property, see Appendix C.3 for details.
As CiPOT vertically scales the ISI) curves, it preserves the relative order of survival probabilities at any single time point. This preservation means that the discrimination power, measured by the area under the receiver operating characteristic (AUROC) at any time, remains intact (Theorem C.4). Furthermore, Antolini's 's time-dependent C-index (Cta) 32), which represents a weighted average AUROC across all time points, also guaranteed be maintained by our method (Lemma C.5 a comparison, CSD does not have such a guarantee for neither AUROC nor Crd
Evaluation metries
We measure discrimination using Harrell's C-index (1 rather than Antolini's Ctdd (32. already established that Ctd is not changed by CiPOT. We aim to assess our performance using a measure that represents relative weakness of our method.
Lemma C.5
As to the calibration metries, the marginal calibration score evaluated on the test set Diest is calculated 3 g
Calmargin (5:P) = [P]
i Σ (P(S(e m:)) ( (ests p) PEP
(6)
where P(Š(ez e [0, i ZT is calculated by combining ① and (): see ( in Appendix A Based the marginal calibration formulation, natural way for evaluating the conditional calibration could be: (i) heuristically define finite feature space set (2,, e.g. S₁ is the set of divorced elder males, S2 is females with children, etc.; and (ii) calculate the worst calibration score all the predefined sub-spaces. This is similar to fairness settings, researchers normally select age, sex, or race as the sensitive attributes to form the feature space. However, this metric does scale to nigher-dimensional settings because it is challenging to create the feature space set that contains all possible combinations of the features.
Motivated by Romano et al. 333, we proposed a worst-slab distribution calibration, Calws- We start by partition the testing set into 25% exploring set Dexplore and 75% exploiting set Dexplait The exploring set is then used to find the worst calibrated sub-region in the feature space Rá:
Se,a. {z; Rd
v x. and IP P( (x, &S, Iexplore
Σ (P (p(((e:) xx)) [], i Texplore x, &e,a,))) p)2
where v,a,b= b= arg max max weR .cbbeR
In practice, the parameters U, a, and are chosen adversarially by sampling i.i.d. vectors v on the unit sphere in IR then finding the arg max using grid search on the exploring set. λ is predefined threshold to ensure that we only consider slabs that contain at least k% of the instances (so that we do encounter pregnant-man situation). Given this slab, can calculate the conditional calibration score on the evaluation set for this slab:
(SP = PP| Σ
(P(((e)) x.) é [0. pl. i Txxploit x, & p)² (7)
Besides the above metrics, we also evaluate using other commonly used metries: integrated Brier score (IBS) 34, and mean absolute error with pseudo-observation (MAE-PO) [35]; see Appendix
5
Experiments
The implementation of CiPOT method, worst-slab distribution calibration score, and the code to reproduce all experiments in this section are available at https:///iithb. com/shi-ans valesurrivaalllllbattteegggaggi
5.1 Experimental setup
Datasets We use 15 datasets to test the effectiveness of our method. Table 3 in Appendix F.1 summarizes the dataset statistics, and Appendix E. also contains details of preprocessing steps,

---

HFCR
FLCHAIN
Employee
0.95
464
90p
C-index 1
990
0.90
0.6
0.85
0.75
0.4
0.80
10"
Caluarga
10
10
10"
10
Methods
Baseline
CSD
CIPOT
Figure 3: Violin plots of C-index and Calmargin performance of our method (CiPOT) and benchmarks. The shape of each violin plot represents the probability density of the performance scores, with the black bar inside the violin indicating the mean performance. The red dashed lines in the lower panels represent the mean calibration performance for KMM, serving as an empirical lower limit.
Table 2: Performance summary of CiPOT. Values in parentheses indicate statistically significant differences (p < 0.05 using two-sided f-test). A tie means the first 3 significant digits are the same. 'The total number of comparisons for Calus is 69, while it is 104 for the other metrics.
C-index
Calmargin
Calus
IBS
MAE-PO
Win
7 (0)
95 (50)
64 (29)
63 (14)
54 (8)
Compare with Baselines
Lose
22 (0)
(1)
(1)
23 (0)
17 (0)
Tic
75
0
0
18
33
Win
11 (1)
68 (37)
51 (26)
53 (15)
39 (8)
Compare with CSD
Lose
26 (0)
36 (20)
18 (7)
35 (11)
39 (4)
Tie
67
0
0
16
26
KM curves, and histograms of event/censor times. Compared with 8 we added datasets with high censoring rates (>60%) and ones whose KM ends with high probabilities (>50%).
Baselines We compared survival algorithms: AFT [3). GB [37]. DeepSurv 18 N-MTLR [39,,
DeepHit [29], CoxTime (27), and CQRNN [40]. We also include KM benchmark (empirical lower bound) for marginal calibration, which is known to achieve perfect marginal calibration B g
Appendix F.2 describes the implementation details and hyperparameter settings.
Procedure We divided the data into a training set (90%) and a testing set (10%) using a stratified split to balance time & and censor indicator 6;- We also reserved a balanced 10% validation subset from the training data for hyperparameter tuning and early stopping. This procedure was replicated across 10 random splits for each dataset.
5.2 Experimental results
Due to space constraints, the main text presents partial results for datasets with high censoring rates and high KM ending probabilities (HFCR, FLCHAIN, Employee, MIMIC-IV). Notably, CQRNN did not converge on the MIMIC-IV dataset. Thus, we conducted total of 104 method comparisons (15 datasets x 7 baselines 1). Table 2 offers a detailed performance summary of CiPOT versus baselines and CSD method across these comparisons. Appendix E.4 presents the complete results.
The upper panels of Figure F indicate minimal differences in the C-index among the methods, with notable exceptions primarily involving Deepifit. Specifically, CiPOT matched the baseline C-index in 75 instances and outperformed it in out of 104 comparisons. This suggests that CiPOT maintains discriminative performance in approximately 79% of the cases.
Discrimination
Marginal Calibration The lower panels of Figure 3 show significant improvements in marginal calibration with CiPOT. It often achieved near-optimal performance, marked by the red dashed

---

FLCHAIN
Employee
MIMIC-IV
8
10"
10 "3
1
10
10
10
66
NAIER
DeneSun
Dhaposa
CraTine
coeses
NF
6B
NATER
DempSion
Chspeti
ccpser
AFT
WAITER
DompSoer
Templi
Civiton
Methods
Baseline
CSD
CIPOT
Figure 4: Violin plots of Calws performance, where the shape and black bars represent the density and mean. Smaller values represent better performance. Note CQRNN did not converge MIMIC-IV.
lines. Table 2 also shows that CiPOT provided better marginal calibration than the baselines in 95 (and significantly in 50) out of 104 comparisons (91%).
CiPOT's marginal calibration was better than CSD most of the time (68/104, 65%). The cases where CSD performs better typically involve models like Deepftit or CQRNN. This shows that our approach often does perform as well as CSD when the original model is heavily miscalibrated, which suggests a minor limitation of our method. Appendix C.4 discusses why our method is sub-optimal for these models.
Conditional Calibration For small datasets (sample size 1000), in some random split, we can find a worst-slab region S the exploring set with P(x;; Su,a,b, i Texpplore 2 33% but still no subjects in this region in the exploiting set. This is probably because we only ensure that the times and censored indicators unbalanced. Therefore, we only evaluated conditional calibration on the 10 larger datasets, resulting in 69 comparisons. Among them, CiPOT improved conditional calibration in 64 cases (93%) compared to baselines and in 51 cases (74%) compared to CSD.
balanced during the partition, however, the features can still be
Case Study We provide 4 case studies in Figure 13 in Appendix E.4 where CSD leads to signifi- cant miscalibration within certain subgroups, and CiPOT can effectively generate more conditional calibrated predictions in those groups. These examples show that CSD's miscalibration is always located at the www-bbbbbbiity regions, which corresponds to our statement (in Section 3.4) that the conditional KM sampling method that CSD used is problematic for the tail of the distribution.
Other Metrics Results in Table and Appendix E.4 show that CiPOT also showed improvement in both IBS and MAE-PO, outperforming 63 and 54 out of 104 comparisons, respectively.
Computational Analysis Appendix E.5 shows the comprehensive results and experimental setup. In summary, CiPOT significantly reduces the space consumption and running time.
Ablation Studies We conducted two ablation studies assess (i) the impact of the repetitions value (R) and (ii) the impact of predefined percentiles (P) on the method; see Appendix E.6
6 Conclusions
Discrimination and marginal calibration are two fundamental yet distinct elements in survival analysis. While marginal calibration is feasible, it overlooks accuracy across different groups distinguished by specific features. this paper, we emphasize the importance of conditional calibration for practical applications and propose principled metric for this purpose. By generating conditionally calibrated Individual Survival Distributions (ISDs), we can better communicate the uncertainty in survival analysis models, enhancing their reliability, fairness, and real-world applicability.
We therefore define the Conformalized survival distribution using Individual Survival Probability at Observed Time (CiPOT) post-pree pesss-rrcessing framework that enhances both marginal and conditional calibration without compromising discrimination. It addresses common issues in prior methods, particularly under high censoring rates or when the Kaplan-Meier curve terminates at high prob- ability. Moreover, this post-prose pros-prcessiig adjusts the ISDs by adapting the heteroskedasticity of the distribution, leading to asymptotic conditional calibration. Our extensive empirical tests confirm that CiPOT significantly improves both marginal and conditional performance without diminishing the models' discriminative power.
10

---

Acknowledgments and Disclosure of Funding
This research received support from the Natural Science and Engineering Research Council of Canada (NSERC), the Canadian Institute for Advanced Research (CIFAR), and the Alberta Machine Intelligence Institute (Amii). The authors extend their gratitude to the anonymous reviewers for their insightful feedback and valuable suggestions.
References
[1] Frank E Harrell Jr, Kerry L Lee, Robert M Califf, David B Pryor, and Robert A Rosati. Regression modelling strategies for improved prognostic prediction. Statistics in medicine, 3(2): 143 152, 1984.
[2] Frank E Harrell Jr, Kerry L. Lee, and Daniel B Mark. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in medicine, 15(4):361-33) 1996.
[3] Humza Haider, Bret Hoehn, Sarah Davis, and Russell Greiner. Effective ways to build and evaluate individual survival distributions. Journal of Machine Learning Research, 21(85):1-63, 2020.
[4] Paidamoyo Chapfuwa, Chenyang Tao, Chunyuan Li, Irfan Khan, Karen Chandross, Michael J Pencina, Lawrence Carin, and Ricardo Henao. Calibration and uncertainty in neural time-to- event modeling. IEEE transactions neural networks and learning systems, 2020.
[5] Anand Avati, Tony Duan, Sharon Zhou, Kenneth Jung, Nigam H Shah, and Andrew Y Ng. Countdown regression: sharp and calibrated survival predictions. In Uncertainty in Artificial Intelligence, pages 145-155. PMLR, 2020.
[6] Mark Goldstein, Xintian Han, Aahlad Puli, Adler Perotte, and Rajesh Ranganath. X-cal: Explicit calibration for survival analysis. Advances in neural information processing systems, 33:18296-188 18307, 2020.
[7] Fahad Kamran and Jenna Wiens. Estimating calibrated individualized survival curves with deep learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(1):240- 2482248 May 2021.
[8] Shi-Ang Qi, Yakun Yu, and Russell Greiner. Conformalized survival distributions: A generic post-process to increase calibration. In Proceedings of the 41 International Conference Machine Learning, volume 235, pages 4 41303-41339, PMLR, 21-27 Jul 2024.
[9] Sahil Verma and Julia Rubin. Fairness definitions explained. In Proceedings of the international workshop on software fairness, pages 1-7, 2018.
[10] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005.
[1I] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression.
Advances in neural information processing systems, 32, 2019.
[12] Emmanuel Candès, Lihua Lei, and Zhimei Ren. Conformalized survival analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(1):24 45, 2023.
[13] John E Angus. The probability integral transform and related results. SIAM review, 36(4): 652-654, 1994.
[14] Edward L Kaplan and Paul Meier. Nonparametric estimation from incomplete observations. Journal of the American statistical association, 53(282):457-481. 1958.
[15] Frank LJ Visseren, François Mach, Yvo M Smulders, David Carballo, Konstantinos Koskinas, Maria Bäck, Athanase Benetos, Alessandro Biffi, Jose-Manuel Boavida, Davide Capodanno, al. 2021 ESC Guidelines on cardiovascular disease prevention in clinical practice: Developed by the Task Force for cardiovascular discase prevention in clinical practice with representatives
11

---

of the European Society of Cardiology and 12 medical societies With the special contribution of the European Association of Preventive Cardiology (EAPC). European heart journal, 42 (34):3227-3337, 2021.
[16] John P Klein and Melvin L Moeschberger. Survival analysis: techniques for censored and Iruncated data. Springer Science & Business Media, 2006.
[17] Na Liu, Yanhong Zhou, and J Jack Lee. IPDfromKM: reconstruct individual patient data from published Kaplan-Men Meinrr survival curves. BMC medical research methodology, 21(1):111, 2021.
[18] Jing Lei and Larry Wasserman, Distribution- free prediction bands for non parameetri regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):71-96, 2014.
[19] Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455-45 2021.
[20] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution- free predictive inference for regression. Journal of the American Statistical Associ- ation, 1136523) 2018.
[21] Matteo Sesia and Emmanuel J Candès, A comparison of some conformal quantile regression methods. Stat, 9(1):226,, 2020.
[22] Rafael Izbicki, Gilson Shimizu, and Rafael Stem. Flexible distribution-free conditional predic- tive bands using density estimators. In International Conference on Artificial Intelligence and Statistics, pages -0768 PMLR, 2020.
[23] Victor Chernozhukov, Kaspar Wüthrich, and Yinchu Zhu. Distributional conformal prediction. Proceedings of the National Academy of Sciences, (1848))e2100779111 18, 2021.
[24] Rafael Izbicki, Gilson Shimizu, and Rafael B. Stern. CD-split and HPD-split: Efficient Conformal Regions in High Dimensions. Journal of Machine Learning Research, 23(8))11-32, 2022.
[25] Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In Machine learning: ECML 2002: 13th European conference machine learning Helsinki, Finland, August 19-23, 2002 proceedings 13, pages 345-356. Springer, 2002.
[26] David R Cox. Regression models and life-tables. Journal of the Royal Statistical Society: Series (Methodological), 34(2):187 202, 1972.
[27] Havard Kvamme, Ørnulf Borgan, and Ida Scheel. Time-to-event prediction with neural networks and cox regression. Journal of Machine Learning Research, 20:1-30, 2019.
[28] Chun-Nam Yu, Russell Greiner, Hsiu-Chin Lin, and Vickie Baracos. Learning patient-specific cancer survival distributions as sequence of dependent regressors. Advances in Neural Information Processing Systems, 24: 14:1855-1853, 2011.
(29) Changhee Lee, William R Zame, Jinsung Yoon, and Mihaela van der Schaar. Deephit: A deep learning approach to survival analysis with competing risks. In Thirty-second AAAI conference on artificial intelligence, 2018
[30] Joseph D Challenger, Cher Y Foo, Yue Wu, Ada WC Yan, Mahdi Moradi Marjaneh, Felicity Liew, Ryan S Thwaites, Lucy C Okell, and Aubrey J Cunnington. Modelling upper respiratory viral load dynamics of sars-cov-2. BMC medicine, 20:1-20, 2022.
[31] Victor Chernozhukov, Iván Fernández Val, and Alfred Galichon. Quantile and probability curves without crossing. Econometrica, 78(3):1093-111 1125, 2010.
[32] Laura Antolini, Patrizi Boracchi, and Elia Biganzoli. A time-dependent discrimination index for survival data. Statistics in medicine, 24(24):3927-3944 2005.
12

---

[33] Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems, 33:3581-3591, 2020.
[34] Erika Graf, Claudia Schmoor, Willi Sauerbrei, and Martin Schumacher, Assessment and comparison of prognostic classification schemes for survival data. Statistics in medicine, 18 (17-18):2529 25442, 1999.
[35] Shi-Ang Qi, Neeraj Kumar, Mahtab Farrokh, Weijie Sun, Li-Hao Kuan, Rajesh Ranganath, Ricardo Henao, and Russell Greiner. An effective meaningful way to evaluate survival models. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 28244-28276. PMLR, 23-29 Jul 2023.
[36] Winfried Stute. Consistent estimation under random censorship when covariables are present. Journal of Multivariate Analysis, 45(1):99-103 1993.
[37] Torsten Hothorn, Peter Bühlmann, Sandrine Dudoit, Annette Molinaro, and Mark Van Der Laan. Survival ensembles. Biostatistics, 7(3):555-775 2006.
[38] Jared L Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC medicul reseurch methodology, 18(1):1-12 2018.
[39] Stephane Fotso. Deep neural networks for survival analysis based on multi-task framework. arXiv preprint /801.00801.005 05512, 2018
[40] Tim Pearce, Onng-Hyeon Jeong, Jun Zhu, et al. Censored quantile regression neural networks for distribution-free survival analysis. In Advances in Neural Information Processing Systems, 2022.
[41] Frederick N Fritsch and Judy Butland. A method for constructing local monotone piecewise cubic interpolants. SIAM journal on scientific and statistical computing, 5(2):300-304, 1984.
[42] Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction. Foundations and Trends@ in Machine Leurning, 16(4):494-51, 2023.
[43] Shi-ang Qi, Neeraj Kumar, Jian- Yi Xu, Jaykumar Patel, Sambasivarao Damaraju, Grace Shen- Tu, and Russell Greiner. Personalized breast cancer onset prediction from lifestyle and health history information. Plas one, 17(12))e0279174, 2022.
[44] Shi-ang Qi, Weijie Sun, and Russell Greiner. SurvivalEVAL: comprehensive open-source python package for evaluating individual survival distributions. In Proceedings of the AAAI Symposium Series, volume 2, pages 453-457, 2023.
[45] Davide Chicco and Giuseppe Jurman. Machine learning can predict survival of patients with heart failure from serum creatinine and ejection fraction alone. BMC medical informatics and decision making, 20: 10,1, 2020.
[46] Heart Failure Clinical Records.
UCI Machine Learning Repository, 2020.
DOL:
[47] Terry Therneau and Patricia Grambsch. Modeling Survival Data: Extending The Cox Model, volume 48. Springer, 01 2000. ISBN 978-1-4419-3161-0, doi: 00.1007/978--4455-22222 1-7577-39--77
[48] Terry M Therneau. A Package for Survival Analysis in R, 2024. URL https [tttps://Of ://CRAN R- p-rroject.ooo org/ http:/paaakg--sriiaaalllll R package version 3.6-4.
[49] David W Hosmer, Stanley Lemeshow, and Susanne May. Applied survival analysis. John Wiley & Sons, Inc., 2008
[50] John N Weinstein, Eric A Collisson, Gordon Mills, Kenna R Shaw, Brad Ozenberger, Kyle Ellrott, Ilya Shmulevich, Chris Sander, and Joshua M Stuart. The cancer genome atlas pan-cancer analysis project. Nature genetics, 45(10): 2013.
[51] Patrick Royston and Douglas Altman. External validation of cox prognostic model: principles and methods. BMC medical research methodology, 13:1-15, 2013.
13

---

[52] Stephane Fotso et al. PySurvival: Open source package for survival analysis modeling. 2019-
URL wwwp://///ww..p.uvvviia....aa
[53] Angela Dispenzieri, Jerry A. Katzmann, Robert A. Kyle, Dirk R. Larson, Terry M. Therneau, Colin L. Colby, Raynell J. Clark, Graham P. Mead, Shaji Kumar, L. Joseph Melton, and S. Vincent Rajkumar. Use of nonclonal serum immunoglobulin free light chains to predict overall survival in the general population. Mayo Clinic Proceedings, 87(6):517-523, 2012.
ISSN 0025-6196.
[54] William A Knaus, Frank E Harrell, Joanne Lynn, Lee Goldman, Russell S Phillips, Alfred F Connors, Neal V Dawson, William Fulkerson, Robert M Califf, Norman Desbiens, al. The support prognostic model: Objective estimates of survival for seriously ill hospitalized adults. Annals of internal medicine, 122(3):191-203; 1995.
[55] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horn, Leo Celi, Anthony, and Roger Mark. MIMIC-IV (version 2.0). PhysioNer (2022). http:://doi.org/10.13000
14., 2022.
[56] Sebastian Pölsterl. scikit-survival: A library for time-to-event analysis built top of scikit- learn. Journal of Machine Learning Research, 21(212):1-6, 2020. URL http: /ttt:://jmlr org/
papers/v21/20-17 729 html
[57] Terry M Therneau. A Package for Survival Analysis in R, 2024. URL https://CRAN R-projject wtg.package-ssrrviiiii.
R package version 3.6-4.
[58] Alistair HW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
[59] Cameron Davidson-Pilon. lifelines, survival analysis in python, January 2024. URL https
/dbol.orgg/1..5588//eedoo 10456828
[60] David R Cox. Partial likelihood. Biometrika, 62(2):269-276, 1975.
[61] Ping Jin. Using survival prediction techniques to learn consumer-specific reservation price distributions. Master's thesis, University of Alberta, 2015.
[62] Norman E. Breslow. Analysis of survival data under the proportional hazards model. Interna- tional Statistical Review/Revue Internationale de Statistique, pages 45-57, 1975.
[63] Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series (The Statistician). 32(1-2):12-2,, 1983.
14

---

A Calibration in Survival Analysis
Distribution calibration (or simply "calibration") examines the calibration ability across the entire range of ISD predictions [3].
This appendix provides more details about this metric.
A.1 Why the survival probability at event times should be uniform?
The probability integral transform 13 states: if the conditional cumulative distribution function
(CDF) Ftt x;) is legitimate and continuous in t for each fixed value of æ,, then F(t æ;)) has standard uniform distribution, Vo.1j. Since Stt æ;) F(t x4), then S(t xx) ~ M(0.1.
Predictions
Histograms
P-P Plots
Survival Prob 600
Survival Prob.
Observed Prob.
Individual
ukx (ith
3
5
1/3 Predicted Prob.
Counts
Figure 5: Individual distribution calibration, illustrated using 3 bins separated and 3. The curve is an oracle's true ISD S(t a;). The stars represent 15 realizations of t xq The vertical coordinates of the star represent the time and the horizontal coordinates of the star are the survival probability at observed time (ee"" æ;;..
A.2 Handling censorship
Definition 2.T defines the marginal calibration for the uncensored dataset. this section, we expand the definition to any dataset with censored instances. Note that Haider et al. [3] proposed the following method, here we just reformulate their methodology to the language in this paper, for better presentation purpose.
Given uncensored subject, the probability of its survival probability at event time in a probability interval of [P1,P2] is deterministic, as:
IP ( ($(e x;)) p21.:2110: 6 [[e(e æ.) E [21, 6, -].
For censored subjects, because we do not know the true event time, so there no way we can know whether the predicted probability is within the interval not. We "blur" the subject uniformly to the probability intervals after the survival probability censored time S(c; xx,) 3
P (S(es w:) [2), - 0) ((S(e xx) E [21,)2), =0)
P(o: = 0)
P (S(ec x.) [21, (x)) S(c; xx))
P(S(ez a.) < S(c mx))
P(((e x.) [21. (1;) S(c: S() x.,) p2)
P(S(e, 2;) x;) S( x;))
P (Š(e: wx) [21, xx) S(c; x.), S(c: x.) p2))
PPS(ee (x) < 5(c (x)))
P (S(e) xx)) [2), S(e mx) $(c; æ;), S(c; x;) s pT)
P(Š(e: mx) 5(c; xx)))
15

---

P (pr s S(e; x;) s p2) ((S(cc x0.) 2
S(4i mi)
P( s S(e; x.) S(c: xx)) P ($(c, x.) Pel) P(Ø) +
S(1 az))
S(1 x;)
(p2 -1- x.) 2 p2] (5(t. x.)) - p1) 1 x.) p2]
S(t x;)
where the decomposition of probability in the second last equality is because of the conditional independent censoring assumption. Therefore, for the entire dataset, considering both uncensored and censored subjects, the probability
(S(e,) x;) e [pI, Pel.ieT))
Eee [6. (G-P(((())))) ax)) [p1, P2] p2]]):))
(8)
Therefore, given the above derivation, we can provide a formal definition for marginal distributional calibration for any survival dataset with censorship.
Definition A.1 (Marginal calibration). For survival dataset, model has perfect marginal calibration i [PL, p2] = [0. 1].
PP (S((e: æ;)) [21, iE = P Pl.
where the probability calculated using
B
Algorithm
Here present more details for the algorithm: Conformalized survival distribution using Individual survival Probability at Observed Time (CiPOT). The pseudo-code presented in Algorithm !
Algorithm 1 CiPOT
Input: Dataset D, testing data with feature 2n+1 E Rd, survival model M, predefined percentile levels p= {P1, .)...... repetition parameter Output: Calibrated ISD curve for In+1
1: Randomly partition D into a training set Dimin and a conformal set Deon
2: Train survival model M using Ditain
3: Make ISD predictions (S wx))
4: (Optional) apply the interpolation and extrapolation to make the ISDs continuous.
5: Initialize conformity score set: Р Ø, and pseudo-uniform array: un = [r/R]
6: for E 2 do
7: Ti.M (t: ax))
>> calculate the iPOT value
8: if 8; = then
9:
Пм I' Y.M ug}
sample R times from S (1/12.)]
10: else
11: M' = TM (H.M. HiM)
> repeat the iPOT value, times
times
12: end if
13:
end for
V peP, A,P, (n-1)= = Syy (Percentile(;; I'M)
St((1) xn+1) = inf(p: 5 mat1) 1) >> transform the inverse ISD into a ISD curve
Our method is motivated by the split conformal prediction 25 The algorithm starts by partitioning the dataset in a training and conformal set (line 1 in Algorithm I]. Previous methods [1] 12 recommend mutually exclusive partitioning, i.e., the training and conformal set must satisfy D train Deon = D and Dimin n Deon Ø. However, this can cause one problem: reducing the training set size, resulting in underfitting for the models (especially for deep-learning models) and sacrificing discrimination performance. Instead, we consider the two partitioning policies proposed by Qi et al. 8 (1) using the validation set as the conformal set, and (2) combining the validation and training sets the conformal set.
16

---

Predictions
Histograms
P-P Plots
Survival Prob.
Survival Prob.
Observed Prob.
Raw Outputs
Marginal
2
1/3
2/3
Days
(b)
Counts
Predicted Prob.
Survwai Probabiity
Survwal Probabillt
Observed Prob.
Nargnal
Survivl
CSD
2
2
3
2
2/3
Days
Days
Counts
(e)
Predicted
x=
Survival Prob.
Sunvival Prob.
Observed Prob.
GiPOT
Survval
2
2
3
1/3
2/3
(h)
Daye
Days
Counts
(4)
Predicted Prob.
Figure 6: Comparison of CSD and CiPOT. The first mirrors the original Figure 2 first row, and the third row reflects its second row. CSD steps include: (d) discretizing the ISDs into predicted percentile times (circles), and calculating conformity scores using the horizontal differences between circles and stars (true outcomes): then (e) adjusting the circles horizontally via conformal regression.
Another algorithm detail is interpolation and extrapolation (line Algorithm D for ISDs. This optional operation is required for non-parametric survival algorithms (including semi- parametric [26] and discretize time/quantile models [28 22 For interpolation, we can use either linear piecewise cubic Hermit interpolating polynomial (PCHIP) [41] which both maintain the monotonic property the survival curves. For extrapolation, we extend ISDs using the starting point
[o, 1] and ending point [feed. S (fend x-
We also include side-by-side visual comparison of our method to CSD 8 in Figure Both CSD and CiPOT approaches use the same general framework conformal prediction, but they differ in how to calculate the conformity score and adjust the positions of the predictions. Note that in Figure 6 the horizontal positions of the circles remain unchanged for CSD while the vertical positions of the circles remain unchanged for CiPOT. This unique approach for calculating the conformity score, and the downstream benefit for handling censored subjects (thanks to the conformity score design), together provide theoretical guarantee for the calibration, elaborated in the next section.
C Theoretical Analysis
This appendix offers more details on the theoretical analysis in Section 3.4
C.1 More the marginal and conditional calibration
This section presents the complete proof for Theorem 3.1 and Theorem
3.2
For the completeness, start by restating Theorem
17

---

Theorem C.I (Asymptotic marginal calibration). If the instances in D are exchangeable, and follow the conditional independent censoring assumption, then for new dataset pt P2 [0. 1].
P2 P1
(P(SM(nnn / (mnr)) (21,
P2 P
(9)
Doun
Proof. This proof is inspired by the proof in Theorem D.I and D.2 from Angelopoulos et al. [42]. the main difference is that our conformity scores are different, and we also have censoring subjects.
Given the target percentile levels P1 and P2. As we remember in the CiPOT procedure, essentially vertically shift the prediction at Percentile p; I) to (see (4). Therefore, the condition in (9)
(tut Øn-1) [21.
can be transformed into this equivalent condition
(aat (n-1))
[Percentile(p:::MM Percentile(p2: [M)]
As we recall, the conformity score set consists of the iPOT score for each subject in the conformal set. Let consider the easy case of an uncensored dataset, then
M = (SM x.))
Without loss of generality, we assume that the conformity scores in I' are sorted. This assumption purely technical as it aims for simpler math expression in this proof, i.e. (S x1) (M22)) 2²2 < ... SM ( (pppoon ppon.. Therefore, by exchangeability assumption (the order between subject 1 and subjects i E Ieea do not matter), the iPOT score of subject n is equally likely to fall into any of the [Deog] + intervals between the [0, 1]. separated by the |Doon| conformity scores. Therefore,
TP (1 xn+1) [Percentile(p:: [,), (ercentile(p2::TMM)) M))) [(2 - p1) 1)]
(iDoon) 1)
This value is always higher than P2 Pt. and if the conformity scores in I'M do not have any tie can also see that the above equation is less than P2 P1
D
Now let's consider the censored case. Based on the probability integral transform, for a censored subject j. the probability that its iPOT value falls into the percentile interval, before knowing its censored time, is (from here we omit the subscript M for simple expression)
PP (S(e)) sp) P. and P ((p S(e,) æ;) p2) =P P1
'This is a technical assumption in conformal prediction. This assumption is easy to solve in practice because users can always add a vanishing amount of random noise to the scores to avoid ties.
18

---

Now given its censored time Cj, we can calculate the conditional probability
P P( s S(e, Sz)) =
xÿ) s p2 S(e, æ3) s S(c,) x2))
P(n See 1.) s s2, S(e, m2) Sc; |
P(S(e) B3) S(c) x.)))
P(l s Ste) 1.) S p2, S(e) (1)) S(c; S(c) xz) mm)
S(c, zz))
p(p s S(e, æ3)) S p2. S(e,
23) S S((, x)). S(c, 23) s P2)
S(c; xj)
((r s S(ee m.)) s p2: S(e; m.)) ((c)) 5.)) x.))) p2)
$(c, x.))
P ((1 s S(e, (z)) s S(cj xz). & S(c, æ.)) s P2)
0
S(G, mz)
S(cj x3))
P(m S(ee) wz)) S p2; S(gj (1)) p2)
S(c, x3)
($(e, - [p s S(e, [t)) S p2] + (p2 (12) xx,) > p2]
$(c; x3)))
where the probability decomposition in the last equality is because the conditional independent censoring assumption, i.e., ej cj æy. This above derivation means the S(e, xx)) follows the uniform distribution (c, Therefore, if we do one sampling for each censored subject using (1) the above proof asymptotically converges to the upper and lower bounds for uncensored subjects, for any survival dataset with censorship.
For the conditional calibration, we start by formally restating Theorem 3.2
Theorem C.2 (Asymptotic conditional calibration). With the conditional independent censoring assumption, if we have these additional three assumptions:
(i) the non-processed prediction S(t x:)) is also a consistent survival estimator with:
p E sup ($(f(2z)) S(t xx)) $ Ta ons s.t. 7n =,== (10)
(ii) the inverse ISD estimation S (t) x.)) is differentiable,
(ii) there exist some M such that inf, a (cle..)) -
then the CiPOT process can asymptotically achieve the conditional distribution calibration:
Š(1 Ta+1) S(L (a+1) ) + op(1).
Proof. In order to prove this theorem, it enough to show that (i) š(t xn-1) = S((t) xn-1)) + or (1), and (ii) S(L xx+1))= S(L xn-11) (1).
The second equality is obvious to see under the consistent survival estimator assumption ITP.
Now let's focus on the first equality. We borrow the idea from Lemma 5.2 and Lemma 5.3 in Izbicki al. (22] It states under the consistent survival estimator assumption, can have S- "PPeeeettil((;;; P' (m++)) S-1 (pl Ta+1) o1().
The only difference between the above claim and the original claim in Izbicki et al. 22 that they use the CDF while we use the survival function, i.e., the complement of the CDF.
19

---

According to 4. the above claim can be translate to (pl En+1) = = Ix+1) + (1) Then if ( xn+1) SS-
- (pl [o11 = 0p()) if the is differentiable, we can have
[$(t) mn++)) 5tt (n++)] = o,(1) inf d (pl x:;)
d
Therefore, if there exist some M such that inf, d d5 (pkemn)) proved.
M-1
We can get the first equality
p
For the censored subjects, we can use the same reasoning and steps in Theorem conditional independent censoring assumption. This will finish the proof.
3.1 under the
We are aware that linear interpolation and extrapolation will make the survival curves nondiffer- entiable. Therefore, recommend using PCHIP interpolation [41] for CiPOT. And extrapolation normally does not need to apply for the survival prediction algorithm because the iPOT value SM (4 æz) can be obtained within the curve range for those methods. However, for quantile-based algorithms, sometimes need extrapolation. And recognize this future direction improve.
C.2 More the monotonicity
CSD is a conformalized quantile regression based [|]] method. It first discretized the curves into quantile curve, and adjusted the quantile curve at every discretized level [8] Both the estimated quantile curves (q(p)) and the adjustment terms (adj())) are monotonically increasing with respect the quantile levels. However, the adjusted quantile curve calculated as the original quantile curve minus the adjustment- - is no longer monotonic. For example, if q (50%) = 5 and ((60% 6, with corresponding adjustments of adj(50%) 5 --2 2 at 50% and 6 4 at 60%, demonstrating non non-monotonicity. For a detailed description of their algorithm, readers are referred Qi et al. X
2 and a60)) 4, the post-CSD quantile curve will be
However, CiPOT has this nice property. Here we restate the Theorem C.4
Theorem C.3, CiPOT process preserves the monotonic decreasing property of the ISD, s.t.,
Vi V VasbelR,
(m m;) (bb xx)--
(11)
Proof. The proof of this theorem is straightforward. The essence of the proof lies the monotonic nature of all operations within CiPOT.
First of all, the percentile operation Percentile(p: MM) is monotonic function, i.e., for all Pr p2, I ()++ Percentilc((2;;1
Second, because the non-post-processed ISD curves Š(1 w:) are monotonic. Therefore, the inverse survival function (p æ;)) is also monotonic. Therefore, after the adjustment step as detailed in for all Pi P2, it follows that (p #n.1) S5 P2 1)..).
Lastly, by converting the inverse survival function 5 (pl xm+1) back to survival function (t(()) 1. the monotonicity is preserved.
These steps collectively affirm the theorem's 's proof through the intrinsic monotonicity of the operations involved in CiPOT.
C.3 More the discrimination performance
This section explores the discrimination performance of CiPOT in the context of survival analysis. Discrimination performance, which is crucial for evaluating the effectiveness of survival models, is typically assessed using three key metrics:
Harrell's concordance index (C-index)
Area under the receiver operating characteristic curve (AUROC)
Antolini's 's time-dependent C-index
20

---

We will analyze CiPOT's performance across these metrics and compare it to the performance of CSD. The comparative analysis aims highlight any improvements and trade-offs introduced by the CiPOT methodology.
Harrell's C-index C-index is calculated the proportion of all comparable subject pairs whose predicted and outcome orders are concordant, defined as
Σi.err 6,- 1[t; t,] 1[, n7,] Eijezua 5; [[t < t,]
(12)
where ij denotes the model's predicted risk score subject i, which can be defined as the negative of predicted mean/median survival time (E,[S(t) æx)] or $0. xx)).
CSD has been demonstrated to preserve the discrimination performance of baseline survival models, as established in Theorem 3.1 byin Qi al. [8. In contrast, CiPOT does not retain this property. To illustrate this, we present counterexample that explains why Harrell's C-index may not be maintained when using CiPOT.
Survival Probability
0.75
0.75
0.5
0.5
0.25
Peroentile(50%;
0.25
0
0
100
150
(a)
(b)
Figure 7: Counter examples of (a) Harrell's C-index performance is not preserved by CiPOT; and (b)
AUROC performance is not preserved by CSD.
As shown in Figure a). two ISD curves cross at a certain percentile level 25% < p < 50%. Initially,
the order of median survival times (where the curves cross 50%) for these curves indicates that patient precedes patient B. However, after applying the adjustment as defined in (4) which involves
vertically shifting the prediction from empirical percentile level Percentile( (50%; MM) desired percentile level 50%. The order of the post-processed median times for the two curves (indicated by the hollow circles) is patient B ahead of patient A. That means this adjustment leads reversal in the order of risk scores, thereby compromising the C-index.
The area under the receiver operating characteristic curve (AUROC) is a widely recognized metric for evaluating discrimination in binary predictions. Harrell's C-index can be viewed as a special case of the AUROC (2). if we use the negative of survival probability at specified time t* S((") ax) the risk score.
AUROC
The primary distinction lies in the definition of comparable pairs. In Harrell's C-index, comparable pairs are those for which the event order is unequivocally determined. Conversely, for the AUROC evaluation at time t", comparable pairs are defined as one subject experiencing an event before t* and another experiencing it after t². This implies that for AUROC at t", pair of uncensored subjects both having event times before (or both after) t", is not considered comparable, whereas for the C-index, such pair is indeed considered comparable.
The AUROC can be calculated using:
AUROC(Š,t)) En,jezis d, 1[t st 1[,, > #* 1[ xx)) < S(t") |x,]]
(13)
Enjer 6, 1t, S ++++ 1[t, ++]
From this equation, because the part of 6; [4 2 (²) 1[4 ["] is independent of the prediction (it only relates to the dataset labels). As long as the post post-prcesssing does not change the order of the survival probabilities, we can maintain the same AUROC score.
21

---

Theorem C.A. Applying the C&POT adjustment to the ISD prediction does not affect the relative order of the survival probabilities at any single time, therefore does not affect the AUROC score of the model. Formally, V i.3 I and V 1" Th,, given
(t xe.) < <Sm(t" (z)),
we must have
(t x21) < Sm((t xz)).
Here 5M is calculated using]
in Section 3.2
Proof. The intuition is that if we scale the ISD curves vertically. Then the vertical order of the ISD curves at every time point should not be changed.
Formally, we first represent (t" | x;)) by p7, and represent Smi(t xz)) by Pj. Then, by applying (5) and then (4), we can have
( x.)) S (Perceetile x;))
(D) xz)) S (Percentile mzy)
where Percentile(2f; [M) is the original predicted probability at t", ie,, Syi(" | |x;) Percentile(ft::
Because the Percentile operation and inverse functions are monotonic (Theorem 3.3 theorem holds.
therefore, this
CSD adjusts the survival curves horizontally (e.g., along the time axis). Hence, while the horizontal order of median/mean survival times does not change as proved in the Theorem 3.1 from Qi et al. g the vertical order, represented by survival probabilities, might not be preserved by CSD.
Let's use a counter-cxample to illustrate our point. Figure b)) shows two ISD predictions S(t x and Š(t æR) for subjects A and B. Suppose the two ISD curves both have the median survival time at L 100, and the two curves only cross once. Without loss of generality, we assume S(t" xx)= < Stt" xB) holds for all t* 100 and S(t" (<t) S(t" 2B) holds for all t" 100. Now, suppose that CSD modified the median survival time from t = 100 to 150 for both of the predictions. Then the order between these two predictions at any time in the range of [100, 150]
is changed from S(t" AA) > S(t* xB) to S((* A) < S(t" xg).
It is worth mentioning that in Figure 2 blue curve is partially at the top in (a). intersecting an orange curve around 1.7 days, while the orange curve consistently at the top in (e). This might raise concerns that the pre- and post-adjustment curves do not maintain the same probability ordering at every time point, suggesting potential violation. In fact, this discrepancy arises from the discretization step used in our process, which did not capture the curve crossing at 1.7 days due to the limited number of percentile levels (2 levels at 3 and } used for simplicity in this visualization. The post-discretizatoon positioning of the orange curve above the blue curve in Figure 2 e) does not imply that the post-processing step alters the relative ordering of subjects. Instead, reflects the limitations of using only fewer percentile levels. Note that other crossings, such as those at approximately 1.5 and 2.0 days, are captured. In practice, we typically employ more percentile levels (e.g., 9, 19, 39, or 49 as in Ablation Study #2 Appendix (E.6). which allows for more precise capture of all curve crossings, thereby preserving the relative ordering.
Antolini's C-index Time-dependent C-index, Ctd,, is modified version of Harrell's C-index 32 Instead of estimating the discrimination over the point predictions, it estimates the discrimination over the entire curve.
Ctd Cid(()) Er 8G 1[t: < kttl. 1[1(t, (x)) < S(tt x1)1
ΣnjeeDi d. 1[t, <t,]
Compared with (12). the only two differences are: the risk score for the earlier subject n. is represented by the iPOT value S(t; æx), and the risk score for the later subject ny is represented by the predicted survival probability for at t,, S(t, æ.)).
22

---

Baseline
CSD
CIPOT
P.P Plots
10
1.0
1.0
1.0
Survival Prokabiityy
Surviva Probabilty
Surevel Probabilly
Obsrved Probobiity
CSD
Baseline
CSD-POT
0.6
0.6
0.8
0.8
044
0.4
0.4
0.4
0.0
0.0
0.0
0.0
6
2000
4000
0
2000
4000
0
2000
4000
0.00
0.5
0.50
077
1.0
Time (8)
Time (b)
Time
Predioled Probebiily (d)
Figure 8: An real example using DeepHit as the baseline, on the FLCHAIN dataset. The predicted curves in the panels are for the same 4 subjects in the test set. The dashed green line represents the KM curve on the test set. (a) Non post-proe post-processed baseline. (b) CSD method on DeepHit. (c) CiPOT method on DeepHit. (d) P-P plots comparison of the three methods.
Ctd can also be represented by the weighted average of AUROC over all time points (Equa- tion 7 and proof in Appendix 1 from Antolini et al. (3 Suppose given a series of time grid .t..............................
Crd (R)($) ΣK AUROC(S 1x) (tk))
(14)
Σx-o w(x)
where
w((x) En &i 1[t st"] L[t, >t]]
(15)
Eijezue 1[t, <ty]
The physical meaning of ((+) measures the proportion of comparable pairs l over all possible pairs. Therefore, we can have the following important property of our CiPOT.
Lemma C.5. Applying the CiPOT adjustment to the ISD prediction does not affect the time- -ependenn C-index of the model.
Because the ((x)) in 15 is independent of the prediction, and also given Theorem AUROC(S, tx) AUROC(S, tx), easily have (5) (S) from the formula in 14
Proof.
As in the previous discussion, CSD does not preserve the vertical order represented by survival probabilities. Therefore, is natural to see that CSD also does not preserve the Ctd performance.
C.4 More on the significantly miscalibrated models
Compared to CSD, our CiPOT method exhibits weaker performance on the Deeplfit baselines. This section explores the three potential reasons for this disparity.
First of all, DeepHit tends to struggle with calibration for datasets that have high KM ending probability [43] and has poor calibration compared to other baselines for most datasets 87. This is because the Deepřlii formulation assumes that, by the end of the predefined mmax. every individual must already have had the event. Hence, this formulation incorrectly estimates the true underlying survival distribution (often overestimates the risks) for individuals who might survive beyond Imax-
Furthermore, apart from the standard likelihood loss, DeepHit also contains ranking loss term that changes the undiffcrentiated indicator function in the C-index calculation in (12) with an exponential decay function. This modification potentially enhances the model's discrimination power but compromises its calibration.
Lastly, Figure B(a) shows an example prediction using DeepHit on the FLCHAIN (72.48% censoring rate with KM curve ends 68.1666. 16%). The solids curves represent the ISD prediction from Deepllit for 4 randomly selected subjects in the test set. And the dashed green curve represents the KM curve for the entire test set. evident that DeepHit tends to overestimate the subjects' risk scores (or underestimate the survival probabilities), see Figure d). Specifically, at the last time point (t 5215),
23

---

KM predicts that most of the instances (68.16%) should survive beyond this time point. However, the ISD predictions from DeepHit show everyone must die by this last point (Speptii (5215 æ;) = 0 for all x see Figure aa)). This clearly violates the unbounded range assumption proposed in Section 3.1 which assumes (L xx)) > for 2 O. This violation the main reason why CiPOT exhibits weaker performance the DeepHit baseline.
CSD can effectively solve this overestimate issue (Figure as shift the curves horizontally, i.e. no upper limit for right-hand side for shifting. CiPOT, on the other hand, scale the curves vertically. In such case, the scaling must be performed within the percentile [0. 1]. Furthermore, CiPOT does not have any intervention for the starting and ending probability (p 1 and p 0) of the curves. So no matter how the post-process changes the percentile in the middle of the curves, the starting and ending points should not be changed, just like the curves in Figure (jc)) whereas the earlier parts of the curve are similar as CSD's, the last parts gradually drop to O.
Consequently, while CiPOT significantly improves upon DeepHit, as shown in Figure d), it still underperforms compared CSD when dealing with models that are notably miscalibrated like DeepHit.
D Evaluation metrics
We use Harrell's C-index [T] for evaluating discrimination performance. The formula is presented in (12). Because we are dealing with a model that may not have proportional hazard assumption, therefore, as recommended [2]. we use the negative value the predicted median survival time as the risk score, i.e., fi: = S- (0.5| x;).
The calculation of marginal calibration for a censored dataset is presented in Appendix A and calculated using (8) and (6).
Conditional calibration, Calws. is estimated using Here we present more details on the imple- mentation. First of all, the evaluation involves further partitioning the testing set into exploring and exploiting datasets. Note that this partition does not need to be stratified (wrt to time f, and event indicator 6,). Furthermore, for the vectors v, we sampling M i.i.d. vectors on the unit sphere in Rd Generally, we want to select a high value for M to enable all possible exploration. For small or mediumm datasets, we use
1000. However, due to the computational complexity, for large datasets, we gradually decrease the value of M [100, 1000] to get an acceptable evaluating time. set k 33% in (7) for finding the Sy.a.b. that means we want to find a worst-slab that contains least 33% of the subjects in the testing set.
Integrated Brier score (IBS) measures the accuracy of the predicted probabilities over all times. IBS for survival prediction is typically defined as the integral of Brier scores (BS) over time points:
tax
IBS(S; tmax) = tmax
BS(t) dt,
fo frn
55 1 [14 S t]
5() xx)² 1[4 (]-(1-11111-11111 S(t mx))²)
dt,
Tiest
Emax
G(t,)
G(t)
where G(t) is the non-censoring probability at time t. It is estimated with KM on the censoring distribution (flip the event indicator of data), and its reciprocal 0(0) is referred to as the inverse probability censoring weights (IPCW). fmax is defined as the maximum event time of the combined training and validation datasets.
Mean absolute error calculates the time-to-event precision, i.e., the average error predicted times and true times. Here use MAE-pseudo observation (MAE-PO) (5) for handling censorship in the
24

---

calculation.
MAEpo ((t)bx)) )
Σ
wi x (1 (1) epo I d Eg}
Σ Everie Wi ieIiol
N E, [SKM(IWO) (NV 1) E. (KM(I (t)
if 8,=0, otherwise,
where (ro
t
Wi 1 1- 3SKM(
if 65 =,
and
otherwise.
Here SKM(Z=1) (t) represents the population level KM curve estimated the entire testing set Z and SKM(T (t) represent the KM curves estimated on all the test subjects but exclude subject i. C-index, margin (also called D-cal), ISB, and MAE-PO are implemented in the Surviva1EVAL package 44 For Calus, please see our Python code for implementation.
E
Experimental Details
E.1
Datasets
We provide brief overview of the datasets used in experiments.
In this study, we evaluate the effectiveness CiPOT across 15 datasets. Table summarizes the data statistics. Compared to the datasets used in [8] we have added HFCR, WHAS, PaM, Churn, FLCHAIN, Employee, and MIMIC-IV IV. Specifically, we use the original GBSG dataset, as opposed the modified
version by Katzman et al. (38 used in S which has higher censoring rate and more features. For
the rest of the datasets, we employ the same preprocessing methods as Qi et al. [8] see Appendix of their paper for details about these datasets. Below, describe these newly added datasets:
Table Key statistics of the datasets. We categorize datasets into small, medium, and large, based on the number of instances, using thresholds of 1,000 and 10,000 instances. The bolded number represents datasets with a high percentage of censorship (2 60%) or its KM estimation ends high probability (2 50%). Numbers in parentheses indicate the number of features after one-hot encoding.
Dataset
#Sample
Censor Rate
Max
#Feature
KM End Prob.
HFCR 45
46
299
67.89 %
285
11
57.57 %
PBC 47
418
61.48 %
4,795
17
35.34%
WHAS 49
500
57.00%
2,358
14
0%
GBM [50 8
595
17.23%
3,881
8 (10)
0%
GBSG [1] 48
686
56.41%
2,659
8
34.28%
PdM [52]
1,000
60.30%
93
5
0%
Churn [52
1,958
52.40%
12
12 (19)
24.36%
NACD 3
2,396
36.44%
84.30
48
12.46%
FLCHAIN 53
48
7,871
72.48%
5,215
(23)
68.16% %
SUPPORT 54
9,105
31.89%
2,029
26 (31)
24.09%
Employee
[5
11,991
83.40%
10
8 (10)
50.82% %
MIMIC-IV [55
35
38,520
66.65%
4404
93
0%
SEER-brain
8
73,703
40.12%
227
10
26.58%
SEER-liver
8
82,841
37.57%
227
14
18,01% %
SEER-st.oach
100,360
43.40%
227
14
28.23%
contains medical records 299 patients with heart failure, aiming to predict mortality from left ventricular systolic dysfunction. This dataset can be downloaded from UCI Machine Learning Repository [46].
Heart Failure Clinical Record dataset (HFCR.
45
Worcester Heart Attack Study dataset (WHAS) 49 contains 500 patients with acute myocardial
infarction, focusing on the time death post-hospital admission. The data was already post-processed and can be downloaded from the scikit-sune scikit-survival package (56).
25

---

Predictive Maintenance (PdM) contains information on 1000 equipment failures. The goal is to predict the time to equipment failure and therefore help alert the maintenance team to prevent that failure. It includes 5 features that describe the pressure, moisture, temperature, team information (the team who is running this equipment), and equipment manufacturer. We apply one-hot encoding on the team information and equipment manufacturer features. The dataset can be downloaded from the PySurvives package [52].
The customer churn prediction dataset (Churn) focuses on predicting customer attrition. We apply one-hot encoding on the US region feature and exclude subjects who are censored at time 0. The dataset can be downloaded from the PySurvival package [52].
Serum Free Light Chain dataset (FLCHAIN) is stratified random sample containing half of the subjects from a study on the relationship between serum free light chain (FI.C) and mortality [53]. This dataset is available in R's survival package [57]. Upon downloading, we apply few preprocessing steps. First, we remove the three subjects with events at time zero. We impute missing values for the "creatinine" feature using the median of this feature. Additionally, we eliminate the chapter feature (a disease description for the cause of death by chapter headings of the ICD code) because this feature is only available for deceased (uncensored) subjects hearectts-enne,,,, knowing this feature will be equivalent to leaking the event indicator label to the model.
Employee dataset contains employee activity information that can used to predict when an employee will quit. The dataset can be downloaded from the PySurvival package [52]. It contains duplicate entries; after dropping these duplicates, the number of subjects in the dataset is reduced from 14,999 to 11,991. We also apply one-hot encoding to the department information.
IV database (58] provides critical care data information for patients within the hospital. We focus on a cohort of all-cause mortality data curated by [35], featuring patients who survived at least 24 hours post-ICU admission. The event of interest, death, is derived from hospital records (during hospital stay) or state records (after discharge). The features are laboratory measurements within the first 24 hours after ICU admission.
German Breast Cancer Study Group (GBSG) [1] contains 686 patients with mode-positive breast cancer, complete with prognostic variables. This dataset is available in R's survival package [57]. While the original GBSG offers higher rate of censoring and more features, X utilized modified version of CBSG from [8, merged with uncensored portions of the Rotterdam dataset, resulting in fewer features, a lower censor rate, and larger sample size.
Figure 9 shows the Kaplan-Meier (KM) estimation (blue curves) for all 15 datasets, alongside the event and censored histograms (represented by green and orange bars, respectively) in a stacked manner, where the number of bins is determined by the Sturges formula: [log([D)) + 17.
E.2 Baselines
In this section, we detail the implementation of the seven baseline models used in our experiments, consistent with Qi et al. 8
Accelerate Failure Time (AFT) 36 with Weibull distribution is a linear parametric model that uses a small 12 penalty on parameters during optimization. It is implemented in lifelines packages [59]
Gradient Boosting Cox model (GB) (37 is an ensemble method that employs 100 boosting
stages with a partial likelihood loss 60 for optimization and 100% subsampling for fitting each base learner. The model implemented
teikki-survival packages (56).
Neural Multi-Tass Logistic Regression (N-MTLR) [39 is discrete-ting model which is an NN-extension of the linear multi-task logistic regression model (MTLR) (28). The number of discrete times is determined by the square root of the number of uncensored patients. We use quantiles to divide those uncensored instances evenly into each time interval, as suggested in 61 3. We utilize the N-MTLR code provided in Qi et al. 8
DeepSurv
38 is NN-extension of the Cox proportional hazard model (CoxPH) 26 To make ISD prediction, we use the Breslow method [2] to estimate the population-level bascline hazard function. We utilize the DeepSurv code provided in Qi et al. (8).
26

---

HFCR
PBC
WHAS
1.0
190
Survivl Probabilit
0.8
Censored
50
Survival Probability
0.8
Censored
00
0.8
Consored
100
40
0.6
30
Courtte
0.6
Counts
0.8
Count
0.4
0.4
0.4
10
0.-
20
0.0
0.0
0.0
100
200
1000
2000
3000
4000
0
500
1000
1500
2000
Time
Time
Time
GBM
GBSG
PdM
1,0
Event
300
1.0
Event
120
Survivel Prebability
Censarad
250
Survival Probability
0.8
Censered
100
0.8
Censored
150
0.6
200
Counts
0.6
Counts
150
60
100
0.4
-1
0.2
50
02
0.2
0.0
Time
Time
Churn
NACD
FLCHAIN
Survivel Prebability
0.8
Cansrred
400
Survival Probability
0.8
Cansond
600
Probabilitty
0.88
Cantaoond
Counte
Counts
1000
0.1
200
0.4
0.4
500
0.0
0.0
0.0
0.0
2.5
5.0 Time
7.5
10.0
20
40
60
80
0
2000
4000
Time
Time
SUPPORT
Employee
MIMIC-IV
1,0
Event
4000
1.0
Event
5000
1.0
Event
Survival Probability
-3000
Survival Probabbiity
0.8
Canscred
4000
Probabillty
Censored
30000
0.5
Counts
0.6
3000
0.6
20000
Counts
2000
2000
0.2
1000
02
1000
0.2
10000
0.0
Time
Time
Time
SEER-brain
SEER-liver
SEER-stomach
30000
40000
Surviel Probabiltt
0.8
Cersored
26000
Survival Probability
0.8
Canacred
Survwal Probability
0.8
Cansored
40000
30000
20000
Counts
30000
Surveel
0.4
0.4
0.4
20000
10000
5000
10000
10000
0.0
0.0
0
0.0
50
130
150
200
50
100
150
200
0
50
100
150
200
Time
Time
Time
Figure 9: Kaplan Meier curves and event/censored histograms for all 15 datasets.
DeepHit [29 is also a discrete-time model where the number and locations of discrete times determined in the same way the N-MTLR model (the square root of numbers of uncensored patients, and quantiles). The model is implemented pyсox packages [27].
CoxTime 27 a non-proportional neural network extension the CoxPH. The model implemented in pycox packages 277
27

---

Censored Quantile Regression Neural Network (CORNN) 40 is a quantile regression based method. We add the gootstrap-rearrangee post- post-processing
31
correct non con-monooo
predictions. We use the CQRNN code provided in Qi et al. g
E.3 Hyperparameter settings for the main experiments
Full hyperparameter details for NN-Based survival baselines In the experiments, all neural network-basse methods (including N-MTLR, DeepSurv, DeepHit, CoxTime, and CQRNN) used the same architecture and optimization procedure.
Training maximum epoch: 10000
Early stop patients: 50
Optimizer: Adam
Batch size: 256
Learning rate: le-3
Learning rate scheduler: CosineAnnellingLR
Learning rate minimum: 1c-6
Weight decay: 0.1
NN architecture: [64, 64]
Activation function: ReLU
Dropout rate: 0.4
Full hyperparameter details for CSD and CiPOT
Interpolation: (Linear, PCHIP)
Extrapolation: Linear
Monotonic method: (Ceiling, Flooring, Booststraping)
Number percentile: (9, 19, 39, 49)
Conformal set: (Validation set, Training set Validation set)
Repetition parameter: (3, 10, 100, 1000)
E.4
Main results
In this section, we present the comprehensive results from our primary experiment, which focuses evaluating the performance of CiPOT compared to the original non-post-processed baselines and CSD.
Note that for the MIMIC- IV datasets, CQRNN fails to converge with any hyperparameter setting possibly due to the extremely skewed distribution. As illustrated in Figure O 80% of event and censoring times happen within the first bin, and the distribution exhibits long tails extending the 17th bin.
Discrimination In Figure 10 we demonstrate the discrimination performance of CiPOT compared benchmark methods, measured by C-index. The panels are ordered by dataset size, from smallest to largest. In each panel, the performance of the non non-post-proot post-processed baselines is shown with blue bars, CSD with orange bars, and CiPOT with green bars. We can see from the figure that the three methods exhibit basically the same performance across all datasets. Indeed, as we can see from the summary Table CiPOT ties with the baselines in out of 104 times. In the remaining 29 times they do not tie, none of them are significantly different from each other. For the 22 times CiPOT underperforms, most of them are wrt to DeepHit or CQRNN baselines (e.g., DeepHit for HFCR, PBC, GBSG, etc.). And CiPOT can even outperform 7 times (e.g., g., GB in HFCR).
28

---

HFCR
PBC
WHAS
0
B8B YAP
0.8
0.8
0.6
0.7
0.4
0.6
GBM
GBSG
PdM
960
406 100
1.0
009
PCo
0.6
0.8
0.4
NACD
FLCHAIN
0.80
0.900
0.80
0.875
0.850
0.70
0.75
0.625
SUPPORT
Employee
MIMIC-IV
0.70
0.95
M+ ge0
009
0.75
000
80g
000
0.68
0.90
0.70
888
0.85
PIP
0.06
400
0.80
0.85
SEER-brain
SEER-liver
BEER-stomach
0.75
1dd f!|
000
900
860
808
0.74
0.75
800
000
000
856
000
66
0.70
888
000
0.72
0.70
668
000
A&A
AFT
GB
NAITRR
DacpSorr
Deopta
Comeer
QB
NAILRR
DevedSorv
Desple
Cortone
APT
N-MTUR
DaepSury
Caatime
comee
Methods
Baseline
CSD
CIPOT
Figure 10: Violin plots of C-index performance of our method (CiPOT) and benchmarks. A higher value indicates better performance. The shape of each violin plot represents the probability density of the performance scores, with the black bar inside the violin indicating the mean performance.
Marginal calibration In Figure II we present the marginal calibration performance of CiPOT versus the benchmark models. The arrangement and color schemes of the panels are consistent with those used previously. For marginal calibration evaluation, we add a "dummy" model Kaplan Meeer (KM) curve (depicted by red dashed lines in each panel) to serve as the empirical lower limit. We calculate each KM curve using the training set and apply it identically to all test samples. It is called a "dummy" because it lacks the ability to discriminate between individuals. However, it asymptotically achieves perfect marginal calibration (see Appendix B in 88).
Our results in Figure I1 indicate significant improvement in calibration performance with CiPOT over both baselines and CSD. Overall, our method outperforms the baselines in 95 out of 104 times (Table 2 In the remaining 9 times where it does not outperform the baseline, only once is the difference statistically significant, and the marginal calibration score in this single case (GB for SEER-liver) is still close to the empirical lower bound (red dashed line).
For datasets characterized by higher censoring rates or high KM ending probabilities (HFCR, PBC, PdM, FLCHAIN, Employee, MIMIC-IV), our method shows superior performance. For those datasets, CSD tends to produce non-calibrated predictions compared with baselines. This outcome likely stems from the inaccuracies of the K.M-sapliig method under conditions of high censor rates or ending probabilities, as discussed in Section 2 Nonetheless, our approach still improves the marginal calibration scores for the baselines under these circumstances.
Conditional calibration Figure 12 showcases the conditional calibration performance of CiPOT versus the benchmarks, evaluated using Calas.
29

---

HFCR
PBC
WHAS
10"
10"
10"
10
10
10"
bdg
10
10"
10"
10
GBM
GBSG
PdM
10
10
10"
10
10
10"
10
10"
10"
NACD
FLCHAIN
10
10"
10
10
10"
SUPPORT
Employee
MIMIC-IV
10"
10
10"
10
10
0 I/ l6t
10"
10"
10"
10
10
SEER-brain
SEER-liver
SEER-stomach
10
T
10
10
10"
10"
10
8
10"
AFT
GB
NAITLR
DaepSurr
Deopha
Comter
AFT
GB
NAILRR
DeveSuev
Desple
Cortine
APT
GB
H-MTUR
DaupSury
Castime
comer
Methods
Baseline
CSD
CIPOT
Figure 11: Violin plots of Calmargin performance of our method (CiPOT) and benchmarks. A lower value indicates superior performance. The shape of each violin plot represents the probability density of the performance scores, with the black bar inside the violin indicating the mean performance. The red lines represent the mean calibration performance for KM, serving as an empirical lower limit.
Compared with marginal calibration, we cannot use any method to establish the empirical upper bound. One might think that once we establish the worst slab, we calculate the KM curve this slab and then use it as the empirical upper bound for the conditional calibration. However, identifying universal worst slab across all the models is impractical. That means different models exhibit varying wlrst-slabs. e.g., N-MTLR can have the worst calibration for overweighted males, while DeepSurv might perform relatively good calibration for this group but poorly for disabled cardiovascular patients.
The results in Figure 11 indicate significant improvement in conditional calibration performance using our method over both baselines and CSD. Overall, our method improves the conditional calibration performance of baselines 64 out of 69 times, with significant improvements 29 times out of 64 (Table
Our method outperforms CSD in 51 out of 69 cases. Most instances where our method underperforms are relative to the DeepHit and CQRNN baselines (the reasons are explained in Appendix C.4). To evaluate the practical benefits of CiPOT over CSD, we present four case studies in Figure 13 The figure showcases 4 concrete examples where CSD (orange) leads to significant miscalibration within certain subgroups (i.e., elderly patients, women, high-salary, and non-white-racine raciall,,, but CiPOT (green) can effectively generate more conditional calibrated predictions which are closer to the optimal line. Moreover, all four examples illustrate that the miscalibration of CSD consistently occurs in low-probability regions, corroborating our assertion in Section 3.4 that the conditional Kaplan-Mett Meier sampling method employed by CSD is problematic if the tail of the distribution is unknown.
30

---

PoMd
Chum
MACD
10
10
10
10"
FLCHAIN
SUPPORT
Employee
10
10
10"
10"
10
10"
10
10"
10" 2
10
10
10
10
SEER-stomach
AFT
NARTA
DempBurr
Daspie
Cortima
AFT
GB
NATTR
Donaftry
Despric
GoxTose
CORRd
10
Methods
10
CSD
CIPOT
AFT
GB
AATTER
Imaple
Cartin
Gomee
Figure 12: Violin plots of Calws performance of our method (CiPOT) and benchmarks. A lower value indicates superior performance. The shape of each violin plot represents the probability density of the performance scores, with the black bar inside the violin indicating the mean performance.
HPGR AFT
FLCHAIN
Employee
DeepSurv
MIMIC-IV MILA
Oberve Oheived 9unhhel Pros.
1.0
Obeeved Sunthel Pres.
1.0
1.0
0.8
0.8
0.8
0.8
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
DPOT
Predicted Survivll Prob.
Case studies of the conditional calibration between CSD and CiPOT. (a) For the elder age subgroup on HFCR, with AFT as the baseline; (b) For women subgroup on FLCHAIN, with GB as the baseline; (c) For the high salary subgroup on Employee, with DeepSurv as the baseline; (d) For the non-white-racin subgroup on MIMIC-IV, IV, with MTLR as the baseline, All four cases show that CiPOT is close to the ideal, while CSD is not.
Figure 13:
IBS Figure 14 illustrates the IBS performance of CiPOT versus the benchmarks. According to DeGroot and Fienberg (63] BS can be decomposed into a calibration part and a discrimination part. This implies that IBS, an integrated version of BS, assesses aspects of calibration. Our results in Figure and Table 2 show that our method improves the IBS score in most of the cases (63 wins, 18 ties, and 23 losses).
MAE-PO Our results Figure 15 and Table show that our method improves the MAE-PO in general (54 wins, 33 ties, and 17 losses).
31

---

0.6
HFCR
PBC
WHAS
0.4
040
888
808
0.2
0/0
W/
B8
868
0
795
668
668
GBM
GBSG
PdM
0.15
0.4
#6
000
090
000
0.05
0.00
0.25
0.20
460
0.20
Bog
0
0.15
0.10
SUPPORT
Employee
MIMIC-IV
0.20
0.045
0.15
&g
0.20
0.040
0.10
060
888
806
868
0.035
SEER-brain
SEER-liver
SEER-stomach
$88
999
88
0.20
0.20
690
BBB
6
0:15
086
0.1G
800
999
69d
666
66
888
g
86
AFT
NAMTLR
DaepSury
Deopha
CoRter
GB
NAITR
DeveSuovv
Desphe
Cortone
APT
H-MTUR
DaepSury Deeaph
Methods
Baseline
CSD
CIPOT
Figure 14: Violin plots of IBS performance of our method (CiPOT) and benchmarks. A lower value indicates superior performance. The shape of each violin plot represents the probability density of the performance scores, with the black bar inside the violin indicating the mean performance.
E.5
Computational analysis
Space complexity Although most of the computational cost arises from ISD interpolation and extrapolation, most of the memory cost of our method stems from storing the conformity scores into an array and performing the Percentile(-) operation.
Let's reuse the symbol N = [Deom] as the number of subjects in the conformal set, P (p1,P2 P2.... is the predefined discretized percentiles, that [P] be the number of predefined percentiles. And R is the repetition parameter for K.M-sampling.
CSD S process first discretized the ISD curves into percentile times (PCTs) using P. Then it calculates a conformity score for each individual at every percentile level p (i.e., each individual will contribute for [P] conformity scores). Lastly, the "KM-sampling" process involves repeating each individual by R times. Therefore, the memory complexity of CSD is O(N |Pl R). In contrast, CiPOT method only caleulates one iPOT score (as the conformity score) for each duplicated individual. After sampling, the total memory complexity of CiPOT is O(N Rt).
Let consider an example of using SEER-stomach dataset in our main experiment in Appendix E.4
with a repetition parameter conformity score matrix requires N x [P] R (100360 0.9) 19 x 1000 8 bytes 13.73 3373 Gb.
1000, and number of predefined percentiles [P] 1g CSD's
2 As recommend by Qi et al. 35 that the predetined percentile levels are (5%, 10%. 95%).
the performance reaches the optimal when |P] = 9 or 19. [P] 19 means
32

---

HFCR
PBC
WHAS
4000
400
2000
000
3000
300
200
2000
1000
860
0
608
878
GBM
GBSG
PdM
7.5
600
4000
180
000
100
W
100
5.0
400
660
600
808
498
888
909
2.5
006
6
FLCHAIN
6000
SUPPORT
Employee
MIMIC-IV
Bg
300
12
200
700
10
260
SEER-brain
SEER-liver
SEER-stomach
69
668
$88
600
000
64
800
880
80
800
989
000
888
6A4
838
e0g
600
AFT
NAMTLR
DeepSurv
Deopha
Comter
AFT
GB
NMILR
DeveSuov
Desphi
Cortene
APT
GB
HAMTUR
DaupSury
Caathoee
comee
Methods
Baseline
CSD
CIPOT
Figure 15: Violin plots of MAE-PO performance of our method (CiPOT) and benchmarks. A lower value indicates superior performance. The shape of each violin plot represents the probability density of the performance scores, with the black bar inside the violin indicating the mean performance.
Our CiPOT method needs only N R 0.72 Gb to store the conformity score. If we change even larger dataset or increase (P], CSD may become infeasible.
Other memory costs, e.g., storing features and ISD predictions, incur negligible memory costs. This because the number feature d, and the length of the ISDs are much smaller than the repeat parameters R.
Time complexity
The primary sources of time complexity in CiPOT are two-fold:
* ISD interpolation and extrapolation (line 4 in Algorithm
An optional monotonic step for the CQRNN model (see discussion below).
Note that other time complexity, e.g., running the Percentile operation or adjusting the ISD curves using lines 14-15 in Algorithm incur negligible time cost.
Here we analyze two kinds of time complexity: training complexity and inference complexity. Figure 16 and Figure [7 empirically compare the training time and inference time of the CiPOT method with those of non-post- processdesssed baselines and CSD across 10 random splits. Both CSD and CiPOT use the following hyperparameters to enable a fair comparison:
Interpolation: PCHIP
Extrapolation: Linear
Monotonic method: Bootstrapping
33

---

HFCR
PBC
WHAS
GBM
10
10
10
2
g00gle
GBSG
PdM
Churn
NACD
10
FLCHAIN
SUPPORT
Employee
MIMIC-IV
600
75
75
40
20
25
SEER-brain
SEER-liver
SEER-4tomach
Methods
100
200
200
Baseine
100
100
200
CSD
CIPOT
M1
wath
Figure 16: Training time comparisons (mean with 95% confidence interval).
HFCR
PBC
WHAS
GBM
1.5
0.3
1.0
0.1
0.5
0.0
0.0
0.0
0 0-0-0-00
GBSG
PdM
Churn
NACD
1.0
0.2
0.0
0-0
0.0
FLCHAIN
SUPPORT
Employee
MIMIC-IV
30
10
5
10
SEER-brain
SEER
StERcstomach
15
20
Mathods
10
10
Baseline
10
CSD
CIPOT
Figure 17: Inference time comparisons (mean with 95% confidence interval),
34

---

Number of percentile: 9
Conformal set: Training set + Validation set
Repetition parameter: 1000
Each point in Figure 16 represents an average training time of the method, where for the non-post- processed baselines is purely the training time of the survival model, while for CSD and CiPOT, it is the training time of the baselines plus the running time for the conformal training/learning, In Figure we see that CiPOT can significantly reduce the additional time imposed by the CSD step on all survival analysis models (AFT, GB, N-MTLR, DeepSurv, DeepHitt, and CoxTime). The only exceptional is the quantile-based method CQRNN, where for the 4 small datasets (HFCR, PBC, WHAS, and GBM) CiPOT actually increase the training time. This is because, for those datasets, the quantile curves predicted by CQRNN are not monotomic. address this, we attempted three methods before we directly apply CiPOT, including ceiling, flooring, and bootstrap rearranging Chernozhukov et al. with the bootstrap method proving most effective, albeit at a significant computational cost.
31
Similarly, each point in Figure 17 represents an average inference time of the method, where for the non-post-processar baselines it purely the inference time of the ISD predictions from the survival model, while for CSD and CiPOT it is the inference time of the ISDs plus the post- post-processing time. We observe the inference time follows the same trend the training time. The extra cost for the small datasets (HFCR, PBC, WHAS, and GBM) is still due to the non-monotonic issues. In those cases, the predicted curves by CQRNN become monotonic after applying CSD. However, surprisingly. after applying CiPOT, the curve remains non-monotonic. That is why the inference time for baselines and CiPOT higher than CSD on those four datasets.
E.6 Ablation Studies
Ablation Study #1: impact of repetition parameter R As we proved in Theorem 3.1 for censored subject j. if we sample its iPOT value using (<slr)) we can asymptotically achieve the exact marginal calibration. However, empirically, due to limited sample sizes, we find that only making sampling for each censored subject will not achieve good calibration performance. Instead, we propose the method repetition sampling, i.e., sampling R times from U (s.la)))
This ablation study tries to find how this repetition parameter R affects the performance, terms of both discrimination and calibration. We gradually increase R. from 3 to 1000, and assume the perfor- mance should converge a certain level. This ablation study uses the following hyperparameters to enable fair comparison:
Interpolation: PCHIP
Extrapolation: Linear
Monotonic method: Bootstrapping
Number of percentile:
Conformal set: Training set Validation set
Repetition parameter: 5, 10, 100, 1000
Figure 18 Figure 19 and Figure 20 present the C-index, marginal calibration, and conditional
calibration performances, respectively.
TL;DR The repetition parameter value has barely any impact on the C-index, and increasing R can benefit the marginal and conditional calibration, with convergence observed around R 100.
In Figure 18
we see that the C-index for CiPOT using 5 different repetition numbers has no visible differencees, for almost all baselines and all datasets. The only exception is there are slight differences for DeepHir baseline on HFCR and Employee datasets, where higher R will slightly decrease the C-index performance insignificantly.
In Figure 19 we can clearly see the repetition parameter has a great impact on the marginal calibration. For most datasets, higher repetition will significantly improve (decrease) the marginal calibration score. This trend is more clear for high censoring rate datasets (FLCHAIN, Employee, MIMIC-IV, etc), while for low censoring rate datasets (GBM, NACD, SUPPORT, which have censoring rates less
35

---

HECR
PBC
WHAS
BUR MAII
0.8
0.6
0.7
CBSC
0.7
1000 MINO
MW
1.0
01000
8:000
0.6
0.5
0.8
0.80
0.900
0.80
0.675
0.850
0.70
0.75
0.025
SUPPORT
Employee
MIMIC-VV
0.70
0.95
NWW neM
01000
0.68
0.90
0.70
0.85
RM
0.66
0.80
0.65
BraR-brain
SEER-liver
SEER-stomach
0.75
Mid WNY
01000
65858
pe9p9
09000
85858
0.74
0.75
8008 DIGG
00000
86468
60000
0.70
0.72
0.70
89998
AFT
NATTER
Detpllurr
Cheppiat
Cortins
CB
NAMILR
Denalluvv
Desegiii
Cos Tomeime
CORRA
NANTUR
Deegitor
Daepii
Contine
Repetitin Parameter
10
100
1000
Figure 18: Violin plots of C-index performance for Ablation Study #1: impact of repetition parameter. A higher value indicates superior performance.
than 40%), the trends still exist but the differences performance usually converges when R 2 100 (the pink and green violins are almost the same for all datasets and all baselines). As higher repetition parameter will increase the memory usage and computational complexity (Appendix E.5). we suggest using R = 100.
very significant. The marginal calibration
In Figure 20 the trends in conditional calibration are similar to those in marginal calibration. Most of the differences are more significant for high censoring rate datasets and less significant for low censoring rate datasets. And the conditional calibration performance also converges at R 2 100 (the pink and green violins are almost the same for all datasets and all baselines).
Ablation Study #2: impact of predefined percentiles? P Different choices of P may lead to slightly different survival distributions, all of which allow to obtain provable distribution calibration, as discussed next. Theoretically, discretizing a continuous curve into a series of discrete points may result in some loss of information. However, this be mitigated by using sufficiently fine grid for percentile discretization. Therefore, we anticipate that if we select P that contains finer grid percentiles, the performance (both calibration and discrimination) will be better.
Previous studies have commonly employed 10 equal-width probability intervals for calculating distribution calibration 3, 6 8
making an intuitive starting choice for 9 percentile levels at P
(10% 20%........................ 90%
'Note that 0% and 100% are excluded because S51(1 æ.) 0 and 5yd (0) x.) = +max oo are fixed endpoints.
36

---

HFCR
PBC
WHAS
10"
0000
19119
10"
10
10
10
joto 1f1 tit Mem iibbl Myys
MAM DOA M0X Bakd
MOM baad
10
10"
10
10"
GBSG
M616
10
10"
10"
Wite HAM
10
10"2
fith Mout jout toop
10
10""
Chum
NACD
FLCHAIN
WM "ild bott Bnpy Mog #11
10"2
A6463
10
10
10"
10 MMM
10""
10
10
10"
SUPPORT
Employee
MIMIC-V
|10000
10"
doogá
10
10
66656
MNN HAME AuN HH
MMM
10
10"
10
9g0
10
SraR-brain
SEER-liver
SEER-stomach
17111
10
10"
10"
10
10
1100
1009
AFT
08
NATTER
Detpllurr
Despiia
Cortins
APT
CB
N-MTLA
Donalluvv
Devepiit
Cox Contime
NANTLR
Dessiior
Daspie
Contime
Repetiion Parameter
10
100
1000
Figure 19: Violin plots of Calmargin performance for Ablation Study #1: impact of repetition parameter. A lower value indicates superior performance.
We compare this standard setting with 19 percentile levels (P 15%, 10%. .... 95%)). 39 percentile levels (P- = {2.5%, calibration evaluations (for both marginal and conditional) are performed 10 equal-width intervals to maintain comparability, following recommendations by 3 This ablation study uses the following hyperparameters to enable a fair comparison:
97.5%%)). and 49 percentile levels (P = (2%, 4%,... 98%)) )). All the
Interpolation: PCHIP
Extrapolation: Linear
Monotonic method: Bootstrapping
Number percentile: 9, 19, 39,
Conformal set: Training set Validation set
Repetition parameter: 1000
Figure 21 Figure and Figure 23 present the C-index, marginal calibration, and conditional
calibration performances, respectively.
TL;DR The number of percentiles has impact on the C-index, and has slight impact on marginal
and conditional calibration.
we see that the C-index for the 4 percentile numbers has no visible differences, for all baselines except CQRNN. It is worth mentioning that the difference for CQRNN is due to its quantile regression nature, requiring more trainable parameters percentiles increase.
In Figure 21
37

---

PdM
Churn
NACD
10
10
MM
10
10
FLCHAIN
SUPPORT
Employee
10"
10
10
bag
10
10
10
MIMIC-V
SEER-brain
Wit
10
10
Angp Mit ayn Opmn Beooo
10
10
10
SEER-stomach
MILR
Despetr
Time
comea
NATTR
DeonSury
10
10
Repelition Parameter
10
100
1000
10
APT
08
NATTUR
DeapSurr
Carrimu
Figure 20: Violin plots of Calws performance for Ablation Study #1: impact of repetition parameter. A lower value indicates superior performance.
In Figure 22 we can the number of percentiles has some impact on the marginal calibration. For example, the results for Employee and MIMIC-IV show some perturbation as we increase the number of percentiles. However, trends are inconsistent across models, egg, fewer percentile levels are preferable by AFT, GB, N-MTLR, DeepSurv for Employee, while more percentile levels are preferable by Deepffit and CoxTime. Also, most differences in Figure 22 are insignificant.
In Figure 23 the trends in conditional calibration are similar to those in marginal calibration. Most of the differences are insignificant and the trends are inconsistent.
These results suggest the choice of predefined percentiles has minimal impact on the performance. In practical applications, the reader can choose the best- rest-performing P at their preference. However, it is worth noticing that a higher percentile will result in more computational cost. Therefore, we suggest that it is generally enough to choose P (10%, 20%%)., 90%).
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and the last paragraph in Section I clearll state the contribution of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
38

---

HECR
PBC
WHAS
N010
1000
HON
0.8
0.6
0.6
0.4
CBSC
1.0
1.00
0000
8060
2000
0.6
0.5
NACD
0.80
0.900
0.675
0.850
0.76
0.70
0.825
SUPPORT
Employee
MIMIC-V
0.70
(010
0:00
0.95
0.00
0.68
0.90
0.70
0.85
0.66
0.80
0.65
SEER-brain
SEER-liver
SEER-stomach
0.75
0%00
86888
6999
0009
0.74
0.75
0000
8858
0:90
8088
0.70
0.72
0.70
8660
AFT
68
NATTR
Deepllury
Drsppia
Cartios
NAMILR
Denalluvv
Devephii
Cox Contime
NANTUR
Desgitor
Daepii
Contiie
Number of percentiles P
19
39
Figure 21: Violin plots of C-index performance for Ablation Study #2: impact of predefined percentiles. A higher value indicates superior performance.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The CiPOT method does not have theoretically guarantee for the preservation of Harrell's s C-index. This is mentioned Section 3..4ion3..4am and Table and theoretically justified Section measures this perspective empirically.
in Appendix C
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be
39

---

HFCR
PBC
WHAS
10
10
10"
10
8868
10
10"
0000
10
GBSG
PaM
10
10
10"
10
10
10"
11000
00X0
10
10
10"
10"2
10
10
10"
10
10"
6466
10
10""
10
SUPPORT
Employee
MIMIC-V
ALA
10
6000
10"
8646
16
10
SEER-brain
SEER-liver
SEER-stomach
10
M
10"
10
10
8058
AM 10
10
MaMi
9200
Y00
10
AFT
68
NATTUR
DeeplBurr
Desgit
Cantion
NAMTLR
Donalluvv
Devepiit
Con ConTime
NANTLR
Desgltor
Daepii
Contine
cosse
Number of percentiles P
19
39
Figure 22: Violin plots of Calmargin performance for Ablation Study #2: impact of predefined percentiles. A lower value indicates superior performance.
used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- lant role in developing norms that preserve the integrity of the community, Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Section 3.4
presents all the theories with the necessary assumptions and Appendix presents the complete proofs for those theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
40

---

PdM
Churn
NACD
10
IN
10"
FLCHAIN
SUPPORT
Employee
10"
10
10
10
10""
10"
10.
10
10
10"
6966
10
10"
SEER-stomach
WAILR
Deopttr
Time
comta
AFT
NATTRR
DeenSury
10
0600
0:00
Number percentlees
39
49
AFT
68
N-MTUR
DeapSurr
Chesphal
Carroou
Figure 23: Violin plots of Calws performance for Ablation Study #2: impact of predefined per- centiles. lower value indicates superior performance.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 5 outlines the pipelines of the experiments and directs the reader to the GitHub repository for the complete code. Further experimental details, e.g., evaluation metrics, dataset preprocessing, baseline architecture, hyperparameters, etc., included in the Appendix D and E
Guidelines:
The answer NA means that the paper does not include experiments. the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully
41

---

might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All the codes for reproducing the experiments are included in the GitHub link in Section along with 11 publicly available datasets. For MIMIC-IV and SEER datasets, we provide the preprocessing code, and users need to apply for the data by themselves, following the instructions in Appendix E.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https) //ips.cc/pubbll wttp://cccgggguueee////e////ce// htlp://guiiese/CCCo Caaeeee//ooSSSmmsseeeee
www.isssioopyccyc.oo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
42

---

Answer: [Yes]
Justification: Section states how we split the data in stratified manner. Further experi- mental details and hyperparameters information are included the Appendix E.2 and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
The paper uses violin plots to show the density distribution for the results from 10 random split of the experiments. We also use two-sided (-test to check whether the differences are significant (Table
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper includes those information in the computational efficiency analysis in Appendix
E.5
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
43

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The authors are sure that this research conforms the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This research motivates the use of conditional distribution calibration in survival analysis. This objective aligns with fairness perspective, where clinical decision sys- tems should guarantee equalized calibration across any protected groups (in Section TT. The authors anticipate our proposed metric and method can better communicate the uncertainty, enhancing the trustworthiness. fairness, and applicability (in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
44

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix E. has properly cited all the datasets used in our experiments. Appendix E.2 has properly cited all the baselines and codes.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pww.::s...aaseiiiia...... thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
The authors provide the well-documented code in the GitHub repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
45

---

Answer:
[NA]
Justification:
This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
46


Reviews:
[{"review_id": "GmUlknw5Pu", "reviewer": "Reviewer_rRGm", "summary": "While previous work focuses on calibration in a marginal sense, this paper proposes a post-processing approach that also imposes conditional calibration on all individual features. Therefore, the proposed method can guarantee equal calibration for different groups, ensuring fairness.\n\nContributions:\n\n1. The paper proposes the post-processing approach to ensure conditional distribution calibration in survival analysis (accommodating censorship) and develops the corresponding metric.\n2. The paper provides asymptotic guarantees for both marginal and conditional calibration and demonstrates the computation complexity.\n3. The paper validates the method by conducting experiments on 15 datasets.", "questions": "As mentioned in the weakness:\n\nQ1: How does the choice of percentiles affect the performance? Are there any empirical results that can illustrate the effect?\n\nQ2: Does the approach keep the Harrel C-index?\n\nQ3: There are plenty of survival analysis methods exploring the heterogeneity of groups. How does the proposed method compare to the SOTA heterogeneity methods? Will it improve the performance?", "limitations": "It remains unknown if this method would improve/hurt fairness in survival analysis. The study is missing.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Originality: The paper motivates the use of conditional distribution calibration in survival analysis via post-processing. The idea is intuitive as both the class-specific and the general survival probability at observed time should be uniformly distributed. \n\nQuality and significance: Although the calibration method is simple and known, the paper accommodates it to the censored setting and proves its effectiveness empirically. Besides, the paper provides corresponding theoretical guarantees. Lastly, it applies to various SOTA methods in survival analysis and improves the performance in most cases. \n\nClarity: The paper clearly conveys the idea and method of calibrating survival probability curves.", "weaknesses": "1. Missing related work in conformal prediction: the paper claims the method is based on conformal prediction but it is not mentioned and illustrated clearly in the main content. Thus, this knowledge gap hampers the understanding.\n\n2. The method seems to be sensitive to the percentiles. As shown in Figures 2 (d) and (e), the post-processed curves are piecewise functions highly related to the percentiles (1/3, 2/3). Thus, the resulting curves would be sensitive to the choice of percentiles. Therefore, the number of percentiles should be included in the complexity and asymptotic behavior analysis.  \n\n3. The discrimination properties of the method are unclear. The authors first claim it lacks the discrimination performance preservation property for Harrel's C-index, which conflicts with the results in Table 1.\n\n4. The empirical exploration and validation of the group fairness is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and suggestions. To address the concerns:\n> W1: Missing related work in conformal prediction: the paper claims the method is based on conformal prediction but it is not mentioned and illustrated clearly in the main content …\n\nThe current method (detailed in Section 3) was intended to incorporate the detailed algorithm of conformal prediction so that readers with no conformal prediction background should be able to understand it, and the current Section 2.3 (lines 96-110) has summarized the related work on conformal prediction with censored data. However, thanks to the reviewer’s feedback, we will include a more detailed related work about standard conformal prediction.\n\n> W2 and Q1: How does the choice of percentiles affect the performance? Are there any empirical results that can illustrate the effect?\n\nAs noted in line 314, our Ablation #2 study explored the impact of predefined percentiles, with results presented in the Appendix. The short summary (lines 998-999) notes that the number of percentiles has no impact on the C-index and only a slight impact on marginal and conditional calibration. More details on the experimental settings and findings can be found in Appendix E.6.\n\nAs to the space complexity analysis, the number of percentiles indeed has an impact on that complexity. However, this impact is so subtle that in the big-O notation, it will be ignored. To expand more on this, Appendix E.5 establishes that the complexity of storing the conformity scores is $O(N \\cdot R)$. After the method gets all the conformity scores, we will apply the percentile operation to all the conformity scores $\\Gamma_{M}$ (as presented in Equation 4). The space complexity of this operation is just $O(|\\mathcal{P}|)$ because there are only $|\\mathcal{P}|$ scores to be saved (each for a unique percentile level) for later use. Therefore, the total space complexity is $O(N \\cdot R + |P|) = O(N \\cdot R)$, because $|\\mathcal{P}|$ (a user-specific hyperparameter – here between 9-49) is significantly smaller than $N \\cdot R$. \n\n> W3 and Q2: The discrimination properties of the method are unclear…\n\nWe thank the reviewer for carefully examining this discrepancy between the statement and Table! This is indeed a typo in Table 1. For the proposed CSD-iPOT, the “Monotonic” should be “check”, and “Discrimination guarantee Harrell’s” should be “cross”. The rest of the paper, including the statement in the main text and proofs in the Appendix, supports the above claim.\n\n> W4 and Limitation 1: The empirical exploration and validation of the group fairness is missing…\n\nWe thank the reviewer for bringing this up. Throughout the paper, we have considered the proposed conditional calibration metric also as a fairness metric. The resemblance to fairness metrics has been discussed multiple times in the paper, including lines 34-36 and 225-226. \n\nTo briefly illustrate their resemblances, the proposed conditional calibration score evaluates the calibration performance across all possible subgroups (and reports the worst score in all the subgroups). This aligns with some fairness definition that clinical decision systems should guarantee equalized performance (e.g., accuracy) across any protected groups.\n\nHowever, we acknowledge that fairness has different definitions, and in this paper, we consider the proposed conditional calibration metric, $\\text{Cal}_{\\text{ws}}$, also as a fairness metric. We provide empirical results in Figures 4 and 11.\n\n> Q3: … How does the proposed method compare to the SOTA heterogeneity methods? Will it improve the performance?\n\nThe proposed method is a **model-agnostic** post-processing method, meaning it can be adapted to any heterogeneity survival methods, as long as they can generate individual survival distributions (ISDs). Our experiments have extensively demonstrated, using 7 SOTA heterogeneity models, that our approach can improve the marginal and conditional calibration without decreasing the discrimination performance. \n\nAs we noted, this approach can be very helpful in model development, as it allows researchers to simply seek models with superior discriminative abilities and subsequently apply the proposed method to improve calibration. This simplifies the model development process while ensuring robust performance across these key metrics.\n\nFurthermore, the proposed method aims to maximize conditional calibration, a score that evaluates the calibration performance by considering the heterogeneity of all possible subgroups. Ideally, one would seek extreme heterogeneity – i.e., individual calibration – which means the prediction is calibrated conditioned on any possible combination of features. However, for each unique combination of features, there is only one realization and therefore is impossible to perform the evaluation. Instead, we reach a middle ground – conditional calibration. This idea is the motivation of our paper and is discussed in detail in Section 2.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H0gLwER2dK", "reviewer": "Reviewer_kAoa", "summary": "The paper enhances the Conformalized Survival Distribution (CSD) post-processing framework to account conditional calibration. The proposed framework, CSD-iPOT, utilizes a conformal set to adjust survival curves vertically, aligning them with predetermined percentiles at test time. Unlike CSD, which relies on Kaplan-Meier curves, CSD-iPOT leverages Individualized Survival Distributions (ISD) for censored events when constructing the conformal set. The paper provides theoretical guarantees for marginal calibration, conditional calibration, and the monotonicity of ISD. Comprehensive experimental results from 15 datasets demonstrate that CSD-iPOT enhances both the marginal and conditional calibration of baseline models with minimal impact on the concordance index (C-index).", "questions": "- Given that CSD-iPOT lacks theoretical guarantees for preserving Harrell's concordance index, is there a hyperparameter that explicitly controls this trade-off?\n- It seems that CSD-iPOT achieves calibration lower than the empirical lower limit set by Kaplan-Meier. Are these the instances that result in an impact on the C-index?\n- Why does CSD-iPOT struggle with both calibration and the C-index for models such as DeepHit?\n- Are the calibration metrics obtained at different percentiles than those used for the post-processed ISD? What happens if these two sets are different?\n- Why does CSD-iPOT improve marginal and conditional calibration models better than CSD? Is the difference due to how CSD-iPOT handles censored events?", "limitations": "I encourage the authors to discuss the limitations of their work, including any violations of modeling assumptions.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. \n- The reviewer appreciates that the visual plots provided offer great intuitive illustrations of the proposed approach. \n- This is the first paper to address conditional calibration in survival analysis, an important yet under-explored problem. \n- The paper provides theoretical guarantees for the proposed approach in terms of marginal calibration, conditional calibration, and monotonicity. \n- CSD-iPOT is more computationally efficient in terms of storage than CSD.\n- Extensive experimental results across 15 datasets and 7 baselines demonstrate that CSD-iPOT significantly improves the calibration (both marginal and conditional) of baseline models with minimal loss in the concordance index (C-index).", "weaknesses": "*The paper highlights that CSD-iPOT lacks theoretical guarantees for preserving Harrell's concordance index. This is a major limitation of this work; nevertheless, experimental results demonstrate that the impact is minimal*\n\n*The description in lines 142-162 requires some improvements:*\n\n- Eqn. 4: Needs to be adjusted to something like $\\tilde{S}^{-1}(p | x_{n+1}) = T(\\hat{S}^{-1}(\\text{Percentile}(\\cdot) | x_{n+1}))) $, where $T$ is the proposed vertical transformation. Then provide all the necessary numbered steps to finally obtain the calibrated ISD (Eqn. 5).\n\n*For completeness, I encourage the author(s) to also benchmark competitive baseline models that directly model event times, e.g., Chapfuwa et al. 2018 and Miscouridou et al. 2018*\n\n*Minor:*\n- In Table 1, CSD-iPOT should be marked with an 'X' under Harrell’s concordance index category.\n- Line 297: Typo -> \"Compare to CSD\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for these wonderful comments and suggestions! Wrt your insightful concern:\n>The paper highlights that CSD-iPOT lacks theoretical guarantees for preserving Harrell's concordance …\n\nYes, we acknowledged this limitation in the paper. However, we argue this is not a big issue for two reasons:\n1. While our method does not have a preservation guarantee for Harrell’s C-index, it does have this guarantee for two other discrimination metrics: AUROC and Antonili’s C-index, which are also (arguably more) commonly used in application studies. Due to their distinct nature, we do not know any method that is guaranteed to preserve all these three notions of discrimination (and suspect this might not be possible). \n2. While our method is not guaranteed to preserve Harrell’s C-index, our extensive experimental results (on 104 comparisons) show that, in 84 cases, our method does not decrease the C-index. And for the 22 cases that it does decrease, none of them are significant. This really shows that this lack-of-formal-guarantee seems minor.\n\n>The description in lines 142-162 requires some improvements …\n\nThanks for the suggestion. This helped us to identify a possible misunderstanding. The proposed vertical transformation pertains solely to Eq 4. It does not apply to Eq 5, which transforms the inverse-ISD back to the ISD function (i.e., from Figure 2(e) to curves similar to Figure 2(a)). However, we realized that the position of lines 158-162 might cause this misunderstanding (now it is right after Eq 5); we will move this description ahead, to lead to a better presentation of the method. \n>I encourage the author(s) to also benchmark competitive baseline models that directly model event times …\n\nWe appreciate this suggestion. However, the proposed CSD-iPOT method requires the original survival models to be able to generate survival distributions (Section 3.1). If a baseline model can only generate a scalar value, it is not clear how to convert the scalar value into a distribution. \n\nAlso, because such a model only generates a time prediction (which is not a probability), it is also unclear what calibration means for time prediction.\n\nHowever, if the reviewer has any insight on this, please let us know and we are happy to use your approach to benchmark these models.\n>Q1: … is there a hyperparameter that explicitly controls this trade-off?\n\nNo, there is no hyperparameter to control how much the CSD-iPOT process modifies the C-index. However, we do not consider this as an issue because (1) in 75/104 cases, the CSD-iPOT process did not affect the C-index, (2) for the remaining cases, the effect is not always negative: in 7/104 cases, CSD-iPOT improves the C-index.\n>Q2: It seems that CSD-iPOT achieves calibration lower than the empirical lower limit set by Kaplan-Meier. Are these the instances that result in an impact on the C-index?\n\nYes. The same columns in Figure 3 correspond to the same instances. The reviewer might think that there is an implicit trade-off – that increasing calibration is associated with decreasing C-index. However, this is not the case for our method: the level of calibration improvement is not associated with the level of C-index decreasing. For example, in the HFCR datasets with the GB baseline, our CSD-iPOT improves both C-index and calibration (Figure 3, left). \n>Q3: Why does CSD-iPOT struggle with both calibration and the C-index for models such as DeepHit?\n\nAppendix C.4 discusses why our method is sub-optimal for these models (DeepHit and CQRNN), as indicated by lines 300-301 in the main text. In summary, this is because such models are often significantly miscalibrated, by implicitly assuming that, by the end of the predefined $t_{\\text{max}}$, every individual must have had the event, and therefore their predicted survival distributions must drop to 0% at that time. In such cases, because CSD-iPOT can not have any intervention for the ending probability (0%), therefore, the ending position cannot be moved. Figure 7 shows an example of this issue, showing that the post-processed distribution has a sharp-dropping tail, which leads to miscalibration in those regions.\n>Q4: Are the calibration metrics obtained at different percentiles than those used for the post-processed ISD? ...\n\nThe percentiles for calibration evaluation are always set to be 10%, 20%, … 90%, as recommended by Haider et al. However, our Ablation #2 explores the impact of different predefined percentiles for the proposed CSD-iPOT. The short summary (lines 998-999) is that the number of percentiles has no impact on the C-index and a slight impact on marginal and conditional calibration. More details on the experiment settings and results can be found in Appendix E.6.\n>Q5: Why does CSD-iPOT improve marginal and conditional calibration models better than CSD? Is the difference due to how CSD-iPOT handles censored events?\n\nYes, that is exactly right! \n\nFor marginal calibration, because the CSD-iPOT shifts the survival probability values **vertically** (while CSD shifts the time prediction **horizontally**), it avoids the challenges of interpolation and extrapolation of the distribution, which is particularly problematic when the censoring rate is high and when KM ends at a high probability level. \n\nFor conditional calibration, CSD-iPOT is better than CSD because CSD-iPOT considers the **heterogeneity** of the features by sampling the event times from the *individual* survival distributions, while CSD samples from the conditional KM distribution do not consider the features. \n\nThe current paper discusses these two aspects in detail, in lines 206-215 and 194-205, resp.\n>I encourage the authors to discuss the limitations of their work, including any violations of modeling assumptions.\n\nThanks for the excellent suggestion. As mentioned above, the current paper discusses two limitations of our methods. However, the revised version will also include a discussion about the violations of assumptions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WqYvMUZo56", "reviewer": "Reviewer_a6Gj", "summary": "The authors study the problem of how to create individual survival distribution (ISD) models which are well-calibrated, both in a marginal and conditional sense, without negatively affecting the discriminative performance.\n\nThey refine the \"Conformalized Survival Distribution\" (CSD) approach and propose \"Conformalized Survival Distribution using Individual survival Probability at Observed Time\" (CSD-iPOT). Both are post-processing methods which utilize conformal prediction, and can be applied on top of various survival analysis models to improve their calibration. CSD-iPOT is however designed to not only improve marginal calibration, but also conditional calibration.\n\nThe method is evaluated on 15 datasets, comparing baseline (without neither CSD nor CSD-iPOT), CSD and CSD-iPOT versions of 7 survival models. The results are quite promising overall.", "questions": "- 189: \"Unlike CSD [8], which adjusts the ISD curves horizontally (changing the times, for a fixed percentile), our refined version scales the ISD curves vertically\", could you perhaps visualize how both CSD and CSD-iPOT modify the ISD curves in an example? I think that could help illustrate how these two methods differ.\n\n- I thought that CSD-iPOT shouldn't affect the relative ordering of subjects, but when comparing the ISD curves before and after in Figure 2(d) and (e), the top orange curve is always above the top blue curve in (e), whereas it is partially below the blue curve in (d)?\n\n- 236: \"Qi et al. [8] demonstrated that CSD theoretically guarantees the preservation of the original model’s discrimination performance in terms of Harrell’s concordance index (C-index) [1]. However, CSD-iPOT lacks this property\", but both methods have a check mark for \"Discrimination guarantee Harrell’s\" in Table 1?\n\n\n\n\nMinor things:\n- 77: \"In summary, individual calibration is ideal but not impractical\", impractical --> practical?\n\n- 219: \"with adequate modifications to the accommodate our method\", remove \"the\"?\n\n- 297: \"Compared CSD\" --> \"Compared to CSD\"?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper studies an interesting and important problem, how to build well-calibrated and discriminative survival analysis models. The special focus on conditional calibration is also important.\n\n- The paper is well-written and very solid overall, the authors definitely seem knowledgeable.\n\n- The general idea of the proposed method, to utilize the conformal prediction framework in order to adjust predicted survival distribution curves on calibration data, makes sense.\n\n- The evaluation is quite extensive with 15 datasets and 7 baseline survival analysis models, and a direct comparison with CSD.\n\n- The results are quite promising overall, CSD-iPOT improves the calibration of CSD in most cases without negatively affecting the discrimination very often.", "weaknesses": "- I found parts of Section 3 and 4 a bit difficult to follow, the proposed method/metric could perhaps be described in a more intuitive way. Figure 2 is neat, but would be helpful to have a similar visualization to illustrate the difference between CSD and CSD-iPOT.\n\n- The technical novelty/innovation compared to CSD is perhaps somewhat limited, both methods employ the same general approach (post-processing methods which utilize conformal prediction).\n\n- The experimental results could be more convincing. CSD-iPOT usually improves the calibration compared to CSD, but far from always, and the gains also seem to be relatively small quite often. Overall, I find it quite difficult to judge how much added benefit CSD-iPOT actually would have compared to CSD in practice. Are there concrete examples where CSD leads to significant mis-calibration within a certain patient subgroup, which is addressed by CSD-iPOT? The current results/metrics are a bit abstract / difficult to interpret.\n\n\n\nSummary:\n- Well-written and very solid paper overall that studies an important/interesting problem. The technical novelty compared to CSD, and how much better the proposed method actually would be in practice, is however a bit unclear. I am leaning towards accept.\n\n\n\n\n***\n***\n***\n***\n\n**Update after the rebuttal:**\n\nI have read the other reviews and all rebuttals.\n\n2/3 other reviews are also positive, and I think the authors respond well to the third.\n\nAll my questions were addressed, Figure 1 and 2 in the provided pdf are neat.\n\nI will increase my score to \"7: Accept\", this is a very well-written and solid paper that I think should be accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and suggestions. To address the concerns:\n\n>W1 & Q1: … would be helpful to have a similar visualization to illustrate the difference between CSD and CSD-iPOT.\n\nGreat suggestion! The revised version will include a side-by-side visual comparison of our method to CSD – see Figure 1 in the additional 1-page PDF. After the paper's acceptance, we will add two animated GIFs to the GitHub repo, to better visually illustrate the difference between our method and CSD.\n\n>W2: The technical novelty/innovation compared to CSD is perhaps somewhat limited, both methods employ the same general approach (post-processing methods which utilize conformal prediction).\n\nIt is true that both CSD and CSD-iPOT use the same general idea of conformal prediction. However, it is important to highlight that conformal prediction is an active area of research, with a variety of methods aimed at enhancing prediction coverage and calibration more effectively and efficiently.\n\nSpecifically, the novelty of this work stems from the unique design of the conformity score, and the downstream method for handling censored subjects (thanks to the conformity score design), which together provide a theoretical guarantee for the calibration. \n\n>W3: .. Overall, I find it quite difficult to judge how much added benefit CSD-iPOT actually would have compared to CSD in practice. Are there concrete examples where CSD leads to significant mis-calibration within a certain patient subgroup, which is addressed by CSD-iPOT? …\n\nWhile our extensive experimental results did not find universal improvement, we did observe CSD-iPOT superior to CSD in 68/104 cases (significantly in 37 cases) for marginal calibration, and in 51/69 cases (significantly in 26 cases) in conditional calibration – see Table 2 in the paper. So in real applications, this method should be in your quiver of tricks to consider. \n\nFurthermore, thanks to the reviewer’s great suggestion, we provide 4 case studies in Figure 2 of the 1-page PDF – concrete examples where CSD leads to significant miscalibration within certain subgroups (elderly patients, women, high-salary, and non-white-racial), but CSD-iPOT can effectively generate more conditional calibrated predictions. Furthermore, all 4 examples show that CSD’s miscalibration is always located at the low-probability regions, which corresponds to our statement (lines 208-212) that the conditional KM sampling method that CSD used is problematic for the tail of the distribution (low-probability regions). \n\n>Q2: … the top orange curve is always above the top blue curve in (e), whereas it is partially below the blue curve in (d)?\n\nThank you for your insightful comment. Yes, the blue curve is partially at the top in (d), intersecting the orange curve around 1.7 days, while the orange curve is consistently at the top in (e). This discrepancy arises from the discretization step used in our process, which did not capture the curve crossing at 1.7 days due to the limited number of percentile levels (2 levels at 1/3 and 2/3) used for simplicity in this visualization.\n\nThe post-discretization positioning of the orange curve above the blue curve in Figure 2(e) does not imply that the post-processing step alters the relative ordering of subjects. Instead, it reflects the limitations of using only fewer percentile levels. Note that other crossings, such as those at approximately 1.5 and 2.0 days, are captured. In practice, we typically employ more percentile levels (e.g., 9, 19, 39, or 49 as in Ablation #2), which allows for a more precise capture of all curve crossings, thereby preserving the relative ordering. \n\n>Q3: …both methods have a checkmark for \"Discrimination guarantee Harrell’s\" in Table 1?\n\nWe thank the reviewer for carefully examining this discrepancy between the statement and Table – yes, there is a typo in Table 1. For the proposed CSD-iPOT, the “Monotonic” should be a check, and “Discrimination guarantee Harrell’s” should be a cross. We are sorry for this mistake and will revise this and other typos in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2089KIcGni", "reviewer": "Reviewer_b9k1", "summary": "This paper proposes a new postprocessing method, CSD-iPOT, for survival analysis based on conformal prediction.", "questions": "Nothing.", "limitations": "This paper does not discuss the hardness of conditional calibration.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The proposed method tries to achieve the conditional calibration, which is known to be hard.", "weaknesses": "This paper puts emphasis on achieving conditional calibration (not marginal calibration), but the conditional calibration is known to be hard to achieve even for datasets without any censored data point.  See, e.g.,\n+ Lei, J. and Wasserman, L. (2014). Distribution-free prediction bands for non-parametric regression. Journal\nof the Royal Statistical Society: Series B (Statistical Methodology), 76(1):71–96.\n+ Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, Ryan J Tibshirani, The limits of distribution-free conditional predictive inference, Information and Inference: A Journal of the IMA, Volume 10, Issue 2, June 2021, Pages 455–482.\n+ Zhao et al., Individual Calibration with Randomized Forecasting, ICML 2020.\n\nThis paper does not cite these papers, and this paper does not discuss the hardness of conditional calibration.  Lack of extensive discussion on the hardness of conditional calibration is a serious problem of this paper.\n\nDespite the hardness, this paper claims that the conditional calibration can be achieved in Theorem 3.2.  However, this theorem actually shows nothing: it shows only that, if we have an estimator that achieves the conditional calibration (i.e., Eq. (10) is satisfied), the output of the proposed method achieves the conditional calibration, too.  I think that if we have an estimator that achieves the conditional calibration, the proposed method is not required.\n\nFurthermore, there are many other problems in this paper:\n+ While $\\Gamma_{M}$ should be a set of scalar values according to line 9 of Algorithm 1, $\\Gamma_{M}$ is a set of pairs of scalar values according to line 11 of Algorithm 1.\n+ According to lines 180-183, $R$ copies of comformity scores $\\gamma_{i,M}$ are generated for each uncensored data point.  If so, the equation between lines 537 and 538 is incorrect.\n+ According to lines 180-183, $R$ copies of comformity scores $\\gamma_{i,M}$ are generated for each uncensored data point.  Neverthless, this paper assumes that the set $\\Gamma_{M}$ does not have any tie in line 538.\n+ The proof of Theorem C.1 are not fully described.   Even though the goal of the proof is to prove equation (9) on $\\rho_{1}$ and $\\rho_{2}$, the proof for censored data points does not argue $\\rho_{1}$ and $\\rho_{2}$.  An equation for censored data points analogous to the equation for uncensored data points (between lines 537-538) must be presented.\n+ The proof of Theorem C.1 completely ignores $R$, even though Theorem C.1 does not hold if $R=1$.  (I think that the main idea of the proposed method, CSD-iPOT, is to use a sufficiently large $R$ to \"blur\" the censored data points.)\n+ The assumptions used in the proof of Theorem C.2 are not clearly stated before the proof: many implicit assumptions are used in lines 563 and 564.   In particular, during the proof of (i), a statement (i.e., an implicit assumption) similar to (i) is used in line 563.\n+ In the proof of Theorem C.2, the alleged proof of (i) on $x_{n+1}$ does not include any discussion on $x_{n+1}$.\n\n\n------------------------\n# Additional Comments\n\nI submitted the following comments in the Author-Reviewer discussion period, but it accidentally did not visible to the authors (probably due to the complex system on comments visibility of OpenReview).  I noticed this fact during the Reviewer-AC discussion period, and the AC allowed me to post the comments here.\n\n==comments begin==\n\nThank you for your comments. I will keep my score.\n\nThis paper has several critial problems.\n\n+ The authors' comments did not give any evidence that the proposed algorithm achieves conditional calibration (claimed in lines 47-48). Since the hardness of the conditional calibration is an important topic in machine learning as already studied by many researchers, the authors must pay careful attention when they discuss on this topic.\n+ Huge discrepancy between the implemented algorithm (with R=1000) in the experimental section and the provided proof (valid only for R=1), even though the key idea of the proposed algorithm CSD-iPOT is to \"blur\" a censored subject with a large \n.\nRegarding the presentation, all the assumptions must be clearly presented before the proof. The assumption is stated at the end of the proof (in lines 546-548).\n\nMinor thing:\n\nThe authors violated the rule on the 1-page pdf in the rebuttal phase: \"Please use this PDF only for figures (including tables) and captions that describe the figure.\"\n\n==comments end==", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing interesting comments but are sorry that you did not agree with the other reviewers about the merit of our paper. To deal with your concern:\n\n>W1: … Lack of extensive discussion on the hardness of conditional calibration is a serious problem of this paper.\n\nThank you for stating your concern. However, we believe that this concern is mostly because this reviewer’s perspective is very different from ours and the other reviewers'. While our paper clearly relates to statistical issues, our goal was a method that survival analysis researchers can use to effectively achieve good calibration and maintain discrimination. That is why the main text de-emphasized theoretic proofs, and instead focused on intuitive explanations (using many real-world examples) of why our method should work. This stems from our concern that if our paper focused too much on conformal prediction and its history, the readers (mainly survival analysis researchers, because it is a tool for them) would lose interest. \n\nFurthermore, Lei and Wasserman (2014) and Barber et al (2021) both note that conditional calibration is hard as they have shown that the finite sample guarantee for conditional calibration is impossible to achieve. This is why our paper does not try to provide any finite sample guarantee. Instead, we follow the route of many other researchers (Romano et al. (2019); Sesia and Candes (2019) [1]; Izbicki et al. (2020); Chernozhukov et al. (2021) [2]) by providing only an asymptotic guarantee for infinite samples. The main idea of this asymptotic conditional guarantee is that the construction of the post-conformal predictions relies on the original prediction, and we only hope for conditional calibration for the class of predictions that can be learned well (which correspond to the assumptions explicitly made in the paper, see also in W2).\n\nLastly, despite the hardness of conditional calibration and the limitation of our assumptions, our extensive experiments have shown the effectiveness and robustness of the method for 15 datasets and 7 baselines – to our best knowledge, no previous conformal prediction paper has provided such a broad empirical validation. While this conditional guarantee is hard in theory, our experiments certainly support it – which we view as a merit of our paper.\n\nWe again appreciate the suggestion. Towards reaching a broad audience, the revised manuscript will include a brief discussion about the hardness issue in the main text and also a new Appendix section to extensively discuss this.\n\n[1] A comparison of some conformal quantile regression methods. Stat\n\n[2] Distributional conformal prediction. PNAS\n\n>W2: … Theorem 3.2 actually shows nothing\n\nWe acknowledge that the consistent estimator assumption in Theorem 3.2 is weak. As mentioned earlier, our paper focuses on asymptotical guarantee – only providing conditional calibration in the class of predictions that can be learned well. This assumption strictly follows Assumption 3 in Sesia and Candes (2019); Assumption 2.3 in Izbicki et al. (2020); and Assumption 1 in Chernozhukov et al. (2021). Therefore, our Theorem 3.2 has the same practicality as those previous methods.\n\nWe also thank the reviewer for confirming that proving the conditional calibration is hard. Moreover, to our best knowledge, there are no results that can prove conditional calibration with more relaxed assumptions. Finally, note that our paper focuses on empirical performance more than theoretical results.\n\n>W3: … $\\Gamma_M$ is a set of pairs of scalar values according to line 11 of Algorithm 1.\n\nThank you for noting this. Algorithm 1 uses this problematic expression (and also adds a direct comment behind this expression to explain) as we did not have a better notation to represent repeating a value by R times. The revised version will reduce this confusion by including an underbrace to express repeating a value.\t\n>W4: … If so, the equation between lines 537 and 538 is incorrect …\n\nAs shown under W7, here we prove that this theorem holds for $R=1$. Therefore, the equation between lines 537-538 is correct.\n>W5: … this paper assumes that the set $\\Gamma_M$ does not have any tie in line 538 …\n\nThis “no tie” assumption is a technical assumption in conformal regression to make the proof. This assumption is easy to solve in practice because users can always add a vanishing amount of random noise to the scores to avoid ties.\n>W6: …the proof for censored data points does not argue $\\rho_1$ and $\\rho_2$\n\nThank you for pointing this out. The original proof only considers the upper bound to be adaptive and sets the lower bound to 0. We have included the revised derivation to incorporate an adaptive lower bound – see Figure 3 in the 1-page PDF. The revised derivation is almost the same as the original but has an extra term when breaking the joint probability into pieces. Fortunately, this modification does not compromise the validity of the conclusion.\n\n>W7: … Theorem C.1 completely ignores 𝑅, even though Theorem C.1 does not hold if 𝑅=1 …\n\nIn fact, the proof shows the theorem holds for $R=1$, as stated in lines 546-548: “if we do one sampling for each censored subject …, the above proof asymptotically converges…”. The repeating strategy is just to get a more accurate estimation for finite samples. \n\n>W8: …many implicit assumptions are used in lines 563 and 564.… during the proof of (i), a statement (i.e., an implicit assumption) similar to (i) is used in line 563.\n\nSorry about the confusion. The revised version will put the assumption (now in line 564) before the proof. Note however that the statement in line 563 is directly derived from line 562, so it is not an assumption.\n\n>W9: In the proof of Theorem C.2, the alleged proof of (i) on $𝑥_{𝑛+1}$ does not include any discussion on $𝑥_{𝑛+1}$.\n\nThanks for catching this typo! In lines 559-565, the conditions should be $x_{n+1}$, not $x_i$, which will be corrected in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their time and thoughtful feedback! In particular, we are grateful for the largely positive reception of our work. To mention a few key points from the reviewers:
>The paper is well-written and easy to follow. The visual plot offers great intuitive illustrations and is neat (`kAoa`, `a6Gj`)

>This is the first paper to focus on conditional calibration in survival analysis  (`rRGm`, `kAoa`), which is important and hard (`kAoa`, `a6Gj`, `b9k1`).

>This paper provides theoretical asymptotic guarantees for marginal and conditional calibration (`rRGm`, `kAoa`)

>This method applies to various SOTA methods in survival analysis and improves the performance in most cases, which is promising overall. (`rRGm`, `kAoa`, `a6Gj`)

The paper extensively validates the method by conducting experiments on 15 datasets and 7 baselines, and a direct comparison with CSD. (`rRGm`, `kAoa`, `a6Gj`)

We are encouraged by the positive evaluations (scores of 7, 5, and 5) and are keen to address the concerns underlying the lower score (3) to bridge any gaps in understanding.

We have carefully considered each point of criticism raised by the reviewers and our individual rebuttals provide detailed clarifications and justifications. Perhaps the main critique is from `b9k1`’s concern that “this paper does not discuss the hardness of conditional calibration", and “the proof of Theorem 3.2 is nothing” – leading to poor ratings on soundness. We appreciate the reviewer’s comments and hope our direct response clarifies the concerns.

We hope our individual responses and the additional 1-page PDF effectively address all of the concerns raised. We are eager to engage further during the discussion phase and to answer any additional queries that may arise.

Meta-review: The paper studies conditional calibration in the context of survival analysis, proposing a method based on conformal prediction that uses the model's predicted individual survival probabilities at observed times. This method effectively improves that model's marginal and conditional calibration, without compromising discrimination. The authors provide theoretical guarantees for both marginal and conditional calibration and test it's performance across 15 real-world datasets.

The majority of the maintained their positivity towards the paper during the discussion phase. One reviewer remained negative, on the principle that the paper contains a significant flaw. In short, the theoretical guarantee provided by the authors holds under an assumption for asymptotic consistency of the distribution estimator; this negates the need for calibration. Moreover, conditional calibration is a hard problem, as cited by many works that the reviewer referenced, and the authors did not include in their original submission.  Hence, the paper should be rejected on principle: the claim by the authors that they solve a hard problem is sufficient grounds for rejection, tantamount to claiming that the paper solves an NP hard problem, and subsequently proceeds to provide experimental results on its performance.

The authors responded in their rebuttal that their theoretical claim does not contradict prior work, that shows hardness under finite samples (as their result is asymptotic). They also acknowledged that, nevertheless, this assumption is indeed strong, even if quite common in conformal prediction literature. They also indicated that their contribution (and therefore their emphasis) was on the experimental results, that indeed establish excellent performance on 15 datasets. 

Remaining reviewers, and the AC, agreed with the authors on this front. They all agreed that the authors have adequately addressed these concerns in their rebuttal and that, even if theoretical results are weak, they are (a) correct, in that there are no errors in the statements of theorems and proofs, (b) rely on standard (albeit strong) assumptions in the field of conformal prediction, and (c) they are supported by very strong and exhaustive experiments, that make the paper worthy of reporting to the community. There was also an agreement with the authors that no results can prove conditional calibration under more relaxed assumptions.

It was also pointed out that a similar argument about weakness of theoretical guarantees could be made for marginal calibration; for example, Kaplan-Meier is a consistent estimator for the survival function but cannot be used for time-to-event prediction. In practice, one is interested in accurate time-to-event prediction models, where calibration is of secondary concern. For any given problem, numerous solutions can yield well-calibrated predictions that are not necessarily accurate and, thus, not useful. Hence, focusing on a strong theoretical result that guarantees conditional calibration might not be of practical use if the models derived from such an estimator do not yield accurate predictions. As the paper demonstrates, via extensive experimental results across 15 datasets and 7 baselines, that CSD-iPOT significantly improves the calibration (both marginal and conditional) of baseline time-to-event models with minimal loss in the concordance index (C-index). Therefore, even if the theoretical results may not be robust, the empirical results are compelling. 

The authors are strongly encouraged to address the concerns raised by all reviewers, particularly the reviewer that remained negative. Papers showing hardness should be cited and contrasted to. The claim regarding theoretical guarantees should be toned down and stated in the appropriate context. Additional comments made by remaining reviewers, that were sufficiently addressed by the authors, should also make it to the paper, even if in the supplement.
Decision: Accept (poster)