KSP: Kolmogorov-Smirnov metric-based Post-Hoc Calibration for Survival Analysis
Authors: Jeongho Park
Daheen Kim
Cheoljun Kim
Hyungbin Park
Sangwook Kang
Gwangsu Kim
Venue: NeurIPS 2025 poster ()
Keywords: survival analysis
calibration
Kolmogorov-Smirnov metric
post-processing
Primary area: general_machine_learning
TL;DR: We propose a KS-based calibration method for survival models that avoids discretization and improves calibration while preserving predictive accuracy across real-world datasets and models.

Abstract: We propose a new calibration method for survival models based on the Kolmogorov–Smirnov (KS) metric. Existing approaches—including conformal prediction, D-calibration, and Kaplan–Meier (KM)-based methods—often rely on heuristic binning or additional nonparametric estimators, which undermine their adaptability to continuous-time settings and complex model outputs. To address these limitations, we introduce a streamlined $\textit{KS metric-based post-processing}$ framework (KSP) that calibrates survival predictions without relying on discretization or KM estimation. This design enhances flexibility and broad applicability. We conduct extensive experiments on diverse real-world datasets using a variety of survival models. Empirical results demonstrate that our method consistently improves calibration performance over existing methods while maintaining high predictive accuracy. We also provide a theoretical analysis of the KS metric and discuss extensions to in-processing settings.

KSP: Kolmogorov-Smirnov metric-based Post-Hoc Calibration for Survival Analysis
Jeongho Park
Daheen Kim
Cheoljun Kim?
Hyungbin Park
Sangwook Kang*
Gwangsu 2im
Department of Statistics and Data Science, Yonsei University, Scoul, Republic of Korea 2 *Department of Statisties, Jeonbuk National University, Jeonju, Republic of Korea Reeeeacc Institute for Materials and Energy Sciences, Jeonbuk National University Janggii)@yonsei..
{wjdgh4325, ddaccong99, kanggi1 @yonsei......... wwwosseiaa....... .r. {fefe2, hbag89503, s88012) bnu. ac.kr
Abstract
We propose a new calibration method for survival models based on the Kol- mogorov-Sminan Smirnov (KS) metric. Existing
conformal pre- diction, D-calibration, and Kaplan-Meier (KM)-based wethods-often rely on heuristic binning or additional nonparametric estimators, which undermine their adaptability to continuous-time settings and complex model outputs. To address these limitations, we introduce a streamlined KS metric-based post-processing framework (KSP) that calibrates survival predictions without relying on discretiza- tion KM estimation. This design enhances flexibility and broad applicability. We conduct extensive experiments on diverse real-world datasets using a variety of survival models. Empirical results demonstrate that our method consistently improves calibration performance over existing methods while maintaining high predictive accuracy. We also provide theoretical analysis of the KS metric and discuss extensions to in in-rrccessing settings.
Introduction
Calibration plays a vital role in ensuring reliable risk estimation for decision- making in survival analysis. Since poor calibration can misrepresent failure risk, leading to suboptimal or misguided decisions, it is especially critical high-stakes areas such as healthcare and infrastructure systems. Recently, deep neural networks (DNNs) have significantly improved predictive performance in survival analysis (Wiegrebe et al., 2024). as evidenced by models such as DeepSurv (Kim al.) 2019), Transformer based architectures (Hu et al., 2021), and GurvTrace (Wang and Sun 2022). These
models are typically evaluated using the concordance index (C-index: [Harrell et al. 1996). which measures model's 's ability to correctly rank individuals. Although DNNs improve discrimination, they often suffer from over-confidence, a widely recognized issue classification (Guo al.) 2017 Kumar et al.) 2018 Mukhoti et al. 2020), which results in poor calibration. Nevertheless, survival calibration remains underdeveloped and is further complicated by censoring and time-dependent risks.
Haidee One notable contribution to survival calibration is D-calibration (Distributional calibration; Haider et al. 2020), which compares the predicted probability with the observed failure proportion within
"Comesponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

predefined intervals. Building on this idea, X-cal (eXplicit calibration; Goldstein et al. 2020) incorporates D-calibration directly into model training. However, relies on binning, which may underestimate calibration error (Kumar et al.
2019) and often substantially reduces discrimination performance (Qi et al. 2024a Park et al. 2025 To reduce such trade-offs, recent studies (Avati et al.) 2020, Fuhlert et al. 2022, Qi et al. 22024a by Lee et al. 2024) have aimed to improve calibration while maintaining predictive performance. Some leverage conformal inference (Qi et al.) 2024a b) or outcome-aware sampling (Lee et al. 2024), but their reliance on fixed sampling schemes or predefined percentiles may limit adaptability.
A promising direction for addressing these limits comes from classical statistical tools. The Kol- Pogorov-Sminnng (KS) metric has long been used to assess discrepancies between empirical and estimated distributions (Fernández and Gretton| (219)), and has also been employed as a calibration metric in classification (Gupta et al. 2021] Arrieta- Ibarra et al. (022), offering a nonparametric, distribution-level evaluation of calibration error. Although the KS metric has also been applied in survival analysis (Rao, 1998; Fleming et al.) 1980; Schumacher]] 1984 Wul 2018 Ansin 2015] Cox and Snell 1968), such studies have predominantly focused testing, with limited attention given to its role in assessing or improving calibration error. A recent study (Park et al. 2025) introduced a KS metric for survival models, using the maximum deviation between predicted and empirical survival functions as a calibration measure. Building on this insight, we propose an efficient post- processing method that leverages the KS metric to improve calibration without compromising predictive accuracy. Our approach avoids binning, surrogate losses, and quantile estimation, providing a bin-free, global measure of calibration error.
Our main contributions are as follows:
1. We introduce simple and scalable post-processing method for calibrating survival models, eliminating the need for surrogate losses, sampling, and quantile estimation.
2. We provide theoretical and algorithmic insights into the KS metric, including its connection to calibration in survival analysis.
3. We conduct extensive empirical evaluations across diverse real-world datasets and models, showing that the method improves calibration without sacrificing predictive performance.
The remainder of this paper reviews the background and motivation for calibration, examines the theoretical properties of the KS metric, and presents the post-processing method along with experimental results. All proofs of the theorem and proposition are deferred to Appendices and
2 Related work
Evaluation In medical research, the calibration of survival models is often assessed graphically using calibration curves, where perfect calibration aligns with the 45-degree line representing equality between observed and predicted probabilitics (Alonzo | Crowson et al. 2016). Goodness-of test (Hosmer and Lemeshow 1980). For DNN- based survival models, 1-calibration, which evaluates predicted survival probabilities at a specific time point, is commonly used (Yan et al. 2022 Li al.] 2023; Xia et 2023), though it does not capture calibration across the full survival distribution. To address this, D-calibration was introduced to assess calibration over the entire distribution (Haider et al. 2020). However, its reliance on fixed bins may underestimate calibration error (Kumar et al. 2019). Yanagisawa 2023) uses KM-calibration as an evaluation metric.
can be evaluated using Hosmer-Lemeshow--ye
In-processing Calibration has also been incorporated into the model training process Goldstein et al. 2020; Chapfuwa et al. 2020 Avati et al. 2020 Lee et al., 2024 Park et al.) 2025). X-cal (Goldstein et al. (0200) introduces a ban-based D-calibration penalty term, while S-cal Park et al. 2025) replaces it with random- mendoominterval-eee interval-based term. Chaptuwa et al. (2020) integrates KM estimates directly into training. These methods enhance marginal calibration, but often at the cost of reduced discrimination performance. A notable exception is the contrastive learning-base method by Lee et al. (2044), which aims to enhance discrimination without sacrificing calibration.
Post- cest-processing To balance calibration and discrimination, post-processing approaches have been proposed primarily using conformal prediction techniques (Candès et al., 2023 Qi et al. 2024a b
Gui ct al. 2024 Qin et al. 2025 Davidov et al.) 2025). The CSD method (Qi ct al. 2024a) refines

---

survival probabilities via KM-based percentile adjustments, ensuring marginal calibration across models. offers a model-agnostic, post-hoc calibration framework, QQ al. (2024b) further extends this to calibrate conditional distributions for censored data.
3 Discrimination and calibration in survival analysis
We consider right-censored survival data, where the non-negative failure time T is subject to censoring. Let C' be the censoring time, and define the observed time as Y= min (T, C*) and the censoring indicator
I (T' I C). where II (A) denotes the indicator function, taking the value when A is true and O otherwise. Each observation consists of (Y, 6, z). where is a vector of covariates. Assuming independent censoring given covariates, T TL C z. the full dataset is denoted by {(Y,, 6,. >;)};== for a sample size of N. Let F (t x) = P (T < t z) denote the conditional cumulative distribution function (CDF). The corresponding survival function S, hazard function X, and cumulative hazard function A are as follows:
P P < t A T 2 122) and (t 2) = P (T > t 2) = 1 F 2)
A (t z) lim A.0
A
with the relationship (t =) = exp { {-A 2)), where (t) 2) = S (s ds
3.1
Discrimination
The C-index is a standard discrimination metric in survival analysis, assessing how well the model ranks individuals by risk. Harrell's C-index (Harrell Jr al. 1996) defined as
C-index Eies 6,11 (Y; < I M,) Eie, 8,1 (Y; < <,,)
where Mj denotes the predicted mean related quantity. The C-index ranges from 0 to with higher values indicating better discrimination. In our case, we use the predicted mean as M,.
3.2 Calibration and metrics
Calibration in survival analysis refers to the agreement between the estimated survival distributions and actual event occurrences. While discrimination is important, calibration must also considered to assess the uncertainty in predicted survival probabilities. A well-calibrated model should accurately estimate the probability of an event occurring by a given time. One fundamental approach to assessing calibration relies on the estimated CDF of event times. Specifically, if T F, then F (T) Unif (0, 1]; that is, the CDF evaluated at the true event time follows a uniform distribution over the interval [0, 1]. This probabilistic property forms the basis of several calibration metrics, including D-calibration and KM-calibration.
3.2.1
D-calibration
Haider et al. (2020) introduced D-calibration, measure designed to evaluate calibration under censoring by leveraging the distributional properties the estimated CDF. For uncensored data, the CDF values satisfy F (T) 2) Unif [o, 1]. In the presence of right censoring (i.e., T > C), the ~ Unif [F (C 1], Combining both cases, the following equality
distribution becomes F (T 2)
holds:
(I) Here, = b C [0.1] is a subinterval. The term P (T) 2) T) is computed as P (F (T) 2) = 1- defined as the sum of bin-wise squared differences:
By,5,= [51 (F (Y =) E) (1 8)P (T' 2) 1)] =
F--i I (Y x) +)+ b-q
h-
(F (Y 2x) < a). Based on this, D-cal
D-cal = Σ (BY
[oll Fo ( (2)1z) + +1 (1-5)p 8)P Fo (T 2) 2)) -)) -11)
TEI
where I = [T,,.... ..., T is set of B (>> 0) disjoint intervals partitioning [0. 1). and Fo denotes the estimated CDF with model parameters 0. D-cal measures the discrepancy between the observed proportion of estimated CDF values falling into each bin and the corresponding predicted probability.
3

---

3.2.2 K.M-caibrratonn
While D-calibration evaluates calibration in the probability space by assessing the uniformity of the estimated CDF values, an alternative approach focuses on the time domain. Specifically,, Yanagisawa (2023) and Qi et al. (2024a) proposed calibration metrics based on the KMM estimator. One such
2 metric is defined as KM-cal Fmax _o _ (SKM (t) Ex So (t +))) dt, where SKM (t) is the
1
KM estimator and Sg (t =) = Fe (t x) is the model-based survival function. This metric assumes that the KM estimator is well-calibrated and measures the squared discrepancy between the empirical and predicted survival probabilities over time.
Motivation for new algorithm
Several recent algorithms leverage D-calibration and KM-calibration metrics, including X-cal (Gold-
stein al.] 2020), SFM (Survival Function Matching: Chapfuwa et al., 2020), CSD (Conformalized
Survival Distributions; Qi et al.) 2024a), and CSD-iPOT (individual survival Probability at Observed Time; Qi et al. 2024b). X-cal and SFM incorporate empirical variants of D-cal and KM-cal as penalty terms during model training. For example, X-cal replaces the discontinuous indicator I (xr E [a, 6)) with a smooth surrogate Sy (z; [a, bl) = T T-expl (oon---(a--aa ((a=))(b==)) (for Y > 0), which enables gradient- based optimization. In contrast, CSD and CSD-iPOT adjust estimated survival distributions using conformal inference over quantile levels. These approaches are effective in reducing calibration error while maintaining predictive accuracy, but they have some limitations.
X-cal Binning-based methods, such as those underlying D-cal and X-cal, may underestimate cal- ibration error (Kumar et al.] 2019),, a limitation that extends to the survival setting. Additionally, benalty-based calibration can lead to degradation in discriminative performance, observed in Qi
ct all... 2024a); Park et al. (0055).
SFM This method involves computing [z (t =)], which can be computationally intensive. Moreover, when the model's estimated survival function is already better calibrated than the KM estimator, the KM-based correction may not further reduce calibration error.
CSD and CSD-iPOT CSD relies on the KM estimator for handling censored observations, whereas CSD-iPOT avoids KM-based estimation. However, both methods involve sampling procedures, which can introduce additional variance, especially in the tail regions, and may result in non-monotonic quantiles (i.e., quantile crossing).
These limitations motivate a calibration approach that (i) avoids discretization and binning, (ii) preserves discriminative performance (e.g.. C-index), (iii) does not require computationally expensive sampling, and (iv) remains stable in tail regions.
The KS metric is a classical tool for quantifying discrepancies between empirical and target distri- butions. In the calibration context, has been applied to classification tasks (Gupta et al. 2021 Arrieta-lbarra et al. 2022) as nonparametric, bin-free measure of calibration error. Among existing tools, Cox-Snell residuals (Cox and Snell 1968) can be used to evaluate calibration error. How- ever, they rely on the cumulative hazard function and are less practical than CDF-based approaches. Therefore, we focus on an alternative form of calibration error. Previous work of Park et al. (2025) proposed using the KS metric calibration measure in survival analysis. We refer to this metric as KS-cal and build upon it by introducing a post-processing algorithm that improves calibration while preserving predictive accuracy.
Let U. = Po (Y: 21). The KS-cal is defined as
N Σ 1(0 s x) { (1 -6, 5,) x -
KS-cal = sup F (x) - x/, where F (x) =E(0.1]
(2)
1- U:
For finite samples, approximate KS-cal using the evaluation points q, = Fo (v, |z,) as KS-cal maxig/SN Dj. where D, max(D,.s Dr,} with Dj.,. [F((g) - 9n and D FF(g))
5,/N 1.l. The term Dj.1 accounts for the left-limit behavior of J at uncensored points.
We observe that İ (x:) exhibits jumps at uncensored points while increasing linearly over censored points. Figure D illustrates this behavior, showing how Dj is computed at uncensored observations

---

Pictoraal example how to find
Uncaliarated model
Calibrated modal
Asi
1.54
0.08pa
D2
0.4
9-293G
6.0
0.00
020
12.m max (
0160
040
40
642
044
0/66
0.40
3.50
0.4
3.20
2.40
0.50
0.00
1.00
3.0
0.20
0.50
0.00
1.80
1.5
Figure 1: Examples of (zr). The solid blue line represents Fxx), and the dashed line shows the identity line y = I. The left panel indicates how Dj computed at uncensored observations. The middle and right panels illustrate uncalibrated and calibrated models, respectively, with respect to KS-cal. Red arrows indicate points of evaluation KS-cal.
and contrasting uncalibrated and calibrated models in terms of KS-cal. This metric offers bin- free assessment of calibration error and is conceptually related to D-calibration (Haider et al. 2020), although key differences in how calibration is measured remain. D-cal evaluates deviations within fixed bins, whereas KS-cal captures the maximum discrepancy at data-adaptive points where miscalibration is most evident, as determined by Pg. In the following section, we formalize the theoretical properties of the KS-cal, including consistency and convergence behavior.
4.1 Property of KS-cal
We now analyze the theoretical behavior of KS-cal. We begin with the simplified setting without covariates, i.e., U, = Po (Y2), where calibration corresponds to Pg = F, with F denoting the true CDF of T. The following result has been established in the previous work Pf et al. (2025).
Proposition 4.1. Assume the Regularity Conditions in Appendix A if and only if
Then calibration holds
|F (2) x| op (1) as N + (0.1
sup
We now extend this result to the setting with covariates, i.e., U. = Fg (Y; 2;). Following Goldstein ct al. (020), calibration with covariates defined as:
I (F)))))))
{ 8 8+ {0+(1-0) 8) (Y z) 1
Ey
1 Fo (Y 2) {
x, Vr 10.
Then, we have the following theorem.
Theorem 4.1. Assume the Regularity Conditions in Appendix covariates) holds if and only if
Then calibration (with/without
sup |8 (x) x| = op (1) as N DC. xE(0.1]
This result implies that fe converges to the CDF of Unif [0, 1] when the model is calibrated, where the theorem has two folds (with/without covariates). Therefore, we can establish KS-cal theoretically grounded, bin-free metric for evaluating calibration survival models. Moreover, under the regularity conditions in Appendix the convergence of (and thus KS-cal) occurs the rate of ON -122) up to log N.
3). an alternative calibration metrie termed S-cal (Summation-bian Calibration) has been proposed (Qi et al.) 2024a by Park et al. 2025). This metric captures cumulative deviations from ideal calibration and defined as:
Remark. Based Eqn.
Ey.8.z 1 (Foo 2) (x)
{ 8 88+ 6+(11 8)
n. (Y)
2) -
27
S-cal = E,
Fo (Y
2)
5

---

S-cal, D-cal, and KS-cal are all variants of D-calibration, derived from the same foundational definition, but they emphasize different aspects of calibration behavior. We include all three in our evaluation to provide a comprehensive view of model calibration.
5 Calibration method based on KS-cal
In this work, we propose a post-hoe calibration method based on the KS-cal introduced in Section 4. We also briefly describe how the KS-cal can be incorporated penalty term during model training.
5.1 Post-processing
We propose KS-cal based post-processing (KSP), similar to Platt scaling (Plau 1999) in classification tasks. We simply transform the original Fe into modified Fa S [0, 1] that minimizes the KS-cal. The procedure is summarized the following algorithm.
Algorithm. KSP
1: Input: Estimated CDFs Fo. strictly monotone increasing link function G: [0,1] (-00,
2: Initialize parameters a (> 0), (> 0)
3: Sort Fg for computational efficiency
4: while KS-cal not improved do
5: Compute transformed CDF: Fg = fa- G[Fo) + }
6: Compute KS-cal on validation set: max Dj, where Dj denotes Dy evaluated using Fj ISiEN
7: Update (a, b, or) via gradient descent (ADAM) to minimize the KS-cal
8: end while
9: Apply final calibrated transformation to the test set using optimized (a, b, a)
Output: Calibrated CDF F
The function G can be any link function satisfying the two predefined conditions described in the Input, such as the logit function, inverse hyperbolic tangent function, or an inverse CDF. Although a DNN-based transformation feasible, it is challenging to preserve the ordering of the transformed CDFs consistent with the original ones. In this work, we adopt the logit function as G.
The three hyperparameters a, b, and control different aspects of the CDF transformation:
influences the tails, shifts the distribution relative to the target x, and a controls nonlinearity. Their effects visualized and discussed in the Appendix As long as 0 and a > the transformation preserves the ordering of CDF values, thereby maintaining the time-dependent C-index (Antolini et al.
2005), consistent with Qi et al. 2022b). Specifically, if (e x1) Fe x2). then the transformed
CDF also satisfies (t 21) (t 22). While the CDF ordering is preserved, the ordering of expected survival times may not be, as the transformation can nonlinearly distort the survival curve.
This is particularly relevant when survival curves intersect. In such cases, monotonicity is maintained at each time point, but the area under the curve may change, which turn affects the expected value. In practice, such changes are typically small, and overall concordance may even improve. We also formalize the conditions under which the mean ordering is preserved. In particular, the original survival curves do not cross (preserving the order across all times), the transformation preserves the ordering expected survival times. Theoretically, these are summarized as:
Proposition 5.1. Ler E [T 2 Jo S* (t z) dt denote the expected survival time under the KSP, where S* (t 2) = 1 F* (t 2). If the original survival curves S 21) and S (t 22) do not cross, then the ordering of expected survival times is preserved under the KSP:
E 2 T 22]
E E
21 > E 22].
The proof is provided Appendix C The non-crossing condition holds for some standard models, including the Cox proportional hazards (PH) model (Cox 1972) and the Weibull Accelerated Failure
Time (AFT) model (Stute) 1993). By Proposition 5.1 the C-index is preserved since we use the predicted mean of survival time for the C-index.

---

Compared to prior methods such as CSD [Qi et al. 2024a) and CSD-iPOT ( et. 2024b), KSP provides better computational efficiency. CSD requires O ON (P]- R) operations, and CSD-iPOT requires O (N R), where N and [P] are the sample size and the number of quantiles. respec- tively, and R is the number of samples used for censored data. In contrast, KSP requires only O ( N + N log N), which simplifies to O [ N) in practice, where B is the number of optimiza- tion iterations. The log N term accounts for the initial sorting of CDF values, and each iteration involves linear-time operations. Although the number of iterations may increase when the initial calibration error is large, the overall overhead remains modest and can be controlled through the leamingg rate. This scalability makes KSP particularly suitable for large datasets.
5.2 In- In-processing
KSP is introduced as a post-prosee post-proesssing method to reduce calibration errors while preserving predictive accuracy. Alternatively, the KS-cal defined in ( can be incorporated directly into training penalty term: Ps (0) = =ズ (P(N-++1)) where the subscript denotes the order statistic. Unlike the original KS-cal, which considers only the maximum deviation, this variant aggregates the top-k largest deviations. Using the squared form improves gradient stability, making it more suitable for optimization. Since the deviations are evaluated only at observed points, the indicator function in contrast, methods like X-cal
F (I) naturally disappears when the CDF values are pre-sorted. require surrogate function to approximate the indicator for gradient-based learning. We refer to this variant as KK)cal(k). While our primary focus is on post-processing, we also explore this in-processing approach in Appendix []] Although the penalty function is not convex, we observe that training remains stable and converges well in practice. However, using excessively small batch sizes can lead to large approximation errors; thus, recommend employing relatively large batch sizes for reliable performance.
6 Experiment
We first compare post-processing methods, including CSD (Qi et al. 2024a), CSD-iPOT (Qi et al., 2024b), and KSP, across various models and datasets. We also evaluate in-processing approaches such as X-cal (Goldstein et al., 2020) and SFM (Chapfuwa et al. 2020). While SFM originally combines a calibration penalty with discriminative loss, we use only the calibration component for consistency. For all in-processing methods, we tune the regularization parameter over {1. 10, 100, 1000}. Exper- iments are conducted with Intel Xeon Silver 6226R CPU and NVIDIA GeForce RTX 3090 GPU.
6.1 Experimental setup
Baselines We consider six baseline models: DeepSurv-based Cox PH model (Katzman et al. 201 8
Multi-task Logistic Regression (MTLR: Yu et al. (01I), I), a parametric model (Goldstein et al. 2020
Survival CRPS (Avati et al.) 2020), DeepHit (Lee et al. 2018), and the Weibull AFT model (Stute
1993). As a reference, we include the KM estimator fitted on the training set and evaluated on the test set, following Qi al. (2024a b). The KM estimator serves
empirical lower bound for calibration error (see Appendix B in Qi et al. (2024a)) ). Further details on the baseline models are provided in Appendix
Datasets We evaluate methods on ten benchmark datasets: WHAS, METABRIC, GBSG, NACD, NB-SEQ, SUPPORT, MIMIC-III, SEER-liver, SEEER-ttomchh, and SEER- These are grouped by sample size into three categories: Small (WHAS, METABRIC, GBSG, NACD), Medium (NB- SEQ. SUPPORT, MIMIC-III), and Large (SEER-liver, SEER- SEER-lung). Details on the datasets and preprocessing steps are provided in Appendix D Each dataset is randomly split into training. validation, and test sets in 3:1:1 ratio, with balanced censoring rates. All experiments are repeated 30 times with different random seeds. For CSD and CSD-iPOT, we use the validation set as the conformal set to enable fair comparison with KSP.
Evaluation metries For predictive accuracy, we report the (time-independent) C-index (Harrell Jr et al. 1996) rather than the time-dependent variant, as the latter may favor our method and CSD- IPOT Qi al. 2024b), potentially inflating their discrimination performance. For calibration, we evaluate five metries: S-cal(20), D-cal(20), KS-cal, KM-cal, and the Integrated Brier Score (IBS;
7

---

Table Summary of pairwise comparisons between pistcesssssing methods. The table shows the number of cases where KSP outperforms its counterpart, is outperformed, or yields a tie. Numbers in parentheses indicate statistically significant differences based on a sided I-test at the 0.05 level.
Method
C-index
S-cal(20)
D-cal(20)
KS-cal
KMMcall
IBS
KSP
20 (12)
46 (45)
46 (43)
47 (45)
47 (35)
44 (25)
Non-calibrated
18 (1)
(7)
14 (6)
13 (5)
13 (10)
16 (1)
Ties
22
0
0
KSP
13 (2)
36 (29)
48 (45)
51 (42)
37 (32)
42 (25)
CSD
34 (2)
(19)
12 (10)
9 (
23 (19)
18 (10)
Ties
13
0
0
0
KSP
21 (0)
32 (21)
46 (39)
44 (29)
45 (36)
38 (9)
CSD-iPOT
25 (1)
28 (19)
14 (13)
16 (11)
15 (10)
22 (10)
Ties
14
0
0
0
0
0
C-index
S-cal(20)
D-cal(20)
KSP
0.00
CSD
CSD. CSD/PPOT
Inference Runtime sec.)
80
1e40
fede
o0e
60
KS-cal
KM-cal
IBS
Small
Medium
Large
Sample Size
Figure 2: Boxplots of metric values (left) and inference runtime by sample size (right), aggregated across all datasets and models.
Graf et al. 1999). S-cal(20) is computed over 20 equally spaced points {0.05, 0.10, .... 1.00],
while D-cal(C (20) uses 20 equal-width bins. The results based on a 10-bin evaluation are provided in Appendix H KS-cal is bin-free metric. KM-cal and IBS assess 1-calibration and offer complemen- tary perspectives. Each model is thus evaluated using one discrimination metric and five calibration metrics. For CSD and CSD-iPOT, we follow the original setup by using P {0.1, 0.2, .... 0.9) for quantile adjustment and set R = 1000 to sample censored observations.
6.2 Experimental results
Comparison with post-procsssing methods We conduct x 60 experiments across models and datasets. For each setting, we perform pairwise comparisons among the non-calibrated baseline, CSD, CSD-iPOT, and KSP, counting how often each method outperforms the others. summarized in Table D non-calibrated baseline, KSP consistently achieves better calibration and significantly improves the C-index in 10 cases, especially for the parametric model and CRPS. Against CSD, KSP shows better calibration, while CSD slightly leads in the C-index due to its guaranty of preserving the iime-independent C-index. Compared to CSD-iPOT, KSP offers similar C-index with elearly better calibration.
KSP outperforms the alternatives in approximately 70% of cases. Compared to the
KSP's improvements are more evident in D-cal(20) and KS-cal, although it still performs competi- tively S-cal(20), which tends favor CSD and CSD-iPOT due their quantile- based structure. similar pattern appears in discretized models like MTLR and DeepHit, where KSP shows slightly weaker calibration, possibly due to structural mismatch. Nonetheless, KSP outperforms in KM--aal and IBS, confirming its overall calibration strength. Figure further illustrates that CSD and CSD- iPOT often exhibit calibration mismatches near the left tail due to boundary interpolation, which D-cal(20) and KS-cal capture effectively. Full results presented in the Appendix F
8

---

KSP also offers efficiency advantages. As shown in Figure 2 runtime increases with sample size for all methods, but the growth is slower for KSP. This because CSD and CSD-iPOT require additional bootstrap sampling for the monotonicity of quantiles, while KSP depends only on the number of optimization steps. The complete runtime summary is provided in Appendix KK] Figure also presents the distribution of each metric across all experiments, with the median values in the boxplots serving as a key point of comparison. All methods show comparable C-index values, confirming that post- processing can improve calibration without compromising discrimination. KSP consistently achieves the best performance in S-cal(20), D-c,l(20), and KS-cal. Although CSD records the lowest median KM-call, KSP surpasses it more frequently in pairwise comparisons, indicating greater robustness. KSP shares a similar property with CSD-iPOT in preserving time-dependent discrimination, as reflected in their comparable performance on KM-cal and IBS.
Calibration plot Figure 3 shows calibration plots for MTLR and CRPS on the MIMIC-II dataset. Although all three post-processing methods achieve similar quantitative scores, CSD and CSD-iPOT exhibit slight mismatches near the left tail, likely due to boundary interpolation adjustments. In contrast, KSP reduces the maximum deviation, resulting in more uniform calibration across the entire range. This pattern aligns with the D-cal(20) and KS-cal results in Table [ where KSP outperforms other methods by better capturing localized discrepancies. Full results are provided in the Appendix G
their quantile
MTLR
MTLR
CRPS
KSP CRPS
KSP
KSP
Obppeed P
procadie
Obsaredd Deperedddaaaaa by
"Predicied probabillty
Prodikled probabilty"
Predicied probbiiy
"Prddicied Prodkciedrrocbaii
Figure 3: Calibration plot comparison among KSP, CSD, and CSD-iPOT for MTLR and CRPS on the MIMIC-III dataset.
Comparison with in-processing methods We report comparisons with in- in-processing methods (SFM and X-cal) for CRPS on the MIMIC-III dataset, which illustrate typical trade-off patterns. As X increases, both in-processing methods reduce calibration error but degrade the C-index. SFM, which penalizes KM-calibration, lowers K.Mccal and IBS but performs worse on D-calibration. X-cal improves D-calibration but incurs a greater C-index loss and increases KM-cal and IBS, particularly 1000. In contrast, KSP achieves a better trade-off with stable performance across metries and no need for tuning. While in-processing may be helpful when calibration is the sole priority, KSP offers more practical and balanced solution when both discrimination and calibration are important. Figure 4 summarizes these results, showing the mean and standard error across multiple À values.
C-index
S-cw||20)-
D-cal(20)
KS-cal
KM-cal
I
3.
0.00
0.55
200
I
0.10
0.13
506
1
0.7
0.0
0.0
5.00
Non-oalibratad
KSP
SFM
X.oal
Figure 4: Comparison of KSP and in-processing methods (SFM, X-cal) the MIMIC-III dataset with CRPS. Points show means and standard errors across À E (0, 10, 100, 1000} for in-processing methods. Dashed lines indicate KM-level calibration errors.
Ablation study We conduct an ablation study to assess the effects of various design choices in the KSP optimization procedure. Specifically, we investigate (1) the choice of link function G, (2) the role of hyperparameters b, and a, and (3) alternative loss formulations beyond the default max, DJ.

---

Table Ablation results on the MIMIC-III dataset using DeepSurv, showing the effect of including hyperparameters a, b, and in the KSP.
Hyperparameter setting
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
Non-calibrated
0,95839
0.036649
0.064547
0.250907
0.104279
0.276509
0,95826
0,011849
0.041974
0.183154
0.077582
0,230868
a, b o, b. &
0,95826
0,005908
0.025273
0.127957
0.076690
0.213331
0.95826
0.003305
0.007044
0.091109
0.071617
0.203039
as detailed the Appendix ! In the main text, we present partial results for (2) on the MIMIC-III dataset using DeepSurv, summarized in Table with full results deferred to the Appendix We find that using b alone substantially improves calibration, reducing D-cal(20) and KS-cal by 35% and 27%, respectively, without degrading the C-index. Introducing and yields further gains, especially in D-cal(20), by enabling scale adjustment and smoothing. While the magnitude of improvement may vary across datasets and models, all three hyperparameters contribute positively the KSP procedure. Note that "Non-calibrated" in Table refers to the default setting with = 1, b = 0, = 1, that is, when KSP is not applied.
Conclusion
Our work highlights the effectiveness of KS-cal a calibration metric and introduces KSP as a practical post-processing method for improving the reliability of survival models. Beyond its empirical and computational advantages, KSP offers a theoretically grounded framework and avoids the limitations of bin-based or sampling-dependenn approaches. While KSP shows clear advantages, tends be less effective in datasets with heavy ties or severe skewness; however, robust when calibration errors are large and in capturing local discrepancies, such as tails. As an important direction for future research, we aim to extend KSP toward conditional calibration, enabling personalized and subgroup- reliability assessments. Such developments are particularly valuable for high-stakes domains where accurate uncertainty quantification is critical.
more
Acknowledgment
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) (RS-2023-00218377), Global Learning & Academic research institution for Master's-Phi students, and Postdocs (LAMP) Program of the NRF grant funded by the Ministry of Education (RS-2024-00443714)
References
Alonzo, T. A. (2009). Clinical prediction models: A practical approach to development, validation, and updating: By ewout w. steyerberg. American Journal of Epidemiology, 170(4):528.
Ansin, E. (2015). An evaluation of the cox-snell residuals. Master's thesis, Uppsala University, Department of Statistics, Uppsala, Sweden.
Antolini, L., Boracchi, P., and Biganzoli, E. (2005). A time-dependent discrimination index for survival data. Statistics in Medicine, 294244439
Arrieta-lbarra I., Gujral. P., Tannen, J., Tygert, M.. and Xu, C. (2022). Metrics of calibration for probabilistic predictions. Journal of Machine Learning Research, 23:1-54.
Avati, A., Duan, T., Zhou, S., Jung, K., Shah, N. H., and Ng, A. Y. (2020). Countdown regression: sharp and calibrated survival predictions. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, pages 145-155. PMLR.
Candès, E., Lei, I.., and Ren, Z. (2023). Conformalized survival analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(1):24-45.
10

---

Chapfuwa, P., Tao, Li, C.. Khan, I., Chandross, K. J., Pencina, M. J., Carin, L., and Henao, R. (2020). Calibration and uncertainty in neural time-to-event modeling, IEEE Transactions on Neural Networks and Learning Systems, 34(4):1666-168))
Cox, D. R. (1972). Regression models and life-tables. Journal of the Royal Statistical Society: Series B (Methodological), 34(2:187-2022
Cox, D. R. and Snell, E. J. (1968). A general definition of residuals. Journal of the Royal Statistical Society: Series B (Methodological), 30(2):248-288
Crowson, C. S., Atkinson, E. J., and Therneau, T. M. (2016). Assessing calibration of prognostic risk scores. Statistical Methods Medical Research, 25(4):1692-1700
Davidov, H., Feldman, S., Shamai, G., Kimmel, R., and Romano, Y. (2025). Conformalized survival analysis for general right-censored data. In the International Conference on Learning Representations. https://openrevier net
Fernández, T. and Gretton, A. (2019). A maximum--nann--iseeeaaaa goodness-of-fe test for censored data. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 89, pages 2966-2975. PMLR.
Fleming, T. R., O'Fallon, R., Brien, P. C., and Harrington, D. P. (1980), Modified kolmogorov- smirnov test procedures with application to arbitrarily right-censored data. Biometrics, pages 607-625.
Fuhlert, P., Emstt, A., Dietrich, E., Westhaeusser, F., Kloiber, K., and Bonn, S. (2022). Deep learning- based discrete calibrated survival prediction. Proceedings of the IEEE International Conference on Digital Health, pages 169-174.
Goldstein, M., Han, X., Puli, A., Perotte, A., and Ranganath, R. (2020). X-cal: explicit calibration for survival analysis. In Advances in Neural Information Processing Systems, volume 33, pages 18296-18307.
Graf, E., Schmoor, C., Sauerbrei, W., and Schumacher, M. (1999). Assessment and comparison of prognostic classification schemes for survival data. Statistics in Medicine, ((17-18))2529)
Gui, Y., Hore, R., Ren, Z., and Barber, F. (2024). Conformalized survival analysis with adaptive cut-offs. Biometrika, 11 111(2):459-477.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the International Conference on Machine Learning, volume 70, pages 1321 1330. PMLR.
Gupta, K., Rahimi, A., Ajanthan, T., Mensink, T., Sminchisescu, C., and Hartley, R. (2021). Calibra- tion of neural networks using splines. In the International Conference Learning Representations. https://openrevierz wet.eewwweeeeeeeee.ee n http://ooruumnddeeeeeeeeene
Haider, H., Hochn, B., Davis, S., and Greiner, R. (2020). Effective ways to build and evaluate individual survival distributions. Journal of Machine Learning Research, 21(1):3289-3311.
Harrell Jr, F. E., Lee, K. I.,, and Mark, D. B. (1996). Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine, 154):361-3377
Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., and Galstyan, A. (2019). Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1):96.
Hosmer, D. W. and Lemeshow. S. (1980). Goodness of fit tests for the multiple logistic regression model. Communications in Statistics-Theory and Methods, 1403
Hu, S., Fridgeirsson, E.,, van Wingen, G., and Welling, M. (2021). Transformer-based deep survival analysis. In Proceedings of Survival Prediction: Algorüthms, Challenges and Applications, pages 132-148.
11

---

Karandikar, A.. Cain, N., Tran, D., Lakshminarayanan, B., Shlens. J., Mozer, M. C., and Roelofs, B. (2021). Soft calibration objectives for neural networks. Advances Neural Information Processing Systems, volume 34, pages
Katzman, J. L., Shaham, U.. Cloninger, A., Bates, J., Jiang, T., and Kluger, Y. (2018). Deepsurv: Personalized treatment recommender system using a cox proportional hazards deep neural network. BMC Medical Research Methodooogy, 18(1):24.
Kim, D. W., Lee, S., Kwon, S., Nam, W., Cha, I.-H., and Kim, H. J. (2019). Deep learning-based survival prediction of oral cancer patients. Scientific Reports, 9(1):6994.
Kumar, A., Liang, P. S., and Ma, T. (2019). Verified uncertainty calibration. In Advances in Neural Information Processing Systems, volume 32, pages 3792-3803.
Kumar, A., Sarawagi, S., and Jain, U. (2018). Trainable calibration measures for neural networks from kernel mean embeddings. In Proceedings of the International Conference Machine Leurning, volume 80, pages 2805-2814. PMLR.
Lee, C., Zame, W., Yoon, J., and Van Der Schaar, M. (2018). Deephit: A deep learning approach to survival analysis with competing risks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 2314-2321.
Lee, D., Park, H., and Lee, C. (2024). Toward well-calibrated discrimination via survival outcome- aware contrastive learning. In Advances in Neural Information Processing Systems, volume 37, pages 10914-31014.
Li, R., Qu, W., Liu, Q. Tan, Y., Zhang, W., Hao, Y., Jiang, N., Mao, Z., Ye, J., Jiao, J., et al. (2023). Development and validation of deep learning survival model for cervical adenocarcinoma patients. BMC Bioinformatics, 1-(1)
Lin, D. Y. (2007). On the breslow estimator. Lifetime Data Analysis, 13(4):471-480)
Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P., and Dokania, P. (2020). Calibrating deep neural networks using focal loss. In Advances in Neural Information Processing Systems, volume 33, pages 15288-15299.
Park, J., Kang. S., and Kim, G. (2025). Stochastic explicit calibration algorithm for survival models. IEEE Access, 13:78679-78706.
Platt, J. C. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, pages 61-74. MIT Press
Qi, S.-A., Yu, Y., and Greiner, R. (2024a). Conformalized survival distributions: A generic post- process to increase calibration. In Proceedings of the International Conference on Machine Learning, volume 235, pages 41303-41339, PMLR.
Qi. S.-A., Yu, Y., and Greiner, R. (2024b). Toward conditional distribution calibration in survival prediction. In Advances in Neural Information Processing Systems, volume 37, pages 86180- 86225.
Qin, J., Piao, J., Ning, J., and Shen, Y. (2025). Conformal predictive intervals in survival analysis: a resampling approach. Biometrics, 8 (2)ajaf80633
Rao, M. (1998). Survival analysis, techniques for censored and truncated data. Technometries, 40(2):159-160.
Schumacher, M. (1984), Two-sample tests of cramér-von mises- and kolmogorov-smimoo--ypee for randomly censored data. International Statistical Review, 22(3)226.-281
Stute, W. (1993). Consistent estimation under random censorship when covariables are present. Journal of Multivariate Analysis, 45(1):89-103.
Van der Vaart, A. W. (2000). Asymptotic Statistics, volume 3. Cambridge University Press.
12

---

Wang, Z. and Sun, J. (2022). Survtrace: Transformers for survival analysis with competing events. In Proceedings of the ACM International Conference on Bioinformatics, Computational Biology & Health Informatics, pages 1-9.
Wiegrebe, S., Kopper. P., Sonabend, R., Bischl, B., and Bender, A. (2024). Deep learning for survival analysis: a review. Artificial Intelligence Review, 57(3)65..
Wu, T. (2018), Randomized survival probability residual for assessing parametric survival models. Master's thesis, University of Saskatchewan, School of Public Health, Saskatoon, Canada.
Xia, Y. Zhang, B., and Zhang, Y. (2023). Deep survival analysis using pscudo values and its application to predict the recurrence of stage iv colorectal cancer after tumor resection. Computer Methods in Biomechanics and Biomedical Engineering, 27(15):1-10.
Yan, L.., Gao, N., Ai, F., Zhao, Y., Kang, Y., Chen, J., and Weng. Y. (2022). Deep learning models for predicting the survival of patients with chondrosarcoma based on a surveillance, epidemiology, and end results analysis. Frontiers in Oncology, 12:967758.
Yanagisawa, H. (2023), Proper scoring rules for survival analysis. In Proceedings of the International Conference on Machine Learning. volume 202, pages 39165-39188 PMLR.
Yu, C.-N., Greiner, R., Lin, H.-C., and Baracos, V. 1). Learning patient-specific cancer survival distributions sequence of dependent regressors. In Advances in Neural Information Processing Systems, volume 24, pages 1845-1853.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We included our limitations on Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
13

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide the theoretical result in Sections 4.1 and 5.1 and the proofs in Appendices A and C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimental settings, hyperparameters, and evaluation protocols are fully described in the main paper and Appendix. The corresponding code is also publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may
14

---

be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All datasets used are public benchmarks, and we provide sounc code with instructions to reproduce all experiments at at/htps://gtt thub. com cotp://jghh233//SS-aaa as also noted in the Appendix F.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
pebllic/guide... http://guiddee//eeuubmm.eeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips. ctaips.cc/ublic//gee gwtp:/idddee/eee wtt.:e/ee/oooeeeueeeeeeeeeeaeeea
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and Non- canccabbbatted. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
15

---

Answer: [Yes]
Justification:
We specify details on data split in Appendix D, other details in Appendix K.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Table shows statistical significance, and we provide error bars for experimen- results in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify compute resources in Section 6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
16

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htt.iggtttts///oooorooooooooo//t
Answer: [Yes]
Justification: We follow NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impact in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No high-risk models or datasets are used or released in this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
17

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We used publicly available code assets under the MIT License:
httpe///gihabb.com/rjjjettllb
httpes//gitaabb.cco//hhhaaa/gg/
All licenses and attributions have been properly acknowledged, and relevant papers are cited in the Appendix Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi www.ww..tth.....o... com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We will open new assets after the paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [No]
Justification: Not applicable.
18

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [No]
Justification: We did not the LLM for the parts of the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.M policy (https: /att:g///earrrssccc///offereeeee for what should or should not be described.
19

---

Appendix
Broader impacts
Improving survival calibration has broader impacts on clinical decision-making, fairness, resource allocation, and policy development. By producing accurate risk estimates, it supports more reliable and balanced decisions across medical and public health domains.
Code availability
The source code for reproducing our experiments is available at: https://github. com/ wjdgh4432/KS-cal
A Proofs of Proposition 4.1 and Theorem 4.1
We denote Fo Fa for notational clarity.
Regularity Conditions
1. With or without covariates, 1'3" (x) exists due to the required properties of Fgg such as left-continutty, the existence of right-hand limit, and monotonicity. For simplicity, we omit the dependence
in the notation when covariates present.
2. For the case without covariates, when non-calibration holds,
there exists an x² E (0, 1) such that
7 1= (FF(FF (x') x² 2 1/-0 F (F (s) (F5((F5(s))
10122/
and for the above x², J dG
F (s)
<1
where F denotes the true CDF of the failure time, and G denotes the CDF of the censoring time.
Proof of Sufficient Condition in Proposition 4.1
For the proof, we F (x)
N EN F (z) where Fg (x) = 1 (U; x) 8 + (1 6,) Up for
0 S x $ = Fg (Y.). By Eqn. in Lemma A.2 we have E. F. (x)) = x. F. (a)) =
h-(1- (1-x) -0 -ー dG (F-1 (s) < and 0 F < 1. Then, by the Bernstein incquality Van der
Vaart 2000 stating that for independent random variable X, such that E [X,] 0, XX < M.
P
Σ
x
x,
S exp
2
B[X Me/3
For any < < <<<
1/2, we have a set S of points between [0, 1] such that (0, e, 2e, [1/ele, 1}. implies
AI E [0, 1], 3k E s.t. k| < e. This implies that
sup
F (z) x 1
sup [0
(() x s 2
JES
Then,
P sup
[F [F)) x >
<P
sup
F (x) >e
Σ
P [1)
zE(0.1]
reS
20

---

By the Bernstein inequality, P (x) - >C) 2 (xp((=2/(2N++/33) due the symmetry of the absolute value and the expectation of the square is less than 2. This implies
P (1)
<) s 2exp (- Ne/3))
exxp (-Ne
Finally, we have
Ne (f(x) x > 2c s exp
sup =E(0.1]
- 0
(4)
as N + 00.
Proof of Necessary Condition in Proposition 4.1
Assuming the regularity conditions, the fact that non-calibration implics that
F(x)= N E I(U ((U.5x) 8, }
5,) x U3 U4
has the deviation en 0 from some x² is sufficient for the proof. By Lemmas Regularity Conditions imply that when the non-calibration holds, suPxeju.1] with probability approaching as N This completes the proof.
A.2
A.3
and the
| x en/
Proof of Theorem 4.1
For simplicity, we let
U:
0U) (U,SSU()) 5t) 8, 8,)
U,
Lemma A.1 and the definition of calibration such as [z] (U1,6,) imply
sup F (z) 20 xe [0
in probability as
Furthermore, when non-calibration holds such as - x 8, we have (uup |F (x) ø| 6/2) + 1 as N co.
N
su [F
(x)]
This implies that calibration holds if IP (supxeju.1) F (x) - x > 6/2) - 0 as N + co. it is by contraposition.
Lemma A.1. Assuming that the Regularity Conditions hold, we have that for any 0 < <<< 1/2
N
sup =E
Σ
[
5,) -E (U,, 8:)]] >(
N
t=1
goes to 0 as N where 2/ (U,, 5,) I (U, S x) { 6, (1 5:) i-; U, )
} and U; Fg (Yi 2,).
Note that we have x (U, 8.) < 1 and (z (U,, 5,) - EU, 61) < 2. Then, by the Bernstein inequality (Van der Vaart 2000) stating that for an independent random variable X, such that
[X,] = 0, M,
2
P
Σ
X; >E
S Sexp
[X + Me/3
21

---

For any 0 < 6 < 1/2, we have set S of points between [0, 1] such that {0, c, 2e, [1/cle, 1}. It immples
Vt [0, 1], 3k s.t.
$ e. This implies that
N
sup #ES
2 (Us, 6:)
E (Us, 6:)l S
N
C
sup
2 (U:, 6:) - E 1 (U),
S 220
=E|0.1]
N
1=1
Then,
sup
Σ 1=1
b (U,, - B (U, 6,]]]
> 2e
x=|o,11
P
sup
2 (U4,6)) -B (U2 6:)1l
sE8
N
P
[2] (U, 5.) E |2 (U,, (U,,
ZES
By the symmetric applications of Bernstein inequality,
N
P
x
(U, 6.) E [2 (U,, 6:)] > 2
S 2expp (-2/(2N + c/3)),
N
which implies
N
Σ
2
(U,
[ (Uz, 6:)1] > < 2exp (-N22 + Ne/3))
N
This is due to the fact that the square-expectation less than Finally, we have
Ne
sup
( (U,, 6.) - B (U,, 6:)]] > 2s < exp
+0 0
(5)
N
1=1
as + 0o.
Remark. If we c Mx log N for any diverging sequence MN, then the probability in the VN
right term of Eqn. 5 approaches 0 as N + co. This ensures the convergence rate of (,(1/vN) up to log
Lemmas for the proof of Proposition 4.1
Lemma A.2. Under the Regularity Conditions and Fg F, we have
E [((x) =
and E [((x)2] = x x-(1- x) I 1
2
dG (F-1(s))
Proof. Regularity Conditions and calibration of F' imply that
F (F, (s) s.
Proof of P. (4)
22

---

6 + (1 0)) ワーー U
T FF F.C G and 2d (U,6) IU 5x)
1-UJ
Then Fi del - (U,, 6.). We have
IP (U S xx 11 = PP (min (Fo(T), S CC) = P st,
P P (x), (0) //- ()) (a)
dFF(S)
(+)
[] d())
1/ [ - G FFo (s) ] dF (F)
(6)
Similarly,
P (U =2,8= 0) = P (min (Fo Fg (C)) 12, 0) P (F)(()) 22,T> C)
F. (x)
= p (c < F (x), 0)
( =
P > dG(s)
Fr (x)
--F(s))dG(
[
FF (s) dG (F (s))
=
By combining 66) and 77), and using integration by parts with respect to dG((),
B BU
1 G
(
(8) dF (102(0)
x -= dG (r))
+
F
(Fo
+
(F (s)
+
F
F
dG
0 (s)
1 1- -
o0 { F Pr (2)) + F (
1 (1))
+ [ F
F (s)
4-8 dG FFg (s)
x
[" x 8
2. Proof of E [(())2] = =x- - (1-s) r)
dG (F (s))
Ju
1 1-s s
23

---

E 20'
+ (1 (1 x x - U 2
= E
I(U - x))
1- {
8) x - (1-U) I(U sx)
= E
8 8 (1-
p (8)) (0)(
[==)) Dp (r))
+
F
(x)) + S-" F (Fz)(s) dG (F)
F
(
(z))
F 0 (z)
x
(r (s)) dG (0))
+
It implies that
E (U,))
- (F-(F=(()) + [ sdG (F) / (1- 8) (x s)2
dG (F-
(1 s)²
0
x+ 10¹ -x
(x s) dG (F- (s)
1-s
(" x dG (F)
I - (1-11x)
(8)
5 s
Lemma A.3. Under the non- non-calibration assumption such as the Regularity Conditions, we have that for some e > O and x E [0, 1].
I (U, 21 en
Proof. We use the notations from the proof of Lemma A. such Z (U,8), and denote that E 1 (U, = Eu,ax 1 (U, Using the proof in Lemma A.2
we have
Eiz" (U,8))
F F (x)
1 - F F (s)
{
(
F 0
(x))
+ F (F)
+ F
dG
(F (s)
(9)
1
Furthermore, the right term of Eqn. is
- (1))
F (s)
x x-5
+ x 8
dG
F (
1
1
1 F (x)
1 1-x x
=
+
(s)
1-8
1 r
x
dG
(s)
= / 22
F F (z)) = + F F (s) -
x
dG [ (s) (s))
{ F F (x) - x dG (s)
F (s) - - 1 1
dG (s)
=
+
24

---

By the Regularity Condition 2, we can pick x" for æ such that F (F) -x =1> 0.
B (U,6)] x²
F 1 (x)))x)) x²" - - F. (x-) 2²) J/ dG (F)
{ 0 (s)
$ 1 dG (s)
x²
= 3 3 0 (x*) I.a
{.3 1
dG (F)
=
+ %/ { F (s) $f 1-s dc (s)
x²
F (F
|// {²² dG (Fo
1o
r
(F)2(s)) - _8] 1 1-x" dG F (s))
by the triangle inequality. Note that
1
F
F
(8)
x²
dG Fr 1 (s)
0
$
F-1
(s)
-
x
dG (s)
J0
s
(s)
dG
F 1 (s)
0
F
F-
0
(x*)
17.
Therefore, we can let Finally, we have
{
F
(s)
1 x²
dG F (s))
o'n for some d' e (0, 1).
1
E|Zz (U,8)] x²1 2 17 1 - {" dG (F) - c'n = en
where c 1 10d dG F (8)) - d Furthermore, E (0, 1) by Regularity Condition 2. This
completes the proof.
L
Lemma A.4. Under the Regularity Conditions, we have
F(x) - x 2 on/2
sup xE|0.1]
goes as N- + oc.
Proof. By Eqn. 4 replacing x with the [x)(, we have that for any 0,
F(s)
B[F(s)] < <6)= +1
sup sE(o.1]
25

---

as N cc. Lemma A.3 implies [[F(x*)] - -21 2z/2 on for x- in the Regularity Conditions.
Consequently, the probability of [(x*) - x² 2 on/ goes to as N - co. suffices for the proof.
B Likelihood function
B.1
DeepSurv
The classical Cox PH model (Cox 1972). A (t 2) do exp (GTz). assumes linearity for the effect of the variables z. Here, we can adopt a DNN to estimate the effects of variables beyond linearity. The hazard function is then modeled nonlinear function z obtained through the DNN. The original Cox PH model optimizes the logarithm of the partial likelihood which given by
X (t z) Au (t) exp {g (2;0)} where g is
8
g
log
Σ JER.
exp (BT
where R; (y 2 Y1 is risk set and ,3 is the vector of regression coefficients. Maximizing this partial likelihood or minimizing the negative log-partial likelihood yields the estimated coefficients G. If we adopt DeepSurv (Ktzman et al. 2018), the negative log-partial likelihood is changed to EN 84 9 (z1; 0) log EjER, exp {g (z,; 0)} for the loss function. In DeepSurv, we only use the estimated g (z,; 0) to calculate the C-index since the baseline hazard function Ån (1) is equivalent for all subjects if the time is the same. But get Fe, we need to estimate Au (t). Rather than estimating itself, we estimate the cumulative baseline hazard function Ao (t) using the Breslow
estimator, denoted by Ås (t) (Lin 2007). Then, by using the relationship between the distribution function and the cumulative hazard function, we have
Fo (t 2) =1 exp -o (t) exp(g(z;(0)
It is known that Åg (t) is consistent for Ao (t). However, Fe includes the effect of which tells us Fg is not consistent for F, even when (t) is consistent. Therefore, need to consider the calibration. Instead, the calibration error we observed is lower than that of other models.
B.2 MTLR
MTLR Yu et al. 2011) models the probability of failure using logistic function. Let (f1, f2. .t)
be the previously determined time points, (y1, ,2,-...932 be the survival status according to (f1, 12, tBB), and (81, , be the actual survival times of the subjects. The probabil- ity of failure after the 1-th time point is modeled as (1 + exp (-0]=)) When the survival time observed over the 1-th point, the survival time is vectorized into 100,,..,0.11. 1......111 when 5 = 1,
of points
and (0, .......) when 0. For the uncensored case, we obtain the probability of observing the # of points
status (y1 (si), 9/2 (8:) :-.-. (s:))
exp EB yk (si) (0) EB k
kmm
fo (s: 2;) = P (Y -1/1 Ve..... ..,)) 2;)
exp
(0) 21
where 0 -0,, 00.,0 0T)
For the censored case, we only know that the failure would observed after the censoring time 8; that we have observed. Thus, if we denote tı the closest time point following s,, the likelihood of
26

---

censored subject is given by
Ex=l exp { 2jz=+1 (0]z.)
So (8: 2.) = Po (T; 2 zi)
ER=0 exp
B
Σ -jz=++ (0)
By combining these, the negative log- lillliiiioood for the loss function is defined as
N
e (
= Σ (s log So (st 2;) + (1 - 6:) log So (8; zz]]
Here, for Sg are the conventional notations, not the true probability density function and survival function of failure time. For more details, refer to Yu et al. 2011). For calibration error, Fe would be obtained, which is accomplished by
Fo (1 2) Ek=o exp
Eー= 0]
(10)
B Σ exp 2j=k+1 0}²]
where E [/,, (141). We use = 20 for all experiments, except for SEER-stomach and SEER-lung,
where = 10 is used.
B.3 Parametric model
We use the parametric model proposed by Goldstein et al. (2020). referred to LognormalNN, parametrize the location and scale parameters (H, σ) of the lognormal distribution. Let and S be the probability density function and survival funetion of the lognormal distribution. respectively, with parameters estimated by DNN. The negative log-likelihood
as & (0) E'e [6; log fe (y: zi) + (1 5,) log So (y: 2;)] where 0 (μ, o) and is an observed time of subject Then, we calculate the distribution function with two parameters at the given
observed time.
B.4 CRPS
Continuous Ranked Probability Score (CRPS: Avati al. 2020) aims minimize
[" ( (1) (112)) di +
{1 {1- Ro (1 (()=)] di
where y is an observed time and 5 a censoring indicator. See Appendix in Avati et al. 2020) for lognormal distribution. Since there is a closed form for the loss, we use the lognormal distribution for Fe.
B.5 DeepHit
DeepHit (Lee et al. 2018) is a discrete time survival model that estimates the probability mass function I (T =t over discrete time intervals. Let Thaxx denote the maximum time point. Given
2, the model outputs a probability vector P = (p.....,ff....) with Emmoe dr using the softmax function. The model is trained to minimize the negative log-likelihood. For an uncensored sample with event time the loss is - log Pt., and for censored sample with censoring time the loss is log Tuna P.. corresponding to the estimated survival probability S 2). We adopt DeepHit in the single-event setting and train it using only the log-likelihood loss, without the ranking loss originally proposed in Lee et al. (2018). Similar to MTLR, DeepHit depends on the number bins, and different number of bins depending the dataset.
B.6 Weibull AFT model
The Weibull AFT model
(Stute 1993) assumes the following regression form:
log T 3T + oe
27

---

where follows a standard Gumbel distribution. Under this formulation, T follows a Weibull distribution with scale parameter exp (33²) and a shape parameter 1/a. The survival function is given by:
(1 2) = exp (- (-
The likelihood function defined similarly to the parametric model.
C Proof for Proposition 5.1
In this section, we demonstrate that if survival curves do not cross, the KSP preserves the ordering of mean survival times. Furthermore, we show that this property holds for both DeepSurv and the Weibull AFT model after applying KSP.
First, recall that for individuals with variables 2; and Zj. if
F (s z;) < F
[z) for 2 0,
then the CDFs after the KSP also satisfy F (s z,) < F* (s 22) due to the monotonicity of the transformation.
For non-negative random variable it is known that its expectation can be written as
/ (1 - (s)) ds.
B
Applying this identity to both the original and the transformed CDFs, we obtain
E T zi]
F(s zi)) ds
[1 F (s 23) ds
E T" 2
> E
T
where T F*. Thus, the ordering of the mean survival times is preserved; consequently, the C-index based on the mean survival time remains unchanged after KSP.
DeepSurv Under the PH assumption, the ordering of mean survival times preserved. The model assumes that
T. < T
explg (z 0)) explg (zy: 0)}
An (s) exple (z: d) < exp [- L'" do (s) explo
(2;; 0)]ds At 2 0
exp
E T
z,] < E T zj]
E [ z.] < < |,,
where (z; 0) is the log-risk function. Hence, the C-index derived from the risk scores is preserved under KSP.
Weibull AFT The Weibull AFT model defines the survival function
(t 2) exp (- ftexp (=
And,
T3 < T;
exp (zTz;) < exp (BTz)
S 2 $ (t 2,), Vt20
E [ zi < R T 2j]
E T z E T
Therefore, the expected survival times maintain their original ordering.
28

---

Table 3: Summary of the ten real-world datasets.
Group
Dataset
Samples (Train / Validation / Test)
% Censored
# Covariates
WHAS
982 / 328 / 328
58%
METABRIC
1,142 / 381 / 381
42%
Small
GBSG
1,340 / 446 / 446
43%
NACD
1,436 / 480 / 480
36%
51
NB-SEQ
2,873 / 958 958
31 %
24
Medium
SUPPORT
5,325 1,774 / 1,774
32%
14
MIMIC-III
5,353 1,785 / 1,785
0%
15
SEER-liver
16,575 5,5255 5,526
22%
15
Large
SEER-stoman
20,615 6,872 6,872
27%
15
SEER-lung
125,254 41,751 /41,752
16%
D
Data generation and description
We first summarize the ten datasets used this study. Then, we describe each dataset in detail, along with the corresponding pre-processing steps.
D.1 Details for WHAS, METABRIC, GBSG, and SUPPORT
The DeepSurv package (Katzman et al. 2018) Python includes four pre- rre-processed real-world datasets: the Worcester Heart Attack Study (WHAS), Molecular Taxonomy of Breast Cancer Interna- tional Consortium (METABRIC) German Breast Cancer Study Group (GBSG), and the Study to Understand Prognoses Preferences Outcomes and Risks of Treatment (SUPPORT). To ensure consis- tency across experiments, we combined the training and test sets provided by the package and re-split the data into training, validation, and test sets a 3:1:1 ratio. No additional data pre-processing was performed.
D.2 Details for NACD
The Northern Alberta Cancer Dataset (NACD) (Qi al. 2024a) includes patients diagnosed with various cancers, such as lung, colorectal, head and neck, esophageal, stomach, and other types of cancer. The event of interest in this dataset is failure time. We obtain the dataset from http://sssp.sv.uullbere
under the "Public Predictors' section. It consists of 2,396 patients with 51 Teatures. For our experiments, we split the dataset into training, validation, and test sets (1,436 / 480 480), ensuring that the censoring rate remains balanced at 36%.
D.3 Details for NB-SEQ
NB-SEQ consists of neuroblastoma data and other supervised penalty learning benchmarks specif- ically designed for censored regression in the context of supervised penalty function learning for change-point detection. The dataset includes various summary statistics, as well as the minimum and maximum values of X, achieving high performance, where is a hyper-paramete for the penalty function. The primary response variable (considered failure time) the maximum value of the estimated X. It is important to note that the maximum value of À is not directly observed in some cases; we only have information that the value exceeds certain threshold, indicating that the true value is censored. We analyze 4,789 sequence data points with 24 variables. The list of the 24 variables used the NB-SEQ dataset is provided in Table 4 All the variables are continuous variables. The censoring rate is 31%. The dataset is partitioned into sizes 2,873, 958, and 958 subsets.
D.4 Details for MIMIC-III
MIMIC-III consists de-identified health-related information from patients admitted to the Beth Israel Deaconess Medical Center (BIDMC) in Boston, Massachusetts, USA, with primary focus on intensive care unit (ICU) admissions. Following the pre-procsssing steps introduced in Harutyunyan This dataset involves
al. 2019), provided by http://gtttbbcccm///evwaanWmaann

---

Table 4: Twenty four variables in NB-SEQ
quartile.25. quartile.50/ quartile.75 quartile.100) mean sd bases data log. wrg....qurtile..2. log. wer.s.l.urttteeee. log. 1.i.rtill.. / log. l.quartilee 100 log.I.mean / log.1.so 1.sd log. 1.bases log. I.data/ / log.quartile.100 100 log. mean / log.sd / log. log.data log.log. www..uuattll....... 100 / log. log. bases log.log. data
multiple instances of length-of-stay data, with each subject having repeated measures at different time points. After excluding ICU transfers and patients under 18, the training and test sets consist of 2,925,434 and 525,912 instances, respectively. Recognizing that using the provided data directly is not suitable due to the correlation among data points for single subject, we undertake additional steps. First, we exclude variables with excessively high missing rates. Next, we create one instance per subject by defining survival time as the duration between enrollment and discharge, which has no censoring. The dataset includes list of variables, as shown in Table [ For variables that are repeatedly measured, we transform them into a single representative value, such as the mean for continuous variables and the mode for categorical variables. Finally, we exclude subjects whose variables are still missing, even after applying imputation, resulting in a dataset with 8,873 instances that combines the training set with the test set. We partition the entire dataset into training, validation, and test sets, with sizes of 5,353, 1,785, and 1,785, respectively.
Table 5: Fifteen selected variables in MIMIC-III
Name
Type
Diastolic blood pressure
Continuous
Glascow coma scale eye opening
Categorical
Glascow coma scale motor response
Categorical
Glascow coma scale total Glascow coma scale verbal response Glucose
Continuous
Categorical
Continuous
Heart Rate
Continuous
Height
Continuous
Mean blood pressure
Continuous
Oxygen saturation
Continuous
Respiratory rate
Continuous
Systolic blood pressure
Continuous
Temperature
Continuous
Weight pH
Continuous
Continuous
D.5 Details for SEER-liver, SEER-stomach, and SEER-lung
The Surveillance, Epidemiology, and End Results (SEER) Program dataset (National Cancer Institute, DCCPS, Surveillance Research Program, 2015) is a large-scale, population-based cancer registry that covers approximately 49% of the U.S. population (Qi al. 2024a). It contains comprehensive information on cancer incidence, patient demographics, treatments, and survival outcomes. In this study, we utilize three distinct subsets from the SEER database SEER-liver SEER-stomach, and which respectively include patients diagnosed with liver, stomach, and lung cancers. The objective is to model the time from cancer diagnosis a failure event, specifically death, with time measured in months. For our analysis, we selected a subset of clinically relevant variables: Sex, Race, Summary stage, Malignant behavior, Record number, Total tumor size, and Age at diagnosis. Variables with high missing rates or a large number of categories were excluded, and patients without recorded follow-up (due to immediate death) were removed. Additionally, we applied filtering criteria to exclude patients with extreme or implausible values. Specifically, we retained only patients with record number less than or equal to total tumor size less than or equal to 13, and a survival time greater than or equal to 10 months. The dataset can be downloaded from https://ser..cancerrst
30

---

E Comparison of post-processing methods: Figure
In this section, we present the results across real datasets using figures. The error bars represent 95% confidence intervals.
WHAS Sore
C-index
S-cal(20)*
D-cal(20)
KS-call
KM-call
IBS
Coozz C,440 cuenn Caseo
n.15
0,000
0.500
00000
0.00
C-index
S-cal(20)-
D-cal(20)
KS-cal
KM-cal
IBS
WN05 MTUR
0.00
D21
0.30
0.30
Esta
201
0.02
DG0 ear
Cio von
100
C-index
(-c))
2-c/
KS-cal
KM- KM-cal
IBS
WHeNe Foaretie
100
202
1116
2.23
C40
A30
640
C-Index
S-cal(20))
D-cal(20)-
KS-cal
KM-cal
IBS
WH0S DRPS
225
3.5
0.16
opr
oat
POT
C-index
S-cal(20)-
D-cal(20)
KS-cal
KM-cal
IBS
WHVS D300HR
9220
0329
C.044
0010
833
0000
8210
0332
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
WHYS AFT
0.12
9.9
020
023
055
t2
caize
600
0.30
80
DP 201
4DP
610
610
Figure 5: Comparison of post-processing methods for the WHAS dataset. Higher C-index values
indicate better discrimination, while lower values are preferred for all calibration metrics. The dashed line indicates the calibration error of the KM estimator.
31

---

METIGRIC DropSun
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
ond
God1
conegt
Now aetree
00010
0.18
METNBRO. MTLR
C-Index'
S-cal(20)
(2c)
KS-cal
KM-cal
IBS
020123
E0003
24013
800115
001
113
(allany
C50
69
calam
NETNERIO Panseet
C-index
S-cal(20)
D-cal(20)+
KS-cal -cal
KM-cal
IBS
colo
2.106
2000
2820
5.170
0000
8116
ADP
Ci zon
0901
METIORIO CRPS
C-Index
S-cal(20)+
D-cal(20)-
Ks-cal
KM-cal
IBS
0300
8.00
New
NETASRIC Degg4t
C-index
S-cal(20)
D-cal(20)+
KS-cal
KM-cal
IBS
Bold
Tood
033700
23322
4004
0 2029
8046
023000
0,00
0330
020
8
90.00
650
010
cover
C-Index
S-cal
D-cal(20)-
KS-cal
KM-cal-
IBS
METNORO AFT
11
C200
0.10
0.00
220
620
Figure 6: Comparison of post- pottpprcesssing methods for the METABRIC dataset. Higher C-index values indicate better discrimination, while lower values are preferred for all calibration metries. The dashed line indicates the calibration error of the KM estimator.
32

---

C-Index
S-cal(20)
D-c)
KS-cal
KM-cal
IBS
0830 Doygury
0.30
G661
0030
0.24
0,29
0.000
cop
Croeett
Com ap
Now
conee
C-Index
S-cal(20)
2-c)
KS-cal
KM-cal
IBS
0000 MTER
0.72
0.23
0.04
C.000
8.1
0880 Pooneero
C-index
S-cal(20)
(-c)
KS-cal
KM-cal
IBS
0.000
0.200
Cote
303
0.37
0.3
302
0.35
0332
0220
190
Cip
C-Index
(-cal(20)
D-cal(20)+
KS-cali
KM-cal
IBS
0080 CRPS
1E3
b.77
0.2
E.00
He
C-index
S-cal(20)
D-cal(20)
KS-cal+
KM-cal
IBS
0003 D00014
0/96
021
0.000n
2210
C.0000
6
020
620
620
C-Index
S-cal
D-cal(20)-
KS-cal
KM-cal-
IBS
0850 AFT
0.004
3.13
6:0
0.000
E
6:9
220
U19
650
620
Figure 7: Comparison of post- post-processing methods for the GBSG dataset. Higher C-index values indicate better discrimination, while lower values are preferred for all calibration metrics. The dashed line indicates the calibration error of the KM estimator.
33

---

C-Index'
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
NMGD 0vee5
nuep
code
pama
nap
0.00
wou
0:6
620
Cobeot
conaut
Canagg
Coheat
C-Index
S-cal(20)
D-cal(20)-
KS-cal
KM-cal
IBS
4000 MTLS
0.33
C3700
00070
00200
0,7000
Hen www.ee
(40
Co
(30.00)
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
NAC0 Pagnett
2.00
Cume
boon
61
16
690
cop
C-Index
S-cal(20) !
D-cal(20) 4
KS-cal
KM-cal
IBS
NACD CPPS
2250
622
im
2.04
2200
2223
C-index
S-cal(20)+
D-cal(20)
KS-cal
KM-cal
IBS
NADD DoogHe
204
0.01
0,7000
003
022
0.5
0001
0.02
ADH
CIPPOOT
850
C-index
S-cal(20)
D-c)
KS-cal-
KM-cal-
IBS
0.36
0.7085
0.24
E/700
0.22
6307
0.20
A50
620
C20
Figure 8: Comparison of post- post-processing methods for the NACD dataset. Higher C-index values indicate better discrimination, while lower values are preferred for all calibration metrics. The dashed line indicates the calibration error of the KM estimator.
34

---

C-Index
(-C)
D-c)
KS-cal
KM-cal
IBS
NB 580 D008%
0.445
0.00055
caso
LAM
0.300
0.100
co coeen
Conarot
convor
C-Index
S-cal(20)-
D-cal(20)
KS-cal
KM-cal
IBS
C.400
Bod4
0.24
0.204
6.00
6.772
Aw
0821
0023
0.34
0.33
0.391
130
Cal
CLD
C50
calbelee
C50
C50
NB 3E0 F3Cawott
C-index
2-c)
D-c)
KS-cal
KM-cal
IBS
00z0
enit
0.33
0o0
6010
0/10
0.11
0.22
0000
0,30
050
6:0
C-Index
S-cal(20)-
D-cal(20)-
KS-cal
KM-cal
IBS
020
2 R C
004
9.0
60t
Callaro
NB 550 DV0044
C-index
S-c)
D-cal(20)
KS-cal
KM-cal
IBS
4od4
0,22
0.30
000
204
Duse
Exec
050
D60
Dio
C-Index
S-cal(20)-
D-cal(20)
KS-cal
KM-cal
IBS
21
0.00
LAA
0/00
310
210
620
Figure 9: Comparison of post-pree proccessossingg methods for the NB-SEQ dataset. Higher C-index values indicate better discrimination, while lower values are preferred for all calibration metrics. The dashed line indicates the calibration error of the KM estimator.
35

---

SUPPORT Degpour
C-Index
(-0)
(-c)
KS-cal
KM-call
IBS
Lone
cuce
do14
EX012
GoTD
lode
C-Index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
BUPRORT METR
0202
60151
0000
0 3330
8218
130
cmam
SJPP Rurooons
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
c.00e
3215
0,00
021
0.0
0,000
BoXt
Bap
0,00
cip
co
C-Index
S-cal(20)
D-cal(20)-
Ks-cal-
KM-cal
IBS
BUPPORT CSPS
L1D
Hew
Nen
New
SUPPORT Docg4)
C-index
S-cal(20)
D-c)
KS-cal
KM-cal
IBS
n nnne
cote
DUCG
C001
2000
0.200
0.90
690
050
coven
C-Index
S-cal(20)-
D-cal(20)
KS-cal
KM-cal-
IBS
SIPPORTAST
3:90
Carp
3370
3.50
024
3329
3330
420
C20
Figure 10: Comparison of post-procee cost-processig methods for the SUPPORT dataset. Higher C-index values indicate better discrimination, while lower values are preferred for all calibration metrics. The dashed line indicates the calibration error of the KM estimator.
36

---

C-Index
S-cal(20)
(20)
KS-cal
KM-cal
IBS
MANIC II Dhgg3.
Bic
1000
B01
Coneot
Goneet
C-Index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
MINO ut MTLS
0019
Ge
300
0000
301
0.00
C50
C50
NIMD i1 Papenoro
C-index
S-cal(20)
D-cal(20)+
KM-cal
IBS
0,0500
204
840
a28
Lieens
8d1
200
330
06000
06210
6.1
000
cr
C10
C-Index
S-cal(20)-
D-c/
KS-cal
KM-cal
IBS
MANIC C3P8
2003
0.004
5,000
MINC II Doogit
C-index
S S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
2.00
6/00
000122
cabrie
610
050.00T
850
620
050
020
C-Index
S-cal
D-cal(20)-
KS-cal
KM-cal-
IBS
AUMIO M AFT
2015
1010
D01
005
D00
(1))
2109
620
Figure 11: Comparison of post-processing methods for the MIMIC-II dataset. Higher C-index values indicate better discrimination, while lower values are preferred for all calibration metrics. The dashed line indicates the calibration error of the KM estimator.
37

---

SEER We D0rp3.r
C-Index
(-0)
D-cal(20)
KS-cal
KM-cal
IBS
cncess
C.08
co0012
de
nnrent
0.22
2014
0.94
0.0
0.11
Covept
eent
Comerro
copeert
C-Index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
SEER 840 MTLS
16-0t
ROC0
6:122
1.001
E33
D040
C.00
2o06
b040
cup
Hon
SEER bet Psoronto
C-index
S-cal(20)
D D-c)
KS-cal
KM-cal
C.04
00000
0.2205
0.0220
0.556
0.00
6a0
0.34
0.2000
0.33
0.000
8000
det
C-Index
(-c)
D-cal(20)
KS-cal
KM-cal-
IBS
SEER Bee CRPS
6.10
c.00e
calrodee
How
Han
Ham
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
0.000
0834
833
D045
0492
E200
b040
C.02
5066
850
ADP
C-Index
S-cal(20)-
D-cal(20)-
KS-cal
KM-cal
IBS
6630
R.1
000 15
0,12
6.11C
E4530
0,302
914
0030
Figure 12: Comparison of post-processing methods for the SEER-live dataset. Higher C-index values indicate better discrimination, while lower values are preferred for all calibration metries. The dashed line indicates the calibration error of the KM estimator.
38

---

BEER congch Diecuuy
C-index
S-cal(20)
D-c)
Ks-cal
KM-cal
IBS
dock
2005
tods
2004
0.10
2003
0.15
2000
0.16
conegi
sel
conegr
SEER nomach MTLA
C-Index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
0250
0.18
Cose
Sod4
014
0144
2o54
600400
C50
BEER dbeee Pasmork
C-index
S-c)
D-cal(20)
KS-cal
KM-cal
0.0075
Go0D
Cose
0.000o
0,000
200
K2P
eat
050
SEER sonne DRP6
C-Index
2-c
D-cal(20)
KS-call
KM-cal
IBS
Core
0.10
Eara
0.39
He
Hew
53E4 donean Diopht
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
0150
0,03
0149
0.77
delt
1002
0.99
1r81
050
Po
D50
600 D0000 AFT
C-Index
S-cal(20)-
D-0)
KS-cal
KM-cal-
IBS
L.000
1000
L.000
1004
0.00
Figure 13: Comparison of post-processing methods for the values indicate better discrimination, while lower values are preferred for all calibration metries. The dashed line indicates the calibration error of the KM estimator.
dataset. Higher C-index
39

---

SEER kig Deo/Strr
C-Index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
BOKKS
onss
Coend
09006
00000
00006
un22
2.0
2044
0:115
8385 u0g MTLR.
C-Index'
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
Diee
2232
Dree
0000
46
123
C.004
desco
Ar1
C30
New ORltaAN
C50
Hon
SEER kin. Fygmatt
C-index
S-ca
D-cal(20)
KS-cal
KM-call
Bol4
029
Soote
Gusto m
050
6EER CSPS
C-Index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
BEER Lng Dooges
C-index
S-cal(20)-
D-cal(20)
KS-cal
KM-cal
IBS
Gu0ce
60-34
E.00
E206
9oC4
0150
Me-3
A150
Loone Calno
115
Destd
clpoor
PID
ADK
D50
C-index
S-ca)
D-c
KS-cal
KM-cal
SEER ung 6FT
0.006
3310
Core
0/15
0303
0/12
620
Figure 14: Comparison of post-processing methods for the SEER-lung dataset. Higher C-index values indicate better discrimination, while lower values are preferred for all calibration metries. The dashed line indicates the calibration error of the KM estimator.
40

---

Comparison of post-processing methods: Table
In this section, we present the results tabular form, in contrast to Appendix E where the same results are visualized as figures. Similar to Table | in the main text, we summarize the num- ber of cases in which each post-processing method outperforms the others for each model. We implemented X-cal (Goldstein et al., 2020 in the main paper), CSD (Qi et al. (024a). and CSD- iPOT (Qi et al. 2024b) with reference to publicly available code repositories under the MIT Li- cense: https: / /gggtbtt..b..... www. 1ab/X- https: http:://ttttsttt//ti// //github. cwm/cooo/oaaooooo fom/shi-- httpsshhannnn///e CSD, and
httpss:/Dg www............... com www.s//saagg//aaaaaaaaaaaaaaaaaa haaaaaaaaaaaaaaaeaarraaaaaaaaaea
Table 6: Summary of pairwise comparisons between post-processing methods for DeepSurv. The table shows the number of cases where KSP outperforms its counterpart, outperformed, yields tie. Numbers in parentheses indicate statistically significant differences based on a one-sided t-test at
the 0.05 level.
Model
Method
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-ccal
IBS
KSP
(0)
6 (5)
(5)
(6)
(1)
(1)
Non-calibrated
1 (0)
3 (1)
4 (0)
3 (1)
6 (5)
4 (0)
Ties
9
KSP
(0)
(9)
10 (10)
10 (10)
10 (9)
(5)
DeepSurv
CSD
6 (0)
1 (0)
(0)
(
(
(
Ties
4
KSP
1 (0)
(4)
10 (8)
8
7
(1)
CSD-iPOT
(0)
4 (0)
(
(0)
(1)
(0)
Ties 4
0
0
0 0
Table Summary of pairwise comparisons between post-pro post-processing methods for MTLR. The table
shows the number of cases where KSP outperforms its counterpart, is outperformed, or yields tic. Numbers in parentheses indicate statistically significant differences based on a one-sided t-test at the 0.05 level.
Model
Method
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
KSP
(0)
(6)
7 (5)
(5)
9 (6)
7 (2)
Non-calibrated
6 (1)
4 (3)
3 (2)
4 (2)
(1)
3 (0)
Ties
1
0
0
0
0
0
KSP
(0)
(1)
6 (6)
(4)
33)
(1)
MTLR
CSD
(1)
(6)
(3)
(3)
(6)
7 (4)
Ties
0
0
0
KSP
(0)
5 (2)
(8)
6 (4)
(6)
6 (0)
CSD-iPOT
6 (0)
5 (5)
(2)
4 (2)
(2)
(2)
Ties
0
0
41

---

Table 8: Summary of pairwise comparisons between post-processing methods for Parametric model.
The table shows the number of cases where KSP outperforms its counterpart, is outperformed. or yields tie. Numbers in parentheses indicate statistically significant differences based on one-sided t-test at the 0.05 level.
Model
Method
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-call
IBS
KSP
8 (7)
10 (10)
10 (10)
10 (10)
(6)
(5)
Non-calibrated
(0)
(0)
0
0 (0)
(1)
2 (0)
Ties
0
0
0
0
0
KSP
(0)
(7)
10 (10)
10 (9)
(6)
10 (4)
Parametric
CSD
5 (0)
(1)
(0)
(
1 (1)
(
Ties
0
0
KSP
(0)
(2)
(7)
9
(5)
(2)
CSD-iPOT
(0)
5 (3)
(1)
((1)
(2)
2
Ties
0
0
0
Table Summary pairwise comparisons between post-processing methods for CRPS. The table
shows the number of cases where KSP outperforms its counterpart, is outperformed, or yields tie.
Numbers in parentheses indicate statistically significant differences based on a one-sided t-test at the
0.05 level.
Model
Method
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
KSP
7 (5)
10 (10)
10 (10)
10 (10)
10 (10)
10 (10)
Non-calibrated
2 (0)
0 (0)
(
0 (
(0)
0
Ties
0
0
KSP
(2)
8 (8)
(5)
8 (8)
10 (10)
10 (10)
CRPS
CSD
(0)
(2)
4 (4)
(2)
(
(
Ties
0
0
KSP
(0)
7 (6)
53)
7 (5)
6 (6)
5 (4)
CSD-iPOT
3 (0)
(3)
5 (4)
3 (3)
4 (3)
(4)
Ties
0
0
Table 10: Summary of pairwise comparisons between post-prost post-processing methods for DeepHit. The table shows the number of cases where KSP outperforms its counterpart, is outperformed, or yields
tie. Numbers in parentheses indicate statistically significant differences based on a one-sided t-test at
the 0.05 level.
Model
Method
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
KSP
2 (0)
4 (4)
3 (3)
4 (4)
7 (4)
6 (0)
Non-calibrated Ties
8 (0)
6 (3)
7 (4)
6 (2)
3 (2)
(
0
0
KSP
1 (0)
4 (3)
8 (6)
8 (6)
3 (2)
4 (0)
DiepHlit
CSD
9 (1)
6 (5)
2 (1)
(1)
(6)
6 (4)
Ties
0
0
0
0
0
KSP
5 (0)
3 (2)
8 (7)
(3)
8 (5)
6 60)
CSD-iPOT
(1)
7 (4)
2 (2)
32)
2 (2)
4
Ties
0
0
42

---

Table 11: Summary of pairwise comparisons between post-processing methods for Weibull AFT model. The table shows the number of cases where KSP outperforms its counterpart, is outperformed. or yields a tie. Numbers in parentheses indicate statistically significant differences based on
one-sided t-test the 0.05 level.
Model
Method
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
KSP 0 (0)
10 (10)
10 (10)
10 (10)
9 (8)
(7)
Non-calibrated
(0)
(0)
0
(
1 (1)
(0)
Ties
10
0
0
0
0
0
KSP
0 (0)
3 (1)
(8)
(5)
(2)
5
AFT
CSD
(0)
7 (5)
(2)
(2)
8 (6)
5 (2)
Ties
6
0
0
0
0
KSP
(0)
(5)
6 (6)
(
10 (9)
(2)
CSD-iPOT
4 (0)
4 (4)
(4)
(3)
(
(2)
Ties
6
0
0
0
0
Table 12: Comparison of post-processing methods for the WHAS dataset. Higher C-index values
indicate better discrimination, while lower values are preferred for all other calibration metrics.
Model
Method
C-index
S-cal(20)+
D-cal(20)+
KS-cal
KM-call
IBS-
Non-calibrated
0.83934
0,000184
0,001324
0.029978
0.000664
0.111616
KSP
0.83934
0,000189
0.001483
0.031463
0.000526
0.11
DeepSurv
CSD
0.83934
0.016320
0.010881
0.191019
0.010207
0.129705
CSD-iPOT
0.83934
0.000163
0.001513
0.031388
0.000563
0.112224
Non-calibrated
0.81940
0.004684
0.009308
0.131427
0.008072
0.140025
KSP
0.81956
0.000766
0.004767
0.055003
0.007428
O. D..40783
MTLR
CSD
0.81935
0.000540
0.002696
0.045507
0.001938
0.130369
CSD-iPOT
0.81955
0.001610
0.003647
0.069353
0,006033
O. 0.136362
Non-calibrated
0.80364
0.000858
0.002488
0.055345
0.003171
0.116109
KSP
0,84049
0.000379
0.001994
0.039706
0.003100
0.146
Parametric
CSD
0,83970
0,031220
0.019276
0,258418
0.028841
0. 15884242
CSD-iPOT
0.84078
0.000439
0.002074
0.042612
0.002768
O. 0.146
Non-calibrated
0.79922
0.052393
0.146602
0.377683
O. 1058559
0.289911
KSP
0.78974
0.001444
0.018794
0.081137
0.003376
0.144378
CRPS
CSD
0,80142
0.036708
0.030670
0,280881
0.043592
0.191666
CSD-iPOT
0.79730
0.007258
0.017252
0.142694
0.016375
O. 17767
Non-calibrated
0.84397
0.001856
0.004547
0.163803
0.003720
O. 109808
KSP
0.84328
0.000604
0.003602
0.051873
0.004851
0.1 13300
DeepHit
CSD
0.84364
0.000963
0.003509
0.057144
0.002235
1.330
CSD-iPOT
0.84319
0.000875
0.002621
0.052076
0.003261
0.109295
Non-calibrated 0.81692
0.159936
0.115956
0.672936
0.218693
0.384233
KSP
0.81692
0.000211
0.001899
0.034510
0.012560
O. D4242276
AFT
CSD
0.81693
0.000800
0.003983
0.057077
0.002907
0.177157
CSD-iPOT
0.81693
0.000744
0,002625
0.051983
0.012709
0.140417
43

---

Table 13: Comparison of post-pnn post-processing methods for the METABRIC dataset. Higher C-index
values indicate better discrimination, while lower values are preferred other calibration metrics.
Model
Method
C-index
S-cal(20)*
D-cal(20)+
KS-cal
KMMcall
IBS
Non-calibrated
0.64572
0.000074
0.001168
0.024570
0.000129
0.162487
KSP
0.64572
0.000138
0.001300
0.028909
0.000192
0.162597
DeepSurv
CSD
0.64585
0.000554
0.002511
0.045971
0.000882
0.164027
CSD-iPOT
0.64573
0.000132
0,001810
0.030531
0.000244
0.162564
Non-calibrated
0.62866
0.000991
0.002344
0.050756
0.002398
0.164533
KSP
0.62912
0.000517
0.001843
0.046266
0.001261
0.163358
MTLR
CSD
0.62873
0.000520
0.002336
0.046683
0.001120
0.163325
CSD-iPOT
0.62880
0.000532
0.002220
0.046963
0,001767
0.163472
Non-calibrated
0.60856
0.012025
0.007692
0.169293
0.011532
O. 17849
KSP
0.62147
0.000501
0.001772
0.044165
0,003312
O. 0.169834
Parametric
CSD
0.62146
0.001267
0.003573
0.063101
0.003366
0.1 70954
CSD-iPOT
0.61973
0.000617
0.002269
0.049259
0.002470
0.169168
Non-calibrated
0,59211
0.085305
0.186298
0.457565
0,208641
0.387240
KSP
0.60051
0.000786
0.003384
0.057540
0.004217
0.175699
CRPS
CSD
0.59756
0.008325
0.009216
0.131548
0.019657
0.222708
CSD-iPOT
0.59810
0.002275
0.004011
0.085995
0.013559
O. 0.185530
Non-calibrated
0.63846
0.000539
0.001943
0.044299
0.001352
O. 162155
KSP
0.63823
0.000541
0.001957
0.048270
0.001 197
0.162350
DeepHit
CSD
0.63793
0.000566
0.002485
0.049948
0.001040
0.162360
CSD-iPOT
0.63746
0.000518
0.002548
0.049504
0,001268
O. 0.162362
Non-calibrated
0.62854
0.170157
0.157761
0.699554
0.150305
0.323121
KSP
0.62854
0.000696
0.003680
0.055747
0.019585
O. 0.195871
AFT
CSD
0,62854
0.000492
0.002619
0.047241
0.001815
0.176144
CSD-iPOT
0.62854
0.001917
0.004316
0.085610
0,033533
0.203033
Table 14: Comparison of post- pest-prcesssig methods for the GBSG dataset. Higher C-index values
indicate better discrimination, while lower values are preferred for all other calibration metrics.
Model
Method
C-index
S-cal(20)+
D-cal(20)+
KS-cal
KM-cal
IBS
Non-calibrated
0.67055
0,000153
0.001249
0.031010
0.000037
0.177364
KSP
0,67055
0.000131
0,001108
0.027683
0.000246
0.177451
DeepSurv
CSD
0.67055
0.001139
0.002872
0.064236
0.001849
0.180139
CSD-iPOT
0.67055
0.000137
0,001822
0.029647
0.000152
0.177440
Non Non-calibrated
0.66518
0.015595
0.047053
0.305066
0.027289
0.212137
KSP
0.66422
0.008840
0.046362
0.161603
0.015227
0.206787
MTLR
CSD
0.6651
0.000277
0.002028
0.038923
0.000476
0.184225
CSD-iPOT
0.66508
0.002382
0.008590
0.112182
0.002627
0.186904
Non-calibrated
0.64735
0.004308
0.003882
0.097194
0.006469
0.189363
KSP
0.65704
0.000404
0.001651
0.040159
0.000619
0.182844
Parametric
CSD
0.65748
0.000673
0.002655
0.046418
0.001565
0. 184407
CSD-iPOT
0.65686
0.000528
0.002326
0.042402
0.000863
0.183205
Non-calibrated
0,59357
0.060048
O. 1.129900
0.389503
O. 00103373
0.299431
KSP
0.60163
0.000383
0,001487
0,041330
0.000531
0.194706
CRPS
CSD
0.60237
0.004895
0.006940
0. 0.109091
0.007319
0.227361
CSD-iPOT
0.60250
0.000579
0.002102
0.046562
0.001
0.195382
Non-calibrated
0.66395
0,013217
0,042233
0.286991
0.022726
0.207548
KSP
0,66380
0.009500
0.047356
0.167647
0.016446
0.208659
Diep
CSD
0.66390
0.000286
0.002090
0.038490
0.000545
0.184770
CSD-iPOT
0.66401
0.002268
0.008670
0.110511
0.002322
O. 1.188600
Non-calibrated
0.66068
1.262242
0.139443
0.694223
0.178947
0.372955
KSP
0.66068
0.000388
0.001406
0.039076
0.001558
0.185855
AFT
CSD
0.66068
0.000316
0.001880
0.039659
0.000557
0,201437
CSD-iPOT
0.66068
0.000733
0.002273
0.051802
0,005238
O. 18684443
44

---

Table 15: Comparison of post-processing methods for the NACD dataset. Higher C-index values
indicate better discrimination, while lower values are preferred for all other calibration metrics.
Model
Method
C-index
S-cal(20)*
D-cal(20)+
KS-cal
KM-cal
IBS
Non Non-calibrated
0.74482
0.002403
0.003013
0.069770
0.002771
0. 0.148203
KSP
0.74482
0.003181
0.003339
0.071627
0,002512
0.148069
DeepSurv
CSD
0.74483
0.006150
0.006106
0.109203
0.002737
0.150490
CSD-iPOT
0.74481
0.002585
0.003708
0.069948
0.002100
0.148205
Non-calibrated
0.75232
0.002727
0.004358
0.078424
0,008610
0.158760
KSP
0.75232
0.000394
0.002154
0.040311
0.003399
0.151300
MTLR
CSD
0.75155
0.000434
0.002150
0.043359
0.002094
0.150096
CSD-iPOT
0.75168
0.000470
0.002770
0.043038
0,004965
0.153054
Non-calibrated
0.73739
0.002980
0.007046
0.091661
0.000875
0.151496
KSP
0.73811
0.000413
0.001920
0.040499
0.002716
0.149560
Parametric
CSD
0.73877
0.001803
0.003523
0.070777
0.000750
O. 1.98155
CSD-iPOT
0.73868
0.000715
0.003684
0.056430
0.000980
O. 149651
Non-calibrated
0.71759
0.043051
0.063536
0.301582
0.089002
0.250414
KSP
0.73131
0.000485
0,001559
0,041403
0.006499
0.156702
CRPS
CSD
0.72759
0.009327
0.009646
0.142301
0.025258
O. 0.187515
CSD-iPOT
0.72959
0.001852
0.003063
0.071834
0.013876
0.164684
Non-calibrated
0.73923
0.001121
0.003564
0.059614
0.003978
0.156312
KSP
0.73889
0.000408
0.001647
0.039227
0.003167
0.152809
DeepHit
CSD
0.73909
0.000612
0.002525
0.051245
0.002205
0.150479
CSD-iPOT
0,73895
0.000484
0.002530
0.043515
0.003355
0.154051
Non-calibrated
0.75251
0.163167
0. 0.126591
0.658877
0.080887
0.248429
KSP
0.75251
0.000341
0.001746
0.038208
0.006366
0.149097
AFT
CSD
0.75251
0.000373
0.002111
0.040763
0.005790
0.173913
CSD-iPOT
0.75251
0.000640
0.002636
0.048710
0.009616
0.151260
Table 16: Comparison of post-processing methods for the NB-SEQ dataset. Higher C-index values
indicate better discrimination, while lower values are preferred for all other calibration metrics.
Model
Method
C-index
S-cal(20)+
D-cal(20)4
KS-cal
KM-cal
IBS
Non-calibrated
0.74406
0.000353
0.001233
0.036594
0.000117
0.048694
KSP
0.74406
0.000124
0.001029
0.024159
0.000114
0.048526
DeepSurv
CSD
0.74406
0.000817
0.004176
0.058940
0.000592
0.049299
CSD-iPOT
0.74406
0.000244
0.004492
0.053483
0.000413
0.048943
Non Non-calibrated
0,67586
0.000151
0.000985
0.026283
0.000248
0.056395
KSP
0.67598
0.000201
0.000969
0.028721
0.000229
0.056408
MTLR
CSD
0.67564
0.000592
0.005005
0.058357
0.000697
0.057094
CSD-iPOT
0.67570
0.000283
0.005322
0.055453
0.000551
0.056798
Non-calibrated
0.71868
0.020123
0.054895
0.217614
0.039669
0.088938
KSP
0.71868
0.000194
0.000847
0.028462
0.011985
0.057502
Parametric
CSD
0.71868
0.010833
0.008191 192
0.155574
0,012509
0.066510
CSD-iPOT
0.71868
0.001833
0,005667
0.082260
0.018628
0.059964
Non-calibrated
0.74602
0.049548
0.082799
0.399175
0.075753
O. 0.130153
KSP
0.74602
0.000289
0,001012
0.033242
0.001293
0.049608
CRPS
CSD
0.74602
0.002066
0.004542
0.079292
0.013118
0.071306
CSD-iPOT
0.74602
0.005656
0.009961
0.140358
0.013339
0.059066
Non-calibrated
0.74238
0.000169
0,000836
0.027303
0.000137
0.047711
KSP
0.74243
0.000196
0.000896
0.028329
0.000125
0.047706
Diep
CSD
0.74267
0.000534
0.004087
0.061122
0.000552
0.048342
CSD-iPOT
0.74273
0.000275
0.004866
0.057137
0.000414
0.048151
Non-calibrated
0.45964
0.195759
0.279953
0.750892
0.177522
0.249208
KSP
0.45964
0.001428
0.005948
0.068017
0.040891
0. 0.134525
AFT
CSD
0.45964
0.001277
0.006713
0.069315
0.001830
0.071353
CSD-iPOT
0.45964
0.001775
0.005227
0.080487
0.048066
O. D.128660
45

---

Table 17: Comparison of post-processing methods for the SUPPORT dataset. Higher C-index values
indicate better discrimination, while lower values are preferred for all other calibration metrics.
Model
Method
C-index
S-cal(20)*
D-c)l(20)+
KS-cal
KM-cal
IBS
Non Non-calibrated
0.61233
0.000055
0.000474
0.019310
0.000017
0.191039
KSP
0.61233
0.000057
0.000475
0.018416
0.000057
0.191034
DeepSurv
CSD
0.61233
0.000336
0.001659
0.036604
0.000435
0.191591
CSD-iPOT
0.61233
0.000057
0.001501
0.027010
0.000129
0.191146
Non-calibrated
0.57371
0.000834
0.003760
0.061424
0.012056
0.220228
KSP
0,57312
0.000490
0.002672
0.040363
0.007750
0.215751
MTLR
CSD
0,57336
0.000163
0.002190
0.031640
0.000372
0.208278
CSD-iPOT
0.57380
0.000389
0.003534
0.039351
0,007746
0.215638
Non-calibrated
0.59260
0.002252
0.003988
0.084550
0.005979
O. 0.198560
KSP
0.60439
0.000243
0.001349
0.034436
0.000275
0.193467
Parametric
CSD
0.60485
0.000440
0.001883
0.037064
0.004318
O. 0.197069
CSD-iPOT
0.60386
0.000182
0.001047
0.025885
0,001020
O. 0.193504
Non-calibrated
0,55840
0.141252
0,434019
0.658101
0,401180
0.615518
KSP
0.57109
0,015244
0,209804
0.248033
0.011795
0.224882
CRPS
CSD
0.57160
0.007093
0.017108
0.127136
0.023034
0.256595
CSD-iPOT
0.57113
0.025275
0.101719
0.380558
0.003503
0.216977
Non-calibrated
0.60214
0.000890
0.003056
0.057871
0.010317
0.206596
KSP
0.60133
0.000356
0.001975
0.035817
0.007007
0.203254
DeepHit
CSD
0.60162
0.000192
0.002228
0.033087
0.001072
0.196510
CSD-iPOT
0.60138
0.000334
0.003204
0.035553
0.007413
0,202907
Non-calibrated
0.56930
0.150578
0.101386
0.641556
0.056436
0.268950
KSP
0.56930
0.000226
0.001612
0.032202
0.000245
0.207191
AFT
CSD 0,56930
0.000175
0.003102
0.041830
0.000897
0.214866
CSD-iPOT
0.56930
0,000210
0.000876
0.029300
0.001 178
0.208210
Table 18: Comparison of post-processing methods for the MIMIC-II dataset. Higher C-index values
indicate better discrimination, while lower values are preferred for all other calibration metrics.
Model
Method
C-index
S-cal(20)+
D-cal(20)+
KS-cal
KM-cal
IBS
Non-calibrated
0.95839
0,036649
0.064547
0.250907
0. 0.104279
0.276509
KSP
0.95826
0,003305
0.007044
0.091109
0.071617
0.203039
DecpSurv
CSD
0.95842
0.002873
0.022981
0.113186
0.082850
0.222409
CSD-iPOT
0.95842
0,003150
0.016172
0.099244
0.069556
0.200904
Non Non-calibrated
0.94494
0.015402
0.035474
0.232722
0.000410
0.014391
KSP
0.94218
0.003581
0.004200
0.084666
0.000405
0.012505
MTLR
CSD
0.94423
0.004565
0.007386
0.091212
0.000573
0.013220
CSD-iPOT
0,94230
0.003792
0.005829
0.084427
0.000439
0.012597
Non-calibrated
0,96204
0.042814
0.113358
0.339424
0.000130
0.012483
KSP
0,96204
0.019627
0.020349
0.170588
0.000163
0.006789
Parametric
CSD
0.96204
0.020498
0.031328
0,243077
0.000174
0.010146
CSD-iPOT
0,96204
0.019147
0.029920
0.173745
0.000178
0.006854
Non-calibrated
0,95342
0.214466
0,258780
0.759872
0.020505
0.072193
KSP
0,95339
0,013238
0.012086
0.149600
0.000378
0.012503
CRPS
CSD
0.95398
0.001911
0.013170
0.111758
0.009601
0.052380
CSD-iPOT
0.95339
0.013594
0.017937
0.153465
0.000357
0.012327
Non-calibrated
0.96406
0,013780
0.012673
0.157457
0.000064
0,005148
KSP
0.96425
0.021969
0.024353
0.174115
0.000084
0.005210
Diep
CSD
0.96437
0.021553
0.038450
0.181557
0.000085
0.005288
CSD-iPOT
0.96423
0.021580
0.037331
0.179908
0.000091
0.005276
Non-calibrated
0.88037
0.224182
0.416392
0.821818
0.019950
0.057193
KSP
0.88037
0.005902
0.019573
0.125796
0.000679
0.021909
AFT
CSD
0.88037
0.000424
0.009171
0.050827
0.002049
0.035906
CSD-iPOT
0.88037
0.000573
0.003317
0.046279
0.000941
0.020705
46

---

Table 19: Comparison of post-processing methods for the SEER- liver dataset. Higher C-index values
indicate better discrimination, while lower values are preferred for all other calibration metrics.
Model
Method
C-index
S-cal(20)+
D-cal(20)+
KS-cal
KM-cal
IBS
Non Non-calibrated
0.63374
0,000070
0,000569
0.022519
0.000013
0.137621
KSP
0.633744
0.000030
0.000178
0.012719
0.000051
0.137654
DeepSurv
CSD
0.63406
0.000201
0.0055399
0.053343
0.000521
0.138246
CSD-iPOT
0.63406
0.000131
0.004846
0.049048
0.000371
0.138145
Non-calibrated
0,63083
0.000188
0.000557
0.036149
0.002314
0.142314
KSP
0.63077
0.000373
0.000820
0.039137
0.002094
O. 0.142062
MTLR
CSD
0.63087
0.000169
0.005012
0.052282
0.000762
0.140381
CSD-iPOT
0.63088
0.000156
0.004599
0.046668
0,002966
O. 1.2442
Non-calibrated
0.62927
0.001078
0.005251
0.056526
0.001723
O. 0.137791
KSP
0.63209
0.000167
0.000810
0.023572
0.000895
0.137527
Parametric
CSD
0.63227
0.000144
0.003663
0.039992
0.001917
O. 0.8899
CSD-iPOT
0.63208
0.000143
0.003052
0.034825
0,001836
O. 1.822
Non-calibrated
0.60369
0.165436
0,455876
0.659311
0,277786
0 0,434392
KSP
0.60940
0,001002
0,008353
0.056664
0.000493
O. 0.142757
CRPS
CSD
0.60585
0.001337
0.003582
0.063189
0.014840
O. 0.116125
CSD-iPOT
0.60835
0.000326
0.002620
0.037597
0.001070
0.145338
Non-calibrated
0.63377
0.000210
0.000613
0.036041
0.002363
O. 142056
KSP
0.63366
0.000438
0.000832
0.039767
0.002216
O. 0.141142
DeepHit
CSD
0.63383
0.000173
0.004892
0.053137
0.000805
0.140133
CSD-iPOT
0.63364
0.000167
0.004800
0.047828
0.002833
O. 0.141158
Non-calibrated
0.62661
0.002288
0.008219
0.081952
0.003858
O. 0.141141
KSP
0.62661
0.000460
0.002309
0.036500
0.003495
O. D4142036
AFT
CSD
0.62691
0.000119
0.003200
0.039183
0.002418
0.140872
CSD-iPOT 0.62691
0,000120
0.001753
0.026187
0,003806
O. 0.141118
Table 20: Comparison of post- post-processing methods for the BEER-stomach dataset. Higher C-index values indicate better discrimination, while lower values are preferred for all other calibration metries.
Model
Method
C-index
S-cal(20)+
D-cal(20)+
KS-cal
KM-cal
IBS+
Non-calibrated
0.701
0,000180
0.000908
0.027420
0.000017
03188
KSP
0.701 16
0.000020
0,000190
0.010721
0.000096
0.137950
DeepSurv
CSD
0.70134
0.000790
0.005599
0.052845
0.000740
0.139080
CSD-iPOT
0.70134
0.000120
0.004336
0.044332
0.000230
0.138501
Non- Non-calibrated
0.69538
0.000247
0.000548
0.036700
0.002508
0.147517
KSP
0.69529
0.000480
0.000860
0.040512
0.002250
0. 0..47458
MTLR
CSD
0.69543
0.000195
0.004382
0.050593
0.000739
0.145290
CSD-iPOT
0.69532
0.000161
0.004533
0.048057
0.002610
0.147009
Non-calibrated
0.69706
0.000673
0.003591
0.046981
0.001048
0.137726
KSP
0.70141
0.000161
0.000725
0.023063
0.001019
0.137680
Parametric
CSD
0.70097
0.000867
0,003156
0,046788
0.001731
0. 0.136664
CSD-iPOT
0.70094
0.000110
0,001523
0.031439
0.001431
0.138318
Non-calibrated
0.67345
0.146871
0,398096
0.615755
0.271412
O. 0.440772
KSP
0.67678
0,002785
0.018437
0,087667
0.003845
0.161579
CRPS
CSD
0.67524
0.005308
0.007145 148
0. 0.109374
0.016138
0.146344
CSD-iPOT
0.67611
0.000700
0.004436
0.063063
0.001403
0.159007
Non-calibrated
0.70345
0,000286
0.000591
0.036736
0.002516
O. 0.144300
KSP
0.70331
0.000479
0.000787
0.041623
0.002041
O. 0.4440
Diep
CSD
0.70349
0.000207
0.004104
0.048694
0.000907
0.142015
CSD-iPOT
0.70327
0.000151
0.004071
0.046705
0.002762
O. 1.143716
Non-calibrated
0.68993
0.001444
0.005714
0.064478
0.001918
0.144443
KSP
0.68993
0.000211
0.001181
0.025636
0.002197
0. 144483
AFT
CSD
0.69010
0.000216
0.003218
0.036171
0.002119
O. 0.1445
CSD-iPOT
0.69010
0.000098
0.001515
0.028447
0.002509
O. D.145438
47

---

Table 21: Comparison of yost-processing methods for the SEER-lung dataset. Higher C-index values
indicate better discrimination, while lower values are preferred for all other calibration metrics.
Model
Method
C-index
S-cal(20)4
D-cal(20)+
KS-cal
KM-call
IBS
Non-calibrated
0.68068
0.000254
0.001268
0.032123
0.000012
0. 1.60811
KSP
0.68068
0.000032
0.000129
0.011249
0.000162
0.106655
DeepSurv
CSD
0.68087
0.000211
0.006196
0.056819
0.000573
0.107374
CSD-iPOT
0.68087
0.000139
0.005479
0.050332
0.000455
0.107197
Non-calibrated
0.67676
0.000083
0.000531
0.044664
0.000273
0.109589
KSP
0.67671
0.000307
0.000624
0.044878
0.000338
O. 0.109683
MTLR
CSD
0.67695
0.000127
0.004891
0.051604
0.000394
0. 0.109700
CSD-iPOT
0.67699
0.000119
0.004575
0.044352
0,000584
0.110014
Non-calibrated
0.67932
0.000732
0.004613
0.051627
0.000897
0.106561
KSP
0.68106
0.000243
0.001297
0.026156
0.000780
0.106744
Parametric
CSD
0.68114
0.000179
0.004534
0.047126
0.001264
O. 107087
CSD-iPOT
0.68096
0.000095
0,002838
0.044666
0,001082
0.107186
Non-calibrated
0.65632
0.192684
0,499422
0.692678
0,287116
0.419652
KSP
0.66622
0,000913
0.013576
0.061543
0.001476
O. 1.111111
CRPS
CSD
0,65819
0.001 192
0.004206
0.069645
0.014257
O. 148662
CSD-iPOT
0.66566
0.000407
0.004188
0.052539
0.000993
0.114890
Non-calibrated
0.681555
0.000093
0.000546
0.044664
0.000268
0.108290
KSP
0.68151
0.000305
0.000648
0,044670
0.000327
O. 0.108375
DeepHit
CSD
0.68214
0.000127
0.004850
0.051287
0.000388
0.108417
CSD-iPOT
0.68218
0.0001 18
0.004547
0.043773
0.000590
O. 0.108151
Non-calibrated
0,67360
0.002282
0.007794
0.077526
0.001585
0.111226
KSP
0.67360
0.000331
0.001690
0.029455
0.001573
0.111337
AFT
CSD
0.67378
0.000136
0.005 106
0.047482
0.001409
0.111230
CSD-iPOT
0.67378
0,000080
0.002245
0.041804
0,001808
0.111826
48

---

G
Calibration plot
In this section, we present calibration plots across datasets and models.
DeepSurv
MTLR
Parametric
CRPS
DeepHit
AFT
Hith
ogn
DeepSurv
MTLR
Parametric
CRPS
DeepHit
AFT
CaO
PricONt prbbbiit
Tredbee (rolnclt)
TrEDE prinob
Teddaa pcddait
Teddadprbsiiy
DeepSurv
MTLR
Parametric
CRPS
DeepHit
AFT
Kit
Figure 15: Calibration plot across six models for the WHAS dataset. Each model is evaluated under four calibration methods; Non-calibrated, KSP, CSD, and CSD-iPOT. A calibration slope close to indicates better agreement between predicted and observed probabilities.
DeepSurv
DeepHit
Parametric
CRPS
DeepHit
AFT
koge
Kopn
oon
DeepSurv
MTLRR
Parametric
CRPS
DeepHit
AFT
0G0
0G0
CsD
CGD
Cap
Crinen propatty
Pricols prbeciey
Trideder incbelle
"Hidcie brineity
Teddciad peladity
Teidadppballly
Trieched prbibiryy
DeepSurv
MTLR
Parametric
CRPS
DeepHit
AFT
USP
Figure 16: Calibration plot across six models for the METABRIC dataset. Each model is evaluated under four calibration methods: Non-calibrated, KSP, CSD, and CSD-iPOT. calibration slope close to indicates better agreement between predicted and observed probabilities.
49

---

DoepSurv
MTLR
Parametric
CRPS
DoepHit
AFT
KSP
DeepSurv
Parametric
DeepHit
520
C20
-S0
C50
Pricott prbeciit
Tridbe proseol
Teddiad pcoddity
Teddaaprbaily
DoepSurv
MTLR
Parametric
CRPS
DoepHit
AFT
sse
KSP
-e-
Figure 17: Calibration plot across six models for the GBSG dataset. Each model is evaluated under four calibration methods: Non-calibrated, KSP, CSD, and CSD-iPOT. A calibration slope close to indicates better agreement between predicted and observed probabilities.
DeepSurv
MTLR
Parametric
CRPS
DeopHit
AFT
ASP
XSP
XIP
XSP
KSP
Agh
DeepSurv
DeopHit
1971
Priccder Inbeciee
Tricdder intbelit
"ridecec ininaity
Trededed pnaaeity
Triede/ppcaally
"rieclef
DeepSurv
MTLR
Parametric
KIP CRPS
DeepHit
AFT
ASP
XSP
XSP
KSP
HS2
Figure 18: Calibration plot across six models for the NACD dataset. Each model is evaluated under four calibration methods: Non-calibrated, KSP, CSD, and CSD-iPOT, A calibration slope close to indicates better agreement between predicted and observed probabilities.
50

---

DeepSurv
MTLR
Parametric
CRPS
DoepHit
AFT
infurite
DeepSurv
Parametric
DeepHit
520
C20
-S0
C50
Pricodtd prbbbiit
Tridbet proseol
"Ndder prinaity
Trddiad pcodiity
Teddaaprbaily
DoepSurv
MTLR
Parametric
CRPS
DoepHit
AFT
KPP
Figure 19: Calibration plot across six models for the NB-SEQ dataset. Each model is evaluated under four calibration methods: Non-calibrated, KSP, CSD, and CSD-iPOT. A calibration slope close to indicates better agreement between predicted and observed probabilities.
DeepSurv
MTLR
Parametric
CRPS
DeopHit
AFT
ASP
XSP
XIP
KSP
KSP
Aga
Wildlse
DeepSurv
DeopHit
1971
-e-Ckt
Triccder Inbeslee
Tricdder incbelit
"ridecec inieatty
Tedided priaaity
Triede/pehaally
Trieclef
DeepSurv
MTLR
Parametric
CRPS
DeepHit
AFT
ASP
XSP
XSP
KIP
KSP
HS2
Figure 20: Calibration plot across six models for the SUPPORT dataset. Each model is evaluated under four calibration methods: Non-calibrated, KSP, CSD, and CSD-iPOT. A calibration slope close to indicates better agreement between predicted and observed probabilities.
51

---

DeepSurv
MTLR
Parametric
CRPS
DoepHit
AFT
sal
Kal
DeepSurv
Parametric
DeepHit
520
000
x -50
C50
Pricoltd prbebiit
Tridbe proseole
Teddcad pcadiit
Teddaaprbaily
DoepSurv
MTLR
Parametric
CRPS
DoepHit
AFT
sse
KPP
-e-
Figure 21: Calibration plot across six models for the MIMIC-III dataset. Each model is evaluated under four calibration methods: Non-calibrated, KSP. CSD, and CSD-iPOT. A calibration slope close to indicates better agreement between predicted and observed probabilities.
DeepSurv
MTLR
Parametric
CRPS
DeopHit
AFT
ASP
ISP
XIP
KGP
KSP
Rg?
Wildls
DeepSurv
DeopHit
1971
Triicder Inbeslee
Tricdder incbelit
"ridecec inieaity
Trededed prioaity
Triede/pchallyy
Trieclef
DeepSurv
MTLR
Parametric
KIP CRPS
DeepHit
AFT
ASP
XSP
XSP
KPP
HS2
Figure 22: Calibration plot across six models for the SEER-liver dataset. Each model is evaluated under four calibration methods: Non-calibrated, KSP, CSD, and CSD-iPOT. A calibration slope close to indicates better agreement between predicted and observed probabilities.
52

---

DeepSurv
MTLR
Parametric
CRPS
DoepHit
AFT
KSP
Wiislee jollo
DeepSurv
Parametric
DeepHit
520
-S0
C50
-c0
Pricott prbebiit
Tridber proseol
Teddiad poodiity
Teddadprbally
DoepSurv
MTLR
Parametric
CRPS
DoepHit
AFT
KPP
-e-
Figure 23: Calibration plot across six models for the SEER-stomach dataset. Each model evaluated under four calibration settings: Non-calibrated, KSP, CSD, and CSD-iPOT, A calibration slope close to indicates better agreement between predicted and observed probabilities.
DeepSurv
MTLR
Parametric
CRPS
DeepHit
AFT
ASP
ISP
XIP
KSP
KSP
Ag?
Wildle
DeepSurv
DeopHit
1971
-e-CAt
Triicder Inbeslee
Tricdder incbelit
"ridecec inieaity
Trededed prioaity
Triede/pehallly
Trieclef
DeepSurv
MTLR
Parametric
CRPS
DeepHit
AFT
ASP
XSP
XSP
KIP
KPP
RS2
Figure 24: Calibration plot across six models for the SEER-lung dataset. Each model is evaluated under four calibration settings: Non-calibrated, KSP, CSD, and CSD-iPOT, A calibration slope close to indicates better agreement between predicted and observed probabilities.
53

---

H Evaluation of calibration errors with 10-bin
In this section, we compare the performance of S-cal and D-cal using evaluations with bins. To determine bin locations, we consider two strategies: uniformly spaced bins and scheme that allocates denser bins in the tails. The results are reported a format similar to Table L in the main manuscript. For uniformly spaced bins, use knot points at {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9), while for tail-focused bins, we use {0.05, 0.10, 0.20, 0.40, 0.50, 0.60, 0.80, 0.90, 0.95). We denote the former case as S S-cal(10)), D-cal(10)*, and the latter case as S-cal(10)", D-cal(10)².
When comparing non-calibrated baselines, KSP consistently outperforms them, regardless of the metric. For CSD, evaluations with S-cal(10) and D-cal(10)' slightly favor CSD over KSP, whereas CSD-iPOT outperforms when evaluated using the same metrics. This natural, since these methods adjust quantile levels exactly at the knot points of S-cal(10)* and D-cal(10)*. However, when calibration errors are measured with denser bins in the tails, KSP again outperforms all methods, consistent with the results from S-cal(20), D-c,l(20), and KS-cal. From this perspective, metrics that depend on the choice of intervals are neither consistent nor fair, highlighting the advantage of bin-free metrics for robust comparison.
Table 22: Summary of pairwise comparisons between post-processing methods. The table shows the
number of cases where KSP outperforms its counterpart, is outperformed, or yiclds a tic. Numbers in parentheses indicate statistically significant differences based on one-sided t-test at the 0.05 level.
Method
S-cal(10)*
D-cal(10)*
S-cal(10)²
D-cal(10)2
S-cal(20)
D-cal(20)
KS-cal
KSP
48 (45)
46 (43)
47 (44)
45 (42)
46 (45)
46 (43)
47 (45)
Non-calibrated
12 (8)
14 (11)
13 (7)
(9)
13 (7)
14 (6)
13 (5)
Ties
1
0
0
0
KSP
36 (28)
32 (29)
37 (28)
41 (37)
36 (29)
48 (45)
51 (42)
CSD
(19)
28 (26)
23 (15)
19 (17)
24 (19)
12 (10)
(
Ties
KSP
26 (14)
(14)
36 (20)
44 (33)
32 (21)
46 (39)
(29)
CSD-iPOT
34 (23)
35 (27)
24 (13)
16 (15)
28 (19)
14 (13)
16 (11)
Ties
0
0
0
0
0
0
Ablation study on KSP
In this section, we present an ablation study investigate the effectiveness of KSP. For simplicity, we conduct experiments using the MIMIC-III dataset with three models: DecpSurv, MTLR, and CRPS. This dataset characterized by relatively high calibration error, making a suitable benchmark for evaluating post-prost most-processing methods.
L11 Ablation study 1: link function
We evaluate KSP using various link functions, including the logit, inverse hyperbolic tangent (atanh), three inverse CDFs with range (-00, ac)--coobil inverse Cauchy, and inverse Laplace- and the complementary loglog (cloglog) function. Since atanh has domain of (-1, 1, 1), we rescale the CDFs using 2F for compatibility.
Table 23 summarizes the performance across DeepSurv, MTLR, and CRPS. While the logit function does not al always yield the best score for every metric, it consistently delivers near-optimal performance across both discrimination and calibration. Atanh and probit show reasonable performance but are prone to numerical instability. The inverse Cauchy and inverse Laplace functions exhibit good performance on some metrics; however, they inherently assume specific parametric forms, making them less suitable for general-purpose use. Cloglog, intended to emphasize tail behavior, results in moderate performance. Considering both empirical stability and robustness, we adopt the logit function as the default link function in KSP.
54

---

Table 23: Comparison of different link functions G used in KSP for DeepSurv, MTLR, and CRPS on the MIMIC-III dataset. Bold values indicate the best performance within each model.
Model
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
logit 0.95826
0,003305
0.007044
0.091109
0.071617
0.203039
atanh 0,94989
0,003882
0.022601
0.099237
0.071447
0.201032
probit 0,93952
0.005747
0.04607 I
0. 0.120879
0.076192
0.203363
DeepSurv
inverse Cauchy
0.95803
0.003273
0.0095
0.096415
0.069989
0.200894
inverse Laplace
0.95804
0,004483
0.0146499
0.111702
0.074329
0.208606
cloglog 0.95797
0.004379
0.014051
0. 0.109403
0,074808
0,209548
logit 0,94089
0.003552
0.004132
0.084090
0.000409
0.0125 12
atanh
0.94086
0.003632
0,004106
0.084987
0.000408
0.012503
probit 0.94155
0.004588
0.011219
0. 0.1999
0.000405
0.012455
MTLR
inverse Cauchy
0.94150
0.004397
0.009183
0.103863
0.000417
0.012737
inverse Laplace
0,94112
0,003970
0.006271
0.095429
0.000397
0,012535
cloglog 0.94090
0.003831
0.005342
0.091252
0.000409
0.012462
logit 0.95339
0.013238
2086
O. 149600
0.000378
0.012503
atanh 0.95339
0,013067
0.011765
0.149861
0.000381
0.012514
probit 0,95339
0.012670
0.012858
0.145194
0.000379
0,012507
CRPS
inverse Cauchy
0.95341
0.015722
0.027449
O. 0.184878
0.000755
0.012856
inverse Laplace
0.95339
0.013383
0.012086
0.158158
0.000399
0.012402
cloglog 0.95339
0.013458
0.013700
0. 0.149499
0.000377
0.012588
1,2
Ablation study 2: impact of using top-k deviations
The proposed KSP in the main text minimizes the KS-cal, which corresponds the maximum deviation between the predicted and empirical CDFs. In this ablation study, we evaluate an alternative formulation that minimizes the sum of the & largest deviations instead of just the maximum alone. We report the performance of KSP under different values of & to examine whether aggregating multiple top deviations yields improved calibration or stability.
As shown in Table 24 increasing & leads to marginal improvements calibration metrics; however, the gains are minimal. Notably, for MTLR, larger values result in decline in the C-index and a rise in calibration errors. These findings indicate that minimizing only the maximum deviation (k: 1) is sufficient to ensure effective calibration without sacrificing predictive performance.
Table 24: Ablation study on the choice of top-k deviations in the KSP. We report results for is {1, 5, 10, 50, 100) across three models (DeepSurv, MTLR, CRPS) on MIMIC-III. The case k = 1 corresponds to the original KSP formulation using only the maximum deviation.
Model
Top A
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
0.95826
0.003305
0.007044
0.091109
0.071617
0.203039
0.95826
0.003313
0.007051
0.090815
0.071621
0.203101
DeepSurv
10
0.95826
0.003252
0.006966
0.090368
0.071437
0.202842
0.95826
0.003233
0.007143
0.090056
0.071202
0.202644
100
0.95826
0.003217
0.007276
0.089720
0.071107
0.202672
0.94089
0.003552
0.004132
0.084090
0.000409
0.012512
5
0.94086
0.003703
0.004485
0.084696
0.000410
0,012518
MTLR
10
0,94082
0.003764
0.004679
0.085963
0.000411
0.012511
50
0.94074
0.003840
0.005624
0.085756
0.000415
0.012516
100
0.94078
0.003854
0.005538
0.085496
0.000416
0.012523
0.95339
0.013238
0.012086
0. 0.149000
0.000378
0.012503
5
0.95339
0.013079
0.012191
0.148102
0.000373
0.012438
CRPS
10
0,95339
0.013475
0.012308
0.150593
0.000378
0.012480
50
0.95339
0.013189
0.012114
0.1097
0.000372
0.012440
100
0.95339
0.012869
0.012076
0.148883
0.000370
0.012400
55

---

1.3 Ablation study 3: Form of the deviation
Originally, the KSP was defined using the absolute value deviations Step 4. We also experiment with an alternative formulation using the squared error. Since the quadratic form penalizes large devi- ations more heavily than the absolute value, may potentially lead to better calibration. Additionally, we report the optimization time in seconds (mean and standard deviation) to compare computational efficiency.
As shown in Table 25 although the squared objective slightly emphasizes larger deviations, the absolute form achieves nearly identical performance in both discrimination and calibration. More importantly, it is more efficient in training time.
Table 25: Effect of using squared vs. absolute deviation in KSP optimization. Time is reported in seconds (mean and standard deviation).
Model
Form
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
Time
Absolute Squared
0.95826
0.003305
0.007044
0.091109
0.071617
0.203039
3,0163 (0.4331)
DeepSurv
0.95826
0.003318
0.007094
0.090898
0.07 1853
0.203535
3.4383 (0.6512)
Absolute 0,94089
0,003552
0,004132
0.084090
0.000409
0.012512
3,0810 (0.8526)
MTLR
Squared 0.94216
0.003563
0.004232
0.084132
0.000407
0.012502
4.1825 (0.6761)
Absolute 0.95339
0.013238
0.01 2086
0. 14946000
0.000378
0.012503
3,4411 (0.8088)
CRPS
Squared
0.95339
0.013238
0.012086
0.149600
0.000378
0.012503
6.6034 (1.8035)
56

---

1.4 Ablation study 4: hyperparameters
To assess the contribution of each hyperparameter used in ASP-namelly a, and a we conduct an ablation study by selectively choosing each component. This results in a total of configurations, and the performance of each setting is reported evaluate the individual and joint effects of the hyperparameters.
As shown in Table 26 using all three hyperparameters a, b, and yields consistently strong perfor- mance across models. While the relative importance each parameter may vary depending on the model, all three appear contribute comparably the overall calibration and discrimination quality.
Table 26: Ablation study on the three hyperparameters a, and a used KSP. Each row shows the performance when a subset of these hyperparameters used.
Model
Hyperparameter
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
Non-calibrated
0.95839
0.036649
0.064547
0.250907
0.104279
0.276509
a
0.95826
0.032545
0.050528
0.231501
0.102629
0.269723
b 0,95826
0.01
0.041974
0.183154
0.077582
0,230868
0,95827
0.004572
0.016652
0.114728
0.074552
0.214151
DeepSurv
a,
0.95826
0.005908
0.025273
0.127957
0.076690
0.213331
a.
0.95826
0.004852
0.017134
0.113908
0.075403
0,210912
b, a
0.95826
0.004119
0,011969
0.105770
0.073772
0,207425
b, a
0,95826
0.003305
0.007044
0.091109
0.071617
0.203039
Non-calibrated
0.94494
0.015402
0.035474
0.232722
0.000410
0,014391
0,94212
0.002954
0.010749
0.092681
0.000403
0,012762
b
0,94075
0.016474
0.037806
0.209210
0.000380
0.013446
MTLR
a
0.94077
0.017945
0.041025
0.216149
0.000415
0.013717
a,
0.94168
0.004578
0.011120
0.106987
0.000403
0.012481
0,
0,94164
0,004438
0.010614
0.105226
0.000406
0.012481
b, c
0.94175
0.01
0.029399
0.185595
0.000346
0.012655
a, b,
0.94089
0.003552
0.004132
0.084090
0.000409
0.012512
Non-calibrated
0.95342
0.214466
0,258780
0.759872
0.020505
0.072193
0.95342
0.083571
0.942654
0.499678
0.194741
0,236811
0.95341
0.029452
0.065234
0.278981
0.000986
0.014843
CRPS
a
0.95342
0.048102
0.153166
0.357446
0.002043
0,021537
a,b 0,95339
0.013067
0,012301
0.151768
0,000396
0.012490
a.
0.95340
0.043578
0.125829
0.340774
0,000908
0.017905
b, 0.95340
0.022179
0.043438
0.242998
0.000822
0.013355
0, b,
0.95339
0.013238
0.012086
0.149600
0.000378
0.012503
On the next page, Figure 25 illustrates how each hyperparameter affects F(x).
57

---

am1, am
0.5. D. m
a 2.0, b. 0.0
Avl
Aut
Pefest Beaice
Prtlct bratinn
Perfec Callbration
0.2
6.0
6.0
0.2
6.2
Effect of varying a with fixed a
3=1,0=0,a-
1= D 1, am
J=1.b=1,aa1
Fiit
Avl
Aul
0.0
0.2
0.2
0.2
(b) Effect of varying with fixed
3, 1,
a=1, 1, 0.5
mm
Avi
Avi
67-
Effect of varying with fixed 1, b
Figure 25: Visualization of how each hyperparameter (a, b, a) influences F(x)) The leftmost plot in each row is the baseline: = 1, b = 0, a = 1.
58

---

Adaptability of KS-cal to in-processing procedures
As discussed in the main paper, the original KS-c can be adapted for in-processing by reformulating it as penalty term:
KS-cal(k) Σ (D(N-sti)²
Here, k: determines how many of the largest deviations are penalized, controlling the trade-off between discrimination and calibration. Using only the maximum deviation (k = 1) can be unstable, as the evaluation point i varies during training with changes in Fg. To reduce this instability, we aggregate the top-k deviations. The squared form improves gradient behavior and allows for smoother optimization.
We evaluate E (1,5, 10} on the MIMIC-III dataset using DeepSurv, MTLR, and CRPS, and compare the results to X-cal with 20 bins, denoted as X-cal(20). To assess each method's ability to manage the trade-off between calibration and discrimination, we plot calibration error against the C-index for various values of X and compute the area under the resulting curve. A smaller area indicates better trade-off.
As shown in Figure 26 increasing À generally reduces calibration error but also lowers the C-index. Larger values of tend yield more favorable trade-offs. While S-cal(20), D-cal(20), and KS-cal show consistent decreases with increasing X, KM-cal and IBS do not, particularly for models other than DeepSurv.
Although KS-cal(10) clearly outperforms X-cal(20) in terms of area, the comparison may be mislead- ing, it includes points with low C-index values that rarely acceptable in practice. Moreover, computing the full curve requires densely sampling X, which incurs a high computational cost. Thus, to selecting single optimal À per method.
59

---

MIMIC-III, SeepSurv
MIMIC-II, MITUR
MIMIC-III CRPS
KS--aals) 1-27 82%))
1, 000344
0.015
0.000410
28%)
0.20
KSS c 1 (-38 48%)
0.0021 en
0.03
-al-call 10 (26.68)))
3.000416
KS oat (0), 0.000237 (-12.02%)
I6S ca11 0.0019/ (68.011))
S-ca(
4 (-ca((0))
0.01 0,010
S-cal(20)
0.10
0.005
0.01
0.00
0.000
0.00
0.6
0.7
0.8
0.0
0.80
0.85
0.50
0.91
0.575
0,000
0.925
0.050
C-index
C-index
C-index
MIMIC-III, DeepSurv
MIMIC-II MTLR
MIMIC-III, CRPS
30%)
001208
cReac1
KS-cwlb) 51, 8 003113 (+25, 65%
KSS-catb) 0.00104 (4.29%)
KS-ccllt) a 612 1:25 (2%)
(+1.00%%)
(-0275)))
-x.all20)_ 6 0.000886
-X.aa,20), 0,301036
0.2
=X.al2201, 0.003406
D-cali20)
(-cal(20)
D-cal(20)
0.00
1
0.02
0.00
0.00
0.0
0.80
0.55
0.90
0.80
3.675
0.500
0.625
0.950
C-index
C-index
C-index
MIMIC-III, DeepSurv
MIMIC-II, MTLR
MIMIC-III, CRPS
0.25 KB 0.018127 (4.40%) KS-cal5s 0.018731 1-11,77%)
K3 0.012582 (5755%)
calt) 0.027967 (e15: 74%)
KB ca15) 1011788 (11,7755
KK-calls) 0C18820 /-22 193)
5891
0.20
X-oil(200), 0.0 3.011992
X-cak22)) 0.015326
0.8
X-cakl20), 0 0.02:906
KS-cal
0:15
KS.C cal
0.18
KS-c8
0.4
0.10
0.10
0.2
0.05
0.0
0.90
0,500
0.626
C-index
C-index
C-index
MIMIC-II, DeepSurv
MIMIC-II, MTLR
MIMI-III
CRPS
KB call) D 0702117 11 33%)
KBB catt) 8.72+ 11%)
0.08
KB calt 0.001540 (1071%)
0,00
KS (S--al(1C), 10), 0,50 0,501254 (45.811%) X-ea1220), 0.002367
KS-cal 9.40 (-22,91%) X-cw)/20 7.00a55
KB- 0 (331199))
X-CwiA0) 0.001725
0 0.075
KM-cal
KM cal
KM-cal
0.050
Be-04
0.025
0.000
0e100
0.7
0.8
0.9
0.80
0.50
0.90
0.575
0,900
0.850
C-index
C-index
C-index
MIMIC-IL, SeepSurv
MIMICIII MTLR
MIMICIII CRPS
1KB-cwk1) o 3.02188 580 (-8 8P%) 5)
KSSoailt 008952 1-4 08%)
KS-oal (0), 0.019130 30 (-10.50%
0.04
K.S cai [S] 0 (-742%
KS cwil1, 003872 (-5 (-5.2)%)
KS ca DJ D 6.007700 (-20.515)
(20)
2X)ai:20), 0.003615
0.000102
0.10
(20),
IBS
0.00
1BS
0.1
0.32
0.06
0.8
0.7
0.8
0.80
0.55
0.30
0.8
0.04
0.575
0 0,503
0.925
0.950
C-index
C-index
C-index
Figure 26: Trade-off between C-index and calibration errors on the MIMIC-III dataset using DeepSurv, MTLR, and CRPS. The legend indicates the area under the curve for each method, with the percentage decrease relative to X-cal shown in parentheses.
60

---

Example of Selection
Calibration Error
Increasing from
0.04
0.03
0.02
Optimal
0.00
0.86
0.88
0.90
0.92
0.94
0.96
0.98
1.00
C-index
Figure 27: Trade-off between C-index and calibration error across different values of X. The optimal point minimizes the Euclidean distance to (1, 0).
Table 27: Example of finding optimal X.
C-index
S-cal(20)
D=cal(20)
KS-cal
CE
1- C-index (scaled)
CE (scaled)
Distance
Optimal
0.95839
0.036661
0.064546
0,250918
0.054722
0.6135
1.0000
1.1732
X
0.95749
0.010718
0.017268
0,151156
0,016945
0.6268
0.2021
0.6586
O
0.954833
0.006342
0.014011
0.132187
0.012609
0.6660
0.1105
0.6751
X
0.94375
0.004108
0.010805
0.113729
0.009282
0.8294
0.0403
0,8304
X
0.100064
0.0073776
0.0000
J.1 Selection of the regularization parameter
The core challenge in in-processing methods is choosing a regularization parameter À that balances calibration and discrimination. While prior work often prioritizes one over the other (Karandikar et al. 202T). ). our approach seeks to balance both by minimizing the Euclidean distance to the ideal error space:
point (1, 0) in the
Distance V(1 C-index)² + (calibration error)?
We use S-cal(20), D-cal(20), and KS-cal to compute calibration error and exclude K.M-cal and IBS due to their non-monotonic behavior with respect X. Since each metric shows different sensitivity to Ả (Figure 26). average the three and denote this composite as CE. We use the squared KS-cal in this average to account for its magnitude.
To ensure both components contribute equally to the distance, we apply the following min-max normalization:
- C-index
C-index
minx (C-index)
CE minx (CE)
CE
max, (CE) min, (CE)
This normalization brings both metrics to the same scale before computing the distance. An example of the scaled values and computed distances is shown in Table 27
Using this criterion, we compare the best cess-performin configurations of KS-cal(k), X-cal(20), and KSP. The results are summarized in Table 28] In-processing methods can sometimes yield lower calibration error than post-processing approaches, but this often comes at the cost of a reduced C-index. They may be appropriate when calibration is prioritized over predictive accuracy, despite a higher computational cost. Addressing this trade-off remains an important direction for future research.
61

---

Table 28: Comparison of in- in-processing methods with their optimal À values and KSP on the MIMIC- III dataset using DeepSurv, MTLR, and CRPS.
Model
Method
C-index
S-cal(20)
D-cal(20)
KS-cal
KM-cal
IBS
(1-
0,93170
0.003199
0.010169
0.110807
0.018586
0.103577
KS-cal(5)
0.92872
0.003000
0.009330
0.101107
0.010723
0.077831
DeepSurv
(1))
0.93218
0.003108
0.009007
0.100064
0.009279
0.072327
X-cal(20)
0.93422
0.003895
0.007950
0.117502
0.020680
0.104861
KSP
0.95826
0.003305
0.007044
0.091109
0.071617
0,203039
(1)
0.88891
0.002555
0.007000
0.083445
0.000590
0.01
(S)
0,90702
0.003735
0.009094
0.098823
0.000527
0.016721
MTLR
(1))
0.90546
0.003365
0.008331
0.094365
0.000539
0.016840
X-cal(20)
0,89491
0.002862
0.006477
0.096721
0,000586
0.018059
KSP
0.94218
0.003581
0.004200
0.084666
0,000405
0.012505
(1)
0.91555
0.068442
0.062303
0.349527
0.021180
0.088294
KS-cal(5)
0,93445
0.013617
0.016511
0.168024
0.012609
0.072749
CRPS
KS-cal(10)
0,93854
0.005743
0.008530
0.118517
0.009256
0.060236
X-cal(20)
0,92507
0.051386
0.028927
0,335169
0.014853
0.072469
KSP
0.95339
0.013238
0.012086
0.149600
0.000378
0.012503
K
Additional information on experiment
Table 29 reports the post-processing time (in seconds) for KSP, CSD, and CSD-iPOT across all real datasets. Table (30 summarizes the hyperparameter settings used for training. All models were optimized using the ADAM optimizer with early stopping (patience 200).
Table 29: Post-processing time (in seconds) based on KSP, CSD, and CSD-iPOT across datasets, reported as Mean (SD).
Dataset
Model
KSP
CSD
CSD-PPT
Dullsct
Model
KSP
CSD
CSD-IPOT
Decpure
0.7675
6015)
DaepSurv
(0.2225)
(0.041 7)
MILR
MILK
WHAS
CRPS
2.2840 (1.49))
0.5500 (0.5245)
04331
0,2533 (0.0130)
(0.0059)
SUPPORT
CRPS
2.6104
1.4599
3.5057 (0.0425)
4.7179 01000)
1,7300 (0.0464)
1.7611 (0.0655)
Dosplit
116656003111
0.4000
01000
0:2544 (0.0075)
Desplii
1.588 (0.647)
1.3576 1231)
1.7854 (0:00779)
AFT
1.0208 (0.1975)
15,9979 (1.241))
0.2419 (0.01 (6)
APT
0.7851 (0.2624)
30,0743 (2.5449)
1.5943 (0.0723)
DecpSurv
0,7030 (0.5403)
0.6364 (8.250%
0,2940 (0.01
DeepSurv
5,0368 (0.6841)
2,4220 (0.4372)
L.4008 (0.0412)
MILK
1188550000 00335605794)
Bastt (4344))
0,2993 (0.0167)
MTL.K
3,871 (08218)
42.1541 (2
1.75% (0.00611
METABRIC
Permmae
CRPS
0.582
11175 7185)
(0.4417)
1,1755 (116))
1.111 (1
0:00150000114
(0.0105)
MIMIC-III
Premmtric
309977
1.519 (0.512)
11,5253
14 (2.959)
L7573
1.779
Desplii
(04121)
AFT
1,4389
14,3036
(0.6115)
7240553 (1.919))
L7750 (0,0672)
DeepSurv
0.9000 (0.4123)
19335 (1.1640)
0,3427 (0.0125)
DespSurv
1,2076 (0.3057)
10.9716 (0.1532)
7. 7,5048 (C.1496)
MTLR
1.5767
(0.7258)
0.6345 01065
033005 (0.0126
MTLR
0.3957 (0. 1692)
12.7019 (0. (0.1489)
9. 5,5456 (S.CA95)
GBSG
CRPS
1.2178
1.0633.0 (0 54003
(0.593)
06355 (855
2000 [1 92083
03655 (0.0192)
1:3455 (0.01
SEER-Eier
Preantrrc:
CRPS
5,4409
3,0441
(0.7013)
10.2443 (0 (16000
10.2083 (n 10000
2. 7.5644 1871)
7, 7,4120 (0.1000) 1100)
Deeplix
45256 (0.0892)
08:03 (10223)
022050 (0.0083))
Despliit
0.3226
1697)
12.7441 (0 121111
90000 1991))
AFT
(0, (961)
1600711 (0.033))
(0.0213)
5,2206 (0.8557)
(2.4466)
7,0094
(06432)
1.0106
MILK
1.500
0.2451
(0.004))
MTLK
0.4129 (0.3515)
17.40644 2004)
13 3391 (14855)
NACD
Prrmutric
CRPS
0.913
(3.5080
(04075)
0.825 05000
0.5506 0126)
63743 (0.0130
63766 (0.011))
SEER stomach
Pomametric
CRPS
3,5959 (0.5606)
3,1459 (0.8141)
15.0044 (413)
14,3505 (0.9472)
10. 10.1 (1.1355)
10.3314 (0.3136)
Doepiite
(1939
10776 (0 7445)
6'1901 (0.0099)
Desplir
0.6400 (0 1965)
17.8186 (0 73.55
13 19A722 (0.3191)
AFT
0.5895 (0.2225)
(1.1591.1139)
0.3255 (0.0004)
AFT
2,5866 (0.644)
38,4594 (4.0547)
9,5965 (0.1246)
DespSur
0.6366
1 (0.0315)
(0.0197)
DecpSurv
2.5002 (0.5320)
(4.3867)
194
MITR
MILK
(0.1773)
(XM33T)
NB-5BQ
CRPS
19,0665
(0.0385)
SEER-kung
CRPS
6,3627
(1.6321)
(2.4764)
174. 2676)
Desplia
(17653 (0. 3043)
14451 (5.75)))
6.3161 (0.0157)
Decplii
0.5815 (0.22%)
270 3440
(21. (6116)))
200 6404 (12.412))
AFT
2,5707 (0.3950)
1T,5392 (2.2009)
0.7256 (0.0251)
AFT
7,3328 (1.5618)
354.4645 (10.3814)
175 9503 (2.5399)
62

---

Table 30: Set-ups used in experiments.
Dataset
Model
Batch size
Learning rafe
Maximum epoch
Dropout rate
Hidden layer
Number of bins
DeepSurv
64
le-4
1000
0.1
24/12/12
Parametric
64
lo-3
1000
0.1
24/12/12
WHAS
CRPS
1e-3
1000
0.1
24/12/12
DeepHit
64
1o-3
2000
0.1
24/12/12
40
AFT
Io-1
1000
DecpSurv
lo-4
1000
0.3
36/18/18
MILR
64
1o-3
2000
20
METABRIC
Purametric
lo-3
1000
0.3
36/18/18
CRPS
256
1e-3
1000
0.3
36/18/18
36/18/18
AFT
128
le-1
1000
DeepSurv
128
le.4
1000
0.3
28/14/14
MILR
128
1e-3
2000
20
Parametric
128
le:3
1000
0.3
28/14/14
GBSG
CRPS
256
le-3
1000
0.3
28/14/14
Deepllit
128
le:3
2000
0.3
28/14/14
40
AFT
128
le-1
1000
DeepSurv
128
le.4
1000
0.1
51/51/51
le-3
NACD
Parametric
CRPS
128
le-3
1o-3
1000
1000
0.1
0.1
51/51/51
51/51/51
DecpHit
128
1o-3
2000
0.1
51/51/51
40
AFT
128
le-1
1000
DeepSurv
96/48/48
MTLR
128
le:3
2000
20
Parametric
1o-3
96/48/48
NB-SEQ
CRPS
512
le:3
1000
0.3
96/48/48
DeepHir
AFT
256
le-1
1000
DeepSurv
512
le4
1000
0.3
56/28/28
MTLR
512
le.3
2000
20
SUPPORT
Parametric
512
le-3
1000
0.3
56/28/28
CRPS
lo-2
1000
0.3
56/28/28
DeepHit
512
l6-3
2000
0.3
56/28/28
40
AFT
1o-1
1000
DecpSurv
512
lo4
1000
0.1
60/30/30
MILR
512
1e-3
2000
20
MIMIC-III
Parametric
512
le:3
1000
0.1
60/30/30
CRPS
1024
1e:3
1000
0.1
60/30/30
Deepllit
512
le:3
1000
0.1
60/30/30
100
AFT
512
le-1
1000
DeepSurv
60/30/30
MiLR
1000
le-2
2000
20
60/30/30
SFRR-livve
CRPS
2000
lo-3
1000
0.1
60/30/30
DeepHit
l6-3
2000
0.1
60/30/30
AFT
1000
1o-1
1000
DeepSurv
1000
le.4
1000
0.1
60/30/30
MTLR
1000
lo-3
2000
10
Parametric
1000
16-3
1000
0.1
60/30/30
SEER-stomace
CRPS
2000
le-3
1000
0.1
60/30/30
DeepHit
1000
1e-3
2000
0.1
60/30/30
10
AFT
DeepSurv
5000
le-4
1000
0.1
60/30/30
MTLR
5000
le:3
2000
10
SEER-
Parametric
5000
le-3
1000
0.1
60/30/30
CRPS
10000
le:3
1000
0.1
60/30/30
DeepHit
5000
le-3
2000
0.1
60/30/30
10
AFT
5000
le-1
1000
63


Reviews:
[{"review_id": "4L3WzBATsu", "reviewer": "Reviewer_64Lb", "summary": "The paper introduces KSP (Kolmogorov-Smirnov post-hoc calibration), a novel method for calibrating survival models using the Kolmogorov-Smirnov metric. The authors address an important challenge in survival analysis by providing a robust, flexible, and theoretically grounded approach that enhances calibration without significantly compromising predictive performance.", "questions": "1. Why does KSP sometimes underperform compared to other methods in specific datasets? What steps can be taken to enhance its performance across these scenarios?\n2. How can KSP be adapted to effectively handle cases with extremely small sample sizes or high censoring rates? Is this feasible? \n3. What future work is planned to further validate and generalize the method across a broader range of survival analysis scenarios, including extreme cases and challenging data conditions?", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Some of the key strengths of this paper are:\n1. The use of the well-established KS metric ensures a solid theoretical basis for the method, making it credible and reliable. Further, KSP demonstrates effectiveness across various survival models and datasets, showcasing its versatility.\n2. The computational efficiency of KSP, requiring fewer iterations compared to other methods like CSD or CSD-iPOT, makes it practical for handling large datasets and complex models.\n3. The method maintains high predictive accuracy while improving calibration, which is crucial in high-stakes domains such as healthcare and infrastructure.\n\nSome of the aspects where this can be improved upon are:\n1. There are instances where KSP performs less well than other methods across specific dataset conditions or small sample sizes. It is important to analyze these scenarios thoroughly to identify \"operating regions\" of the metric\n2. Continuing on the above, the method may not capture calibration error as effectively in very small sample sizes, which could be a limitation in certain scenarios. \n\n\nOverall, the paper presents an innovative solution to a significant problem in survival analysis, offering a method that is both theoretically sound and empirically validated. It has the potential to significantly impact the development and deployment of survival models in real-world applications. The authors are encouraged to explore further enhancements and validations to address its limitations and expand its applicability.", "was_revised": true, "final_justification": "Overall I am satisfied by the response to my questions. I am curious to see how the other reviewers consider the responses to their questions, but to be fair in my review and the ability of the authors to respond to particular queries, I am keeping my current review and scores as is", "rebuttal": "We thank the reviewer for their thoughtful and constructive comments. We respond to each point below and will make all necessary clarifications and corrections in the final version.\n\n$\\\\textsf{Q1}$. Why does KSP sometimes underperform compared to other methods in specific datasets? What steps can be taken to enhance its performance across these scenarios?\n\nThe SUPPORT dataset contains more than 100 tied events at multiple time points, with a particularly severe accumulation of ties at early time points. In the GBSG dataset, approximately 19\\% of the observations are censored at the final recorded time. Due to the presence of heavy ties and skewness in both datasets, the performance of KSP appears to be inferior to quantile-based approaches such as CSD and CSD-iPOT. We believe that the performance of KSP could be improved by incorporating tie-breaking strategies or reformulating the method to explicitly account for ties. We will further elaborate on this issue in the conclusion.\n\n$\\\\textsf{Q2}$. How can KSP be adapted to effectively handle cases with extremely small sample sizes or high censoring rates? Is this feasible?\n\nOne possible direction for improvement is to enhance the nonlinearity of the KSP formulation using deep neural networks, or alternatively, to simplify its complexity. Incorporating appropriate weighting schemes may also help. However, challenges such as small sample size or high censoring rates are not specific to KSP. These are fundamental difficulties that are generally hard to overcome, regardless of the method used.\n\n$\\textsf{Table 1. Average performance for pbc dataset using KSP.}$\n\\\\begin{array}\n\\\\hline\n\\textbf{Method} & \\textbf{C-index} & \\textbf{S-cal(20)} & \\textbf{D-cal(20)} & \\textbf{KS-cal} & \\textbf{KM-cal} & \\textbf{IBS} \\\\\\\\\n\\\\hline\n\\textbf{Non-calibrated} & 0.73462 & 0.009487 & 0.029233 & 0.139288 & 0.037962 & 0.204758 \\\\\\\\\n\\textbf{KSP} & 0.75688 & 0.002120 & 0.021412 & 0.096729 & 0.016621 & 0.182144 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\nWe applied KSP to the pbc dataset from the R survival package, which contains 250, 84, and 84 samples for training, validation, and testing, respectively, with a censoring rate of 61\\%. As shown in the table, the KSP method clearly reduces calibration error. Notably, it also improves the C-index, indicating that KSP can enhance performance in small-sample settings where baseline models often struggle. However, we acknowledge that the improvement may not yet be sufficient to satisfy practical or research expectations fully. Addressing this limitation remains an important direction for future work.\n\n$\\\\textsf{Q3}$. What future work is planned to further validate and generalize the method across a broader range of survival analysis scenarios, including extreme cases and challenging data conditions?\n\nAs discussed in the two points above, a promising direction for future work is to revise the KSP formulation to better accommodate datasets with a large number of tied events or extreme characteristics such as small sample sizes and high censoring rates. This may involve drawing inspiration from existing methods, such as approaches for handling ties like the Efron approximation, or weighting ideas that underlie the log-rank test, while developing solutions specifically designed for calibration-focused objectives."}, {"review_id": "ew78rriqGk", "reviewer": "Reviewer_UH7y", "summary": "A new calibration method is presented, which is based on the Kolmogorov-Smirnov statistic. It aims to avoid computational limitations and the reliance on binning of some earlier methods.", "questions": "- How exactly do the theoretical results presented provide guarantees for KSP's performance? It is unclear how the theoretical results presented carry over to the actual finite-sample behavior of the method.\n\n- When can we expect the current method to work best and under what settings will one or more of the other methods probably outperform the current one?", "limitations": "Yes.", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- As all approaches considered are basically heuristics, it would have been good to try and understand which method performs best in what situation. Currently, some benchmark datasets are used, but carefully designed synthetic data could provide much better insight into the inner behavior of the various methods.\n\n- The suggested method overcomes limitations of earlier methods due to computational complexity and discretization. It is, however, unclear how those earlier methods would perform, in principle, if we could have (close to) unlimited compute and the binning can be coarsened arbitrarily. Are the reported gains primarily the result of computational improvements and the continuous formulation or is there more?\n\n- I think the part of the paper leading up to Theorem 3.1 isn't clear to me, which probably makes me underappreciate the claim in Theorem 3.1. For one, I don't readily see that \"Eqn. (3) is equivalent to (...)\". Also, the text says \"We begin with the simplified setting without covariates,\" but then the general result still follows. The paragraph at line 133 offers some observations and links to other methods, but it is unclear to me what I should take away. In the actual formulation of the theorem, (and maybe this is a silly question but) what does it mean for \"calibration to hold\"? Ultimately, the theorem is an asymptotic result and doesn't necessarily say anything about the finite sample behavior of KS-cal.\n\n- There are some statements that sound appealing, but remain unsubstantiated, as far as I can see. Some examples:\n - Abstract: \"Existing approaches (...) often rely on heuristic binning or nonparametric estimators, which undermine their adaptability to continuous-time settings and complex model outputs.\" (Besides, KS is also a nonparametric estimator.)\n - Page 1: \"but their reliance on fixed sampling schemes or predefined percentiles may limit adaptability.\"\n - Page 9: \"KSP offers a theoretically grounded framework that avoids the limitations of bin-based or sampling-dependent approaches.\"\n\n- I find related work sections \"after the fact\" not very helpful. Apparently, these are works that did not inspire the current contribution, nor are they important to put the work in context nor are they put in a different context based on the results in this contribution. I believe this whole section should be taken out, or whatever is really relevant should be moved to the front.\n\n- A critical discussion in which the pros and cons of this new method in the context of earlier methods and results is lacking. In addition, none of the related works are discussed in the new context provided by the proposed method", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and constructive comments. We respond to each point below and will make all necessary clarifications and corrections in the final version.\n\n$\\textsf{Q1}$. As all approaches considered are basically heuristics, it would have been good to try and understand which method performs best in what situation. Currently, some benchmark datasets are used, but carefully designed synthetic data could provide much better insight into the inner behavior of the various methods.\n\nThe SUPPORT dataset contains more than 100 tied events at multiple time points, with a particularly severe accumulation of ties at early time points. In the GBSG dataset, approximately 19\\% of the observations are censored at the final recorded time. Due to the presence of heavy ties and skewness in both datasets, the performance of KSP appears inferior to that of quantile-based approaches, such as CSD and CSD-iPOT. We believe that the performance of KSP could be improved by incorporating tie-breaking strategies or reformulating the method to account for ties explicitly.\n\nIn addition, we observe that KSP tends to perform more effectively when applied to datasets with fewer ties, to models that are not discretized, and in settings where the calibration error is relatively large. As illustrated in several calibration plots, KSP also appears to capture tail probabilities better. This may be attributed to its objective of minimizing the maximum discrepancy, which encourages uniformly improved calibration across the entire range of probabilities, regardless of the interval. To gain a clearer understanding of these behaviors, it may be valuable to conduct additional controlled experiments on synthetic data that allow us to systematically vary key factors such as the degree of censoring or the presence of ties. We will further elaborate on this issue in the conclusion.\n\n$\\textsf{Q2}$. The suggested method overcomes limitations of earlier methods due to computational complexity and discretization. It is, however, unclear how those earlier methods would perform, in principle, if we could have (close to) unlimited compute and the binning can be coarsened arbitrarily. Are the reported gains primarily the result of computational improvements and the continuous formulation or is there more?\n\nWe believe there are three primary reasons for the superiority of KSP.\n\nOne of the main advantages of KSP lies in its continuous formulation, which allows it to achieve calibration without relying on predefined intervals. This formulation makes it fundamentally different from binning-based approaches and enables more consistent calibration behavior across the probability space.\n\nIn addition, KSP offers significantly lower computational cost, especially on large-scale datasets. Even with unlimited computing resources, KSP remains far more efficient due to its formulation, which avoids costly sampling or discretization procedures. By approximating the supremum of the calibration error, KSP also has the potential to promote uniform calibration across the entire probability range.\n\nAnother important strength of KSP lies in its explicit optimization of the calibration error. This makes the method more interpretable and accessible for practitioners, as it directly aligns with the calibration objective. In contrast, conformal prediction-based approaches rely on implicit calibration driven by theoretical guarantees under assumptions such as exchangeability. When these assumptions do not hold in practice, the reliability of the calibration guarantee may deteriorate.\n\n$\\textsf{Q3}$. I think the part of the paper leading up to Theorem 3.1 isn't clear to me, which probably makes me underappreciate the claim in Theorem 3.1. For one, I don't readily see that \"Eqn. (3) is equivalent to (...)\". Additionally, the text states, \"We begin with the simplified setting without covariates,\" yet the general result still applies. The paragraph at line 133 offers some observations and links to other methods, but it is unclear to me what I should take away. In the actual formulation of the theorem, (and maybe this is a silly question but) what does it mean for \"calibration to hold\"? Ultimately, the theorem is an asymptotic result and doesn't necessarily say anything about the finite sample behavior of KS-cal.\n\nSorry for the confusion. The Eqn (2) is the sample mean where each sample has the same distribution as that of the random quantity in the bracket of Eqn. (3). Therefore, the expectation of (2) is equal to that of the random quantity in the bracket of Eqn. (3) when given $x=s.$ Also, on line 133, we merely describe the shape of $\\tilde{F}(x)$ having jumps on uncensored points and linearly interpolated on censored points. These mixture types are the main characteristic of $\\tilde{F}(x),$ not depending on fixed bins.\n\nFurthermore, we'll provide more detailed explanations for the Theorem. Due to the limitations of space on paper, we have merged two theorems, which may cause some confusion. We state cases with/without covariates. The first is the case of non-covariate (i.e., independent and identically distributed, or iid) cases, and the second is the case of having covariates (i.e., independent cases). Without covariates, the $ \\mathbb{E}_{Y, \\delta } [ \\tilde{F}(x) ]$ following the c.d.f. of uniform distribution implies the calibration. When considering the covariates, calibration is defined as follows: (3) with an expectation over $(Y, \\delta, {\\boldsymbol z}).$ The theorem states that the KS-cal goes to 0 if and only if the calibration holds. If the space limit of the paper is relaxed, we can state two theorems. The calibration with/without covariates has relatively common and different definitions.\n\nIn addition, the finite sample behavior can be stated by letting $\\epsilon$ be a sequence depending on $N$ since we used a large deviation such as Bernstein's inequality in the proof. It reveals the finite behavior. More rigorously, with probability $ 1 - \\delta$, we can have $a_N$ such that  \n\\begin{eqnarray*}\n\\vert \\tilde{F}(x) - x \\vert < a_N / \\sqrt{N}  \n\\end{eqnarray*}\nwhere $a_N = M ( \\log N )^{1/2}.$ Here, $M>0$ and $\\delta>0$ are large and small, respectively.\n\n$\\\\textsf{Q4}$. There are some statements that sound appealing, but remain unsubstantiated, as far as I can see. Some examples:\n\n$\\\\bullet$ Abstract: ``Existing approaches (...) often rely on heuristic binning or nonparametric estimators, which undermine their adaptability to continuous-time settings and complex model outputs.\" (Besides, KS is also a nonparametric estimator.)\n\nThank you for the comment. We agree that KS itself is a nonparametric estimator. Our point was to highlight that KM-based methods rely on an additional nonparametric estimator, namely the KM estimator, as part of their calibration procedure. Unlike our approach, which evaluates the discrepancy between $\\tilde{F}(x)$ and the theoretical target $x$, KM-based methods assess calibration using the KM estimator. This reliance can introduce additional variance or instability in the calibration process. We will revise the sentence to clarify this distinction more explicitly.\n\n$\\\\bullet$ Page 1: ``but their reliance on fixed sampling schemes or predefined percentiles may limit adaptability.\"\n\nIn contrast to KSP, prior methods are designed based on predefined quantities such as the number of quantiles, which limits their flexibility and adaptability. We intended to emphasize this distinction, and we will make an effort to improve the clarity of this point in the revised version.\n\n$\\\\bullet$ Page 9: ``KSP offers a theoretically grounded framework that avoids the limitations of bin-based or sampling-dependent approaches.\"\n\nWe acknowledge that other methods also have solid theoretical foundations. We intended to highlight that KSP avoids the need for additional components, such as binning in X-cal or KM sampling in CSD. We apologize that the original phrasing may have caused confusion, and we will revise the sentence to reflect our intended meaning better.\n\n$\\\\textsf{Q5}$. I find related work sections ``after the fact\" not very helpful. \n\nIn the “Related Work” section, our intention was to highlight notable general studies on evaluating survival predictions, as well as specific works on in-processing and post-processing methods for survival prediction. We agree with the reviewer that some of these references may not be directly relevant to our current work. As suggested, we will move the “Related Work” section to an earlier part of the paper and remove those references that are not closely related to the present study.\n\n$\\\\textsf{Q6}$. A critical discussion in which the pros and cons of this new method in the context of earlier methods and results is lacking. In addition, none of the related works are discussed in the new context provided by the proposed method.\n\nThe merit of our algorithm lies in its use of a continuous metric, such as KS, which offers a stronger foundation than bin-wise approaches. Despite its theoretical rigor, the computational burden remains low, and empirical results demonstrate clear improvements in calibration. One limitation is that the post-processing transformation may have a somewhat lower representational capacity. However, in practice, our results consistently validate the effectiveness of our method, suggesting potential for further enhancement in future work. We also apologize for the earlier confusion in articulating the main motivation behind KSP. It is indeed inspired by the works of [1] and [2], and we will revise the manuscript to highlight this point.\n\n[1] Fernández, T., Gretton, A.. A maximum-mean-discrepancy goodness-of-fit test for censored data.\n\n[2] Gupta, K. et al. Calibration of Neural Networks using Splines."}, {"review_id": "LTzbE60sQe", "reviewer": "Reviewer_cvLJ", "summary": "This paper proposes a post-processing method to improve calibration in survival models. Existing approaches often rely on heuristic binning, nonparametric estimators, or incur significant computational cost. Inspired by Platt scaling, the proposed method introduces a transformation function optimized via backpropagation to minimize a Kolmogorov-Smirnov (KS) version of D-calibration. Empirical results demonstrate that the proposed method outperforms existing techniques in terms of both calibration performance and computational efficiency.", "questions": "1. Why is Park et al.’s in-processing method not included in the comparison? \n2. What is the purpose of the left panel in Figure 2, which aggregates performance across all datasets and models? It shows little difference among the four methods, including the uncalibrated one. Does this imply that in most settings, the original models are already well-calibrated, and post-processing yields only marginal improvements?\n3. The grid used for tuning the regularization parameter is large: {1, 10, 100, 1000}. With such strong regularization, wouldn’t the parameters shrink toward 0s, effectively yielding little adjustment from the original CDF? Again, this raises the question of whether the original models are already sufficiently calibrated.", "limitations": "No limitation is discussed in the paper.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The motivation is clearly presented.\n2. The empirical section is extensive and includes useful ablation studies.\n3. The paper is generally well-written and accessible.\n\n\n\nWeaknesses:\n1. The paper claims that the KS test has been underutilized in survival calibration, which is misleading. Numerous adaptations of the KS test for right-censored data already exist and are covered in standard survival analysis textbook and literature [1–3]. Furthermore, the proposed use of a KS-based loss function resembles techniques based on Cox–Snell residuals and their calibration assessments [4, 5]. A more thorough discussion of these related methods and their differences from the proposed approach is essential.\n2. While the motivation and background are well-explained, the main methodological section (Section 4) is surprisingly brief. Although simplicity can be a virtue, the proposed method currently appears heuristic and lacks theoretical depth. Several technical questions remain unanswered:\n   * Can the transformation function in line 5 of the algorithm approximate arbitrary survival functions (i.e., is it identifiable)?\n   * Is the objective function (when combined with the KS loss) convex or well-behaved during optimization?\n   * Why are exactly three trainable parameters sufficient, and under what conditions? Table 25 shows that 3-parameter is better than 0/1/2, but the justification needs to be made more rigorous for 4+ parameters.\n   * Is there any formal guarantee on calibration, as provided in recent conformal-based approaches?\n\n3. The evaluation uses 20 bins, while the competing methods (e.g., CSD and CSD-iPOT) were originally designed or evaluated using 10 bins. This discrepancy could influence the relative performance. A fair comparison using consistent bin counts is necessary.\n\n4. Proposition 4.1 assumes the non-crossing property of survival curves, which limits its applicability. The statement that “this condition is satisfied by many commonly used models” is inaccurate. In practice, only CoxPH and WeibullAFT (and their NN-extensions) satisfy this condition. This limitation should be discussed more.\n\n5. Figures 3 and 15–24 use histograms to compare calibration performance, but these are difficult to interpret and obscure the actual differences between methods. For instance, in Figure 3’s leftmost panel, the CSD method appears slightly better than the proposed one. A QQ plot might better illustrate deviations from perfect calibration. Also, the small size of panels in Figure 4 makes error bars and trends hard to see.\n\n\n6. The transformation function is described as “monotonic,” but this should be refined to “strictly monotonic.” A merely monotonic function may contain flat regions, leading to tied CDF values and potentially compromising evaluation metrics such as the time-dependent concordance index. Since all functions evaluated are strictly monotonic, this is a minor but important clarification.\n\n[1] Chapter 7, Survival Analysis Techniques for Censored and Truncated Data. Klein and Moeschberger\n\n[2] Modified Kolmogorov-Smirnov Test Procedures with Application to Arbitrarily Right-Censored Data. Biometrics, 1980\n\n[3] Two-Sample Tests of Cramér--von Mises- and Kolmogorov--Smirnov-Type for Randomly Censored Data. International Statistical Review. 1984.\n\n[4] An evaluation of the Cox-Snell residuals. Elin Ansin. Master Thesis\n\n[5] Randomized Survival Probability Residual for Assessing Parametric Survival Models. Tingxuan Wu. Master Thesis", "was_revised": true, "final_justification": "Thanks to the author for their rebuttal. My concerns are mostly resolved.", "rebuttal": "We thank the reviewer for their thoughtful and constructive comments. We respond to each point below and will make all necessary clarifications and corrections in the final version.\n\n$\\\\textsf{Q1}$. The paper claims that the KS test has been underutilized in survival calibration, which is misleading. Numerous adaptations of the KS test for right-censored data already exist and are covered in standard survival analysis textbook and literature [1–3]. Furthermore, the proposed use of a KS-based loss function resembles techniques based on Cox–Snell residuals and their calibration assessments [4, 5]. A more thorough discussion of these related methods and their differences from the proposed approach is essential.\n\nThank you for the insightful comment and for providing the five references. When we described the KS test as underutilized, what we meant was that in the context of survival analysis, there have been no existing methods specifically designed to reduce calibration error based on the KS metric. We recognize that our original wording may have caused confusion, and we apologize for any inconvenience this may have caused.\n\nIn fact, we have previously used Cox-Snell residuals in our experiments. However, since Cox-Snell residuals are based on the cumulative hazard function, they are often less straightforward to use than CDFs. In addition, for censored observations, Cox-Snell residuals typically rely on surrogate estimates such as $\\ln{2}$, which motivated us to explore alternative formulations. We appreciate the references provided and will ensure that we cite and discuss them in our revised manuscript.\n\n$\\\\textsf{Q2}$. While the motivation and background are well-explained, the main methodological section (Section 4) is surprisingly brief. Although simplicity can be a virtue, the proposed method currently appears heuristic and lacks theoretical depth. Several technical questions remain unanswered:\n\n$\\\\bullet$ Can the transformation function in line 5 of the algorithm approximate arbitrary survival functions (i.e., is it identifiable)?\n\nKSP is a one-to-one transformation and thus identifiable, which gives it a clear structural advantage. While we acknowledge that this is still an early stage of research, the current formulation of KSP has demonstrated strong and consistent performance across various models. We believe this formulation offers a solid foundation, and we are optimistic about its ability to approximate a wide range of survival functions. Nevertheless, we recognize that further theoretical and empirical exploration will help us better understand its capacity and limitations.\n\n$\\\\bullet$ Is the objective function (when combined with the KS loss) convex or well-behaved during optimization?\n\nAlthough the KS loss is not convex, its behavior is mild under gradient-based optimization, and it converges reliably in practice. Compared to X-cal, KS loss is slightly faster and does not require a surrogate loss function, which simplifies implementation. One important consideration, however, is that using a very small batch size can lead to noticeable approximation error. Therefore, selecting an appropriate batch size is crucial for achieving stable performance.\n\n$\\\\bullet$ Why are exactly three trainable parameters sufficient, and under what conditions? Table 25 shows that 3-parameter is better than 0/1/2, but the justification needs to be made more rigorous for 4+ parameters.\n\nThanks for your comment. It is indeed possible to introduce more than four parameters while preserving the time-dependent C-index. However, based on our preliminary experiments using deep neural networks having numerous parameters, we observed a tendency toward overfitting, indicating that careful design and regularization are needed. We will include additional experimental results related to this observation in the revised version.\n\n$\\\\bullet$ Is there any formal guarantee on calibration, as provided in recent conformal-based approaches?  \n\nYes. While we did not emphasize this in the paper, by letting $\\epsilon$ be a sequence that depends on $N$, we can infer the convergence rate from the proofs in the appendix. It shows that $\\tilde{F}(x)$ converges at the rate of $1 / \\sqrt{N}$ up to $(\\log N)^{1/2}.$ We agree that highlighting this point in the paper would strengthen the presentation, and we will revise the manuscript accordingly.\n\n$\\\\textsf{Q3}$. The evaluation uses 20 bins, while the competing methods (e.g., CSD and CSD-iPOT) were originally designed or evaluated using 10 bins. This discrepancy could influence the relative performance. A fair comparison using consistent bin counts is necessary.\n\nTo evaluate the metrics that depend on bin locations, we used 10 equally sized bins: $[0, 0.1], [0.1, 0.2], \\ldots, [0.9, 1.0]$. The table below reports only the D-calibration metrics, following the format of Table 1 in the main text, where the number of wins for each method is counted. Under this 10-bin setting, somewhat surprisingly, KSP still outperforms CSD in most cases, although with a reduced number of wins. It is worth noting that, with 10 bins, the bin boundaries exactly align with the quantile locations used during inference. This alignment tends to favor quantile-based methods such as CSD and may not provide an equally favorable setting for other methods.\n\nA truly well-calibrated method should maintain consistent performance regardless of the specific binning scheme. Therefore, using bins that do not align with a model’s inference quantiles can provide a fairer basis for comparison. Since the X-cal paper adopted 20 bins of width 0.05, we followed the same configuration in our main experiments.\n\n$\\\\textsf{Table 1. Summary of pairwise comparisons between post-processing methods.}$\n\n\\\\begin{array}\n\\\\hline\n\\\\textsf{Method} & \\\\textsf{S-cal(10)} & \\\\textsf{D-cal(10)} & \\\\textsf{S-cal(20)} & \\\\textsf{D-cal(20)} & \\\\textsf{KS-cal} \\\\\\\\\n\\\\hline\n\\\\textsf{KSP} & \\\\textsf{48 (45)} & \\\\textsf{46 (43)} & \\\\textsf{46 (45)} & \\\\textsf{46 (43)} & \\\\textsf{47 (45)} \\\\\\\\\n\\\\textsf{Non-calibrated} & 12 (8) & 14 (11) & 13 (7) & 14 (6) & 13 (5) \\\\\\\\\n\\\\textsf{Tie} & 0 & 0 & 1 & 0 & 0 \\\\\\\\\n\\\\hline\n\\\\textsf{KSP} & \\\\textsf{36 (28)} & \\\\textsf{32 (29)} & \\\\textsf{36 (29)} & \\\\textsf{48 (45)} & \\\\textsf{51 (42)} \\\\\\\\\n\\\\textsf{CSD} & 24 (19) & 28 (26) & 24 (19) & 12 (10) & 9 (8) \\\\\\\\\n\\\\textsf{Tie} & 0 & 0 & 0 & 0 & 0 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n$\\\\textsf{Q4}$. Proposition 4.1 assumes the non-crossing property of survival curves, which limits its applicability. The statement that “this condition is satisfied by many commonly used models” is inaccurate. In practice, only CoxPH and WeibullAFT (and their NN-extensions) satisfy this condition. This limitation should be discussed more.\n\nWe acknowledge that the statement may have been overstated. Among the models we used, only two strictly satisfy the required condition. However, as shown in our experimental results, the time-independent C-index is nearly preserved even for the other models. Regardless of the discrimination measure used, KSP does not significantly degrade discrimination power. We agree that this point deserves further discussion and will revise the corresponding section to reflect this nuance better.\n\n$\\\\textsf{Q5}$. Figures 3 and 15–24 use histograms to compare calibration performance, but these are difficult to interpret and obscure the actual differences between methods. For instance, in Figure 3’s leftmost panel, the CSD method appears slightly better than the proposed one. A QQ plot might better illustrate deviations from perfect calibration. Also, the small size of panels in Figure 4 makes error bars and trends hard to see.\n\nThank you for the suggestion. We will revise the figure accordingly.\n\n$\\\\textsf{Q6}$. The transformation function is described as “monotonic,” but this should be refined to “strictly monotonic.” \n\nThank you for your constructive comments. We will revise the wording to ``strictly monotonic\" for clarity and accuracy.\n\n$\\\\textsf{Q7}$. Why is Park et al.’s in-processing method not included in the comparison?\n\nWhen comparing in-processing methods, we selected representative approaches from each line of work. The method proposed by Park et al. is a variant of X-cal. Therefore, we did not include it separately. While their method may outperform X-cal in some settings, the fundamental trade-off between discrimination and calibration inherent to in-processing approaches still applies. Therefore, we did not consider it essential to include this method in our comparison.\n\n$\\\\textsf{Q8}$. What is the purpose of the left panel in Figure 2, which aggregates performance across all datasets and models? It shows little difference among the four methods, including the uncalibrated one. Does this imply that in most settings, the original models are already well-calibrated, and post-processing yields only marginal improvements?\n\nSince the figure integrates results from all 60 cases, some overlapping occurs in the bar plot. This is because both low and high calibration error scenarios are included. For models that already exhibit low calibration error, post-processing tends to make minimal difference. However, for models with higher calibration error, such as CRPS, the reduction is clearly substantial. What we aimed to highlight is the median value. KSP consistently reduces calibration error to near-best levels across all cases. We included this figure to emphasize that point, and we will revise the plot to improve clarity and interpretation.\n\n$\\\\textsf{Q9}$. The grid used for tuning the regularization parameter is large: {1, 10, 100, 1000}.\n\nUsing a regularization parameter up to 1000 did not pose a significant issue in our experience. Even with $\\lambda=1000$, the models trained properly, although they may require more training epochs. We followed a similar range as reported in the X-cal paper, which also experimented with values of $\\lambda$ up to 1000."}, {"review_id": "4lq9VdadQB", "reviewer": "Reviewer_vgeG", "summary": "The authors propose a post-processing method (KS metric-based post-processing, or KSP for short) based on the Kolmogorov-Smirnov statistic for improving calibration in survival models. Similar to Platt scaling for calibrated classification, KSP applies a linear transformation to the logits of the predicted survival time CDF with an exponential adjustment to the transformed CDF. The three hyperparameters specifying this transformating can be chosen via gradient descent to minimize an empirical version of the KS metric on a validation set. The authors then conduct extensive experiments across 6 base models and 10 datasets, comparing KSP with other post-processing and in-processing methods for survival calibration. They find that KSP improves competitor methods in most scenarios without sacrificing discriminative performance, as measured by the C-index.", "questions": "Are there some situations where KSP provides a larger absolute improvement in calibration as compared to the baselines?\n\nWhat is a typical value of B (i.e., number of iterations for KSP to converge)?\n\nHow does KSP compare to training a survival with a proper scoring rule?\n\nWhy should the KM estimator be considered an empirical lower bound on calibration error (lines 223-224)?", "limitations": "Some limitations are discussed, mainly in the Experiments section (Section 5).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\nThe paper is generally well-written and easy to follow.\n\nThe topic is relevant to the NeurIPS community and the contribution is solid. As the authors discuss, uncertainty quantification is important for the safe and effective application of ML techniques in high-stakes scenarios like healthcare, a domain in which survival analysis is particularly relevant. As the proposed method can be combined with any survival model which estimates a survival function, it is widely applicable.\n\nThe experimental validation is very extensive. The authors considered 60 total test scenarios (6 base models x 10 datasets) which cover a variety of model types and dataset sizes. The results are generally in favor of their method, and most of the time they obtain statistically significant improvements across a variety of calibration metrics as compared to the baseline methods. In scenarios where KSP underperforms, they give insightful discussion as to why this may be the case.\n\nIn addition to improved calibration, the method may also be less computationally expensive than other competitor methods for survival calibration.\n\nI would also like to add that I reviewed an earlier version of this paper, and it is clear that the authors have made a substantial effort to address the weaknesses in the previous version.\n\n---\n# Weaknesses\n\nWhile KSP does frequently rank first in terms of the various calibration metrics as compared to the baselines, the absolute improvement is frequently not very large. For instance, in Table 12, the improvement of KSP over the second-best baseline is usually on the order of 10^-3 or 10^-4. Thus, the significance of the contribution (in the practical sense, not the statistical sense) may be somewhat limited.\n\nThere is a line of related work on proper scoring rules for survival analysis which is only briefly referenced in the paper (Yanigasawa, 2023 in the Related work section). The relationship is that proper scoring rules are loss functions which are minimized only by the ground truth survival function (at the population level), so implicitly the use of such a loss function will lead to calibration. It would strengthen the paper to include a model trained with such a loss function as one of the \"uncalibrated\" baselines to see how much gain is to be had from explicit calibration techniques.", "was_revised": true, "final_justification": "I appreciate the authors' attempts to address my concerns, and the additional experiments were helpful. Unfortunately, it seems that the results on models trained using proper scoring rules is quite mixed. The positive correlation between the number of iterations B and the calibration error is also odd; one would hope that using more computational resources on the method would improve performance, but the opposite effect is observed. I also appreciate the clarification on the reason for considering the KM estimator as a lower bound on the calibration, but this actually raises another question, namely, why would one not use the KM estimator instead of any of the proposed methods if it is considered optimal?\n\nOverall, it seems that KSP can indeed improve the calibration of pretrained survival models with little additional overhead, meaning that it may be useful in some settings, but I will refrain from enthusiastic acceptance due to the issues mentioned above.", "rebuttal": "We thank the reviewer for their thoughtful and constructive comments. We respond to each point below and will make all necessary clarifications and corrections in the final version.\n\n$\\\\textsf{Q1}$. While KSP does frequently rank first in terms of the various calibration metrics as compared to the baselines, the absolute improvement is frequently not very large. For instance, in Table 12, the improvement of KSP over the second-best baseline is usually on the order of $10^{-3}$ or $10^{-4}$. Thus, the significance of the contribution (in the practical sense, not the statistical sense) may be somewhat limited.\n\nThank you for the thoughtful comment. While it is true that the absolute magnitude of improvement in calibration error may appear small (e.g., on the order of $10^{-3}$ or $10^{-4}$), we believe these differences can still be meaningful for several reasons.\n\nFirst, calibration error metrics are typically bounded between 0 and 1, and in well-trained models, absolute errors tend to be quite small. As shown in Table 12 of the main text, the CRPS model, when calibrated using KSP, achieves more than an 80\\% reduction in S-cal(20) and a 43\\% reduction in KS-cal, compared to calibration with CSD-iPOT. While the absolute improvements may seem modest, they are substantial in relative terms and reflect meaningful gains. Even modest improvements — especially in the low-error regime — can signify a meaningful reduction in systematic miscalibration. In safety-critical applications such as clinical risk prediction or reliability analysis, small but consistent calibration gains can be translated into increased trust in model decisions. Although improvements may appear modest for models with an already low calibration error, we believe that this is primarily a matter of scale.\n\nSecond, as seen in our experiments across diverse datasets and models, KSP consistently ranks among the top performers and often achieves the best results across multiple metrics. We believe this consistency, rather than a few large gains in specific cases, is a key strength of KSP.\n\nLastly, we emphasize that the KSP achieves these improvements with minimal computational overhead and without relying on binning or sampling. This practical efficiency further supports the utility of our method, even if the absolute gains may sometimes appear modest. Nonetheless, we agree that future work could further investigate settings in which calibration improvements lead to tangible benefits in downstream decision-making.\n\n$\\\\textsf{Q2}$. There is a line of related work on proper scoring rules for survival analysis which is only briefly referenced in the paper (Yanagisawa, 2023 in the Related work section). The relationship is that proper scoring rules are loss functions which are minimized only by the ground truth survival function (at the population level), so implicitly the use of such a loss function will lead to calibration. It would strengthen the paper to include a model trained with such a loss function as one of the ``uncalibrated\" baselines to see how much gain is to be had from explicit calibration techniques.\n\nWe appreciate your suggestion. Both the use of proper scoring rule-based loss functions and our proposed post-processing method share the common goal of approximating the true survival function. Therefore, we agree that combining them could potentially yield a synergistic effect.\n\nWe are currently conducting experiments in this direction. However, based on our findings, the models trained with proper scoring rule-based loss already exhibit low calibration error, so applying our post-processing method tends to offer only marginal gains. We will continue our experiments, and if we identify any cases where the post-processing method provides meaningful improvement, we will share the results during the rebuttal period.\n\n$\\\\textsf{Q3}$. Are there some situations where KSP provides a larger absolute improvement in calibration as compared to the baselines?\n\nThis observation is especially evident in the CRPS, which tends to show the largest calibration errors across most scenarios. As it is not based on a discretized structure, KSP’s explicit focus on minimizing calibration error appears to be particularly effective in this case.\n\n$\\\\textsf{Q4}$. What is a typical value of B (i.e., number of iterations for KSP to converge)?\n\nThe table below shows the number of iterations for each dataset and model averaged by 30 repetitions.\n\n$\\\\textsf{Table 1. Number of B iterations for each datasets and models using KSP.}$\n\n\\\\begin{array}\n\\\\hline\n\\\\textsf{Dataset} & \\\\textsf{DeepSurv} & \\\\textsf{MTLR} & \\\\textsf{Parametric} & \\\\textsf{CRPS} & \\\\textsf{DeepHit} & \\\\textsf{AFT} \\\\\\\\\n\\\\hline \n\\\\textsf{WHAS} & 329 & 300 & 989 & 409 & 188 & 426 \\\\\\\\\n\\\\textsf{METABRIC} & 338 & 488 & 523 & 378 & 423 & 599 \\\\\\\\\n\\\\textsf{GBSG} & 388 & 673 & 468 & 528 & 606 & 508 \\\\\\\\\n\\\\textsf{NACD} & 640 & 646 & 389 & 612 & 383 & 378 \\\\\\\\\n\\\\textsf{NB-SEQ} & 272 & 269 & 685 & 958 & 330 & 1087 \\\\\\\\\n\\\\textsf{SUPPORT} & 306 & 635 & 1107 & 609 & 649 & 314 \\\\\\\\\n\\\\textsf{MIMIC-III} & 1310 & 1310 & 698 & 1352 & 395 & 1660 \\\\\\\\\n\\\\textsf{SEER-liver} & 517 & 161 & 1288 & 1484 & 131 & 1362 \\\\\\\\\n\\\\textsf{SEER-stomach} & 431 & 169 & 1317 & 1578 & 184 & 1249 \\\\\\\\\n\\\\textsf{SEER-lung} & 430 & 114 & 1231 & 1199 & 112 & 1344 \\\\\\\\\n\\\\hline\n\\\\end{array}\n\nAs discussed in the paper, we observed a tendency for the calibration error to increase as $B$ increases. When computing the Spearman rank correlation between $B$ and KS-cal, the result was 0.234.\n\n$\\\\textsf{Q5}$. Why should the KM estimator be considered an empirical lower bound on calibration error (lines 223-224)?\n\nAs claimed in the CSD and CSD-iPOT papers, the KM estimator is considered an optimally calibrated estimator. Although it does not always represent a lower bound, we believe it serves as a suitable reference point. We acknowledge that we only mentioned this without explicitly citing it, and we will add the corresponding references."}]

Author rebuttal:

Meta-review: This paper proposes a post-hoc calibration approach for survival analysis based on the Kolmogorov-Smirnov metric. This proposed approach is well-motivated and theoretically sound, and the experiments were extensive. During the course of the author/reviewer discussion, the authors addressed the weaknesses well with helpful clarifications and additional experiments. Of the 4 reviewers, 3 ended up favoring acceptance (1 accept, 2 borderline accept), and a dissenting reviewer voted for borderline rejection (note however that the dissenting reviewer stopped being responsive and never acknowledged the author rebuttal; I found that their concerns were sufficiently addressed by the authors). Thus, I am recommending acceptance for this paper.
Decision: Accept (poster)