Toward a Well-Calibrated Discrimination via Survival Outcome-Aware Contrastive Learning

Authors: dongjoon  lee
hyeryn  park
changhee  lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:31:13

Abstract: Previous deep learning approaches for survival analysis have primarily relied on ranking losses to improve discrimination performance, which often comes at the expense of calibration performance. 
To address such an issue, we propose a novel contrastive learning approach specifically designed to enhance discrimination without sacrificing calibration. 
Our method employs weighted sampling within a contrastive learning framework, assigning lower penalties to samples with similar survival outcomes. This aligns well with the assumption that patients with similar event times share similar clinical statuses. Consequently, when augmented with the commonly used negative log-likelihood loss, our approach significantly improves discrimination performance without directly manipulating the model outputs, thereby achieving better calibration.
Experiments on multiple real-world clinical datasets demonstrate that our method outperforms state-of-the-art deep survival models in both discrimination and calibration. Through comprehensive ablation studies, we further validate the effectiveness of our approach through quantitative and qualitative analyses.

Main paper: 
 introduction: 
 Deep learning models have gained significant attention in survival analysis (also known as time-toevent analysis), which focuses on predicting and understanding the occurrence of an adverse event (e.g., death) as a function of time. The utility of survival models is typically evaluated through two key aspects: discrimination and calibration. Discrimination measures a model's ability to differentiate between patients with varying risks, prioritizing those more likely to experience the event. Calibration, on the other hand, assesses how well a model's predictions align with the observed event distribution. In other words, a well-calibrated model offers predictions that closely match actual survival outcomes, offering crucial prognostic value to clinicians.In an effort to enhance the discriminative power of survival models, ranking loss functions [1, 2, 3, 4] are frequently employed. These functions aim to maximize a relaxed proxy of the concordance index (C-index), a well-established metric for evaluating the quality of patient rankings based on the risk predictions of survival models [5] . However, a notable improvement in discriminative power often comes at the expense of calibration performance, which can negatively affect the clinical utility of predicted survival outcomes. For example, a poorly calibrated model that overestimates risk may lead to unnecessary treatment or testing for patients. Conversely, underestimating risk may result in patients with adverse conditions not receiving appropriate care.Contrastive learning [6, 7, 8] is a framework that aims to learn an embedding space where similar samples are mapped to nearby locations, while dissimilar samples are pushed farther apart. Notably, although commonly used in an unsupervised fashion, this framework shares a fundamental principle explicitly incorporate a calibration loss function. Nevertheless, it achieves comparable or even superior calibration performance, demonstrating the effectiveness of our contrastive learning framework in implicitly preserving calibration alongside discrimination. 
contrastive learning: 
 Contrastive learning [6, 7, 17] is a framework that learns an embedding space that effectively discriminates among samples, by mapping positive pairs (similar samples) closer together, while pushing negative pairs (dissimilar samples) farther apart. While contrastive learning has made significant progress, our review of related work will focus on methods that explore various strategies for injecting inductive bias to enhance representation learning.Exploiting Continuous Labels. Inspired by the success of incorporating label information into contrastive learning for classification tasks [8] , recent research has extended this concept to regression tasks by exploiting continuous label information for constructing positive and negative pairs. Zha et al. [18] employ hard thresholding on label differences to generate positive and negative samples, capitalizing on the idea that samples with similar target values should be mapped closer in the latent space. Kerdabadi et al. [19] apply a similar approach in a dynamic time-to-event setting, defining positive pairs as those within a specific time window and weighting negative pairs proportionally to their time difference. However, this weighting scheme's heavy reliance on absolute time differences can destabilize the contrastive loss during training.Weighting Negative Samples. Recent works have significantly improved contrastive learning by incorporating inductive bias in the selection of negative samples. This includes focusing more weights on hard negatives that challenge the model for more effective discrimination [20, 21, 22, 23] and avoiding potential false negatives that might mislead the model [24, 25] . Many of these studies address negative sample selection by employing informed sampling techniques to identify informative hard negatives or potential false negatives. Notably, Robinson et al. [20] leverage importance sampling by proposing a similarity-based distribution that prioritizes hard-to-distinguish negative samples.3 Problem Formulation 
preliminary: discrete-time survival analysis: 
 Suppose we are given a discrete-time survival dataset comprising N patients, denoted as D = {(x i , τ i , δ i )} N i=1 . Each patient i is represented by the input feature x i ∈ X where X is the input space, and the observed survival outcomes, τ i ∈ T and δ i ∈ {0, 1}. Here, τ i and δ i indicate the time elapsed until either an event of our interest (e.g., death) or censoring (e.g., lost to follow-up) occurs and whether the event was observed or not (i.e., right-censored), respectively. Throughout, we treat survival time as discrete and the time horizon as finite such that the set of possible survival times is defined as T = {0, . . . , T max } with a pre-defined maximum time horizon T max .The conditional hazard function, λ : X × T → [0, 1], is the instantaneous risk of the event at time t given feature x and is defined as λ(t|x) = P(T = t|T ≥ t, x). Then, we can represent the survival function S : X × T → [0, 1] as follows:EQUATIONwhich is a non-increasing function of t, indicating the probability of the event occurring after time t given feature x. Equivalently, we could estimate the risk function, R : X × T → [0, 1], which represents the probability of the event occurring at or before time t given feature x, i.e., R(t|x) = P(T ≤ t|x) = 1 -S(t|x).Then, we can achieve likelihood-based estimates for the hazard function, λ, by minimizing the following negative NLL loss:EQUATIONwhere p(t|x) = λ(t|x) Ŝ(t -1|x) represents the estimate for the probability of an event occurring at time t, i.e., P(T = t|x). Here, (2) exploits two pieces of information from the survival data: (i) when the event is observed (i.e., δ i = 1), we know that the event occurred at time τ i , and (ii) when the event is not observed (i.e., δ i = 0), we are aware that the event will occur after time τ i .Ranking loss specifically targets enhancing the discriminative power of survival models, which is crucial for better distinguishing between patients based on their risk. Suppose we are given the risk function, R, that quantifies the risk of a given patient with x at any time t. We aim to penalize instances where the risks assigned to a pair of patients are incorrectly ordered (i.e., assigning a lower risk to patient i who died before patient j). This can be achieved by the ranking loss, which is formally given as follows:EQUATIONwhere η is a function that relaxes the non-differentiable indicator function, I. Here, A i,j = I(δ i = 1, τ i < τ j ) indicates acceptable pairs whose assigned risks are comparable.Different deep survival models have employed various types of quantities for implementing the ranking loss. For example, Lee et al. [1] have applied the risk function, R, and set η(x, y) = exp(-(x -y)/κ), a convex function that both penalizes the wrongly ordered pairs and encourages the correctly ordered pairs. The same convex function has been employed utilizing the survival function, S [2] . Similarly, Steck et al. [13] utilizes η(x, y) = log σ(y -x), which is a lower bound of the C-index, with the risk function, R. Chi et al. [3] employ the hazard function h with a linear ranking function η(x, y) = -(x -y).Challenges. While combining NLL with ranking loss shows a notable improvement in discriminative power, it often comes at the expense of calibration performance, potentially harming the clinical utility of predicted survival outcomes. We suspect this is primarily due to how ranking loss directly modifies model outputs to order predicted risks, potentially leading to misalignment with the actual risk distribution. In this paper, we propose a different approach. Our method focuses on increasing discriminative power not by directly utilizing survival outcomes but by exploiting the embedding space through our novel contrastive learning framework. This approach preserves the calibration performance achieved by the NLL loss while enhancing discriminative capabilities. 
method: 
 To address the challenges described above, we propose a novel Contrastive learning approach to a deep Survival model, which we refer to as ConSurv. The proposed method consists of three key components as illustrated in Figure 1 :• Encoder (parameterized by θ), f θ : X → H, takes features x ∈ X as input and outputs latent representations, i.e., h = f θ (x). • Projection head (parameterized by ψ), g ψ : H → R d , maps latent representations h to the embedding space where contrastive learning is applied, i.e., z = g ψ (h). • Hazard network (parameterized by ϕ), f ϕ : H × T → [0, 1], predicts the hazard rate at each time point t ∈ T given the input latent representation h, i.e., λ(t|x) = f ϕ (h, t) = f ϕ (f θ (x), t).Motivated by the core concept of contrastive learning frameworks, we aim to differentiate each sample from other semantically different samples based on their survival outcomes. This allows us to overcome the limitation of ranking loss, which arises from the direct comparison of model outcomes in the form of risk/survival functions. Instead, we contrast samples in the latent space based on their survival outcomes and utilize the model outcome solely for the NLL loss to encourage better calibration.Our novel contrastive learning framework imposes lower penalties on potential false negative samples based on the similarities in their survival outcomes. More specifically, given an anchor sample, we define potential false negatives as samples with a small difference in the corresponding time-to-event. This inherently aligns with our inductive bias that patients with similar survival outcomes should share similar clinical status, which manifests through similar representations.For each anchor sample x ∼ p X , the noise-contrastive estimation (NCE) objective aims to learn mapping f = g ψ • f θ utilizing a positive sample with the same semantic meaning, i.e., x + ∼ p + , and negative samples with (supposedly) different semantic meanings, i.e., x -∼ q, as follows [20] :EQUATION)where M is the scaling term which is set to the batch size and s : X × X → [-1, 1] is the similarity score between two samples. Here, we use cosine similarity in the embedding space, i.e., s(x,x ′ ) = f (x) ⊤ f (x ′ ) ∥f (x)∥∥f (x ′ )∥ .For notational convenience, we omit the corresponding temperature ν and write e s(x,x + ) to denote e s(x,x + )/ν .The key aspect in (4) for obtaining embeddings with discriminative power lies in how we select negative samples that the anchor sample should be distinguished from. To reflect the differences in the time-to-events in the embedding space, we design a novel negative distribution, q, by utilizing the available information from survival outcomes.Importance Sampling using Survival Outcomes. To accurately distinguish patients based on their time-to-event outcomes, we fully utilize the time-to-event information for designing q based on the following inductive bias: similar patients are more likely to experience the event at similar time points than the ones who are not. Hence, given an anchor (x, τ ) and a negative sample (x -, τ -), we define the weight function as follows:EQUATIONwhere σ > 0 is a temperature coefficient. From this point forward, we will slightly abuse the notation and write w(x -; x) to denote w(τ -; τ ) to be consistent with the notation used for the negative distribution. This function is a variant of the Laplacian kernel, where the weight increases as the time difference increases. That is, we assign larger weights for samples with large differences in the time-to-events, and smaller weights for samples with small time differences. Utilizing the weight function in (5) , we can define the negative distribution, q, as:EQUATIONwhere Z is the normalizing constant. Then, using the importance sampling technique, we can approximate the expectation of the similarity score of negative samples drawn from the negative distribution in (6) as:EQUATIONwhere the normalizing constant for the empirical distribution can be given as Z = 1 M M j=1 w(x - j ; x). Overall, the survival outcome-aware NCE (SNCE) loss in (8) , mitigating the effect of potential false negatives that have similar survival outcomes, can be given as the following:EQUATIONHowever, in a survival dataset, there exist samples that are right-censored, offering partial information about the corresponding survival outcomes which indicates that the event will occur sometime after the censoring time. Consequently, not every pair in the survival dataset has comparable survival outcomes. Specifically, there are three different cases of sample pairs to consider:• Case 1: Both samples are uncensored (i.e., have observed events). We can directly compare the two time-to-events of any given pair. • Case 2: One is uncensored and the other is censored. Similar to the acceptable pairs in ranking loss, we can only compare a pair when sample i experiences an event while sample j is censored after that event time (implying no event has occurred by that time), i.e., τ i < τ j . • Case 3: Both samples are censored. No comparison is possible as we do not know when the actual events have occurred for both samples.Considering these cases, we redefine the weight function considering the right-censoring as w(τ j ; τ i ) = I i,j • w(τ j ; τ i ) where I i,j indicates the comparable cases as follows:I i,j = 1 if (δ i = 1, δ j = 1) ∨ ((δ i = 1, δ j = 0) ∧ (τ i < τ j ) ∧ (|τ i -τ j | ≥ α)) 0 otherwisewhere α is a hyperparameter that represents a margin to ensure a minimum time difference. Please see Appendix C. 3 for more details about the effect of α.In this subsection, we provide a detailed network description of ConSurv and outline the learning procedure using a discrete-time survival dataset, D = {(x i , τ i , δ i )} N i=1 . Negative Log-likelihood. Based on the encoder and the hazard network, we can provide the estimated hazard function given the input feature x at each time point t ∈ T as λ(t|x) = f ϕ (f θ (x), t). As our hazard estimate is defined as a function of time given an input feature, we can naturally model the time-varying effect of input features (thus, more complex relations) on risk/survival functions. Then, we can compute the NLL loss, L θ,ϕ NLL , in (2) by plugging in f ϕ (f θ (x), t) into p and Ŝ as follows:EQUATION)Contrastive Learning. To train the proposed method using our contrastive loss introduced in (8), we construct augmented samples based on the state-of-the-art contrastive learning framework specifically designed for tabular data [26] . For each sample in a batch of M samples, we construct a corrupted version of the original sample following the marginal corruption process. So, given x i as an anchor, we set the corrupted sample, i.e., xi , as positive and all the other corrupted samples, i.e., xj for j ̸ = i, as negative. By passing the original, positive, and negative samples through f = g ψ • f θ , we can compute our survival outcome-based contrastive learning loss function as defined in (8) .Overall, we can estimate the hazard function by training ConSurv with a loss function that combines the NLL loss and the SNCE loss as the following:EQUATION)where β is a balancing coefficient that trade-offs between the two loss terms; the effect of β is provided in C.3. Please find the pseudo-code of ConSurv in H.  
experiment: 
 In this section, we evaluate the performance of ConSurv and multiple survival models using several real-world clinical datasets. Further details about all the experiments are available in Appendix D.Datasets. We compare our proposed method and the benchmarks with the following four commonly used real-world clinical datasets: METABRIC, NWTCO, GBSG, FLCHAIN, SUPPORT, and SEER.For detailed descriptions of these datasets, please refer to D.1.Benchmarks. We compare ConSurv with six survival models that were selected based on their respective loss functions, which are critical for understanding trends in survival analysis performance.The evaluated models are based on i) partial log-likelihood including CoxPH [27] and DeepSurv [28] , ii) ranking loss including DeepHit [1] and DRSA [10] , and iii) calibration loss including DCS [2] and X-CAL [15] . Detailed methodological descriptions are provided in Appendix D.3.Performance Metric. We evaluate the discriminative performance of ConSurv and benchmarks using the integrated time-dependent C-index (CI) [29] across all time points, where higher values indicate better performance. For calibration, we utilize the integrated Brier score (IBS) [30] and distributional divergence for calibration (DDC) [2] , where lower values indicate better performance. Additionally, D-Calibration (D-CAL) [31] assesses calibration with results reported based on p-values; those exceeding 0.05 are noted as statistically significant. Comprehensive details on IBS, DDC, and D-CAL are provided in Appendix D.2. In Appendix C.1, we further provide time-dependent C-index [5] and time-dependent Brier score [30] at different time points.We report the CI, BS, DDC, and D-CAL of our proposed method and the benchmarks in Table 1 .The results in Table 1 demonstrate that our method significantly outperforms all the benchmarks in discriminating among individual risks at the evaluated all time points, consistently across the four real-world clinical datasets. ConSurv achieves such gains in the discriminative performance while providing the best or comparable calibration performance. Notably, as evidenced by the metrics IBS, DDC, and D-CAL, our method yields exceptional calibration performance. Contrarily, ranking loss-based models such as DeepHit and DRSA, which are designed primarily to enhance discriminative power, usually exhibit poor calibration. Even without being specifically designed through the loss function for calibration, our model outperforms benchmarks such as DCS and X-CAL, which incorporate loss functions used to enhance calibration power. This indicates that our approach inherently balances both discriminative and calibration power effectively.Ablation Study. We conduct an ablation study to better understand the contribution of different loss functions within our framework employing the following variants of ConSurv: (i) 1 highlights the performance trade-offs between discrimination and calibration power when different loss functions are utilized in survival models. Using only NLL loss typically leads to lower discriminative performance while maintaining reasonably good calibration. Augmenting a ranking loss to the NLL significantly enhances discriminative performance but decreases calibration. In contrast, our proposed SNCE loss significantly enhances the model's performance compared to (i) using NLL only and (iii) using NLL with ranking loss. This suggests that the SNCE loss not only boosts discriminative power but also preserves or even improves the model's calibration. Additionally, when compared to (ii) using NLL with InfoNCE, which pushes all negative pairs away equally, we confirm that ConSurv, which assigns weights to negative samples based on their time difference until the event, ensures better performance. L NLL only, (ii) L NLL & L NCE , (iii) L NLL & L Rank , and (iv) ConSurv (i.e., L NLL & L SNCE ). TableTo further demonstrate how survival outcome-aware contrastive learning promotes discrimination among samples in the latent space, we compare two-dimensional t-SNE visualizations of latent representations trained with L NLL only, L SNCE only and ConSurv (i.e., L NLL & L SNCE ) for the METABRIC dataset in Figure 2 . Here, we only display uncensored samples with event times for clarity. Figure 2 shows that when trained with L NLL alone, latent representations are mixed regardless of their corresponding event times. However, incorporating L SNCE into the loss function significantly improves the alignment of representations with event time information, enabling the predictor to better discriminate the risks associated with each sample. Notably, even when trained solely with L SNCE , latent representations exhibit a clear alignment with event times. This finding strongly suggests that our contrastive learning approach effectively encourages discrimination by capturing and reflecting the underlying event time information. (See Appendix C.2 for more datasets.)Figure 3 shows calibration plots comparing the calibration of ConSurv with the deep learning-based survival models. Here, we evaluate calibration by matching predicted cumulative densities to observed event frequencies at the quantiles of the predicted cumulative density. In these plots, the x=y line represents the ideal state where predicted probabilities perfectly match the observed outcomes. The evaluated survival models show different trends depending on the loss function used for training. The ranking-based models (i.e., DeepHit and DRSA), which directly manipulate the model outputs, experimentally demonstrated the lowest calibration power. Contrarily, the calibration-based models (i.e., X-CAL and DCS), designed to improve calibration power, show relatively better calibration but do not match the calibration performance of DeepSurv, likely due to the assumption of proportional hazards. Compared to these survival models trained based on three different types of loss functions, our proposed method demonstrates calibration performance that is similar or even superior.We further validate the calibration performance of the survival models by comparing their survival plots with the Kaplan-Meier (KM) curve, a non-parametric estimate of the survival function at a population level. For this comparison, we examine two binary hormone receptor statuses in the METABRIC dataset: estrogen receptor (ER) and human epidermal growth factor receptor 2 (HER2), which are crucial for determining hormone therapies and can result in significantly different prognoses for breast cancer patients. In Figure 4 , we average the survival functions of different survival models based on the subgroups of patients with the corresponding feature values (e.g., ER+ vs ER-). The results confirm that our method aligns well with the KM curve compared to other deep learning-based survival models, especially those with ranking loss.To quantitatively assess calibration performance across different subgroups, we compare the survival predictions of each model with KM curves for each subgroup using the Wasserstein distance. This metric is well-suited for comparing survival curves as it considers both the overall shape and the discrepancies in predicted survival probabilities at each time point. The results in Table 2 demonstrate that our proposed model achieves well-calibrated predictions within each subgroup. (See Appendix C.4.2 for more subgroups.)Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Our code is currently included in the supplementary material and will be made available on GitHub upon acceptance. Detailed instructions are provided in the readme.md file within the supplementary material. Real-world datasets used in our experiments are cited with their respective references; no new datasets were generated for this work.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
conclusion: 
 In this paper, we propose ConSurv, a novel contrastive learning approach for deep survival analysis.Our method leverages weighted sampling to incorporate the intuitive assumption that patients with similar event times likely share similar clinical statuses. Unlike most existing deep survival models, ConSurv does not directly manipulate hazard or survival predictions during contrastive learning. This approach allows our method to maintain well-calibrated predictions based on the negative log-likelihood loss. Experiments on multiple real-world datasets demonstrate the superiority of our method over state-of-the-art deep survival models, particularly in achieving superior calibration performance compared to ranking loss-based models. 
 
 Appendix: The notations used in this work are listed in Table 3 . The indicator for acceptable pairs in the ranking loss αThe margin to ensure a minimum time difference βThe balancing coefficient between the loss function δ iThe Event indicator for patient i DThe survival dataset f ϕThe hazard network function f θThe encoder function g ψThe projection head function HThe latent representation space λThe conditional hazard function L NLLThe negative log-likelihood loss L RankThe ranking loss L SNCEThe survival outcome-based contrastive learning loss MThe number of samples in a batch RThe risk function SThe survival function σThe temperature coefficient for the weight function νThe temperature coefficient for the SNCE loss TThe set of all possible discrete survival times τ iThe observed survival time for patient i T maxThe maximum time horizon xiThe corrupted version of the original sample x i wThe weight function ZThe normalization constant x iThe input feature vector for patient i XThe input feature spaceIn our experiments with tabular datasets, we compared augmentation methods, focusing on traditional noise injection (NI) and the state-of-the-art method, SCARF [26] . For noise injection, we introduced random Gaussian noise into the latent space, determining the optimal corruption rate through a hyperparameter search consistent with SCARF. SCARF consistently showed performance that was similar to or better than NI. Consequently, we adopted SCARF as our primary augmentation method for our experiments. While the CI and IBS provide valuable overall assessments of a given survival model (over the entire time horizon), these metrics cannot fully capture variations in model performance across different time points. To address this, we included time-dependent performance evaluations, namely the time-dependent C-index [5] and the time-dependent Brier score [30] at three different time points (i.e., 25%, 50%, and 75% percentiles of time-to-events as in [32, 33, 34] . It is worth highlighting that utilizing these time-dependent metrics may reveal subtle differences that might be obscured when using CI and IBS alone. As shown in Table 5 and 6, we demonstrate the superiority of our method in discriminative power while preserving calibration. To evaluate the performance of survival models at various time points, we select the 25%, 50%, and 75%-percentile of each dataset.In Figure 5 , When using L NLL alone, representations tend to cluster tightly in a narrow space regardless of time. However, when applying L SNCE (contrastive learning with a time weight) to L NLL , representations of features with similar times are more likely to be positioned close to each other.Sensitive Analysis on Coefficient α. α is the margin to ensure a minimum time difference between samples in acceptable pairs. All the weights for the entire sample are precomputed. For setting the margin, we calculate the percentile for only the weights corresponding to Case 2 mentioned in section 4.2. By defining this minimum time difference, the margin aids in reducing the ambiguity in survival outcomes, leading to a more precise comparison of patient risks. Specifically, this enhancement is vital for improving the model's discrimination capabilities, as it removes potential false negative samples with little difference in their survival outcomes. In Table 7 , to determine the sensitivity of the ConSurv with respect to the value of α, we report the performance results on four datasets by varying (Samples are listed in ascending order of their event times.)the value of α according to the following percentiles of time differences: {5.0, 7.0, 10.0, 12.0, 14.0}. Additionally, we compare the performance metrics with no margin (i.e., α = 0).Effect of Margin α. In Figure 6 , we compare time differences based on censoring times and the (unobserved) ground-truth event times. For each event sample, we compute the time differences between its event time and: 1) the (observed) censoring time of its corresponding comparable censored sample, and 2) the (unobserved) ground-truth event time of that same censored sample.To show this effect, we have conducted an experiment using additional synthetic data by adapting the time-to-event generation process in DeepHit [1] for both event times T i and censoring times C i :EQUATION) Here, we randomly generated 1,000 random samples, where a sample is censored whenT i > C i .Without a margin, censored samples with censoring times close to event times may be incorrectly treated as having similar event times, leading to very low weight assignments. To avoid this, we introduce a margin to exclude such samples when computing our contrastive loss. Overall, the margin in |τ i -τ j | ≥ α helps prevent failures in accurately distinguishing risk between comparable censored samples.Sensitive Analysis On Coefficient β. β is the balancing coefficient in our overall loss function where increased β makes the impact of L SNCE more dominant. Table 8 shows that when β (using the optimal α for each dataset) is relatively small (here, β = 0.01), contrastive loss struggles to effectively learn similarities between survival outcomes, leading to poor discrimination performance. However, increasing the β ratio consistently improves performance. This suggests that our proposed contrastive learning approach does not prevent the survival model from learning effectively guided by the NLL loss. Figure 7 shows calibration plots of ConSurv in comparison with the benchmarks on all the evaluated datasets. Here, similar to our observation in Section 5.3.2, ConSurv consistently exhibits calibration performance that is similar to or better than the benchmarks, effectively balancing discriminative and calibration power. In this section, we show that our model learns well about the differences between subgroups on realworld datasets, with good calibration performance by comparing the averaged survival predictions and the observed KM estimation within each subgroup. In Figure 8 , we subgroup patients based on cellularity, which is a categorical variable representing the degree of density of cells in the tumor. Similarly, in Figure 9 (a), we partition patients based on the size of the tumor (with + indicating 70% or more and -indicating 30% or less of the total size), which have a significant impact on the event.In Figure 9 (b), we examine binary hormone receptor statuses progesterone receptor (PR).Our results show that our method closely aligns with the KM curve, outperforming other deep learningbased survival models, especially those with ranking loss. We assess calibration performance using the Wasserstein distance to compare survival predictions with KM curves across different subgroups.In Table 9 indicate that our model provides well-calibrated predictions within each subgroup. Table 11 shows baseline statistics of the real-world datasets. METABRIC [35] comprises clinical features and gene expression profiles (overall 21 features) used to determine the risk of breast cancer subgroups for 1,981 patients. Among the total number of patients, 1,093 (55.2%) are censored, and 888 (44.8%) are uncensored. NWTCO [36] is a study related to Wilms' tumor, containing 4,023 patients with 6 clinical and histology features. Among the total number of patients, 3,457(85.8%) are censored and 571(14.2%) are uncensored. GBSG [37] investigates the effects of hormone treatment on recurrence-free survival time, with the event of interest being breast cancer recurrence. The dataset consists of 21 clinical and tumor-related features for 2,232 patients. Among them, 965 (43.2%) are censored and 1,267 (56.8%) are uncensored. FLCHAIN [38] contains subjects from a study of the relationship between serum-free light chain and mortality, where each subject is described by 8 features. Among 6,524 subjects, 1,962 (30.1%) are censored and 4,562 (69.9%) are uncensored. SUPPORT [39] focuses on the survival time of seriously ill hospitalized adults. The dataset consists of 14 clinical features for 8,873 patients, having 2,837 (31.9%) censored and 6,036 (68.1%) uncensored. SEER [40] provides data on prostate cancer patients, containing 12 clinical and cancer-related features for 54,544 patients. Among them, 604 (1.11%) are uncensored and 53,940 (98.9%) are censored. We split the data into train, test, and validation sets with a ratio of 0.64:0.20:0.16, and then apply min-max normalization to the input features.Integrated Brier Score (IBS) is metric to evaluate the mean squared error between the predicted survival curve with step function of the observed event. The IBS is the integration of the Brier score across all time points.Distributional Divergence for Calibration (DDC) is a novel metric introduced in [2] to assess the calibration quality of survival models. This metric computes the divergence between the empirical distribution of estimated survival probabilities at the observed event times and a uniform distribution. Specifically, DDC utilizes the Kullback-Leibler (KL) Divergence between these two distributions.• It first bins the estimated survival probabilities Ŝ(t i |x i ) at the observed event times into ten equal-width intervals across the [0,1] range. • It then calculates the KL divergence D KL (P ∥ Q) between this binned distribution P and a theoretical uniform distribution Q. • DDC is scaled so that its values range between 0 and 1, with lower values indicating better calibration. A perfectly calibrated model, where the estimated probabilities perfectly match a uniform distribution, would yield a DDC value of 0.DDC serves as a more nuanced measure compared to traditional metrics like the Brier score or C-index because it directly evaluates how closely the distribution of the model's probabilistic predictions matches the expected uniform distribution of true probabilities. This direct assessment helps in determining whether the model's predictions are systematically biased or well-aligned with actual outcomes.D-Calibration (D-CAL) assesses how well predicted survival probabilities align with observed outcomes based on a goodness-of-fit test. D-CAL bins the predicted survival probabilities at the true event times into ten equal-width intervals from 0 to 1, and performs a chi-squared test to ascertain the uniformity of the distribution. As suggested in [31] , we reports p-values from the D-CAL test and the number of datasets with p-values greater than 0.05.Benchmarks. We compare ConSurv with six survival models including one traditional statistical method and four state-of-the-art deep learning methods: CoxPH [27] is a commonly used statistical method under the proportional hazard assumption 4 , DeepSurv [28] is an extension of the Cox model which utilizes a DNN to predict individual hazard rates 5 , DeepHit [1] is a DNN model which directly models the joint distribution of the event times 6 , DRSA [10] is an RNN-based model which predicts the likelihood of the true event occurrence and estimates the survival rate over time7 , DCS [2] is an RNN-based model that estimates calibrated individualized survival curves while maximizing the discriminative power 8 , and X-CAL [15] is a survival model designed to focus on the calibration performance by directly utilizing a differentiable version of D-CAL as a part of the objectives. 9 Especially with DCS and X-CAL, we made every effort to faithfully reproduce the code with hyperparameter optimization, as the original code had several errors and we were unable to use it for our experiments.We discuss implementation of ConSurv below. The source code for ConSurv is available in https://github.com/dongzza97/ConSurv. Throughout the experiments, training ConSurv and its variants takes approximately 60 minutes to 2 hours on a single GPU machine.10 We perform a random search for hyperparameter optimization -including the batch size, hidden dimension, depth, learning rates, corruption rates, σ, α, and ν -on the validation set and choose the settings with the best performance for ConSurv on each dataset. Table 12 describes the model specifications for the evaluated datasets. The candidates for random search can be given as {32, 64, 128, 256}, {16, 32, 64}, and {3, 4, 5} for batch size, hidden dim, and depth respectively. We carefully choose the σ, α and ν for L SNCE over {0.25, 0.5, 0.75}, {5.0, 7.0, 10.0, 12.0, 14.0} and {0.03, 0.05, 0.07}. Corruption rates are selected from a range of 0.1 to 0.9. To train multiple components in our network simultaneously, we need to find different learning rates for each network f θ , g ψ , and f ϕ , where the learning rate is chosen from a set {1e-3, 1e-4}. In this work, we highlight the direct and indirect influence of our approach through experiments on real-world clinical datasets. The use of these datasets was in accordance with the guidance of the respective data providers and domain experts. We acknowledge the potential for both positive and negative impacts associated with survival analysis techniques. While these methods can provide valuable insights for personalized treatment and risk stratification, we emphasize the importance of careful interpretation and validation by domain experts before applying these predictions in clinical practice. Misuse or misunderstanding of survival analysis could lead to unintended consequences for individuals or groups, such as inequitable access to care or discriminatory practices. Therefore, we advocate for transparency, accountability, and ongoing dialogue among researchers, clinicians, and the public to ensure the responsible and ethical use of survival analysis in healthcare.Survival analysis datasets often include features where a clear event time is not available, indicating a significant number of censored patients. This makes it challenging to assign weights based on time differences and to compare them against each other. Therefore, utilizing such data for learning purposes presents difficulties. Overcoming these limitations requires further research. Potential avenues include modifying models to account for such uncertainties or developing alternative learning approaches. Additionally, there is a need for new evaluation metrics or model developments that consider the characteristics of such data.Furthermore, the tabular datasets we utilized often lack diverse augmentation techniques. Future research could explore augmentation methods suitable for survival datasets to enhance contrastive learning performance. For example, techniques considering temporal variations or reflecting uncertainties in events could be investigated. Such research endeavors are expected to enhance understanding in the fields of survival analysis and contrastive learning, contributing to improved model performance.Algorithm 1 Pseudo-code for the ConSurvInput: Dataset D = {(x i , τ i , δ i )} N i=1, batch size M , margin α, coefficient β encoder θ, projection head ψ, hazard network ϕ, learning rate η 1 , η 2 . Output: ConSurv parameters (θ, ψ, ϕ)Initialize parameters (θ, ψ, ϕ) repeat Sample a mini-batch of {(x i , τ i , δ i )} M i=1 ∼ D for i = 1, ..Draw augmentation function: s ∼ S Augmentation pair:x i , x + i ← -x i , s(x i ) Latent representation: h i , h + i ← -f θ (x i ), f θ (x + i ) Map to the embedding space: z i , z + i ← -g ψ (h i ), g ψ (h + i ) end for for all i = 1 • • • , 2M and j = 1 • • • , 2M doCosine similarity in the embedding space: s(x i , x j ) = z ⊤ i z j /(∥z i ∥∥z j ∥) Weight function for negative distribution: w(τ - j ; τ i ) = 1 -e-|τ i -τ - j | σRedefined weight function considering the right-censoring: w(τ - j ; τ i ) = I i,j • w(τ - j ; τ i ) end for Update the encoder and projection head parameter θ, ψ:EQUATIONfor i = 1, ..., M do Compute latent representations:h i ← f θ (x i ) Compute estimates for survival function: Ŝi (τ i |x i ) ← t≤τi (1 -f ϕ (h i , t))Compute estimates for probability mass function: pi (τi |x i ) ← f ϕ (h i , τ i ) t<τi (1 -f ϕ (h i , t)) end forUpdate the encoder and hazard network parameters (θ, ϕ):(θ, ϕ) ← (θ, ϕ) -η 2 ∇ (θ,ϕ) -1 M M i=1 δ i log p(τ i |x i ) + (1 -δ i ) log Ŝ(τ i |x i )until convergence Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Source code for ConSurv is available in https://github.com/dongzza97/ConSurvTable 1 presents the performance results obtained using 10 random seeds. For enhanced stability verification, we conducted additional experiments with 25 random seeds. Please refer to Table 10 in appendix C.5.Python package scikit-survivalhttps://github.com/havakv/pycoxhttps://github.com/chl8856/DeepHithttps://github.com/rk2900/DRSAhttps://github.com/MLD3/Calibrated-Survival-Analysishttps://github.com/rajesh-lab/X-CALThe specification of the machine is CPU: INTEL XEON Gold 6240R, GPU: NVIDIA RTX A6000 


Decision: I would like to thank the authors and reviewers for engaging in a fruitful discussion. The reviewers mostly leaned toward acceptance (1 strong accept, 2 weak accepts, 1 borderline reject). I would consider this paper borderline, especially as the strong accept review was unusually terse, and the borderline reject review asks for additional baselines, which while reasonable to request, I realize can be challenging for the authors to fully address within the author/reviewer discussion period. I have decided to recommend acceptance for this paper, as I view the contribution of this paper as useful for the community to know even if there are very clear limitations (that I think the authors should address either with additional experiments or at least with some discussion in the manuscript text).

Specifically regarding the additional baselines requested by reviewer Lfcy: I agree with the authors that CSD was published so recently that it is fine not to compare against it, and that, furthermore, CSD is quite different in that it can be used to modify different existing survival models. However, I agree with reviewer Lfcy that comparison with DATE is actually possible (one can compute standard survival analysis metrics using DATE), so adding DATE as a baseline would be a helpful comparison.

I would encourage the authors to acknowledge that DATE and CSD exist and could potentially be good at both discrimination and calibration. Even if the authors do not plan on adding experiments, at least discussing these other methods as worthwhile to compare against in the future would be helpful. Specifically for CSD, since it works quite differently, an interesting question would be how much of a performance boost does CSD applied to various models (including the proposed method) yield (with the idea that if the base model being modified already does a great job with calibration and discrimination, then you'd expect applying CSD to be unnecessary/to lead to less of an improvement).

Reviews: review: paper_summary: paper_summary: This paper presents an approach to handle survival analysis datasets using a method to improve both calibration and discrimination. This adds a bunch of novelties to the field like handling right-censoring and an SNCE loss to handle calibration and ranking simultaneously (NLL + contrastive loss). This paper presents a reliable way to look at survival analysis, write a hazard function which is quite reasonable and also present a ranking loss to take care of the domain-specific problem of stratification by risk-levels using generated patient rank list.
strength_weakness: strength_weakness: -> lack of using unstructured data
-> more clarity of censored data
questions: questions: I understood why there is censored data, but if the authors could explain with an example of a time series what censored data looks like that would make it easier.
limitations: limitations: maybe added unstructured data like doctors notes
paper_summary: paper_summary: The paper proposes a contrastive learning loss to regularize maximum likelihood learning of discretized survival times. The main contribution of this work is the utilization of the Laplace kernel to weigh negative pairs inversely proportional to their time difference from the anchor sample, while also accounting for censoring. Specifically, negative pairs with event times close to that of the anchor sample contribute less to the contrastive loss. Experimental results on four datasets demonstrate that the proposed approach achieves a high Concordance Index (C-Index) without a significant loss in calibration when compared to baselines.
strength_weakness: strength_weakness: *Given that the main contribution of this work is the application of contrastive loss to survival analysis, the paper lacks complete details for reproducibility:*
- There are no details on how the model functions $ f_{\theta}, f_{\phi}$, and $g_{\psi} $ are parametrized.
- Details on the construction of positive pairs are very sparse. No motivation is provided in terms of why the SCARF approach is chosen and why it should be expected to work with survival data.
- The paper proposes the Laplace Kernel for weighing samples; it is unclear if alternative kernels were also considered.
- The notation seems sloppy; symbols ${f}$ and $Z$ are overloaded.
- I expect calibration and the C-Index should be sensitive to $\beta $; the paper does not discuss how to choose $\beta $ and what values resulted in the performance results reported. This phenomenon has been discussed in detail in [1]. I encourage the authors to include a comprehensive discussion on this trade-off.
- It is unclear how the margin parameter $\alpha$ was chosen. I encourage the authors to provide visualizations in the main paper to demonstrate sensitivity to both $\alpha$ and $\beta $.

*The proposed approach relies on discretizing event times, which is sensitive to the time-binning approach. Additionally, it seems the contrastive loss could be an indirect way to achieve consistency in event times. I encourage the authors to also benchmark with approaches that directly predict event times such as [2].*

*The four datasets used seem to be of low sample size. I encourage the authors to also consider larger datasets such as SUPPORT and SEER.*

**References**

- [1] Qi et al., "Conformalized Survival Distributions: A Generic Post-Process to Increase Calibration", ICML 2024
- [2] Chapfuwa et al., "Calibration and Uncertainty in Neural Time-to-Event Modeling", IEEE TNNLS 2020
questions: questions: - What are the values of $\alpha$ and $\beta $ for the reported results?
- Were alternative weighting kernels considered?
- Were alternative approaches for constructing positive pairs considered?
limitations: limitations: - The paper does not discuss modeling assumptions; for example, the assumed independent censoring mechanism could be violated in practice.
paper_summary: paper_summary: The authors study discrete-time survival analysis, proposing to train models using a loss function that combines the NLL loss and a modified NCE loss. This NCE loss is modified to take the survival outcomes (event times) into account, mitigating the effect of potential false negatives that have small event time differences.

The method is evaluated on 4 tabular clinical survival datasets, comparing against 6 baseline models and 3 method variations. The proposed method performs well compared to all baselines.
strength_weakness: strength_weakness: - The technical novelty/innovation is perhaps somewhat limited ("just" adding a contrastive loss term).

- The experimental evaluation could perhaps be more extensive, include more datasets. Would have been nice to see the method being applied also to some non-tabular dataset.




Summary:
- Somewhat limited technical novelty, but this is a well-written paper with a simple/intuitive method that seems to perform well compared to reasonable baselines. Thus, I am definitely leaning towards accept.
questions: questions: - Could you perhaps apply the proposed method also to some non-tabular dataset?

- The visualization in Figure 2 is really neat, could you add this for the three other datasets as well (to the appendix)?




Minor things:
- 130: "by minimizing the following negative NLL loss", remove "negative"?

- Perhaps add 1-3 sentences to the start of Section 3, just to describe what is covered here (and why this is covered, why this is relevant for your method)?

- 248: "We compare our proposed method and the benchmarks with...", "benchmarks" --> "baselines"? The same also for line 251?
limitations: limitations: Yes.
paper_summary: paper_summary: This work concerns deep learning for survival analysis. Deep learning models for survival analysis are typically desired to have good discriminative performance, where a model can differentiate between patients of different risk profile, and calibration performance, where the time-to-event is accurately predicted by the model. Methods which seek to improve discrimination performance have often led to poorer model calibration. Consequently, this paper proposes an auxiliary contrastive loss to improve the discriminative performance of deep survival models, while still achieving high calibration performance. The novelty in this method is adapting the SOTA contrastive learning method for tabular data, SCARF, by weighting the negative pairs according to the difference in time-to-event. Hence, samples with greater difference in time-to-event are considered more important negative pairs, enforcing an inductive bias to differentiate the latent representations of samples with different time-to-event.  After providing a detailed problem formulation, the contrastive loss and generation of positive and negative pairs is described. The results of several experiments are provided which compare the proposed method with current state-of-the-art methods when applied to a number of real world datasets, and demonstrate the proposed model performance under ablation of each term in the loss function. From these experiments it is concluded that the auxiliary contrastive loss achieves superior discriminative and calibration performance.
strength_weakness: strength_weakness: While the technical parts of this paper are clearly explained, the organisation and presentation of the paper could be improved. In particular, the introduction uses quite a lot of terminology specific to survival analysis that is unexplained or explained in detail only later in the paper. As a specific example, censoring is first mentioned on line 67, but is not clearly explained. Explaining this terminology, which may not be familiar to all of NeurIPS wide readership, at first use would improve the clarity of the paper. Presentation and clarity of the paper would also be improved if all captions were self-contained. In some cases, for example in Table 1, it is difficult to interpret the results as presented, and this could have been avoided if the captions were more detailed. Specifically, it is not clear what the standard deviations represent in this table i.e. variation across model initialisations or data splits.   

This work should be praised for providing a list of experiments that should provide the evidence needed to make the claim that the proposed model provides superior discriminative performance while maintaining calibration performance. However, the clarity and quality of these experiments is sometimes below expectations, and this ultimately reduces the overall significance of this work. Table 1 is used to report that the proposed model is the best performing model. However, it seems that the model is within 1 standard deviation of the second best performing model, and hence the statistical significance of this result is unclear. As previously mentioned, while mean and standard deviation of performance metrics are reported, it is not entirely clear what distribution they refer to – whether it be different data splits, or model initialisations via random seed. This makes it difficult to interpret the significance of these results, and also makes it more difficult to reproduce. 

The paper, therefore, addresses an important problem, proposes a novel and well thought out method to address this issue, but the significance is undermined by a lack of clarity and ambiguity surrounding the statistical significance of reported results.
questions: questions: The questions below focus on the details of the experiments that were unclear. Clarity on these questions may change the overall opinion of the paper. 

Table 1: In the caption you refer to the standard deviation - are the reported values in this table the mean? 
Table 1: Does the standard deviation refer to variation over different data splits, model initialisations or something else? 
Table 1: What do bold and underlined values represent? 
Table 1: How do you determine that the superior performance of ConSurv is statistically significant? 

Line 528: Can you confirm whether min-max normalisation was performed before or after data was split into train, test and validation sets? 

Line 236: What is the marginal corruption process? By what augmentation were positive pairs generated?

Line 197: What does the temperature coefficient scale? What impact does this have on model training? 

Line 104: Does ref. [18] show that model training is destabilised by the contrastive loss? If not, what supports this statement? Could this model have been included in your benchmarks?


NOTE: Due to the discussion in the author-reviewer discussion period, I feel that many of these points were clarified. In particular new results have better demonstrated the statistical significance of the performance of ConSurv. I raise my score from a 4 to a 6 to reflect this.
limitations: limitations: The authors have addressed the potential negative societal impact of their work, stating that model predictions from such work should be under scrutiny by domain experts before being applied in a clinical setting. They stress that incorrect use could lead to inequitable access or discriminatory practices. While this is welcome, it is not specific to this work, but is a concern of all works proposing machine learning for clinical practice. A potential impact more specific to this work might consider why the inductive bias introduced by the auxiliary contrastive loss would lead to unintended consequences for AI-supported decisions on treatment and intervention. For example, the unsupervised nature of the contrastive learning procedure could align patients based on features which may act as a proxy for protected traits and lead to inequitable model performance amongst different demographics. 

Additionally, a discussion of limitations is provided but is limited. The authors correctly identify that a significant number of patients are censored in real world datasets, and this undermines the importance sampling based on difference in time-to-event, as weights are assigned according to a function in equation (5). It would improve the discussion of the limitations if it was pointed out that seeking alternatives or amendments to (5) would be a specific avenue to improve upon this work. 

The limitation section could benefit from greater specificity. The authors propose “modifying models to account for such uncertainties or developing alternative learning methods”. The uncertainties they refer to are unclear, and a preferred statement might be: future avenues of work would be to develop a model which retains the benefit of contrastive learning with respect to discriminative performance, whilst also being agnostic to the number of censored samples within the training set.

They then state “there is a need for new evaluation metrics or model developments that consider the characteristics of such data”. The clarity of this proposal might be clearer if stated with more specificity i.e. "to develop a model that retains high discriminatory and calibration performance for datasets with arbitrary proportions of censored samples, new metrics must be established that can compare the same model applied to different proportions of censored samples."

Comments: title: Thanks
Thanks
Response to Reviewer Lfcy
Official Response by Reviewer Lfcy
reminder from your area chair to respond to this author feedback
reminder from your area chair to respond to this author feedback
Response to Reviewer VjbS
Response to Reviewer VjbS
Response to Reviewer Lfcy
Response to rebuttal
comment: Dear Reviewer VjbS,

Thank you so much for raising the score from 4 to 6. We kindly ask if you could please ensure that the updated score is reflected in the main system.

We greatly appreciate your consideration.

Thank you, Paper 15583 Authors
I have happily raised my score from a 4 to a 6 in agreement with one of the other reviewers.
Dear Reviewer VjbS,

We are sincerely grateful for your time and energy in the review process. In light of your satisfaction with our response, we wonder whether the reviewer would kindly consider revising the rating.

Thank you, Paper 15583 Authors
Thank you to the authors for taking the time to reply again and provide further discussion and clarification. 

I have no further questions. Thank you for pointing me to the relevant code - I agree that min-max normalisation has not led to data leakage. And thanks for clarifying how your results demonstrate a statistical significant improvement over current baselines. I find the arguments above convincing.
# [ Response to Reviewer Lfcy ]

**Regarding the Comparisons with DRAFT and DATE**:

We appreciate the reviewer's insightful comments regarding the comparison of ConSurv with DRAFT and DATE.

We opted to include DRAFT, not DATE, in our comparative analysis due to the following considerations: DRAFT posits a log-normal distribution for the underlying time-to-event process, $p(t∣x)$. This allows for a direct calculation of the survival function, $S(t∣x)$, facilitating a clear and equitable comparison using the performance metrics presented in our study.

Conversely, while DATE represents a significant advancement in deep survival models by directly generating time-to-event outcomes, it employs a generator that learns the underlying distribution, $p(t∣x)$, implicitly. Consequently, computing $S(t∣x)$ necessitates sampling a substantial number of samples, complicating the comparison against the reported performance metrics.

Therefore, we restricted our comparison to DRAFT as a deep learning counterpart that explicitly focuses on AFT models.


**Regarding the comparison with CSD (Qi et al.’s Method)**:

We appreciate the suggestion to compare ConSurv with CSD (Qi et al.'s post-hoc calibration method), but we believe a direct comparison is not within the scope of our work. Our focus lies in improving discriminative power while preserving calibration, a distinct advancement from traditional ranking losses. CSD addresses a post-hoc approach to improve calibration of survival models, making a direct comparison less relevant.

Furthermore, Qi et al.'s work was published after our submission deadline and thus couldn't be included in our original analysis. However, we acknowledge its value and will incorporate a comparison in the appendix of our final version by applying their post-hoc calibration to existing ranking-based models.


**The Connection Between Contrastive Learning and Model Calibration**:

As the reviewer pointed out,  Kamran et al. and Qi et al. have discussed the trade-off between discriminative power and calibration, where the trade-off arises because discrimination is typically assessed at an individual level, whereas calibration is evaluated at the population level. However, the parameter, $/lambda$, in the loss term proposed by Kamran et al. does not directly explain this trade-off. Instead, it controls the balance between two losses in the composite loss function, where each loss influences both the discrimination and calibration of the survival model. 


To the best of our knowledge, there is no theoretical proof on why utilizing the ranking loss may lead to poor calibration. We hypothesize that directly modifying the output of survival models (whether the hazard function in DRSA or the PMF of event times in DeepHit) can bias the network towards optimizing the approximated C-index, potentially sacrificing accurate modeling of event time distributions, as conjectured by Kamran et al.

In contrast, our approach employs a contrastive learning framework that adjusts latent representations based on the similarity in the event times, implicitly promoting discriminative power without directly impacting the distribution of event times. Consequently, we were not able to observe a clear trade-off between the two loss functions, a phenomenon more commonly seen with traditional ranking losses.


**Clarification on Notations**: 

We've taken steps to improve clarity by clarifying notations and providing comprehensive implementation details in both the appendix and this rebuttal. If you have specific areas that you feel still require attention, please point them out and we'll be happy to address them further.
Thank you for your response and for providing additional experimental results on DRAFT, large datasets, and sensitivity analysis.

- Unfortunately, I don't think the comparison to DRAFT is adequate; a fairer comparison would be against DATE, since Chapfuwa et al. have already demonstrated that DATE is superior to DRAFT.
- The connection of contrastive learning to model calibration appears to be superficial. The paper does not provide theoretical guarantees, only experimental results. Given that the experimental results highlight that $\beta$ controls the calibration-discrimination trade-off, this is a key weakness of the work. This weakness is not shared by the conformalized post-processing approach detailed by Qi et al. Hence, revisions should include comparisons to CSD and the Kaplan-Meier estimator.
- The paper will require significant revisions to address the sloppy mathematical notations and provide complete details on the modeling choices.

Considering these issues, as well as the other reviews, I am inclined to maintain my score.
Hello reviewer VjbS,

Thanks for already engaging in discussion with the authors on this paper. The authors have responded in detail to your most recent comments. Please try to provide a response before the end of the author/reviewer discussion period (Aug 13 11:59pm AoE).

Thanks,
Your AC
Hello reviewer Lfcy,

The author/reviewer discussion period ends very soon (Aug 13 11:59pm AoE). It would be great if you could respond to the authors as they've taken the time to respond to your review.

Thanks,
Your AC
**Evaluating Calibration from Different Perspectives**: 
We further provided the comparison in the model calibration by evaluating alternative calibration metrics -- Distributional Divergence for Calibration (DDC) [R6] and D-Calibration (D-CAL) [R7] -- that assess model calibration from a slightly different perspective. These metrics provide a more nuanced perspective on calibration quality, directly evaluating the alignment of predicted survival probabilities with uniform distributions. 
For the assessment of statistical significance of the performance gain in terms of model calibration, we have additionally included the p-values for the DDC in Table I. (Please note that the D-CAL test already incorporates p-values by its definition and thus is not considered for comparing the statistical significance of the performance gains.) Table I highlights that ConSurv achieves significant calibration gain over the benchmarks -- especially the ranking-based deep survival models -- across most of the datasets. 

*Table I. p-value for DDC*
|Dataset|M|N|G|F|SU|SE|
|-|-|-|-|-|-|-|
|CoxPH|-|**0.012**|-|**0.018**|**0.000**|**0.017**|
|DeepSurv|**0.030**|**0.000**|-|**0.018**|**0.000**|**0.000**|
|DeepHit|**0.000**|**0.000**|**0.000**|**0.000**|**0.000**|**0.005**|
|DRSA|**0.013**|-|**0.000**|**0.049**|**0.000**|**0.001**|
|DCS|0.341|-|**0.000**|0.318|**0.000**|-|
|X-CAL|0.926|-|0.964|**0.016**|**0.002**|-|
|SupWcon|-|**0.005**|-|**0.043**|**0.045**|**0.000**|

---
## *Reference*

[R1] H. Uno et al., “On the C‐statistics for evaluating overall adequacy of risk prediction procedures with censored survival data,” Statistics in Medicine, 2011. 

[R2] E. Graf et al., “Assessment and comparison of prognostic classification schemes for survival data,” Statistics in Medicine, 1999.

[R3] Z. Wang et al., "Survtrace: Transformers for survival analysis with competing events," ACM-BCM, 2022.

[R4] C. Nagpal et al., "Deep survival machines: Fully parametric survival regression and representation learning for censored data with competing risks," IEEE Journal of Biomedical and Health Informatics, 2021.

[R5] C. Lee et al., "Temporal quilting for survival analysis," AISTATS, 2019.

[R6]  F. Kamran et al., "Estimating calibrated individualized survival curves with deep learning," AAAI, 2021.

[R7] H. Haider et al., “Effective ways to build and evaluate individual survival distributions,” JMLR, 2020.
# [ Response to Reviewer VjbS ]
We appreciate the reviewer for the feedback and acknowledging our efforts in addressing the reviewer’s comments. Here, we would like to provide further clarification on the following points:

**Normalization and Data Leakage**:
We apologize for any confusion caused by our initial explanation of the min-max normalization process. To clarify, we performed the min-max normalization **after** splitting the dataset, **not before**. This inadvertent miscommunication may have raised concerns about potential data leakage. We assure you that our method prevents any leakage, as normalization is conducted based solely on the training and validation sets. For verification, this procedure can be confirmed in the supplementary $\texttt{dataset.py}$ file that was submitted at the initial submission deadline.

**Statistical Significance of the Performance Gain**:
We would like to start by emphasizing that our work introduces a deep survival model leveraging contrastive learning to enhance discriminative power without sacrificing calibration. Therefore, a fairer comparison showcasing our contribution would involve comparing our model with i) variations of our model trained solely with the NLL loss or with both the NLL and ranking losses, and ii) deep survival models utilizing traditional ranking loss. 
In this context, we have evaluated the statistical significance of our results based on the p-values calculated by the Welch’s t-test. As shown in Table G, ConSurv achieves superior discriminative power (while maintaining its calibration) over its variant trained solely with the NLL loss. The results are statistically significant except for the NWTCO dataset. On the other hand, our variant trained with both the NLL and ranking losses often loses its calibration power. Contrarily, when compared with the variant trained with the NLL and ranking losses, ConSurv significantly outperforms in terms of calibration power for all the evaluated datasets while providing gains in discriminative power (but not statistically significant for the NWTCO dataset) except for the SUPPORT dataset. These results are consistent with our motivation described in the Introduction.
A similar trend can be observed when compared with traditional ranking-based deep survival models (i.e., DeepHit and DRSA). While the gain in discriminative power is sometimes not statistically significant, our method provides statistically significant improvements in terms of calibration power, as shown in Table G.

*Table G. p-value for the variants of our method* 
|Model|Metric|METABRIC|NWTCO|GBSG|FLCHAIN|SUPPORT|SEER|
|-|-|-|-|-|-|-|-|
|**NLL**|CI|**0.000**|0.308|**0.001**|**0.000**|**0.000**|**0.008** |
||IBS|**0.000**|**0.001**|0.174|0.194|**0.034**|**0.000**|
|**NLL+Rank**|CI|**0.000**|0.424|**0.000**|**0.000**|-|**0.000**|
||IBS|**0.000**|**0.003**|**0.000**|**0.000**|**0.000**|**0.018** |

**Analysis of Time-Dependent Performance Metrics**: 
While the CI and IBS provide valuable overall assessments of a given survival model (over the entire time horizon), these metrics cannot fully capture variations in model performance across different time points. To address this, we included time-dependent performance evaluations, namely the time-dependent C-index [R1] and the time-dependent Brier score [R2], in Appendix C.2, at three different time points (i.e., 25%, 50%, and 75% percentiles of time-to-events as in [R3, R4, R5]). It is worth highlighting that utilizing these time-dependent metrics may reveal subtle differences that might be obscured when using CI and IBS alone. 
Table H shows the number of statistically significant performance improvements over each survival model across different datasets. The improvement is considered significant if ConSurv provides statistically significant gains at at least one of the three time points. Again, we use the Welch's t-test to calculate the p-values for both time-dependent discriminative and calibration performance throughout the datasets. 
ConSurv offers superior calibration performance compared to ranking-based deep survival models (DeepHit and DRSA) and outperforms calibration-focused survival models (DCS and X-CAL) in discriminative power. Overall, compared to the deep learning baselines, ConSurv achieves statistically significant improvements in at least 4 out of 6 datasets, demonstrating its robust and consistent performance gains across various scenarios.

*Table H. Number of Significant Performances Across Datasets*
|Model|C-index|Brier score|
|-|-|-|
|CoxPH|3|1| 
|DeepSurv|6|6|
|DeepHit|4|6|
|DRSA|5|6|
|DCS|5|5|
|X-CAL|5|5|
|SupWcon|5|4|
# [ Response to Reviewer Lfcy ]

Once again, we would like to thank you for your invaluable feedback! We were wondering whether our response from  Aug 7 has sufficiently addressed your comment. If you have any remaining comments, please let us know. We would be happy to do our utmost to address them!
Thanks for the rebuttal. Agree with the future work.
Thanks to all authors for their rebuttal. They have clearly gone to great lengths to address the comments that all reviewers have made. I have read the reviews from each of the reviewers, and the authors' response to them, as well as the response of the authors to my own review. 

I thank the authors for their clear explanation of the temperature coefficient, censoring, and the generation of positive and negative pairs. The manuscript would be improved by the inclusion of this text. 

I also thank the authors for performing further statistical analysis, specifically by providing the p-values associated with the Welch's t-test, to assess the performance of ConSurv relative to the other baselines. Unfortunately, I remain unconvinced by the performance of ConSurv. The proposed benefit of this model is to enhance discrimination, without sacrificing calibration. However, table E shows us that ConSurv outperforms the following models on **both** discrimination (CI) and calibration (IBS):
- METABRIC:  DRSA, DCS and X-CAL 
- NWTO:  DeepSurv and X-CAL 
- GBSG: DeepHit
- FLCHAIN: DRSA and DCS 
- SUPPORT: DRSA and DCS
- SEER: CoxPH and DeepSurv 

In short, it seems that for 36 model-dataset pairs, only 12 show evidence for the proposed property. 

I also believe the choice to perform min-max normalisation before data splitting may have inadvertently led to data leakage, and I would expect the performance reported to be an overestimate were normalisation applied using the train/validation sets only. 

My score would change if I were convinced that the statistical performance was significant, and may be changed after further discussion.
Thank you for the response.

I have read the other reviews and all rebuttals.

While two of the other reviewers were leaning towards reject, I think the authors responded well overall to their concerns. I don't see any clear reason to lower my score.