Title: Toward a Well-Calibrated Discrimination via Survival Outcome-Aware Contrastive Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Survival Analysis

Contrastive Learning

Deep Learning

Healthcare
Abstract: Previous deep learning approaches for survival analysis have primarily relied on ranking losses to improve discrimination performance, which often comes at the expense of calibration performance. 
To address such an issue, we propose a novel contrastive learning approach specifically designed to enhance discrimination without sacrificing calibration. 
Our method employs weighted sampling within a contrastive learning framework, assigning lower penalties to samples with similar survival outcomes. This aligns well with the assumption that patients with similar event times share similar clinical statuses. Consequently, when augmented with the commonly used negative log-likelihood loss, our approach significantly improves discrimination performance without directly manipulating the model outputs, thereby achieving better calibration.
Experiments on multiple real-world clinical datasets demonstrate that our method outperforms state-of-the-art deep survival models in both discrimination and calibration. Through comprehensive ablation studies, we further validate the effectiveness of our approach through quantitative and qualitative analyses.

Toward a Well-Calibrated Discrimination via Survival Outcome-Aware Contrastive Learning

Abstract

Previous deep learning approaches for survival analysis have primarily relied on ranking losses to improve discrimination performance, which often comes at the expense of calibration performance. To address such an issue, we propose a novel contrastive learning approach specifically designed to enhance discrimination without sacrificing calibration. Our method employs weighted sampling within a contrastive learning framework, assigning lower penalties to samples with similar survival outcomes. This aligns well with the assumption that patients with similar event times share similar clinical statuses. Consequently, when augmented with the commonly used negative log-likelihood loss, our approach significantly improves discrimination performance without directly manipulating the model outputs, thereby achieving better calibration. Experiments on multiple real-world clinical datasets demonstrate that our method outperforms state-of-the-art deep survival models in both discrimination and calibration. Through comprehensive ablation studies, we further validate the effectiveness of our approach through quantitative and qualitative analyses.

1 Introduction

Deep learning models have gained significant attention in survival analysis (also known as time-toevent analysis), which focuses on predicting and understanding the occurrence of an adverse event (e.g., death) as a function of time. The utility of survival models is typically evaluated through two key aspects: discrimination and calibration. Discrimination measures a model's ability to differentiate between patients with varying risks, prioritizing those more likely to experience the event. Calibration, on the other hand, assesses how well a model's predictions align with the observed event distribution. In other words, a well-calibrated model offers predictions that closely match actual survival outcomes, offering crucial prognostic value to clinicians.

In an effort to enhance the discriminative power of survival models, ranking loss functions [1, 2, 3, 4] are frequently employed. These functions aim to maximize a relaxed proxy of the concordance index (C-index), a well-established metric for evaluating the quality of patient rankings based on the risk predictions of survival models [5]. However, a notable improvement in discriminative power often comes at the expense of calibration performance, which can negatively affect the clinical utility of predicted survival outcomes. For example, a poorly calibrated model that overestimates risk may lead to unnecessary treatment or testing for patients. Conversely, underestimating risk may result in patients with adverse conditions not receiving appropriate care.

Contrastive learning [6, 7, 8] is a framework that aims to learn an embedding space where similar samples are mapped to nearby locations, while dissimilar samples are pushed farther apart. Notably, although commonly used in an unsupervised fashion, this framework shares a fundamental principle

with ranking losses: both focus on the relative ranking of samples for better discrimination. However, unlike ranking losses, which directly modify differences in model outputs, contrastive learning operates on the relative distances of representations in the embedding space. This distinction offers the potential to overcome misalignment issues caused by direct comparisons and modifications of model outputs inherent in the ranking-based approaches. To leverage this potential and further enhance survival models, we propose incorporating informed sampling into the contrastive learning framework. By assigning lower penalties to potential false negative pairs based on the similarity of their survival outcomes, we instill an inductive bias in the model, encouraging it to learn that samples with similar event times likely share similar clinical statuses. This allows us to achieve strong discrimination without directly comparing or manipulating model outputs, thus potentially improving calibration as well.

Contribution. In this paper, we propose a novel contrastive learning approach, specifically designed for survival analysis, to enhance both discrimination and calibration performance. 2 Motivated by the assumption that patients with similar event times share similar clinical status, we contrast samples in the latent space based on their survival outcomes. However, unlike standard contrastive methods, we leverage the model output solely for maximum likelihood estimation (MLE), thereby promoting better calibration. This is achieved by incorporating importance sampling into our contrastive learning framework, where we assign weights proportional to the differences in survival outcomes. This guides our model to learn appropriate similarity relationships between clinical features relevant to survival analysis. Through experiments on multiple real-world clinical datasets, we demonstrate the superiority of our method in both discrimination and calibration performance, outperforming state-of-the-art deep survival models. We further provide comprehensive quantitative and qualitative analyses through ablation studies to showcase the impact of our novel contrastive learning approach tailored for survival analysis.

2 Related Works

2.1 Deep Learning Approaches to Survival Analysis

Deep learning-based survival models have garnered significant attention due to their ability to provide non-parametric estimations of the underlying discrete-time survival distribution, particularly conditional hazard or survival functions. Negative log-likelihood (NLL) has been widely used in survival analysis to estimate various survival quantities, such as conditional hazard functions [9, 10, 11], probability mass functions of event times [1], or survival functions [4]. MLE, based on NLL, provides unbiased estimates of these quantities under the assumption of ignorable censoring, where the event and censoring are independent given the input features. Building on this foundation, recent deep learning-based survival models often incorporate auxiliary losses alongside the NLL loss to further enhance model performance.

Ranking Loss. Ranking loss, often augmented with the NLL loss, has been widely employed in recent deep survival models to enhance the discriminative power of survival predictions [1, 2, 3, 4]. These ranking loss functions, which are differentiable approximations or upper bounds of the negative C-index [12], are typically based on exponential [1, 2], log-sigmoid [4, 13], or linear [3] functions. In particular, these approaches directly utilize model outputs, such as conditional hazard or survival functions, to establish pairwise ordering of comparable individual risks for better discrimination. However, since ranking loss primarily focuses on samples with earlier event times (often ignoring censored samples), these models may struggle to capture the full distributional characteristics of time-to-event outcomes. This can adversely affect calibration performance, potentially leading to inaccurate predictions regarding the observed time-to-event outcomes [2].

Calibration Loss. In healthcare applications, while achieving high discriminative power through ranking loss is crucial, well-calibrated predictions are equally important for effective clinical decisionmaking. Notably, recent approaches [14, 2] have prioritized enhancing calibration by minimizing the rank probability score (RPS), a prediction error metric specifically tailored for survival analysis. Furthermore, the authors in [15] have introduced a novel approach that transforms D-Calibration [16] into a differentiable objective, calculating the squared difference between observed and predicted events across multiple time intervals. In contrast to these methods, our proposed approach does not

explicitly incorporate a calibration loss function. Nevertheless, it achieves comparable or even superior calibration performance, demonstrating the effectiveness of our contrastive learning framework in implicitly preserving calibration alongside discrimination.

2.2 Contrastive Learning

Contrastive learning [6, 7, 17] is a framework that learns an embedding space that effectively discriminates among samples, by mapping positive pairs (similar samples) closer together, while pushing negative pairs (dissimilar samples) farther apart. While contrastive learning has made significant progress, our review of related work will focus on methods that explore various strategies for injecting inductive bias to enhance representation learning.

Exploiting Continuous Labels. Inspired by the success of incorporating label information into contrastive learning for classification tasks [8], recent research has extended this concept to regression tasks by exploiting continuous label information for constructing positive and negative pairs. Zha et al. [18] employ hard thresholding on label differences to generate positive and negative samples, capitalizing on the idea that samples with similar target values should be mapped closer in the latent space. Kerdabadi et al. [19] apply a similar approach in a dynamic time-to-event setting, defining positive pairs as those within a specific time window and weighting negative pairs proportionally to their time difference. However, this weighting scheme's heavy reliance on absolute time differences can destabilize the contrastive loss during training.

Weighting Negative Samples. Recent works have significantly improved contrastive learning by incorporating inductive bias in the selection of negative samples. This includes focusing more weights on hard negatives that challenge the model for more effective discrimination [20, 21, 22, 23] and avoiding potential false negatives that might mislead the model [24, 25]. Many of these studies address negative sample selection by employing informed sampling techniques to identify informative hard negatives or potential false negatives. Notably, Robinson et al. [20] leverage importance sampling by proposing a similarity-based distribution that prioritizes hard-to-distinguish negative samples.

3 Problem Formulation

3.1 Preliminary: Discrete-Time Survival Analysis

Suppose we are given a discrete-time survival dataset comprising N patients, denoted as D = { ( x i , τ i , δ i ) } N i =1 . Each patient i is represented by the input feature x i ∈ X where X is the input space, and the observed survival outcomes, τ i ∈ T and δ i ∈ { 0 , 1 } . Here, τ i and δ i indicate the time elapsed until either an event of our interest (e.g., death) or censoring (e.g., lost to follow-up) occurs and whether the event was observed or not (i.e., right-censored), respectively. Throughout, we treat survival time as discrete and the time horizon as finite such that the set of possible survival times is defined as T = { 0 , . . . , T max } with a pre-defined maximum time horizon T max.

The conditional hazard function, λ : X × T → [0 , 1] , is the instantaneous risk of the event at time t given feature x and is defined as λ ( t | x ) = P ( T = t | T ≥ t, x ) . Then, we can represent the survival function S : X × T → [0 , 1] as follows:

which is a non-increasing function of t , indicating the probability of the event occurring after time t given feature x . Equivalently, we could estimate the risk function, R : X × T → [0 , 1] , which represents the probability of the event occurring at or before time t given feature x , i.e., R ( t | x ) = P ( T ≤ t | x ) = 1 -S ( t | x ) .

Then, we can achieve likelihood-based estimates for the hazard function, ˆ λ , by minimizing the following negative NLL loss:

where ˆ p ( t | x ) = ˆ λ ( t | x ) ˆ S ( t -1 | x ) represents the estimate for the probability of an event occurring at time t , i.e., P ( T = t | x ) . Here, (2) exploits two pieces of information from the survival data: (i)

when the event is observed (i.e., δ i = 1 ), we know that the event occurred at time τ i , and (ii) when the event is not observed (i.e., δ i = 0 ), we are aware that the event will occur after time τ i .

3.2 Ranking Loss for Survival Analysis

Ranking loss specifically targets enhancing the discriminative power of survival models, which is crucial for better distinguishing between patients based on their risk. Suppose we are given the risk function, R , that quantifies the risk of a given patient with x at any time t . We aim to penalize instances where the risks assigned to a pair of patients are incorrectly ordered (i.e., assigning a lower risk to patient i who died before patient j ). This can be achieved by the ranking loss, which is formally given as follows:

̸

̸

where η is a function that relaxes the non-differentiable indicator function, I . Here, A i,j = I ( δ i = 1 , τ i < τ j ) indicates acceptable pairs whose assigned risks are comparable.

Different deep survival models have employed various types of quantities for implementing the ranking loss. For example, Lee et al. [1] have applied the risk function, R , and set η ( x, y ) = exp( -( x -y ) /κ ) , a convex function that both penalizes the wrongly ordered pairs and encourages the correctly ordered pairs. The same convex function has been employed utilizing the survival function, S [2]. Similarly, Steck et al. [13] utilizes η ( x, y ) = log σ ( y -x ) , which is a lower bound of the C-index, with the risk function, R . Chi et al. [3] employ the hazard function h with a linear ranking function η ( x, y ) = -( x -y ) .

Challenges. While combining NLL with ranking loss shows a notable improvement in discriminative power, it often comes at the expense of calibration performance, potentially harming the clinical utility of predicted survival outcomes. We suspect this is primarily due to how ranking loss directly modifies model outputs to order predicted risks, potentially leading to misalignment with the actual risk distribution. In this paper, we propose a different approach. Our method focuses on increasing discriminative power not by directly utilizing survival outcomes but by exploiting the embedding space through our novel contrastive learning framework. This approach preserves the calibration performance achieved by the NLL loss while enhancing discriminative capabilities.

4 Method

To address the challenges described above, we propose a novel Con trastive learning approach to a deep Sur vival model, which we refer to as ConSurv . The proposed method consists of three key components as illustrated in Figure 1:

Encoder (parameterized by θ ), f θ : X → H , takes features x ∈ X as input and outputs latent representations, i.e., h = f θ ( x ) .

Projection head (parameterized by ψ ), g ψ : H → R d , maps latent representations h to the embedding space where contrastive learning is applied, i.e., z = g ψ ( h ) .

Hazard network (parameterized by ϕ ), f ϕ : H×T → [0 , 1] , predicts the hazard rate at each time point t ∈ T given the input latent representation h , i.e., ˆ λ ( t | x ) = f ϕ ( h , t ) = f ϕ ( f θ ( x ) , t ) .

Motivated by the core concept of contrastive learning frameworks, we aim to differentiate each sample from other semantically different samples based on their survival outcomes . This allows us to overcome the limitation of ranking loss, which arises from the direct comparison of model outcomes in the form of risk/survival functions. Instead, we contrast samples in the latent space based on their survival outcomes and utilize the model outcome solely for the NLL loss to encourage better calibration.

4.1 Contrastive Learning for Survival Analysis

Our novel contrastive learning framework imposes lower penalties on potential false negative samples based on the similarities in their survival outcomes. More specifically, given an anchor sample, we define potential false negatives as samples with a small difference in the corresponding time-to-event.

Ouncensored

xcensored

M·.·

S(t]x)

LNLL

t

h

fp()

p(t|x).

Time

x

f()

h

gy()

Z

shared

shared

LSNCE

x

aug()

x

f()

h+

gy()

Z

LatentSpace

This inherently aligns with our inductive bias that patients with similar survival outcomes should share similar clinical status, which manifests through similar representations.

For each anchor sample x ∼ p X , the noise-contrastive estimation (NCE) objective aims to learn mapping f = g ψ ◦ f θ utilizing a positive sample with the same semantic meaning, i.e., x + ∼ p + , and negative samples with (supposedly) different semantic meanings, i.e., x -∼ q , as follows [20]:

where M is the scaling term which is set to the batch size and s : X × X → [ -1 , 1] is the similarity score between two samples. Here, we use cosine similarity in the embedding space, i.e., s ( x , x ′ ) = f ( x ) ⊤ f ( x ′ ) ∥ f ( x ) ∥∥ f ( x ′ ) ∥ . For notational convenience, we omit the corresponding temperature ν and write e s ( x , x + ) to denote e s ( x , x + ) /ν .

The key aspect in (4) for obtaining embeddings with discriminative power lies in how we select negative samples that the anchor sample should be distinguished from. To reflect the differences in the time-to-events in the embedding space, we design a novel negative distribution, q , by utilizing the available information from survival outcomes.

Importance Sampling using Survival Outcomes. To accurately distinguish patients based on their time-to-event outcomes, we fully utilize the time-to-event information for designing q based on the following inductive bias: similar patients are more likely to experience the event at similar time points than the ones who are not. Hence, given an anchor ( x , τ ) and a negative sample ( x -, τ -) , we define the weight function as follows:

where σ > 0 is a temperature coefficient. From this point forward, we will slightly abuse the notation and write w ( x -; x ) to denote w ( τ -; τ ) to be consistent with the notation used for the negative distribution. This function is a variant of the Laplacian kernel, where the weight increases as the time difference increases. That is, we assign larger weights for samples with large differences in the time-to-events, and smaller weights for samples with small time differences. Utilizing the weight function in (5), we can define the negative distribution, q , as:

where Z is the normalizing constant. Then, using the importance sampling technique, we can approximate the expectation of the similarity score of negative samples drawn from the negative distribution in (6) as:

where the normalizing constant for the empirical distribution can be given as Z = 1 M ∑ M j =1 w ( x -j ; x ) . Overall, the survival outcome-aware NCE (SNCE) loss in (8), mitigating the effect of potential false negatives that have similar survival outcomes, can be given as the following:

4.2 Handling Right-Censoring

However, in a survival dataset, there exist samples that are right-censored, offering partial information about the corresponding survival outcomes which indicates that the event will occur sometime after the censoring time. Consequently, not every pair in the survival dataset has comparable survival outcomes. Specifically, there are three different cases of sample pairs to consider:

Case 1: Both samples are uncensored (i.e., have observed events). We can directly compare the two time-to-events of any given pair.

Case 2: One is uncensored and the other is censored. Similar to the acceptable pairs in ranking loss, we can only compare a pair when sample i experiences an event while sample j is censored after that event time (implying no event has occurred by that time), i.e., τ i < τ j .

Case 3: Both samples are censored. No comparison is possible as we do not know when the actual events have occurred for both samples.

Considering these cases, we redefine the weight function considering the right-censoring as ˜ w ( τ j ; τ i ) = I i,j · w ( τ j ; τ i ) where I i,j indicates the comparable cases as follows:

where α is a hyperparameter that represents a margin to ensure a minimum time difference. Please see Appendix C.3 for more details about the effect of α .

4.3 Network Description

In this subsection, we provide a detailed network description of ConSurv and outline the learning procedure using a discrete-time survival dataset, D = { ( x i , τ i , δ i ) } N i =1 .

Negative Log-likelihood. Based on the encoder and the hazard network, we can provide the estimated hazard function given the input feature x at each time point t ∈ T as ˆ λ ( t | x ) = f ϕ ( f θ ( x ) , t ) . As our hazard estimate is defined as a function of time given an input feature, we can naturally model the time-varying effect of input features (thus, more complex relations) on risk/survival functions. Then, we can compute the NLL loss, L θ,ϕ NLL , in (2) by plugging in f ϕ ( f θ ( x ) , t ) into ˆ p and ˆ S as follows:

̸

Contrastive Learning. To train the proposed method using our contrastive loss introduced in (8), we construct augmented samples based on the state-of-the-art contrastive learning framework specifically designed for tabular data [26]. For each sample in a batch of M samples, we construct a corrupted version of the original sample following the marginal corruption process. So, given x i as an anchor, we set the corrupted sample, i.e., ˜ x i , as positive and all the other corrupted samples, i.e., ˜ x j for j = i , as negative. By passing the original, positive, and negative samples through f = g ψ ◦ f θ , we can compute our survival outcome-based contrastive learning loss function as defined in (8).

Overall, we can estimate the hazard function by training ConSurv with a loss function that combines the NLL loss and the SNCE loss as the following:

where β is a balancing coefficient that trade-offs between the two loss terms; the effect of β is provided in C.3. Please find the pseudo-code of ConSurv in H.

5 Experiment

In this section, we evaluate the performance of ConSurv and multiple survival models using several real-world clinical datasets. Further details about all the experiments are available in Appendix D.

5.1 Experiment Setup

Datasets. We compare our proposed method and the benchmarks with the following four commonly used real-world clinical datasets: METABRIC , NWTCO , GBSG , FLCHAIN , SUPPORT , and SEER . For detailed descriptions of these datasets, please refer to D.1.

Benchmarks. We compare ConSurv with six survival models that were selected based on their respective loss functions, which are critical for understanding trends in survival analysis performance. The evaluated models are based on i) partial log-likelihood including CoxPH [27] and DeepSurv [28], ii) ranking loss including DeepHit [1] and DRSA [10], and iii) calibration loss including DCS [2] and X-CAL [15]. Detailed methodological descriptions are provided in Appendix D.3.

Performance Metric. We evaluate the discriminative performance of ConSurv and benchmarks using the integrated time-dependent C-index (CI) [29] across all time points, where higher values indicate better performance. For calibration, we utilize the integrated Brier score (IBS) [30] and distributional divergence for calibration (DDC) [2], where lower values indicate better performance. Additionally, D-Calibration (D-CAL) [31] assesses calibration with results reported based on p -values; those exceeding 0.05 are noted as statistically significant. Comprehensive details on IBS, DDC, and D-CAL

250

250

250

200

eventtime

200

eventtime

200

eventtime

150

150

150

F100

100

100

50

50

50

0

0

(a) NLL only 400 epochs

(b)SNCE only 400 epochs

(c)NLL + SNCE 400 epochs

1.0

1.0

1.0

1.0

1.0

Observed Cumulative Fraction

0.9

DeepSurv

ConSurv

0.9

DeepHit

ConSurv

0.9

ConSurv

DRSA

0.9

ConSurv

DCS

0.9

ConSurv

X-CAL

0.8

0.8

0.8

0.8

0.7

0.7

0.7

0.7

0.7

0.6

0.6

0.6

0.6

0.6

0.5

0.5

0.5

0.5

0.4

0.4

0.4

0.4

0.4

0.3

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

0.1

0.0

0.0

0.0

0.0

0.0

0

N

m

Predicted CDF

6

00

2

La

Predicted CDF

4

00

2

m

Predicted CDF

一

0.0

23

Predicted CDF

4

6

D

D

D

0'0

m

Predicted CDF

6

D

7

(a)vsDeepSurv

(b）vs DeepHit

(c)vs DRSA

(d)vs DCS

(e) vs X-CAL

are provided in Appendix D.2. In Appendix C.1, we further provide time-dependent C-index [5] and time-dependent Brier score [30] at different time points.

5.2 Quantitative Analysis Results

We report the CI, BS, DDC, and D-CAL of our proposed method and the benchmarks in Table 1. The results in Table 1 demonstrate that our method significantly outperforms all the benchmarks in discriminating among individual risks at the evaluated all time points, consistently across the four real-world clinical datasets. ConSurv achieves such gains in the discriminative performance while providing the best or comparable calibration performance. Notably, as evidenced by the metrics IBS, DDC, and D-CAL, our method yields exceptional calibration performance. Contrarily, ranking loss-based models such as DeepHit and DRSA, which are designed primarily to enhance discriminative power, usually exhibit poor calibration. Even without being specifically designed through the loss function for calibration, our model outperforms benchmarks such as DCS and X-CAL, which incorporate loss functions used to enhance calibration power. This indicates that our approach inherently balances both discriminative and calibration power effectively.

Ablation Study. We conduct an ablation study to better understand the contribution of different loss functions within our framework employing the following variants of ConSurv: (i) L NLL only, (ii) L NLL & L NCE , (iii) L NLL & L Rank , and (iv) ConSurv (i.e., L NLL & L SNCE ). Table 1 highlights the performance trade-offs between discrimination and calibration power when different loss functions are utilized in survival models. Using only NLL loss typically leads to lower discriminative performance while maintaining reasonably good calibration. Augmenting a ranking loss to the NLL significantly enhances discriminative performance but decreases calibration. In contrast, our proposed SNCE loss significantly enhances the model's performance compared to (i) using NLL only and (iii) using NLL with ranking loss. This suggests that the SNCE loss not only boosts discriminative power but also preserves or even improves the model's calibration. Additionally, when compared to (ii) using NLL with InfoNCE, which pushes all negative pairs away equally, we confirm that ConSurv, which assigns weights to negative samples based on their time difference until the event, ensures better performance.

Subgroup:ER+

Subgroup: ER-

1.0

1.0

Survival Probability

0.8

0.8

0.6

KM

0.6

KM

ConSurv

ConSurv

0.4

DeepSurv

CoxPH

0.4

DeepSurv

CoxPH

DeepHit

DeepHit

0.2

DRSA

0.2

DRSA

DCS

SOO

X-CAL

0.0

X-CAL

0

50

100

150

200

0

50

100

150

200

time

time

Subgroup:HER2+

Subgroup: HER2-

1.0

1.0

Survival Probability

0.8

0.8

0.6

KM

0.6

KM

ConSurv

ConSurv

0.4

DeepSurv

CoxPH

0.4

Hdxo

DeepSurv

DeepHit

DeepHit

0.2

DRSA

0.2

DRSA

DCS

DCS

0.0

X-CAL

X-CAL

0

50

100

150

200

0

50

100

150

200

time time

5.3 Qualitative Analysis Results

5.3.1 Effect of Contrastive Learning

To further demonstrate how survival outcome-aware contrastive learning promotes discrimination among samples in the latent space, we compare two-dimensional t-SNE visualizations of latent representations trained with L NLL only, L SNCE only and ConSurv (i.e., L NLL & L SNCE ) for the METABRIC dataset in Figure 2. Here, we only display uncensored samples with event times for clarity. Figure 2 shows that when trained with L NLL alone, latent representations are mixed regardless of their corresponding event times. However, incorporating L SNCE into the loss function significantly improves the alignment of representations with event time information, enabling the predictor to better discriminate the risks associated with each sample. Notably, even when trained solely with L SNCE , latent representations exhibit a clear alignment with event times. This finding strongly suggests that our contrastive learning approach effectively encourages discrimination by capturing and reflecting the underlying event time information. (See Appendix C.2 for more datasets.)

5.3.2 Calibration Plot

Figure 3 shows calibration plots comparing the calibration of ConSurv with the deep learning-based survival models. Here, we evaluate calibration by matching predicted cumulative densities to observed event frequencies at the quantiles of the predicted cumulative density. In these plots, the x=y line represents the ideal state where predicted probabilities perfectly match the observed outcomes. The evaluated survival models show different trends depending on the loss function used for training. The ranking-based models (i.e., DeepHit and DRSA), which directly manipulate the model outputs, experimentally demonstrated the lowest calibration power. Contrarily, the calibration-based models (i.e., X-CAL and DCS), designed to improve calibration power, show relatively better calibration but do not match the calibration performance of DeepSurv, likely due to the assumption of proportional hazards. Compared to these survival models trained based on three different types of loss functions, our proposed method demonstrates calibration performance that is similar or even superior.

5.3.3 Subgroup Analysis

We further validate the calibration performance of the survival models by comparing their survival plots with the Kaplan-Meier (KM) curve, a non-parametric estimate of the survival function at a population level. For this comparison, we examine two binary hormone receptor statuses in the METABRIC dataset: estrogen receptor (ER) and human epidermal growth factor receptor 2 (HER2), which are crucial for determining hormone therapies and can result in significantly different prognoses for breast cancer patients. In Figure 4, we average the survival functions of different survival models based on the subgroups of patients with the corresponding feature values (e.g., ER+ vs ER-). The

results confirm that our method aligns well with the KM curve compared to other deep learning-based survival models, especially those with ranking loss.

To quantitatively assess calibration performance across different subgroups, we compare the survival predictions of each model with KM curves for each subgroup using the Wasserstein distance. This metric is well-suited for comparing survival curves as it considers both the overall shape and the discrepancies in predicted survival probabilities at each time point. The results in Table 2 demonstrate that our proposed model achieves well-calibrated predictions within each subgroup. (See Appendix C.4.2 for more subgroups.)

6 Conclusion

In this paper, we propose ConSurv, a novel contrastive learning approach for deep survival analysis. Our method leverages weighted sampling to incorporate the intuitive assumption that patients with similar event times likely share similar clinical statuses. Unlike most existing deep survival models, ConSurv does not directly manipulate hazard or survival predictions during contrastive learning. This approach allows our method to maintain well-calibrated predictions based on the negative log-likelihood loss. Experiments on multiple real-world datasets demonstrate the superiority of our method over state-of-the-art deep survival models, particularly in achieving superior calibration performance compared to ranking loss-based models.

Acknowledgments

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00358602) and the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2019-II190079, Artificial Intelligence Graduate School Program (Korea University).

References

Changhee Lee, William Zame, Jinsung Yoon, and Mihaela Van Der Schaar. Deephit: A deep learning approach to survival analysis with competing risks. In Proceedings of the 32th AAAI Conference on artificial intelligence (AAAI 2018) , 2018.

Fahad Kamran and Jenna Wiens. Estimating calibrated individualized survival curves with deep learning. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI 2021) , 2021.

Shengqiang Chi, Yu Tian, Feng Wang, Yu Wang, Ming Chen, and Jingsong Li. Deep semisupervised multitask learning model and its interpretability for survival analysis. IEEE journal of biomedical and health informatics , 25(8):3185-3196, 2021.

Eleonora Giunchiglia, Anton Nemchenko, and Mihaela van der Schaar. Rnn-surv: A deep recurrent model for survival analysis. In Proceedings of the 27th International Conference on Artificial Neural Networks and Machine Learning (ICANN 2018) , 2018.

Hajime Uno, Tianxi Cai, Michael J Pencina, Ralph B D'Agostino, and Lee-Jen Wei. On the c-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in medicine , 30(10):1105-1117, 2011.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020) , 2020.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020) , 2020.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Proceedings of the 34th Conference on Neural Information Peocessing Systems (NeurIPS 2020) , 2020.

Michael F Gensheimer and Balasubramanian Narasimhan. A scalable discrete-time survival model for neural networks. PeerJ , 7:e6257, 2019.

Kan Ren, Jiarui Qin, Lei Zheng, Zhengyu Yang, Weinan Zhang, Lin Qiu, and Yong Yu. Deep recurrent survival analysis. In Proceedings of the 33th AAAI Conference on Artificial Intelligence (AAAI 2019) , 2019.

Alicia Curth, Changhee Lee, and Mihaela van der Schaar. Survite: Learning heterogeneous treatment effects from time-to-event data. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021) , 2021.

Frank E Harrell, Robert M Califf, David B Pryor, Kerry L Lee, and Robert A Rosati. Evaluating the yield of medical tests. Jama , 247(18):2543-2546, 1982.

Harald Steck, Balaji Krishnapuram, Cary Dehing-Oberije, Philippe Lambin, and Vikas C Raykar. On ranking in survival analysis: Bounds on the concordance index. In Proceedings of the 20th Conference on Neural Information Processing Systems (NeurIPS 2007) , 2007.

Anand Avati, Tony Duan, Sharon Zhou, Kenneth Jung, Nigam H. Shah, and Andrew Y. Ng. Countdown regression: Sharp and calibrated survival predictions. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference (UAI 2020) , 2020.

Mark Goldstein, Xintian Han, Aahlad Puli, Adler Perotte, and Rajesh Ranganath. X-cal: Explicit calibration for survival analysis. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020) , 2020.

Humza Haider, Bret Hoehn, Sarah Davis, and Russell Greiner. Effective ways to build and evaluate individual survival distributions. Journal of Machine Learning Research , 21(85):1-63, 2020.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-n-contrast: Learning continuous representations for regression. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2024) , 2024.

Mohsen Nayebi Kerdabadi, Arya Hadizadeh Moghaddam, Bin Liu, Mei Liu, and Zijun Yao. Contrastive learning of temporal distinctiveness for survival analysis in electronic health records. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM 2023) , pages 1897-1906, 2023.

Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592 , 2020.

Mike Wu, Milan Mosse, Chengxu Zhuang, Daniel Yamins, and Noah Goodman. Conditional negative sampling for contrastive learning of visual representations. arXiv preprint arXiv:2010.02037 , 2020.

Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021) , 2021.

Ruijie Jiang, Thuan Nguyen, Prakash Ishwar, and Shuchin Aeron. Supervised contrastive learning with hard negative samples. arXiv preprint arXiv:2209.00078 , 2022.

Zhaoyang Zhang, Xuying Wang, Xiaoming Mei, Chao Tao, and Haifeng Li. False: False negative samples aware contrastive learning for semantic segmentation of high-resolution remote sensing image. IEEE Geoscience and Remote Sensing Letters , 19:1-5, 2022.

Lingling Xu, Haoran Xie, Fu Lee Wang, Xiaohui Tao, Weiming Wang, and Qing Li. Contrastive sentence representation learning with adaptive false negative cancellation. Information Fusion , 102:102065, 2024.

Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. In Proceedings of the 10th International Conference on Learning Representations (ICLR 2022) , 2022.

David R Cox. Regression models and life-tables. Journal of the Royal Statistical Society: Series B (Methodological) , 34(2):187-202, 1972.

Jared L Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC medical research methodology , 18(1):1-12, 2018.

Thomas A Gerds, Michael W Kattan, Martin Schumacher, and Changhong Yu. Estimating a time-dependent concordance index for survival prediction models with covariate dependent censoring. Statistics in medicine , 32(13):2173-2184, 2013.

Erika Graf, Claudia Schmoor, Willi Sauerbrei, and Martin Schumacher. Assessment and comparison of prognostic classification schemes for survival data. Statistics in medicine , 18(17-18):2529-2545, 1999.

Humza Haider, Bret Hoehn, Sarah Davis, and Russell Greiner. Effective ways to build and evaluate individual survival distributions. Journal of Machine Learning Research , 21(85):1-63, 2020.

Zifeng Wang and Jimeng Sun. Survtrace: Transformers for survival analysis with competing events. In Proceedings of the 13th ACM international conference on bioinformatics, computational biology and health informatics , 2022.

Changhee Lee, William Zame, Ahmed Alaa, and Mihaela Schaar. Temporal quilting for survival analysis. In The 22nd international conference on artificial intelligence and statistics , 2019.

Chirag Nagpal, Xinyu Li, and Artur Dubrawski. Deep survival machines: Fully parametric survival regression and representation learning for censored data with competing risks. IEEE Journal of Biomedical and Health Informatics , 25, 2021.

Christina Curtis, Sohrab P Shah, Suet-Feung Chin, Gulisa Turashvili, Oscar M Rueda, Mark J Dunning, Doug Speed, Andy G Lynch, Shamith Samarajiwa, Yinyin Yuan, et al. The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups. Nature , 486(7403):346-352, 2012.

Norman E Breslow and Nilanjan Chatterjee. Design and analysis of two-phase studies with binary outcome applied to wilms tumour prognosis. Journal of the Royal Statistical Society: Series C (Applied Statistics) , 48(4):457-468, 1999.

M Schumacher, G Bastert, H Bojar, K Hübner, M Olschewski, W Sauerbrei, C Schmoor, C Beyerle, RL Neumann, and HF Rauschecker. Randomized 2 x 2 trial evaluating hormonal treatment and the duration of chemotherapy in node-positive breast cancer patients. german breast cancer study group. Journal of Clinical Oncology , 12(10):2086-2093, 1994.

Terry Therneau et al. A package for survival analysis in s. R package version , 2(7), 2015.

William A Knaus, Frank E Harrell, Joanne Lynn, Lee Goldman, Russell S Phillips, Alfred F Connors, Neal V Dawson, William J Fulkerson, Robert M Califf, Norman Desbiens, et al. The support prognostic model: Objective estimates of survival for seriously ill hospitalized adults. Annals of internal medicine , 122(3):191-203, 1995.

Changhee Lee, Alexander Light, Ahmed Alaa, David Thurtle, Mihaela van der Schaar, and Vincent J Gnanapragasam. Application of a novel machine learning framework for predicting non-metastatic prostate cancer-specific mortality in men using the surveillance, epidemiology, and end results (seer) database. The Lancet Digital Health , 2021.

A Notations

The notations used in this work are listed in Table 3.

B Augmentation Method

In our experiments with tabular datasets, we compared augmentation methods, focusing on traditional noise injection (NI) and the state-of-the-art method, SCARF [26]. For noise injection, we introduced random Gaussian noise into the latent space, determining the optimal corruption rate through a hyperparameter search consistent with SCARF. SCARF consistently showed performance that was similar to or better than NI. Consequently, we adopted SCARF as our primary augmentation method for our experiments.

C Additional Results

C.1 Performance on Time-dependent C-index and Brier score

While the CI and IBS provide valuable overall assessments of a given survival model (over the entire time horizon), these metrics cannot fully capture variations in model performance across different time points. To address this, we included time-dependent performance evaluations, namely the time-dependent C-index [5] and the time-dependent Brier score[30] at three different time points (i.e., 25%, 50%, and 75% percentiles of time-to-events as in [32, 33, 34]. It is worth highlighting that utilizing these time-dependent metrics may reveal subtle differences that might be obscured when using CI and IBS alone. As shown in Table 5 and 6, we demonstrate the superiority of our method in discriminative power while preserving calibration. To evaluate the performance of survival models at various time points, we select the 25%, 50%, and 75%-percentile of each dataset.

C.2 Effect of Contrastive Learning

In Figure 5, When using L NLL alone, representations tend to cluster tightly in a narrow space regardless of time. However, when applying L SNCE (contrastive learning with a time weight) to L NLL , representations of features with similar times are more likely to be positioned close to each other.

C.3 Sensitive Analysis

Sensitive Analysis on Coefficient α . α is the margin to ensure a minimum time difference between samples in acceptable pairs. All the weights for the entire sample are precomputed. For setting the margin, we calculate the percentile for only the weights corresponding to Case 2 mentioned in section

60

60

60

50

50

50

eventtime

eventtime

eventtime

3

20

10

10

10

0

0

NLLonlyaftertraining

SNCEonlyaftertraining

(a)SUPPORT

160

160

160

140

140

140

eventtime

eventtime

eventtime

0 60

60

40

40

20

20

20

0

0

0

NLL only after training

SNCEonlyaftertraining

(b)FLCHAIN

4.2. By defining this minimum time difference, the margin aids in reducing the ambiguity in survival outcomes, leading to a more precise comparison of patient risks. Specifically, this enhancement is vital for improving the model's discrimination capabilities, as it removes potential false negative samples with little difference in their survival outcomes. In Table 7, to determine the sensitivity of the ConSurv with respect to the value of α , we report the performance results on four datasets by varying

time-to-censoring

time-to-event(groud-truth)

Time Difference

60

60

50

40

30

20

10

10

11

12

14

15

16

18

19

20

21

22

23242526272

0

(a)Censoredsampleswithoutmargin

Time Difference

6

60

504030

50

40

OE

℃

20

10

(b)Removed censored sampleswithmarginα

Time Difference

60504030

60

50

40

0℃

20

10

10

11

12

13

14

15

16

17

18

19

24

25

26

27

28

29

Patients Index (with Event)

(c) Censored samples within margin α

the value of α according to the following percentiles of time differences: { 5 . 0 , 7 . 0 , 10 . 0 , 12 . 0 , 14 . 0 } . Additionally, we compare the performance metrics with no margin (i.e., α = 0 ).

Effect of Margin α . In Figure 6, we compare time differences based on censoring times and the (unobserved) ground-truth event times. For each event sample, we compute the time differences between its event time and: 1) the (observed) censoring time of its corresponding comparable censored sample, and 2) the (unobserved) ground-truth event time of that same censored sample.

To show this effect, we have conducted an experiment using additional synthetic data by adapting the time-to-event generation process in DeepHit [1] for both event times T i and censoring times C i :

Here, we randomly generated 1,000 random samples, where a sample is censored when T i > C i .

Without a margin, censored samples with censoring times close to event times may be incorrectly treated as having similar event times, leading to very low weight assignments. To avoid this, we introduce a margin to exclude such samples when computing our contrastive loss. Overall, the margin in | τ i -τ j | ≥ α helps prevent failures in accurately distinguishing risk between comparable censored samples.

Sensitive Analysis On Coefficient β . β is the balancing coefficient in our overall loss function where increased β makes the impact of L SNCE more dominant. Table 8 shows that when β (using the optimal α for each dataset) is relatively small (here, β = 0 . 01 ), contrastive loss struggles to effectively learn similarities between survival outcomes, leading to poor discrimination performance. However, increasing the β ratio consistently improves performance. This suggests that our proposed contrastive learning approach does not prevent the survival model from learning effectively guided by the NLL loss.

C.4 Calibration

C.4.1 Calibration Plot

Figure 7 shows calibration plots of ConSurv in comparison with the benchmarks on all the evaluated datasets. Here, similar to our observation in Section 5.3.2, ConSurv consistently exhibits calibration performance that is similar to or better than the benchmarks, effectively balancing discriminative and calibration power.

CumulativeFraction

98888

ConSurv

ConSurv

ConSurv

ConSurv

ConSurv

CoXPH

CoxPH

CoxPH

CoXPH

CoxPH

0.6

0.6

0.5

0.4

Observed

0.3

0.21

0

12

3

6 8

6 6

0.0

0.0

0.0

0

89

Predicted CDF

Predicted CDF

9882

Predicted CDF

Predicted CDF

CumulativeFraction

Predicted CDF

ConSurv

ConSurv

ConSurv

ConSurv

ConSurv

DeepSurv

DeepSurv

DeepSurv

DeepSurv

DeepSurv

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.0

0.0

0

°

2

°

882

ConSurv

Predicted CDF

ConSurv

Predicted CDF

ConSurv

Predicted CDF

!

CumulativeFraction

ConSurv

Predicted CDF

CumulativeFraction

88

ConSurv

Predicted CDF

DeepHit

DeepHit

0.8

DeepHit

0.8

0.7

DeepHit

0.7

DeepHit

0.6

0.6

0.5

0.5

0.5

0.4

0.4

Observed

0.3

021

0.2

0.1

0.0

0.0

0.0

1.0

60

2

6

2

Predicted CDF

Predicted CDF

9882

PredictedCDF

Observed Cumulative Fraction

Predicted CDF

ObservedCumulativeFraction

9882

Predicted CDF

ConSurv

ConSurv

ConSurv

ConSurv

ConSurv

DRSA

DRSA

DRSA

DRSA

DRSA

0.7

0.7

0.7

0.6

0.7

0.6-1

0.5

0.5

0.4

0.4

0.3

E0

02

.2

0.2

0.0

0.1

o0

0.1

00

0.0

2

6 8

°

6

1.0

Predicted CDF

Predicted CDF

PredictedCDF

CumulativeFraction

Predicted CDF

CumulativeFraction

98888

Predicted CDF

ConSurv

ConSurv

ConSurv

ConSurv

ConSurv

DCS

DCS

DCS

DCS

DCS

!!

0.7

0.6

0.6

0.6

0.5

0.5

9.041

Observed

0.3

0.3

0.2

0.0

0.0

2

89

24

6 6

2

°

1.0

0.0

0.0

0

89

Predicted CDF

Predicted CDF

CumulativeFraction

9882

PredictedCDF

988

Predicted CDF

CumulativeFraction

9888

Predicted CDF

ConSurv

ConSurv

ConSurv

ConSurv

ConSurv

X-CAL

X-CAL

X-CAL

X-CAL

X-CAL

o.7

0.7

0.6

0.7

0.6

0.5

0.4

Observed

0.3

Observed

0.3

0.2

Predicted CDF

Predicted CDF

Predicted CDF

Predicted CDF

Predicted CDF

(a) NWTCO

(b) GBSG

(c) FLCHAIN

(d) SUPPORT

(e) SEER

C.4.2 Subgroup Analysis

In this section, we show that our model learns well about the differences between subgroups on realworld datasets, with good calibration performance by comparing the averaged survival predictions and the observed KM estimation within each subgroup. In Figure 8, we subgroup patients based on cellularity, which is a categorical variable representing the degree of density of cells in the tumor. Similarly, in Figure 9 (a), we partition patients based on the size of the tumor (with + indicating 70% or more and - indicating 30% or less of the total size), which have a significant impact on the event. In Figure 9 (b), we examine binary hormone receptor statuses progesterone receptor (PR).

Our results show that our method closely aligns with the KM curve, outperforming other deep learningbased survival models, especially those with ranking loss. We assess calibration performance using the Wasserstein distance to compare survival predictions with KM curves across different subgroups. In Table 9 indicate that our model provides well-calibrated predictions within each subgroup.

Subgroup:cellularity1

Subgroup:cellularity2

Subgroup:cellularity3

1.0

1.0

1.0

obability

0.8

0.8

0.8

Prob

0.6

0.6

0.6

Survival

0.4

KM

ConSurv

0.4

KM

ConSurv

0.4 -

ConSurv

KM

CoxPH

CoxPH

CoxPH

DeepSurv

DeepSurv

DeepSurv

0.2

DeepHit

DRSA

0.2

DeepHit

DRSA

0.2

DeepHit

DRSA

DCS

DCS

DCS

0.0

X-CAL

X-CAL

X-CAL

0

50

100

time

150

200

0

50

100

time

150

200

0

50

100

time

150

200

Subgroup:size+

Subgroup:size-

Subgroup:PR+

Subgroup:PR-

1.0

1.0

1.0

1.0

Survival Probability

0.8

0.8

0.8

0.8

0.6

0.6

90

0.6

KM

KM

KM

KM

0.4

ConSurv

CoxPH

0.4

ConSurv

CoxPH

0.4

ConSurv

CoxPH

0.4

ConSurv

CoxPH

DeepSurv

DeepSurv

DeepSurv

DeepSurv

0.2

DeepHit

DRSA

0.2

DeepHit

DRSA

0.2

DeepHit

DRSA

0.2

DeepHit

DRSA

DCS

DCS

DCS

DCS

X-CAL

+00

X-CAL

X-CAL

0'0

X-CAL

0

50

100

time

150

200

0

50

100

time

150

200

0

50

100

time

150

200

0

50

100

time

150

200

(a) size

(b) PR

C.5 Evaluating Performance Across Extended Seeds

For enhanced stability verification, we conducted additional experiments with 25 random seeds in Table 10.

D Dataset Description and Experimental Setup

D.1 Real-World Datasets

Table 11 shows baseline statistics of the real-world datasets. METABRIC [35] comprises clinical features and gene expression profiles (overall 21 features) used to determine the risk of breast cancer

subgroups for 1,981 patients. Among the total number of patients, 1,093 (55.2%) are censored, and 888 (44.8%) are uncensored. NWTCO [36] is a study related to Wilms' tumor, containing 4,023 patients with 6 clinical and histology features. Among the total number of patients, 3,457(85.8%) are censored and 571(14.2%) are uncensored. GBSG [37] investigates the effects of hormone treatment on recurrence-free survival time, with the event of interest being breast cancer recurrence. The dataset consists of 21 clinical and tumor-related features for 2,232 patients. Among them, 965 (43.2%) are censored and 1,267 (56.8%) are uncensored. FLCHAIN [38] contains subjects from a study of the relationship between serum-free light chain and mortality, where each subject is described by 8 features. Among 6,524 subjects, 1,962 (30.1%) are censored and 4,562 (69.9%) are uncensored. SUPPORT [39] focuses on the survival time of seriously ill hospitalized adults. The dataset consists of 14 clinical features for 8,873 patients, having 2,837 (31.9%) censored and 6,036 (68.1%) uncensored. SEER [40] provides data on prostate cancer patients, containing 12 clinical and cancer-related features for 54,544 patients. Among them, 604 (1.11%) are uncensored and 53,940 (98.9%) are censored. We split the data into train, test, and validation sets with a ratio of 0.64:0.20:0.16, and then apply min-max normalization to the input features.

D.2 Performance Metrics

Integrated Brier Score (IBS) is metric to evaluate the mean squared error between the predicted survival curve with step function of the observed event. The IBS is the integration of the Brier score across all time points.

Distributional Divergence for Calibration (DDC) is a novel metric introduced in [2] to assess the calibration quality of survival models. This metric computes the divergence between the empirical distribution of estimated survival probabilities at the observed event times and a uniform distribution. Specifically, DDC utilizes the Kullback-Leibler (KL) Divergence between these two distributions.

It first bins the estimated survival probabilities ˆ S ( t i | x i ) at the observed event times into ten equal-width intervals across the [0,1] range.

It then calculates the KL divergence D KL ( P ∥ Q ) between this binned distribution P and a theoretical uniform distribution Q .

DDC is scaled so that its values range between 0 and 1, with lower values indicating better calibration. A perfectly calibrated model, where the estimated probabilities perfectly match a uniform distribution, would yield a DDC value of 0.

DDCserves as a more nuanced measure compared to traditional metrics like the Brier score or C-index because it directly evaluates how closely the distribution of the model's probabilistic predictions matches the expected uniform distribution of true probabilities. This direct assessment helps in determining whether the model's predictions are systematically biased or well-aligned with actual outcomes.

D-Calibration (D-CAL) assesses how well predicted survival probabilities align with observed outcomes based on a goodness-of-fit test. D-CAL bins the predicted survival probabilities at the true event times into ten equal-width intervals from 0 to 1, and performs a chi-squared test to ascertain the uniformity of the distribution. As suggested in [31], we reports p -values from the D-CAL test and the number of datasets with p -values greater than 0.05.

D.3 About Benchmarks

Benchmarks. We compare ConSurv with six survival models including one traditional statistical method and four state-of-the-art deep learning methods: CoxPH [27] is a commonly used statistical method under the proportional hazard assumption 4 , DeepSurv [28] is an extension of the Cox model which utilizes a DNN to predict individual hazard rates 5 , DeepHit [1] is a DNN model which directly models the joint distribution of the event times 6 , DRSA [10] is an RNN-based model which predicts the likelihood of the true event occurrence and estimates the survival rate over time 7 , DCS [2] is an RNN-based model that estimates calibrated individualized survival curves while maximizing the

discriminative power 8 , and X-CAL [15] is a survival model designed to focus on the calibration performance by directly utilizing a differentiable version of D-CAL as a part of the objectives. 9 Especially with DCS and X-CAL , we made every effort to faithfully reproduce the code with hyperparameter optimization, as the original code had several errors and we were unable to use it for our experiments.

E Implementation

We discuss implementation of ConSurv below. The source code for ConSurv is available in https://github.com/dongzza97/ConSurv. Throughout the experiments, training ConSurv and its variants takes approximately 60 minutes to 2 hours on a single GPU machine. 10

E.1 Hyperparameter Specification

We perform a random search for hyperparameter optimization - including the batch size, hidden dimension, depth, learning rates, corruption rates, σ , α , and ν - on the validation set and choose the settings with the best performance for ConSurv on each dataset. Table 12 describes the model specifications for the evaluated datasets. The candidates for random search can be given as {32, 64, 128, 256}, {16, 32, 64}, and {3, 4, 5} for batch size, hidden dim, and depth respectively. We carefully choose the σ , α and ν for L SNCE over {0.25, 0.5, 0.75}, {5.0, 7.0, 10.0, 12.0, 14.0} and {0.03, 0.05, 0.07}. Corruption rates are selected from a range of 0.1 to 0.9. To train multiple components in our network simultaneously, we need to find different learning rates for each network f θ , g ψ , and f ϕ , where the learning rate is chosen from a set {1e-3, 1e-4}.

F Broader Impact

In this work, we highlight the direct and indirect influence of our approach through experiments on real-world clinical datasets. The use of these datasets was in accordance with the guidance of the respective data providers and domain experts. We acknowledge the potential for both positive and negative impacts associated with survival analysis techniques. While these methods can provide valuable insights for personalized treatment and risk stratification, we emphasize the importance of careful interpretation and validation by domain experts before applying these predictions in clinical practice. Misuse or misunderstanding of survival analysis could lead to unintended consequences for individuals or groups, such as inequitable access to care or discriminatory practices. Therefore, we advocate for transparency, accountability, and ongoing dialogue among researchers, clinicians, and the public to ensure the responsible and ethical use of survival analysis in healthcare.

G Limitation and Future Work

Survival analysis datasets often include features where a clear event time is not available, indicating a significant number of censored patients. This makes it challenging to assign weights based on time differences and to compare them against each other. Therefore, utilizing such data for learning

purposes presents difficulties. Overcoming these limitations requires further research. Potential avenues include modifying models to account for such uncertainties or developing alternative learning approaches. Additionally, there is a need for new evaluation metrics or model developments that consider the characteristics of such data.

Furthermore, the tabular datasets we utilized often lack diverse augmentation techniques. Future research could explore augmentation methods suitable for survival datasets to enhance contrastive learning performance. For example, techniques considering temporal variations or reflecting uncertainties in events could be investigated. Such research endeavors are expected to enhance understanding in the fields of survival analysis and contrastive learning, contributing to improved model performance.

H Pseudo-code for ConSurv

Algorithm 1 Pseudo-code for the ConSurv

Input: Dataset D = { ( x i , τ i , δ i ) } N i =1 , batch size M , margin α , coefficient β encoder θ , projection head ψ , hazard network ϕ , learning rate η 1 , η 2 .

Output: ConSurv parameters ( θ , ψ , ϕ )

Initialize parameters ( θ , ψ , ϕ )

repeat

Sample a mini-batch of

{

(

x

i

, τ

i

, δ

i

)

}

i M

=1

Draw augmentation function: s ∼S

Augmentation pair:

(

x

∼ D

i

)

i

x

,

x

+

i

← -

x

i

, s

Latent representation: h i , h + i ← -f θ ( x i ) , f θ ( x + i )

Map to the embedding space:

end for

+

i

Cosine similarity in the embedding space: s ( x i , x j ) = z ⊤ i z j / ( ∥ z i ∥∥ z j ∥ )

-

j

w

(

τ

;

τ

i

) = 1

-

e

-

σ

Weight function for negative distribution:

Redefined weight function considering the right-censoring: ˜ w ( τ -j ; τ i ) = I i,j · w ( τ -j ; τ i ) end for

Update the encoder and projection head parameter θ , ψ :

Compute latent representations: h i ← f θ ( x i )

Compute estimates for survival function: ˆ S i ( τ i | x i ) ← ∏ t ≤ τ i (1 -f ϕ ( h i , t ))

Compute estimates for probability mass function:

x

i

)

end for

t<τ

i

Update the encoder and hazard network parameters ( θ, ϕ ) :

until convergence

ˆ

p

i

(

τ

i

|

f

ϕ

(

h

i

, τ

i

)

∏

-|

τ i

τ

-

j

|

z

i

,

z

← -

g

ψ

(

h

i

)

, g

(1

-

f

ϕ

ψ

(

(

h

i

h

+

i

, t

)

))

̸

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims in the abstract and introduction correctly describe the paper's contributions. We provide experiments in Section 5 to support these claims.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have provided Limitations after the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In (6), we demonstrate how importance sampling can be used to approximate the negative distribution of our loss function, defined in (8).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Full experimental details are provided in Appendix D, and implementation details are provided in Appendix E.1. We have included our code in the supplementary material and plan to make it publicly available on GitHub upon acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code is currently included in the supplementary material and will be made available on GitHub upon acceptance. Detailed instructions are provided in the readme.md file within the supplementary material. Real-world datasets used in our experiments are cited with their respective references; no new datasets were generated for this work.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Complete experimental details, including training procedures and the hyperparameter configurations tested and selected for each dataset, are provided in Appendix E.1. Train/validation/test splits and preprocessing steps for each dataset are detailed in Appendix D.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In Section 5, we report the mean and standard deviation across five random training/validation/testing splits.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We conduct all the experiments on a local machine with CPU: Xeon Intel Xeon Gold 6240R and GPU: NVIDIA RTX A6000. Please refer to Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper fully complies with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have included a Broader Impact statement following the Conclusion. Given that both this research field and our specific work are still in the early stages of development and not yet deployed, we do not anticipate any immediate adverse effects.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our research does not entail such risks, as elaborated in the Broader Impact statement. We engage in foundational exploration of methodologies and do not proceed with deploying a fully trained model.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have appropriately referenced all methodologies and datasets used in our study. Whenever necessary, we have provided the licenses for datasets in the Dataset Details (Appendix D.1) and Implementation Details (Appendix E). Furthermore, we specify the date of access for any dataset or method utilized.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our novel contribution lies in our methodology and its practical application. The code accompanying this paper is available in the supplementary materials and will be made publicly accessible on GitHub upon acceptance.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our research does not rely on crowdsourcing; all datasets utilized are publicly accessible.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Since we employed openly accessible datasets that have been deidentified, there is no necessity for IRB approval for our study.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work concerns deep learning for survival analysis. Deep learning models for survival analysis are typically desired to have good discriminative performance, where a model can differentiate between patients of different risk profile, and calibration performance, where the time-to-event is accurately predicted by the model. Methods which seek to improve discrimination performance have often led to poorer model calibration. Consequently, this paper proposes an auxiliary contrastive loss to improve the discriminative performance of deep survival models, while still achieving high calibration performance. The novelty in this method is adapting the SOTA contrastive learning method for tabular data, SCARF, by weighting the negative pairs according to the difference in time-to-event. Hence, samples with greater difference in time-to-event are considered more important negative pairs, enforcing an inductive bias to differentiate the latent representations of samples with different time-to-event.  After providing a detailed problem formulation, the contrastive loss and generation of positive and negative pairs is described. The results of several experiments are provided which compare the proposed method with current state-of-the-art methods when applied to a number of real world datasets, and demonstrate the proposed model performance under ablation of each term in the loss function. From these experiments it is concluded that the auxiliary contrastive loss achieves superior discriminative and calibration performance.\n\nSTRENGTHS:\nThe strength of this work lies in its originality, quality and clarity. The paper proposes an interesting adaptation to SCARF, a contrastive method for tabular data, by weighting the negative pairs such that samples with greater difference in time-to-event are considered more important negative pairs. Recent work on contrastive learning and deep survival methods are cited and appropriately discussed. \n\nTechnically, the paper is sound, with a good mathematical foundation for the importance sampling in the weighted contrastive loss term, and a thorough set of experiments which establishes performance against important baselines and provides good support for modelling choices via ablation studies. Furthermore, the clarity of the paper is for the most part good: the description of the background, methodology, and experiments are clear and detailed.\n\nWEAKNESSES:\nWhile the technical parts of this paper are clearly explained, the organisation and presentation of the paper could be improved. In particular, the introduction uses quite a lot of terminology specific to survival analysis that is unexplained or explained in detail only later in the paper. As a specific example, censoring is first mentioned on line 67, but is not clearly explained. Explaining this terminology, which may not be familiar to all of NeurIPS wide readership, at first use would improve the clarity of the paper. Presentation and clarity of the paper would also be improved if all captions were self-contained. In some cases, for example in Table 1, it is difficult to interpret the results as presented, and this could have been avoided if the captions were more detailed. Specifically, it is not clear what the standard deviations represent in this table i.e. variation across model initialisations or data splits.   \n\nThis work should be praised for providing a list of experiments that should provide the evidence needed to make the claim that the proposed model provides superior discriminative performance while maintaining calibration performance. However, the clarity and quality of these experiments is sometimes below expectations, and this ultimately reduces the overall significance of this work. Table 1 is used to report that the proposed model is the best performing model. However, it seems that the model is within 1 standard deviation of the second best performing model, and hence the statistical significance of this result is unclear. As previously mentioned, while mean and standard deviation of performance metrics are reported, it is not entirely clear what distribution they refer to – whether it be different data splits, or model initialisations via random seed. This makes it difficult to interpret the significance of these results, and also makes it more difficult to reproduce. \n\nThe paper, therefore, addresses an important problem, proposes a novel and well thought out method to address this issue, but the significance is undermined by a lack of clarity and ambiguity surrounding the statistical significance of reported results.\n\nQUESTIONS:\nThe questions below focus on the details of the experiments that were unclear. Clarity on these questions may change the overall opinion of the paper. \n\nTable 1: In the caption you refer to the standard deviation - are the reported values in this table the mean? \nTable 1: Does the standard deviation refer to variation over different data splits, model initialisations or something else? \nTable 1: What do bold and underlined values represent? \nTable 1: How do you determine that the superior performance of ConSurv is statistically significant? \n\nLine 528: Can you confirm whether min-max normalisation was performed before or after data was split into train, test and validation sets? \n\nLine 236: What is the marginal corruption process? By what augmentation were positive pairs generated?\n\nLine 197: What does the temperature coefficient scale? What impact does this have on model training? \n\nLine 104: Does ref. [18] show that model training is destabilised by the contrastive loss? If not, what supports this statement? Could this model have been included in your benchmarks?\n\n\nNOTE: Due to the discussion in the author-reviewer discussion period, I feel that many of these points were clarified. In particular new results have better demonstrated the statistical significance of the performance of ConSurv. I raise my score from a 4 to a 6 to reflect this.\n\nLIMITATIONS:\nThe authors have addressed the potential negative societal impact of their work, stating that model predictions from such work should be under scrutiny by domain experts before being applied in a clinical setting. They stress that incorrect use could lead to inequitable access or discriminatory practices. While this is welcome, it is not specific to this work, but is a concern of all works proposing machine learning for clinical practice. A potential impact more specific to this work might consider why the inductive bias introduced by the auxiliary contrastive loss would lead to unintended consequences for AI-supported decisions on treatment and intervention. For example, the unsupervised nature of the contrastive learning procedure could align patients based on features which may act as a proxy for protected traits and lead to inequitable model performance amongst different demographics. \n\nAdditionally, a discussion of limitations is provided but is limited. The authors correctly identify that a significant number of patients are censored in real world datasets, and this undermines the importance sampling based on difference in time-to-event, as weights are assigned according to a function in equation (5). It would improve the discussion of the limitations if it was pointed out that seeking alternatives or amendments to (5) would be a specific avenue to improve upon this work. \n\nThe limitation section could benefit from greater specificity. The authors propose “modifying models to account for such uncertainties or developing alternative learning methods”. The uncertainties they refer to are unclear, and a preferred statement might be: future avenues of work would be to develop a model which retains the benefit of contrastive learning with respect to discriminative performance, whilst also being agnostic to the number of censored samples within the training set.\n\nThey then state “there is a need for new evaluation metrics or model developments that consider the characteristics of such data”. The clarity of this proposal might be clearer if stated with more specificity i.e. \"to develop a model that retains high discriminatory and calibration performance for datasets with arbitrary proportions of censored samples, new metrics must be established that can compare the same model applied to different proportions of censored samples.\"", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study discrete-time survival analysis, proposing to train models using a loss function that combines the NLL loss and a modified NCE loss. This NCE loss is modified to take the survival outcomes (event times) into account, mitigating the effect of potential false negatives that have small event time differences.\n\nThe method is evaluated on 4 tabular clinical survival datasets, comparing against 6 baseline models and 3 method variations. The proposed method performs well compared to all baselines.\n\nSTRENGTHS:\n- The paper is well-written overall.\n\n- The proposed method is described well in Section 4.\n\n- The proposed method is conceptually quite simple and intuitive. To add a contrastive loss term, and for this use a modified contrastive loss that takes the survival outcomes into account, makes intuitive sense.\n\n- The proposed method seems to perform well compared to baselines, both in terms of discrimination and calibration. In particular, it performs well compared to the baseline of replacing the proposed modified NCE loss term with standard NCE.\n\nWEAKNESSES:\n- The technical novelty/innovation is perhaps somewhat limited (\"just\" adding a contrastive loss term).\n\n- The experimental evaluation could perhaps be more extensive, include more datasets. Would have been nice to see the method being applied also to some non-tabular dataset.\n\n\n\n\nSummary:\n- Somewhat limited technical novelty, but this is a well-written paper with a simple/intuitive method that seems to perform well compared to reasonable baselines. Thus, I am definitely leaning towards accept.\n\nQUESTIONS:\n- Could you perhaps apply the proposed method also to some non-tabular dataset?\n\n- The visualization in Figure 2 is really neat, could you add this for the three other datasets as well (to the appendix)?\n\n\n\n\nMinor things:\n- 130: \"by minimizing the following negative NLL loss\", remove \"negative\"?\n\n- Perhaps add 1-3 sentences to the start of Section 3, just to describe what is covered here (and why this is covered, why this is relevant for your method)?\n\n- 248: \"We compare our proposed method and the benchmarks with...\", \"benchmarks\" --> \"baselines\"? The same also for line 251?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a contrastive learning loss to regularize maximum likelihood learning of discretized survival times. The main contribution of this work is the utilization of the Laplace kernel to weigh negative pairs inversely proportional to their time difference from the anchor sample, while also accounting for censoring. Specifically, negative pairs with event times close to that of the anchor sample contribute less to the contrastive loss. Experimental results on four datasets demonstrate that the proposed approach achieves a high Concordance Index (C-Index) without a significant loss in calibration when compared to baselines.\n\nSTRENGTHS:\nThe use of contrastive learning to improve model calibration in survival analysis seems interesting. Experimental results demonstrate the efficacy of this approach in maintaining a high C-index without significant loss in calibration, a trade-off that is difficult to achieve in practice.\n\nWEAKNESSES:\n*Given that the main contribution of this work is the application of contrastive loss to survival analysis, the paper lacks complete details for reproducibility:*\n- There are no details on how the model functions $ f_{\\theta}, f_{\\phi}$, and $g_{\\psi} $ are parametrized.\n- Details on the construction of positive pairs are very sparse. No motivation is provided in terms of why the SCARF approach is chosen and why it should be expected to work with survival data.\n- The paper proposes the Laplace Kernel for weighing samples; it is unclear if alternative kernels were also considered.\n- The notation seems sloppy; symbols ${f}$ and $Z$ are overloaded.\n- I expect calibration and the C-Index should be sensitive to $\\beta $; the paper does not discuss how to choose $\\beta $ and what values resulted in the performance results reported. This phenomenon has been discussed in detail in [1]. I encourage the authors to include a comprehensive discussion on this trade-off.\n- It is unclear how the margin parameter $\\alpha$ was chosen. I encourage the authors to provide visualizations in the main paper to demonstrate sensitivity to both $\\alpha$ and $\\beta $.\n\n*The proposed approach relies on discretizing event times, which is sensitive to the time-binning approach. Additionally, it seems the contrastive loss could be an indirect way to achieve consistency in event times. I encourage the authors to also benchmark with approaches that directly predict event times such as [2].*\n\n*The four datasets used seem to be of low sample size. I encourage the authors to also consider larger datasets such as SUPPORT and SEER.*\n\n**References**\n\n- [1] Qi et al., \"Conformalized Survival Distributions: A Generic Post-Process to Increase Calibration\", ICML 2024\n- [2] Chapfuwa et al., \"Calibration and Uncertainty in Neural Time-to-Event Modeling\", IEEE TNNLS 2020\n\nQUESTIONS:\n- What are the values of $\\alpha$ and $\\beta $ for the reported results?\n- Were alternative weighting kernels considered?\n- Were alternative approaches for constructing positive pairs considered?\n\nLIMITATIONS:\n- The paper does not discuss modeling assumptions; for example, the assumed independent censoring mechanism could be violated in practice.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper presents an approach to handle survival analysis datasets using a method to improve both calibration and discrimination. This adds a bunch of novelties to the field like handling right-censoring and an SNCE loss to handle calibration and ranking simultaneously (NLL + contrastive loss). This paper presents a reliable way to look at survival analysis, write a hazard function which is quite reasonable and also present a ranking loss to take care of the domain-specific problem of stratification by risk-levels using generated patient rank list.

STRENGTHS:
-> novelties added quite a lot: SNCE loss and handling right-censoring
-> ranking loss introduced can be generally used for any method aiming to generate or classify along ranked lists
-> well presented math for different loss functions and easy to follow

WEAKNESSES:
-> lack of using unstructured data
-> more clarity of censored data

QUESTIONS:
I understood why there is censored data, but if the authors could explain with an example of a time series what censored data looks like that would make it easier.

LIMITATIONS:
maybe added unstructured data like doctors notes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13