Conformalized Survival Analysis for General Right-Censored Data

Authors: Hen Davidov, Shai Feldman, Gil Shamai, Ron Kimmel, Yaniv Romano
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: conformal prediction, survival analysis, PAC, covariate shift, uncertainty quantification

Abstract:
We develop a framework to quantify predictive uncertainty in survival analysis, providing a reliable lower predictive bound (LPB) for the true, unknown patient survival time. Recently, conformal prediction has been used to construct such valid LPBs for *type-I right-censored data*, with the guarantee that the bound holds with high probability. Crucially, under the type-I setting, the censoring time is observed for all data points. As such, informative LPBs can be constructed by framing the calibration as an estimation task with covariate shift, relying on the conditionally independent censoring assumption. This paper expands the conformal toolbox for survival analysis, with the goal of handling the ubiquitous *general right-censored setting*, in which either the censoring or survival time is observed, but not both. The key challenge here is that the calibration cannot be directly formulated as a covariate shift problem anymore. Yet, we show how to construct LPBs with distribution-free finite-sample guarantees, under the same assumptions as conformal approaches for type-I censored data. Experiments demonstrate the informativeness and validity of our methods in simulated settings and showcase their practical utility using several real-world datasets.

Decision: accept (poster)
Reviewer scores: 3, 8, 8, 3

Reviews and discussion:
<thread>
creation date:
1729217507820

thread_type:
official_review

<thread_object>
creator: reviewer_bcin

<content>
summary:
The paper proposes a reliable lower predictive bound (LPB) for the survival time under the general right-censored data based on the work in Gui et al. (2024). The proposed LPB is a PCA-type with distribution-free finite-sample guarantee.

strengths:
1.	The LPB is doubly robust by theorem.
2.	The paper extends the result in Gui et al. (2024) to the general right-censoring setting.

weaknesses:
1.	I don't see a strong motivation for using the general right-censoring setting, which appears to be quite similar to the Type-I right-censoring setting. Other than two citations (which only mention the definition), the authors do not provide a clear explanation of why the general right-censoring setting is timely or important in real-world data.
2.	The entire framework builds on Gui et al. (2024), except for the miscoverage estimators. The proposed focus calibration and fused calibration use only part of the information in the calibration set, which may lead to a waste of information. Additionally, I have concerns about the sufficiency of the proposed calibrations in cases of a large imbalance between censored and uncensored data in the calibration set (e.g., when the censoring rate is high). Therefore, there are doubts regarding the novelty of the proposed method.
3.	Many parts of the paper are derived from Gui et al. (2024), including the main idea, the theoretical results, the simulations, and the proofs. Several sections are not clearly explained in this paper, requiring a read of Gui et al. (2024) to understand the context. For example, why is a PAC-type LPB considered instead of a marginally calibrated LPB? How does the PAC-type LPB feature in your method? As a result, phrase like "Notably, ..." in line 162-163 is vague and confusing.
4.	Section 3.2 presents a list of results without a clear interpretation of the underlying ideas. The nested notation further complicates the readability of the section.
5.	In your experiments, there is no comparison between your method and other methods, except the naïve LPB. The experimental section is mainly a collection of results without sufficient interpretations. For instance, the focused and fused LPBs are not consistently better than the naïve LPB in settings 4 and 6. In the real data experiment, the focused and fused LPB are not significantly better than the naïve LPB.
6.	The data used in your experiments does not reflect a general right-censoring setting.
7.	Some notations in the proof are clunky and confusing. For example, in the proof of Lemma B.1, $q=q(X)$ in your $\mathbb{P}_x(\cdot)$ 

is deterministic? In Section B.2, the form 
$\mathbb{E}[\mathbb{P}(\tilde{T}<\hat{q}_{\tau}(X)|X=x,e=1)]$ 

does not make sense, 
as $\hat{q}_{\tau}(X)$ is not random given $X=x$.

8. Some references are not in correct format. For example, the arxiv references should be in arxiv format.

questions:
1.	How do you handle the extreme $\hat{\omega}$?
2.	How does the estimator $\hat{s}_{\tau}(x)$ affect your method?
3.	The fused and focused LPB may not as tight as the LPB in Gui et al. (2024), then how loss are they? Can you quantify?
4.	In what settings, the fused and focused LPB are not better than the naïve LPB?
5.	Why the authors not include the average LPB in the experiments?
6.	What is the coverage result of the real data experiment? Can you run experiments on other real data set? For example, the data used in Candes et al. (2023) and Gui et al. (2024).


Emmanuel Cande`s, Lihua Lei, and Zhimei Ren. Conformalized survival analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(1):24–45, 2023.

Yu Gui, Rohan Hore, Zhimei Ren, and Rina Foygel Barber. Conformalized survival analysis with adaptive cut-offs. Biometrika, 111(2):459–477, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730519157250

thread_type:
official_review

<thread_object>
creator: reviewer_ovap

<content>
summary:
The manuscript is well-structured, offering solid theoretical support and a thorough experimental evaluation. The authors present a framework to quantify predictive uncertainty in survival analysis with general right-censoring, specifically addressing cases where the censoring time is not directly observed. The main contribution lies in adapting conformal prediction methods to this broader right-censoring context, providing a lower predictive bound (LPB) for patient survival times. Unlike previous approaches that assume known censoring times, this work extends the methodology to general right-censored data, where either the censoring or survival time is observed, but not both.

strengths:
- The paper extends conformal prediction methods to general right-censored data, addressing a significant gap where censoring times are unknown.
- The framework is backed by strong theoretical support, providing distribution-free, finite-sample guarantees for lower predictive bounds.
- The authors provide both simulated and real-world data experiments, showcasing the method’s validity and practical effectiveness.

weaknesses:
- In Section 4.1, the proposed methods appear to underperform compared to the naive method in Setting 6 regarding the lower predictive bound (LPB). Could the authors provide additional insights into why the naive method performs exceptionally well in this specific setting?

questions:
- This is a general question. Why do researchers tend to prioritize the lower predictive bound rather than the upper predictive bound for survival time? Is there any existing work on the upper predictive bound for survival time?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730524066794

thread_type:
official_review

<thread_object>
creator: reviewer_rrgu

<content>
summary:
This paper extends on Gui et al.'s paper (Conformalized survival analysis with adaptive cut-offs), to construct valid and reliable lower predictive bound for survival analysis in the more general right-censoring settings, where either censoring time or event time can be observed but not both. The paper provides distribution-free finite-sample guarantees under some necessary conditions, and provides some experiments to demonstrate the effectiveness.

strengths:
This paper is well-written and a pleasure to read. It tackles the important topic of achieving calibrated (probably approximately correct) predictions in survival analysis with the right censoring settings, an area where previous research has not provided methods with finite-sample distribution-free guarantees. The authors try to fill this gap by utilizing the concept of covariate shift. The paper is well-motivated and introduces three methods to address the problem. Starting with the simplest approach, it gradually highlights the limitations of each method and proposes solutions to overcome them. The authors provide finite-sample guarantees for the fused method, and the results on synthetic datasets support the theoretical findings.

weaknesses:
Despite all the merits I mentioned, I have concerns regarding the experimental results. The experiments are limited to oversimplified synthetic datasets and a single small real-world dataset, without comparisons to existing methods. This raises questions about the practical effectiveness and generalizability of the proposed approach.

Below are my major concerns with the paper:
1. **Lack of Comparison with Existing Methods:** 
The paper does not compare the proposed method with any existing approaches. Although it is the first to provide finite-sample guarantees, this does not inherently mean it outperforms methods without such guarantees. Previous studies, such as Cresswell et al. [1], have shown that methods lacking finite-sample guarantees can perform equally well or better in practice. It is crucial to include comparative analyses with methods like those proposed by Qin et al., Meixide et al., and Qi et al. [2] to compare the empirical performance of the proposed method.
2. **Simplistic Synthetic Datasets:** The main claim relies solely on 6 synthetic datasets with simplistic settings where survival and censoring times follow lognormal and exponential distributions. These settings are not reflective of real-world scenarios. For example, event and censor distributions can be non-parametric and censoring can stem from various sources (e.g., a combination of loss to follow-up, competing risks, and administrative censoring). I recommend designing more complex synthetic datasets that better mimic the complexities encountered in practice.
3. **Inadequate Evaluation on Real-World Data:**
The evaluation of the real-world dataset is insufficient. The TCGA dataset used is small (n=1044), and the chosen evaluation metrics do not adequately support the paper's main findings. There are established metrics for assessing coverage and calibration in censored datasets, such as the distribution calibration metric discussed in Haider et al. [3]. The distribution calibration is proved to be asymptotically unbiased under the conditional independent assumption (Assumption 1.1 in this paper). Although it is not perfect (as it doesn't have a finite sample guarantee), incorporating such metrics would strengthen the validity and reliability of the experimental results.
4. **Sensitivity to High Censoring Rates:** 
The proposed method employs inverse probability weighting, which is known to have high variance and be sensitive to the number of uncensored subjects, especially in datasets with high censoring rates. The paper should report the censoring rates in both synthetic and real datasets and assess the method's performance across varying levels of censoring. Additionally, testing on larger datasets would provide more robust evidence of the method's practical applicability.
5. **Challenges with Double Robustness in Practice:** While the double robustness property is theoretically appealing, achieving either condition in practical settings is challenging. The paper should discuss the practical implications of this limitation and explore potential strategies to mitigate its impact.

---

Below are my additional thoughts on the lower predictive bound (LPB) and marginal coverage/calibration:

Achieving a Probably Approximately Correct (PAC) lower predictive bound is important, but focusing solely on calibration or coverage is insufficient. Simple methods like the Kaplan-Meier estimator can provide calibrated predictions without the need for complex modeling. For example, a PAC-type LPB at 90% can be contained by finding the time when the KM curve drops below 90%.  A more important question is: how does the proposed method affect (or possibly decrease) other performance metrics by achieving a better calibration/coverage? Some important metrics include ranking accuracy (Concordance index), probabilistic accuracy (Brier score), and time-to-event prediction accuracy (mean absolute/squared error).

The requirement of a separate calibration set reduces the number of samples available for training, which could negatively impact predictive performance. Qi et al. [2] proposed a conformal-based method that addresses this issue by enhancing calibration without sacrificing other metrics. Discussing such alternatives would provide valuable context and highlight potential limitations for the current manuscript for improvement.

Moreover, providing only the LPB may not offer sufficient information for real-world applications compared to generating survival or hazard functions (which are the standard prediction outcomes for survival analysis models). These functions provide a comprehensive view of the time-to-event distribution, which is crucial for decision-making in clinical and other applied settings. LPB is a degenerated output that can be obtained from the survival function while losing a lot of other information.


[1] Cresswell et al., Conformal Prediction Sets Improve Human Decision Making. ICML 2024

[2] Qi et al., Conformalized Survival Distributions: A Generic Post-Process to Increase Calibration. ICML 2024

[3] Haider et al., Effective Ways to Build and Evaluate Individual Survival Distributions. JMLR 2020


---
**Update after rebuttal**
Thank you for your comprehensive rebuttal and the additional experimental results on complex synthetic datasets, real-world datasets, and supplementary baselines. Your responses have effectively addressed my concerns. I find the current draft to be a strong paper and am now more inclined toward acceptance. Accordingly, I am revising my rating to 8.

questions:
Lines 294-295 state "inaccurate estimation of $\hat{s}_\tau$ does not impact the validity of the calibration procedure", what do you mean by that exactly? Theorem 3.1 states that the calibration is achieved with *accurate weights*, which seems to contradict the first statement.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730592664452

thread_type:
official_review

<thread_object>
creator: reviewer_r8tu

<content>
summary:
This paper introduces an extended conformal survival analysis framework specifically designed to handle general right-censored data, aiming to provide a reliable lower predictive bound (LPB) for patient survival times. The main contribution of the paper is the development of a novel, doubly robust calibration method that constructs valid LPBs with finite-sample guarantees, even under uncertain censoring conditions.

strengths:
The paper proposes a doubly robust calibration method capable of constructing valid lower predictive bounds (LPBs) for general right-censored data, addressing a gap in survival analysis. The proposed method provides finite-sample guarantees, which are particularly important for ensuring model reliability in high-stakes survival analysis applications.

weaknesses:
Although the paper introduces a doubly robust calibration method, it fundamentally relies on existing conformal prediction methods (e.g., the work by Gui et al., 2024). The real innovation lies in extending these techniques to the general right-censored data setting, representing a relatively incremental contribution. The comparisons in the paper are primarily against a relatively simple "naive" calibration method rather than other state-of-the-art survival analysis methods.

questions:
1. In the "focused calibration" and "fused calibration" methods, the estimation of weights w(x) is crucial. How much impact does error in weight estimation have on the prediction results? Are there established standards for evaluating the accuracy of weight estimation, or strategies to reduce its sensitivity?

2. In the experiments, the proposed method is compared with the "naive method." Are there other commonly used methods for handling censored data that could serve as benchmarks? For example, traditional weighted Cox regression models or other survival analysis methods.

3. What is the computational complexity of the proposed "fused calibration" method when handling large-scale datasets in practice? How does its computational efficiency compare to other methods?

4. The proposed method relies on the estimation of weights and conditional quantiles; however, the article does not include a sensitivity analysis of these critical hyperparameters. How do the hyperparameters impact model performance? Is it possible to provide a robust hyperparameter selection strategy?

5. The effectiveness of doubly robust calibration depends on the accuracy of conditional quantile or weight estimation. Could explicit theoretical bounds be provided to describe the calibration method’s performance under inaccurate estimations? It is recommended to clearly specify how such biases impact the final coverage.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732130973939

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
GLOBAL RESPONSE: EXPERIMENTS

comment:
The main criticism raised was about the relatively simple synthetic experiments, limited comparison to baseline methods, and limited real-world data experiments. In response, we will revise the experimental section and include the following: 

(1) More realistic synthetic datasets: the synthetic datasets now include a more complex relationship of $(T,C) | X$, early censoring, early time-to-event, and end-of-trial censorship.

(2) We now include the results of the uncalibrated base model: a deepsurv model, which is a Cox model with a deep learning model instead of a linear one.

(3) We also include the conformalized survival distributions (CSD) method by Qi et al. (2024) for right censored data—an asymptotic calibration method.

(4) We have added 5 more real-world data sets, and compared the performance of our method to the uncalibrated base model and CSD methods.

(5) We conducted a robustness analysis of our methods to extreme censorship rates.

All these experiments, and additional ones that are more reviewer-specific, appear in a PDF titled “rebuttal_experients.pdf” attached to the Supplementary Material. We will include all the new experiments in the text; in the interest of space, we will migrate the synthetic experiments of the initial version of the paper to the appendix.

**Remark:** The supplementary PDF includes additional calibration metrics (c-index, IBS, and D-Cal) suggested by Reviewer rRgu. While these metrics are helpful for assessing distribution calibration, our experiments show that these metrics are not necessarily indicative of valid but conservative coverage. In the interest of space, we are not discussing this here and kindly refer to our response to Reviewer rRgu (Evaluating distribution calibration metrics) for details.

> Results: synthetic data

In all experiments, we fit a deepsurv model on 200 training samples, apply early stopping using a validation set to avoid overfitting, and have 1000 samples reserved for calibration. The precise description of the synthetic datasets is in Section A.1 of the supplementary “rebuttal_experients.pdf.”

Following Figure 1 in the supplementary PDF, we can see that our methods (naive, focused, fused) always attain valid coverage, as expected. Our advertised method—the fused approach—outperforms the naive and focused methods in terms of relative LPB (higher is better). In contrast to our calibration methods, the uncalibrated model does not attain valid coverage in all cases. Similarly, the asymptotic CSD method does not achieve valid coverage in all cases.


> Results: real data

In addition to the breast cancer dataset (‘TCGA’), we now added the ‘SUPPORT’ (n=8873), ‘METABRIC’ (n=1904), ‘GBSG’ (n=686), ‘NACD’ (n=2401), and ‘churn’ (n=2000) datasets available at github.com/shi-ang/MakeSurvivalCalibratedAgain and github.com/havakv/pycox. Refer to Qi et al. (2024) for more information about these datasets.
 
We follow the same training and calibration protocol as in the synthetic experiments, except that here, we randomly split the available data into disjoint training (60% of the data), validation (10%), calibration (20%), and testing (10%) sets. 

The results are summarized in Figure 3 in the supplementary “rebuttal_experients.pdf.” In contrast with the synthetic experiments, it is impossible to evaluate the coverage in such a real-world setting. With this in mind, we can see that the fused method performs the best among the three calibration methods we proposed, which are supported by finite-sample coverage guarantees. A conclusion that is in line with our synthetic experiments. The baseline methods—uncalibrated model and CSD—often attain higher LPBs, but these might be invalid; recall the synthetic experiments where these baseline methods do not always achieve the desired coverage level in practice.

> Results: Robustness analysis to extreme censorship rates

We evaluate the performance of our calibration methods for varying levels of censorship, using the dataset from Setting 3 of our new synthetic experiments. We vary the censorship rate by changing the end-of-trial time. 

We follow the same experimental protocol as in the synthetic experiment, and present the results in Figure 5 in the supplementary “rebuttal_experients.pdf.” As depicted in that figure, we can see that all the methods attain valid coverage regardless of the censorship rate, as expected. The fused approach outperforms the other methods, providing valid LPBs that are closer to the desired coverage level. The focused method performs similarly to the fused method, only under a low censorship rate, and the naive method is more conservative than the others.


**References**

Shi-Ang Qi, Yakun Yu, and Russell Greiner. Conformalized Survival Distributions: A Generic Post-Process to Increase Calibration. ICML, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732131017167

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
GLOBAL RESPONSE: CONTRIBUTIONS

comment:
We appreciate the reviewers' engagement with our paper and their valuable comments and suggestions. We will integrate their feedback into the revised paper.

We are glad that reviewers found our paper to be “well-structured” (Reviewer ovAp) and “well-written and pleasure to read” (Reviewer rRgu). We are also gratified that Reviewers ovAp and rRgu recognize the importance of our contribution and its novelty. To quote Reviewer rRgu: “[This paper] tackles the important topic of achieving calibrated (probably approximately correct) predictions in survival analysis with the right censoring settings, an area where previous research has not provided methods with finite-sample distribution-free guarantees.“ Thank you!

We also appreciate the concerns that Reviewers R8TU and bciN raised about the novelty of our method, and thank them for providing us the opportunity to elaborate both on the importance of our work and its key technical novelty.

> Importance of reliable survival analysis under general right-censored data
 
General right-censoring is a very common setup in survival analysis, where the observed data include either the event time $T$ or the censoring time $C$, but not both. We understand that this setup may be confusing, as it is often conflated with Type-I censoring. However, the Type-I setting—where the censoring time $C$ is always available—is a special case of the general right-censoring framework for which $C$ is observed only for surviving subjects. Therefore, the ability to provide reliable survival analysis for general right-censoring is crucial in many real-world scenarios, as recognized by Reviewer rRgu and Reviewer ovAp. To be sure, this setup is widely explored in the literature, see, e.g., Clark et al. (2003); Jenkins et al. (2005); George et al. (2010); Qi et al. (2024); Lagakos et al. (1979), to name a few papers. 

> Key novelty

We are the first to propose conformal prediction methods that handle the broader right-censoring setting, providing a lower predictive bound (LPB) for patient survival times with finite-sample guarantees.
While our framework builds on the foundations of Gui et al. (2024), it introduces significant innovations, particularly in the design of miscoverage estimators that do not require access to censorship times $C_i$ for all samples $i=1,...,n$. This stands in striking contrast with the miscoverage estimator of Gui et al. (2024), which can only be implemented when $C_i$ is known for all $i$. To be sure, the method by Gui et al. (2024) is not applicable under general right-censored data.
The transition from the naive approach to the focused method, and then to our advertised fused method is highly non-trivial: the challenge is to carefully select a subset of observations that make the LPB more informative while rigorously ensuring the validity of our methods. We motivated and backed our design choices analytically; see, e.g., Proposition 1.
We should also stress that the theoretical validity of our proposal does not trivially follow from Gui et al. (2024), and our analysis requires additional crucial steps. Indeed, the key theoretical challenge is to prove that even though $T_i$ and $e_i$ can be dependent conditional on $X$, our calibration methods are valid. This goes beyond the technical contribution of Gui et al. (2024).

**References**

TG Clark, MJ Bradburn, SB Love, and DG Altman, Survival Analysis Part I: Basic concepts and first analyses. British Journal of Cancer, 2003.

Stephen Jenkins. Survival analysis. 2005.

Brandon George, Samantha Seals, and Inmaculada Aban. Survival analysis and regression models. Journal of Nuclear Cardiology, 2010.

Shi-Ang Qi, Yakun Yu, and Russell Greiner. Conformalized Survival Distributions: A Generic Post-Process to Increase Calibration. ICML, 2024.

SW Lagakos. General right censoring and its impact on the analysis of survival data. Biometrics, 1:139-56, 1979.	

Yu Gui, Rohan Hore, Zhimei Ren, and Rina Foygel Barber. Conformalized survival analysis with adaptive cut-offs. Biometrika, 111(2):459–477, 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The work represents a interesting advancement in survival analysis by addressing the more realistic and challenging general right-censoring scenario, where either the survival time ($T$) or the censoring time ($C$) is observed, but not both. A core novelty is the introduction of the indicator variable $e$ (denoting whether the observation is censored or uncensored), which naturally encodes whether $T$ or $C$ is observed without requiring explicit access to $C$. This is a powerful and natural idea because $e$ reflects the fundamental structure of censored data and allows the framework to handle both cases robustly without making restrictive assumptions. The work further introduces novel calibration techniques—focused calibration for uncensored data and fused calibration for selectively incorporating censored data—that ensure statistical efficiency while maintaining finite-sample, distribution-free guarantees. Its double robustness property ensures valid results if either censoring weights or survival quantile models are well-specified, making it highly reliable even under model misspecification. By leveraging $e$, improving efficiency, and expanding applicability to complex, real-world censoring patterns, this work marks a foundational step beyond incremental improvements in conformal survival analysis.