Title: Uncertainty-Calibrated Prediction of Randomly-Timed Biomarker Trajectories with Conformal Bands
Venue: NeurIPS 2025
Accepted: yes
Keywords: Biomarker Trajectories

Conformal Prediction

Uncertainty Quantification
Abstract: We introduce a novel conformal prediction framework for constructing conformal prediction bands with high probability around biomarker trajectories observed at subject-specific, randomly-timed follow-up visits. Existing conformal methods typically assume fixed time grids, limiting their applicability in longitudinal clinical studies. Our approach addresses this limitation by defining a time-varying nonconformity score that normalizes prediction errors using model-derived uncertainty estimates, enabling conformal inference at arbitrary time points. We evaluate our method on two well-established brain biomarkers—hippocampal and ventricular volume—using a range of standard and state-of-the-art predictors. Across models, our conformalized predictors consistently achieve nominal coverage with tighter prediction intervals compared to baseline uncertainty estimates. To further account for population heterogeneity, we develop group-conditional conformal bands with formal coverage guarantees across clinically relevant and high-risk subgroups. Finally, we demonstrate the clinical utility of our approach in identifying subjects at risk of progression to Alzheimer’s disease. We introduce an uncertainty-aware progression metric based on the lower conformal bound and show that it enables the identification of 17.5\% more high-risk subjects compared to standard slope-based methods, highlighting the value of uncertainty calibration in real-world clinical decision making. We make the code available at \href{https://github.com/vatass/ConformalBiomarkerTrajectories}{\texttt{github.com/vatass/ConformalBiomarkerTrajectories}}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a nice application of conformal prediction, for the prediction of biotracker trajectories with random observation times.\nThe task is as follows: in an initial consultation (at time step t_0 = 0) we collect covariates $X$, potentially with some biomarker reading $Y_0$.\nWe then want to predict 'confidence bands' $C_t(X)$ for the trajectory $Y_t$ (for $t \\in [0, T]$ or some discretization thereof) such that, given some random times $\\mathcal{T} = \\{T_1, ..., T_k\\}$, it holds that\n$\\mathbb{P}[\\forall t \\in \\mathcal{T},\\ Y_t \\in C_t(X)] \\geq 1 - \\alpha$ for some $\\alpha$ chosen a priori.\nThe authors approach this via split conformal prediction techniques -- namely (vanilla) split conformal and group-conditional split conformal prediction -- with a conformity score designed for their setting.\nThey then go on to show the applicability of their approach in a real-world setting involving the prediction of brain biomarkers, highlighting the increased reliability when using their method, as well as the advantage of doing group-conditional CP with well-chosen covariates.\nThey conclude by showing how their uncertainty quantification can be used to confidently identify high-risk individuals, by building upon a metric that is reasonably well-established in the existing practice.\n\nSTRENGTHS AND WEAKNESSES:\nThe application presented in the paper is very nice, and the writing is easy to follow.\nThe methodology is sound, well-motivated and achieves good results.\nMy understanding of the specific example of brain biomarkers is limited, but the larger problem of biomarker prediction with random observation times is broadly applicable and relevant.\n\nMy main concern is that, at the end of the day, the proposed method reduces to simple split conformal and group-conditional split conformal prediction, treating the biomarkers at the random times as the labels. The only significant difference is a novel conformity score (which is a straightforward modification from existing ones, involving only adding a maximum over the time).\n\nThat said, the methodology and experimental validation are quite sound.\nThe section on group-conditional conformal prediction also cleanly presents the advantage of conditional guarantees. It's worth mentioning that the covariates used for the groups are some of the more comprehensive I have seen in applied conformal prediction (i.e., more than the usual applied CP paper), and clearly were carefully thought through.\n\nI think it's also worth noting that there is some more recent work on CP that might be of interest to the authors. For example, stronger conditional guarantees than simple group-conditional CP can be attained via the work of https://arxiv.org/abs/2305.12616.\nAlso, similar ideas of involving maxima/suprema in the computation of the conformity scores have been considered in other contexts, e.g. in strategic CP (https://arxiv.org/abs/2411.01596) and in CP with adversarial robustness (e.g. https://arxiv.org/abs/2503.05239).\n\nFinally, some minor nitpicks and suggestions:\n- line 153-154: $\\sigma$ might not be a function of (just) $\\hat{Y}_t^{(i)}$, right?\n- It would be best to make it clear in Problem 3.1 (line ~135) that though the guarantee cares only for $t \\in \\mathcal{T}$, the conformal bands are defined for all $t \\in [0, T]$ (or some discretization thereof).\n- Regarding the checklist:\n    - Question 2 is marked as [TODO], but was answered with [YES] (judging by the justification).\n    - Data&code: while I don't think it is essential that the code for this paper be reviewed, I strongly recommend that it be indeed published with the paper.\n    - CIs on metrics: the authors claim in the checklist to provide CIs of all metrics, however I cannot find them for the tables with Precision, Recall & F1 (e.g. Table 1, and in the supplementary material). These can be easily computed e.g. with bootstrap, right?\n    - Declaration of LLM usage is missing?\n\nQUESTIONS:\nMy main concern is as stated above, and led me to give a borderline accept rather than a straight accept.\nCould the authors perhaps elaborate more on either (i) how their methodology differs from existing CP methods besides the conformity score, and/or (ii) why the problem being solved would be of particular interest to the NeurIPS community, in contrast to the many other problems straight-forwardly solvable by CP? Are there any additional properties enjoyed by CP in the specific setting being considered?\n\nLIMITATIONS:\nLimitations were appropriately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to estimate continuous biomarker trajectories with uncertainty bands from sparsely and irregularly sampled data, such as patient visits during a chronic disease journey.  The technical novelty is to use conformal methods, which previously rely on fixed time points, but to extend the formulation to accommodate irregular time points.  The authors demonstrate the method on an Alzheimer’s data set using standard image-derive biomarkers, hippocampal and ventricular volume.  Results show that a method that uses the lower bounds of the uncertainty bands more reliably detects progressors from MCI to AD than a standard method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe formulation seems nice, although I am not particular familiar with conformal methods.\n\nThe problem is real – both the reconstruction of biomarker trajectories from irregular time points and the estimation of uncertainty bands – and has genuine utility in clinical and application and not doubt other application areas.\n\nResults show some potential benefit of the approach.\n\nWeaknesses\n\nThere’s no comparison against other methods that can estimate uncertainty bands.  The obvious choice would be Gaussian process models.\n\nFurther to the point above, one limitation of the proposed method is that it only considers biomarkers individually.  Other work on disease progression modelling – see https://pubmed.ncbi.nlm.nih.gov/38191721/ for an overview – typically creates models of multiple biomarker trajectories on the same time axis, which is more powerful.  In particular, Lorenzi et al’s Gaussian Process disease progression model https://www.sciencedirect.com/science/article/abs/pii/S1053811917307061, considers multiple biomarkers with uncertainty bands so seems on the face it more advanced than the proposed method.\n\nQUESTIONS:\nWhat advantages does the conformal method offer over alternatives that can estimate uncertainty bands such as the Gaussian process model mentioned above?\n\nLIMITATIONS:\nFine", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a conformal prediction method for longitudinal biomarker trajectories with irregular follow-up times. The proposed time-adaptive nonconformity scores leverage uncertainty estimates to enable conformal predictions of trajectory values at arbitrary time points. The method is demonstrated to outperform baseline approaches in terms of coverage and interval width on brain biomarker trajectories. \nTo address population heterogeneity (e.g., gender, race, and existing risk factors), the authors propose stratifying the calibration data by relevant groups. This approach is demonstrated to attain nominal coverage, unlike their unconditional (non-stratified)  counterparts.\nTo further demonstrate the clinical utility, the authors propose a *lower-bound rate of change* (LRoC) derived from the lower bound of the conformal prediction band, as a tool for identifying patients at high risk of Alzheimer’s disease.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n\n1. This work presents a novel extension of the conformal prediction framework to handle longitudinal data with irregularly spaced time points, a critical advantage in clinical and biomedical applications where sampling times are often inconsistent across patients. The reviewer agrees that this addresses an important limitation of existing approaches, which typically require uniformly sampled time points for all trajectories.\n\n2. The proposed method achieves nominal coverage with tight prediction bands compared to baseline approaches that rely on model-specific uncertainty estimates. This is demonstrated on a brain biomarker example, where the necessity of uncertainty calibration—particularly in the presence of population heterogeneity—is clearly illustrated through comparisons with baseline and/or non-stratified methods.\n \n**Weaknesses:**\n\n1. The method relies on time-wise uncertainty estimates to construct normalize nonconformity scores, yet it does not explicitly account for registration problem (e.g., misalignment due to irregular or asynchronous sampling schedules). Such misalignment could distort both the estimated mean trajectory and the uncertainty estimates, which assume a consistent temporal structure. It is unclear whether the authors considered how registration (or lack thereof) might affect the validity of these uncertainty estimates in their data analysis. This omission raises questions about whether the current approach adequately captures the true variability in trajectory dynamics or whether the results could be sensitive to unaccounted temporal misalignment. Future work could explore the impact of registration problem on the proposed uncertainty-aware predictions.\n\n2. The specification of groups (e.g., line 263) is not thoroughly discussed in the manuscript, despite being a critical modeling choice that could influence results. See also my later Question 2. The authors could benefit from a deeper discussion in Section 8.\n\n3. The contrast between RoC (rate of change) and LRoC could have been more effectively leveraged to highlight the clinical value of the proposed uncertainty-aware metrics. However, the current presentation (e.g., Table 1) may not fully clarify such advantage, as the gain in recall could stem from either reduced precision or improved discriminative power. Alternative summaries, such as ROC-AUC or PR-AUC, would provide a more comprehensive evaluation of performance across thresholds, rather than focusing solely on a single Youden-optimized $\\tau^*$ point.\n\nQUESTIONS:\n1. Figure 2, what the mean interval width is averaging over, the subject index $i$, the time index $t$, or both? If averaging over $t$ is involved, does the irregularly sampled $t$ affect the result?\n\n2. Figure 4, while coverage is attained, why is it further away for the nominal level compared to those in figure 2? Is this due to the baseline prediction model $\\hat Y$ and $\\sigma$ not using the grouping $G$? If so, why the baseline models not using them?\n\n3. Line 349, how is the prediction horizon $t_N$ specified? Further, RoC's current definition in line 361 -- 364 is ambiguous. It is beneficial to add formula for RoC as well, preferably side-by-side to that of LRoC in (3).\n\n4. Line 370, trial eligibility? Isn't this $\\tau$ is just used for identifying high-risk or not?\n\n5. While Table 4 in the appendix includes all results utilizing different base predictors, why Table 1 only shows DRMC? \n\n6. Is it possible to also show threshold $\\tau^*$ in certain way in figure 5 for better illustration?\n\nLIMITATIONS:\nThe manuscript lacks a dedicated discussion of limitations, despite Section 8 providing some directions for future work. While these future directions may implicitly highlight constraints of the current approach, a more explicit and constructive elaboration of limitations, particularly in light of the discussed weaknesses, would be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to construct conformal prediction bands for randomly timed trajectories, where outcomes are observed only at random time points and may neither be consecutive nor follow any fixed time grid. The authors propose to utilize normalized nonconformity scores originally introduced in [Yu et al.], in conjunction with the standard split conformal prediction pipeline. Furthermore, they extend their study to group-conditional conformal prediction, leveraging the method from [Romano et al.]. The proposed methodology is applied to two well-established longitudinal clinical datasets. Additionally, the authors introduce a new metric based on conformal prediction bounds to assess the risk of progression from Mild Cognitive Impairment (MCI) to Alzheimer’s disease.\n\n[Yu et al.] Xinyi Yu, Yiqi Zhao, Xiang Yin, and Lars Lindemann. Signal temporal logic control synthesis among uncontrollable dynamic agents with conformal prediction.\n[Romano et al.] Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel Candès. With malice toward none: Assessing uncertainty via equalized coverage\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \nThe paper is clearly written, well-structured, presents a clear problem statement, and is easy to follow.\n\nWeaknesses: \n1. Novelty: My main concern lies with the paper's technical novelty. The proposed method essentially applies existing conformal prediction frameworks (as introduced in [Yu et al.]) to the setting where outcomes are observed at irregular, random time points. The normalized nonconformity score presented in Equation (2) appears to be identical to that of [Yu et al.], and the overall methodology follows the standard split conformal prediction paradigm. Similarly, the group-conditional conformal procedure mirrors the approach in [Romano et al.] without apparent modification. That being said, while the setting of randomly-timed observations is practically relevant and underexplored, the methodological contribution appears to be more an adaptation rather than an innovation. If there are subtle but important theoretical or algorithmic contributions to handling the randomness in time stamps, I encourage the authors to clearly articulate and highlight them.\n2. Empirical Evaluation: Some aspects of the experimental results are counterintuitive. If I have misunderstood, I would appreciate clarification. In particular, regarding Figure 4: when reporting stratified coverage by features such as Race, did you partition the test set into mutually exclusive and collectively exhaustive groups (e.g., Asian, Black, and White)? If so, it is puzzling that the conformal coverage appears to fall below the nominal level (e.g., 90%) for all subgroups. If each subgroup undercovers, then by the law of total probability, the overall marginal coverage would also be below target - contradicting the guarantees of marginal conformal prediction. A more detailed discussion or clarification of this point would be helpful.\n3. Limitations: The paper does not provide a dedicated discussion on its limitations.\n\nQUESTIONS:\n1. Compared to [Yu et al.], what unique challenges arise in extending the method to randomly-timed observations, aside from substituting the predictive model with one capable of handling such data?\n2. In Figure 2, could you please report the standard deviations or confidence intervals associated with the coverage estimates?\n\nLIMITATIONS:\nNo, please refer to the weaknesses section for further details.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}