Title: Scalable and adaptive prediction bands with kernel sum-of-squares
Venue: NeurIPS 2025
Accepted: yes
Keywords: kernel sum-of-squares

uncertainty quantification

conformal prediction

kernel methods
Abstract: Conformal Prediction (CP) is a popular framework for constructing prediction bands with valid coverage in finite samples, while being free of any distributional assumption. A well-known limitation of conformal prediction is the lack of adaptivity, although several works introduced practically efficient alternate procedures. In this work, we build upon recent ideas that rely on recasting the CP problem as a statistical learning problem, directly targeting coverage and adaptivity. This statistical learning problem is based on reproducible kernel Hilbert spaces (RKHS) and kernel sum-of-squares (SoS) methods. First, we extend previous results with a general representer theorem and exhibit the dual formulation of the learning problem. Crucially, such dual formulation can be solved efficiently by accelerated gradient methods with several hundreds or thousands of samples, unlike previous strategies based on off-the-shelf semidefinite programming algorithms. Second, we introduce a new hyperparameter tuning strategy tailored specifically to target adaptivity through bounds on test-conditional coverage. This strategy, based on the Hilbert-Schmidt Independence Criterion (HSIC), is introduced here to tune kernel lengthscales in our framework, but has broader applicability since it could be used in any CP algorithm where the score function is learned. Finally, extensive experiments are conducted to show how our method compares to related work. All figures can be reproduced with the accompanying code at https://www.gitlab.com/drti/ksos-bands.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method for building adaptive prediction intervals using conformal prediction. \nThe main idea is to learn both a mean function and a non-negative scaling function using a kernel sum-of-squares framework. \nThe method produces adaptive intervals that adjust to the local uncertainty in the data and still guarantee valid coverage. \nThe authors provide a solid theoretical foundation, a dual formulation that scales well, and a new tuning strategy based on HSIC to guide adaptivity.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper's biggest strength is how well it connects ideas from different areas—conformal prediction, kernel methods, and structured low-rank learning—and turns them into something coherent. \nIt doesn't just throw these tools together. Each part is motivated: kernel SoS is used to learn a non-negative function, the split CP step gives finite-sample guarantees, and HSIC is brought in to tune adaptivity. \nThe method is scalable too, thanks to a dual formulation that avoids solving SDPs. \nAnd the hyperparameter section is thoughtful. The authors actually explain what each parameter does and how they interact.\n\nThat said, the evaluation is limited. \nAll experiments are done on synthetic data. \nThese examples are useful for showing how the method behaves under different types of noise, but they don’t really tell us how it handles the messiness of real-world data—things like outliers, covariate shifts, or higher-dimensional structure. Even a single real regression benchmark would have gone a long way. \nAs it stands, the results mostly validate the theory, not the practical utility.\n\nThere's also no discussion of limitations, as requested.\nSome of these issues are mentioned in passing (like the difficulty of local coverage), but there's no structured effort to acknowledge the boundaries of the approach. \nA short limitations section would have made the paper feel more complete and self-aware.\n\nQUESTIONS:\nSee Strengths and Weaknesses\n\nLIMITATIONS:\nThe paper is missing a clear limitations section. The authors should briefly mention key assumptions and when the method might not work well. Even a short note would help readers understand the boundaries of the approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the lack of adaptivity problem in conformal prediction by reframing the construction of prediction intervals as a constrained learning problem in RKHS. Specifically, the authors introduce adaptivity to the score function by estimating the uncertainty measure ($\\sigma^2$) through kernel sum of squares, which is a natural technique for positive function estimation. The new algorithm is termed, kernel sum of squares (kSoS). The authors derive a new representor theorem for the band prediction problem, which shows the unique solution is supported on the finite samples. Dual formulations are provided, which can be solved through accelerated gradient descent, therefore achieving significant speed up. Furthermore, by linking local coverage to HSIC, the authors provide a principled way for hyperparameter tuning (kernel bandwidth). Synthetic data simulation is provided to show the efficacy of the proposed kSOS algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The paper is mostly well written and easy to follow.\n2. Strong theoretical results are provided. The new representor theorem ensures that the algorithm can operate under finite dimensional case. Dual formulation achieves impressive speed up. The HSIC based hyperparameter tunning also intuitively makes sense.\n\nWeakness:\n1. (Major) The paper only have a set of synthetic data simulation and lacks real data simulation. One suggestion is to add a small set of real data simulation comparison just like [1]\n[1] Romano, Yaniv, Evan Patterson, and Emmanuel Candes. \"Conformalized quantile regression.\" Advances in neural information processing systems 32 (2019).\n2. (Major) Following weakness 1, the actual merit of Prop. 4 and the hyperparameter tuning strategy is questionable. Since the HSIC is simply appearing as part of the lose lower bound for local coverage, there is no guarantee that optimizing $\\theta_f$ according to HSIC will not harm the actual performance. With very limited synthetic data simulation, the claim that such a tuning scheme would work is not well supported.\n3. (Minor) The choice of hyperparameters (other than tunable parameters) are discussed in Section 3.2 with claims supported in Section 4. However, with the lack of real data simulations, it would be difficult to judge the claims. (For example, b=10 is sufficiently large)\n4. (Minor) Notation definitions can be improved. For example, nuclear and Frobenius norm are defined after Eq. (4), but used as early as Eq. (2). Hyperparameter a and b are not defined.\n\nQUESTIONS:\nPlease see weakness.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConformal Prediction (CP) is a popular framework for uncertainty quantification in machine learning, valued for its ability to provide valid, non-asymptotic coverage guarantees regardless of the data distribution or underlying model. However, standard CP methods do not adapt prediction intervals to individual inputs, which has motivated recent work on adaptive extensions. This paper introduces a new adaptive CP method based on kernel sum-of-squares, a technique for approximating nonnegative functions in a reproducing kernel Hilbert space. The key idea is to learn the CP score function by solving an optimization problem regularized with a kernel sum-of-squares term (eqs. 4 to 6), which enables the construction of adaptive prediction bands. The authors generalize the representer theorem from Marteau-Ferey et al. (2020) and Muzellec et al. (2022) to their CP setting (Theorem 2), and derive a dual formulation that can be efficiently optimized with gradient-based methods, making their approach scalable to large datasets. They also study the influence of each hyperparameter and propose a novel tuning practical method using the Hilbert-Schmidt Independence Criterion (HSIC) to target improved marginal coverage in local regions of the test point, based on the theoretical bound in Proposition 4. Finally, they empirically evaluate their method in synthetic regression problems and compare against a few methods (Conformalized Quantile regression with random forests, and rescaled scores with Gaussian Processes) in terms of mean width, mutual information and local coverage.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper is very well-written, with a clear motivation that aligns well with the interests of the NeurIPS community\n- The theoretical contributions and the derived methodology appear sound and well-justified\n- I particularly appreciate the authors' efforts in providing an in-depth discussion of their framework, including Section 3.2, where they examine the influence and interactions of the multiple hyperparameters. This reflects their commitment to understand the method's behavior and improving its interpretability. Additionally, their analysis leads to novel theoretical results with potential relevance beyond the scope of this work. For example, the discussion on hyperparameter tuning directly motivates Proposition 4, which establishes weak conditional validity based on a lower bound involving HSIC, and could be of interest for other adaptive CP methods\n- The empirical results show that the proposed CP method is promising\n\n### Weaknesses\n\n1) Some aspects of the theoretical coverage guarantees would benefit from further clarification:\n\na) The proof of Proposition 3 is missing. While this suggests that the result is a trivial extension of standard CP arguments to their adaptive setting, it would be helpful for the authors to explicitly explain why this is the case. In particular, they should clarify how the exchangeability of the learned scores is preserved, and why their approach based on a simple split conformal scheme suffices, whereas prior related works have relied on more elaborate calibration procedures [Liang, 2022; Fan et al., 2024].\n\nb) In Proposition 4, HSIC is actually subject to a constraint to ensure that the square root expression remains defined (i.e., its argument is nonnegative). Since this constraint depends on the value of $\\alpha_2$, and hence on the kernel choice, it would be useful to discuss whether this restricts the applicability or effectiveness of the theoretical bound. If I am not mistaken, this aspect is not addressed in the paper, and it raises the question of whether there exist scenarios in which the proposed methodology (based on maximizing HSIC) may be affected.\n\n\n2) The experimental evaluation is promising but limited in scope:\n\na) Some implementation details could be clarified. For instance, the choice of kernel used in their experiments is not given, and the sensitivity of the results to this choice is not discussed. Additionally, while Figure 3 claims scalability to \"thousands of training points\", Figure 3 left goes up to $n=1000$, which limits the strength of this claim.\n\nb) More importantly, the experiments are conducted only on low-dimensional (mostly $d=1$ and up to $d=5$) synthetic regression tasks with relatively small to moderate sample sizes. While the main text reports figures showing that the proposed method performs the best, the supplementary material questions this conclusion. For example, in Cases 3 and 4, the kernel SoS CP does not always outperform competing methods. In particular, in Case 3, heteroscedastic GP performs better than kernel SoS; in Case 4 (or Case 2 with larger dimensions, in light of Remark 2) where the ideal prediction intervals are constant, kernel SoS appears to struggle in capturing this, which seems a bit counterintuitive as I would expect its adaptivity to naturally account for such case. This makes me question the relevance of kernel SoS for more realistic or complex benchmarks.\n\nQUESTIONS:\nGiven my comments in \"Strengths And Weaknesses\", I would be glad if the authors could address the following.\n\n**Q1.** On clarifying Proposition 3: why is the exchangeability assumption preserved when applying a standard split conformal scheme to their adaptive learned score function? How does this justify bypassing more elaborate calibration procedures used in prior work?\n\n**Q2.** On the validity of the HSIC-based bound (Proposition 4): how does the dependence of this bound on the kernel and the parameter $\\alpha_2$ influence its applicability? Are there cases where the square root constraint limits the practical use of this result?\n\n**Q3.** What kernel function was used in the experiments, and how sensitive are the results to this choice?\n\n**Q4.** Could the authors discuss and illustrate the relevance of their method to more realistic or complex regression tasks?\n\nLIMITATIONS:\nWhile the authors offer a detailed discussion of their method, the paper would benefit from a more explicit acknowledgment of its limitations, maybe through a short dedicated paragraph toward the end.\n\nConcerning potential negative societal impacts, I did not find a specific discussion on this topic. However, I think the content of this paper does not raise any major concerns in this regard.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConformal prediction provides a framework to construct prediction bands with coverage guarantees. This paper builds on adaptive prediction band methods for regression tasks. Specifically, the paper targets coverage and adaptivity using reproducible kernel Hilbert spaces and kernel sum-of-squares methods. It derives a dual formulation for efficient optimization. Further, it tunes hyperparameters to provide better adaptivity.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__\n\n1.\tThe paper studies the problem of adaptive prediction bands for regression problems, which is of practical relevance.\n\n2.\tThe paper is well written and is easy to understand.\n\n3.\tThe theoretical results are promising. They are also well explained in the paper, providing a good intuitive understanding of the same.\n\n__Weaknesses__\n\n1.\tThe experiments include results on synthetic datasets, good for analyzing the behavior of the proposed algorithm. However, experiments on real-world datasets are highly encouraged and will add to the comparison between the proposed algorithm and baseline methods.\n\nQUESTIONS:\n__Questions and Comments__\n\n1.\tHow is the local coverage metric computed?\n\n2.\tWhat is the reason for opting to have a 100% coverage constraint on the training data (Eq. 5)? Is it for convexity reasons only?\n\n3.\tIt is not easy to read the legends of the figures in the main paper.\n\n4.\tIt is worth highlighting that the width in lines 91-92 is constant for a fixed train and calibration data, not otherwise.\n\n5.\tIt is worth highlighting that the conformal prediction procedure detailed in Section 2.1 assumes $\\alpha \\geq 1/(m+1)$.\n\n6.\tThe paper uses the nuclear and Frobenius norms for the first time in Eq. 2. It would be helpful to define the two norms there.\n\n__Typos__\n\n1.\t\"Then Equation (4) admits a unique solution...\" $\\rightarrow$ \"Then the problem defined in Equations (4), (5), and (6) admits a unique solution...\" (lines 219-220).\n\nLIMITATIONS:\nA more thorough discussion on limitations would benefit the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-21"}