Title: Valid Selection among Conformal Sets
Venue: NeurIPS 2025
Accepted: yes
Keywords: Conformal Prediction

Selection after Inference

Uncertainty Quantification
Abstract: Conformal prediction offers a distribution-free framework for constructing prediction sets with coverage guarantees. In practice, multiple valid conformal prediction sets may be available, arising from different models or methodologies. However, selecting the most desirable set, such as the smallest, can invalidate the coverage guarantees. To address this challenge, we propose a stability-based approach that ensures coverage for the selected prediction set. We extend our results to the online conformal setting, propose several refinements in settings where additional structure is available, and demonstrate its effectiveness through experiments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of constructing a conformal set when K different conformal predictors are given. Both the batch setting (i.e. standard split conformal prediction) and the online setting (without stationarity assumptions) are considered. A central point of departure is the notion of algorithmic stability, which allows for constructing valid conformal sets when an optimization step over the K existing conformal sets is needed (Theorem 1 and Corollary 1). In Lemma 3 a concrete algorithm that has theoretical coverage guarantees is derived. In Section 4 the online setting is considered, and a theoretical comparison with the existing COMA algorithm is made. In Section 5 the method is further improved by calibration after selection using effective ranks. This leads to Theorem 2, which shows that the method achieves marginal coverage after optimizing for the best among K conformal predictors. In Section 6 the new methods are experimentally compared to a few baselines on synthetic data and three real-world regression datasets.\n\nSTRENGTHS AND WEAKNESSES:\nFor me the main strength of the paper is that it introduces quite some concepts that are new in the area of conformal prediction (according to my knowledge of the literature). These concepts can be useful when aggregating different conformal predictors in an ensemble learning setting. The authors clearly show that they have a strong knowledge of the theory in this field. The paper is written in a mathematically rigorous way. \n\nThe main weakness of the paper is that it is hard to follow the story (even for researchers with prior knowledge of conformal prediction I would say). The paper is written in a very dense manner, and many design choices are not really justified. I have read the paper two times. The first time I had a hard job to understand Section 3-6. The second time it was better, but I still have a lot of questions. Here I am listing the points that were not clear to me. \n\n- It is not really discussed in which ML setting the proposed algorithms would be useful. I would guess that the main application domain is ensemble learning, either ensembles constructed using bootstrapping, or ensembles that involve randomizing model parameters (such as deep ensembles). Please be more specific in this. \n- For such a setting, an obvious baselines would be performing conformal prediction on the mean of the ensemble. Why is this not a useful approach and inferior to the complicated aggregation that the authors propose?\n- In the experiments on synthetic data, an ensemble is created by randomizing the kernel function and the regularization parameter in kernel ridge regression. Why would that be useful? I would argue that one should first optimize the hyperparameters using grid search, and run conformal prediction on the best performing model in terms of MSE. Would such an approach not perform better? Why not consider bootstrapping or another more useful technique to construct an ensemble?\n- In the experiments the baselines that are implemented are references [11] and [12]. For [11] it is more or less clear to me what it does, but for [12] the authors only say that it is the \"best-on-average\" conformal predictor. I have no idea what this means without reading carefully [12]. More information is needed for readers that don't know [12].  \n- From Corollary 1 I conclude that the theoretical coverage highly depends on the hyperparameters \\nu and \\tau. In the experiments they are just set to one and zero. Isn't this like a very loose coverage bound? (Instead of 1-\\alpha coverage one obtains 1-\\alpha\\epsilon coverage, which is much lower?  \n- In the introduction the authors write: \"In contrast to [11] and [12] our method allow pointwise selection depending on each X\". I don't see why this claim would be the case. The way I interpret the MinSE optimization problem in Lemma 3 is that the best convex combination of model is found. The same p_i values are used for all X, so the convex combination is not specific for each X? \n- In the experiments on real data, what are the K models? Are these the six different regression models from scikit-learn, so K=6? Not 100% clear. \n- The absolute residuals are considered as nonconformity scores, leading to prediction intervals that have the same length for all instances. This might be a clear disadvantage when interval size is a criterion, so it should be relatively easy to outperform baselines using this nonconformity score. I would be interested to see experiments where estimated quantiles are used as nonconformity scores (i.e. conformalized quantile regression). I expect that it becomes much more difficult to outperform the baselines using such a nonconformity score.    \n- Would the proposed method also be useful in classification settings? For such settings, specific nonconformity scores that optimize set size have been proposed, such as the adaptive prediction sets (APS) and its variants. A comparison to such methods could also be useful.\n\nA lot of these questions are related to explaining the experiments better and comparing to different baselines. The experiments were in general a bit underwhelming, but I did not check the additional experiments in the appendix.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nLimitations of the presented methods are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAssume k different conformal predictors (e.g. score taken from an ensemble of models). Directly picking the smallest set among the prediction sets, results in breaking the marginal guarantee in conformal prediction. Same applies to any selection that tunes some objective as it becomes data-dependent. The paper addresses this by valid stable selection. The authors show a set of selections that are stable, and they bound the coverage guarantee of a selection parameterized by the stability parameters. Furthermore they discuss the same setup for time-series data.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength.**  I think the paper is mathematically solid, and also the contributions are noteworthy, despite their limited evaluations, I was convinced to accept the paper even around Section 3.3.\n\nAlso the paper addresses an interesting problem in CP. Basically they define a lower bound on the prediction set defined from an ensemble of CPs while it is data dependent tuning for a specific goal; e.g. smaller set size.\n\n**Weakness: Empirical evaluation.**  I believe in contrast with the complicated theory, there could be evaluation done on larger datasets, and more complicated model. Of course, authors can easily examine their method on an ensemble of classifiers for the CIFAR10 dataset. \n\nI suggest some rephrasing, however they are only for the readability; feel free to ignore them.\n\n**Line 29.** It is better to reorder the words: In section 4, we extend our work …\n\n**Line 115.** It makes it more understandable if you use an example for the inner-randomness of an algorithm. To me, one example is the uniform random value in the APS score function.\n\n**Line 117.** I suggest to use another notation for algorithms that is dissimilar to random variable, unless there is an inherent similarity in the nature of both entities.\n\n**Line 130.** The random algorithm is stable not stability. Right?\n\n**Examples.** I think the authors have used a block similar to theorem for their examples. The readers eye gets tired as there is an italic text for several paragraphs. I suggest the authors to try some other type of environment that has font in normal style.\n\n**No conclusion, and limitations.** Also why there is no conclusions section?\n\nQUESTIONS:\n1. What is the probability in Eq. 3 over?\n2. Why the evaluation on real datasets are so limited? Why didn’t the authors also discuss classification. From what I understand application on classification setup is not a limit of their framework. The models here are not as complicated as a deep model. \n3. If I understood correctly, your definition of stability is based on a reference (intuitively center) distribution. But can we also redefine this based on two random samples of the distribution. Intuitively I think if two samples are not too far with a high probability, they are also both close to a central reference sample.\n4. What prevented authors to evaluate their result on a more complicated setup like ensemble image classification?\n5. Also since the paper defines stable way to predict time-series data, why didn’t the authors try the algorithm on some time series task; e.g. stocks forecast, electricity demand forecast, etc.\n6. I did not fully understood your definition of adversarial setup, and solution for it. To me adversarial setup is defined by a threat model including the set of possible perturbations, and the objective the adversary can achieve. Can you elaborate on this definition?\n\nIf my questions are answered and mentioned weaknesses are addressed, I’ll review the work again and try to increase my score.\n\nLIMITATIONS:\nNo limitations mentioned. But I do not see specific limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a compelling framework for selecting among multiple predictive sets based on any criteria, such as selecting the smallest set, while maintaining the coverage guarantees via algorithmic stability. The proposed procedure has coverage guarantees in both batch and online settings. Notably, their method encourages selection on a pointwise basis.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-written and enjoyable to read. I believe it makes a nice contribution, the core idea and the results are elegant, especially the pointwise selection aspect, which makes this work distinct from other approaches.  \n\nHowever, despite the nice derivations and theoretical results, the proposed approach does not appear to meaningfully outperform the baseline methods, particularly YK-BASE. From what I understand, the method alone does not surpass the baseline unless the recalibration step described in Section 5 is applied. This makes the comparison somewhat unfair, as it relies on an additional calibration set.\n\nHowever, it seems that the method demonstrates its true strength primarily in the online setting. Could you confirm if this is the case? If it is true, could you give more intuition on this difference?\n\nQUESTIONS:\nCould you clarify how AdaMinSE automatically optimises the parameters? I've already looked at the proof, but I might be missing something. It seems to me that the approach is mainly bounding the hyperparameters to eliminate them and using $\\tilde{\\alpha}$ instead (Update after further reading: It seems that the parameters become variables rather than inputs to the linear program. Still, I'd appreciate your take if you have a clearer or more accurate explanation.)\n\nAdditionally, if AdaMinSE is truly optimising the parameters, why does MinSE sometimes perform better? Along the same lines, can you prove that your proposal will outperform any single confidence set?\n\nDo you have any intuition on how to select the prior, especially in the batch setting?\n\nI saw in the appendix that you extended your idea for conditional coverage. Could it be extended for PAC coverage?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper takes up an interesting challenge of comparing conformal prediction (CP) sets that are obtained via different learning methods. To this end, the paper proposes a stability-based approach that focuses on coverage for the selected prediction set. It introduces a few stable selection rules, of which MinSE is proved to be optimal. The paper also shows the applicability of this approach to online CP and validates the contributions through a reasonable suite of experiments. It also improves the empirical performance on split CP through a recalibration step.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The idea of using algorithmic stability to get the smallest prediction set in CP that satisfies coverage is novel and interesting.\n+ The case studies in Sec 3.3 was nice to see.\n+ The additional perspectives on extension to online CP, formulation of adaptive MinSE and recalibration for split CP show a good depth of analysis of the proposed approach.\n\nWeaknesses:\n- The paper largely relies on [4] for their work on algorithmic stability and applies those ideas to CP. However, I do think the application to CP is non-trivial and done well.\n- For the contribution in Sec 3, a limitation of this work seems to be that you needs to run CP multiple times and get multiple sets/intervals, and then apply this method to find the smallest one. What is the computational overhead of this approach, and what is its practicality? Considering the paper largely focuses on adopting algorithmic stability to CP, it may be important to also comment on the additional complexity introduced. \n- Continuing with the above point, the empirical results for this contribution may need to be compared against methods that also have this advantage of seeing multiple prediction sets. Would there be a more fair comparison where the proposed method is compared against other approaches that find the smallest prediction set given multiple CP predictors? \n- In Line 139, the paper refers to $\\lambda$ as the \"size\" of set $C_i$, and in the subsequent analysis, $\\lambda$ is assumed to be $\\in [0,1]$. How would one define such $\\lambda$s in practice that actually capture the size of a set (considering the objective of this work is to find the smallest prediction set)? This seems to be key to the stable selection procedure, and I could not find how this was defined in the results section of the main paper too.\n- AdaCOMA (Sec 4.1) does not make any comments on the size of the prediction set in this setting. Is it possible to comment on this?\n- In the main results (Fig 2), YK-Base seems to do as well/outperform the proposed method on the interval length in Fig 2. While the paper says YK-Base does not guarantee coverage, if $\\alpha=0.1$ (as stated in Line 343), YK-Base seems the most calibrated on Bike Sharing and California Housing.\n- Also, it is concerning that while the paper compares with references [11] and [12], the datasets used are different from what is used in those papers. This makes it hard to know if the datasets used herein are just favorable for the proposed results, or if these generalize to the datasets used in [11] and [12].\n- (Editorial point, but relevant here) It would have been nice to specify $\\alpha$ in the caption of Fig 2 itself. It was hard to parse the figure by looking through the text for this value. This issue is exacerbated in the appendix where it was hard to find the confidence level, without which it is hard to interpret coverage results.\n\nQUESTIONS:\nPlease answer the concerns in the weaknesses above. Below are some additional questions:\n* In Corollary 1, how does one find $\\hat{S}$ in practice?\n* It is interesting to see that when $b$ is a uniform prior (in Sec 3.2), MinSE reduces to deterministically selecting the smallest set. What would this be if $b$ were a prior based on the class frequencies?\n\nAlthough I have marked the score as BR, I am on the borderline on this paper. The approach is interesting and nicely done; however, the lukewarm results and lack of discussion around the size of prediction sets in later sections of the paper make the final takeaways not as consistent as may be required. I look forward to seeing the authors' responses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}