Title: Conformal Prediction for Ensembles: Improving Efficiency via Score-Based Aggregation
Venue: NeurIPS 2025
Accepted: yes
Keywords: uncertainty quantification

conformal prediction

conformal aggregation

ensembling
Abstract: Distribution-free uncertainty estimation for ensemble methods is increasingly desirable due to the widening deployment of multi-modal black-box predictive models. Conformal prediction is one approach that avoids such distributional assumptions. Methods for conformal aggregation have in turn been proposed for ensembled prediction, where the prediction regions of individual models are merged as to retain coverage guarantees while minimizing conservatism. Merging the prediction regions directly, however, sacrifices structures present in the conformal scores that can further reduce conservatism. We, therefore, propose a novel framework that extends the standard scalar formulation of a score function to a multivariate score that produces more efficient prediction regions. We then demonstrate that such a framework can be efficiently leveraged in both classification and predict-then-optimize regression settings downstream and empirically show the advantage over alternate conformal aggregation methods.


Peer reviews:
comments: SUMMARY:
Authors propose conformal score aggregation (CSA) a new method for adapting conformal prediction to ensembles. Instead of averaging the scores of each model, they construct a data-driven ordering of the score vectors. This pre-ordering is achieved using a set of nested sets, each defined by a threshold parameter. Once the conformal scores are ordered, the coverage guarantee can be computed. Then, they go over a binary search over a separate parameter to obtain the desired coverage based on that ordering. They prove the theoretical consistency of CSA and provide experiments demonstrating the improvements it offers.

STRENGTHS AND WEAKNESSES:
Strengths:

S1. The CSA framework is novel and takes a different approach on the adaptation of conformal prediction to ensembles. (Originality).

S2. CSA experimentally proves to yield significantly less conservative regions than previous methods. (Significance).

S3. Computational efficiency.

S4. The writing is clear and the exposition and methods are sound. (Quality, Clarity).  

Weaknesses:

W1. There is no comment on the limitations of the proposed method.

W2. The article lacks a conclusion section.

Minor comments:
E1. Line 230 grammatical error.

QUESTIONS:
Q1. The conclusions section appears as Discussion, it should be self-contained and have more detail.

Q2. The paper can be improved with a comment on the limitations of CSA. 

Q3. We are not sufficiently familiar with this specific field to provide a confident assessment, however the papper seems technically sound and consistent. This is spetially the case for sections 3.2 and 4.2.

LIMITATIONS:
No, the authors should comment on limitations of the proposed method.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to merge prediction sets from different predictors, each being a valid prediction set with at least $1-\\alpha$ coverage rate. They take advantage of the quantile envelope idea but make substantial changes. Most importantly, they develop a nontrivial method to correct the coverage rate for the intersection of many prediction sets, which seems to be a promising alternative to Bonferroni. Performance on real datasets is shown.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: It is impressive to see the idea of splitting the calibration set into two (disjoint) parts, with the first part used to calculate a $\\beta^\\ast$ for coverage correction and the second part for providing a further correction $\\hat t$ for constructing the final prediction set. In my opinion, this is the most important contribution of this work to the study of conformal prediction.\n\nWeaknesses: \n1) I do not think Theorem 1 precisely covers the setting of Algorithm 1. A complete proof of the correctness of Algorithm 1 must be provided. Theoretical guarantees are perhaps the most valuable feature of conformal prediction. Also, in the statement of the theorem, as well as in its proof (Appendix B) and elsewhere in the paper, the authors should carefully and clearly specify with respect to which random variables the probability is taken. I will raise my rating once these are done satisfactorily.\n\n2) The readability of some parts of the paper needs to be improved, especially on page 5. This is important, as this page contains the most essential idea of the paper. For example, $\\mathcal{S}^{K-1}$ is not defined; just above Theorem 1,  what does it mean by ``with this choice of $\\{ (u_m, \\widetilde{q}_m) \\}$''? Does $\\widetilde{q}_m$ already contain the information of $\\beta^\\ast$?\n\nQUESTIONS:\n1) What are the typical values of $\\beta^\\ast$ in your numerical experiments? I guess it might be very close to $\\alpha$ in many cases as the random forest is one of the predictors and its performance is almost as good as that of the proposed method (Appendices H and I). What about $\\hat t$?\n\n2) What are the typical correlations between scores from different predictors in your experiments? In general, the random forest and the XGBoost could provide very similar scores. \n\n3) Overall, LASSO or OLS is not as good as more advanced algorithms, so if we drop them from the ensemble, does it really matter? In other words, under what situations, including such ``simple'' predictors will help? (In terms of coverage and/or size of the resulting prediction set.)\n\n4) Is it possible to establish some proper concentration inequality that quantify the fluctuation of a finite number of random directions $M$?  Or, could you provide some empirical evaluation on the influence of $M$ on coverage and size of the prediction set? (Similar to Table 4.)\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary:\n\nThe paper introduces a novel method called Conformal Score Aggregation (CSA) to enhance uncertainty estimation in ensemble models using conformal prediction. While existing approaches aggregate prediction regions after conformalization, potentially causing overly conservative regions, CSA instead aggregates predictons in the score space itself. The main contribution is using a multivariate score function and defining prediction regions via quantile envelopes. This enables tighter, more data-driven prediction regions. The authors demonstrate that their method achieves distribution-free coverage guarantees. It is less conservative, and improves efficiency in both classification and predict-then-optimize regression tasks through empirical evaluations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe introduction of multivariate score functions and quantile envelopes in conformal prediction is innovative and addresses limitations in existing methods.\n\nThe method leverages quantile envelopes and concepts of higher-dimensional quantiles for aggregation, enhancing predictive efficiency.\n\nAuthors conduct experimentation, including classification (ImageNet) and regression benchmarks, providing strong evidence of CSA's practical advantages.\n\nCSA can be applied across both classification and regression tasks, unlike some existing methods restricted to specific scenarios.\n\nWeaknesses:\n\nThe approach involves multiple stages (splitting calibration sets, directional quantile computation). Authors should comment on the computational overhead, especially with larger ensemble sizes.\n\nThe necessity of tuning (e.g., binary search for optimal quantiles) could limit practical scalability.\n\nPerformance is likely sensitive to the calibration set split (e.g.,\nand\n); suboptimal splitting could substantially affect predictive efficiency and coverage.\n\nWhile the paper extensively compares to previous methods, authors should comment extensively on the computational cost relative to these alternatives.\n\nQUESTIONS:\nQuestions for Authors:\n\nHow does computational complexity scale with increasing ensemble size (K) or calibration dataset size (NC)? Could the method become infeasible at larger scales?\n\nHow sensitive is CSA to the choice of projection directions {u_m}? Would alternative direction-sampling methods improve performance further?\n\nHow should practitioners optimally select the calibration split ratio (DC^(1):DC^(2)) in practice, and how sensitive are results to variations in this choice?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to improve conformal prediction for ensemble models. While existing work has already proposed aggregating ensemble prediction regions to retain coverage guarantees, this paper introduces a new framework called Conformal Score Aggregation (CSA). CSA leverages quantile envelopes to enable data-driven uncertainty estimation. The method can be directly incorporated into Predict-Then-Optimize tasks. It is evaluated on both regression and classification tasks and demonstrates strong performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper is well-written and well-organized.\n\nThe method provides theoretical guarantees for convergence.\n\nIt is applicable to both regression and classification tasks.\n\nWeaknesses:\n\nI am not very familiar with this line of work, and I found the paper quite difficult to understand. Section 3.1.1 is especially complex, and I am unsure what the main novelty is and why it is important.\n\nI suggest adding illustrative figures or diagrams to help visualize the process of conformal prediction in ensemble models. This would also help clarify the specific contributions and steps introduced in the paper.\n\nThe phrase “improving efficiency” in the title is unclear to me. Does it refer to optimization speed or something else? Clarification would be helpful.\n\nI also found the evaluation results hard to interpret. For instance, what evaluation metrics are used for the regression and classification tasks?\n\nWhile the abstract mentions uncertainty quantification, the paper does not clearly demonstrate how uncertainty is estimated, nor does it provide an evaluation of the uncertainty quantification quality.\n\nI may improve my score based on further discussion.\n\nQUESTIONS:\nPlease see above for the weaknesses. All the questions are there.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}