Title: What Does It Take to Build a Performant Selective Classifier?
Venue: NeurIPS 2025
Accepted: yes
Keywords: selective prediction

selective classification

calibration

ranking

abstention

rejection

uncertainty quantification
Abstract: Selective classifiers improve model reliability by abstaining on inputs the model deems uncertain. However, few practical approaches achieve the gold-standard performance of a perfect-ordering oracle that accepts examples exactly in order of correctness. Our work formalizes this shortfall as the selective-classification gap and present the first finite-sample decomposition of this gap to five distinct sources of looseness: Bayes noise, approximation error, ranking error, statistical noise, and implementation- or shift-induced slack. Our analysis reveals that monotone post-hoc calibration—often believed to strengthen selective classifiers—has limited impact on closing this gap, since it rarely alters the model’s underlying score ranking. Bridging the gap therefore requires scoring mechanisms that can effectively reorder predictions rather than merely rescale them. We validate our decomposition on synthetic two-moons data and on real-world vision and language benchmarks, isolating each error component through controlled experiments. Our results show that (i) Bayes noise and limited model capacity can account for substantial gaps, (ii) only non-monotone or feature-aware calibrators consistently reduce the ranking term, and (iii) distribution shift introduces a separate slack that demands distributionally robust training. Together, our decomposition yields a quantitative error budget as well as actionable design guidelines that practitioners can use to build selective classifiers which approximate ideal oracle behavior more closely.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an analytical framework for selective classification that decomposes the gap between a selective classifier and the optimal \"oracle\" into a number of well-defined components: \n\n1. Bayes (Aleatoric uncertainty) resulting in natural label ambiguity.\n2. Approximation error of the model to the true conditional distribution.\n3. Ranking error of the confidence score.\n4. Statistical error from using empirical estimates of expectations\n5. Miscellaneous covering \"the rest\" such as optimisation imperfections.\n\nThe authors then demonstrate empirically the impact of each component on model performance.\n\nThroughout the paper, the authors aim to provide practical takeaways to the reader for improving the gap.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The knowledge presented could be of practical use to a range of readers and practictioners, facilitating better real-world systems for selective classification. I think there is real value in being able to disentangle the contributing causes to a problem in order to build better systems in a more informed way. For example, in existing research aimed at improving selective classification, especially *training* based approaches, it is not always clear whether improved selection is due to better ranking (as is typically how methods are motivated) or actually simply due to better accuracy due to improved generalisation performance. \n- Attractive and clear presentation. The authors structure the paper in a way that is easy to follow, with important information visually highlighted.\n- Good balance of theory and practical information. Although the paper is theory heavy, practical takeaways are presented intuitively in natural language, reducing the barrier-to-entry for the reader. Thumbs up especially for the inclusion of Appendix C.\n\n## Weaknesses\n\nAs a reviewer, I can certainly make mistakes. Due to the large reviewing load for NeurIPS I may have missed information in the paper, or misinterpreted details. I welcome the authors to correct me if that is the case. \n\n\n1. **Missing nuance in the presentation of the oracle**: I believe there are some presentation issues here \n    1. It is not made clear in Definition 3 that the upper bound is achieved under no Bayes noise. \n    2. It is not clear whether the oracle is the bayes-optimal ranker (in the presence of Bayes noise) or a ranker with \"oracle\" knowledge of the true sampled label and thus the correctness of the classifier.\n    3. It is not made clear that lower approximation error (higher mean accuracy $a_\\text{full}$) also raises the upper bound rather than just decreasing the selective gap. How this interacts with Eqs. 9-12 is also not discussed. In fact, this is clearly shown in Fig. 2 but not emphasised.\n2. **Overstated contribution on monotone calibration**. The fact that *monotone calibration cannot improve selective classification* should indeed be emphasised, but to claim this result as a contribution is odd to me as it seems to obviously arise from basic knowledge of binary classification. I would recommend the authors temper this claimed contribution.\n3. **Space for improvement presenting Eqs. 9-12**. Although I have previously praised the presentation, I feel like Sec. 3.3 would benefit from guiding the reader a bit more. For example:\n    1. Making it clear that Eq. 10 compares the learned classifier to the optimal classifier, *whilst keeping the selection function constant*. Note that this is not necessarily intuitive as in practice changing the classifier generally leads to changing the selection function. \n    2. Bringing in discussion (mentioned in the limitations section) about how a single design choice might affect multiple components. E.g. Typically, MSP is used by default, so improving the approximation error of the model $P_\\theta(y|x)$ to the true conditional will simultaneously improve both $\\varepsilon_\\text{rank}$ and $\\varepsilon_\\text{approx}$. This sort of clarification would be especially useful to practitioners.\n4. **Numerous issues with calibration discussion**\n    1. The claim that *temperature scaling rarely reorders* is supported by incorrect calculations in Appendix B.2 (simply sanity checking the provided logits in Eq. 34 gives different softmax outputs, and Eq. 32 seems to be wrong). Besides, existing research [1,2] (mentioned by the authors) clearly demonstrates empirically that temperature scaling has a large effect on selective classification. \n    2. Poor discussion on scoring. Histogram binning, which explicitly introduces ties by many-to-one mappings, is claimed to preserve ordering (even though ties are later discussed in the paper). Recent work [2,3] that both empirically demonstrates and analytically explains the effectiveness of post-training logit normalisation for improving selection, as well as investigates the effects of the calibration approach label smoothing, are not discussed.\n5. **Various missing discussion**. I believe a number of recent research directions should be at least briefly discussed as being relevant (e.g. for future work).\n    1. Unified selective classification [4,5,6] including OOD data to be rejected. How could the gap be extended in this case?\n    2. Uncertainty for open-ended language generation [7,8]. What challenges exist as our models are no longer directly trained to model the Bayes posterior?\n7. **Advice not backed by empirical evidence**. Although I have praised the presence of Appendix C, much of the advice given is not based on experiments, but simply the authors' understanding of existing approaches. This could lead to potential issues. For example, [3] explains how label smoothing, an approach that superficially should improve both approximation error and calibration, in fact degrades selective classification. Additionally, they recommend binning approaches, even though binning introduces ties.  Authors ought to roll back advice or provide additional experiments/references to other works with results for specifically selective classification, to support the advice given.\n6. **Minor complaints**.\n    1. The use of colour, although aesthetic, is a little arbitrary. I initially was confused as I thought definitions and takeaways were the same colour.\n    2. Sec. 2 Accuracy-coverage tradeoff is missing a reference [9].\n    3. Both MSP and SR are used to refer to the same thing.\n\n**Note**: I do not see the paper's lack of a novel method/algorithm as a weakness, as I believe this sort of framework/analysis paper can be just as, if not more impactful than a paper that proposes new solutions.\n\n[1] Galil et al. What Can we Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers? *ICLR 2023*\n\n[2] Cattelan and Silva How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks *UAI 2024*\n\n[3] Xia et al. Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It *ICLR 2025*\n\n[4] Jaeger et al. A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification *ICLR 2023*\n\n[5] Xia and Bouganis Augmenting the Softmax with Additional Confidence Scores for Improved Selective Classification with Out-of-Distribution Data *IJCV 2024*\n\n[6] Narasimhan et al. Plugin estimators for selective classification with out-of-distribution detection *ICLR 2024*\n\n[7] Kuhn et al. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation *ICLR 2023*\n\n[8] Kang et al. Scalable Best-of-N Selection for Large Language Models via Self-Certainty 2025\n\n[9] Traub et al. Overcoming Common Flaws in the Evaluation of Selective Classification Systems *NeurIPS 2024*\n\nQUESTIONS:\nSee weaknesses, I am keen to raise my score if the authors are able to sufficiently address the issues I've presented above. In fact, I am generally positive on the paper at a high level, and would like to see it improve.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThey formulate the performance difference of a selective classifier to its perfect ordering scoring (optimality) measured with the *AUACC* (Area Under the Accuracy–Coverage Curve) as the *coverage-uniform selective-classification gap*. Further they differentiate the finite-sample decomposition of the gap dividing it into five parts: 1)“Bayes noise/irreducible”, 2)“appoximation error/capacity”, 3)“ranking error/ranking”, 4)“statistical noise/data” and 5)“implementation or shift-induced slack/optimization & shift”.\n\nThey provide a theoretical analysis of closing the gap and show that only calibration that modifies the ranking improves selective classification, with a specific focus on temperature scaling and the reason for its limited effect on the AUACC.\n\nFinally they support their theoretical analysis by means of empirical experiments on: the Toy Two Moons dataset, CIFAR10 & CIFAR100, Stanford Cars and Camelyon 17-WILDS.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper provides a clear theoretical thought framework for silent failure cases in selective classification, which the selective classifier does not catch due to the five aforementioned sources of errors.\n- The paper is well structured and beautiful if I may say so. Especially, the takeaway boxes substantially improve parsing of the text.\n- The appendix is well organized and, in combination with the code provided in the supplementary material, which is also well structured, seems to provide all the information necessary to replicate their empirical results.\n- All of this combined can make it a nice starting point for people working themselves into the field of selective classification and on which areas to focus to improve selective classifiers, especially in combination with Appendix C, the “Practitioner Checklist for Tightening the Selective-Classification Gap”.\n\n## Weaknesses\n**Formulation**:\n- Given the stated purpose in (lines 35-36): “For my finite model on finite data, what aspects of the learning setup will actually move my trade-off curve closer to the upper bound?” \n- The algebraic decomposition of the selective classification gap (here a) in equation 27 boils down to an inequality of the form $a <= a + b$ without a clear reasoning why we should care about $b$ if the goal is to improve $a$ as the upper bound appears to be defined as the perfect ranking upper bound, which is defined as $a$.\n    - Please clarify what the exact goal of the finite-sample decomposition is in a simple manner, and especially explain the reason for using this decomposition.\n- As the formulation of the selective classification gap seems to be very similar –if not identical– to that of the e-AURC [D], this should be discussed.\n- Overall, the theoretical formulations could be greatly improved by clarifying the relations to existing literature and metrics.\n\n\n**Related Works**:\n- Currently the results of this paper are not discussing some recent works which have been highly prominent in the community like [A] (ICLR Oral), [B] (NeurIPS Spotlight) and [C] (NeurIPS). This substantially reduces the quality of the text and reduces its value as starting point for people to familiarize themselves with the topic of selective classification.\n    - Specifically [B] introduces the AUGRC which should be discussed in Section 2 paragraph “Accuracy-coverage tradeoff evaluation.” as it essentially reweights the AURC with the coverage which reduces the impact of the very low coverage area.\n    - The finding of [A] based on a large-scale empirical study reveals that “None of the prevalent methods proposed in the literature is able to outperform a softmax baseline across a range of realistic failure sources,” which would help motivate the selection of MSP in Section 4.\n        - Further, the reasoning in [A] goes against the reasoning here because the goal is to reduce the selective classification gap instead of the best overall classifier. This is worth discussing what your view means for the practitioner.\n    - A detailed discussion of the difference to the e-AURC [D] is missing where it is analyzed based on formulas.\n- See last point in “Formulation”.\n\n\n**Novelty**:\n- Playing devils advocate here: That label noise (Bayes), overall model performance (capacity), sample size (data) and optimization errors or distribution shifts (optimization & shifts) influence the overall model performance and that uncertainty estimation (ranking) influences the selection process is hardly new to anyone in the field.\n    - I see the main value in the thought framework and handing solutions. Based on this I believe that more citations to the work referenced as solutions in App. C would greatly increase the value of this work as well as the works which indicate that this is a reliable solution for the use case of selective classification to make it more valuable for future reference.\n- Further the novelty that methods which induce changes in the ranking lead to substantial improvements is rather limited as this is true for basically any metric which is based on ordering of scores (e.g. AUROC, AUPRC…)\n\n\n**Evaluation**:\n- I do not agree with the authors that they perform a “large scale” evaluation. The evaluation works like [A] and [B] spans more methods and more datasets while [C] uses more methods and ImageNet instead of the small scale CIFAR, Stanford and Camelyon datasets. \n    - Solutions: (1) rephrase this, (2) In an optimal scenario I encourage the authors to reinterpret the results from [A], [B] and [C] in their framework.\n- General weaknesses with regard to the evaluation: They only evaluate four different selective classification methods 1) MSP, 2) Temp. Scaled Softmax, 3) Self-Adaptive Training, 4) Deep Ensemble. All of which are simple to implement and 1 & 2 are essentially identical with regard to the overall setup. Further many of the Practical Guidelines in App. C are not explicitly shown (e.g. how RandAugment improves the model [see point above for a solution not necessitating extension of the evaluation]).\n\n\n**Minor points**:\n- In Sections 4.1 and 4.2 in Setup: there are too many enumerations with (i) to (iii) I would appreciate to do (i)...(iii) and (a)... (z) to make the distinction across them clearer.\n\n\n\n### Citations\n[A] Jaeger, P. F., Lüth, C. T., Klein, L., & Bungert, T. J. (2023). A call to reflect on evaluation practices for failure detection in image classification. The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=YnkGMIh0gvX\n\n[B] Traub, J., Bungert, T. J., Lüth, C. T., Baumgartner, M., Maier-Hein, K., Maier-hein, L., & Jaeger, P. F. (n.d.). Overcoming common flaws in the evaluation of selective classification systems. The Thirty-Eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=2TktDpGqNM&noteId=MJx9cPynNy\n\n[C] Mucsányi, B., Kirchhof, M., & Oh, S. J. (2024). Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, & C. Zhang (Eds.), Advances in neural information processing systems (Vol. 37, pp. 50972–51038). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2024/file/5afa9cb1e917b898ad418216dc726fbd-Paper-Datasets_and_Benchmarks_Track.pdf\n\n[D] Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. arXiv preprint arXiv:1805.08206, 2018\n\nQUESTIONS:\nFor questions, I refer to the Weaknesses, where I already detailed ways to improve the quality of the work. In general, I am positive about the value of this work, but see it more as introducing a thought framework and guiding beginners in the field, which could be greatly improved by aggregating findings of previous work, as I see the overall novelty and scale of their experiments as limited.\n\nLIMITATIONS:\nyes, all limitations except the ones mentioned in the Weaknesses section are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the problem of learning selective classifiers, which are classifiers that have the option to abstain on inputs deemed unreliable, reducing the risk of costly errors. Their analysis is based on the gap between the accuracy-coverage tradeoff of the classifier, which quantifies the degradation of performance as the model accepts a broader set of inputs, and a perfect-ordering upper bound, which assumes an oracle that orders the example according to the likelihood of error by the classifier. They show that this gap can be decomposed into five (not necessarily non-overlapping) components that measure different potential sources for this gap: Bayes noise, approximation error, ranking error, statistical noise and implementation of shift-induced slack. Through controlled experiments with both synthetic and real data, they isolate these components, showing the effect of different design choices for selective classifiers that impact the accuracy coverage tradeoff gap. In particular, their results show that calibration methods that are monotonic (i.e. do not affect ranking) do not improve the accuracy coverage tradeoff gap, which was an open question in previous research with contradictory results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n-The paper is well-written, with a good description of the area and recent research, positioning well the contributions of the paper, the conclusions derived from their theoretical and empirical validation and the limitations of the work. \n-The experimental analysis both on synthetic and real data is very well designed to show the effect of the different sources of errors \n-The paper presents an original analysis of the selective classifier learning problem, introducing new concepts (the selective classification gap) and different ways to understand the impact of different design choices.\n\n\nWeaknesses:\n-Although the formulation is clean and elucidates the problem, it is a little hard to see practical, non-obvious, design guidelines for building selective classifiers coming out of this formulation. Even monotonic calibration could in principle improve approximation error, so we can only find out whether it works through controlled experiments (as done in the paper). \n-The paper does not mention overfitting and the effects that this may cause on the selective classification gap — all the experiments show increased performance when the capacity of the model increases in terms of the accuracy-coverage tradeoff gap, but we know that in practice this is not always true because of overfitting.\n\nQUESTIONS:\nHow is the selective classification gap affected by overfitting? What component does it affect? Have you seen cases in practice where increasing model capacity resulted in a worse accuracy coverage tradeoff?\n\nIn the abstract you say that you provide concrete design guidelines for building selective classifiers. Could you list exactly are these concrete guidelines? (I haven’t read the Appendix).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of selective classification: a model can choose to abstain from making a prediction on a subset of examples. They first define a formalism for studying this problem: given a classifier h and a scoring function g, we abstain if g is lower than a threshold tau. They then define the central quantity of interest: the selective classification gap \\Delta(c) (given a coverage level c) which measures the gap between the performance of the system (h,g) compared to the optimal selective classification decisions. The main contribution is to decompose this gap into 5 factors (irreducible gap, capacity, ranking, data, and misc) formally and study each of these terms and the impact of common solutions (calibration) on this gap. They also provide empirical results to help illustrate the theory.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: This paper is very solid overall. The theoretical results are sound. The empirical methodology is sound, however, the experiments rely on old datasets and models. Not to be cliche, but it would have been good to scale the experiments with LLM based classifiers (one can still extract logits from those). \n\nClarity:\nOverall the paper is very easy to read and the presentation is excellent.\n- please make Figure 1 bigger so that its easier to read\n\nOriginality:\nWhile the presentation of the results is very well done and thoughtful, I am not sure there is much novelty in the decomposition presented. The main contribution here is framing of the selective classification gap as a quantity, the decomposition and the bound is then immediately obtained from prior work in learning theory. In practice, we can't compute the true gap but have to resort an upper bound that relies on whether the model got each example right or wrong (abstain from only wrong examples as much as needed based on coverage level c). Therefore, I am not sure how the bound can be used to inform practice. \n\nSignificance:\nI think this paper is very enjoyable to read and will help future researchers in more easily framing the area. However, I do not believe the contributions of this paper make much of a contribution to the theory or practice of selective classification.\n\nQUESTIONS:\n- Can you expand the experiments to include newer models and datasets, notably with LLMs?\n\n- How does this analysis and how does the gap Delta(c) relate to a gap we can compute in practice based on the oracle selective classification rule? Or is there other ways to compute this gap?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-03"}