Towards Better Selective Classification
Authors: Leo Feng
Mohamed Osama Ahmed
Hossein Hajimirsadeghi
Amir H. Abdi
Venue: ICLR 2023 poster (2023)
Keywords: Selective Classification
Semi-Supervised Learning

Abstract: We tackle the problem of Selective Classification where the objective is to achieve the best performance on a predetermined ratio (coverage) of the dataset. Recent state-of-the-art selective methods come with architectural changes either via introducing a separate selection head or an extra abstention logit. In this paper, we challenge the aforementioned methods. The results suggest that the superior performance of state-of-the-art methods is owed to training a more generalizable classifier rather than their proposed selection mechanisms. We argue that the best performing selection mechanism should instead be rooted in the classifier itself. Our proposed selection strategy uses the classification scores and achieves better results by a significant margin, consistently, across all coverages and all datasets, without any added compute cost. Furthermore, inspired by semi-supervised learning, we propose an entropy-based regularizer that improves the performance of selective classification methods. Our proposed selection mechanism with the proposed entropy-based regularizer achieves new state-of-the-art results.

TOWARDS BETTER SELECTIVE CLASSIFICATION
Leo Feng
Mohamed Osama Ahmed
Mila Université de Montréal & Borealis Al
Borealis Al mohamed. o. ahmeedebrrellllse......... com
leo. Fengêmila la. quebec
Hossein Hajimirsadeghi
Amir Abdi
Borealis AI hossein ha hts.imaaaaeeeel.era.ii.........
Borealis AI
amir abd..abbbrrrrraisaaa. com
ABSTRACT
We tackle the problem of Selective Classification where the objective is to achieve the best performance on predetermined ratio (coverage) of the dataset. Recent ttat-oof-th-au selective methods come with architectural changes either via in- troducing separate selection head or an extra abstention logit. In this paper, we challenge the aforementioned methods. The results suggest that the superior performance of state-of-the-ar methods is owed to training more generalizable classifier rather than their proposed selection mechanisms. We argue that the best performing selection mechanism should instead be rooted in the classifier itself. Our proposed selection strategy uses the classification scores and achieves better results by significant margin, consistently, across all coverages and all datasets, without any added compute cost. Furthermore, inspired by semi-supervised learn- ing, we propose entropy-basee -assed regularizer that improves the performance of selective classification methods. Our proposed selection mechanism with the proposed entropy- based regularizer achieves new Mate-of-the-arr results.
INTRODUCTION
A model's ability to abstain from decision when lacking confidence is essential in mission-critical applications. This is known as the Selective Prediction problem setting. The abstained and uncertain samples can be flagged and passed to human expert for manual assessment, which, in turn, can improve the re-training process. This crucial in problem settings where confidence critical or an incorrect prediction can have significant consequences such in the financial, medical, or autonomous driving domains. Several papers have tried to address this problem by estimating the uncertainty in the prediction. Gal & Ghahramani (2016) proposed using MC-dropout. LLkshminarayanan
al. (211)) proposed to use an ensemble of models. Dusenberry et al.) (2020) and Maddox et al. (2019) examples of work using Bayesian deep learning. These methods, however, are either expensive to train or require lots of tuning for acceptable results.
In this paper, we focus on the Selective Classification problem setting where classifier has the option to abstain from making predictions. Models that come with an abstention option and tackle the selective prediction problem setting are naturally called selective models. Different selection approaches have been suggested such as incorporating a selection head Geifman & EI- Yaniv (0019) or an abstention logit (Huang et al., 2020 Ziyin et al.) 2019). In either case, threshold is set such that selection and abstention values above or below the threshold decide the selection action. SelectiveNet
Geifman HI- Yaniv (2019) proposes learn model comprising of selection head and a prediction head where the values returned by the sclection head determines whether the datapoint is selected for prediction or not. Huang et al. (2020) and Ziyin et al. (019) introduced an additional abstention logit for classification settings where the output of the additional logit determines whether the model abstains from making predictions on the sample. The promising results of these works suggest that the selection mechanism should focus the output of an external head/logit.
On the contrary, in this work, we argue that the selection mechanism should be rooted in the classifier itself. The results of rigorously conducted experiments show that (1) the superior

---

performance of the ofte--of-the-ar methods is owed to training more generalizable classifier rather than their proposed external head/logit selection mechanisms. These results suggest that future work in selective classification (i) should aim to learn more generalizable classifier and (ii) the selection mechanism should be based on the classifier itself rather than the recent research directions of architecture modifications for an external logit/head. (2) We highlight connection between selective classification and semi-supervised learning. To the best of our knowledge, this has has not been explored before. We show that entropy-mining munoopy-mnnimization regularization, common technique in superviueee learning. significantly improves the performance of the state-of- of-te-of-the-ar selective classification method. The promising results suggest that additional research is warranted to explore the relationship between these two research directions.
From a practical perspective, (3) we propose a selection mechanism that outperforms the original selection mechanism of state-of-the-art methods. Furthermore, this method can be immediately applied to an already deployed selective classification model and instantly improve performance at no additional cost. (4) We show a selective classifier trained with the entropy-regularised loss and with selection according to the classification scores achieves new state-of-the-arr results by a significant margin (up to 80% relative improvement). (5) Going beyond the alleady-satuurale datasets often used for Selective Classification research, we include results on larger datasets: StanfordCars, F0od101, Imagenet, and Imagenet 00 to test the methods on wide range of coverages and ImagenetSubset to test the scalability of the methods.
RELATED WORK
The option to reject prediction has been explored in depth in various learning algorithms not limited to neural networks. Primarily, Chow (Chow 1970) introduced a cost-based rejection model and analysed the error-reject trade-off. There has been significant study in rejection in Support Vector
Machines (Bartlett & Wegkamp 2008: Fumera & Roli 2002. [Wekkmpp 2007 Wegkamp & Yuan 2011). The same is true for nearest neighbours (Hellman 1970) and boosting (Cortes et al.) (2166).
In 1989, LeCun et al. (1899) proposed a rejection strategy for neural networks based on the most activated output logit, second most activated output logit, and the difference between the activated output logits. Geeifman& & El- Yaniv (2017) presented a technique to achieve target risk with certain function. As examples of confidence-rate functions, the authors suggested selecting according to Softmax Response and MC-Dropout selection mechanisms for vanilla classifier. We build on this idea to demonstrate that Softmax Response, if utilized correctly, is the highest performing selection mechanism in the selective classification settings. Beyond selective classification, max-logit (Softmax Response) has also been used in anomaly detection (Hendrycks & Gimpel 2016 Dietterich & Guyer 2022).
probability for a given confidence
Future work focused on architectural changes and selecting according to a separately computed head/logit with their own parameters. The same authors, /Geffman & El-yaniv (2019) later proposed SelectiveNet (see Section 3.2. a three-headed model, comprising of heads for selection, prediction, and auxiliary prediction. Deep Gamblers (Ziyin et al.) 2019) (see Appendix A. and Self-Adaptive Training (Huang et al. 2020) (see Section 3.3.1) propose (C + 1)-way classifier, where C/ is the number of classes and the additional logit represents abstention. In contrast, in this work, we explain how selecting via entropy and max-logit can work as proxy to select samples which could potentially minimise the cross entropy loss. In general, we report the surprising results that the selection head of the SelectiveNet and the abstention logits in Deep Gamblers and Self Self-Adaptive Training are suboptimal selection mechanisms. Furthermore, their previously reported good performance rooted in their optimization process converging to a more generalizable model.
Another line of work which tackles the selective classification is that of cost-sensitive classification (Charoenphakdee et al. 202I). However, the introduction of the target coverage adds new variable and changes the mathematical formulation. Other works have proposed to perform classification in conjunction with expert decision makers (Mozannar & Sontag 2020).
In this work, we also highlight a connection between semi-supervised learning and selective clas- sification, which, to the best of our knowledge, has not been explored before. As result, propose an entropy-regularizee loss function in the Selective Classification settings to further improve the performance of the Softmax Response selection mechanism. However, entropy minimization
2

---

objectives have been widely used for Unsupervised Learning (Long et al.
2016
Semi Supervised 2019 Wu et al. 2020).
Learning (Grandvalet & Bengio 2004). and Domain Adaptation (Vu et al.
BACKGROUND
In this section, we introduce the Selective Classification problem. Additionally, we describe the top methods for Selective Classification. To the best of our knowledge, Self-Adaptive Training (Huang al. 2020) achieves the best performance on the selective classification datatests.
3.1 PROBLEM SETTING: SELECTIVE CLASSIFICATION
The selective prediction task can be formulated follows. Let X' be the feature space, ) be the label space, and P(X; y)
of a prediction function represent the data distribution over X' x V. selective model comprises X' Y and selection function g X + {0, 1}. The selective model decides to make predictions when g(x) = 1 and abstains from making predictions when g(z) = O. The objective is to maximise the model's predictive performance for given target coverage Curget E [0, 1]. where coverage is the proportion of the selected samples. The selected set is defined as {x: g(x) 1). Formally, optimal selective model, parameterised by 8" and th",, would
be the following:
0", v Begning,.EP[((a((a v) (x)], s.t. Ep(gg(x) 2 Cuarget
(I)
where Er((([(()), v) go (x)] is the selective risk. Naturally, higher coverages are correlated with higher selective risks.
In practice, instead of hard selection function (1)), existing methods aim to learn a soft selection function Fy : X' - IR such that larger values of (x) indicate the datapoint should be selected for prediction. At test time, threshold selected for a coverage such that
(x(x) {
if go (x) 2 T otherwise
s.t. ((g((())
Crarget
(2)
In this setting, the selected (covered) dataset defined as {a Do(x) 2 T). The process of selecting the threshold is known as calibration.
3.2
APPROACH: LEARN TO SELECT
3.2. SELECTIVENET
SelectiveNet (Geifman & EI- Yaniv, 2019) is a three-headed network proposed for selective learning. A SelectiveNet model has three output heads designed for selection ø. prediction f, and auxiliary prediction h. The selection head infers the selective score of each sample, as a value between to 1, and is implemented with sigmoid activation function. The auxiliary prediction head is trained with a standard moon-selective) loss function. Given a batch {(z,, Mc) Jmm } where is the label, the model trained minimise the loss L where is defined as:
L (Lselective + +Ao)) + (1 Cauu
(3)
Em ((s(s,,, (1))
Lselective
(4)
Σx (Ix.)
L. =mx (0. (Cuagged
Σ g(x:)))),
Lauz 1 Σ E(h(z;).
(5)
where is any standard loss function. In Selective Classification, é is the Cross Entropy loss function. The coverage loss L. encourages the model to achieve the desired coverage and ensures g(æ;) > for at least Carget proportion of the batch samples. The selective loss Lselective discounts the weight of difficult samples via the soft selection value g(z) term encouraging the model to focus more on easier samples which the model is more confident about.
The auxiliary loss Lauz ensures that all samples, regardless of their selective score (g(x)). contribute to the learning of the feature model. X and hyper-parameters controlling the trade-off of
3

---

different terms. Unlike Deep Gamblers and Self-Adaptive Training, SelectiveNet trains separate model for each target coverage Gurget- In the SelectiveNet paper (Geifman & EI- Yamiv 2019). it has been suggested that the best performance is achieved when the traming target coverage is equal to that of the evaluation coverage.
3.3 APPROACH: LEARN TO ABSTAIN
Self-Adaptive Training (Huan et al. 2020) and Deep Gamblers (Ziyin et al. 2019) propose to tackle the selective classification problem by introducing (C++1) 1)-th class logit where the extra class logit represents abstention. Let Po (x) represent the prediction network with softmax as the last layer. This family of methods abstain if p + 1|z) above a threshold. Here, we unify the notations of these abstention and selection methods under the same Selective Classification framework (See Section 3.1) with the following soft selection function: g(æ) = pa (C + 1/x). Due to the space limitation, the formulation for Deep Gamblers is included in the Appendix.
3.3.1 SELF-ADAPTIVE TRAINING
In addition learning logit that represents abstention, Self-Adaptive Training (Huang al. 2020 proposes to use convex combination of labels and predictions as dynamically moving training target instead of the fixed labels. Let y; be the one-hot encoded vector representing of the label for datapoint (x;, Mi) where y; is the label.
Initially, the model trained with cross-entropy loss for series of pre-training steps. Afterwards, the model is updated according to dynamically moving training target. The training target 4; initially set equal to the label t; f y, such that the training target is updated according to + x f + (1 a) p=(+(x;;) s.t. a E (0, 1) after each model update. Similar to Deep Gamblers, the model trained to optimise a loss function that allows the model to also choose to abstain on hard samples instead of making prediction:
L =
Min log (v()))); + (1 - li log Po + 1(2;)),
(6)
where m the number of datapoints the batch. As training progresses, approaches (=)2x;). The first term is similar to the Cross Entropy Loss and encourages the model to learn good classifier. The second term encourages the model abstain from making predictions for samples that the model is uncertain about. This use of dynamically moving training target l; allows the model to avoid fitting on difficult samples as the training progresses.
4 METHODOLOGY
We motivate alternative selection mechanism Softmax Response for Selective Classification models. We explain how the known state-of-the-art selective methods can be equipped with the proposed selection mechanism and why it further improves performance. Inspired by semi-supereised learning, we also introduce an catropyrreglaarize loss function.
4.1 MOTIVATION
Recent caate-ff-the-at methods have proposed to learn selective models with architecture modifica- tions such as an external logit/head. These architecture modifications, however, act regularization mechanisms that allow the method to train more generalizable classifiers (see Table IT. As a result, the claimed improved results from these models could actually be attributed to their classifiers being more generalizable. For these selective models to have strong performance in selective classification they require the external logit/head to generalise in these sense that the external logit/head must select samples for which the classifier is confident of its prediction. Since the logit/head has its own set of learned model parameters, this adds another potential mode of failure for a selective model. Specifically, the leamned parameters can fail to generalise and the logit/head may (1) suggest samples for which the classifier is not confident about and (2) reject samples for which the classifier confident about. In the appendix (See Figure and 5), include examples of images that fail due
4

---

Model
Accuracy
Vanilla Classifier
85.68 = 0.14
SelectiveNet
86.23 = 0.14
Deep Gamblers
86.51 # 0.52
Self-Adaptive Training
86,40 # 0.30
Table 1: Results at 100% coverage on Imagenet100.
this introduced mode of failure. As such, we propose that selective mechanisms should stem from the classifier itself instead of an external logit/head, avoiding this extra mode of failure.
4.2 SELECTING ACCORDING TO THE CLASSIFIER
The cross entropy loss function is a popular loss function for classification due to its differentiability. However, during evaluation, the most utilized metric accuracy, i.e. whether a datapoint is predicted correctly. In the cross-entropy objective of the conventional classification settings, p(c|2,) is one-hot encoded vector; therefore,
the cross-entropy loss can simplified as CE (p(-)æa), (o po(-2r)) =
> p(u Ir;) log Pa (u(uaz (2;) = - log Pe (3: [x,), L.e,, during optimization, the logit of the correct class maximised. Accordingly, the maximum value of logits can be interpreted the model's relative confidence of its prediction. Therefore, simple selection mechanism for model would be select according the maximum predictive class score, g(z) max (1....)))) (o(w)z,) (aka Softmax Response (Geifiman& & El- Yaniv 2017)). Alternatively, model can also select according to predictive entropy g(z)
H [po x)), a metric of the model's uncertainty. in-depth
discussion is included in in the Appendix B
4.3 RECIPE FOR BETTER SELECTIVE CLASSIFICATION
The recipe that we are providing for better selective classification is as follows:
Train a selective classifier (e.g., SelectiveNet, Self-Adaptive Training, or Deep Gamblers).
2. Discard selection mechanism:
For SelectiveNet:
Ignore the selection head
For Self-Adaptive Training and Deep Gambler: Ignore the additional abstain logit and compute the final layer's softmax the original C' class logits.
3. Use classifier-based selection mechanism (e.g., Softmax Response) to rank the samples.
4. Calculate the threshold value based on the validation set, to achieve the desired target coverage and select samples with max logit greater than T.
Empirically, we show that selecting via entropy or Softmax Response both outperform selecting according to the external head/logit. From these results, we can conclude that the strong performance of these recent sfute-of-the-ar methods were due learning a more generalizable classifier rather than their proposed selection mechanisms. In Step 3, we experimented with both entropy-based selection mechanism and Softmax Response we found that Softmax Response performed better. Notably, Softmax Response does not require retraining and can be immediately applied to already deployed models for significant performance improvement at negligible cost.
4.4 ENTROPY-REGULARTEN LOss FUNCTION
Here, we highlight a similarity between supi-supe learning and selective classification, which to our knowledge has not been explored before. In the semi-supervised learning setting the training dataset consists labelled and unlabelled data. A simple approach is to train solely on the labelled data and ignore the unlabelled data, i.e., training model via supervised learning to the labelled data. This is equivalent to having a weight of 1 for the labelled samples and for the unlabelled samples. However, this is suboptimal because it does not use any information from the unlabelled samples. Similarly, in Selective Classification, samples that are selected tend to have a high weight close to 1 (see, for example, in Section 3.2.1 the g(r) term in the objective) and samples that not selected
5

---

have a low weight close to 0. One way that semi-supervised learning have proposed to tackle this is via an entropy minimization term.
Entropy minimization is one of the most standard, well-studied, and intuitive methods for semi- supervised learning. It uses the information of all the samples and increases the model's confidence in its predictions, including the unlabelled samples, resulting in better classifier. Inspired by the similarity in the objective of selective classification and the setting of semi-supere superiisee learning, we propose an entropy-minn cutropy-miiimistionn term for the objective function of selective classification methods:
fnew = /3 (pp(()))))
(7)
where 3 is a hyperparameter that controls the impact. In our experiments, we found = 0.01 to perform well in practice. The entropy minimization term encourages the model he more confident in its predictions, ie.. increasing the confidence of the predicted class and decreasing the predictive entropy during training. Thus, it allows for better disambiguation between sample predictions. The larger coefficient on the cross-entropy term compared to that of the autropy-minimizate term ensures that increasing the confidence of correct predictions are prioritised, benefitting Softmax Response. In Section we show that this proposed loss function based on semi-supervised learning improves the performance Selective Classification by significant margin. These results opens the door to future exploration of the connection between Selective Classification and semi-supervised learning.
EXPERIMENTS
For the following experiments, we evaluate the following state-of-the-arr methods (1) Selec- tiveNet (SN), (2) Self-Adaptive Training (SAT), and (3) Deep Gamblers. Furthermore, compare the performance of these methods with the following selection mechanisms (1) original selection mechanism and (2) SR: Softmax Response (our proposed method). Due to space limitations, the table results for a vanilla classifier is included in the Appendix with several additional results.
The goal of our experimental evaluation to answer the following questions: (1) Is the superior performance of recent state-of-the-art methods due to their proposed external head/logit selection mechanisms? Taking this further, what is the state-of-the-art selection mechanism? (2) Does the proposed
loss function improve the effectiveness of Softmax Response for Selective Classification? (3) What is the new state-of-the-oe method for Selective Classification? (4)
How scalable are selective methods for larger datasets with larger number of classes?
5.1 DATASETS
We introduce new datasets: StanfordCars, 101, Imagenet, Imagenet100 and ImagenetSubset, for the selective classification problem setting and benchmark the existing state-of-the-e methods. We propose StanfordCars, Food101, Imagenet, and Imagenet100, as realistic non-saturated datasets that can be evaluated at a wide range of coverages (10 100%). In addition, we propose ImagenetSubsett as a collection of datasets to evaluate the scalability of the methods for different number of classes. This is in contrast to the existing Selective Classification research which mainly have focused on small datasets such as CIFAR- with 10 or less classes, low resolution images (64x64 or less), and very low eтror (The error at 80% coverage is already lower than 1 1%) this dataset is limited to high coverages (70%+). ). The results of the previously introduced datasets indicate saturation, e.g., 0.3% error at 70% coverage, discouraging experiments with lower coverages, which, in turn, prevents researchers from achieving conclusive results.
Imagenet (Deng et al. 2009) comprises of 1,300 images per class and evaluation data comprising of 50,000 images split into 1,000 classes. Imagenet 100 (Tian) al. 2020) a subset of Imagenet which comprising of 100 classes. ImagenetSubset is created as collection of datasets with varying number of classes (difficulty) from 25 to 175 in increments of 25. The classes are sampled randomly such that datasets with less classes are subsets of those with more classes. The complete list of selected classes in each dataset subset is available in the Appendix ImagenetSubset evaluates the models' performance with respect to the difficulty (scalability) of the task.
Note that the created dataset of ImagenetSubset with 100 classes different than that of Imagenet 100.

---

SelectiveNet (SN)
Deep Gamblers (DG)
Self-Adaptive Training (SAT)
Coverage
SN
SN+SR
DG
DG+SR
SAT
SAT+SR
100
13.77 0.14
13.77 + 0.14
13.49 = 0.52
13.49 : 0.52
13.58 0.30
13.58 0.30
90
9.44 0.28
7.89
8.42 = 0.44
8.11 = 0.48
8.80 0.41
8.04 = 0.25
80
6.22
4.47
5.21 = 0.32
4.32
5
4
70
3.28
=
3.30 0.40
2.58 0.21
2.71 0.29
2.33
60
0.15
1.57
0.06
2.14 0.37
1.71 = 0.32
0.12
+
50
1.05 0.17
0.02 0.85
1.55 0.27
1.31 0.22
0.88 0.07
40
0.58 0.08
0.53
1.23 0.38
1.07 0.19
0.06
0.60
30
1.04 0.37
0.10
1.09 + 0.31
0.96 - 0.21
0.67 + 0.06
0.59
20
48.87 6.15
1.03 + 0.31
0.90
0
0.26
10
99.00 0.00
99.00 0.00
0.80
0.28
0.53 0.25
0.100
0.16
Table 2: Comparison of the selective classification error between SelectiveNet (SN), Deep Gamblers (DG), Self-Adaptive Training (SAT) with their original selection mechanisms vs. using Softmax Response (SR) as the selection mechanisms on ImageNet100.
Food101. The Food dataset (Bossard et al. 2014) contains 75750 training images and 25250 testing images split into 101 food categories.
The Cars dataset (Krause et al. 2013) contains 8, 8,144 training images and 8,041 testing images split into 196 classes of cars. Unlike prior works which typically evaluate StanfordCars for transfer learning, in this work, the models are trained from scratch.
StanfordCars.
C-FAR-10. The CIFAR- 10 dataset (Krizhevsky) 2009) comprises of small images: 50,000 images for training and 10,00 images for evaluation split into 10 classes. Each image is of size 32 x 32 x 3.
5.2 EXPERIMENT DETAILS
For our experiments, we adapted the publicly available official implementations of Deep Gamblers and Self-Adaptive Training 2
Experiments on SelectiveNet were conducted with our Pytorch implementation of the method which follow the details provided in the original paper (Geifman & Yaniv 2019). For the StanfordCars, F01 Imagenet100. and ImagenetSubset datasets, we use a ResNet34 architecture for Deep Gamblers, Self-Adaptive Training, and the main body block of SelectiveNet. Following prior work, we use a VGG16 architecture for the CIFAR-10 10 experiments.
We tuned the entropy minimization loss function hyperparameter with the following values: / E (0.1, 0.01, 0.001, 0.0001). CIFARIO, Food101, and StanfordCars experiments were run with 5 seeds. Imagenet-related experiments were run with 3 seeds. Additional details regarding hyperparameters are included in the Appendix.
5.3 RESULTS
5.3.1 CORRECTING THE MISCONCEPTION ABOUT THE SELECTION MECHANISM
In Table we compare the different selection mechanisms for a given selective classification method (SelectiveNet, Deep Gamblers, and Self-Adaptive Training). The results show that for each of these trained selective classifiers, their original selection mechanisms are suboptimal; in fact, selecting via Softmax Response outperforms their original selection mechanism. These results suggest that (1) the strong performance of these methods were due to them learning a more generalizable model rather than their proposed external head/logit selection mechanisms and (2) the selection mechanism should stem from the classifier itself rather than a separate head/logit. We see that Softmax Response is the ofate-of-the-are selection mechanism. is important to note that this performance gain achieved by simply changing the selection mechanism of the pre-trained selective model without any additional computational cost. This observation applies to SN, DG, and SAT models.
An interesting result from this experiment is that at low coverages (30%, 20%, and 10%), Selec- tiveNet's 's performance progressively gets worse. We hypothesize that this is due to the optimisation process of SelectiveNet that allows the model to disregard (i.e., assign lower weight to their loss) a vast majority of samples during training at little cost, i.e., g(x) Rs 0, especially when the target
"The code is available at http://gghhhh...ooooamaaaal////s

---

StanfordCars
Food101
Cov.
SAT
SAT+SR
SAT+EM+SR
SAT
SAT+SR
SAT+EM-SR
100
37.68
1.11
37.68 1.11
32,49 4 2.33
16.41
16141
4.32
90
32.34 + 1.19
32.04 + 1.18
26.60 # 2.39
11.87 + #.13
10184
10.77 0.36
80
26.39 1.13
20.87 + 2.33
7.91
6.57 0.13
6.57
70
2..3
20.70 1.23
15.84 1.98
0.11
3.05
3.52 0.19
60
16.21
1.09
11.09
0.09
1.05
1.75
50
0.75
10.25
1.13
0.09
1.06
0.96
40
0.76
6.32
4.00 0.87
0.05
0.56
0.49 0,08
30
0.35
3.34
2.20 + 0.44
0,48 0.07
0.04
20
2.42
1.99
1.17 0.28
0.25 0.01
0.15
0.09 0.05
10
1.49 0.00
1.20 # 0.21
0.80 + 0.22
0.15 0.07
0.09 = 0.02
0.03 # 0.02
Table 3: Comparison of the selective classification with the original selection mechanisms vs. using Softmax Response (SR) and the proposed entropy minimization loss function (EM) on StanfordCars and Food101
between Self-Adaptive Training (SAT)
Imagenet
Imagenet100
Cov.
SAT
SAT+EM+SR
SAT
SAT
SR
SAT + EM
SAT+EM+SR
100
27.41
0.08
27.27 # 0,05
13.58 0.30
13.58 0.30
13.18
13.18 0.24
90
22.67
0.24
21.57 # 0.19
8.80
8.04 0.25
8,69 0.32
7.73 0.22
80
18.14
0
16.83 0.06
+
0.13
5.03
3,90 0.34
70
13.88
12.34 0.11
2.71
0.19
2.61 0.22
1.81
60
0.15
8.45 0.05
1.72
0.12
1.19
0.15
50
0.07
5.57
1.14
0.07
1.22
0.09
40
0.33
3.77
0.02
0.10
0.11
0.34
30
0.14
2.32
0.67 + 0.06
0.19
0.11
0.25 0.10
20
1.82
1.35 0.20
0.48 0.18
0.26
0.12
0.15 + 0.08
10
0.34
0.55 = 0.05
0.32 # 0.10
0.12 0.16
0.32 0.20
0.12 0.07
Table 4: Results on Imagenet and demonstration of the impact of our SAT+EM+SR method over using SR alone or EM alone.
coverage is low 10%. In contrast, Deep Gamblers and Self-Adaptive Training models equally optimised over all samples regardless of their selection.
5.3.2 POWER OF SOFTMAX RESPONSE SELECTION WITH ENTROPY MINIMIZATION
In these experiments, we focus Self-Adaptive Training as is the state-of-the-art selective model.
CIFAR10
Coverage
SAT
SAT+EM+SR
In Table and [ we compare Self-Adaptive Training (SAT), SATwith SR (Softmax Response) selection mechanism, and SAT with SR and EM caatropy-Minimizatiion on the Imagenet, Stanford- Cars, Food101, and CIFAR10 10 datasets. The results show that SAT+EM+SR achieves state-of-the-art performance across all coverages. For example, in StanfordCars, at 70% coverage, we see a raw 5.5% absolute improvement (25% relative reduction) in selective classification error by using our proposed method EM+SR. In Food101, at 70% coverage. we see raw 1.37% absolute reduction (28% rel- ative reduction) in selective classification error. The clear and considerable improvement across all coverages when using Softmax Response selection mechanism rather than the original selection mechanism. These results further confirm surprising finding that existing selection mechanisms are suboptimal. In the Appendix we further include (I) risk-coverage curves and (2) results for several network architectures. The results of those experiments show that proposed methodology generalises across different network architectures.
100
5.91 = 0.04
5,91 0.04
95
3.73 0.13
3.63 0.10
90
2.18 2 0.11
2.11 + 0.06
85
1.26 0.09
1.18 + 0.07
80
0.69 = 0.04
0.64
75
0.37 0.01
0.36 + 0.03
70
0.27 = 0.02
0.23 + 0.05
Table 5: Results on CIFAR10
For the CIFAR- 10 experiments (Table the results for the different methods within confidence intervals. Since the selective classification errors are very small, it is difficult to draw conclusions from such results. On CIFAR-10, SAT achieves 99+ % accuracy 80% coverage. In contrast,
8

---

30% Coverage
50% Coverage
70% Coverage
Classes
SAT
SAT+EM+SR
SAT
SAT+EM+SR
SAT
SAT+EM
175
0.69 # 0.12
0,46 0.05
1.27 = 0.12
0.91 0.16
0.13
2.73
150
0.44 0.13
0.16 0.02
0.81 0.11
0.47 = 0.05
2.23
1.71 0.15
125
0.44
0.13
0.52 0.07
0.25
1.84 0.14
100
0.11
0.06
1.11
0.56 0.06
2.19
1.81 0.08
75
0.15
0.09 -
1.1
0.40 = 0.03
2.10
1.68 + 0.27
50
0.76 + 0.06
0.37 = 0.10
2.86 + 0.34
1,47 + 0.12
25
0.53
0.00
0.08 0.11
0.64 0.23
0.21 + 0.08
1.79 + 0.53
1.14 0.25
Table 6: Comparison between the Selective classification error for Self-Adaptive Training (SAT) and SAT with Entropy Minimization (EM) and Softmax Response (SR) ImagenetSubset.
Imagenet100, SAT achieves 95% 80% coverage. The saturation of CIFAR-10 further highlighted
in previous works which show improvements on the dataset (Geifman & El- Yaniy 2019 Ziyin et al.
2019
Huang et al. 2022) the scale of 0.1 0.2%.
5.3.3
SCALABILITY WITH THE NUMBER OF CLASSES: IMAGENETSUBSET
To evaluate the scalability of the proposed methodology with respect the number of classes, evaluate our proposed method SAT+EM+SR with the previous state-of-the-art SAT on ImagenetSub- In T we see once again that Self- Adaativee Training with our proposed antropy-regularis loss function and selecting according to Softmax Response outperforms the previous state-of-the-as (vanilla Addativeee Training) by a very significant margin (up to 85% relative improvement) across all sizes of datasets. Due to the space limitations, the results for the other coverages of Table 6 are included the Appendix.
5.3.4 CIRIOP--MMIMIZZATION ONLY, SOFTMAX RESPONSE SELECTION ONLY, OR BOTH?
In this experiment, we show that applying EM or SR alone provide gains. However, to achieve state-of-the-arr results by large margin, it crucial to use the combination of both SR and EM. Table 4 Isoows that using only the entropy- entropy-minimization minimization (SAT-EM) slightly improves the performance of SAT. However, SAT+EM+SR (MAT+EM in conjunction with SR selection mechanism) improves upon SAT+SR and SAT+EM significantly, achieving new state-of-ft art results for selective classification.
6 CONCLUSION
In this work, we analysed the state-of- state-of-the-ane art Selective Classification methods and concluded that their strong performance owed to learning a more generalisable classifier rather, yet their suggested selective solutions are suboptimal. Accordingly, we showed that selection mechanisms based on the classifier itself outperforms the state-of-the-as selection methods. These results suggest that future work in selective classification should explore selection mechanisms based on the classifier itself rather than following recent works which proposed architecture modifications. Moreover, we also highlighted connection between selective classification and semi-supervised learning, which to knowledge has not been explored before. We show that a common technique in semi-supervised learning, namely, entropy-minimizatie opening the door to further exploration of the relationship between these two fields.
greatly improves performance in selective classification,
From practical perspective, showed that selecting according to classification scores is the SOTA selection mechanism for comparison. Importantly, this method can be applied to an already deployed trained selective classification model and instantly improve performance at negligible cost. In addition, we showed a selective classifier trained with the entropy- loss and with selection according to Softmax Response achieves new state-of of- the-hrt results by a significant margin.
REPRODUCIBILITY STATEMENT
In our experiments, we build on the official implementations Self-Adaptive Train- ing available http://ggthbbccmmllaallllSSSssss Our code is available at

---

http://shhauoooo/////XXX///////2 The experiments with Deep Gamblers (Link: are run using the official implementaiton. Our Pytorch implementation of SelectiveNet follows the details in the original paper. The implemen- tation details are available in Section The hyperparameters are available in Section 5 and Appendix
ACKNOWLEDGEMENTS
The authors acknowledge funding from the Quebec government.
REFERENCES
Peter L. Bartlett and Marten Wegkamp. Classification with a reject option using a hinge loss. Mach. Learn. Res., 9:1823-1840, jun 2008. ISSN 1532-4435.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014.
Nontawat Charoenphaklee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning, pp. 1507-1517. PMLR, 2021.
C. Chow. On optimum recognition and reject tradeoff. IEEE Transactions on Information 19T.
Theory, 16(1):41-46, 1970. doi:
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. Advances in Neural Information Processing Systemx, 29, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Thomas G Dietterich and Alex Guyer. The familiarity hypothesis: Explaining the behavior of deep open methods. Pattern Recognition, 132:108931, 2022.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable Bayesian neural nets with rank-1 factors. In International Conference on Machine Learning, 2020.
Giorgio Fumera and Fabio Roli. Support vector machines with embedded reject option. In Interna- fional Workshop Support Vector Machines, pp. 68-88 82. Springer, 2002.
Yarin Gal and Zoubin Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.
Yonatan Geifman and Ran Yaniv. Selective classification for deep neural networks. Advances in
neural information processing systems, 30, 2017.
Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject
option. In International Conference on Machine Learning. pp. 2151-2159. PMLR, 2019.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Martin E. Hellman. The nearest neighbor classification rule with reject option. IEEE Trans. Syst. Sei. Cybern., 6:179-185, 1970.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv: 1et.:0//0222200 2016.
10

---

Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. Advances in neural information processing systems, 33:19365- 2020.
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: Bridging supervised and
self-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-17, 2022. doi: 10.1 109/TPAM120022.2277999
Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang. Selective classification can magnify disparities across groups. In International Conference on Learning Representations, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), 13), Sydney, Australia, 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back propagaopnnttion network. Advances in neural information processing systems, 2, 1989.
Joshua K Lee, Yuheng Bu, Deepta Rajan, Prasanna Sattigeri, Rameswar Panda, Subhro Das, and Gregory Wornell. Fair selective classification via sufficiency. In Marina Meila and Tong Zhang (eds.), International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6076-6086. PMLR, 18-24 Jul 2021.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. Advances in neural information processing systems, 29, 2016.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 2019.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference Machine Learning, pp. 7076-7087. PMLR, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pp. 776-794. Springer, 2020.
Tuan-Hume Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adver- sarial entropy minimization for domain adaptation semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2517-2526, 2019.
Marten Wegkamp. Lasso type classifiers with a reject option. Electronic Journal of Statistics, 1: 155-168, 2007.
Marten Wegkamp and Ming Yuan. Support vector machines with a reject option. Bernoulli, 17(4): 1368-1385, 2011.
Xiaofu Wu, Quan Zhou, Zhen Yang, Chunming Zhao, Longin Jan Latecki, et al. Entropy minimization vs. diversity maximization for domain adaptation. arXiv preprint arXiv:2002.01690, 2020.
Liu Ziyin, Zhikang T Wang, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippa Morency, and Masahito Ueda. Deep gamblers: learning to abstain with portfolio theory. In Proceedings of the 33dd International Conference Neural Information Processing Systems, pp. 10623-10633, 2019.
11

---

A APPENDIX: ADDITIONAL BACKGROUND AND BROADER IMPACT
A.1 DEEP GAMBLERS
Inspired by portfolio theory, Deep Gamblers proposes to train the model using the following loss function:
C Σ p(u)la)) log (o(usl -
where is the number of datapoints in the batch and is hyperparameter controlling the impact of the abstain logit. Smaller values of encourages the model to abstain more often. However, o I 1 makes it ideal to abstain for all datapoints and > C makes it ideal to predict for all datapoints. As result, o is restricted to be between 1 and C. Note that the corresponding loss function with large values of is approximately equivalent to the Cross Entropy loss.
A.2 BROADER IMPACT
The broader impact of this work depends the application of the selective model. In terms of the societal impact, fairness in selection remains a concern as lowering the coverage can magnify the difference recall between groups and increase unfairness (Jones al. 2021 Lee et al.) 2021).
The calibration step performed on the validation set assumes the validation and test data are sampled from the same distribution. Hence, in the case of out-of-distribution test data, a selective classifier calibrated to, for example, 70
When evaluating, Selective Classifiers may choose to predict samples with easier classes more than hard to predict classes. Thus, it would be undesirable in fairness applications that require equal coverage amongst the different classes.
B APPENDIX: ALTERNATIVE MOTIVATION
In the selective classification problem setting, the objective is to select Grarge proportion of samples for prediction according to the value outputted by selection function, g(x). Since each datapoint (I,, y4) is an i..dd sample, is optimal to iteratively select from the dataset D the sample x that maximizes the selection function, ie.,, x= argmax ((r)., until the target coverage Curget proportion of the dataset is reached. In other words, to select Curget proportion of samples (coverage Carge)). sufficient define the criterion and select a threshold such that exactly Carget proportion of samples satisfy ((x) >
B. SELECTING VIA PREDICTIVE ENTROPY
At test time, given dataset of datapoints D, if the labels were available, the optimal criterion to select the datapoint & E DD that minimise the loss function would be according
argmingeD CF (p((11z),
However, at test time, the labels are unavailable. Instead, we the model's belief over what
the label is, i.e., the learned approximation po(-(z) se p(1z)) We know CE (-(2a, pe(-(x)) = H Hppo where H is the entropy function. As such, we select samples according to
argmingeD CE (p((ll))
argminged H (po(-lx))-
In other words, entropy is an approximation for the unknown loss function. Accordingly, with respect to the discussed selection framework (Section 3.1.. the samples with the largest negative entropy value, i.e., g(x)
H (po (po(+(x)) (-(x)) are best nominees for selection.
In Figure la we show the distribution of entropy for a trained vanilla classifier, empirically showing entropy to be strongly inversely correlated with the model's ability to correctly predict the labels. As a result, entropy is a good selection mechanism. We include results on CIFAR-10 10 and Imagenet100 for vanilla classifier in Table and Table
12

---

Cissilet Predictive Enéraay Courk
Logit
3230
3300
CENTT
300
500
160
00821 o0n o/e one out 8:44 oue 9-w-tte 3 322 06400
8.203
(a) Entropy
(b) Max Class Logit
Figure 1: A bistogram of the number of datapoints according to a vanilla classifier trained on Imagenet100. The orange bar indicates samples for which the model correctly predicts the class of the sample. The blue bar represents the samples for which the model incorrectly predicted the class. the case of entropy, a lower value correponds to higher model confidence. contrast, in the case of max class logit, higher value correponds higher model confidence.
SelectiveNiet Predictive Entropy Count
selected Lerd Talse
selected Tue
400
0.0.0.11 0.0).002 0.620.04 0.04-.00
0.0.0.16..15. 0.116.6.. 632-0 06668 0.0001 000-002 6.62-004 D'L..00 6.000 1.00.16 16-6..22 6.3206 8664800
Predictive Entropy
Predictive Entropy
Figure 2: Entropy Comparison. SelectiveNet trained on Imagenet100 for a target coverage of 0.8 and evaluated on coverage 0.8. In the case of entropy, a lower value correponds higher model confidence. The histogram represents the counts of samples that were incorrectly predicted by the model. The left image indicates datapoints that were not selected by the selection head, i.e., datapoints with low selection value h(x)) r. The right image indicates datapoints that were selected by the selection head, ie., h(x)= 2 T.
B.2 SELECTING VIA MAXIMUM PREDICTIVE CLASS LOGIT (SOFTMAX RESPONSE)
inter- pretation of po (x])) a probability estimate of the true correctness likelihood, i.e., po u ()) the likelihood that is the correct label of æ. Let y/s be the correct label for 2;. For example, given 100 with Pa (u|z;) = 0.8, we would expect approximately 80% of the samples to have u as its label. As result, po (yr [xx])) is the model's probability estimate that the correct label is yi- In classification, the probability that the calibrated model predicts a datapoint x correctly is equivalent (o(u|z). Logically, the sample z; that should be selected for clasification is the sample the model's most likely to predict the sample correctly. i.e., = argmax, (ma.,e (I....))Po(u( (2)r)) This selection equivalent to selecting according to the following soft selection function g(x) = max..(1....C) (u(ax). Simply put, this is equivalent to selecting according to the maximum predictive class logit (aka Softmax Response (Geifman &
Given model with well-calibrated confidences (Guo al. 2017 Minderer et al. 2021).
samples {x),.... F0ooo! 1000}
the value of the max class logit, i.e., max
El Yaniy (017)).
In practice, neural network models are not guaranteed to have well-calibrated confidences. In Selective Classification, however, we threshold according to r and select samples above the threshold for classification, so we do not use the exact values of the confidence (max class logit). As result, we do not need the model to necessarily have well-calibrated confidences. Instead, suffices samples with higher confidences (max class logit) have higher likelihood of being correct. In Figure [b] we show the distribution max class logit for trained vanilla classifier, empirically showing larger class logit to be strongly correlated with model's ability to correctly predict the label.
13

---

As result, max class logit is good selection mechanism. We include results CIFAR-10 and Imagenet100 for a vanilla classifier in Table and Table 9
SelectiveNet Max Class Logit Count
selectod False
selected Tru
1200
1000
000
COMNcE
0.0-5.
0.507
0.5.4
09:..05
A.55..00 a 0.98-C.9E 0.960-10
0.0-0
D.507
0.7-g.
0.9-6.05
0 1.5-0.56 0.80-0 0.900-10
Lagit
Lugit
Figure 3: Max Class Logit Comparison. SelectiveNet trained on Imagenet100 for a target coverage of 0.8 and evaluated on coverage of 0.8. In the case of max class logit, higher value correponds to higher model confidence. The histogram represents the counts of samples that were incorrectly predicted by the model. The left image indicates datapoints that were not selected by the selection head, ie., datapoints with low selection value (xx) < r. The right image indicates datapoints that were selected by the selection head, iee., h(x) 2 T
B.3 RECIPE FOR BETTER SELECTIVE CLASSIFICATION
In this section, we further illustrate how SelectiveNet's original selection mechanism is suboptimal. The optimisation of SelectiveNet's ' selective loss selecttive (See Section 3.2.1 aims to learn selection head (soft selection model) g that outputs low selection value for inputs with large cross-entropy loss and high selection value for inputs with low cross-entropy loss. At test time, good performance of SelectiveNet depends on the generalisation of both the prediction and selection heads. However, learned models can at times fail generalise. In Figure 2 and Figure we show the distribution of entropy and max class logit for selected and not-selected samples according to SelectiveNet model. In the plots, samples with large entropy and low max class logit. In Table we see that the selection mechanisms based on entropy and max class logit outperforms the original selection mechanism. This comparison further supports our argument that the selection mechanism should be rooted in the objective function instead of separately calculated score.
SelectiveNet's 's original selection mechanism selects several
C APPENDIX: ADDITIONAL EXPERIMENTAL DETAILS
C.1 HYPERPARAMETERS
Following GGeifman & EI- Yaniv 2019), SelectiveNet was trained with target coverage rate and evaluated on the same coverage rate. result, there are different models for each experimental coverage rate. In contrast, target coverage does not play role in the optimization process of Deep Gamblers and Self-Adaptive Training, hence, the results for different experimental coverages are computed with the same models.
All CIFAR-10 10 experiments were performed with 5 seeds. All Imagenet-related experiments were performed with 3 seeds. For hyperparameter tuning, we split Imagenetl 00's training data into 80% training data and 20% validation data evenly across the different classes. We tested the following val- for the entropy minimization coefficient 3 e {0.1, 0.01, 0.001, 0.0001}. For the final evaluation, we trained the model on the entire training data.
Self Self-Adaptive Training models trained using SGD with initial learning rate of 0.1 and momentum of 0.9.
Food 0tt////mageee1100mmaaaaaaa size of 128. The learning rate was reduced by 0.5 every 25 epochs. The entropy- minimizationn term B = 0.01.
The models were trained for 500 epochs with mini-batee
14

---

CIFAR-10. The models were trained for 300 epochs with mini-bbtchh size of 64. The learning rate was reduced by 0.5 every 25 epochs. The entropy-min minimization term was 3 = 0.001.
StanfordCars. The models were trained for 300 epochs with mini-batch size of 64. The learning rate was reduced by 0.5 every 25 epochs. The entropy-ming minimization term was 3 0.01.
Imagenet. The models were trained for 150 epochs with mini matt-bbthh size of 256. The learning rate was reduced by 0.5 every 10 epochs. The entropy-min minimizationn term was 3 = 0.001.
C.2 COMPUTE
The experiments were primarily run GTX 1080 Ti. The CIFARIO experiments took 1.5 hours for Self-Adaptive Training and Deep Gamblers. SelectiveNet experiments took hours each. The Imagenet100 experiments took 2 days for Self-Adaptive Training and Deep Gamblers. SelectiveNet experiments took 2.75 days each. The ImagenctSubset experiments took 0.5-4.5 days each for Self-Adaptive Training and Deep Gamblers, depending the number of classes. SelectiveNet experiments took 0.75-5.5 days each, depending on the number of classes.
C.3 IMAGENETSUBSETT CLASSES
ImagenetSubset comprises of multiple datasets ranging from 25 to 175 classes in increments of 25, ie., (D2s, Ds0, D7s, D1v0. D125, D130. D17s]. Let C2s; Cso..... C175 represent the classes of the respective datasets. The classes for ImagenetSubset are uniform randomly sampled from the classes of Imagenet such that the classes the smaller datasets are subsets of the classes the larger datasets, iee. D25 D50 D75 .... Di75 and C25 C. The list of Imagenet classes in each dataset is included below for reproducibility.
C.3.1 C25
n03133878 n03983396 n03995372 n03776460 n02730930 n03814639 n03666591 n031 10669
n04442312 m02017213 n04265275 n01774750 n03709823 n09256479 n07715103 n04560804
n02120505 n04522168 n04074963
n02268443
n03291819
n02091467
n02486261
n03180011
n02100236
C.3.2 C50 C25
n02106662 n01871265
n12057211 n04579432 n07734744 n02408429 n02025239 n03649909
n03041632 n02484975 n02097209 n03854065 n03476684 n04579145 n01739381 102319095
n01843383 n02229544 n09288635
n02138441
n021 19022
n07583066
n03534580
n02817516
n04356056
C.3.3 C75 C50
n03424325 n04507155 n02112350 n03450230 n01616318 n01641577 n03630383 n01530575
N02102973 n04310018 n02134084 n01729322 n03250847 n02099849 n03544143 n03871628
n03777754 n04465501 n01770081 m03255030
n01910747
n03016953
n03485407
n03998194
n02129604
C.3.4 C100 C75
n02128757 n03763968
n01677366 n03483316 n02177972 n03814906 n01753488 n02116738
n01755581 n02264363 n03290653 nI3133613 n03929660 n04040759 n02317335 n02494079
n02865351 n03134739 n02102177 mo4192698
n02814533
no4090263
n01818515 n01748264
104328186
C.3.5 C125 C100
n03930313 n02422106 n07714571 m02111277 n03706229 n03729826 n03344393 n07831 146
n02090379 n06596364 n03187595
n04317175 nl 11199949
n04277352 n01807496
n02804610
15

---

Coverage
Self-Adaptive Training
Deep Gamblers
SelectiveNet
MC-Dropout
100
5.91 = 0.04
6.08 0.00
6.47 + 0.22
6.79 0.03
95
3.73 # 0.13
3.71 = 0.00
4.12
4.58 1.05
90
2.18 = 0.11
2.27 0.00
0.13
0.01
85
1.26 # 0.09
1.29 = 0.00
#.08
1.82
80
0.69 + 0.04
0.81 0.00
0.86
1.08 = 0.05
75
0.37 + 0.01
0.44 = 0.00
0.53 = 0.06
0.66 + 0.05
70
0.27 + 0.02
0.30 0.00
0.42 = 0.04
0.43 0.05
Table Comparison of existing Selective Classification baselines with MC-Dropout. The results of MC-Dropout are originally from Geifman & El- (2017). For given coverage, the bolded result indicate the lowest selective risk (i.e. best result) and underlined result indicate the second lowest selective risk.
n02093991 n09428293 nO3207941 n02132136 n04548280 n02793495 n03924679 n021 12137 n02107312
C.3.6
C150 C125
n03376595 n03467068 n02837789
n04467665 n04243546
n03530642 n04398044
n02113624
n13044778 n031 88511 nt1729977 n01980166 n02101388
n01629819 n01773157 n0168981
n02109522 n03938244 n02123045
n04548362
2254
n04264628
n02108551
n04311174
n02276258
C.3.7 C175 C150
n03724870 n02087046 n09421951 n02799071 n07717410 n02906734 n02206856 n03877472
n01740131 n04523525 n03496892 n04116512 n03743016 n03759954 n04462240 n03788195
n02137549 n03866082 n02233338 n02219486 m02445715 n02974003
n01924916 n12620546
n02992211
D APPENDIX: ADDITIONAL RESULTS
Briefly summarised, the additional interesting results found are as follows: (1) In low coverage vanilla classifier trained via the
settings, selecting based on Softmax Response and Entropy cross entropy loss outperform both SelectiveNet and Deep Gamblers. (2) SelectiveNet outperforms Deep Gamblers moderate coverages (60%, 50%, and 40%) which reported results. We attribute the interesting results to our work being the first to evaluate these methods on large datasets wide range of coverages. Since previous works have mainly focused on toy datasets and high coverages (70 + %), they failed to capture these patterns, The main takeaway of these results, however, that, across the reported methods, selecting via Softmax Response best.
in par with the previously
D.I ANALYSIS OF SELECTED IMAGES: SELECTIVENET
We include in Figure and Figure examples of images where the selection head of SelectiveNet fails to generalise.
D.2 SELECTION MECHANISMS: MC-DROPOUT
In Table 7 we see that MC-Dropout performs worse than the existing state-of-the-ee methods for Selective Classification.
16

---

Unconfident and Incorrect Predictions but Selected Images according to Selection Head Selection Score Threshold: 0.93
Selection Daterman pinscher
Salectint Score:
Image Label:
Labsl paw sapaee Power
Inace Label: lamorder standard poodle
Labst Nortson lbbit
58 1ety Sorweeer
nossenee Cull Line rock crat. Lencer mordbus
0.41 (2nck pinccher
28 (2nd: plneappee aranas 0.03 Jrs cnemm. move thesler
C.0 Lind Chesopecie Bay incowver
6:05 Lire) voail. Iaurganan pointer
Selection Score. 0.5g Image Amsnican Nortran (ddeerenhh ne (2nr) Americaa waster Northem Icheser (315) mousttin
Solocior Score, 00 Imsge Chitustia [baly (pedy Thi ania no (trr
5280c101 Score: 0 99 Image tub, out (1et 38 (2nd) 1uh
Selocion Score. 0
Image Dara Ameeran Ststhweehi Stafhertesie 26 12nry vidh, Hinger on ralager 0:10 Vhiler
Solectio mage Labid Great Dane SuAe
Selector mage Laboi drss. Canoer mnreeuu
D.4E passenng
Lebel bontecap
Imag Label troon
(altt moria beard
0.43 (ind): 2 Dara
0,55 (2hd)) cuchtal shaker [Ted)
0.42 (2nd)) (Pasperr theatre cumaain
0.23 theel rack crab. Carcue intratis 6,62 Creaaseanc Bay Nerthsse
Amee ran Stathwrthh temter, Stafthetchire TeRTiNe
thene
Selection Score I. 0.57 (1st). + ddforciso cual, Cancer 0.02 (Bd)
Selector on Scrne: US 0.2 (2st) aidus. piclup bruck 0.03 (hdi muped
Select on scane: U 97 44 (2st: caull flowrr 6.181.3d saltomare
Select Serew: 6,30 11tt ChiPuaTAA 0.30 (3m))
Figure 4: SelectiveNet (at 80% coverage) on Imagenet 100: Incorrect and unconfident predictions according its classifier but selected images according its Selective Head. The selection score threshold for selecting images for prediction is 0.93. Included are the selection score according to the Selection Head, the image's 's label, and the top-3 predicted classes according the classifier and their respective classifier scores.
17

---

Confident and Correct Predictions but Rejected Images according to Selection Head Selection Score Threshold: 0.93
Selection Store 0.01 0.9 (1st; Tansycoms (hed) tvmns 3.00
Selection Score: 84 0.92 (181) treater curtan, theatre curtans
Stiecten Scoee: 25 0.99 (35t]: morta roea sarong
Selecton Sraee: 0 31 0.68 (251)) nonsycome (3rr) thenne
12ne) (3r)) theater theatre
0:00 (fed]
Eduir
Inaget Labrl Durch even 0.02 (2nd): retissene 3rd) mil
Imagh sbet Debetman Debrrnes pireeter
mage Libel notter the
mage Label honeyycamt
0.00 (2na): fosteetlrr lary lathosper
0.02 (208) rock znb. Concer intratus 500 (383) Wathesper
0.00 (2nd): Bonnet, poke bonnit
Hoor shane, arder. sano vil
Selecton Scate, 2,64 U.d (2eil) langur 0.00 3rd pedconal plrnt. Trunstal
Seleciion Stare: 20
Selecien Stare: 0 02 Cai (Jalls masaebres 0.07 3rd) calety
Selector Stare: 0
2.97 (Leie sapole, prente well, braen will no Elids red fea. Wipes wubees
0.97 (aell walknng sish wellengliss slick now
bot (3ei) dung bedii
Selection imag Label mouuetto
Selecten Image Latal Sauit gatelle hound 0.0 (Ands Chituania
Seletior Score: mane Lakel langur
Selectin Score
Image Lacel ingean, lapoce comcuber 0.02 tandi con tule beyboutt heeppd
3.00 (2dd): hort des: hard rice. ford deot
0,22 Land pibenn Pylobuit kar
Advanced
rap pack
Figure 5: SelectiveNet (at 80% coverage) on Imagenet100: Correct and confident predictions according to its classifier but rejected images according to its Selective Head.
18

---

Vanilla Classifier
Dataset
Coverage
Entropy
Softmax Response
100
6.61 = 0.25
6.61 = 0.25
95
4.30 0.19
4.35 0.17
90
2.63 = 0.12
2.63 - 0.11
CIFAR-10
85
1.62 = 0.10
1.63 0.12
80
1.01 = 0.13
0.99 0.10
75
0.72 = 0.08
0.72 0.08
70
0.57 = 0.08
0.55 + 0.07
Table 8: Comparison of selection based on Entropy and Softmax Response for vanilla classifier trained with cross-entropy loss on CIFAR-I0.
Vanilla Classifier
Dataset
Coverage
Entropy
Softmax Response
100
14.32 0.14
14.32 # 0.14
90
9.14 + 0.05
8.96 = 0.13
80
5.34 0.12
4.99 = 0.05
70
3,04 # 0.14
2.83 = 0.12
60
1.80 + 0.14
1.70 = 0.19
Imagenet
50
1.22 + 0.31
1.08 = 0.28
40
0.82 + 0.32
0.77 = 0.39
30
0.63 # 0.33
0.60 = 0.28
20
0.60 + 0.28
0.60 + 0.28
10
0.30 # 0.14
0.20 + 0.28
Table 9:
Comparison of selection based on Entropy and Softmax Response for vanilla classifier trained with cross-entropy loss on Imagenet100.
D.3 SELECTION MECHANISMS: VANILLA CLASSIFER
D.3.1 CIFAR-10
In Table the difference in performance between selecting according to entropy and selecting according to Softmax Response not significant. We attribute this marginal difference to the saturatedness of the CIFAR- 10 dataset.
D.3.2 IMAGENET 100
In Table we see that selecting according to Softmax Response clearly outperforms selecting according to entropy. We see that Softmax Response learns a less generalizeable clasifier (See performance 100% coverage) than Self-Adaptive Training, Deep Gamblers, and SelectiveNet. However, interestingly, we found that Softmax Response outperforms both Deep Gamblers and SelectiveNet on low coverages (10%, 20%, 30%). Previous works failed to capture this pattern due to lack of evaluation on larger datasets and lower coverages.
D.4 SELECTION MECHANISMS: DEEP GAMBLERS
CIFAR-10. these results (Table 10 we see that the difference performance between the various selection mechanisms is marginal. Due to the marginal difference between errors, it is difficult to make conclusions from these results.
Imagenet100. In Table 10 outperforms the original selection mechanism.
that selecting according to Softmax Response and Entropy clearly
ImagenetSubset. In Table II similar to Imagenet100, using Softmax Response as the selection mechanism instead the original selection mechanism. Furthermore, that Entropy also outperforms the original selection mechanism.
clear substantial improvement when
19

---

Deep Gamblers
Dataset
Coverage
DG
DG + Entropy
DG + SR
100
6.08 0.00
6.08 1 0.00
6,08 0.00
95
3.71 0.00
3.79 + 0.00
3.81 0.00
90
2.27 0.00
2.14 0.00
2.16
CIFAR-I0
1.29 0.00
1.31
1.35 0.00
80
0.81 0.00
0.84 # 0.00
0.85 = 0.00
75
0.44
0.57 0.00
0.56 = 0.00
70
0.30 0.00
#.000
0,43 = 0.00
100
13.49
13.49 = 0.52
90
8.42 0.44
8.24
8.11 0.48
80
5.21 0.32
4.76
4.52 = 0.38
70
3.30 0.40
2.20
2.58 0.21
60
2.14 0.37
1.86 + 0.32
1.71 0.32
Imagenet100
1.55 + 0.27
1.35 + 0.25
1.31 0.22
40
1.23 # 0.38
1.20 # 0.11
1.07 + 0.19
30
1.09 0.31
1.00 + 0.19
0.96 = 0.21
20
1.03 # 0.31
0.97 0.21
0.90 # 0.22
10
0.80 0.28
0.73 + 0.25
0.53 # 0.25
Table 10: Deep Gamblers Results on CIFAR-10 and Imagenet100. Comparison of Selection Mecha- nism Results.
Deep Gamblers
Dataset
of Classes
DG
DG + Entropy
DG SR
I75
3,77 0.10
3.75 0.14
3.62 # 0.11
150
2.62 0.03
2.65 = 0.26
2.54 0.24
125
2.58 +
2.19
2.22 # 0.17
ImagenetSubset
100
0.00
2.20 # 0.04
75
2.60 0.20
2.00
2.22 + 0.05
50
2.63 0.12
2.15 0.05
2.08 0.07
25
1.60 + 0.28
1.22 + 0.30
1.30 + 0.19
Table 11: Deep Gamblers Results on ImagenetSubset (70% coverage) with various selective mecha- nisms.
D.5 SELECTION MECHANISMS: SELF-ADAPTIVE TRAINING
ImagenetSubset. In addition the Imagenet 100 experiments, also evaluate Self-Adaptive Training trained with the proposed entropy-rogularised loss function on ImagenetSubset.
In Figure 6 (and Table 12 and Table 13), we see that training with the entropy-regularised loss function improves the scalability of Sell-Adaptive Training when selecting according to Softmax Response
In Figure 6 the results for SelectiveNet, Deep Gamblers, and Self-Adaptive Training on 70% coverage. Consistent with previous experiments, we see that both the selection mechanisms based on the classifier itself (predictive entropy and Softmax Response) significantly outperform the original selection mechanisms of the proposed methods. These results further support our conclusion that (1) the strong performance of these methods were due to them learning more generalizable model and (2) the selection mechanism should stem from the classifier itself rather than a separate head/logit. Similarly,
see that Softmax Response is the state-of-the- the experiments, we see that SelectiveNet struggles to scale to harder tasks. Accordingly, the achieved improvement in selective accuracy with Softmax Response (SR) increases the number of classes increase. This suggests that the proposed selection mechanism is more beneficial for SelectiveNet as the difficulty of the task increases, i.e., improves scalability.
selection mechanism. In
20

---

SelectiveNet
Deep Gamblers
4.0
Selection Head
Abstain Logit
Entropy
3.5
Entropy
Selective Risk (95)
SR
3.0
2.5
2.0
1.5
1.0
25
50
75
100
125
150
175
25
50
75
100
125
150
175
Number Classes
Number Classes
(a) SelectiveNet
(b) Deep Gamblers
Self Adaptive Training
Self Adaptive Training with Entropy Minimization
secivee Risk (86)
2.5
2.5
2.0
2.0
Abstain Logit
1.5
Aostain Logit
Entropy
Entropy
1.0
SR
1.0
SR
100 Number of Classes
Number of Classes
(c) Self-Adaptive Training
(d) with Entropy Minimization
Figure 6: ImagenetSubset at 70% coverage. (a), (b), and (c) Various Selective Models (d) Comparison of Self-Adaptive Training trained with the proposed entropy-regularised loss and without. The loss function improves the scalability of Self Ada--Aeeee Training, particularly when using Softmax Response as the selection mechanism.
21

---

SAT
SAT
Entropy
SAT+ + Softmax Response
# Classes
SAT
EM
S
SAT+EM+E
SAT SR
SAT+EM+SR
175
5.03
0.16
3.01 0.09
2.80 0.07
2.88 3 0.15
2.73 0.07
150
2.13
0.20
2.07
1.13
0.15
1.71 0.15
125
0.25
2.24
0.19
1.90
0.16
1,84 0.14
100
0.19
0.22
2.20
1.87
0.29
1.81 0.08
75
2.60
0.40
1.98
0.29
1.88
1.68 - 0.27
50
2.86
2.34
0.24
1.47
1.98
0.27
1.47 + 0.12
25
1.79
0.53
1,87 = 0.05
1.14 0.19
1.14
1.18
1.14 0.25
Table 12: Self Adaptive Training with Entropy Minimization Loss Function on ImagenetSubset at 70% coverage. We see that SAT+EM+SR performs the best and outperforms SAT by statistically significant margin.
D.6 ABLATION: VARYING ARCHITECTURE
In these experiments, show generalizability across architectures of proposed entropy- minimization and softmax response methodology. In Tables 14 [5) and [6] we see that applying the entropy-minimizznn and Softmax Response methodology improves upon the state-of-the-art method's '' performance significantly.
D.7 RISK-COVERIE PLOTS
Figure shows the risk coverage plots for Imagenet100, Food101, and StanfordCars results.
D.8 LEARNING CURVES PLOTS
Figure shows that SAT and SAT+EM models have converged on StanfordCars.
22

---

Self-Adaptive Training
Dataset
Coverage
# of Classes
SAT
SAT EM SR
175
0.69 = 0.12
0.46 0.05
150
0.44 0.13
0.16 0.02
125
0.44 0.07
0.14
30
100
0.71 0.11
0.15 + 0.06
75
0.10
0.09 0.00
50
0.76
0.16 + 0.05
25
0.00
0.08 + 0.11
I75
0.94
0.14
150
0.04
0.06
125
0.76
0.25
40
100
0.90
0.00
75
0.144
0.23
50
1.17
0.27 2
25
0.25
0.07
I75
1.17
0.11
150
0.11
0.47 2 0.05
125
0.91
0.52
50
100
1.11
0.56
75
1.01
0.40
50
1.34
0.37 +.3
25
0.24
0.21
175
1.77
1.24
150
1.21
125
1.34
0.95
InagenetSubset
60
100
1.07
0.93
75
1.51
0.78
50
1.78
0.69
25
0.93
0.17
I75
3.03
0.73
150
0.16
1.11
125
0.22
1.14
70
100
2.19
1.81
75
0.15
1.68 +.27
50
2.36
1.47
25
1.79
1.14 0.25
175
5.85
0.17
150
4.06
3.88
125
4.28
3.34
80
100
4.94
3.96 0.02
75
4.21
3.78
50
5.14
3.35 0.36
25
4.33
2.80
175
10.14
9.69
150
1.00
8.08 #
125
8.87
8.18
90
100
8.90 0.54
8.28
75
8.57 0.44
50
8.79 # 0.17
6.96 0.64
25
7.79 + 0.43
6.84
Table 13: ImagenetSubset Results
23

---

ResNet34
Coverage
SAT
SAT+SR
SAT+EM+SR
100
37.68 + 1.11
37.68 + 1.11
32.49 + 2.33
90
32.34 + 1.19
32.04 1.18
26.60 2.39
80
26.86 1.15
26.39 + 1.13
20.87 2.33
70
21.34
20.70 + 1.23
15.84
60
16.21:1.1
14.92 1.03
1
50
10.25
7.00
40
0.43
6.69
4.00 = 0.87
30
4.56
0.36
2.20 = 0.44
20
2.42
1.93
1.17 0.28
10
1.49 + 0.00
1,20 + 0.21
0.80 + 0.22
Table 14: ResNet34: StanfordCars results
RegNetX
Coverage
SAT
SAT+SR
SAT+EM+SR
100
31.78 + 2.44
31.78 2.44
27.75 + 1.81
90
26.35
25.68 2.44
=
80
2.440
20.07 2.54
16.21 1.79
70
14.77 2.14
11.22
60
12.13
10.07 + 1.58
7.39
50
8.60
6.43 1.46
4.95
40
5.94
4.04 0.88
2.88 + 0.61
30
0.60
2.47 0.44
1.74
20
2.55 + 0.33
1.55 + 0.00
1.10 = 0.34
1.66 0.26
0.91 2 0.15
0.70 0.26
Table 15: RegNetX: StanfordCars results
ShuffleNet
Coverage
SAT
SAT+SR
SAT+EM+SR
100
34.10
0.73
34.10 0.73
32.90 1.29
90
28.61 # 0.72
28.27 + 0.80
26.94 1.33
80
23.16
22.72 0.63
21.13 1.40
70
1.224
17.14 0.46
15.70 + 1.42
60
13.00
12.10 0.46
10.89 + 1.19
50
9.23 0.10
7.68 0.10
7.11 0.87
40
6.31 0.22
4.77 0.24
4.49 0.51
30
3.81 0.39
2.97 # 0.25
2.83 + 0.28
20
2.07 0.34
1.70 0.30
1.43 2 0.05
10
1.08 0.26
0.99 0.18
0.66 0.31
Table 16: ShuffleNet: StanfordCars results
24

---

Imagenat100 |Coverages
Imagenat100 (Coverages 50)
SAT
SAI
SAT+5n
1.2
SAT-5R
SAT+EN 4cM
SAT-EN
1.0
SAI 4M
Serective Cleechicto
0.0
Coverage
Coverage
Imagenet100 Results: High Coverages
(b) Imagenet100 Results: Low Coverages
Foud101 Results (Coverages 50)
Food100 Results (Covereges 501
16
SAT
Selebbe Clave Piaettn Ervor PS
Selectvv Claesiiicahn brnot
1.2
12
1.0
0.8
50
60
70
80
10b
10
70
30
40
50
Eoownage (56)
Esterrgee 171
(c) Food101 Results: High Coverages
(d) Food101 Results: Low Coverages
StantordCars Results
StantordCars
12-
SAT-SR
S4T-SR
SAT 6 55
10-
SAT SAT-
30
Selectee dlosiraation caltonn
Selective caton
25
10
(c) StanfordCars Results: High Coverages
(f) StanfordCars Results: Low Coverages
Figure 7: Risk Coverage Plots for Imagenet10, Food101, and StanfordCars, All plots show that SAT+EM+SR outperform SAT across all coverages, achieving state-of-the-art results.
StanfordCars: SAT Learning Curve
StanfordCars: SAT+EM Learning Curve
Tain Lows
Tain Laws
Text Loss
Test Loss
100 150
200
250
300
50
100
150
200
250
300
Number Epochs
Number of Eoochs
(a) StanfordCars: SAT
StanfordCars: SAT+EM
Figure 8: Learning Curve (Convergence) Plots for StanfordCars.
25


Reviews:
[{"review_id": "zY8S20T07n1", "reviewer": "Reviewer_F9DQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The problem of Selective Classification considers how to train the model’s ability to abstain from a decision when the credibility is low. In this paper, the authors analyzed the state-of-the-art methods and found that learning a more generalizable classifier is more important than using some selection mechanisms in the problem of Selective classification. Besides, they found that the Softmax Response performs better than the entropy-based selection mechanism. Through researching the relationship between Selective Classification and Semi-supervised learning, the authors proposed an entropy-regularized loss function to improve the performance. The experimental results show the effectiveness of the proposed method.", "summary_of_the_review": "* The authors may consider adding some visual expressions to help readers better understand the proposed method.\n* The proposed method is mainly based on the  entropy-regularized loss function, which has a small amount of space in the article. The authors may point out more properites of the proposed method, so that readers can better appreciate the method. Besides, the authors may consider demonstrating the advantages of the proposed method compared with the state-of-the-art methods from the theoretical level.\n* In the experiment, $\\beta$ is chosen as 0.01 for the proposed method, the authors may consider giving some specific experiments to illustrate the effectiveness of the method under different settings.  The experiments do not seem to work well on the Cifar10 dataset, while the good results are on the two datasets constructed by the authors themselves. The authors need to add some datasets which also used by other researchers to improve the credibility of the experiment.\n\n* Typo: the the -> the (cf. the last paragraph line 4 on page 4)", "strength_and_weaknesses": "**Strengths**\n\n1. The authors introduced the state-of-the-art methods (SelectiveNet, Self-Adaptive Training, and Deep Gamblers) and the proposed method through mathematical expressions. \n2. This paper analyzes the actual effective part of the state-of-the-art methods and the correlation between the Selective Classification problem and Semi-supervised learning. The proposed method has a better research premise and innovation.\n3. The experiment compares the state-of-the-art methods with the proposed method and illustrates the point of view of the paper.\n\n**Weakness**\n\n1. The proposed method is a bit hard to follow.  \n2. The technique of the proposed method seems to be simple (i.e., mainly from the  entropy-regularized loss function).\n3. The experiment is not well executed (e.g., $\\beta$ is chosen as 0.01 for the proposed method, and not results under different settings).  The experimental results do not seem to work well on the Cifar10 dataset, while performing well on the two synthetic datasets generated by the authors. This kind of experimental results is unconvincing, as they may look artificial.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JJMgS8E800c", "reviewer": "Reviewer_wKA4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors point out that predictive distribution base classifier within in selection classification captures\nbetter signal for selection rather than a separate selection/abstention logit present in various approaches of selective classification within deep learning. \n", "summary_of_the_review": "Though authors make an interesting point and demonstrate effectiveness of their observations. For lack of novelty, or lack of additional insights, currently in my humble opinion not convinced if the work warrants an ICLR acceptance. ", "strength_and_weaknesses": "Strengths:\n\nPaper is well written and easy to understand, and authors include various techniques of selective classification.\n\nWeakness:\n\nThough the point made is effective in experiments its only marginally better, including entropy regularized loss.\nOn CIFAR, we see the difference as we decrease coverage to 70 but even at 80% coverage selection mechanism and softmax response are at par and already <1% error. In case of imagenet we see 1.5% improvement at 80% coverage.\n\nAlso, authors do not provide further insight into why the observation exists, inspired by OOD detection work which demonstrate that max-logit is best performing OOD metric, and [3] makes an argument that max-logit is better because within Deep learning we check for familiarity of an input rather than absence of key features. So, based on OOD/anomaly detection signals & familiarity hypothesis we know it's always easy to detect presence than reject/ do good uncertainty quantification within our current deep learning practices.\n\nIs there any reason for authors to not consider \n\t1. H. Mozannar et al. Consistent Estimators for Learning to Defer to an Expert (Learning to Defer) (ICML 2020)\n\t2. N. Charoenphakdee et al. Classification with Rejection Based on Cost-sensitive Classification (ICML 2021)\n        3. T.G. Dietterich et al. The Familiarity Hypothesis: Explaining the Behavior of Deep Open Set Methods (arXiv: 2203.02486)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CXmePsLj8g", "reviewer": "Reviewer_hDSo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles an interesting question regarding model decision making. Specifially, the authors are looking into the ability of a model to abstain from making a decision, when it has low confidence scores: The “Selective Classification problem”. \n\nThe authors demonstrate that: \n1) the current SOA models’s performance can be attributed to their ability to train a more classifier than is more generalizable (than the logit selection mechanism).\n2) semi-supervised learning-based models have better selective classification ability. \n3) the new selective classifier (SR: SoftMax response) could improve current SOA models\n", "summary_of_the_review": "Overall the paper addresses an important question about Selective Classification. I was wondering if there are practical applications of this specific decision making approach. It would be a good addition to articulate the utility of this approach, especially in CV related tasks. The relation with the semi-supervised learning approaches is also interesting but less well understood. I encourage the authors to spend more time fleshing that section out in the paper (or in future work). ", "strength_and_weaknesses": "Pros:\nI found the paper to be generally very well written and the experiments well motivated. The most interesting part of the paper to me was the connection with self-supervised models.\nI also appreciated the authors reporting the results with errorbar statistics. I wish more CS/AI researchers emulate this\nI found the results to be quite thorough. I liked seeing the results generalize to different tests and reported across the number of classes (where selective decision becomes more of a problem). \n\nCons:\nIn some instances, I felt the results were slightly overstated. The difference between the entropy loss and the SR loss is relatively subtle across the board. \nThe relation with semi-supervised learning-based models could be better explored. At the moment it is an observation. Readers would like to get an intuition for why this happens. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AUdSnu_imK", "reviewer": "Reviewer_6JTV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to revise the conclusions of two different selective classification methods. They argue this methods obtain betters results by just providing a more robust training. Thus, at inference step, the selective solution can be discarded. With this variation, they are able to increase the results obtained by the original paper. Furthermore, they provide an entropy regularization term that can be used in semi-supervised learning. ", "summary_of_the_review": "Although the idea is interesting, the proposed solutions are either not new (entropy-regularization term) or not properly explained. Thus, I cannot recommend to publish this contribution in its actual form.", "strength_and_weaknesses": "Strength\n* The idea is simple and easy to implement\n* The results are interesting\n\nWeaknesses\n* The semi-supervised entropy regularization is not new [1].\n* Only one network was tested.\n* The paper organization is poor.\n\n[1] Grandvalet, Y., & Bengio, Y. (2004). Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster