Curriculum-inspired Training for Selective Neural Networks
Authors: Rui Liu
Reza Soroushmehr
Barzan Mozafari
Venue: Submitted to ICLR 2023 (2023)
Keywords: curriculum learning
selective classification

Abstract: We consider the problem of training neural network models for selective classification, where the models have the reject option to abstain from predicting certain examples as needed. Recent advances in curriculum learning  have demonstrated the benefit of leveraging the  example difficulty scores in training deep neural networks for typical classification settings. Example difficulty scores are even more important in selective classification as a lower prediction error rate can be achieved by rejecting hard examples and accepting easy ones. In this paper, we propose a curriculum-inspired method to train selective neural network models by leveraging example difficulty scores. Our method tailors the curriculum idea to selective neural network training by calibrating the ratio of easy and hard examples in each mini-batch, and exploiting difficulty ordering at the mini-batch level. Our experimental results demonstrate that our method outperforms both the state-of-the-art and alternative methods using vanilla curriculum techniques for training selective neural network models.

Under review as a conference paper at ICLR 2023
CURRICULUM-INSPIRED TRAINING FOR SELECTIVE NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the problem of training neural network models for selective classifi- cation, where the models have the reject option to abstain from predicting certain examples needed. Recent advances in curriculum learning have demonstrated the benefit of leveraging the example difficulty scores in training deep neural net- works for typical classification settings. Example difficulty scores important selective classification as lower prediction error rate can be achieved by rejecting hard examples and accepting easy ones. In this paper, propose curriculum-inspired method to train selective neural network models by leverag- ing example difficulty scores. Our method tailors the curriculum idea selective neural network training by calibrating the ratio of easy and hard examples in each mini-batch, and exploiting difficulty ordering at the mini-batce level. Our exper- imental results demonstrate that our method outperforms both the state-of-the-ar and alternative methods using vanilla curriculum techniques for training selective neural network models.
INTRODUCTION
In selective classification, the goal is to design predictive model that is allowed abstain from making prediction whenever is not sufficiently confident. A model with this reject option is called selective model. In other words, a selective model will reject certain examples as appropriate, and provide predictions only for accepted examples, In many real-life scenarios"] such as medical diagnosis, robotics and self-driving cars (Kompa al. 202T), selective models are used to minimize the risk of wrong predictions on the hard examples by abstaining from providing any predictions and possibly seeking human intervention.
In this paper, we focus on selective neural network models, which are essentially neural network models with the reject option. These models have been shown to achieve impressive results (Geff- man & Fl- Yaniv, 2019 2017 Liu et al., 2019). Specifically, Geifman & EI- Yaniv/ (2019) proposed a neural network model, SELECTIVENET, that allows end-to end-to-end optimization of selective models. SELECTIVENET contains a main body block followed by three heads: one for minimizing the error rate among the accepted examples, one for selecting the examples for acceptance or rejection, and one for the auxiliary task of minimizing the error rate all examples. These three heads are illus- trated later in Figure [ examples while satisfying a coverage constraint terms of the least percentage of examples that need to be accepted. The coverage constraint is imposed to avoid the trivial solution of rejecting all examples get a 0% eтror rate. Ideally, the model should reject hard examples and accept easy ones to lower its overall error rate. While it is clear that difficulty scores are helpful, they are typically un- known in most settings. Therefore, to leverage difficulty scores we must overcome two challenges: (1) how to obtain the difficulty scores as accurately as possible and (2) how to best utilize them in selective neural network model.
The final goal of this model is to minimize the error rate among the accepted
Recent curriculum learning techniques have investigated how to use example difficulty scores to improve neural network models' performance (Hacohen & Weinshall 2019 Wu et al. (020). To the best of our knowledge, these techniques only consider the typical classification setting where the error rate on all examples should be minimized. Curriculum learning techniques often use a
Thhse further elaborated in Section A.1 the appendix.

---

Under review as a conference paper at ICLR 2023
scoring function to estimate the difficulty scores. There are different approaches to constructing a scoring function from a reference model, such as (1) confidence score (Hacohen & Weinshall
2019). (2) learned epoch/iteration (Wu et al., 2020), and (3) estimated c-score (Jiag et al. 2220)_ Prior work has shown that these scoring functions are highly correlated with one another and lead to similar performance (Wu et al. 2020). Existing curriculum learning techniques
the estimated
difficulty scores only to decide the order of examples being exposed to the model during the training process. Typically, they expose easy examples the model during the early phase of the training process, and gradually transition to the hard examples as the training progresses. In this paper, we consider selective classification, which has more relaxed goal: minimize the error rate on the accepted examples for given coverage constraint. Difficulty scores are even more useful in selective classification, because the selection of accepted examples has a direct impact on the model's s final performance (i.e., a lower enror rate can be achieved by rejecting hard examples and accepting easy ones). This inspires us to design new method for training selective neural network models.
Contributions. In this paper, propose new method for training selective neural network models inspired by curriculum learning. Our curriculum- innspuredd method has two benefits. First, existing training methods ignore the coverage constraint when constructing the mini-batch for each iteration, which we show will introduce misguiding noise to the training progress of selective neural network models. Taking advantage of the estimated difficulty scores, our method calibrates the ratio of easy and hard examples in each mini-batch to match the desirable value due to the coverage constraint. Second, our method also adopts the curriculum idea of increasing difficulty levels gradually over the training process, which has been demonstrated to improve neural network training (Hacohen & Weinshall 2019: Wang et al., 202T. However, instead of relying on existing vanilla curriculum techniques that exploit difficulty ordering at the example level, our method is tailored to selective neural network training by exploiting difficulty ordering at the mini-batch level. We will show that this change is necessary because of the need to match coverage constraint as just explained.
To summarize, we make the following contributions:
1. To the best of our knowledge, we are the first to investigate the benefits of leveraging example difficulty scores to improve selective neural network model training.
2. We design new curriculum-inspired method with two benefits for training selective neural networks: (a) calibrating the ratio of easy and hard examples in each been ignored by existing methods (Section 4.1;; (b) adopting a curriculum idea which is tailored to selective neural network training by exploiting difficulty ordering at the mini- batch level (Section 4.2p.
which has
3. We conduct extensive experiments demonstrating that our method improves the converged error rate (up 13% lower) of selective neural network models compared to the state-of- the-art (Section 5.2). We also show that our method is better than alternative designs using vanilla techniques from existing curriculum learning literature (Section (53)
RELATED WORK
Selective classification. Prior work selective classification primarily focuses on adding the reject
option to classical learning algorithms such as SVM (Fumera & Roli 2002). nearest neighbors (Hell-
man 1970). boosting (Cortes al. (016) and online learning methods (Cortes et al.] 2018). In
particular, Geiffman & EI- Yaniv (017)) applies selective classification techniques in the context of deep neural networks (DNNs). They show how to construct a selective classifier given trained neural network model. They decide whether or not to reject each example based on a confidence score. They rely on two techniques for extracting confidence scores from neural network model: Softmax Response (SR), and Monte-Carlo dropout (MC-dropout). SR is the maximal activation in the softmax layer for a classification model, which used as the confidence score. MC-dropout es- timates the confidence score based on the statistics of numerous forward passes through the network with the dropout applied. Unfortunately, MC-dropout requires hundreds of forward passes for each example, incurring massive computational overhead. More recently, Geifman & El- Ylniv (2019) proposes a selective neural network that jointly learns a predictive function and a selection function. This model is trained end-to-end, resulting in selective model that is optimized over the covered domain. They show empirically that this selective neural network outperforms previous methods based SR or MC-dropout. In addition, inspired by portfolio theory, Liu et al. (2019) propose
2

---

Under review as a conference paper at ICLR 2023
new loss function for selective classification based the doubling rate of gambling. Self-adaptive training (Huang et al., 2020) that calibrates training process by model predictions can also be ap- plied to selective classification. In Gangrade et al. (202T), the selective classification is formulated into optimising a collection of class-wise decoupled one-sided empirical risks. However, none of the prior work in selective classification has exploited estimated difficulty scores in curriculum learning, which is what we propose for the first time in this paper.
Curriculum learning. Inspired by how human students are gradually introduced concepts with increasing complexity, curriculum learning techniques train neural networks by feeding easier ex- amples earlier in the training process and gradually transitioning to more difficult examples. There is huge surge of interests in applying curriculum learning to different domains in recent years, e.g., image classification (Guo et al. 2018), machine translation (Platanios et al., 2019), language model pre-training (Li et al. 2021), healthcare prediction (El-Bouri et al. (020) and graph learn- ing (Gong et al.] 2019). Despite the differences among domains, these techniques follow a general pattern where a sequence of training criteria are presented to the model over T training steps and the training criteria affect how examples are selected at each step (Wang et al., 2021; Sooviayy et al, 2022). Typically, example selection works in the following way: the mini-batch at each iteration is uniformly sampled from a working subset, which starts with a few easiest examples, and is being appended with harder and harder examples after every several epochs until all the training examples are added. Our purriculum-inspied method is also based on the above example selection strategy. Note that anti-curriculum method uses the opposite strategy by starting from the hardest examples and transitioning to easier examples. We do not consider the anti- anti-curricllum strategy because it has been shown have worse performance than normal curriculum strategy in recent studies (Wu al., 2020 Hacohen & Weinshall, 2019). Similarly, another type of strategies for example selection that different from curriculum learning example weighting (Shrivastava al. [0166] Katharopoulos & Fleuret 2018). However, some studies (Wang et al.) 2021 Soviany al. 2022) have shown that example weighting methods are more prone to noise and outliers the datasets, because they tend to select the hardest examples which might actually be noise and outliers. In this paper, we focus on
curriculum learning due to its suitability for handling complex real-world scenarios where selective classification is usually applied.
BACKGROUND
In this section we describe the problem formulation for selective classification, the selective neural network model and a baseline method for training it.
3.1 SELECTIVE CLASSIFICATION PROBLEM FORMULATION
Here, we consider the multi-class classification setting. Let X' be the input feature space, 3' the label space and P(X ))) a distribution over X' x y: A predictive model is function f X + Y. Given a loss function 1 : y x y > R+, the risk of the model f is Ep(x,,, (Px [z) v)]. In the classical supervised learning. the goal is find the model f such that the model risk is minimized.
In this paper, we consider the selective model which is a pair of functions (f.9) where f: + y is the predictive model, and g X (0.1) } is selection function. The selection function is used to decide whether each example r is accepted or not:
f(x),
if g()) = 1; ABSTAIN, if g(z) = 0.
(f.)(z)= =
(1)
Thus, the selective model rejects (i.e., abstains from) an example iff g(x) = 0. Note that soft selection function 9 : X'' [0, 1] can also be considered, from which decisions are taken probabilisticaly or deterministically using threshold. The selective risk of (f,g) is
Ep (2)((())))(()) Ep [g(x)]
R(f.g)
The coverage ((g) is defined as Ep [g(x)), i.e., the ratio of the number of accepted examples among all examples. Clearly, there is a trade-off between risk and coverage. In other words, a lower risk could be achieved by sacrificing the coverage. The entire performance of selective model can be
3

---

Under review as a conference paper at ICLR 2023
measured via its risk- curve (El-Yaniv et al. 2010). As with Geifman & El- Yaniv (2019)
we adopt the goal of finding the model (f.g) g) such as its selective risk is minimized given à target coverage
Given a labeled set S {(x, yt) Jm there are empirical counterparts for the selective risk and coverage. The empirical selective risk is
1 E ((f(
F(Sg|S)>
(3)
((g|S)
and the empirical coverage is
m
((g|S) Σ g(.)).
(4)
m
3.2 SELECTIVE NEURAL NETWORK MODEL
In this paper, we foeus the neural network model SELECTIVENET from Geifman & El- Yaniv (2019) because of its impressive empirical performance. SELECTIVENET optimizes both f and in single neural network model. The main body block can be any type of architecture that is typically used for solving the problem at hand. For example, convolutional neural networks or residual neural networks are typically used for image classification (Krizhevsky et al. 2012 He al. 2016), and transformer networks are typically used for natural language understanding (Devlin et al. (018)) SELECTIVENET has three output heads for prediction f. selection g and auxiliary prediction h, respectively. The prediction head f and selection head g are the two functions previously defined defined in Section 3.1 The auxiliary head h handles a prediction task with the goal of exposing the main body block to all training examples throughout the training process. As with Geifman & El-yaniv (201)), the auxiliary head h uses the same prediction task assigned to f in this paper. We only need h during training by adding its loss function (defined later in Equation ( the main selective classification loss (Equation 5). Given a training set S = {xi, Mi ) and a target coverage the selective training objective for the heads f and g
(( = f(f.g) + À max "(g\و))²
(5)
where À is hyper-parameter. The standard loss function is used for auxiliary head h:
Cn 1
(1(.)).).).
(6)
Thus, the overall training objective is
C C= + a))
where a is hyper- pprem-paramtter controlling the relative importance of the above two losses. A stochas- tic gradient descent (SGD))type optimization method is used to minimize the above objective (Geif-] man & EI- Yaniv 2019), which is an iterative process. At each iteration, a mini-batch of examples are picked from the entire training set, which are used to compute the gradient to update the model parameters. We treat this training method as our baseline, and propose a new method inspired by curriculum learning in the next section.
is worth noting that, regardless the value of the target coverage c, all examples participate and are accounted for in minimizing our training objective (Equation 7). This is because the selection head g (used to decide whether an example should be accepted or not) is trained over all examples. Furthermore, the auxiliary head is optimized over all examples it has shown to improve the main body block training (Geifman & EI- Yaniy 2019). Once the training is completed, the auxiliary head is no longer needed. We keep the prediction head f and selection head g together with the main body block, and use them for selective classification according to Equation
4 CURRICULUM-INSPRED TRAINING METHOD
Inspired by curriculum learning, we propose a new method for training the selective neural network model. Our method is presented in Algorithm |
The key idea here is to (1) construct mini-batches
4

---

Under review as a conference paper at ICLR 2023
by calibrating the ratio of easy and hard examples, and (2) exploit the difficulty ordering at the mini- batch level, i.c., control the order in which the mini-batches are exposed by gradually narrowing the difficulty gap between easy and hard examples within each Sections 4.1 and 4.2 respectively. In Section 4.3 we discuss our approach to estimate the example difficulty scores.
These steps are discussed in
4.1 MINH-BATCH CONSTRUCTION
Since we using stochas- tic gradient descent (SGD) optimization method to train the selective neural network model, need to construct mini-
Algorithm Curriculum-inspired method for training selective neural network.
1: Input: training set S, target coverage c, mini-batch size b,
pacing function p and scoring function q
batch of examples to be exposed each iteration. We can show that, when the mini-batch has the proportion of easy examples different from the target cov- erage, as a result of minimiz- ing the loss during the training,
2: Output: sequence of mini- mini-batches [B1, B2....]
3: Sort S according to q in ascending order //easier examples are before the harder examples
4: result + -
5: for i +- to T do /IT is the total number of iterations
6:
size + -p(i;;
7:
sizeg [p(i) ccl; //the size of easy active set Sg sizeH t size sizeg; //the size of hard active set S H
even an already perfect selection function tends to move to the di- rection where mistakes will be made (i.e., reject easy exam- ples or accept hard examples), Therefore, mini-batch which has the proportion of easy ex- amples different from the target coverage will misguide the se- lection function regarding which examples should be accepted or rejected, which further affects the prediction head's learning progress. Fortunately, we can leverage the estimated difficulty scores to alleviate this issue. Specif- ically, we partition all the training examples into an easy set and hard set according their timated scores. Assume the mini-batch size is b, the total number of iterations is T and the total number of training examples m. The easy set contains m c examples with the lowest difficulty scores, whereas the hard set contains the rest of the examples. We construct the mini-batch by se- lecting [ x c] examples from the easy set, called Bg, and b [b * cl examples from the hard set, called Bи, as shown in Lines 11 and 12 of Algorithm [ The union of Bg and Bи becomes the mini- B, for the i-th iteration (A (Algorithm) Line 13). Here, we use the [-] operator because bx may not be an integer. By calibrating the ratio of easy and hard examples in each mini-batch in the above manner, we can ensure that the proportion of easy examples roughly c.
8:
9:
Se S/: sizeg); //select the first sizeg examples
from the ordered training set S
10:
Sh Si-size :): //select the last sizeh examples from ordered training set S)
11:
Bg + uniformly sample [b cl examples from Sg:
12:
Ви + uniformly sample b - [ * c] examples from Sw;
13:
B, = B BBr //form the mini-batch for the i-th
iteration
14: append B, to result;
15:
end for
16: Return: result
4.2 MINI-BATCH ORDERING
Curriculum learning techniques suggest exposing easy examples to the model early on in the training process, and gradually transitioning to hard examples. We still adopt the curriculum idea, but we ar- gue that in selective neural network context, currieulum should be designed at the batch level. We cannot expose examples exactly the same way that curriculum learning does because mini-batch should include both easy and hard examples with their ratio appropriately calibrated to meet the cov- erage constraint, as explained earlier in subsection 4.1 We therefore need to generalize curriculum learning to impose the ordering at the mini batch level, i.e., by exposing easy mini-batches to the model early in the training process. The key intuition is follow. We consider a mini-batch easy for the model (especially its selection head) if its constituent easy examples are clearly separate from its constituent hard examples in terms of their difficulty scores. Similarly, we consider a mini-batch hard if its constituent easy examples are close to its constituent hard examples in terms of
Detailed explanations included in Section A.2 the appendix.
5

---

Under review as a conference paper at ICLR 2023
their difficulty scores. We propose expose easy mini-batches to the model early in the training and gradually transition to hard mini- batche. This enables us to gradually narrow down the difficulty gap between easy and hard examples. Similar to other curriculum learning techniques (Hacohen & Weinshall 2019), we also need to provide two functions as the input to our curriculum-inspree. method: a pacing function p and scoring function q. pacing function p is used to control the curriculum pace, i... the speed at which we transition from casy to hard mini-batches. A scoring function q used indicate how hard each example is (i.e., estimated difficulty score), which will be explained in the next subsection.
Specifically, the pacing function p will output an integer p(i) for each iteration i, which we then use to decide how many examples to consider via uniform sampling when constructing the i-th mini- batch (Algorithm) T Line 6). In this paper, we use the fixed exponential pacing function from Haaoo hen & Weinshall (2099) because of its good performance in practice. At the i-th iteration, instead of picking the p(i) easiest examples from the training set, we choose [p(i) cl easiest examples as the easy active set, called Sg. and p(i) - [p(i)= cl hardest examples as the hard active set, called Sh (Algorithm Line 9 and Line 10). Then, we uniformly sample [b * cl examples from Sg, called Bg, and uniformly sample b b- [b cl examples from SH called Ви (Algorithm Line 11 and Line 12). The union Bg and Bи becomes the mini-batch B; for the i-th iteration (Algorithm Line
13). The entire process is also illustrated in Figure [
4.3 DIFFICULTY SCORE ESTIMATION
We use the following approach for getting the scoring function q. which outputs a scalar value ((x)
Prediction
for each example x as its esti- mated difficulty score. First, we train the same model using uni- formly sampled mini-batches (i.e., vanilla method), and run each ex- ample through it to get the activa-
Sonneettooa es Oretennd Oiteuuy Fecees
Sp
4r.0)
Sampling
BP
Mini- batch
Selection
Main Body
Bg
tion levels of its main body output its condensed representation. We then run an outlier detection algo- rithm, Minimum Covariance Deter- minant (MCD) [Roussceuw] 1984). to compute the Mahalanobis distance for each example with respect to all examples that belong to the same class with this example. The idea be- hind MCD is to find a subset exam- ples whose empirical covariance has the smallest determinant, hence re- sulting in the tightest representation of the overall dataset, which may oth- erwise be quite noisy and have out- liers. This subset of examples is then used to compute the center location and covariance Σ. For any example its Mahalanobis distance is computed as d(x) (x = (x - p.). The larger the Mahalanobis distance d(z). , the noisier the example x. We thus treat d(x:) as the scoring function output q(x:) for example x, i.e., ((x)) d(x:), Va.
Sampling
Sg
Auxiliary
L2
Figure 1: Diagram of our curriculum-inspired method for training the selective neural network model. The training examples
ordered by their difficulty scores shown in the left-most box. Hard examples are at the top (red color) and easy examples at the bottom (green color). At each iteration, the mini-batch is constructed by sampling from both the hard part and the easy part, with their ratio appro- priately calibrated. As training process goes, we gradually narrow the difficulty gap between easy and hard examples, indicated by the boundaries of the easy and hard parts mov- ing towards each other in the diagram.
In practice, the overhead of this new approach for difficulty score estimation should not become a concern. Firstly, the model using uniform sampling (i.e., without rejection) needs much less itera- tions to get an estimation of difficulty scores than the model with rejection. In all of our experiments, use half of the iterations compared to training the model with rejection. Secondly, we run the MCD separately for the examples belonging to each class. The number of examples in each class is typically small enough for rumning MCD (e.g., 5k for CIFARIO, 7k for SVHN and 500 for CI- FAR1100). Both factors limit the overhead of our approach. What's 's more, the overhead of getting
difficulty score estimation can be greatly reduced by simply using an exiting approach based on

---

Under review as a conference paper at ICLR 2023
Table 1: Results of moderate-sized CNN model CIFAR10. Testing classification rates (in %) are shown for different target coverage rates. The relative improvement of error rates of our method w.r.t. SelectiveNet shown the last column.
Error Rate
Coverage
Impr.
Our Method
SelectiveNet
0.95
7.32 = 0.05
7.57 + 0.1
3.30%
0.9
5.63
0.04
6.07 + 0.1
7.25%
0.85
4.30 = 0.04
4.53 0.08
5.08%
0.8
3.02
0.02
12.97%
0.75
2.16 = 0.04
2,45 +.07
11.84%
0.7
1.47 = 0.02
1.67 + 0.08
0.08
11.98%
Table 2: Results of moderate-sized CNN model CIFAR100. Testing classification error rates
(in %) are shown for different target coverage rates. The relative improvement is shown at the last column.
Error Rate
Coverage
Impr.
Our Method
SelectiveNet
0.95
30.63 + 0.06
31.13 + 0.22
1.61%
0.9
28.95 + 0.08
29.05 + 0.14
0.35%
0.85
27.07 0.08
27.32 + 0.09
0.92%
0.8
25.00 : 0.05
25.39 + 0.07
1.54%
0.75
23.16 0.03
23.67
0.05
+ 0.05
2.15%
0.7
21.28 + 0.02
21.97 + 0.03
3.14%
off-the-shelf pre-trained models (e.g., Inception in our ablation study in Section 5.3 Equipped with either approach for getting difficulty score estimation, our method can outperform the baseline. However, our new difficulty estimation approach Icads to better empirical performance. We think this is because MCD offers the robustness needed for handling complex datasets (which might be noisy with outliers).
EXPERIMENTS
In this section, evaluate our proposed method and compare it against a state-of-the-art method for training selective neural network models. Similar to Geifman & EI-Yaniy (2019), we also report both the testing classification eиrоr rates (which is the same selective risk values when using 0-1 loss) as well as the relative improvements of the етгоr rates. In summary, our experiments show the following Fl (1) our method achieves lower error rate (up 10% lower) than the state-of-the-an for the same target coverage rate for a variety of different main body network architectures and datasets (section 5.2pp: (2) our method is better than other alternative designs using difficulty scores or example selection from existing curriculum learning literature (section 5.3
5.1 SETUP
Datasets. Similar to prior work, we use the following three datasets: CIFAR10 (Krizhevsky et al.,
2009). CIFARI00 (Krizhevsky et al. 2009) and SVHN (Netzer al. 2011 in our experiments.
Baseline method and architectures. We compare our method against SELECTIVENET (Geifman & El- Yaniv 2019), which considered the -tate-of-the-art. We vary the target coverage rate e with 6 different values: 0.95, 0.9, 0.85, 0.8, 0. 0.75, 0.7. We use two different neural network architectures for the main body of our selective neural network model: moderate-sized convolutional network ar- chitecture (Hacohen & Weinshall, 2019) and VGG-16 16 architecture (Simonyan & Zisserman) 2014) Details about these architectures can be found the appendix.
In the appendix, we also show that (i) our method has a faster convergence speed (up to 50% faster) than the state-of-the-art (section A.5), and (ii) our method can reduce the coverage violation (i.c., the difference hetween target coverage and empirical coverage) by more than 10% (section A.6

---

Under review as a conference paper at ICLR 2023
Table 3: Results of moderate-sized CNN model on SVHN. Testing classification error rates (in %)
are shown for different targel coverage rates. The relative improvement is shown at the last column.
Error Rate
Coverage
Impr.
Our Method
SelectiveNet
0.95
1.91 = 0.03
2.08 + 0.02
8.17%
0.9
1.03 = 0.04
1.12 0.06
8.04%
0.85
0.63 0.02
0.66
4.55%
0.8
0.49 = 0.03
0.02
3.92%
0.75
0.41 0.01
0.42
2.38%
0.7
0.35 = 0.01
0.39 =
10.26%
Table 4: Results of VGG model on CIFAR100. Testing classification error rates (in %) are shown for different target coverage rates. The relative improvement is shown the last column.
Error Rate
Coverage
Impr.
Our Method
SelectiveNet
0.95
28.41 + 0.07
28.92 + 0..09
1.76%
0.9
25.67 0.04
25.78 + 0.02
0.43%
0.85
22.94
23.56 + 0.08
2.63%
0.8
21.40
21.99 + 0.02
2.68%
0.75
18.25 + 0.01
18.91 + 0.08
3.49%
0.7
16.09 0.02
16.32 + 0.01
1.41%
Training details. We follow the settings from Geifman El-Yaniv (2019) whenever possible. We train each model for 300 epochs, with an initial learning rate of 0.1 which is reduced by half every 25 epochs. The model optimized using stochastic gradient descent (SGD) with momentum 0.9 and weight decay 5e 4. All experiments are repeated 3 times similar to SELECTIVENET. We report both the average value and standard deviation (following +). The classification error rate that we report in our experiments is the percentage of incorrectly predicted examples among all the accepted examples, by evaluating the selective model on the hold-out test sets. Note that classification error rate is the same as empirical selective risk value when using the 0-1 loss. All experiments are run using NVIDIA V100 GPUs.
Post-training calibration. We report the calibrated selective error rate as the metric for comparing against the bascline for each target coverage value. To get the calibrated selective error, we estimate an appropriate threshold for selection head output values validation set, and use the following calibrated rule for our predictions:
f(x). ABSTAIN, otherwise
if g<))
( (z.g)()) =
This calibration process is used to correct for the difference between the post-training empirical coverage è and the target coverage Specifically, given validation set V, to be the 100(1 - e) percentile of the distribution of g(z;), E V.
5.2 SELECTIVE CLASSIFICATION RESULTS
Moderate-sized CNN. As with Hacohen & Weinshall (2019), we conduct experiments using moderate-sized convolutional neural networks (CNN) on all three datasets CIFAR10, CIFAR100 and SVHN. The corresponding results are shown in Table [ Table 2 and Table 3 It can be seen that, the target coverage rate becomes smaller, the error rate is lower for both our method and the baseline. This behavior is expected because, in general, any selective model should be able to trade-off coverage for error rate, i.e., lower the error by reducing coverage. However, achieving lower error becomes more challenging when the target coverage smaller because the model needs to figure out which additional examples should be further rejected.
Overall, our method outperforms the baseline by achieving lower selective enror rates for each target coverage rate (corresponding each in the above mentioned tables). Our method consistently
8

---

Under review as a conference paper at ICLR 2023
Table 5: Ahlation study with moderate-sized CNN on CIFARIO. Testing classification error rates (in %) are shown below.
Coverage
Our Method
SelectiveNet
Inception
Vanilla-Sclection
0.9
5.63 + 0.04
6.07 + 0.1
5.87 + 0.05
5.98 + 0.04
0.8
3.02 + 0.02
3.47 + 0.07
3.14 + 0.05
3.35 + 0.04
0.7
1.47 + 0.02
1.67 + 0.08
1.59 + 0.02
1.66 + 0.03
improves upon the baseline across all three datasets. These improvements FAR10 and SVHN (up to 10%), especially when the target coverage is small. This demonstrates that our method is very effective at trading off the coverage for even lower error rate.
even higher for CI-
VGG. We also conduct experiments using the VGG-16 model (Simonyan & Zisserman, 2014) on the CIFARIO0 dataset, because of VGG-16's strong prediction ability which is desired for the CI- FARI00 dataset. VGG is larger image recognition model with more layers. As shown in Table our method still outperform the baseline SELECTIVENET consistently across all different target cov- erage rates. This verifies that our method can be applied to different backbone models. It is worth noting that, compared with the moderate-sized CNN on CIFAR 100 shown in Table 2 the error rate for either method is smaller smaller in Table 5 for the same coverage rate. This is expected because VGG model larger in size (more layers). addition, the average improvement from Table 2 is 1.61% while the average improvement from Table 5 is 2.07%. This implies that our method can achieve an even greater improvement with larger models.
5.3 ABLATION STUDY
We conduct ablation study validate our design decisions by comparing method against two variations of it. Note that our method is composed of two components: (1) estimating the difficulty scores, and (2) selecting examples for each mini-batch during training. To show that both compo- nents play an important role in the overall success of our method, we implement two variants, each using one component from existing work. Inception a variant of our method using the pre-trained inception model for difficulty estimation (i.e., transfer scoring function in Hacohen & Weinshall 2019). Vanilla-Selection another variant of our method using the vanilla method of selecting examples for each mini-batch that is typically used in existing curriculum literature. Different from our method which considers ordering at the mini-batch level, existing curriculum learning litera- ture typically consider ordering at the example level while selecting examples for each mini-batch. Specifically, they construct the mini-batch by uniformly sampling from a working subset, which starts with few casiest examples, and is being appended with harder and harder examples after ev- ery several epochs until all the training examples are added (Wang et al. 2021 (Soviany et al.) 2022). We adopt this strategy in Vanilla-Selection which can be viewed as applying the vanilla curriculum learning (e.g., Algorithm I from Hacohen & Weinshall (2019) to selective classification domain.
As reported in Table 5 both variants have higher error rates than method, validating that both components are integral to and contribute to our method's impressive performance. Especially, ap- plying the vanilla curriculum learning the selective classification domain as in Vanilla-Selection is not as good as our method which has a specially tailored curriculum learning strategy. In addi- tion, even though our method involves an overhead of model training order get the difficulty score estimation, this overhead is paid only once even if the coverage needs to change. Inception the variant without this overhead since the thellff pre-trained inception model can be readily obtained online. If one does not wish to pay the upfront overhead of our method, these results imply that even Inception outperforms the baseline.
CONCLUSION
In this paper, we proposed a curriculum-inspired method to train selective neural network mod- els by leveraging difficulty scores. Our method calibrates the ratio of easy and hard examples in each mini-batch to match what is best desired by the training objective. We generalize curriculum learning by utilizing difficulty scores to reorder mini-batches rather than simply reorder examples. Extensive experiments show the superior performance of our method compared to state-of offte-of-the-nn SELECTIVENET.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. Advances in Neural Information Processing Systems, 29:1660-1668, 2016.
Corinna Cortes, Giulia DeSalvo, Claudio Gentile, Mehryar Mohri, and Scott Yang. Online learning with abstention. In international conference on machine learning. pp. 1059-1067. PMLR, 2018.
Jacob Devlin, Ming- bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Wei Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
Rasheed El-Bouri, David Eyre, Peter Watkinson, Tingting Zhu, and David Clifton. Student-teacher curriculum learning via reinforcement learning: predicting hospital inpatient admission location. In International Conference on Machine Learning, pp. 2848-2857. PMLR, 2020.
Ran EI- Yaniv al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
Giorgio Fumera and Fabio Roli. Support vector machines with embedded reject option. In Interna- tional Workshop on Support Vector Machines, pp. 68-82. Springer, 2002.
Aditya Gangrade, Anil Kag, and Venkatesh Saligrama. Selective classification via one-sided predic- tion. In International Conference on Artificial Intelligence and Statistics, pp. 2179-2187. PMLR, 2021.
Yonatan Geifman and Ran El- Yaniv. Selective classification for deep neural networks. arXiv preprint 17Xiv:/70.088000 2017.
Yonatan Geifman and Ran El-Yania Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International Conference Machine Learning. pp. 2151-2159. PMLR, 2019.
Chen Gong, Jian Yang, and Dacheng Tao. Multi-modal curriculum learning over graphs. ACM Transactions Intelligent Systems and Technology (TIST), 10(4):1-25, 2019.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European conference on computer vision (ECCV), pp. 135-150, 2018
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep net- works. In International Conference on Machine Learning, pp. 2535-2544 PMLR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp- 770-778, 2016
Martin E Hellman. The nearest neighbor classification rule with reject option. IEEE Transactions on Systems Science and Cybernetics, 6(3):179-188) 1970.
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. Advances in neural information processing systems, 33:19365-19376 2020.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Exploring the memorization- generalization continuum in deep learning. 2020.
Angelos Katharopoulos and François Fleuret. Not all samples created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525-2534. PMLR, 2018.
Benjamin Kompa, Jasper Snoek, and Andrew I. Beam. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digital Medicine, 4(1):166, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Conglong Li, Minjia Zhang, and Yuxiong He. Curriculum learning: A regularization method for efficient and stable billion-scale gpt model pre-training. arXiv preprini arXiv:2108. 2021.
Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ R Salakhutdinov, Louis-Philippe Morency, and Masahito Ueda. Deep gamblers: Learning to abstain with portfolio theory. Advances Neural Information Processing Systems, 32: 12:10623-1000 10003,,,0,,,, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom M Mitchell. Competence- curriculum learning for neural machine translation. arXiv preprini 1rX3v:/903.09844 2019.
Peter Rousseeuw. Least median of squares regression. Journal of the American statistical associ- ation, 79(388):871-880, 1984.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- object detectors with online hard example mining. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 761-769, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, pp. 1-40, 2022.
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2021.
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? arXiv:2012.03107. 2020.
arXiv preprint
A APPENDIX
A.1 SCENARIOS WHERE SELECTIVE CLASSIFICATION IS NEEDED
In nutshell, selective classification is needed in settings where prediction error of the model can cause critical consequences (e.g.. in medical diagnosis or self-driving cars). In these settings, we would like the model to proactively indicate when it's not confident its prediction, so that human intervention is triggered.
Medical diagnosis. In medical diagnosis, a patient has tumor based on the MRI image. When the model is not confident enough, we can ask an oncologist, and even call upon an urgent meeting of multiple oncologists across different hospitals for a group consultation (Kompa et al. 2021).
train machine learning model to predict if
Self-driving cars. In self-driving cars, we train machine learning model detect the traffic light, which will help the self-driving car to decide what to do at cross-roads. However, when the model not confident enough (e.g., traffic light is not clear due rain or obstruction), the human driver should be alerted and take control of the car.
A.2 DETAILED EXPLANATIONS ABOUT TARGET COVERAGE MATCH
We elaborate on why it's ideal to match the proportion of easy examples each mini-batch with the target coverage as briefly mentioned in Section4.1 We start by showing that the empirical coverage e will converge to the target coverage c. Then show that mini-hatch in which the proportion
11

---

Under review as a conference paper at ICLR 2023
of easy examples is the same the target coverage is ideal. Therefore, mini-batch which has the proportion of easy examples different from the target coverage will misguide the selection function regarding which examples should be accepted or rejected, which further affects the prediction head's learning progress.
Empirical coverage converges to target coverage. As result of minimizing Equation 5, the empirical coverage I will converge to the target coverage c. Without being too technically rigorous, demonstrate this by showing that, whether not the empirical coverage ò is larger than target coverage c, the loss (()))) (i.e., left hand side of Equation will become smaller if empirical coverage is moved closer to target coverage.
When the empirical coverage is larger than the target coverage, the second term of Equa- [ion[5] becomes O. Note that the first term f is the average loss of accepted examples. By simply rejecting the example with the highest loss (i.e., decreasing the empirical coverage), the first term f will become smaller while keeping the second term still O. Therefore, when the empirical coverage is larger than the target coverage, decreasing the empirical coverage will make the loss ((r smaller.
When the empirical coverage is smaller than the target coverage, the second term is positive value. As long as the hyperparameter À is large enough to make the second term dominant, the loss L((.g) will become smaller by increasing the empirical coverage. As suggested in Geifman & Ya-iv (2019), the hyperparameter À chosen to be a large value 32 in our experiments. Therefore, when the empirical coverage is smaller than the target coverage, increasing the empirical coverage will make the loss L(f.)) smaller.
Mini-batch should ideally match target coverage. We now show that a mini-batch in which the proportion of casy examples is the same as the target coverage is ideal. We demonstrate it by showing that, when the mini-batch has the proportion of easy examples different from the target coverage, as result of minimizing the loss during the training, even an already perfect selection function tends to move to the direction where mistakes will be made (i.e., reject easy examples or accept hard examples). Note that, by an already perfect selection function, we mean that the selection function accepts or rejects examples based on the ground truth (i.e., reject hard examples or accept easy examples).
When the proportion of easy examples in the mini-batch is larger than the target coverage, a perfect selection function gives the empirical coverage on this mini-batch that is larger than target coverage. As we have just discussed above, when the empirical coverage is larger than the target coverage, decreasing the empirical coverage will make the loss L(t.)) smaller. It implies that the selection function tends to reject more examples, which actually accepted by a perfect selection function.
When the proportion of easy examples in the mini-batch is smaller than the target coverage, perfect selection function gives the empirical coverage this mini-batch that is smaller than target coverage. As we have just discussed above, when the empirical coverage is smaller than the target coverage, increasing the empirical coverage will make the loss L(f.)) smaller. It implies that the selection function tends to accept more examples, which are actually rejected by a perfect selection function.
A.3 MORE DETAILS ABOUT EXPERIMENTAL SETUP
The miderate-sized convolutional network architecture contains 8 convolutional layers with 32,32, 12, 64, 64, 128, 128, 256, 256 filters respectively. The filter size is 3 x 3 for the first 6 layers max-pooling layer and dropout layer with rate
and 2 x for the last layers. There is a 0.25 after every two layers. Following the convolutional layers, there is a fully-connected layer with 512 units and a dropout layer with rate 0.25. At last, another fully connected layer used to match the number of classes in the dataset. We omit the detailed layer-by-layer description of VGG-16 as it is described in Simonyan Zisserman (0014). In addition, the prediction head f and auxiliary head h are fully-connected softmax layers, and the selection head g is culle-coneected hidden layer with 513 neurons, followed by batch normalization, ReLU activation and another fully connected layer to one output neuron with sigmoid activation. These architectures have been used in the
12

---

Under review as a conference paper at ICLR 2023
Table 6: Ablation study with moderate-sized CNN CIFARIO. Testing error (in %) are shown below.
Coverage
Our Method
SelectiveNet
Inception
Vanilla-Selection
0.95
7.32 + 0.05
7.57 = 0.1
7.43 + 0.07
7.51 = 0.04
0.9
5.63 0.04
6.07
5.87 0.05
5.98 1 0.04
0.85
4.30
4.53
4.42
0.8
3.47 3.02 0.02
3.14=0.00
3.35
0.75
2.16 + 0.04
2.23 + 0.04
2.38 + 0.09
0.7
1.47 + 0.02
1.67 0.08
1.59 + 0.02
1.66 + 0.03
recent selective classification and eurriculum learning papers (Geifman & -El Yaniv 2019 Hacohen & Weinshall 2019 Wu et al. 2020).
A.4
MORE EXPERIMENTAL RESULTS
The complete results of ablation study are shown in Table O
SelectiveNet Method
SelectiveNet Our Method
0.4
0.4
Testing Loss
0.3
0.3
0.1
40000
60000
B0000
100000
120000
40000
60000
80000
100000
120000
Number Iterations
Number Iterations
(a) CNN on CIFARIO with c 0.9
(b) CNN on CIFARIO with c = 0.8
SelectiveNet
SelectiveNet Our Method
2.5
Our Method
Testing Lo55
2.0
Testing Loss
2.0
2.5
3,3
1.0
0.5
40000
60005
80000
100000
120000
40000
60000
50000
100000
120000
Number of Iterations
Number of Iterationn
(c) VGG CIFAR100 with = 0.9
(d) VGG on CIFAR100 with c 0.8
Figure 2: Testing loss vs. the number of iterations for different models on different datasets. Our method convergences faster by achieving a lower testing loss for the same number of iterations.
A.5 CONVERGENCE SPEED
It has been shown that curriculum learning can improve the convergence speed when training on various classical neural network models (Hacohen & Weinshall, 2019 Li al.] [02T). In this section, we investigate whether this applies to our method as well. We plot the curves of testing loss versus the number of iterations under different settings in Figure
Specifically, Figures 2 (a) and
(b) correspond to training moderate-sized CNN on the CIFAR 10 dataset with target coverage of c = 0.9 and e = 0.8, respectively, while Figures (c) and (d) correspond to training VGG-16 on the CIFAR 100 dataset with target coverage of 0.9 and 0.8, respectively. We can see that our method has a better convergence rate than the baseline because our method achieves a lower testing loss for the same number of training iterations than SELECTIVENET. To better understand how faster our method's converges, we also measure the number of iterations to reach the same testing loss summarized in Table Specifically, the testing loss reach is set the lowest testing loss
13

---

Under review as a conference paper at ICLR 2023
Table 7: The number of iterations to reach the target loss for training moderate-sized CNN the CIFARIO dataset. The target loss value is the lowest testing loss achieved by SELECTIVENET for each coverage rate.
of Iterations
Coverage
Impr.
Our Method
SelectiveNet
0.95
332k
694k
52%
0.9
73k
116k
37%
0.85
97k
459k
79%
0.8
74k
115k
36%
0.75
75k
109k
31%
Table 8: Coverage violation of moderate-sized CNN on CIFAR1O. Empirical Coverage (in %) are shown for different target coverage rates. Our method achieves smaller coverage violation (differ- ence between target coverage and empirical coverage) than SELECTIVENET.
Our Method
SelectiveNet
Coverage (In %)
Violation Reduction
Empirical Coverage
Violation
Empirical Coverage
Violation
90.2 0.1
90.0
5.0
84.7 0.1
5.9
10.2%
85
80.7 1 0,2
4.3
80.2 1.
4.8
10.4%
80
76.6 + 0.1
3.4
76.3 + 0.2
3.7
8,1%
75
73.1 + 0.1
1.9
72.7 + 0.3
2.3
17.4%
70
69.3 1 0.1
0.7
68.8 1 0.1
1.2
41.7%
achieved by SELECTIVENET. be seen that our method can reduce the the number of iterations by to 80%. This demonstrates that our method is significantly faster than the baseline during training.
A.6 COVERAGE VIOLATION
Our method leverages the difficulty scores to calibrate the ratio of the difficuult and easy examples in each mini-batch. This could effectively reduce the discrepancy between the target coverage and empirical coverage (i.e., the coverage the testing set before applying the calibration process as described in Equation 8). We measure the empirical coverage and how much it differs from the target coverage (i.e., violation) summarized in Table As expected, the empirical coverage achieved by our method is closer to the target coverage, resulting in smaller violation compared to the baseline (i.e.. SELECTIVENET). The relative reduction of the violation achieved by our method is larger than 10% for most values of target coverage (i.e., 90%, 70%). Especially, when the target coverage is small, i.c. 70%. the violation reduction is high as 41.7%.
14


Reviews:
[{"review_id": "b-cXbsdTU1w", "reviewer": "Reviewer_F3no", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper deals with NN models that are allowed to abstain from making a prediction whenever it is not sufficiently confident. These selective models are used to minimize risk of wrong prediction. Their model is based on SelectiveNet, which has 3 heads, one for minimizing the error for selected examples, one for deciding whether to select or reject examples and the last one for minimizing overall error rate. Given an order set of examples based on their difficulty scores, the curriculum-inspired training proposed by this work samples a mini-batch consisting of easy and hard examples to iteratively improve coverage and prediction accuracy of the NN model. They show improved performance of their technique over SelectiveNet. \n", "summary_of_the_review": "This work is clearly presented and well written. The appendix also covers more details on ablation studies and convergence rates. I feel that this work is not very novel but useful nonetheless. ", "strength_and_weaknesses": "Comments:\n1. First paper to leverage the difficulty scores obtained by selective classification in curriculum learning framework.\n2. The contribution of this work is (1) defining the coverage term in the loss function. (2) use of mini-batch learning for curriculum inspired training. \n3. The experiments demonstrate the effectiveness of their technique over SelectiveNet. \n4. Have you tried different sampling strategies, instead of uniformly distributing easy and difficult samples in a mini-batch? Something like importance sampling based on the difficulty scores?\n5. Any theoretical guarantees of using this type of curriculum-inspired training? \n6. Any experiments/datasets where this technique failed to improve over vanilla learning?\n7. Any observations on getting worse results for this way of obtaining mini-batches? Some theoretical guarantees on convergence will be perfect but even empirical evaluation will be good. \n\nThis work is clearly presented and well written. The appendix also covers more details on ablation studies and convergence rates. I feel that this work is incremental and not very novel but useful nonetheless. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JMq2L0cqYY9", "reviewer": "Reviewer_xFiA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces curriculum learning in training selective neural networks. \n- Selective Neural Network, also referred to as the abstaining classifier, have the option to reject an input, i.e., abstain from prediction. Typically, this abstention mechanism is learnt to be a measure of the difficulty of the input. \n- Curriculum Learning trains a standard neural network by sorting the inputs according to some difficulty / uncertainity measure and serving these data points in the order from easy to hard during training.\n\nThis work merges the above two ideas. Specifically, it trains a selective neural network using curriculum learning. Although one could leverage the vanilla curriculum learning strategy, the notion of rejection provides an opportunity to align the difficulty notion used in the curriulum learning and the selective neural networks. This work calibrates the input into easy and hard examples and serves each mini-batch to the abstaining network such that it can enforce a coverage constratint ( amount of examples that the network accepts for prediction ). Empirical evaluation shows that the proposed curriculum learning design for selective neural networks is better than vanilla techniques.\n", "summary_of_the_review": "This work is indeed novel in that in integrates curriculum learning while training selective neural networks. It does not simply follow vanilla curriculum strategy but creates a mini-batch that respects the coverage constraint. \n\nI had expected a bit more thoughts in the design choices such as scoring and pacing functions. Since these are very crucial to the success of proposed method. In addition, choice of selective net is also unclear. There are other abstaining classifiers such as DeepGamblers and NNTD with better performance. I would have expected a few ablations that justify the choice of selectivenet.", "strength_and_weaknesses": "\nStrengths:\n- Incorporating Curriculum learning in Selective Classification is indeed novel \n- Proposed carefully designed mini-batch design is indeed relevant to the coverage constraint being tackled in the Selective Classification\n\nWeaknesses:\n- Poor empirical performance (in many cases the improvement in error is in the range of 0.3-0.5 see Table~1, 2)\n- Lack of ablations that justify the design choices (selectivenet over other abstaining neural networks, pacing function, scoring function)\n- Does not address the computational overhead of the proposed method\n- It is unclear why this work chooses the proposed difficulty score estimator. It is by no means computational cheap as compared to simpler metrics such as entropy, softmax margin, etc. \n\nQuestions for Authors:\n\n- Choice of pacing function : For completeness, could you describe what the fixed exponential pacing function is? Did you try some other pacing functions? Since this problem ( CL + abstaining classifier ) is different than existing CL work, it is unclear why fixed exponential pacing function should be the de-fact choice in this scenario?\n\n- Choice of scoring function : Outlier detection algorithm ( Minimum Covariance Determinant (MCD) ) seems like an expensive way to measure the difficulty of examples. Is there a comparison on the training time split, i.e., how much time gets split between training the auxiliary classifier, how much goes in computing the MCD scores, how much in training the abstaining classifier? Have you tried other simpler scoring functions such as entropy, softmax margin, etc.?\n\n- How often is the MCD invoked? Are the scores refreshed every epoch in the training or only once in a while? \n\n- Could you elaborate the point regarding the use of inception network w.r.t. MCD?\n\n- Could you shed some lights as to why the gains are not impressive in Table 1 and 2?\n\n- Can you plot the hardness measure to see how well separated easy/hard examples are for various coverage levels?\n\n-  Is there a reason why SelectiveNet was used as the abstaining architecture? There are other superior networks than SelectiveNet, for instance, \n     - DeepGamblers : https://proceedings.neurips.cc/paper/2019/file/0c4b1eeb45c90b52bfb9d07943d855ab-Paper.pdf\n     - One-Sided Predictions : http://proceedings.mlr.press/v130/gangrade21a/gangrade21a.pdf\n     - Self-Adaptive Training : https://proceedings.neurips.cc/paper/2020/file/e0ab531ec312161511493b002f9be2ee-Paper.pdf\n     - NNTD : https://arxiv.org/pdf/2205.13532.pdf\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DrKnaLFBM2u", "reviewer": "Reviewer_QZ16", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a variation of SelectiveNet (Geifman & El-Yaniv, 2019) inspired by curriculum learning. The approach can be seen as a combination of selective classification and curriculum training. As illustrated in Figure 1, it aims to minimize a joint loss between the selective loss term constrained by the empirical coverage and standard loss. The authors provide several experimental results on benchmark datasets (CIFAR10, CIFAR100, SVHN) that their approach achieves lower error rates than the baseline (SelectiveNet). They also compare the performance with several variants of their methods to provide a rationale for their design choice.", "summary_of_the_review": "Overall, this paper should be improved by \n(1) providing a good rationale for their design choice (i.e., why do we need to combine the selective classification and curriculum training), and\n(2) providing additional experiments to improve the quality and novelty of the paper.  \n\n\n\n", "strength_and_weaknesses": "- Rationale: It is not clear why we need to combine the two approaches (selective classification and curriculum-inspired training). The authors need to clearly state a rationale for their choice (i.e., the combination of the two). It can be an argument that the two methods are complement each other, or extensive experimental evidence showing that the combination of the two outperform each one. \n\n- Ablation study: In addition to Inception (to provide a rationale for their difficulty score estimation) and Vanilla-Sekection (to provide a rationale for the mini-batch technique), another option can be added to provide a rationale for the selective classification technique in thier model: non-selective classification with the mini-batch technique.  \n\n- Hyperparameter choice: This approach uses several hyperparameters (e.g. coverage rates, alpha, lambda, etc.) that can impact the model performance. Therefore, several experiments are necessary to understand those. For example, all experimental results (Table1 - 4, Supp Table 6) show that the performance improves as it uses lower target coverage rates. Can you provide further experimental results to find what coverage rate is preferable for the model? Also, it would be good to see how the empirical coverage changes along with the target coverage in addition to the supplementary explanations.  \n\n- [Minor] Add missing references (see comment in Clarity - Minor 1)\n\n- [Minor] Make the main paper more self-inclusive (see comment in Clarify Minor 2)\n\n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6x5Kq8YpHQl", "reviewer": "Reviewer_oBun", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies curriculum learning for selective neural networks. These are models that are allowed to reject examples if the confidence is low. The method proposed by the authors relies on carefully selecting the easy vs hard examples to be included in each mini-batch. Experiments are reported on CIFAR10, CIFAR100 and SVHN, comparing the proposed method with vanilla curriculum and non-curriculum baselines.", "summary_of_the_review": "My main concerns are related to the low-level of novelty and the rather small improvements on low-resolution datasets.", "strength_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow.\n\n- The authors study an interesting topic.\n\nWeaknesses:\n- The level of novelty is limited in my opinion. The curriculum learning method is quite standard. The novelty consists in applying curriculum learning to selective nets, which does not represent a significant contribution.\n\n- The authors use the relative improvements in terms of error rate, which is deceiving. The relative gains in terms of accuracy would be very different. Nonetheless, the absolute gains seem rather small.\n\n- The authors use only small-scale and low-resolution datasets. It is thus unclear if the method would help on large-scale datasets where there are more examples per class available.\n\n- The authors should also relate and compare with curriculum methods not using example difficulty scores, e.g. [A].\n\n- In my opinion, the studied problem is related to the idea of dispatching test examples to different models based on image difficulty [B]. It would be nice for the authors to mention that hard examples could be processed by another (more confident) model instead of rejecting them.\n\n[A] Samarth Sinha, Animesh Garg, and Hugo Larochelle, “Curriculum by smoothing.” In Neural Information Processing Systems, pp. 21653-21664, 2020. \n\n[B] Petru Soviany, and Radu Tudor Ionescu. \"Optimizing the trade-off between single-stage and two-stage deep object detectors using image difficulty prediction.\" In International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, pp. 209-214, 2018.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5zOgtAo88cI", "reviewer": "Reviewer_bj8f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work focuses on the training of selective networks, where the network knows then to reject to answer for unconfident examples. They propose a curriculum-inspired method to train selective neural works, where they (1) consider the target converge ratio when sampling batches, and (2) use the example difficulty score to to design the curriculum. They conduct experiments on CIFAR-10/100 and SVHN to demonstrate the usefulness of their method.\n", "summary_of_the_review": "The novelty of this work is not significant as the main ideas in this work are straightforward. In this case, I would like to look for rigorous empirical evaluation. At this point, I feel a ton of experiments need to be added to fully justify the usefulness of the proposed. The current performance improvement is marginal.", "strength_and_weaknesses": "The proposed idea is straightforward and makes sense to me. But I feel the empirical evaluation is not rigorous enough to validate the usefulness of the proposed method. Here are my concerns:\n\n(1) The improvement over the baseline SelectiveNet is quite marginal, e.g., ~0.5% error rate reduction on CIFAR-100 in Table 2.\n\n(2) There are no results on ImageNet. It is strongly encouraged to show ImageNet results for image classification. \n\n(3) There lacks diversity in network architectures. This work only shows results on a small CNN and VGG-16, both of which are probably far from the SOTA network architectures in computer vision, e.g., EfficientNet, FBNet, etc.\n\n(4) An existing work (“Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models” ICLR 2022) shows that a couple of simple of metrics (e.g., maximum confidence, entropy, etc) can be used to make rejection decisions as well (although they use it in the context of model cascades). Due to the simplicity of those metrics, a comparison against them is necessary and we should expect an obvious improvement to justify the usefulness of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject