Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a variation of SelectiveNet (Geifman & El-Yaniv, 2019) inspired by curriculum learning. The approach can be seen as a combination of selective classification and curriculum training. As illustrated in Figure 1, it aims to minimize a joint loss between the selective loss term constrained by the empirical coverage and standard loss. The authors provide several experimental results on benchmark datasets (CIFAR10, CIFAR100, SVHN) that their approach achieves lower error rates than the baseline (SelectiveNet). They also compare the performance with several variants of their methods to provide a rationale for their design choice.
strength_and_weaknesses: - Rationale: It is not clear why we need to combine the two approaches (selective classification and curriculum-inspired training). The authors need to clearly state a rationale for their choice (i.e., the combination of the two). It can be an argument that the two methods are complement each other, or extensive experimental evidence showing that the combination of the two outperform each one. 

- Ablation study: In addition to Inception (to provide a rationale for their difficulty score estimation) and Vanilla-Sekection (to provide a rationale for the mini-batch technique), another option can be added to provide a rationale for the selective classification technique in thier model: non-selective classification with the mini-batch technique.  

- Hyperparameter choice: This approach uses several hyperparameters (e.g. coverage rates, alpha, lambda, etc.) that can impact the model performance. Therefore, several experiments are necessary to understand those. For example, all experimental results (Table1 - 4, Supp Table 6) show that the performance improves as it uses lower target coverage rates. Can you provide further experimental results to find what coverage rate is preferable for the model? Also, it would be good to see how the empirical coverage changes along with the target coverage in addition to the supplementary explanations.  

- [Minor] Add missing references (see comment in Clarity - Minor 1)

- [Minor] Make the main paper more self-inclusive (see comment in Clarify Minor 2)


 
clarity,_quality,_novelty_and_reproducibility: ### Novelty
The novelty comes from the idea that combines the selective classification problem and curriculum-inspired training. However, I evaluate it as a marginal novelty as it is not convincing enough why that combination is needed (as mentioned in Weakness - Rationale).

### Clarity
There were no big obstacles to understanding its main idea and experimental designs. 

Minor)
1. There are missing references for the sentence "In many real life scenarios such as medical diagnosis, robotics and self driving cars ...".
2. If several tables (Table 1-4) are combined into one or two, the other interesting results can be moved from the supplementary to the main paper.

### Quality
The quality of the paper needs to be improved by providing enough experimental evidence to support the approach and to understand the model's behavior (as mentioned in Weakness - Hyperparameter choice and Ablation study). 

### Reproducibility
This paper provides source codes for reproduction and uses publicly available datasets in their experiments. 
summary_of_the_review: Overall, this paper should be improved by 
(1) providing a good rationale for their design choice (i.e., why do we need to combine the selective classification and curriculum training), and
(2) providing additional experiments to improve the quality and novelty of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work introduces curriculum learning in training selective neural networks. 
- Selective Neural Network, also referred to as the abstaining classifier, have the option to reject an input, i.e., abstain from prediction. Typically, this abstention mechanism is learnt to be a measure of the difficulty of the input. 
- Curriculum Learning trains a standard neural network by sorting the inputs according to some difficulty / uncertainity measure and serving these data points in the order from easy to hard during training.

This work merges the above two ideas. Specifically, it trains a selective neural network using curriculum learning. Although one could leverage the vanilla curriculum learning strategy, the notion of rejection provides an opportunity to align the difficulty notion used in the curriulum learning and the selective neural networks. This work calibrates the input into easy and hard examples and serves each mini-batch to the abstaining network such that it can enforce a coverage constratint ( amount of examples that the network accepts for prediction ). Empirical evaluation shows that the proposed curriculum learning design for selective neural networks is better than vanilla techniques.

strength_and_weaknesses: 
Strengths:
- Incorporating Curriculum learning in Selective Classification is indeed novel 
- Proposed carefully designed mini-batch design is indeed relevant to the coverage constraint being tackled in the Selective Classification

Weaknesses:
- Poor empirical performance (in many cases the improvement in error is in the range of 0.3-0.5 see Table~1, 2)
- Lack of ablations that justify the design choices (selectivenet over other abstaining neural networks, pacing function, scoring function)
- Does not address the computational overhead of the proposed method
- It is unclear why this work chooses the proposed difficulty score estimator. It is by no means computational cheap as compared to simpler metrics such as entropy, softmax margin, etc. 

Questions for Authors:

- Choice of pacing function : For completeness, could you describe what the fixed exponential pacing function is? Did you try some other pacing functions? Since this problem ( CL + abstaining classifier ) is different than existing CL work, it is unclear why fixed exponential pacing function should be the de-fact choice in this scenario?

- Choice of scoring function : Outlier detection algorithm ( Minimum Covariance Determinant (MCD) ) seems like an expensive way to measure the difficulty of examples. Is there a comparison on the training time split, i.e., how much time gets split between training the auxiliary classifier, how much goes in computing the MCD scores, how much in training the abstaining classifier? Have you tried other simpler scoring functions such as entropy, softmax margin, etc.?

- How often is the MCD invoked? Are the scores refreshed every epoch in the training or only once in a while? 

- Could you elaborate the point regarding the use of inception network w.r.t. MCD?

- Could you shed some lights as to why the gains are not impressive in Table 1 and 2?

- Can you plot the hardness measure to see how well separated easy/hard examples are for various coverage levels?

-  Is there a reason why SelectiveNet was used as the abstaining architecture? There are other superior networks than SelectiveNet, for instance, 
     - DeepGamblers : https://proceedings.neurips.cc/paper/2019/file/0c4b1eeb45c90b52bfb9d07943d855ab-Paper.pdf
     - One-Sided Predictions : http://proceedings.mlr.press/v130/gangrade21a/gangrade21a.pdf
     - Self-Adaptive Training : https://proceedings.neurips.cc/paper/2020/file/e0ab531ec312161511493b002f9be2ee-Paper.pdf
     - NNTD : https://arxiv.org/pdf/2205.13532.pdf



clarity,_quality,_novelty_and_reproducibility: Paper is written in a simple manner and it is easy to follow. This work is indeed novel in that in integrates curriculum learning while training selective neural networks. It does not simply follow vanilla curriculum strategy but creates a mini-batch that respects the coverage constraint. 

summary_of_the_review: This work is indeed novel in that in integrates curriculum learning while training selective neural networks. It does not simply follow vanilla curriculum strategy but creates a mini-batch that respects the coverage constraint. 

I had expected a bit more thoughts in the design choices such as scoring and pacing functions. Since these are very crucial to the success of proposed method. In addition, choice of selective net is also unclear. There are other abstaining classifiers such as DeepGamblers and NNTD with better performance. I would have expected a few ablations that justify the choice of selectivenet.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work focuses on the training of selective networks, where the network knows then to reject to answer for unconfident examples. They propose a curriculum-inspired method to train selective neural works, where they (1) consider the target converge ratio when sampling batches, and (2) use the example difficulty score to to design the curriculum. They conduct experiments on CIFAR-10/100 and SVHN to demonstrate the usefulness of their method.

strength_and_weaknesses: The proposed idea is straightforward and makes sense to me. But I feel the empirical evaluation is not rigorous enough to validate the usefulness of the proposed method. Here are my concerns:

(1) The improvement over the baseline SelectiveNet is quite marginal, e.g., ~0.5% error rate reduction on CIFAR-100 in Table 2.

(2) There are no results on ImageNet. It is strongly encouraged to show ImageNet results for image classification. 

(3) There lacks diversity in network architectures. This work only shows results on a small CNN and VGG-16, both of which are probably far from the SOTA network architectures in computer vision, e.g., EfficientNet, FBNet, etc.

(4) An existing work (“Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models” ICLR 2022) shows that a couple of simple of metrics (e.g., maximum confidence, entropy, etc) can be used to make rejection decisions as well (although they use it in the context of model cascades). Due to the simplicity of those metrics, a comparison against them is necessary and we should expect an obvious improvement to justify the usefulness of the proposed method.

clarity,_quality,_novelty_and_reproducibility: This paper is well-written and reproducible. The ideas are new for selective neural networks but are very straightforward, so only contains modest novelty.
summary_of_the_review: The novelty of this work is not significant as the main ideas in this work are straightforward. In this case, I would like to look for rigorous empirical evaluation. At this point, I feel a ton of experiments need to be added to fully justify the usefulness of the proposed. The current performance improvement is marginal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies curriculum learning for selective neural networks. These are models that are allowed to reject examples if the confidence is low. The method proposed by the authors relies on carefully selecting the easy vs hard examples to be included in each mini-batch. Experiments are reported on CIFAR10, CIFAR100 and SVHN, comparing the proposed method with vanilla curriculum and non-curriculum baselines.
strength_and_weaknesses: Strengths:
- The paper is well-written and easy to follow.

- The authors study an interesting topic.

Weaknesses:
- The level of novelty is limited in my opinion. The curriculum learning method is quite standard. The novelty consists in applying curriculum learning to selective nets, which does not represent a significant contribution.

- The authors use the relative improvements in terms of error rate, which is deceiving. The relative gains in terms of accuracy would be very different. Nonetheless, the absolute gains seem rather small.

- The authors use only small-scale and low-resolution datasets. It is thus unclear if the method would help on large-scale datasets where there are more examples per class available.

- The authors should also relate and compare with curriculum methods not using example difficulty scores, e.g. [A].

- In my opinion, the studied problem is related to the idea of dispatching test examples to different models based on image difficulty [B]. It would be nice for the authors to mention that hard examples could be processed by another (more confident) model instead of rejecting them.

[A] Samarth Sinha, Animesh Garg, and Hugo Larochelle, “Curriculum by smoothing.” In Neural Information Processing Systems, pp. 21653-21664, 2020. 

[B] Petru Soviany, and Radu Tudor Ionescu. "Optimizing the trade-off between single-stage and two-stage deep object detectors using image difficulty prediction." In International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, pp. 209-214, 2018.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, but the novelty is limited.
summary_of_the_review: My main concerns are related to the low-level of novelty and the rather small improvements on low-resolution datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper deals with NN models that are allowed to abstain from making a prediction whenever it is not sufficiently confident. These selective models are used to minimize risk of wrong prediction. Their model is based on SelectiveNet, which has 3 heads, one for minimizing the error for selected examples, one for deciding whether to select or reject examples and the last one for minimizing overall error rate. Given an order set of examples based on their difficulty scores, the curriculum-inspired training proposed by this work samples a mini-batch consisting of easy and hard examples to iteratively improve coverage and prediction accuracy of the NN model. They show improved performance of their technique over SelectiveNet. 

strength_and_weaknesses: Comments:
1. First paper to leverage the difficulty scores obtained by selective classification in curriculum learning framework.
2. The contribution of this work is (1) defining the coverage term in the loss function. (2) use of mini-batch learning for curriculum inspired training. 
3. The experiments demonstrate the effectiveness of their technique over SelectiveNet. 
4. Have you tried different sampling strategies, instead of uniformly distributing easy and difficult samples in a mini-batch? Something like importance sampling based on the difficulty scores?
5. Any theoretical guarantees of using this type of curriculum-inspired training? 
6. Any experiments/datasets where this technique failed to improve over vanilla learning?
7. Any observations on getting worse results for this way of obtaining mini-batches? Some theoretical guarantees on convergence will be perfect but even empirical evaluation will be good. 

This work is clearly presented and well written. The appendix also covers more details on ablation studies and convergence rates. I feel that this work is incremental and not very novel but useful nonetheless. 

clarity,_quality,_novelty_and_reproducibility: <Covered in comments above.>
summary_of_the_review: This work is clearly presented and well written. The appendix also covers more details on ablation studies and convergence rates. I feel that this work is not very novel but useful nonetheless.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The proposed approach in this paper combines elements of SelectiveNet and curriculum learning to address the challenge of selective classification. While the use of curriculum learning is a valuable contribution, the reliance on SelectiveNet as the only baseline is a limitation of the paper. Furthermore, the focus on Cifar and SVHN as the primary datasets is not as relevant or convincing as using the ImageNet dataset, which is widely considered to be more complex and challenging.

Overall, this paper has some valuable insights, but the limited empirical content and the lack of robust comparison to other baselines make it difficult to fully assess the effectiveness of the proposed approach. We encourage the authors to consider incorporating additional baselines and focusing on the ImageNet dataset in order to strengthen the empirical content of the paper and make it more convincing. Additionally, we recommend that the suggestions and questions raised by the reviewers should be carefully considered and addressed in a revised version of the paper.
Decision: Reject