Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper tackles an interesting question regarding model decision making. Specifially, the authors are looking into the ability of a model to abstain from making a decision, when it has low confidence scores: The “Selective Classification problem”. 

The authors demonstrate that: 
1) the current SOA models’s performance can be attributed to their ability to train a more classifier than is more generalizable (than the logit selection mechanism).
2) semi-supervised learning-based models have better selective classification ability. 
3) the new selective classifier (SR: SoftMax response) could improve current SOA models

strength_and_weaknesses: Pros:
I found the paper to be generally very well written and the experiments well motivated. The most interesting part of the paper to me was the connection with self-supervised models.
I also appreciated the authors reporting the results with errorbar statistics. I wish more CS/AI researchers emulate this
I found the results to be quite thorough. I liked seeing the results generalize to different tests and reported across the number of classes (where selective decision becomes more of a problem). 

Cons:
In some instances, I felt the results were slightly overstated. The difference between the entropy loss and the SR loss is relatively subtle across the board. 
The relation with semi-supervised learning-based models could be better explored. At the moment it is an observation. Readers would like to get an intuition for why this happens. 

clarity,_quality,_novelty_and_reproducibility: I found the paper well written. The experiments were thorough and the relation with semi-supervised learning is indeed novel, albeit less well fleshed out. 
summary_of_the_review: Overall the paper addresses an important question about Selective Classification. I was wondering if there are practical applications of this specific decision making approach. It would be a good addition to articulate the utility of this approach, especially in CV related tasks. The relation with the semi-supervised learning approaches is also interesting but less well understood. I encourage the authors to spend more time fleshing that section out in the paper (or in future work).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The problem of Selective Classification considers how to train the model’s ability to abstain from a decision when the credibility is low. In this paper, the authors analyzed the state-of-the-art methods and found that learning a more generalizable classifier is more important than using some selection mechanisms in the problem of Selective classification. Besides, they found that the Softmax Response performs better than the entropy-based selection mechanism. Through researching the relationship between Selective Classification and Semi-supervised learning, the authors proposed an entropy-regularized loss function to improve the performance. The experimental results show the effectiveness of the proposed method.
strength_and_weaknesses: **Strengths**

1. The authors introduced the state-of-the-art methods (SelectiveNet, Self-Adaptive Training, and Deep Gamblers) and the proposed method through mathematical expressions. 
2. This paper analyzes the actual effective part of the state-of-the-art methods and the correlation between the Selective Classification problem and Semi-supervised learning. The proposed method has a better research premise and innovation.
3. The experiment compares the state-of-the-art methods with the proposed method and illustrates the point of view of the paper.

**Weakness**

1. The proposed method is a bit hard to follow.  
2. The technique of the proposed method seems to be simple (i.e., mainly from the  entropy-regularized loss function).
3. The experiment is not well executed (e.g., $\beta$ is chosen as 0.01 for the proposed method, and not results under different settings).  The experimental results do not seem to work well on the Cifar10 dataset, while performing well on the two synthetic datasets generated by the authors. This kind of experimental results is unconvincing, as they may look artificial.
clarity,_quality,_novelty_and_reproducibility: Clarity: The overall presentation is okay, but the description of the proposed method can be improved (cf. Summary of The Review)

Quality and Novelty: This aspect is average, as the proposed method is mainly based on entropy-regularized loss. The authors may need to clarify or highlight the novelty of the work.

Reproducibility: The results look reproducible.
summary_of_the_review: * The authors may consider adding some visual expressions to help readers better understand the proposed method.
* The proposed method is mainly based on the  entropy-regularized loss function, which has a small amount of space in the article. The authors may point out more properites of the proposed method, so that readers can better appreciate the method. Besides, the authors may consider demonstrating the advantages of the proposed method compared with the state-of-the-art methods from the theoretical level.
* In the experiment, $\beta$ is chosen as 0.01 for the proposed method, the authors may consider giving some specific experiments to illustrate the effectiveness of the method under different settings.  The experiments do not seem to work well on the Cifar10 dataset, while the good results are on the two datasets constructed by the authors themselves. The authors need to add some datasets which also used by other researchers to improve the credibility of the experiment.

* Typo: the the -> the (cf. the last paragraph line 4 on page 4)
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors point out that predictive distribution base classifier within in selection classification captures
better signal for selection rather than a separate selection/abstention logit present in various approaches of selective classification within deep learning. 

strength_and_weaknesses: Strengths:

Paper is well written and easy to understand, and authors include various techniques of selective classification.

Weakness:

Though the point made is effective in experiments its only marginally better, including entropy regularized loss.
On CIFAR, we see the difference as we decrease coverage to 70 but even at 80% coverage selection mechanism and softmax response are at par and already <1% error. In case of imagenet we see 1.5% improvement at 80% coverage.

Also, authors do not provide further insight into why the observation exists, inspired by OOD detection work which demonstrate that max-logit is best performing OOD metric, and [3] makes an argument that max-logit is better because within Deep learning we check for familiarity of an input rather than absence of key features. So, based on OOD/anomaly detection signals & familiarity hypothesis we know it's always easy to detect presence than reject/ do good uncertainty quantification within our current deep learning practices.

Is there any reason for authors to not consider 
	1. H. Mozannar et al. Consistent Estimators for Learning to Defer to an Expert (Learning to Defer) (ICML 2020)
	2. N. Charoenphakdee et al. Classification with Rejection Based on Cost-sensitive Classification (ICML 2021)
        3. T.G. Dietterich et al. The Familiarity Hypothesis: Explaining the Behavior of Deep Open Set Methods (arXiv: 2203.02486)
clarity,_quality,_novelty_and_reproducibility: Authors make a useful observation for using selective classification, but whole idea of selective classification is to obtain a cost-sensitive prediction at training and inference time. As the current paper does not utilize the observed insight to propose a an improved selective classification training paradigm or provide more insights/practices, nor significant performance improvement.
Also, w.r.t entropy minimization regularization it might not be fruitful to minimize entropy on all instances?
Inspired by works which leverage uncertainty in case of pseudo labels[1] or even selective classification. In case of [2] authors discard hard samples to improve overall coverage vs accuracy tradeoff within selective classification. 

References:
	1. M.N. Rizve et al . In Defence of Pseudo-Labeling: An Uncertainty aware pseudo label selection framewrok for semi-supervised learning (ICLR 2021)
	2. Y. Ding et al. Uncertainty-Aware Training of Neural Networks for Selective Medical Image Segmentation (MIDL 2020)

summary_of_the_review: Though authors make an interesting point and demonstrate effectiveness of their observations. For lack of novelty, or lack of additional insights, currently in my humble opinion not convinced if the work warrants an ICLR acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose to revise the conclusions of two different selective classification methods. They argue this methods obtain betters results by just providing a more robust training. Thus, at inference step, the selective solution can be discarded. With this variation, they are able to increase the results obtained by the original paper. Furthermore, they provide an entropy regularization term that can be used in semi-supervised learning. 
strength_and_weaknesses: Strength
* The idea is simple and easy to implement
* The results are interesting

Weaknesses
* The semi-supervised entropy regularization is not new [1].
* Only one network was tested.
* The paper organization is poor.

[1] Grandvalet, Y., & Bengio, Y. (2004). Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17.
clarity,_quality,_novelty_and_reproducibility: Clarity
* The idea is simple and easy to implement.
* Both bold and underlined values presented in all Tables are not properly explained. They should be explained in the regular paper, not only at the appendix.
* It is not clear how the results of Table 5 are obtained. The authors suggest the entropy-regularized loss function for semi-supervised purposes, but I did not find any mention about how they remove some labels of the supervised dataset.
* I am not sure how the authors select the coverage from the state-of-the-art methods. In the case of SelectiveNet, did the authors train the model choosing different $c_{target}$ values? In the case of the self-adaptive training, did the authors use the last Eq. of section 3.1? It seems the self-adaptive training is made with the purpose of discarding a solution only if the $C+1$ class has the highest accuracy.
* All equations should be numbered

Quality
* The provided results are interesting, but only one network was tested. I suggest the authors to introduce more networks with different architectures, in order to ensure the results provided are consistent.

Novelty
* Although the idea is simple, it could be very effective. However, I am not sure if the experimental results are fair, given the concern I had explain regarding to the clarity of the paper.
* Besides that, the entropy-regularization term for semi-supervised learning is not novel [1]. Although, as the entropy regularization is zero whenever all the values are o or 1, the idea is pretty similar to the active learning approach, where the non-labelled examples are labelled after each training epoch.

Reproducibility
* The authors provide enough information to successfully reproduce the networks, but they should include more information regarding how the state-of-the-art methods were trained (see the clarity section for details). 

[1] Grandvalet, Y., & Bengio, Y. (2004). Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17.
summary_of_the_review: Although the idea is interesting, the proposed solutions are either not new (entropy-regularization term) or not properly explained. Thus, I cannot recommend to publish this contribution in its actual form.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper main contribution is the finding that selective classification performance can be boosted by simply using softmax response (SR) on a network that was pretrained with a selection mechanism. The authors consider to the three main selective classification methods: SelectiveNet, Deep Gamblers, and SAT. It is also shown that entropy minimization regularization improves the selective classification performance.
I believe this paper contains interesting and valuable observations and recommend acceptance.
I ask the authors to address the following issues:

1) An analysis over the complete ImageNet dataset would greatly benefit this paper. ImageNet results are highly transferable to real-world, while results on subsets of ImageNet do not always transfer even to the entire ImageNet dataset itself. Moreover, a random selection of a dataset subset can be problematic, unless it is repeated several times.

2) The paper could be streamlined, and while it is good to provide hypotheses that explain some observations, I feel some are not based on enough evidence (such as the claims about generalization).

3) A risk-coverage figure depicting the different methods could greatly improve the comparison (the tables with selective risks for different coverages are okay, but not as informative). If you want to compare only higher coverages, you can start the curves at 50% coverage, for example.

4) In some of the tables (e.g., Table 3) the boldface highlighting of SAT+EM+SR is not correct in all cases. In case of an overlap It would probably be fair to emphasize the entry with the lower standard error or all the overlapping confidence intervals.

I look forward to seeing your paper published once you fix these issues!

justification_for_why_not_higher_score: There are still some remaining issues.
Decision: Accept: poster