Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Selective classifiers have the ability to abstain on certain datapoints and consist of two functions - the classification function and the selection function. This paper considers ensembes of selective classifiers by averaging the probability distributions of the classifiers and also defining the selection function in a certain way. They provide some assumptions and theoretically prove that under those assumptions, the ensemble of selective classifiers has better selective risk than each of the individual models for certain values of coverage. They also verify these assumptions on a few datasets. They also find that the simplest ensemble of classifiers with maximum probability as the selection function has the best performance compared to the previous selective classification approaches.
strength_and_weaknesses: The idea of combining ensembles and selective classification is interesting. 

In assumption 2, why do we need the definite samples probability at confidence approaching 1 to be non-zero? It seems that the entire advantage of ensembles is coming out of the ambiguous samples. 

For the ambiguous samples, assumption 2 says that there are a few ambiguous samples which are predicted with confidence by individual models. Why it can’t be the case that all the models predict the wrong label with confidence approaching 1?

It is very hard to understand assumption 3 in the paper and it is not clear what the intention of that assumption is. It would be good to explain that in the paper. Also, the authors have verified assumptions 1 and 2 but have not commented on the applicability of assumption 3 for practical settings. Can the authors please comment on that?
clarity,_quality,_novelty_and_reproducibility: The proof in the paper is hard to read currently. It would also be good to provide some intuition of why we should expect the proof to go through. The ideas are somewhat incremental and I don't think the proofs provide any new technical challenges.
summary_of_the_review: I think the idea is interesting but the paper lacks clarity and is not well written.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors analyze ensembles in the context of selective prediction. The authors show that under loose assumptions, a selective classifier using the probability assigned to the predicted class to decide whether to abstain from predicting can be improved by ensembling $n$ copies of it (differing only in their random seed). 

The three assumptions required are, paraphrased:
- A1: When all models agree on a prediction, they are less likely to be incorrect than when they disagree. 
- A2: Each individual model has a non-zero probability of assigning predicting a class with probability one (implying that models can be overconfidently incorrect).
- A3: The joint PDF of all ensemble members is bounded on ambiguous queries (no prediction has PDF = $+\infty$).

These assumptions in hand, the authors show (Prop 1.) that on ambiguous queries, an ensemble _cannot_ assign probability 1 to any class. It follows that (Thm. 2) ensembling will improve the selective risk up to a coverage $\phi_0 > 0$.

The authors then evaluate their assumptions on vision and NLP datasets, as well as different ensembling techniques for selective prediction (including, but not restricted to, deep ensembles).
strength_and_weaknesses: # Strengths
- The paper is well structured and motivated; the literature review and general positioning of this paper is clear and well situated within the greater context of the field.
- The paper addresses an important task with clear applications to real life problems (can we automatically detect when to abstain from predicting), and analyzes a well-known, simple technique (ensembling) within this new context.
- The authors conclusively show under intuitive hypotheses that ensembling will improve selective prediction.
- Empirical results confirm that the hypotheses are most of the time reasonable; furthermore, empirical results show that ensembling for selective prediction is beneficial even beyond what theoretical results are provided.
 
# Weaknesses
- I believe the greatest issue with this work is that the theoretical and empirical results could be more detailed, as the current statements would benefit from greater investigation. For example:
  * (Thm. 2) How does $\phi_0$ depend on various ensemble characteristics? Can we bound it away from zero? Under which conditions?
  * Similarly, I think a deeper analysis of what causes the MRPC dataset to violate the assumptions (which datapoints, and understanding why) would significantly strengthen the paper, especially as the scope of the theoretical results is somewhat limited (next point).
- The scope of the theoretical results is limited to deep ensembles (i.e., ensembles whose predicted probability vector is obtained via a softmax). It wasn't very clear to me whether this assumption is required on top of the three assumptions detailed earlier -- if so, could the authors clarify why this is?
- I found some of the notation unclear -- in some cases, the quantities being manipulated are vectors (of length = number of classes), but numerical estimates are scalars. For example, the standard deviation in $\S$ 6.1. Similarly, is $R_\text{ind}$ in Thm. 2 the average selective risk of all ensemble members?

Minor: 
- I would recommend reordering Figure 2, so that each subfigure compares different models on a single dataset.
- This paper could benefit from some quick SPAG proofreading, as there are several minor grammar mistakes throughout the paper.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is reasonably clear, and the results are novel to the best of my knowledge. 

As mentioned above, I think the major issue is with the scope of the results provided in this paper. Although the results are relevant, the limited application of the theory (to highly specific types of ensembles) and the lack of follow-up on the limitations of the results (ensembling improves only up to a threshold $\phi_0$; assumptions can be violated; etc.) limit the impact of this work. 
summary_of_the_review: This is an interesting paper addressing an important question in the ML community. However, this paper could be significantly improved by probing the derived theoretical and empirical results in much more detail. As it stands, the import of this work remains unclear, and it is difficult to place the specific results and limitations within the greater context of the selective prediction problem. 

That being said, I believe that these weaknesses may be easily resolved, making this paper a valuable contribution to the field.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper the authors address the selective classification problem with the goal of reducing the selective risk (a non-convex function of the model prediction). A selective classifier is a pair (f, g), where f is a classifier, and g: X -> {0, 1} is a selection function, that allows to abstain {= 0} on some difficult input. In this paper the approach combines several weak selective classifiers (ensemble wit M = 2 to 5 members). 
The contribution is to prove that an ensemble has a lower selective risk than the individual model under a range of coverage.
The method has been tested on image classification (CIFAR-10/100 and SVHN) and text classification (MRPC, MNLI, QNLI) tasks. The backbones used are a VGG-16 and BERT-base networks respectively. The method is compared against selective classification approaches.

strength_and_weaknesses: **Strength**

- Classification with abstention has gained a lot of attention in recent years. It is an important topic.
- Interesting comparison between methods.



**Weaknesses**

- Novelty.

- Table 1: it is not clear whether the results are presented with the same level of coverage. If the coverage level is different, it is difficult to compare the values. (It is recommended to add the coverage in the table, or to specify and use a fixed coverage).

Ensemble techniques are presently hard to realize in the context of DNNs, for which it could be very costly to train sufficiently many ensemble members [1] [2].

- What is a sufficient number of ensemble members? 

Figure 6 shows that the AURC  for the SR method on the test set decreases as the number of members in the ensemble increases. What is the accuracy of the model and coverage at M=20 for SR? 

- The big backbone is designed to have twice as many filters in every convolutional layer (VGG-16) why the authors didn't consider a ResNet-50/101?



[1] Kush R Varshney. A risk bound for ensemble classification with a reject option. In Statistical Signal Processing Workshop (SSP), 2011 IEEE, pages 769–772. IEEE, 2011

[2] Yoav Freund, Yishay Mansour, and Robert E Schapire. Generalization bounds for averaged classifiers. Annals of Statistics, pages 1698–1722, 2004.
clarity,_quality,_novelty_and_reproducibility: The paper is clear, readable, the images and plots are of good quality. However, there is no technical novelty, the contributions are only marginally significant.
summary_of_the_review: The paper presents a series of experiments that support the hypothesis that a set of selective classifiers reduces selective risk. At the same time it suggests that the number of members of the ensemble should be low (2-5). Obviously, a high number of member classifiers corresponds to a cost in terms of resources, especially in DL. The paper is clear and readable. There is no technical novelty.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission shows that simple randomization-based learning of deep ensembles improves on training a single model when performing selective classification, regardless of the selective classification method that is employed. Given some assumptions that seem to hold in most of the classification scenarios considered in the submission, it is proven that this must indeed be the case. Another, empirical, finding of the submission is that simple selective classification based on a threshold on the maximum class probability is preferable to other methods, including the current state-of-the-art.
strength_and_weaknesses: + selective classification is an interesting and important problem
+ results challenge what is considered the state of the art
+ convincing empirical findings with theoretical support

- the proposed method is not new and the improvements observed with ensemble learning are unsurprising
- no discussion of work on estimating epistemic uncertainty, which seems a highly relevant topic

Questions and comments:

Are the numbers after "+/-" in Table 1 standard deviations? How were these obtained?

"there has been no systematic study of ensemble methods in selective classification" - repeated verbatim in two adjacent sentences

"selective risk ... are"

"which detects samples different greatly from a given dataset" - rephrase

"to the overconfident problem" - rephrase

"For definite samples, given the predictive probability distribution of one member model..." - I could not follow the reasoning here

"AURCs of the individual model"

"for the better performance of the ensemble than the individual selective classifier" - rephrase


clarity,_quality,_novelty_and_reproducibility: The submission is well written and great effort has been taken to spell out all steps of the arguments. The main finding of the submission, that ensemble learning enables more accurate selective classification than using a single model, is unsurprising, but the theory presented in the submission appears to be novel. It is also interesting to see that selective classification based on maximum probability, the most obvious approach, outperforms the state-of-the-art when care is taken to use the same experimental setup for both approaches.
summary_of_the_review: Although the main finding of the submission, that randomization-based ensembles outperform individual models, is unsurprising, particularly considering the experiments reported by Lakshminarayanan et al., it may be useful to have the additional theoretical justification for this method that the submission presents. This justification, based on some assumptions that seem reasonable (and seem to hold in pertinent benchmark problems), appears to be novel. For practitioners, it is also useful to see that selective classification based on maximum probability beats the state of the art in the experiments presented in the submission.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers and meta reviewer all carefully checked and discussed the rebuttal. They thank the authors for their response and their efforts during the rebuttal phase.
The response helped resolve some concerns (e.g., improvement of the experiments with new results for MRPC, together with a few clarifications about notations/assumptions).

The reviewers and meta reviewer all acknowledge that the submission investigates a relevant and important problem (“important task with clear applications to real life problems”, “selective classification is an interesting and important problem”).

At this stage, however, there are still some arguably important aspects that would benefit from further consolidations, in particular:

* As acknowledged by the authors, the core contribution of the paper lies in the theoretical analysis and the new proof technique(s). Right now, this is not properly surfaced by the manuscript. A substantial reorganization of the manuscript around that material would help deliver the main messages.

* In parallel to the point above, several reviewers raised concerns about the clarity of the notation, assumptions and other technical elements around the theoretical results (the changes made during the rebuttal phase go in the right direction). Since those represent the core results of the paper, they need to be more clearly exposed. 

Because of the extremely competitive landscape of the submissions this year, the paper remains under the cut and has ultimately not been selected for acceptance.
We are convinced that the suggestions above will help strengthen the paper for a future resubmission, which the reviewers and meta reviewer all encourage.

justification_for_why_not_higher_score: * Current structure of the paper fails to effectively deliver the main contributions (theoretical analysis/proof techniques)
* Clarity/notation issues around key components of the paper (e.g., assumptions)
Decision: Reject