Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper summarizes various types of classification failures and confidence scoring metrics and identifies their shortcomings with respect to the softmax baseline. Extensive experiments on public image benchmarks (CIFAR-10, CIFAR-100, SVHN, CAMELYON-17-Wilds, iWildCam-2020- Wilds, BREEDS-ENTITY-13) indicate that confidence scoring techniques are sensitive to the dataset choice, integer precision, and calibration.

strength_and_weaknesses: The key strength of the paper is the thorough discussion of evaluation of model classification failures, which is an important and often overlooked part of ML research. The key conclusion is concisely stated in the conclusion: a need for greater introspection and possibly less novelty. 

The key weakness is that the paper is difficult to follow. It is not clear how the pitfalls discussed in Section 2 are evaluated or studied in the remainder of the paper. A number of relevant method definitions are found in the appendix, rather than the main paper. In addition, definitions of techniques is often found after the fact. The paper would benefit from greater organization and extraction of key findings and results that support these conclusions. In Table 2, a number of results are reported, however, it is difficult to parse the metrics, color code, and main results from the paper. One of key findings is that MSR outperforms other techniques, but (a) this is not obvious from the results as its not clear what trends to look for in the metrics (a simple up/down arrow would help) (b) no analysis and discussion of other techniques is provided.  

clarity,_quality,_novelty_and_reproducibility: The paper offers a number of valuable insights into failure analysis, confidence scoring and evaluation practices of classifiers. An extensive evaluation  is generated over a number of public benchmarks. I believe the paper would be very valuable to the research community because it brings up overlooked points in evaluation protocols. A detailed code repository is included. 

However, the manuscript is relatively difficult to follow and understand. 

summary_of_the_review: A useful paper offering insightful, novel analysis, but very difficult to follow.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper offers a set of criteria (and a metric area under risk coverage curve) to compare different flavors of failure detection. It gives experiments  that compare different methods according to these criteria.
strength_and_weaknesses: (strengths)

The paper evidently springs from a substantial understanding of and experience with the topic.

The proposed criteria are thoughtful and offer, agree or disagree, a good point of departure for discussion.

The "step back and look in the mirror" approach is welcome.

The findings section is super interesting.

(weaknesses)

Sometimes the paper covers ground too fast for me (too condensed).
clarity,_quality,_novelty_and_reproducibility: I believe the work is high quality and well informed. I do not know the literature. 

The perspective and approach are thoughtful.

The authors indicate full reproducibility via open codebases.
summary_of_the_review: The proposed criteria are thoughtful and offer, agree or disagree, a good point of departure for discussion.

The "step back and look in the mirror" approach is welcome.

The paper combines many methods that are usually silo'ed.

The findings section is super interesting.

The Conclusions paragraph is wonderful.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: With the advancements of machine learning and boosting availability of compatible hardware as well as cloud services, machine learning-based classification systems have been deployed in numerous scenarios. Regardless of the specific application — some more critical than others — it is essential to detect the failure cases in a timely manner in order to quickly iterate on targeted improvements.

The authors pointed out that currently the dominant means for such failure detection is through checking some type of confidence scores, either directly produced by the model as an output, or indirectly calculated through some other heuristics (e.g., recall the notion of uncertainty in Focal Loss). The authors claimed that these different approaches are aiming to solve the same failure detection problem yet through distinctive methods which led to diverging protocols of various subfields (misclassification detection, out-of-distribution detection, selective classification, predictive uncertainty quantification). The authors then systematically analyzed the existing methods to reveal their pitfalls, and proposed a holistic and realistic approach for failure detection.

The proposed metric is the Area under the Risk-Coverage Curve (AURC), which is proposed in Geifman et al. This metric assess the rate of silent failures of a classifier across different failure filtering thresholds. The authors claimed that the metric avoids all pitfalls they revealed, and recommended wide adoption across all subfields of research within classification failure detection. The authors further delineated the modification of protocols needed to adapt to the new evaluation approach from these aforementioned subfields.
strength_and_weaknesses: Strengths
1. I strongly encourage the intention stated by the authors: to articulate a call to various interrelated yet isolated communities to acknowledge the shortcomings of current practices and adapt to a more reasonable practice that not only better fit their alleged purpose but also help bring the isolated fields together.
2. Section 2 is a deliberate and eye-catching section. The authors discussed three pitfalls of the existing methods and derived the corresponding point-by-point requirements for a desired evaluation protocol for failure detection. Among them, I particularly like (1) the second pitfall where the authors drew out attention towards different sources of classification failure and pointed out that existing methods often cover a subset of them (also illustrated in Figure 1). (2) the third pitfall where most relevant research claim failure detection as their purpose but in fact are evaluated on outlier detection, causing problems due to the mismatch (also illustrated in Figure 2).
3. I also like that the authors pointed out an easily neglected issue: rounding errors during softmax operation may largely affect the resulting evaluation metrics.

Weaknesses
1. Personally I believe reorganizing section 4.2 would be beneficial. For example, consider organizing the CSFs to be compared by their subfields (MisD, SC, PUQ, OoD-D) if possible?
2. Table 1 is a bit difficult for me to quickly process — I understand what is displayed but my brain is not well trained to instantly make sense of this representation. Is this just me or is it a common inconvenience? If it’s that latter, I would suggest adding a column for each dataset showing the “sum of rank order for the current method across all evaluations in this dataset”.

Minor Things
1. Typo? Section 2, Pitfall 2, last sentence. “…it is not realistic to exclusively assume classification failures from label-altering shifts **an** no failures caused by label-preserving shifts.”
2. Typo? Section 4.3, Under heading “Different types of uncertainty are empirically not distinguishable.“ 3rd line. “…we are interested in the **extend** to which such relations can be confirmed…”
3. Is there any reason $y_{f,i}$ in Eq. 24 is highlighted?
clarity,_quality,_novelty_and_reproducibility: This paper is of high clarity and quality. Although it does not propose a new approach/technique/method, I still believe it has considerable novelty.
summary_of_the_review: This paper encouraged researchers to view multiple subfields (misclassification detection, out-of-distribution detection, selective classification, predictive uncertainty quantification) under the umbrella of classification failure detection through a unified perspective, and proposed improvements to the problem statements and walked through the design of a generic evaluation protocol. More remarkably, the authors showed that a simple baseline can match or beat the state-of-the-arts in these subfields on the unified benchmark, since these SOTA techniques only perform in their selective subfield and miss the generalizability in the bigger picture. I like the scope, vision and insight from the authors, and would recommend for acceptance of this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This submission is in some sense a position paper, as well as an enumeration of issues in evaluation of image classification, a set of recommendations, and an empirical study with a code base for classifier evaluation.  The reviewers were unanimous in their praise for the contributions, and the message of taking a step back and understanding common pitfalls in the current state of the literature.  Reviewers were of this opinion that this is a top quality submission, though there were consistent concerns about the (self-contained) presentation.  Some concrete suggestions were discussed between the reviewers and authors, with some potential fixes, though there is acknowledgement that it might be difficult within space limitations.  It was recommended for highlighting within the conference by Reviewer JKia.  A recorded talk/spotlight would provide additional guidance through the paper, which may go some way to making its content more accessible, in addition to bringing the contributions to the attention of a wider audience.
justification_for_why_not_lower_score: The paper received high scores, and perhaps more importantly, is on a topic that is of general interest and will serve to improve the state of best-practices in our community.  It would also be OK to bump it down to a spotlight if there is a shortage of slots for oral presentation, but I think it would make a good oral.
Decision: Accept: notable-top-5%