Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on the trade-off between the error rate of the classifier and the number of examples that need to be processed manually because the system was not confident enough to process them automatically. This trade-off is at the heart of all uses of automatic classifiers for industrial tasks with maximum error rate constraints. The authors propose a curve (Confidence Operating Characteristics (COC)) and a measure on this curve (AUCOC) to compare classifiers by taking into account the rate of examples rejected to a treatment because of a lack of confidence of the classifier. The authors compare the use of a loss function derived from this curve for training neural networks with standard calibration methods. The experiments are conducted on 3 datasets (CIFAR100, TinyImageNet and DermaMNIST)


strength_and_weaknesses: The paper addresses an important but often neglected problem in the field: the search for an operating point in the implementation of an automatic prediction system coupled with a manual validation. This problem is closely related to the search for a good calibration for machine learning models and in particular neural networks.

However, the proposed curve is not new and has been used for a long time in industrial systems.  See for example the read/substitution curves, used to determine the number of rejected bank checks: 

 N. Gorski, V. Anisimov, E. Augustin, O. Baret, and S. Maximov, “Industrial bank check processing: the A2iA CheckReader,” Int. J. Doc. Anal. Recognit., pp. 196–206, 2001.

Experiments seem to show an advantage to the proposed method, but this advantage remains limited. The datasets chosen are perhaps not the most adequate to highlight the contribution of the proposed method because, apart from DemaMNIST, they do not correspond to use cases where a minimum performance is fixed and implies manual processing to achieve it. An illustration on real cases of the contribution of the proposed method in terms of reduction of the percentage of manual processing would be interesting. Figure 4 in the appendix gives an indication in this sense and shows that the difference between the different methods is very small.


clarity,_quality,_novelty_and_reproducibility: The proposed method is simple and sufficiently described. However, the method lacks novelty. The experiments seem simple enough to be reproduced.

summary_of_the_review: The method proposed in this article is not totally new and its experimental evaluation does not show a major gain compared to other calibration methods. The experiments remain somewhat limited, on data sets that do not really correspond to industrial use cases
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a COC curve and area under the COC as a loss to obtain a better trade-off between network confidence and accuracy. The proposed loss together with the standard cross-entropy loss yields a better trade-off as measured by AUCOC.
strength_and_weaknesses: ## Strengths
1. AUCOC is interesting as a measure of a trade-off between confidence and accuracy and also as an auxiliary loss.
2. The experiments show marginal improvements in this trade-off as measured by AUCOC.

## Weaknesses
1. An important paper is not cited [a]. For instance, the COC curve introduced in this paper has similarities to the ones used in [a] and the Kolmogorov-Smirnov (KS) metric has similar properties to AUCOC. Specifically, the counter-example provided in Fig.1 for ECE is not applicable to the KS metric. This raises questions about the value addition of the COC/AUCOC. Please address this.
2. The value of AUCOC as a loss is not very clear to me. Is it improving calibration if so it should be shown in the evaluation. I think it would be better to show the KS metric in table 1 to show calibration. As we know, ECE is susceptible to the binning scheme.
3. Section 2.3 makes multiple approximations and it is not clear about the effect of them in practice.


[a] Gupta K, Rahimi A, Ajanthan T, Mensink T, Sminchisescu C, Hartley R. Calibration of neural networks using splines. ICLR 2021.
clarity,_quality,_novelty_and_reproducibility: ## Clarity 
The paper has some concerns about clarity. For example, what are p(r) and c in eq. 1? Also sec 2.3 is not clear to me. Overall, the clarity of the method section could be improved by explaining the equations in simple words so that they could be understood by non-experts.

## Quality and Novelty
As mentioned previously, there are similarities to the KS metric and it is not clear how much value the presented COC/AUCOC adds to the literature. 

## Reproducibility
Please release the code to ensure reproducibility.

summary_of_the_review: There are concerns about the value addition of the proposed method and an important reference missing. Due to this I'm recommending rejection but looking forward to the response from the authors.

## Post rebuttal
I thank the authors for the detailed response. I'm convinced that the proposed AUCOC loss adds value beyond calibration and I appreciate the effort to include KS and other calibration metrics. The discussion about the relation to calibration should be included in the paper to improve clarity. I'm increasing the score to marginal-accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the fact that there should be an operating point based on the confidence of a model (deep neural network) in which an expert should examine the decisions in human-AI collaboration settings. The authors propose a confidence operating characteristics (COC) curve to summarize a model's performance. In COC, the x-axis is the proportion of samples with confidence scores less than a threshold; that is the proportion of samples that should be delivered to an expert. The y-axis is the model's expected accuracy for samples with higher confidence scores than the threshold. The curve is constructed by varying the threshold value. Then, the authors propose the area under the COC curve (AUCOC) that summarizes the curve as a single metric. They use the metric as an objective (usually combined with other metrics such as cross-entropy) to train deep nets. Kernel density estimation is utilized to compute the (extra) terms in the objective. They claim that the trained networks with this extra objective are more accurate and require fewer samples to be delegated to the experts.
strength_and_weaknesses: The paper is, in general, well-written.

The authors consider an important problem where the expert load versus the confidence/accuracy of a model is examined.

Taking expert load into account might be helpful in some applications.

Their method seems to have good results both for in and out of distribution cases.

While the toy example is intuitive, the problem is raised because of the binning issue in the ECE metric. Binning-free metrics like the Brier score may not suffer from this issue. Evaluating on other metrics such as Brier, NLL, KS[1], and classwise calibration (calibration on other classes rather than the top one) can further strengthen the paper.

It would be better to discuss the computational time and the extra overhead the proposed method may have in the main paper. 
Some details are missing: e.g., the number of bins (M) used for calculating the ECE metric, and the kernel density is computed using batches or the whole dataset.

The ECE errors are sometimes higher than other methods (especially on the DermaMNIST dataset). This brings a few questions: Is AUCOC loss a proper scoring rule? Is this going to have some effect on the results of AUCOC?

Two checkpoints are considered: one by the accuracy of the model and the other by the AUCOC value. Which one do the authors suggest using?

[1] Gupta et al., "Calibration of neural networks using splines," ICLR 2022.
clarity,_quality,_novelty_and_reproducibility: The code is provided and the COC curve, and the AUCOC loss that tries to minimize the expert load are novel.
summary_of_the_review: I think the paper is well-written and well motivated. The results look good and the code is also provided. A bit more details on the experiments and analysis of the introduced loss function might be required.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a trade-off between the model accuracy and the amount of data needed to be validated by domain experts. In particular, the paper presents 1) Confidence Operating Characteristics (COC) curve and 2) a new loss function for optimizing the area under the curve. 
 
Strength: 
- A measure (as well as an auxiliary loss) for computing the trade-off between confidence and accuracy
Generally a well-written paper.
- AUCOC achieves similar calibration performance while improving other metrics in most cases. 
 
Weakness:
- The COC was not new “N. Gorski, V. Anisimov, E. Augustin, O. Baret, and S. Maximov, “Industrial bank check processing: the A2iA CheckReader,” Int. J. Doc. Anal. Recognit., pp. 196–206, 2001. (as pointed out by Reviewer DLK2)
- Limited experimental validation on real datasets.
 
The authors’ responses address some of the concerns. Reviewer FvY2 and Reviewer 4BfS engaged in discussions with the authors and clarified several misunderstandings (e.g., the relationship with the KS metric). The AC, however, has concerned about the novelty issue brought up by Reviewer DLK2. Given that the main contribution is to "apply" an existing work on COC (which is widely used in the literature) to DL, the authors should discuss why this work has not been used in DL and the associated challenges. Then articulate *how* they address the challenge (which will be the main contribution). After reading all the reviews and discussions, the AC is convinced that the paper, in its current state, does not have sufficient merits and recommends to reject. 

justification_for_why_not_higher_score: As multiple reviewers pointed out, the experimental validations in this paper are not particularly strong. Coupled with the issues with the novelty of COC, the AC believes that rejection is a suitable choice.
Decision: Reject