Expert load matters: operating networks at high accuracy and low manual effort
Authors: Sara Sangalli
Ertunc Erdil
Ender Konukoglu
Venue: NeurIPS 2023 poster (2023)
Keywords: human-ai collaboration system
optimization

Abstract: In human-AI collaboration systems for critical applications, in order to ensure minimal error, users should set an operating point based on model confidence to determine when the decision should be delegated to human experts. 
Samples for which model confidence is lower than the operating point would be manually analysed by experts to avoid mistakes.
Such systems can become truly useful only if they consider two aspects: models should be confident only for samples for which they are accurate, and the number of samples delegated to experts should be minimized.
The latter aspect is especially crucial for applications where available expert time is limited and expensive, such as healthcare. 

The trade-off between the model accuracy and the number of samples delegated to experts can be represented by a curve that is similar to an ROC curve, which we refer to as confidence operating characteristic (COC) curve. 
In this paper, we argue that deep neural networks should be trained by taking into account both accuracy and expert load and, to that end, propose a new complementary loss function for classification that maximizes the area under this COC curve.
This promotes simultaneously the increase in network accuracy and the reduction in number of samples delegated to humans.
We perform experiments on multiple computer vision and medical image datasets for classification.
Our results demonstrate that the proposed loss improves classification accuracy and delegates less number of decisions to experts, achieves better out-of-distribution samples detection and on par calibration performance compared to existing loss functions.

Expert load matters: operating networks at high accuracy and low manual effort
Sara Sangalli
Ertune Erdil Computer Vision Lab ETH Zürich
Computer Vision Lab
ETH Zürich
sara. wwwgallliiviiinnnnmommmmmm etaz. ch
Ender Konukoglu
Computer Vision Lab, ETH Zürich
The LOOP Zürich Medical Research Center, Zürich, Switzerland
Abstract
In human-AI collaboration systems for critical applications, in order to minimal error, users should set an operating point based on model confidence determine when the decision should be delegated human experts. Samples for which model confidence is lower than the operating point would be manually analysed by experts avoid mistakes. Such systems can become truly useful only if they consider two aspects: models should be confident only for samples for which they are accurate, and the number of samples delegated to experts should be minimized. The latter aspect is especially crucial for applications where available expert time is limited and expensive. such as healthcare. The trade-off between the model accuracy and the number of samples delegated to experts can represented by a curve that is similar to an ROC curve, which we refer to as confidence operating characteristic (COC) curve. In this paper, we argue that deep neural networks should be trained by taking into account both accuracy and expert load and, to that end, propose new complementary loss function for classification that maximizes the area under this COC curve. This promotes simultaneously the increase network accuracy and the reduction in number of samples delegated humans. We perform experiments on multiple computer vision and medical image datasets for classification. Our results demonstrate that the proposed loss improves classification accuracy and delegates less number of decisions to experts, achieves better -iitt-iiiitiitnoonnn samples detection and par calibration performance compared to existing loss functions!
Introduction
Artificial intelligence (AI) systems based on deep neural networks have achieved state-of-the-art results by reaching or surpassing human-level performance in many predictive tasks (3] 36] Despite the great potential of neural networks for automation, there are pitfalls when using them in fully automated setting, especially pertinent for -eiitcal applications, such healtheare 17. 32 34]. Human-Al collaboration aims at remedying such issues by keeping humans in the loop and building systems that take advantage of both 29) An example of such human-Al collaboration hate speech detection for social media 33, where neural networks could reduce the load of manual
analysis of contents required by humans. Healthcare is another relevant application 5 22 For
Cooe is available at:
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

example, neural network trained to predict whether a lesion is benign or malignant should leave the decision to human doctors if it is likely to make an error [[3]. The doctors' domain knowledge and experience could be exploited to assess such, possibly, ambiguous cases and avoid mistakes.
A simple way of building collaboration between network and a human expert is delegating the decisions to the expert when the network's confidence score for a prediction is lower than a threshold, which we refer to as "operating point". It is clear that the choice of the operating point can only be done through a trade-off between the network performance on the automatically analysed samples, i.e. the number of errors expected by the algorithm, and the number of delegated samples, i.e., experts' workload. The latter is crucial especially for applications where expert time is limited and expensive. For example, in medical imaging, the interpretation of more complex data requires clinical expertise and the number of available experts is limited, especially in low income countries [17]. Hence, predictive models that can analyse a large portion of the samples at high accuracy and identify the few samples that should be delegated to human experts would naturally be more useful with respect to this trade-off.
It is possible to evaluate the performance of a predictive model taking simultaneously into account the accuracy and the number of samples that requires manual assessment from a human expert with a performance curve reminiscent of Receiver Operating Characteristic (ROC) curves, as illustrated in Fig. Ia We will refer to this performance curve as Confidence Operating Characteristics (COC) as is similar to the ROC curve. A COC curve plots for a varying threshold on algorithm confidence, i.c., operating point, the accuracy of model on the samples on which the algorithm more confident than the threshold versus the number of samples remaining below the threshold. The former corresponds to the accuracy of the model on automatically analysed samples while the latter corresponds to the amount of data delegated to the human expert for analysis. In an ROC curve balance is sought after between Sensitivity and Specificity of a predictive model, while a COC curve can be used by domain experts, such doctors, to identify the most suitable balance between the accuracy on the samples that are automatically analysed and the amount of data delegated to be re-examined by a human for the specific task. Variations of this curve have been used to evaluate the performance of automated industrial systems 77.
In this paper, we focus on the trade-off between model accuracy and the amount of samples delegated to domain experts based on operating points on model confidence. Specifically, our goal is to obtain better trade-off conditions to improve the interaction between the Al and the human expert. To this end, we propose a new loss function for multi-class classification, that takes into account both of the aspects by maximizing the area under COC (AUCOC) curve. This enforces the simultaneous increase in neural network accuracy on the samples not analysed by the expert and the reduction in human workload. To the best of our knowledge, this is the first paper to include the optimization of such curve during the training of neural network, formulating it in a differentiable way. We perform experiments on two computer vision and three medical image datasets for multi-class class classification. We compare the proposed complementary AUCOC loss with the conventional loss functions for training neural networks as well network calibration methods. The results demonstrate that our loss function complements other losses and improved both accuracy and AUCOC. Additionally, we evaluate network calibration and out-of-distribution (OOD) samples detection performance of networks trained with different losses. The proposed approach was also able to consistently achieve better OOD samples detection and on par network calibration performance.
2 Related Work
For the performance analysis of human-AI collaborative systems, confidence operating characteristics (COC) curves can be employed, which plot network accuracy on accepted samples against manual workload of a human expert, e.g as in 77. While such curves have been used, to the best of our knowledge, we present the first work that defines a diffeerentialee loss based on COC curve, in order to optimize neural networks to take into account simultaneously accuracy and experts' load for a human-Al collaborative system. Thus, there is no direct literature with which
compare.
The underlying assumption deciding which sample to delegate to human expert based on a threshold on confidence scores is that these scores provided by deep learning models indicate how much the predictions are likely to be correct or incorrect. However, the final softmax layer of network does not necessarily provide real probabilities of correct class assignments. In fact, modern deep neural

---

networks that achieve state-of-the-ar results are known to be overconfident even in their wrong predictions. This leads to networks that are not well-calibrated, i.e., the confidence scores do not properly indicate the likelihood of the correctness of the predictions [8) Network calibration methods mitigate this problem by calibrating the output confidence scores of the model, making the above mentioned assumption hold true. Thus, we believe that the literature on network calibration methods is the closest to our setting because they also aim at improving the interaction between human and AI, by enforcing correlation between network confidence and accuracy, so that confidence can be used to separate samples where networks' predictions are not reliable and should be delegated to human experts. Calibration methods aim to get models that are highly accurate in the samples they are confident, but not the problem of minimising the number of samples delegated to human experts, contrary to the loss proposed here.
Guo et al. defines the calibration error as the difference in expectation between accuracy and confidence in each confidence bin. One category of calibration methods augments or replaces the conventional training losses with another loss to explicitly encourage reducing the calibration error. Kumar et al. [2I] propose the MMCE loss by replacing the bins with kernels to obtain a continuous distribution and a differentiable measure of calibration. Karandikar et al. [16] propose two losses for calibration, called UOC and Soft-ECE, by replacing the hard confidence thresholding in AvUC
[8] and binning in ECE 8 with smooth functions, respectively. All these three functions are used as secondary loss along with conventional losses such as cross-entropy. Mukhoti et al. [26] find that Focal Loss (FL) [23] provides inherently more calibrated models, even it was not originally designed for this, it adds implicit weight regularisation. The authors further propose Adaptive Focal Loss (AdaFI.) with sample-dependent schedule for the choice of the hyperparameter y. The second category of methods are post-hoe calibration approaches, which rescale model predictions after training. Platt scaling [3] and histogram binning [40) fall into this class. Temperature scaling (TS) 8 is the most popular approach of this group. TS scales the logits of a neural network, dividing them by positive scalar, such that they do not saturate after the subsequent softmax activation. TS can be used as a complementary method and it does not affect model accuracy, while significantly improving calibration. A recent work by Gupta et al. 191 fits spline function to the empirical cumulative distribution to re-calibrate post-hoc the network outputs. They also present bimning- ffree calibration measure inspired by the Kolmogorov- -omirooo (KS) statistical test. Lin et al. (24) propose a Kernel-based based method on the penultimate-lay latent embedding using a calibration set.
3 Methods
In this section, we illustrate in detail the Confidence Operating Characteristics (COC) curve. Then, we describe the proposed complementary cost function to train neural networks for classification; the area under COC (AUCOC) loss (AUCOCLoss).
3.1 Notation
Let D = ((an: Un)) denote a dataset composed of N samples from a joint distribution D(X, )), where H,, E X and Yn E y = 21. K} are the input data and the corresponding class label, respectively. Let fo (y) x) be the probability distribution predicted by a classification neural network parameterized by 9 for an input x. For each data point En, In ar argmax,es yy fo (y) denotes the predicted class label, associated a correctness score en 1 (gn = you) and to a confidence score Tn = y foyla,) where r, E [0, 1] and (.) is an indicator function. r = [r1,... represents the vector containing all the predicted confidences for a set of data points, e.g., batch. p(r) denotes the probability distribution over values (confidence space). We assume human-Al AI collaboration system where samples with confidence r lower than a threshold ro would be delegated to human expert for assessment.
3.2 Confidence Operating Characteristics (COC) curve
Our first goal is to introduce an appropriate evaluation method to assess the trade-off between a neural network's prediction accuracy and the number of samples that requires manual analysis from domain expert. We focus on the COC curve, as provides practitioners with flexibility in the choice of the operating point, similarly to the ROC curve.
3

---

3.2.1 X-y axes of the COC curve
To construct the COC curve, first, we define sliding threshold To over the space of predicted confidences Then, for each threshold ru. we calculate the portion of samples that are delegated to human expert and the accuracy of the network on the remaining samples for the threshold ro. which form x-axis and y-axis of COC curve, respectively. These axes are formulated as follows
axis: TO p(r < <rr) - / p(r)dr,
y axis Elelr 2 rol
For each threshold level threshold, i.e., the amount of the samples that are delegated to the expert. E|c/r 2 rul corresponds to the expected value of the correctness score for all the samples for which the network's confidence is equal or larger than ro. i.e., accuracy among the samples for which network prediction will be used. This expected value, i.e. y-axis, can be computed
represents the portion of samples whose confidence is lower than that
Eielr 2 ro] To
Eldr)p(()dr.
(2)
We provide the derivation of Eq. 2 in the Appendix A
3.2.2 Area under COC curve
Like the area under ROC curve, area under COC curve (AUCOC) is global indicator of the performance of a system. Higher AUCOC indicates lower number of samples delegated to human experts or/and higher accuracy for the samples that not delegated to human experts but analysed only by the network. Lower AUCOC on the other hand, indicates higher number of delegations to human experts or/and lower accuracy the samples analysed only by the network. Further explaination reported in Appendix]] []] whole range of Tu [0, 1]:
It can be computed by integrating the COC curve over the
roldTo S/') ( ((ddrl(((r)
) diu TO
AUCOC 2
(3)
Further details are provided in Section 3.5
3.3 AUCOCLoss: Maximizing AUCOC for training neural networks
As mentioned, in human-AI collaboration systems, higher AUCOC means better model. Therefore, in this section we introduce new loss function called AUCOCLoss that maximizes AUCOC for training classification neural networks.
AUCOC's explicit maximization would enforce the reduction of the number of samples delegated to human expert while maintaining the accuracy level the samples assessed only by the algorithm (i.e., keeping E[cr 2 ro] constant) and/or the improvement in the prediction accuracy the samples analysed only by the algorithm while maintaining a particular amount of data to be delegated to the human (i.e., keeping To constant), as illustrated in Figure la
We define our loss function to maximize AUCOC as log(AUCOC).
AUCOCLoss
We use the negative logarithm as AUCOC lies in the interval [0, 1], which corresponds to AUCOCLoSs [0, inf] which suitable for minimizing cost functions. The dependence of AUCOC on the network parameters may not be obvious from the formulation given in Eq. 3 Indeed, this dependence is hidden in how p(r) and E[c[r] are estimated as we show next.
3.3.1 Kernel density estimation for AUCOC
We need to formulate AUCOC differentiable way in order for it to be incorporated a cost function for the training of neural network. For this purpose, use kernel density estimation (KDE) on confidence predictions r, for training samples to estimate p(r) used in Eq.
N
p(r) R
Σ n=1
K(r - -w)
(5)

---

where K is Gaussian kernel and we choose its bandwidth using Scott's 's rule of thumb [3]. Then, the other terms Eq. namely [(clr)p(r) and To- are estimated as
N
N Σ n=1
B(crip(r) ~
N Cn Kr Tn),
2 N
K(r-rail -or--wi rn)dr.
(6)
==
0
Note that 7, = m Zo()) from where the dependency of AUCOC on stems. Further, note that 70 is the x-axis of the COC curve. The AUCOC is defined by integrating over the 7o values, and therefore To should not depend on the network parameters, i.e., its derivative with respect to should be equal zero. We can write the derivative using Leibniz integral rule follows:
dro da
/." dp(r) dr p(ro) dB
dro dB
Then, the constraint that To should not depend on 0 can be enforced explicitly by deriving the derivative dro/ do from Eq. 7 as follows:
drg
T dp(r) dr p(ro)
(8)
d0
where p(r) implemented as in Eq. Derivations are provided the Appendix.
curves
Predicted Confidences
f 01
Increasing accuracy
Predicted Confidences Neural Network føz
85
2) Decreasing the amount of delegated samples
A 0.54 for ALCCC 49
9etth, AUcO 12.00
0.
1th. AUROU 14.9
AUCOClem ALICCC
75
20 40
6
50
100
BCE, BCE
KS
A00 AcCfos
AUCOC AUCOC
Ts: 08 eiples anslysed manuslly 156)
(a)
(b)
Figure 1: (a) shows how to improve AUCOC, 1) increasing the accuracy of the network and/or 2) decreasing the amount of data to be analysed by the domain expert. The pink curve has higher AUCOC than the blue one. (b) illustrates a toy example where two models have the same accuracy, ECE with 5 bins and KS. However, they have different AUCOC values due to different ordering of correctly and incorrectly classified samples according to the assigned confidence by the network.
3.4 Toy example: added value by AUCOC
In this section, we demonstrate the added value of assessing the performance of a predictive model using COC curve and AUCOC through toy example. We particularly compare with the widely used expected calibration error (ECE) [8) binning-free calibration metric called Kolmogorov-S -mirro (KS) O and classification accuracy.
Assume we have two classification models fe, and fez and they yield confidence scores and predic- tions for 5 samples shown in Figure [b] The green circles denote the predicted confidences for correctly classified samples, while the red crosses the confidences of the misclassified ones.
ECE divides the confidence space into bins, computes the difference between the average accuracy and confidence for each bin, and returns the average the differences as final measure of calibration error. If we divide the confidence space into 5 5-bins, indicated with the gray dotted lines the confidence spaces of fa, and fog, ECEs computed for the both models will identical. Furthermore, similar situation can be constructed for any number of bins.
ECE:
KS: KS is binning-free metric, so it is less prone to binning errors than ECE, However, one can prove that there exist some confidence configurations for which the two models also report the same
5

---

KS, in spite of it being a binning-free free metric. This happens, for example, if the confidence values are (from left to right): 0.45, 0.55, 0.65, 0.70, 0.75.
Accuracy: These models have equal classification accuracy, 3/5 for both.
Therefore, looking at these three performance metrics, it is not possible choose model over the other since fe, and fe, perform identically.
AUCOC: On the contrary, the AUCOC is larger for fe, than for fa,, shown in Figure lb The difference AUCOC is due to the different ranking of correctly and incorrectly classified samples with respect to confidence values. It does not depend values, but only on the ranking. By looking at the AUCOC results, one would prefer fa, compared to fez. Indeed, fe. is a better model than fe, because it achieves either equal or better accuracy than Sag. for the same amount of data to be manually examined. Changing point of view, So, delegates either equal or lower number of samples experts for the same accuracy level.
particular binning exact confidence
Comparison of AUCOC and calibration: As also demonstrated with the toy example, AUCOC and calibration assess different aspects of model. The former is rank-based metric with respect to the ordering of the predictive confidences of correctly and incorrectly classified samples. The latter instead assess the consistency between the accuracy of the network and the predictive confidences, i.e. it is sensitive to the numerical values of the confidences themselves.
3.5 Implementation Details
Construction of the COC curve, thresholds To and operating points: To provide flexibility in the selection of COC operating points, we need cover the entire range [0. 1] of To values. As a consequence, the thresholds need to span the confidence range of the predictions In of the neural network. natural choice for such thresholds is employing the predictions rn themselves. This spares us from exploring the confidence space with arbitrarily fine-grained levels. First, we sort the confidences of the whole dataset (or batch) in ascending order. Each predicted confidence is then selected as threshold level ro for the vector corresponding to a certain To (x-value). Subsequently, Elc|r 2 ro] (y-value) is computed. Note that setting the threshold to each rn in the sorted array corresponds to going through TO |1/,, (N 1)/N, 1] for N samples one by one in order.
Modelling correctness: Instead of using E/crrp((r) ~ NV EN= K(I rall) as given in Eq. 6
we approximate it as Ejc|rip(r)= N Σ m== r ((rrrr)), where r" = fo (yn (z,,,,) ) the confidence of the correct class for sample n. The main reason is that the gradient of the misclassified samples becomes zero because 6, is zero when a sample H,, not classified correctly. To deal with this
issue we replace the correctness score Ca. which can be either 0 or 1, with r which can take continuous values between 0 and 1, following Yin et al. [39]. With this new approximation, we can back propagate through misclassified samples and we found that this leads to better results.
Use as secondary loss: We observed in our experiments that using AUCOCLoss alone to train a network leads to very slow convergence with the existing optimization techniques. We believe this is due to the fact that the AUCOCLoss is -log(E,, 2n) with zn, [0. 1]. Optimization of such a form with gradient descent is slow because the contribution of increasing low 2,, 's to the loss function is small. In contrast, cross-entropy is a Σ, log Znd where contribution of increasing low 2, to the loss is much larger, hence tradient-based optimization is faster. One can in theory create an upper bound to AUCOC loss by pulling the log inside the sum, as we show in the Appendix. However, when 2n E [0. 1] this upper bound is very loose and minimizing the upper bound not necessarily correspond to minimizing the AUCOC loss, hence does not maximize AUCOC. On the contrary, when AUCOCLoss is complements primary cost that is faster to optimize, such as cross-entropy, is improves over the primary loss and lead to the desired improved AUCOC while preserving the accuracy. This is obtained within the same amount of epochs and without ad-hoc fine-tuning the training parameerssss for the secondary loss.
4 Experiments
In this section, we present our experimental evaluations on multi-class image classification tasks. We performed experiments on five datasets. We experimented with CIFAR100 [19] and Tiny- ImageNe,,,, a subset of ImageNet 4), following the literature network calibration. Further, used two publicly

---

available medical imaging datasets, DermaMNIST and RetinaMNIST 38). since medical imaging an application area where expert time is limited and expensive. Due to space reasons we report results on third medical dataset, TisSueMNIST 38, in Appendix D as well as information about the datasets in Appendix [
We compared AUCOCLoss (referred to as AUCOCL in the tables) with different loss fuunctions, most of which are designed improve calibration performance while preserving accuracy: cross-entropy (CE), focal-loss (FL) 23 adaptive focal-loss (AdaFL) 226], maximum mean calibration error loss (MMCE) 22 soft binning calibration objective (S-ECE) and soft aceuracy versus uncertainty calibration (S- Av-) [[]]. We optimized MMCE, S-ECE, and S-AvUC losses jointly with primary loss for which we used either CE or FL, consistently with the literature [6]
Loss functiinn:
The same is done for AUCOCLoss, applying KDE batch-wise during the training.
Evaluation metries:
To evaluate the performance of the methods, used classification accuracy and AUCOC. Classification accuracy is simply the ratio between the number of correct samples over the total number of samples, and AUCOC is computed using Eq. 3 We also report some examples of operating points of COC curve given a certain accuracy, we show the corresponding expert load (To @acc), i.e. percentage of samples that need to be analyzed manually, on the COC curves.
Since we compared AUCOCLoss mostly with losses for network calibration, we also assessed the calibration performance of the networks, even though calibration is not direct goal of this work. To this end, we used the following metrics: the widely employed equal-mass expected calibration error (ECH) 28 with 15 bins, the binning-free free Brier score [1] and Kolmogorov-Smirnov (KS) score [9] and the class-wise ECE (cwECE) 20) The evaluation is carried out post temperature scaling (TS) [ as it has been proved to be always beneficial for calibration.
Hyperparameters: In addition to the common hyperparameters for all losses, selected as specified in Appendix C there are also specific ones that need to be tuned for some of them. In this case, we used the best hyperparameter settings reported in the original papers. In cases where the original paper did not report the specific values, carried out validation and selected the setup that provided the best performance on the validation set. We also selected the weighting factor for AUCOCLoss in the same way. We found that optimal weighting values for AUCOCLoss all fell between and 10. We found empirically that models check theck-pointed using ECE provided very poor results. Networks check-pointed using either accuracy or AUCOC provided comparable outcome with respect to accuracy, therefore we reported results on AUCOC as they provided the best overall performance.
OOD experiments: We evaluate the out-of-distribttion (OOD) samples detection performance of all methods since OOD detection is crucial for reliable AI system and it is a common experiment in the network calibration literature. The commonly used experimental setting in this literature is using CIFAR100-C [12] (we report results for CIFAR100 with Gaussian noise in the main paper and the average over all the perturbations in the Appendix) and SVHN [27 as OOD datasets, while the network is trained on CIFAR 100 (in-distribution). We evaluated the OOD detection performance of all methods using Area Under the Receiver Operating Characteristics (AUROC) curve, with MSP and EBM (25
13. ODIN 14. MaxLogit
[1
Finally, in Appendix D we report results for accuracy and AUCOC on imbalanced datasets, being class imbalance present in many domains, such as medical imaging. We report results on the widely used Long-Tailed CIFAR 100 CIFARI00-LTT [37 42] with controllable degrees of data imbalance ratio to control the distribution of training set. We trained with three levels of imbalance ratio, namely 100, 50, 10.
Class imbalance experiments:
Further details on hyper-parameter settings are provided in the the Appendix.
5 Results
First, we report the results for accuracy and COC-related metries. Then, we report results for calibration, even though this is not an explicit goal of this work. Bold results indicate the methods that performed best for each metric, underlined results are the second best. means the higher the better for metric, while the lower the better. The experiments results are averaged over three runs.
7

---

Table 1:
Test results on the natural datasets CIFARI00 and Tiny-Imagenet. We report AUCOC, accuracy for both, To at 90% and 95% accuracy for CIFARI00 and at 65% and 75% accuracy for Tmag-em, as the initial accuracy is also lower. In bold the best result for each metric, underlined the second best. AUCOCL improves, varying degrees, the baselines in all metrics.
Dataset
CIFAR 100
Iiny-ImageNet
7b J @ acc.
acc.
Loss funct.
AUCOC f
Acc.
90%
95%
AUCOC
Ace. f
65%
75%
CE
91,43
75,71
29.03
44,61
72,56
47,39
39,88
$6,29
FL (7=3)
93,91
77.83
24.71
40,48
73,12
47,71
38,40
55,08
AdaFLS3
93,89
77.64
25,08
40.74
73.19
47,81
38,56
55,20
CE+MMCE
92,42
75,35
30,01
44,71
72,47
47.11
39,94
56,70
FL+MMCE
93,90
77.78
25,62
40,90
73,51
48,03
37,83
55,15
CE+S-AvUC
93,99
77,65
24,62
45,00
72,92
47,89
38,28
54,91
FL+ S-AvUC
93,97
77,64
25,97
40,82
74,30
48,69
35,83
53,21
CE+S-ECE
93,88
77.57
24,76
40,14
72,94
47.65
38,81
55,84
FLAS-ECE
93,41
76,69
28,13
43,29
72,61
47,40
39,94
56,71
CE+AUCOCL
94,49
78,94
21,60
36,73
74,56
49,10
34,78
52,01
EL+AUCOCL 94.18
78.31
23,50
36,68
74,30
49,19
34,85
53.15
Table Test results on the medical datasets DermaMNIST and RetinaMNIST. We report AUCOC, accuracy for both, at 90% and 95% accuracy for DermaMNIST and to at 65% and 75% accuracy for RetinaMNIST, as the initial accuracy also lower. In bold the best result for each metric, underlined the second best. AUCOCL improves, to varying degrees, the basclines in all metrics.
Dataset
DermaMNIST
RetinaMNIST
To J. @ acc.
@ acc.
Loss funct.
AUCOC f
Acc.
90%
95%
AUCOC
Ace. f
65%
75%
CE
89,84
71,59
43,51
56,21
71,45
52,10
39,53
68,58
FL (y=3)
90,50
72,64
40,63
53,90
68,57
52,25
44,25
58,25
AdaFL553
90.11
73.10
40,78
55,86
68,85
48,58
48,67
62,00
CE+MMCE
89.71
70.99
45,82
58.07
69.18
48,50
42,92
FLAMMCE
89,34
71.72
46,81
59.10
67,08
50,33
47,50
78,91
CE+S-AvUC
89,67
71.51
43,04
57.09
68,15
51,42
42,71
62,29
FL+ S-AvUC
89,42
71.04
45,47
58,69
66,80
$2,00
45,58
70,54
CE+S-ECE
89,54
71,46
43,48
57,82
71,40
52,05
39,45
55,83
ELE
90,22
72,62
40,95
57,46
70,49
51,33
42,42
79,91
CE+AUCOCL
90.87
74.30
39.01
52,70
72,47
$3,10
38,33
53,81
FLAAUCOCL
91,35
74,80
37,30
53,90
72,31
53,58
39,42
56,12
In Tables I we present our results based on accuracy and AUCOC. The results for TisSueMNIST
be found in the Appendix. We observed that complementing CE and FL with loss, i.e., CE+AUCOCL and FL+AUCOCL, they were consistently better than the other losses in all experi- ments. To evaluate the significance of the accuracy and AUCOC results, we performed permutation test [30 between the best AUCOCLOs and the best baseline with 1000 rounds. In all the cases provided p score 1%, therefore the differences in the models are steadily significant. The advantage of our model was even more apparent in amount of expert loads corresponding to specific accuracy levels. In particular, we measured the percentage of samples delegated expert (7a) at 90% and 95% accuracy for CIFARION DermaMNIST and TissueMNIST, and at 65% and 75% accuracy and RetinaMNIST (as the initial accuracy is also much lower on these datasets). In all the experiments, to varying degrees, AUCOCLoss provided lower delegated samples than the baselines. Noticeably, while some the baselines may be close to the results of AUCOCLoss for certain metrics, none of them were consistently close to AUCOCLOS Loss across all the datasets. For example, when looking at the last two columns for CIFAR 100 and Tmay-emageNet respectively of Table I CE+MMCE is worse by 4% than AUCOCLoss ImageNe,, but by around 9% in
for

---

Table 3: Test results on DermaMNIST and RetinaMNIST for calibration: expected calibration error (ECE), KS score, Brier score and class-wise ECE (cwECE), post TS. In bold and underlined respectively are the best and second best results for each metric. Noticeably, AUCOCL performs comparably to the baselines, even though it does not aim at improving calibration explicitly.
Dataset
DermaMNIST
RetinaMNIST
Loss funct.
ECE!
KSL
Brier
CWECEL
ECE!
KSJ.
Brier
CwECE!
CE
3,07
2,15
38,11
2,22
8,42
6,15
59,75
6,01
FL
4,24
1,77
36,55
2,03
13,61
10,87
63,32
8,19
AdaFL53
3,88
1,95
36,19
1,66
13,63
11,12
65,64
8,47
CE+MMCE
3,59
2,73
38.96
2,20
11,58
9,75
62,18
6,87
FL+MMCE
3.65
2,29
38,91
2,34
12,49
10,54
65.72
7,81
CE+S-AvUC
3,79
2,69
38,28
2,21
8,37
7,01
64,36
6,91
FL+ S-AvUC
3,48
2,13
37,98
1,82
11,60
5,86
64.06
7.12
CE+S-ECE
3,23
2,73
38,39
1,92
9,44
5,88
59,84
5,37
FL+S-ECE
4,5
2,67
36,91
2,08
12,52
10,05
61,16
6,08
CE+AUCOCL
5,70
1,56
36.12
1,78
8,15
4,47
58,69
4,46
FL+AUCOCL
5,10
2,04
35,46
2,04
10,77
8,84
60,52
5,32
Table 4: Test AUROC(%)) OOD detection, training CIFAR100 and testing CIFARI00-C (Gaussian noise) and SVHN, using MSP, ODIN MaxLogit and EBM. Best and second best results in bold and underlined.
Datasct
C100-C AUROCT
SVHN
AUROCT MaxLogit
Loss funct.
MSP
ODIN
MaxLogit
EBM
MSP
ODIN
EBM
CE
74,37
75,22
74,51
66,18
77,42
79,42
67,71
67.71
FL (y=3)
75,12
75,35
73,43
72,83
76.61
77.08
66,69
66,42
AdaPL.53
74,53
74.68
71,29
70,58
80,3
81,28
66,94
66,73
CE+MMCE
74,67
74,53
70,67
70,17
75,79
77.39
66,34
66,23
FLAMMCE
74,42
74,39
70,01
68,70
77.57
77,41
66,97
66,60
CE4S-AvUC
73,63
73.62
72,16
72,05
78,04
78,90
67,93
67,96
FL+ S-AvUC
72.78
76,72
69.53
68,51
79,68
79,68
67.38
67,24
CE+S-ECE
73,29
73,18
71.59
71,36
77.02
77,69
67.98
68,01
FLAS-ECE
74,29
73.97
71.51
70,26
79,68
81,17
67.38
67.78
CEAAUCOCL
76,03
76,90
78,02
78,30
82,03
83,50
69,51
69,69
FL+AUCOCL
76,51
76,82
75,14
74,68
80,51
79,35
69,52
69,46
CIFARI00. Figure La shows examples COC curves. Overall, the plot AUCOCLoss lies above all the baselines, which is a desirable behavior as it corresponds to better operating points.
Even though the proposed loss was not designed to improve calibration, provided on par perfor- mance compared to the other cost functions particularly designed for network calibration, as reported Table 3 for DermaMNIST and RetinaMNIST and in the Appendix for the other datasets.
In OOD experiments reported in Table 4 we used the model trained on CIFAR 100 and evaluated the OOD detection performance on CIFAR 100-C (with Gaussian noise) and SVHN dataset. Results for CIFAR 00-C averaged over all the perturbations are reported in the Appendix. We employed state-of-the-arr OOD detectors, namely MSP, ODIN, MaxLogit and EBM determine whether sample is OOD or in-distribution. The bold results highlight the best results in terms of AUROC. On both OOD datasets, AUCOCLoSs always provided the highest AUROC and in almost all the cases also the second best.
Class imbalance experiments CIFAR100-LT reported in the Appendix. AUCOCLoss obtains best results for both accuracy and AUCOC, with higher benefits at increasing imbalance.
Crucially, CE+AUCOCL, where the proposed loss is used jointly with CE, outperformed every baseline accuracy, AUCOC and To @acc. in all the experiments. It further outperformed all the baselines in OOD detection and class imbalance experiments, presented in the Appendix. Even

---

though the metric is not geared towards calibration, CE+AUCOCL yielded either the best or the second best calibration performance in the majority of the cases compared to the baselines.
In the Appendix, we explore how the performance of a model trained with AUCOCI Loss varies, when changing batch size it can be crucial for KDE-based methods. Even substantially lower batsch sizes do not have a considerable effect on the performance of the proposed method.
Conclusion
In this paper proposed new cost function for multi-class classification that takes into account the trade-off between a neural network's accuracy and the amount of data that requires manual analysis from a domain expert, by maximizing the area under COC (AUCOC) curve. Experiments on multiple computer vision and medical image datasets suggest that our approach improves the other methods in terms of both accuracy and AUCOC, where the latter was expected by design, provides comparable calibration metrics, even though the loss does not aim to improve calibration explicitly and outperforms the baselines in OOD detection.
While we presented COC and AUCOCLoss for multi-class classification, extensions to other tasks possible future work as well investigating different performance metrics embed in the y-axis of COC. Moreover, aware of potential problems with KDE at the boundaries, i.e., boundary bias, we explored corrections like reflection method, which did not provide major improvements, but will further investigate. We believe that this new direction of considering expert load in human-Al system important and AUCOCLoSs will serve as baseline for future work.
Limitations: As described in Section 3.5 AUCOCLoss alone empirically leads to slow convergence, due the small contribution of Loog(nn 2n) with žn E [0, 1] in its formulation. Therefore, recommend to use it as secondary loss, to successfully complement existing cost functions.
7 Acknowledgments
This study was financially supported by: The LOOP Zürich Medical Research Center, Zurich, Switzerland, 2. Personalized Health and Related Technologies (PHRT), project number 222, ETH domain and 3. Clinical Research Priority Program (CRPP) Grant on Artificial Intelligence in Oncological Imaging Network, University of Zürich.
References
[1] Brier, G. W. (1950). Verification of forecasts expressed terms of probability. Monthly Weather Review 78, 1-3.
[2] Chen, L.-C., G. Papandreou, F. Schroff, and H. Adam (2017). Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 077.705000.7777
[3] Davidson, T., D. Warmsley, M. Macy, and Weber (2017). Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM 17. pp. 512-515.
[4] Deng, J., W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei (2009). Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255.
[5] Dvijotham, K., J. Winkens, M. Barsbey, S. Ghaisas, R. Stanforth, N. Pawlowski, P. Strachan, Z. Ahmed, S. Azizi, Y. Bachrach, et al. (2023). Enhancing the reliability and accuracy of ai-enabled diagnosis via complementarity driven deferral to clinicians. Nature Medicine, 1-7.
[6] Esteva, A., B. Kuprel, R. A. Novoa, J. Ko. S. M. Swetter, H. M. Blau, and S. Thrun (2017). Dermatologist-level classification skin cancer with deep neural networks. nature 542(7639), 115-118.
[7] Gorski, N., V. Anisimov, E. Augustin, O. Baret, and S. Maximov (2001). Industrial bank check processing: the a2ia checkreadertm. International Journal on Document Analysis and Recognition 3(4), 196-206.
10

---

[8] Guo, C., G. Pleiss, Y. Sun, and K. Q. Weinberger (2017). On calibration of modern neural networks, In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, 17, pp. 1321-1330. AMLR.org
[9] Gupta, K., A. Rahimi, T. Ajanthan, T. Mensink, C. Sminchisescu, and R. Hartley (2021). Calibra- tion of neural networks using splines. In International Conference on Learning Representations.
[10] He, K., X. Zhang, Ren, and Sun (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778.
[11] Hendryeks, D., S. Basart, M. Mazeika, M. Mostajabi, J. Steinhardt, and D. X. Song (2022). Scaling cut-of-distribution detection for real-world settings. In International Conference on Machine Learning.
[12] Hendrycks, D. and T. Dietterich (2019). Benchmarking neural network robustness to com- mon corruptions and perturbations. Proceedings of the International Conference on Learning Representations.
[13] Hendrycks, D. and K. Gimpel (2016). A baseline for detecting misclassified and out-of- distribution examples in neural networks. arXiv preprint arXiv:1610.021::
[14] Hsu, Y.-C., Y. Shen, H. Jin, and Z. Kira (2020, 06). Generalized odin: Detecting out-of- distribution image without learning from out-of-distribution data. pp- 10948-10957.
[15] Jiang, X., M. Osl, J. Kim, and L. Machado (2012). Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association 19(2), 263-274.
[16] Karandikar, A., N. Cain, D. Tran, B. Lakshminarayanan, J. Shlens, M. C. Mozer, and B. Roelofs (2021). Soft calibration objectives for neural networks. NeurIPS.
[17] Kelly, C. J., A. Karthikesalingam, M. Suleyman, G. Corrado, and D. King (2019). Key challenges for delivering clinical impact with artificial intelligence. BMC medicine 17(1), 1-9.
(18) Krishnan, R. and O. Tickoo (2020). Improving model calibration with accuracy versus uncer- tainty optimization. Advances in Neural Information Processing Systems.
[19] Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical report.
[20] Kull, M., M. Perello-Nieto, M. Kängsepp. T. Filho, Song, and P. Flach (2019). Be- yond Temperature Scaling: Obtaining
Multiclass Probabilities with Dirichlet Calibration. Red Hook, NY, USA: Curran Associates Inc.
[21] Kumar, A., Sarawagi, and U. Jain (2018). Trainable calibration measures for neural networks from kernel mean embeddings. In ICML.
[22] Leibig, C., M. Brehmer, S. Bunk, D. Byng, K. Pinker, and L. Umutlu (2022). Combining the strengths of radiologists and ai for breast cancer screening: retrospective analysis. The Lancet Digital Health 4(7), e507-e519.
[23] Lin, T.-Y., P. Goyal, R. Girshick, K. He, and P. Dollár (2017). Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2999-3007.
[24] Lin, Z., S. Trivedi, and J. Sun (2022). Taking a step back with kcal: Multi-class kernel-based calibration for deep neural networks.
[25] Liu. W., X. Wang. J. D. Owens, and Y. Li (2020). Energy- antoof-distribution detection. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 6-12, 2020, virtual.
[26] Mukhoti, J., Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania (2020). Calibrating deep neural networks using focal loss. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances in Neural Information Processing Systems, Volume 33, pp. 15288-15299. Curran Associates, Inc.
11

---

[27] Netzer, Y., T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng (2011). Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011.
[28] Nixon, J., M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran (2019, June). Measuring calibration in deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
[29] Patel, N.,, L. Rosenberg. G. Willcox, D. Baltaxe, M. Lyons, J. Irvin, P. Rajpurkar, T. Amrhein, R. Gupta, S. Halabi, et al. (2019). Human-machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ digital medicine 2(1), 1-10.
[30] Pesarin, and L. Salmaso (2010, 03). Permutation Tests for Complex Data.
[31] Platt, (2000, 06). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classif. 10.
[32] Quinonero-Candele J., M. Sugiyama, A. Schwaighofer, and N. D. Lawrence (2008). Dataset shift in machine learning. Mit Press
[33] Rajpurkar, P., J. Irvin, R. L.. Ball,
K. Zhu, B. Yang, H. Mehta, Duan, D. Ding, A. Bagul, C. P. Langlotz, al. (2018). Deep learning for chest radiograph diagnosis: retrospective comparison of the chexnext algorithm to practicing radiologists. PLoS medicinee 15(11), e1002686.
[34] Sangalli, S., E. Erdil, A. Hötker, O. Donati, and E. Konukoglu (2021). Constrained optimization train neural networks on critical and under-represented classes. Advances in Neural Information Processing Systems 34, --401-2541..
[35] Scott, D. W. (1979). On optimal and data-based histograms. Biometrika 66(3), 605-610.
[36] Szegedy, C., V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2218 1-28...
[37] Tang, K., J. Huang, and H. Zhang (2020). Long-tailed classification by keeping the good and removing the bad momentum causal effect. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA. Curran Associates Inc.
[38] Yang, J., R. Shi. D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni (2023, 01). Medmnist v2 large-scale lightweight benchmark for 2d and 3d biomedical image classification, Scientific Data 10.
[39] Yin, P., Lyu, S. Zhang. S. J. Osher, Y. Qi, and J. Xin (2019). Understanding straight-through estimator training activation quantized neural nets. In International Conference on Learning Representations.
[40] Zadrozny, B. and C. Elkan (2001). Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, San Francisco, CA. USA, pp. 609-616. Morgan Kaufmann Publishers Inc.
[41] Zagoruyko, S. and N. Komodakis (2016, September). Wide residual networks. In E. R. H. Richard C. Wilson and W. A. P. Smith (Eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1 1-.7.12.. BMVA Press.
[42] Zhou, B., Q Cui, X.S. Wei, and Z.-M. Chen (2020). BBN: Bilateral-brance network with cumulative learning for long-tailed visual recognition. pp. 1-8.
12

---

A Derivations
In this Section we provide the derivation of AUCOC and its gradient formulation.
A.1 Derivation of AUCOC
AUCOC is defined as:
dro f/ Elclr 2 roldTo S.' { E(cr)u(()dr } 1--0
AUCOC=
The y-axis in COC curve expressed mathematically by:
Se' ple, ple, r)dr Sra p(r)dr frs B(cl))((r)dr Sr p(r)dr
Blcr ro] Ep(ch ro)c=
Σ
ple,r 2 200) 200c==
=
plr 2 ro)
Srs p(clr))(r)dr
Sin (p(c)))p(())drr Sr p(r)dr
Σ Srg p(r)dr
S E(clrlp(r)dr
The x-axis in COC curve expressed mathematically by: To ppr ro) J./ To
p(r)dr
Using the To formulation, can rewrite the y-axis
f (lcr)p(()dd
E|c/r 2 rol
Maximizing the area under this curve, over To [0, 1] corresponds to max A max 7.' { B(cl)u(()dr)
dru 1-70 To
Let us assume to use Gaussian kernel with this expression:
K K(Ilr rx 1
(r rT1)²
exp
V2Ta
2a
Developing the equation, the area calculation becomes:
A = /
Elclrpp())dr } dro
1-70
dno
(cc)) Tn 11)dr
1-T0 To
N
1
(r rn)
drg
1(cn)
exp
dr
V2Ta
2a
To
n=1
(10)
(r -
rn
dru
1(cm)
exp
dr
V2TT
2xx
To
n=1
Σ 1(cm)
"ndtr
(L-Tx) Voou
Tw
ndlr
ro In
dm
N
n=1
#thresh
f(TOA-11 f(TO.N. (To To,k-1)
Σ
2
13

---

Where:
rork N
1
70
p(r)dr
Σ
K(llr
rnll)dr=
N
n=l
(11)
N
1
ndtr
ru,k - Fn Voou
-nddrr
nddrr
Tn (Vooe
N
Where ndtr expresses the Gaussian cumulative distribution function and the last row in Equation 10 exploits the trapezoidal rule for integrals computation.
A.2 Derivations of the gradients of AUCOC
d
d ((Eerrrrrr)) d0 {
dro
d0
1-70
Here, we the assumption discussed in Section Methods that To does depend any parameter, thus allowing us to apply Leibnitz's 's integration rule, obtaining:
d0 A ('' d0
dro
Elclrip(r)dr 1-m1
(12)
L'
{
d -/lro/drrr-l/olrrrrrr d0
dra dra
(13)
TO
To can be expressed as:
To p(r S ro) ((" p(r)dr
(14)
Consequently:
dro
dp(r)
drg
dr + p(ro)
d0
do
(15)
dra
So
dp(r) dr do p(ro)
d0
Plugging this expression back into Equation 12 we obtain:
dA
d0 / U d
E(drlp(()) + E[c|ro] do
are dp(r)
d0 dr
dro
(16)
1
Assuming the use of a Gaussian kerne::
exp (- - (r-Tn)²
K([r- - rxll)
(17)
V2Ta
2n
And re-writing:
B(c|rolp(ro) p(ro)
EN 1(c,) rnll)
23
(18)
E K(Irro - rwll)
N
14

---

The gradient of the area becomes:
dA
(/' {1/- drn
Ele L dr
no dp(r)
d7o
dr
dr
1 TO
[ drn v 2 N
(- (r (r(--r))) rn)
Σ ((G,)
-epp
dr+
20x²
E|c/rol L-- To d 1
dr dto
Σ
exp
dra 2 N
20²
TO
n=l
= -/00 1
(r - rmn
(19)
V2ma2N (cn)) ( (r- eexp
dr+
2a²
(- (r rp)
dTa
E|c/ro] V2Ta3N
(r- exp
dr)
2a²
1 TO
= =/o²t
1
(1-rn)
(ro r2n)
{
1(m))
exp
-exp
exp
V2TaN
2a²
2a²
1
(no - r2
(-(=
dto
[[c[ro]
exp
exp
dr)
V2TON
2a²
2a²
TO
Also for the gradients in the code implementation we exploited the trapezoidal rule for the computation of the external integral between [0,1].
B Potential upper bound to AUCOCLOSS
AUCOC has a formulation of the kind -log(En zn.) with 2n E [0, 1]. In contrast, cross-entropy is -En log Zn, where contribution of increasing low z, 's the loss much larger, hence gradient- based optimization is faster. Exploiting Jensen's inequality, one could find the following upper bound of AUCOCLoss to minimise. However, from thorough experiments it has been proven not to be a tight enough bound for the optimisation to be successful. In fact, trying to optimise the rightmost term of the following equation, instead of the correct definition on the left, does not lead to a satisfactory optimisation of AUCOC.
loog (L' U.' ([r rall)dr } dro
x
L/ U. log(r") rall)dr 1-
dra
C Training details
In CIFAR100 experiments, we followed Karandikar et al. [6 and used the network architecture. We trained the models for 200 epochs, using Stochastic Gradient Descent (SGD), with batch of 512, momentum of 0.9 and an initial learning rate of 0. 1, decreased after 60, 120, 160 epochs by factor of 0.1. We set these parameters based on the best validation performance of CE and we keep it for all the losses. In Tiny- Imageet experiments, we used ResNet-50 [IO] as backbone architecture, SGD optimiser with batch size of 512, momentum of 0.9 and base learning rate of 0. divided by 0.1 at 40th and 60th epochs as in Mukhoti et al. (26] In DermaMNIST, RetinaMNIST and MNSTT [8] experiments, we followed the training procedures of the original paper, employing ResNet-50 He et al. [10] Adam optimizer. The batch size is set to 128 for MerST and RetinaMNIST and to 512 for the larger TisSUeMNNIT.. We used the initial learning rate 0.0001 for DermaMNIST and 0.001 for RetinaMNIST and TissueMNIST, and trained the models for 100 epochs by reducing the learning rate by 0.1 after epochs 50 and 75.
41 as
All the models have been trained using either the NVIDIA GeForce RTX 2080 Ti or NVIDIA GeForce RIX 3090. The datasets have been split as follows.
15

---

For CIFARIO0 we used 45000/5000//0000 images respectively as traininggvalidattioneest sets.
Tmay--mageNet is subset of ImageNet with 64 x 64 images and 200 classes. We employed 90000/10000/10000 images as training/validationneesttt set, respectively.
DermaMNIST is composed of dermatoscopic images with 7007/1003/2005 samples for training, validation and test set, respectively, categorised different diseases.
RetinaMNIST is based on the DeepDRiD24 challenge, which provides a dataset of 1,600 retina fundus images. The task is ordinal regression for 5-level grading of diabetic retinopathy severity. Consistently with the original paper, we split the source training set with ratio of 9 into training and validation set, and use the source validation the set.
D Results on a third medical dataset, TissueMNIST
Tables 5 and report results 236,386 human kidney cortex cells, segmented from reference tissue specimens and organized into 8 categories. We split the source dataset with ratio of 7 1 2 into training, validation and test set. The results are consistent with those reported the main paper.
additional medical dataset, TissueMNIST. The dataset contains
Table 5: Test results TissueMNIST for AUCOC and accuracy. The last columns report To corresponding to 90% and 95% accuracy. In bold and underlined respectively the best and second best results for each metric.
4 @ acc.
Loss funct.
AUCOC
Acc.
90%
95%
65,91
83,34
68, 68,60
81,95
FL (7=3)
66,07
82.77
79,41
87.82
AdaFI.53
66,15
83,15
75,01
89,90
CE+MMCE
66,52
84,12
68,81
86,84
FL+MMCR
66,30
82,85
68,96
81,70
CE+S-AvUC
62,42
80.88
73,00
86.03
FL+ S-AvUC
62,75
79.75
74,79
84,74
CE+S-ECE
65,58
83.53
70,67
83.84
FL4S-ECE
65,20
82,59
70,90
84,06
CE+AUCOCL
67,16
84,54
68,60
77,80
FL+AUCOCL
67,04
83,46
67,21
81,55
Table 6: Test results on TissueMNIST for calibration: expected calibration error (ECE), KS score, Brier score and class-wise ECE (cwECE), post TS. In bold and underlined respectively the best and second best results for each metric. Noticeably, AUCOCL performs comparably to the baselines, even though it does not aim at improving calibration explicitly.
Loss funct.
ECEL
KSS.
Brier]
CwECEL
CF
0.89
0,36
46,11
1,65
FL (7=3)
2,06
0,67
46,50
1,58
AdaFL53
2,10
1,11
46,58
1,79
CE+MMCE
0,98
0,53
45,27
1,83
FL+MMCE
1.55
0,57
45.87
1,21
CE+S-AvUC
2.28
1,87
50.32
1.62
FL+ S-AvUC
2,49
1,83
50,99
2,00
CE+S-ECE
2,38
0,98
52,88
2,71
FL+S-ECE
2,45
1,40
47,29
1,83
CE+AUCOCL
1,07
0,71
44,41
1,29
FL+AUCOCL
3,56
0,68
45,54
1,47
16

---

E Class imbalance experiments
In Table 7 we report the results for accuracy and AUCOC the widely employed Long-Tailed CIFARI 100. We employed a tunable data imbalance ratio (Nmax Nmin, where N number of samples in each class), which controls the class distribution in the training set and trained the models with three levels of imbalance ratio, namely 100, 50, 10. Noticeably, when AUCOCLoss complements cross-entropy obtains the best results for both metries all the three settings, with higher benefits at increasing imbalance, while with FL it obtains always the second best AUCOC and comparable accuracy.
Table 7: Accuracy and AUCOC results on CIFARIOO Long-Tailed for 3 degrees of class imbalance the training set, namely 100, 50 and 10. bold and underlined respectively the best and second best results for each metric.
Imbalance ratio
100
50
10
Loss funct.
AUCOC 1
Acc. f
AUCOC
Acc. f
AUCOC
Acc. f
CE FL (y=3)
72,33
47,27
78,02
53,83
89,59
71,15
73,66
47.00
79,53
53,50
91,91
71.59
AdaFLS3
73,72
47.07
79,55
53,55
92,05
71,61
CE+MMCE
72,92
47,80
78,54
53,45
90,66
71,32
FL+MMCE
73,49
46,78
79,50
53,32
91,70
71,71
CE4S-AvUC
73,08
46,62
76,15
54,45
91,72
71,60
FL+ S-AvUC
72,68
46,62
78,51
53,11
89,75
72.11
CE4S-ECE
73,53
47,58
79,31
53,84
91,63
71,67
FLAS-ECE
72,30
46,13
78,56
52,84
91,29
70,6
CE+AUCOCL
75,85
49,63
81,38
55,89
92,59
72,40
FLAUCCOL
74.51
46,91
79.72
53,64
92,21
71,92
Calibration results for Tiny-ImageNet and CIFAR100
In T 8 we report calibration results respectively for CIFAR 100 and Tiny-ImageNet. For all the metries and datasets, AUCOCLoss provides comparable results with respect the baselines.
Table 8: Test results CIFARIO0 100 and Tiny-Imagenet for calibration metrics, namely expected calibration error (ECE), KS score, Brier score and class-wise ECE (cwECE), post temperature scaling. In bold and underlined respectively the best and second best results for each metric. Noticeably, AUCOCL performs comparably to the baselines, even though does not aim at improving calibration explicitly.
Dataset
CIFARI00
Imageee
Loss funct.
ECE|
KSJ.
Brier.
cwECEJ
ECE,
KSJ.
Brier
cwECEJ
CE
2,41
1,1
33,91
0,211
1,54
0,74
66,25
0,163
FL (7=3)
1,92
1,46
31.16
0,184
1,46
1,14
65.91
0,152
AdaFL53
1,47
1,22
31,3
0,183
1,35
0.85
65,71
0,159
CE+MMCE
2,32
1,23
34,6
0,209
1,79
0,93
66,42
0,157
FL+MMCE
1,92
1.82
31,24
0,181
1,94
1,63
65,49
0,151
CE+S-AvUC
3,23
0.53
31.71
0,199
2,13
1,15
65.20
0,146
FL. S-AvUC
1,83
0,53
31,74
0,177
1,51
1,13
64,62
0,15
CE+S-ECE
3,85
1,3
31,85
0,197
1,66
0.81
66,
0,15
FLAS-ECE
1,74
1,56
32,19
0,180
2,6
2,37
66,40
0,161
CE+AUCOCL
1,65
0,75
29.78
0,184
1.65
1,29
64,23
0,157
FL+AUCOCL
1,34
0.95
30,24
0,175
1.86
1.29
64,45
0,155

---

G Results for varying batch size
In Table 9 we report the results on DermaMNIST with reduced batch sizeon which KDE is applied i.c. 64 and 28 samples per batch. The results of AUCOCLoss not significantly affected by it, when complementing cross-entropy and focal loss.
Table 9: Test results on DermaMNIST for accuracy, AUCOC and ECE for batch sizes 64 and 32. Reducing the batch size does not affect significantly the performance of AUCOCI.
Batch
Loss funct.
AUCOC f
Acc. t
ECE
CE+AUCOCL
91,16
75,18
11,00
64
FL+AUCOCL
91,13
74,71
9,35
CE+AUCOCL
90,14
73,80
15,50
32
FL+AUCOCL
91,10
73,47
6,69
H Additional results for CIFAR100 and Immy-Ima
In Table 10 we report test results on CIFAR100 using ResNet-50 and Tiny-Imagenet using WideResnet--28- 10, run for one seed. Best and second best results are in bold and underlined. They show consistency with the main experiments reported in the paper.
Table 10: Test results on CIFAR100 using ResNet-50 and for one seed. Best and second best results are in bold and underlined.
using WideResnet-28-11 10.
Dataset
CIFAR100
Tiny- ImageetaeNet
J. @ acc.
@ acc.
Loss funct.
AUCOC f
Acc.
90%
95%
AUCOC
Acc.
65%
75%
CE
91,10
73,69
34,72
49,42
74,03
49,29
34,74
52,28
FL (v=3)
92,68
75,13
32,30
46,40
75,05
49,56
34,17
51,72
AdaFL53
92,65
74.5
32,01
45.78
75.36
49,49
33,77
50.76
CE+MMCE
92,55
74.75
31,08
44.53
74.12
48,56
35,83
52,48
FL+MMCE
92,46
74.47
31.69
46,63
74.79
49,32
34.57
52,12
CE+S-AvUC
92,63
74,12
31,02
44,83
74,14
49,13
35,00
53,17
TL+S-AVUC
92,71
74,82
31,11
44,51
74,21
49,02
35,22
53,12
CE+S-ECE
92,57
74,27
31,38
45,4
75,62
49,91
31,88
49,55
FL4S-ECE
92,63
75,00
31,35
47,42
74,57
49,16
35,60
52,36
CE+AUCOCL
93,81
75,94
28,39
42,61
76,64
51,40
29,97
47,65
FL+AUCOCL
93.19
75,90
29,45
43,11
75,86
50,84
32,00
50,12
I Further explanation on how improve AUCOC
There are two factors which contribute to an increase in AUCOC: decrease the number of samples delegated to human experts (given the same network accuracy) and increase in the accuracy for the samples that are not delegated but analysed only by the network (given the same human workload).
These two aspects could manifest either individually, if the AUCOC improvement is generated by just shift "up" or "left" of COC, or in a combined way. The example provided in Figure Ia shows an improvement in both axes ("and" case) and the proposed loss function does not favour one specific behaviour. Figure providess an example of shifts "up" and "left" ("or" cases). From the AUCOC metrics alone, it is not possible to infer which mechanism is taking place.
18

---

COC Curves
1.0
Network accurscy
0.6
0.2
0.51)
COC Shift "up" (AUCOC 0.711
0.0
0.0
0.2
0.4
0.8
1.0
Samp los be manuslly analysed
Figure 2: Toy example of possible AUCOC increase by shifting "up" or "left" the blue COC curve.
OOD results on all the perturbations on CIFAR100-C
In table I report the average results on CIFAR100-C for all the perturbations available from Hendrycks and Dietterich [2].
Table 11: Test AUROC(%) on OOD detection, training CIFARIO6 100 and testing CIFAR100-C (all 15 perturbations) for MSP, ODIN, MaxLogit and EBM. Best and second best results are bold and underlined.
Dataset Loss funct.
C100-c all corruptions AUROCI
MSP
ODIN
MaxLogit
EBM
CE 66,18
66,42
67.71
67.71
FL (y=3)
66,57
66,36
66,69
66,42
AdaFL53
66,58
66,33
66,94
66,73
CE+MMCE
66,12
66,22
66,34
66,23
FLAMMCE
66,92
66,14
66,97
66,60
CE+S-AvUC
67,29
67.45
67,93
67,96
FL+ S-AvUC
66,78
66,00
67,38
67,24
CE+S-ECE
67,32
67.50
67,98
68,01
FLAS-ECE
67,34
65,38
67,38
67,78
CE+AUCOCI.
67,77
67,77
69,51
69,69
FL+AUCOCL
68,00
66,83
69,52
69,46
19


Reviews:
[{"review_id": "6QM7Ht8VUy", "reviewer": "Reviewer_Bcd1", "summary": "This paper proposes a new loss, AUCOC loss, to improve the networks accuracy and prediction confidence. The loss aims to reduce the number of errors made by the algorithm and thus also the number of delegated samples to domain experts.  The proposed loss focuses on maximizing the area under the COC curve during training in a differentiable manner. The AUCOC loss is used complementary to the original network’s training loss, resulting in increased classification accuracy, better OOD sample detections, and on par calibration performance.", "questions": "See Weaknesses.\n\nIt would be interesting to see if the proposed AUCOC loss can influence the CNN prediction behavior to be more in line with humans resulting in a lower need for experts.", "limitations": "The paper addressed the limitations of why the AUCOC loss is rather used in conjunction with classical CNN loss functions, like cross-entropy. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The usage of the ''area under the confidence operating characteristics'' curve as an additional loss (being differentiable) is a novel idea. \n\nThe paper is well written, the core method is well explained and the experimental results back the claims of the paper.\n\nThe proposed AUCOC loss is outperforming other loss functions.\n\nThe possible weakness of only using AUCOC loss is clearly described.\n", "weaknesses": "The title is somewhat misleading. \n\nA clear reference of which CNNs are used for the experiments is needed, especially since different CNNs were used for the experiments. \n\nHow does ResNet-50 behave for CIFAR100 using the AUCOC loss and how does Wide-Resnet-28-10 behave on TinyImageNet? Was the latter Wide-Resnet also used for table 4? \n\nWhy was only the corruption type Gaussian noise evaluated for the CIFAR 100-C experiments? \n\nThe paper wants to minimize the expert load and to be able to predict OOD samples correctly. So far, the paper shows that the accuracy increases and the amount of delegated samples decreases in the OOD setting. While in its current state, the paper has important contributions, it would be interesting to explore if the proposed method also influences the prediction behavior of the CNN. [1] analyses the prediction behavior of CNNs and finds a shape-bias cue conflict: CNNs tend to recognize texture rather than shape, which is in contrast to human vision behavior, which recognizes objects based on their shapes. Therefore, an obvious step would be to investigate how the networks predict objects (i.e., shape or texture), and if the AUCOC can also influence that behavior. \n\nLine 149-153: This or/and is confusing. When does a higher AUCOC indicate lower number of samples delegated to humans *and* also a higher accuracy, and when is it one of these results? \n\nTypos: \n\nReferences to Eq. 3 are sometimes misleading, while a reference to the Eq. in line 153 would sometimes be more appropriate. \n\nLine 173: “estimate”: choose\n\nLine 178: Leibniz integral rule\n\n[1] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR 2019\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the proposed work and helping us improve the explanations. We are happy to address the raised doubts and questions.\n\n\n# Comment about employed architectures and additional experiments on CIFAR100 and Tiny-ImageNet.\n\nWe thank the reviewer for pointing this out. Due to space reasons, we provided information on which architectures have been used in Appendix 7. To be consistent with the baseline [1], we used Wide-Resnet-28-10 for CIFAR100 (OOD in Table 4 as well) and Resnet-50 for Tiny-Imagenet. Following the suggestion of the reviewer, we report results on CIFAR100 with Resnet-50 and on Tiny-Imagenet with Wide-Resnet-28-10 in Table 1 of the PDF uploaded in the \"global\" rebuttal response. Consistently with the results in the paper, AUCOCLoss is better than all the baselines in terms of accuracy and AUCOC, and it provides lower delegated samples.\n\n[1] Karandikar, A., N. Cain, D. Tran, B. Lakshminarayanan, J. Shlens, M. C. Mozer, and B. Roelofs (2021). Soft calibration objectives for neural networks. In NeurIPS.\n\n# Comment about OOD setup.\n\nWe evaluated the proposed method on OOD, among the various tasks. Consistently with our baseline [2], we provided an example of stronger dataset shift with SVHN and a weaker one with CIFAR-C under Gaussian noise. Following the reviewer's suggestion, we report the average AUROC(%) on all the 15 corruptions provided in [3] on CIFAR-C. Results are presented in Table 2 of the PDF uploaded in the \"general\" rebuttal response. Please note that we explicitly referred to the two detectors already employed in the main paper with their commonly used names, i.e., MSP and ODIN. Following the suggestion of R3, we added MaxLogit and EBM to further consolidate the presented findings. The new results are consistent with those presented in the paper, i.e. AUCOCLoss provides the best OOD detection performance for all the experiments and, in almost all the cases, also the second-best.\n\nWe would like to highlight that the current method does not explicitly enforce networks to focus on \"human-like\" learning, e.g., focusing more on shapes rather than textures, as we are neither directly acting on representation learning, nor providing human feedback to the network during training, therefore we do not expect at the moment an improvement towards this end. However, it is definitely an interesting future investigation, to reduce the need of the expert analysis in a human-AI system. \n\n\n[2] Mukhoti, J., V. Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania (2020). Calibrating deep neural networks using focal loss. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances in Neural Information Processing Systems, Volume 33, pp. 15288–15299. Curran Associates, Inc.\\\n[3] Hendrycks, D. and T. Dietterich (2019). Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations\n\n# Comment about AUCOC improvement explanation (lines 149-153).\n\nWe thank the reviewer for drawing attention to it, thus helping clarify the explanation. In lines 149-153 we aim to explain what an improvement, i.e. increase, in AUCOC could practically correspond to. In that paragraph, we wanted to clarify that there are two factors which contribute to an increase in AUCOC: decrease in the number of samples delegated to human experts (given the same network accuracy) and increase in the accuracy for the samples that are not delegated but analysed only by the network (given the same human workload).\n\nThese two aspects could manifest either individually, if the AUCOC improvement is generated by just a shift \"up\" or \"left\" of COC, or in a combined way. Hence the \"or/and\". The example provided in Figure 1a of the paper shows an improvement in both axes (\"and\" case) and the proposed loss function does not favour one specific behaviour. Figure 1 in the PDF uploaded in the \"global\" rebuttal response provides an example of  shifts \"up\" and \"left\" (\"or\" cases). From the AUCOC metrics alone, it is not possible to infer which mechanism is taking place."}, {"review_id": "AAvfB3fg4R", "reviewer": "Reviewer_1tYR", "summary": "The paper aims to address the trade-off between model accuracy and model confidence results. The authors propose a novel loss function called AUCOC, which maximizes the area under the confidence operating characteristic curve. They evaluate the performance of their approach on various image classification and medical image classification datasets.", "questions": "See cons above", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The problem being addressed is of significant importance and has been widely studied.\n\n", "weaknesses": "1. In terms of empirical evaluation, the authors only assess the performance on image classification datasets. As this is primarily an empirical paper, it would be beneficial to see results on different types of datasets, including text data, tabular data, and other diverse datasets. Moreover, I am curious to know how the proposed model performs on larger-scale datasets, as the ones evaluated in the paper (e.g., CIFAR100 and Tiny-ImageNet) may not be sufficiently large-scale.\n\n2. Furthermore, when it comes to out-of-distribution (OOD) detection results, it would be valuable for the authors to compare their proposed methods with commonly used approaches such as MSP, MaxLogit, and others.\n\nThe proposed method is commendably simple and straightforward. However, I believe that a more comprehensive evaluation is necessary to demonstrate the effectiveness of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the importance of the task and appreciating the simplicity of the proposed method. We are happy to address the raised concerns.\n\n# Comment about additional datasets.\n\nWe would like to point out that the presented article is not primarily an empirical study, as the reviewer suggests. Considerable modelling effort is required to formulate the loss, which is a novel contribution that is appreciated by R1 and R4. We aimed for a clear and easy explanation, and, to this end, we placed all lengthy derivations in the Appendix.\n\nWhile we appreciate that demonstrations on diverse types of data can be beneficial, there are two main reasons why we refrain from this: (i) the aim here is to present a novel loss and evaluated it consistently with experiments from the baselines [1, 2] and (ii) imaging data is one of the most challenging data types used in many different ML articles. Therefore, we chose to focus on explaining the loss well and demonstrate its different aspects on well established data sets.\n\nLarger-scale datasets are indeed very interesting, however, resource and energy consumption of experimenting with such datasets at the breadth we present - which requires training from scratch and assessing multiple aspects of the model with different datasets - is extremely high. We report some initial results on ImageNet here to address the reviewer's concerns. Results are summarised in Table 3 of the PDF uploaded in the \"global\" rebuttal response, running AUCOCLoss and the cross-entropy baseline for 15 epochs with ResNet-50. Noticeably, compared to the baseline, AUCOCLoss shows favourable preliminary results in all the metrics.\n\n\n\n[1] Karandikar, A., N. Cain, D. Tran, B. Lakshminarayanan, J. Shlens, M. C. Mozer, and B. Roelofs (2021). Soft calibration objectives for neural networks. In NeurIPS.\\\n[2] Mukhoti, J., V. Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania (2020). Calibrating deep neural networks using focal loss. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances in Neural Information Processing Systems, Volume 33, pp. 15288–15299. Curran Associates, Inc.\n\n# Comment about additional OOD experiments.\n\nWe would like to highlight that the detectors provided in the main paper are already the commonly used approaches: AUROC MSP [3] (pre temperature scaling) and ODIN (post temperature scaling) [4], even though we did not mention the original names explicitly, but only described in lines 275-277. This was also adopted by the baseline [5]. We thank the reviewer for pointing this out, and we will add the names to the paper to make it clearer and more explicit.\n\nFollowing the reviewer's suggestion, in addition we report the results with two other OOD detectors, MaxLogit [6] and EBM [7], for SVHN, CIFAR-C under Gaussian noise and additionally CIFAR-C under all the 15 corruptions from [8]. Results are summarised in Table 2 of the PDF uploaded in the \"global\" rebuttal response. The new results are consistent with those presented in the paper i.e., for every detector and dataset, AUCOCLoss provides the best OOD detection performance and, in almost all the cases, also the second-best.\n\n[3] Hendrycks, D., & Gimpel, K. (2016). A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR.\\\n[4] Hsu, Yen-Chang, Yilin Shen, Hongxia Jin and Zsolt Kira. “Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 10948-10957.\\\n[5] Mukhoti, J., V. Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania (2020). Calibrating deep neural networks using focal loss. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances in Neural Information Processing Systems, Volume 33, pp. 15288–15299. Curran Associates, Inc.\\\n[6] Hendrycks, D., Basart, S., Mazeika, M., Mostajabi, M., Steinhardt, J., & Song, D. (2022). Scaling out-of-distribution detection for real-world settings. ICML.\\\n[7] Liu, W., Wang, X., Owens, J., & Li, Y. (2020). Energy-based out-of-distribution detection. NeurIPS.\\\n[8] Hendrycks, D. and T. Dietterich (2019). Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations"}, {"review_id": "Tlas3U97mx", "reviewer": "Reviewer_eeZS", "summary": "The authors present a \"confidence operating characteristic\" curve to represent tradeoff between accuracy and numbers of samples delegated to human experts. To maximize the area under this curve, the authors propose a new loss. The authors run classification experiments on computer vision and medical image datasets.", "questions": "See weaknesses above.", "limitations": "See weaknesses above.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper tackles a problem that is clearly important -- namely, minimizing the human effort needed in a human-in-the-loop system and quantifying the \"human effort\" portion. However, see weaknesses section below.", "weaknesses": "- The references list looks reasonable (35 references), but for whatever reason, the related works section seems very sparse and lacking in relevant bodies of work. Upon closer inspection, it looks like a significant portion of those references (10+) are not related, just part of the motivation. This work is effectively \"active learning,\" which is conspicuously absent from the paper entirely. I'm not super familiar with active learning, but no doubt there is previous work that already tackles the problems introduced here -- and it's not clear how this paper is different. For example, here's a highly cited paper in the area: \"Cost-Effective Active Learning for Deep Image Classification\" 2017, https://arxiv.org/abs/1701.03551. The paper here proposes a metric that minimizes annotation cost -- in particular, they ask humans to annotate highly-unconfident samples. This seems like a reasonable metric and it's not clear why end-to-end minimizing cost is better, for example. (I'm sure it could be justified) Granted, the setup is slightly different, but I'm sure a more thorough search will yield more relevant papers.\n- The paper doesn't cite previous metrics for \"human effort\", how they are fallible, and how the proposed metric fixes that problem. As a result, it's not clear why this metric is preferred to other variants.\n- Experiments are performed on MNIST variants, CIFAR100 and TinyImageNet, but do these results extend to ImageNet for example? A full ImageNet run isn't needed; just a few epochs showing that results are trending positively for your method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Comment about active learning and related work.\n\nWe believe there is a misunderstanding, as the proposed paper is not performing active learning. Active learning and our work have two inherently different goals. Active learning aims to include the human expert in the loop **during training**. In contrast, our work focuses on reducing the load on human expert **at deployment**, i.e., when the model is used for inference. In many applications, e.g., medical imaging, to meet high accuracy requirements, samples for which a model is not certain are delegated to experts. Our goal is to find training losses that will take into account this delegation.\n\nConsequently, our study has primarily concentrated on a different literature than active learning, namely confidence calibration, that also explicitly focuses on predictive confidences and ultimately aims to enhance human-AI interaction. We outlined the most recent and relevant methodologies in this realm. Moreover, our investigation explores the allocation of samples for human analysis during deployment. This quantity is a recognised metric, and analogous curves resembling COC have been previously employed, as in [1]. In response to the reviewer's suggestion, to further improve our work, we will incorporate supplementary references to exemplar applications that utilise akin metrics [2, 3, 4].\n\nTo avoid any future misunderstandings or doubts, we will also add a discussion to the main paper where we explicitly state the differences with active learning.\n\n# Comment about \"human effort\" metrics.\n\nTo the best of our understanding of this comment, we believe this is related to the misunderstanding about the relation between this work to active learning. In this work we are only interested in the number of samples delegated to a human expert for analysis **at deployment time**. This number is already an established metric and curves similar to COC have been used before [1]. Following the reviewer's suggestion, we can add additional citations to example applications using similar quantities [2,3,4]. However, to the best of our knowledge, this is the first work that incorporates AUCOC in a loss function in a differentiable way. We acknowledge that in active learning, \"human effort\" may be quantified in different ways, but this work is fundamentally different than active learning, as we discuss in detail as response to the previous concern of the reviewer.\n\n[1] Gorski, N., V. Anisimov, E. Augustin, O. Baret, and S. Maximov (2001). Industrial bank check processing: the a2ia checkreadertm. International Journal on Document Analysis and Recognition 3(4), 196–206.\\\n[2] Dvijotham, K.(., Winkens, J., Barsbey, M. et al. Enhancing the reliability and accuracy of AI-enabled diagnosis via complementarity-driven deferral to clinicians. Nat Med 29, 1814–1820 (2023). \\\n[3] Leibig, C. et al. Combining the strengths of radiologists and AI for breast cancer screening: a retrospective analysis. Lancet Digit. Health 4, e507–e519 (2022).\\\n[4] Hendrycks, D. & Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of International Conference on Learning Representations (ICLR) (OpenReview.net, 2017).\n\n# ImageNet preliminary results.\n\nLarger-scale datasets are indeed very interesting. However, resource and energy consumption of experimenting with such datasets at the breadth we present - which requires training from scratch and assessing multiple aspects of the model with different data sets - is extremely high. Following the reviewer's suggestion, we report preliminary results on ImageNet in Table 3 of the PDF uploaded in the \"global\" rebuttal response, running AUCOCLoss and the cross-entropy baseline for 15 epochs with ResNet-50. Noticeably, compared to the baseline, AUCOCLoss shows favourable preliminary results."}, {"review_id": "UbR4OSgX3C", "reviewer": "Reviewer_jAPT", "summary": "This work supposes a real-world setting where misclassified examples are reviewed post-hoc by human experts, and offers a near-optimal trade-off between such examples (the expert load) and classifier accuracy. The authors propose to use a curve of confidence versus expert load, using the latter as a sliding scale to arrive at that optimum. They propose a no-binning method that kernelises the said (CFOC) curve, and arrive at a formulation to minimize the expert load value irrespective of the network parameters so that the formulation may be added to the error minimisation loop. \n\nExtensive experiments are performed, such as those on OOD detection where overconfidence is significant, and imbalance, where traditional OC curves are less meaningful because the underlying metrics are too. \n", "questions": "Equation 3 does not mention how AUCOC is determined. It just computes the AUCOC Loss using the former. Please let me know in my assuming it is determined in the same way as the ECE does, i.e.  accuracy vs. confidence, as is strongly evident from the presence of bins? How do you define confidence? \n", "limitations": "yes \n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The idea has a high likelihood of orginality and novelty, to the extent that benchmarking has had to be done with the COC-derived-loss-augmented CE and all sorts of CE-derived losses, and not competing methods. The idea is developed in a simple manner that persuades the reader's logic. Experimentation is fairly complete, even though a discussion of mixture ratios of CE and AUCOC- losses was expected too. \n", "weaknesses": "Studying the dynamic of mixing two losses that have such variable gradient magnitude properties on the rate and preciseness of convergence is something I'd have done. \n\nThe $\\tau$ being the bandwidth of the kernel, seems to be extraordinarily large at both levels in table 1. I'm not sure if I'd be offloading up to a third of testing examples to the expert. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the proposed method and we are happy to answer to the raised questions.\n\nAUCOCLoss (Equation 3), exploits the definition of AUCOC (equation in Section 3.2.2). Specifically, to build a differentiable loss out of this definition, we employ KDE to define E[c|r]p(r) and tau0 as explained in Section 3.3.1.\nThe metrics AUCOC instead, as it does not need to be differentiable, is just built in the same fashion as classic AUROC, i.e. as explained in lines 215-224.\n\nRegarding the study about mixing the two losses, it is definitely an interesting point. We tried various weighting factors and found out that favourable weighting factors for AUCOCLoss fell all between 1 and 10, without no unified preferred value in this range, but with consistent improvement over the baselines."}]

Author rebuttal:
We thank the reviewers for taking the time to read our paper and for providing insightful feedback and input.

In each rebuttal we addressed the individual concerns of the reviewers and we are happy to respond to any additional doubts or questions.
In the PDF uploaded in this "general" response, we provide the following additional results and clarifying figures, following reviewers' suggestions:
- Additional experiments on CIFAR100 and Tiny-Imagenet;
- Additional OOD experiments, with new OOD detectors and more  CIFAR-C corruptions;
- Preliminary ImageNet results;
- A toy example of AUCOC improvement.

Meta-review: This paper assumes a setting where misclassified examples are reviewed by human experts for model analysis at deployment time, and attempts to minimize the human expert load. It proposes an AUCOC loss to improve the networks accuracy and prediction confidence, where the AUCOC loss aims to reduce he number of samples to be reviewed by to domain experts. The proposed loss maximizes the area under the COC curve. Used in addition to the respective original training loss, AUCOC can provide increased classification accuracy, OOD sample detections, and on par calibration performance.
While the paper was generally well received by the reviewers, there was some initial criticism for example due to the experimental evaluation. This was however addressed during the rebuttal phase (including the concerns of reviewer 1tYR w.r.t. the comparison to existing methods) so that after the rebuttal and discussion three out of four reviewers are in favor of accepting the paper. The proposed approach is novel and interesting and the paper is well written. The authors are encouraged to include the feedback from the rebuttal phase into the final version of their submission.
Decision: Accept (poster)