Title: Expert load matters: operating networks at high accuracy and low manual effort
Venue: NeurIPS 2023
Accepted: yes
Keywords: human-ai collaboration system

optimization
Abstract: In human-AI collaboration systems for critical applications, in order to ensure minimal error, users should set an operating point based on model confidence to determine when the decision should be delegated to human experts. 
Samples for which model confidence is lower than the operating point would be manually analysed by experts to avoid mistakes.
Such systems can become truly useful only if they consider two aspects: models should be confident only for samples for which they are accurate, and the number of samples delegated to experts should be minimized.
The latter aspect is especially crucial for applications where available expert time is limited and expensive, such as healthcare. 

The trade-off between the model accuracy and the number of samples delegated to experts can be represented by a curve that is similar to an ROC curve, which we refer to as confidence operating characteristic (COC) curve. 
In this paper, we argue that deep neural networks should be trained by taking into account both accuracy and expert load and, to that end, propose a new complementary loss function for classification that maximizes the area under this COC curve.
This promotes simultaneously the increase in network accuracy and the reduction in number of samples delegated to humans.
We perform experiments on multiple computer vision and medical image datasets for classification.
Our results demonstrate that the proposed loss improves classification accuracy and delegates less number of decisions to experts, achieves better out-of-distribution samples detection and on par calibration performance compared to existing loss functions.

Expert load matters: operating networks at high accuracy and low manual effort

Abstract

In human-AI collaboration systems for critical applications, in order to ensure minimal error, users should set an operating point based on model confidence to determine when the decision should be delegated to human experts. Samples for which model confidence is lower than the operating point would be manually analysed by experts to avoid mistakes. Such systems can become truly useful only if they consider two aspects: models should be confident only for samples for which they are accurate, and the number of samples delegated to experts should be minimized. The latter aspect is especially crucial for applications where available expert time is limited and expensive, such as healthcare. The trade-off between the model accuracy and the number of samples delegated to experts can be represented by a curve that is similar to an ROC curve, which we refer to as confidence operating characteristic (COC) curve. In this paper, we argue that deep neural networks should be trained by taking into account both accuracy and expert load and, to that end, propose a new complementary loss function for classification that maximizes the area under this COC curve . This promotes simultaneously the increase in network accuracy and the reduction in number of samples delegated to humans. We perform experiments on multiple computer vision and medical image datasets for classification. Our results demonstrate that the proposed loss improves classification accuracy and delegates less number of decisions to experts, achieves better out-of-distribution samples detection and on par calibration performance compared to existing loss functions. 1

1 Introduction

Artificial intelligence (AI) systems based on deep neural networks have achieved state-of-the-art results by reaching or even surpassing human-level performance in many predictive tasks [6; 33; 2; 36]. Despite the great potential of neural networks for automation, there are pitfalls when using them in a fully automated setting, especially pertinent for safety-critical applications, such as healthcare [17; 32; 34]. Human-AI collaboration aims at remedying such issues by keeping humans in the loop and building systems that take advantage of both [29]. An example of such human-AI collaboration is hate speech detection for social media [3], where neural networks could reduce the load of manual analysis of contents required by humans. Healthcare is another relevant application [5; 22], . For

Ertunc Erdil

Computer Vision Lab ETH Zürich

example, a neural network trained to predict whether a lesion is benign or malignant should leave the decision to human doctors if it is likely to make an error [15]. The doctors' domain knowledge and experience could be exploited to assess such, possibly, ambiguous cases and avoid mistakes.

A simple way of building collaboration between a network and a human expert is delegating the decisions to the expert when the network's confidence score for a prediction is lower than a threshold, which we refer to as 'operating point". It is clear that the choice of the operating point can only be done through a trade-off between the network performance on the automatically analysed samples, i.e., the number of errors expected by the algorithm, and the number of delegated samples, i.e., experts' workload. The latter is crucial especially for applications where expert time is limited and expensive. For example, in medical imaging, the interpretation of more complex data requires clinical expertise and the number of available experts is limited, especially in low income countries [17]. Hence, predictive models that can analyse a large portion of the samples at high accuracy and identify the few samples that should be delegated to human experts would naturally be more useful with respect to this trade-off.

It is possible to evaluate the performance of a predictive model taking simultaneously into account the accuracy and the number of samples that requires manual assessment from a human expert with a performance curve reminiscent of Receiver Operating Characteristic (ROC) curves, as illustrated in Fig. 1a. We will refer to this performance curve as Confidence Operating Characteristics (COC) as it is similar to the ROC curve. A COC curve plots for a varying threshold on algorithm confidence, i.e., operating point, the accuracy of a model on the samples on which the algorithm is more confident than the threshold versus the number of samples remaining below the threshold. The former corresponds to the accuracy of the model on automatically analysed samples while the latter corresponds to the amount of data delegated to the human expert for analysis. In an ROC curve a balance is sought after between Sensitivity and Specificity of a predictive model, while a COC curve can be used by domain experts, such as doctors, to identify the most suitable balance between the accuracy on the samples that are automatically analysed and the amount of data delegated to be re-examined by a human for the specific task. Variations of this curve have been used to evaluate the performance of automated industrial systems [7].

In this paper, we focus on the trade-off between model accuracy and the amount of samples delegated to domain experts based on operating points on model confidence. Specifically, our goal is to obtain better trade-off conditions to improve the interaction between the AI and the human expert. To this end, we propose a new loss function for multi-class classification, that takes into account both of the aspects by maximizing the area under COC (AUCOC) curve. This enforces the simultaneous increase in neural network accuracy on the samples not analysed by the expert and the reduction in human workload. To the best of our knowledge, this is the first paper to include the optimization of such curve during the training of a neural network, formulating it in a differentiable way. We perform experiments on two computer vision and three medical image datasets for multi-class class classification. We compare the proposed complementary AUCOC loss with the conventional loss functions for training neural networks as well as network calibration methods. The results demonstrate that our loss function complements other losses and improved both accuracy and AUCOC. Additionally, we evaluate network calibration and out-of-distribution (OOD) samples detection performance of networks trained with different losses. The proposed approach was also able to consistently achieve better OOD samples detection and on par network calibration performance.

2 Related Work

For the performance analysis of human-AI collaborative systems, confidence operating characteristics (COC) curves can be employed, which plot network accuracy on accepted samples against manual workload of a human expert, e.g as in [7]. While such curves have been used, to the best of our knowledge, we present the first work that defines a differentiable loss based on COC curve, in order to optimize neural networks to take into account simultaneously accuracy and experts' load for a human-AI collaborative system. Thus, there is no direct literature with which we can compare.

The underlying assumption in deciding which sample to delegate to human expert based on a threshold on confidence scores is that these scores provided by deep learning models indicate how much the predictions are likely to be correct or incorrect. However, the final softmax layer of a network does not necessarily provide real probabilities of correct class assignments. In fact, modern deep neural

networks that achieve state-of-the-art results are known to be overconfident even in their wrong predictions. This leads to networks that are not well-calibrated, i.e., the confidence scores do not properly indicate the likelihood of the correctness of the predictions [8]. Network calibration methods mitigate this problem by calibrating the output confidence scores of the model, making the above mentioned assumption hold true. Thus, we believe that the literature on network calibration methods is the closest to our setting because they also aim at improving the interaction between human and AI, by enforcing correlation between network confidence and accuracy, so that confidence can be used to separate samples where networks' predictions are not reliable and should be delegated to human experts. Calibration methods aim to get models that are highly accurate in the samples they are confident, but not the problem of minimising the number of samples delegated to human experts, contrary to the loss proposed here.

Guo et al. [8] defines the calibration error as the difference in expectation between accuracy and confidence in each confidence bin. One category of calibration methods augments or replaces the conventional training losses with another loss to explicitly encourage reducing the calibration error. Kumar et al. [21] propose the MMCE loss by replacing the bins with kernels to obtain a continuous distribution and a differentiable measure of calibration. Karandikar et al. [16] propose two losses for calibration, called Soft-AvUC and Soft-ECE, by replacing the hard confidence thresholding in AvUC [18] and binning in ECE [8] with smooth functions, respectively. All these three functions are used as a secondary loss along with conventional losses such as cross-entropy. Mukhoti et al. [26] find that Focal Loss (FL) [23] provides inherently more calibrated models, even if it was not originally designed for this, as it adds implicit weight regularisation. The authors further propose Adaptive Focal Loss (AdaFL) with a sample-dependent schedule for the choice of the hyperparameter γ . The second category of methods are post-hoc calibration approaches, which rescale model predictions after training. Platt scaling [31] and histogram binning [40] fall into this class. Temperature scaling (TS) [8] is the most popular approach of this group. TS scales the logits of a neural network, dividing them by a positive scalar, such that they do not saturate after the subsequent softmax activation. TS can be used as a complementary method and it does not affect model accuracy, while significantly improving calibration. A recent work by Gupta et al. [9] fits a spline function to the empirical cumulative distribution to re-calibrate post-hoc the network outputs. They also present a binning-free calibration measure inspired by the Kolmogorov-Smirnov (KS) statistical test. Lin et al. [24] propose a Kernel-based method on the penultimate-layer latent embedding using a calibration set.

3 Methods

In this section, we illustrate in detail the Confidence Operating Characteristics (COC) curve. Then, we describe the proposed complementary cost function to train neural networks for classification: the area under COC (AUCOC) loss (AUCOCLoss).

3.1 Notation

Let D = ⟨ ( x n , y n ) ⟩ N n =1 denote a dataset composed of N samples from a joint distribution D ( X , Y ) , where x n ∈ X and y n ∈ Y = { 1 , 2 , ..., K } are the input data and the corresponding class label, respectively. Let f θ ( y | x ) be the probability distribution predicted by a classification neural network f parameterized by θ for an input x . For each data point x n , ˆ y n = argmax y ∈Y f θ ( y | x n ) denotes the predicted class label, associated to a correctness score c n = ✶ (ˆ y n = y n ) and to a confidence score r n = max y ∈Y f θ ( y | x n ) , where r n ∈ [0 , 1] and ✶ ( . ) is an indicator function. r = [ r 1 , ...r N ] represents the vector containing all the predicted confidences for a set of data points, e.g., a batch. p ( r ) denotes the probability distribution over r values (confidence space). We assume a human-AI collaboration system where samples with confidence r lower than a threshold r 0 would be delegated to a human expert for assessment.

3.2 Confidence Operating Characteristics (COC) curve

Our first goal is to introduce an appropriate evaluation method to assess the trade-off between a neural network's prediction accuracy and the number of samples that requires manual analysis from a domain expert. We focus on the COC curve, as it provides practitioners with flexibility in the choice of the operating point, similarly to the ROC curve.

3.2.1 x-y axes of the COC curve

To construct the COC curve, first, we define a sliding threshold r 0 over the space of predicted confidences r . Then, for each threshold r 0 , we calculate the portion of samples that are delegated to human expert and the accuracy of the network on the remaining samples for the threshold r 0 , which form the x-axis and y-axis of a COC curve, respectively. These axes are formulated as follows

For each threshold level r 0 , τ 0 represents the portion of samples whose confidence is lower than that threshold, i.e., the amount of the samples that are delegated to the expert. E [ c | r ≥ r 0 ] corresponds to the expected value of the correctness score c for all the samples for which the network's confidence is equal or larger than r 0 , i.e., accuracy among the samples for which network prediction will be used. This expected value, i.e., y-axis, can be computed as

We provide the derivation of Eq. 2 in the Appendix A.

3.2.2 Area under COC curve

Like the area under ROC curve, area under COC curve (AUCOC) is a global indicator of the performance of a system. Higher AUCOC indicates lower number of samples delegated to human experts or/and higher accuracy for the samples that are not delegated to human experts but analysed only by the network. Lower AUCOC on the other hand, indicates higher number of delegations to human experts or/and lower accuracy on the samples analysed only by the network. Further explaination is reported in Appendix I. It can be computed by integrating the COC curve over the whole range of τ 0 ∈ [0 , 1] :

Further details are provided in Section 3.5

3.3 AUCOCLoss: Maximizing AUCOC for training neural networks

As mentioned, in human-AI collaboration systems, higher AUCOC means a better model. Therefore, in this section we introduce a new loss function called AUCOCLoss that maximizes AUCOC for training classification neural networks.

AUCOC's explicit maximization would enforce the reduction of the number of samples delegated to human expert while maintaining the accuracy level on the samples assessed only by the algorithm (i.e., keeping E [ c | r ≥ r 0 ] constant) and/or the improvement in the prediction accuracy of the samples analysed only by the algorithm while maintaining a particular amount of data to be delegated to the human (i.e., keeping τ 0 constant), as illustrated in Figure 1a.

We define our loss function to maximize AUCOC as

Weuse the negative logarithm as AUCOC lies in the interval [0 , 1] , which corresponds to AUCOCLoss ∈ [0 , inf] which is suitable for minimizing cost functions. The dependence of AUCOC on the network parameters may not be obvious from the formulation given in Eq. 3. Indeed, this dependence is hidden in how p ( r ) and E [ c | r ] are estimated as we show next.

3.3.1 Kernel density estimation for AUCOC

We need to formulate AUCOC in a differentiable way in order for it to be incorporated in a cost function for the training of a neural network. For this purpose, we use kernel density estimation (KDE) on confidence predictions r n for training samples to estimate p ( r ) used in Eq. 3

where K is a Gaussian kernel and we choose its bandwidth using Scott's rule of thumb [35]. Then, the other terms in Eq. 3, namely E [ c | r ] p ( r ) and τ 0 , are estimated as

Note that r n = max y ∈Y f θ ( y | x n ) , from where the dependency of AUCOC on θ stems. Further, note that τ 0 is the x-axis of the COC curve. The AUCOC is defined by integrating over the τ 0 values, and therefore τ 0 should not depend on the network parameters, i.e., its derivative with respect to θ should be equal to zero. We can write the derivative using Leibniz integral rule as follows:

Then, the constraint that τ 0 should not depend on θ can be enforced explicitly by deriving the derivative dr 0 /dθ from Eq. 7 as follows:

where p ( r ) is implemented as in Eq. 5. Derivations are provided in the Appendix.

(a)

(b)

COC curvesfor fo1 andfo2

E[clr>ro]: accuracy of the network (%)

(%  []

95

OXO+X

0.8

0

0.5

0.66

0.6

1)Increasingaccuracy

0.4

2)Decreasingtheamountofdelegated

85

samples

0.2

f01,AUCOC 0.64

CE,AUCOC 91.60

+×

f02，AUCOC0.49

0

0.5

0.66

1

0.0

0.0

To: samples to be analysed manually (%)

0.1

0.2

0.3

0.4

0.5

0.6

'0

0.8

CE + SoftAvUC,AUCOC 93.97

CE + AUCOCL0SS, AUCOC 94.61

75

0

20

40

60

80

100

ECEfo= ECEfo2

AcCfθ1= AcCf o2

AUCOCfθ1 >AUCOCf02

To: samples to be analysed manually (%)

3.4 Toy example: added value by AUCOC

In this section, we demonstrate the added value of assessing the performance of a predictive model using COC curve and AUCOC through a toy example. We particularly compare with the widely used expected calibration error (ECE) [8], a binning-free calibration metric called Kolmogorov-Smirnov (KS) [9] and classification accuracy.

Assume we have two classification models f θ 1 and f θ 2 and they yield confidence scores and predictions for 5 samples as shown in Figure 1b. The green circles denote the predicted confidences for correctly classified samples, while the red crosses the confidences of the misclassified ones.

ECE: ECE divides the confidence space into bins, computes the difference between the average accuracy and confidence for each bin, and returns the average of the differences as final measure of calibration error. If we divide the confidence space into 5-bins, as indicated with the gray dotted lines in the confidence spaces of f θ 1 and f θ 2 , ECEs computed for the both models will be identical. Furthermore, a similar situation can be constructed for any number of bins.

KS: KS is a binning-free metric, so it is less prone to binning errors than ECE. However, one can prove that there exist some confidence configurations for which the two models also report the same

KS, in spite of it being a binning-free metric. This happens, for example, if the confidence values are (from left to right): 0.45, 0.55, 0.65, 0.70, 0.75.

Accuracy: These models have equal classification accuracy, 3/5 for both.

Therefore, looking at these three performance metrics, it is not possible to choose one model over the other since f θ 1 and f θ 2 perform identically.

AUCOC: On the contrary, the AUCOC is larger for f θ 1 than for f θ 2 , as shown in Figure 1b. The difference in AUCOC is due to the different ranking of correctly and incorrectly classified samples with respect to confidence values. It does not depend on a particular binning nor exact confidence values, but only on the ranking. By looking at the AUCOC results, one would prefer f θ 1 compared to f θ 2 . Indeed, f θ 1 is a better model than f θ 2 because it achieves either equal or better accuracy than f θ 2 , for the same amount of data to be manually examined. Changing point of view, f θ 1 delegates either equal or lower number of samples to experts for the same accuracy level.

Comparison of AUCOC and calibration: As also demonstrated with the toy example, AUCOC and calibration assess different aspects of a model. The former is a rank-based metric with respect to the ordering of the predictive confidences of correctly and incorrectly classified samples. The latter instead assess the consistency between the accuracy of the network and the predictive confidences, i.e. it is sensitive to the numerical values of the confidences themselves.

3.5 Implementation Details

Construction of the COC curve, thresholds r 0 and operating points: To provide flexibility in the selection of COC operating points, we need to cover the entire range [0 , 1] of τ 0 values. As a consequence, the thresholds r 0 need to span the confidence range of the predictions r n of the neural network. A natural choice for such thresholds is employing the predictions r n themselves. This spares us from exploring the confidence space with arbitrarily fine-grained levels. First, we sort the confidences r of the whole dataset (or batch) in ascending order. Each predicted confidence is then selected as threshold level r 0 for the vector r , corresponding to a certain τ 0 (x-value). Subsequently, E [ c | r ≥ r 0 ] (y-value) is computed. Note that setting the threshold to each r n in the sorted array corresponds to going through τ 0 = [1 /N, 2 /N,.. . , ( N -1) /N, 1] for N samples one by one in order.

Modelling correctness: Instead of using E [ c | r ] p ( r ) ≈ 1 N ∑ N n =1 c n K ( ∥ r -r n ∥ ) as given in Eq. 6, we approximate it as E [ c | r ] p ( r ) ≈ 1 N ∑ N n =1 r ∗ K ( ∥ r -r n ∥ ) where r ∗ n = f θ ( y n | x n ) is the confidence of the correct class for a sample n . The main reason is that the gradient of the misclassified samples becomes zero because c n is zero when a sample x n is not classified correctly. To deal with this issue we replace the correctness score c n , which can be either 0 or 1, with r ∗ n which can take continuous values between 0 and 1, following Yin et al. [39]. With this new approximation, we can back-propagate through misclassified samples and we found that this leads to better results.

Use as secondary loss: We observed in our experiments that using AUCOCLoss alone to train a network leads to very slow convergence with the existing optimization techniques. We believe this is due to the fact that the AUCOCLoss is a -log ( ∑ n z n ) with z n ∈ [0 , 1] . Optimization of such a form with gradient descent is slow because the contribution of increasing low z n 's to the loss function is small. In contrast, cross-entropy is a -∑ n log z n , where contribution of increasing low z n 's to the loss is much larger, hence gradient-based optimization is faster. One can in theory create an upper bound to AUCOC loss by pulling the log inside the sum, as we show in the Appendix. However, when z n ∈ [0 , 1] this upper bound is very loose and minimizing the upper bound not necessarily correspond to minimizing the AUCOC loss, hence does not maximize AUCOC. On the contrary, when AUCOCLoss is complements a primary cost that is faster to optimize, such as cross-entropy, it is improves over the primary loss and lead to the desired improved AUCOC while preserving the accuracy. This is obtained within the same amount of epochs and without ad-hoc fine-tuning the training hyper-parameters for the secondary loss.

4 Experiments

In this section, we present our experimental evaluations on multi-class image classification tasks. We performed experiments on five datasets. We experimented with CIFAR100 [19] and Tiny-ImageNet, a subset of ImageNet [4], following the literature on network calibration. Further, we used two publicly

available medical imaging datasets, DermaMNIST and RetinaMNIST [38], since medical imaging is an application area where expert time is limited and expensive. Due to space reasons we report results on a third medical dataset, TissueMNIST [38], in Appendix D, as well as information about the datasets in Appendix C.

Loss functions: We compared AUCOCLoss (referred to as AUCOCL in the tables) with different loss functions, most of which are designed to improve calibration performance while preserving accuracy: cross-entropy (CE), focal-loss (FL) [23], adaptive focal-loss (AdaFL) [26], maximum mean calibration error loss (MMCE) [21], soft binning calibration objective (S-ECE) and soft accuracy versus uncertainty calibration (S-AvUC) [16]. We optimized MMCE, S-ECE, and S-AvUC losses jointly with a primary loss for which we used either CE or FL, consistently with the literature [21; 16]. The same is done for AUCOCLoss, applying KDE batch-wise during the training.

Evaluation metrics: To evaluate the performance of the methods, we used classification accuracy and AUCOC. Classification accuracy is simply the ratio between the number of correct samples over the total number of samples, and AUCOC is computed using Eq. 3. We also report some examples of operating points of COC curve - given a certain accuracy, we show the corresponding expert load ( τ 0 @acc), i.e., percentage of samples that need to be analyzed manually, on the COC curves.

Since we compared AUCOCLoss mostly with losses for network calibration, we also assessed the calibration performance of the networks, even though calibration is not a direct goal of this work. To this end, we used the following metrics: the widely employed equal-mass expected calibration error (ECE) [28] with 15 bins, the binning-free Brier score [1] and Kolmogorov-Smirnov (KS) score [9] and the class-wise ECE (cwECE) [20]. The evaluation is carried out post temperature scaling (TS) [8], as it has been proved to be always beneficial for calibration.

Hyperparameters: In addition to the common hyperparameters for all losses, selected as specified in Appendix C, there are also specific ones that need to be tuned for some of them. In this case, we used the best hyperparameter settings reported in the original papers. In cases where the original paper did not report the specific values, we carried out cross-validation and selected the setup that provided the best performance on the validation set. We also selected the weighting factor for AUCOCLoss in the same way. We found that optimal weighting values for AUCOCLoss all fell between 1 and 10. We found empirically that models check-pointed using ECE provided very poor results. Networks check-pointed using either accuracy or AUCOC provided comparable outcome with respect to accuracy, therefore we reported results on AUCOC as they provided the best overall performance.

OODexperiments: We evaluate the out-of-distribution (OOD) samples detection performance of all methods since OOD detection is crucial for a reliable AI system and it is a common experiment in the network calibration literature. The commonly used experimental setting in this literature is using CIFAR100-C [12] (we report results for CIFAR100 with Gaussian noise in the main paper and the average over all the perturbations in the Appendix) and SVHN [27] as OOD datasets, while the network is trained on CIFAR100 (in-distribution). We evaluated the OOD detection performance of all methods using Area Under the Receiver Operating Characteristics (AUROC) curve, with MSP [13], ODIN [14], MaxLogit [11] and EBM [25].

Class imbalance experiments: Finally, in Appendix E we report results for accuracy and AUCOC on imbalanced datasets, being class imbalance present in many domains, such as medical imaging. We report results on the widely used Long-Tailed CIFAR100 (CIFAR100-LT) [37; 42] with controllable degrees of data imbalance ratio to control the distribution of training set. We trained with three levels of imbalance ratio, namely 100, 50, 10.

Further details on hyper-parameter settings are provided in the the Appendix.

5 Results

First, we report the results for accuracy and COC-related metrics. Then, we report results for calibration, even though this is not an explicit goal of this work. Bold results indicate the methods that performed best for each metric, underlined results are the second best. ↑ means the higher the better for a metric, while ↓ the lower the better. The experiments results are averaged over three runs.

In Tables 1, 2 we present our results based on accuracy and AUCOC. The results for TissueMNIST can be found in the Appendix. We observed that complementing CE and FL with our loss, i.e., CE+AUCOCL and FL+AUCOCL, they were consistently better than the other losses in all experiments. To evaluate the significance of the accuracy and AUCOC results, we performed permutation test [30] between the best AUCOCLoss and the best baseline with 1000 rounds. In all the cases it provided a p score « 1 % , therefore the differences in the models are steadily significant. The advantage of our model was even more apparent in amount of expert loads corresponding to specific accuracy levels. In particular, we measured the percentage of samples delegated to expert ( τ 0 ) at 90% and 95% accuracy for CIFAR100, DermaMNIST and TissueMNIST, and at 65% and 75% accuracy for Tiny-Imagenet and RetinaMNIST (as the initial accuracy is also much lower on these datasets). In all the experiments, to varying degrees, AUCOCLoss provided lower delegated samples than the baselines. Noticeably, while some of the baselines may be close to the results of AUCOCLoss for certain metrics, none of them were consistently close to AUCOCLoss across all the datasets. For example, when looking at the last two columns for CIFAR100 and Tiny-ImageNet respectively of Table 1, CE+MMCE is worse by 4% than AUCOCLoss on Tiny-ImageNet, but by around 9% in

CIFAR100. Figure 1a shows examples of COC curves. Overall, the plot of AUCOCLoss lies above all the baselines, which is a desirable behavior as it corresponds to better operating points.

Even though the proposed loss was not designed to improve calibration, it provided on par performance compared to the other cost functions particularly designed for network calibration, as reported in Table 3 for DermaMNIST and RetinaMNIST and in the Appendix for the other datasets.

In OOD experiments reported in Table 4, we used the model trained on CIFAR100 and evaluated the OOD detection performance on CIFAR100-C (with Gaussian noise) and SVHN dataset. Results for CIFAR100-C averaged over all the perturbations are reported in the Appendix. We employed state-of-the-art OOD detectors, namely MSP, ODIN, MaxLogit and EBM to determine whether a sample is OOD or in-distribution. The bold results highlight the best results in terms of AUROC. On both OOD datasets, AUCOCLoss always provided the highest AUROC and in almost all the cases also the second best.

Class imbalance experiments on CIFAR100-LT are reported in the Appendix. AUCOCLoss obtains best results for both accuracy and AUCOC, with higher benefits at increasing imbalance.

Crucially, CE+AUCOCL , where the proposed loss is used jointly with CE, outperformed every baseline in accuracy, AUCOC and τ 0 @acc. in all the experiments. It further outperformed all the baselines in OOD detection and class imbalance experiments, presented in the Appendix. Even

though the metric is not geared towards calibration, CE+AUCOCL yielded either the best or the second best calibration performance in the majority of the cases compared to the baselines.

In the Appendix, we explore how the performance of a model trained with AUCOCLoss varies, when changing batch size as it can be crucial for KDE-based methods. Even substantially lower batsch sizes do not have a considerable effect on the performance of the proposed method.

6 Conclusion

In this paper we proposed a new cost function for multi-class classification that takes into account the trade-off between a neural network's accuracy and the amount of data that requires manual analysis from a domain expert, by maximizing the area under COC (AUCOC) curve. Experiments on multiple computer vision and medical image datasets suggest that our approach improves the other methods in terms of both accuracy and AUCOC, where the latter was expected by design, provides comparable calibration metrics, even though the loss does not aim to improve calibration explicitly and outperforms the baselines in OOD detection.

While we presented COC and AUCOCLoss for multi-class classification, extensions to other tasks are possible future work as well as investigating different performance metrics to embed in the y -axis of COC. Moreover, aware of potential problems with KDE at the boundaries, i.e., boundary bias, we explored corrections like reflection method, which did not provide major improvements, but we will further investigate. We believe that this new direction of considering expert load in human-AI system is important and AUCOCLoss will serve as a baseline for future work.

Limitations: As described in Section 3.5, AUCOCLoss alone empirically leads to slow convergence, due to the small contribution of -log ( ∑ n z n ) with z n ∈ [0 , 1] in its formulation. Therefore, we recommend to use it as a secondary loss, to successfully complement existing cost functions.

7 Acknowledgments

This study was financially supported by: 1. The LOOP Zürich - Medical Research Center, Zurich, Switzerland, 2. Personalized Health and Related Technologies (PHRT), project number 222, ETH domain and 3. Clinical Research Priority Program (CRPP) Grant on Artificial Intelligence in Oncological Imaging Network, University of Zürich.

References

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review 78 , 1-3.

Chen, L.-C., G. Papandreou, F. Schroff, and H. Adam (2017). Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 .

Davidson, T., D. Warmsley, M. Macy, and I. Weber (2017). Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media , ICWSM '17, pp. 512-515.

Deng, J., W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei (2009). Imagenet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pp. 248-255.

Dvijotham, K., J. Winkens, M. Barsbey, S. Ghaisas, R. Stanforth, N. Pawlowski, P. Strachan, Z. Ahmed, S. Azizi, Y. Bachrach, et al. (2023). Enhancing the reliability and accuracy of ai-enabled diagnosis via complementarity-driven deferral to clinicians. Nature Medicine , 1-7.

Esteva, A., B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun (2017). Dermatologist-level classification of skin cancer with deep neural networks. nature 542 (7639), 115-118.

Gorski, N., V. Anisimov, E. Augustin, O. Baret, and S. Maximov (2001). Industrial bank check processing: the a2ia checkreadertm. International Journal on Document Analysis and Recognition 3 (4), 196-206.

Guo, C., G. Pleiss, Y. Sun, and K. Q. Weinberger (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, pp. 1321-1330. JMLR.org.

Gupta, K., A. Rahimi, T. Ajanthan, T. Mensink, C. Sminchisescu, and R. Hartley (2021). Calibration of neural networks using splines. In International Conference on Learning Representations .

He, K., X. Zhang, S. Ren, and J. Sun (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 770-778.

Hendrycks, D., S. Basart, M. Mazeika, M. Mostajabi, J. Steinhardt, and D. X. Song (2022). Scaling out-of-distribution detection for real-world settings. In International Conference on Machine Learning .

Hendrycks, D. and T. Dietterich (2019). Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations .

Hendrycks, D. and K. Gimpel (2016). A baseline for detecting misclassified and out-ofdistribution examples in neural networks. arXiv preprint arXiv:1610.02136 .

Hsu, Y.-C., Y. Shen, H. Jin, and Z. Kira (2020, 06). Generalized odin: Detecting out-ofdistribution image without learning from out-of-distribution data. pp. 10948-10957.

Jiang, X., M. Osl, J. Kim, and L. Ohno-Machado (2012). Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association 19 (2), 263-274.

Karandikar, A., N. Cain, D. Tran, B. Lakshminarayanan, J. Shlens, M. C. Mozer, and B. Roelofs (2021). Soft calibration objectives for neural networks. In NeurIPS .

Kelly, C. J., A. Karthikesalingam, M. Suleyman, G. Corrado, and D. King (2019). Key challenges for delivering clinical impact with artificial intelligence. BMC medicine 17 (1), 1-9.

Krishnan, R. and O. Tickoo (2020). Improving model calibration with accuracy versus uncertainty optimization. Advances in Neural Information Processing Systems .

Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical report.

Kull, M., M. Perello-Nieto, M. Kängsepp, T. S. Filho, H. Song, and P. Flach (2019). Beyond Temperature Scaling: Obtaining Well-Calibrated Multiclass Probabilities with Dirichlet Calibration . Red Hook, NY, USA: Curran Associates Inc.

Kumar, A., S. Sarawagi, and U. Jain (2018). Trainable calibration measures for neural networks from kernel mean embeddings. In ICML .

Leibig, C., M. Brehmer, S. Bunk, D. Byng, K. Pinker, and L. Umutlu (2022). Combining the strengths of radiologists and ai for breast cancer screening: a retrospective analysis. The Lancet Digital Health 4 (7), e507-e519.

Lin, T.-Y., P. Goyal, R. Girshick, K. He, and P. Dollár (2017). Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV) , pp. 2999-3007.

Lin, Z., S. Trivedi, and J. Sun (2022). Taking a step back with kcal: Multi-class kernel-based calibration for deep neural networks.

Liu, W., X. Wang, J. D. Owens, and Y. Li (2020). Energy-based out-of-distribution detection. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual .

Mukhoti, J., V. Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania (2020). Calibrating deep neural networks using focal loss. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances in Neural Information Processing Systems , Volume 33, pp. 15288-15299. Curran Associates, Inc.

Netzer, Y., T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng (2011). Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 .

Nixon, J., M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran (2019, June). Measuring calibration in deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops .

Patel, B. N., L. Rosenberg, G. Willcox, D. Baltaxe, M. Lyons, J. Irvin, P. Rajpurkar, T. Amrhein, R. Gupta, S. Halabi, et al. (2019). Human-machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ digital medicine 2 (1), 1-10.

Pesarin, F. and L. Salmaso (2010, 03). Permutation Tests for Complex Data .

Platt, J. (2000, 06). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classif. 10 .

Quinonero-Candela, J., M. Sugiyama, A. Schwaighofer, and N. D. Lawrence (2008). Dataset shift in machine learning . Mit Press.

Rajpurkar, P., J. Irvin, R. L. Ball, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. P. Langlotz, et al. (2018). Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists. PLoS medicine 15 (11), e1002686.

Sangalli, S., E. Erdil, A. Hötker, O. Donati, and E. Konukoglu (2021). Constrained optimization to train neural networks on critical and under-represented classes. Advances in Neural Information Processing Systems 34 , 25400-25411.

Scott, D. W. (1979). On optimal and data-based histograms. Biometrika 66 (3), 605-610.

Szegedy, C., V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 2818-2826.

Tang, K., J. Huang, and H. Zhang (2020). Long-tailed classification by keeping the good and removing the bad momentum causal effect. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA. Curran Associates Inc.

Yang, J., R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni (2023, 01). Medmnist v2 - a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data 10 .

Yin, P., J. Lyu, S. Zhang, S. J. Osher, Y . Qi, and J. Xin (2019). Understanding straight-through estimator in training activation quantized neural nets. In International Conference on Learning Representations .

Zadrozny, B. and C. Elkan (2001). Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning , ICML '01, San Francisco, CA, USA, pp. 609-616. Morgan Kaufmann Publishers Inc.

Zagoruyko, S. and N. Komodakis (2016, September). Wide residual networks. In E. R. H. Richard C. Wilson and W. A. P. Smith (Eds.), Proceedings of the British Machine Vision Conference (BMVC) , pp. 87.1-87.12. BMVA Press.

Zhou, B., Q. Cui, X.-S. Wei, and Z.-M. Chen (2020). BBN: Bilateral-branch network with cumulative learning for long-tailed visual recognition. pp. 1-8.

A Derivations

In this Section we provide the derivation of AUCOC and its gradient formulation.

A.1 Derivation of AUCOC

AUCOC is defined as:

The y-axis in COC curve is expressed mathematically by:

The x-axis in COC curve is expressed mathematically by:

Using the τ 0 formulation, we can rewrite the y-axis as

Maximizing the area under this curve, over τ 0 ∈ [0 , 1] corresponds to

Let us assume to use a Gaussian kernel with this expression:

Developing the equation, the area calculation becomes:

Where:

Where ndtr expresses the Gaussian cumulative distribution function and the last row in Equation 10 exploits the trapezoidal rule for integrals computation.

A.2 Derivations of the gradients of AUCOC

Here, we use the assumption discussed in Section Methods that τ 0 does not depend on any parameter, thus allowing us to apply Leibnitz's integration rule, obtaining:

τ 0 can be expressed as:

Consequently:

And re-writing:

Plugging this expression back into Equation 12 we obtain:

Assuming the use of a Gaussian kernel:

The gradient of the area becomes:

Also for the gradients in the code implementation we exploited the trapezoidal rule for the computation of the external integral between [0,1].

B Potential upper bound to AUCOCLoss

AUCOC has a formulation of the kind -log ( ∑ n z n ) with z n ∈ [0 , 1] . In contrast, cross-entropy is a -∑ n log z n , where contribution of increasing low z n 's to the loss is much larger, hence gradientbased optimization is faster. Exploiting Jensen's inequality, one could find the following upper bound of AUCOCLoss to minimise. However, from thorough experiments it has been proven not to be a tight enough bound for the optimisation to be successful. In fact, trying to optimise the rightmost term of the following equation, instead of the correct definition on the left, does not lead to a satisfactory optimisation of AUCOC.

C Training details

In CIFAR100 experiments, we followed Karandikar et al. [16] and used Wide-Resnet-28-10 [41] as the network architecture. We trained the models for 200 epochs, using Stochastic Gradient Descent (SGD), with batch of 512, momentum of 0.9 and an initial learning rate of 0.1, decreased after 60, 120, 160 epochs by a factor of 0.1. We set these parameters based on the best validation performance of CE and we keep it for all the losses. In Tiny-ImageNet experiments, we used ResNet-50 [10] as backbone architecture, SGD as optimiser with a batch size of 512, momentum of 0.9 and base learning rate of 0.1, divided by 0.1 at 40th and 60th epochs as in Mukhoti et al. [26] In DermaMNIST, RetinaMNIST and TissueMNIST [38] experiments, we followed the training procedures of the original paper, employing a ResNet-50 He et al. [10], Adam optimizer. The batch size is set to 128 for DermaMNIST and RetinaMNIST and to 512 for the larger TissueMNIST. We used the initial learning rate 0.0001 for DermaMNIST and 0.001 for RetinaMNIST and TissueMNIST, and trained the models for 100 epochs by reducing the learning rate by 0.1 after epochs 50 and 75.

All the models have been trained using either the NVIDIA GeForce RTX 2080 Ti or NVIDIA GeForce RTX 3090. The datasets have been split as follows.

For CIFAR100 we used 45000/5000/10000 images respectively as training/validation/test sets.

Tiny-ImageNet is a subset of ImageNet with 64 × 64 images and 200 classes. We employed 90000/10000/10000 images as training/validation/test set, respectively.

DermaMNIST is composed of dermatoscopic images with 7007/1003/2005 samples for training, validation and test set, respectively, categorised in 7 different diseases.

RetinaMNIST is based on the DeepDRiD24 challenge, which provides a dataset of 1,600 retina fundus images. The task is ordinal regression for 5-level grading of diabetic retinopathy severity. Consistently with the original paper, we split the source training set with a ratio of 9 : 1 into training and validation set, and use the source validation set as the test set.

D Results on a third medical dataset, TissueMNIST

Tables 5 and 6 report results on an additional medical dataset, TissueMNIST. The dataset contains 236,386 human kidney cortex cells, segmented from 3 reference tissue specimens and organized into 8 categories. We split the source dataset with a ratio of 7 : 1 : 2 into training, validation and test set. The results are consistent with those reported in the main paper.

E Class imbalance experiments

In Table 7 we report the results for accuracy and AUCOC on the widely employed Long-Tailed CIFAR100. We employed a tunable data imbalance ratio (Nmax / Nmin, where N is number of samples in each class), which controls the class distribution in the training set and trained the models with three levels of imbalance ratio, namely 100, 50, 10. Noticeably, when AUCOCLoss complements cross-entropy it obtains the best results for both metrics in all the three settings, with higher benefits at increasing imbalance, while with FL it obtains always the second best AUCOC and comparable accuracy.

F Calibration results for Tiny-ImageNet and CIFAR100

In Table 8 we report calibration results respectively for CIFAR100 and Tiny-ImageNet. For all the metrics and datasets, AUCOCLoss provides comparable results with respect to the baselines.

G Results for varying batch size

In Table 9 we report the results on DermaMNIST with reduced batch sizeon which KDE is applied on, i.e. 64 and 28 samples per batch. The results of AUCOCLoss are not significantly affected by it, when complementing cross-entropy and focal loss.

H Additional results for CIFAR100 and Tiny-ImageNet

In Table 10 we report test results on CIFAR100 using ResNet-50 and Tiny-Imagenet using WideResnet-28-10, run for one seed. Best and second best results are in bold and underlined. They show consistency with the main experiments reported in the paper.

I Further explanation on how improve AUCOC

There are two factors which contribute to an increase in AUCOC: decrease in the number of samples delegated to human experts (given the same network accuracy) and increase in the accuracy for the samples that are not delegated but analysed only by the network (given the same human workload).

These two aspects could manifest either individually, if the AUCOC improvement is generated by just a shift "up" or "left" of COC, or in a combined way. The example provided in Figure 1a shows an improvement in both axes ("and" case) and the proposed loss function does not favour one specific behaviour. Figure 2 provides an example of shifts "up" and "left" ("or" cases). From the AUCOC metrics alone, it is not possible to infer which mechanism is taking place.

0.0

0.2

0.4

0.6

0.8

1.0

Samples to be manually analysed

0.0

0.2

0.4

0.6

0.8

1.0

Network accuracy

COC Curves

COC Curve (AUCOC = 0.51)

COC Shift "up" (AUCOC = 0.71)

COC Shift "left" (AUCOC = 0.71)

J OODresults on all the perturbations on CIFAR100-C

In table 11 we report the average results on CIFAR100-C for all the perturbations available from Hendrycks and Dietterich [12].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work supposes a real-world setting where misclassified examples are reviewed post-hoc by human experts, and offers a near-optimal trade-off between such examples (the expert load) and classifier accuracy. The authors propose to use a curve of confidence versus expert load, using the latter as a sliding scale to arrive at that optimum. They propose a no-binning method that kernelises the said (CFOC) curve, and arrive at a formulation to minimize the expert load value irrespective of the network parameters so that the formulation may be added to the error minimisation loop. \n\nExtensive experiments are performed, such as those on OOD detection where overconfidence is significant, and imbalance, where traditional OC curves are less meaningful because the underlying metrics are too.\n\nSTRENGTHS:\nThe idea has a high likelihood of orginality and novelty, to the extent that benchmarking has had to be done with the COC-derived-loss-augmented CE and all sorts of CE-derived losses, and not competing methods. The idea is developed in a simple manner that persuades the reader's logic. Experimentation is fairly complete, even though a discussion of mixture ratios of CE and AUCOC- losses was expected too.\n\nWEAKNESSES:\nStudying the dynamic of mixing two losses that have such variable gradient magnitude properties on the rate and preciseness of convergence is something I'd have done. \n\nThe $\\tau$ being the bandwidth of the kernel, seems to be extraordinarily large at both levels in table 1. I'm not sure if I'd be offloading up to a third of testing examples to the expert.\n\nQUESTIONS:\nEquation 3 does not mention how AUCOC is determined. It just computes the AUCOC Loss using the former. Please let me know in my assuming it is determined in the same way as the ECE does, i.e.  accuracy vs. confidence, as is strongly evident from the presence of bins? How do you define confidence?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a \"confidence operating characteristic\" curve to represent tradeoff between accuracy and numbers of samples delegated to human experts. To maximize the area under this curve, the authors propose a new loss. The authors run classification experiments on computer vision and medical image datasets.\n\nSTRENGTHS:\nThe paper tackles a problem that is clearly important -- namely, minimizing the human effort needed in a human-in-the-loop system and quantifying the \"human effort\" portion. However, see weaknesses section below.\n\nWEAKNESSES:\n- The references list looks reasonable (35 references), but for whatever reason, the related works section seems very sparse and lacking in relevant bodies of work. Upon closer inspection, it looks like a significant portion of those references (10+) are not related, just part of the motivation. This work is effectively \"active learning,\" which is conspicuously absent from the paper entirely. I'm not super familiar with active learning, but no doubt there is previous work that already tackles the problems introduced here -- and it's not clear how this paper is different. For example, here's a highly cited paper in the area: \"Cost-Effective Active Learning for Deep Image Classification\" 2017, https://arxiv.org/abs/1701.03551. The paper here proposes a metric that minimizes annotation cost -- in particular, they ask humans to annotate highly-unconfident samples. This seems like a reasonable metric and it's not clear why end-to-end minimizing cost is better, for example. (I'm sure it could be justified) Granted, the setup is slightly different, but I'm sure a more thorough search will yield more relevant papers.\n- The paper doesn't cite previous metrics for \"human effort\", how they are fallible, and how the proposed metric fixes that problem. As a result, it's not clear why this metric is preferred to other variants.\n- Experiments are performed on MNIST variants, CIFAR100 and TinyImageNet, but do these results extend to ImageNet for example? A full ImageNet run isn't needed; just a few epochs showing that results are trending positively for your method.\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nSee weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper aims to address the trade-off between model accuracy and model confidence results. The authors propose a novel loss function called AUCOC, which maximizes the area under the confidence operating characteristic curve. They evaluate the performance of their approach on various image classification and medical image classification datasets.

STRENGTHS:
The problem being addressed is of significant importance and has been widely studied.

WEAKNESSES:
1. In terms of empirical evaluation, the authors only assess the performance on image classification datasets. As this is primarily an empirical paper, it would be beneficial to see results on different types of datasets, including text data, tabular data, and other diverse datasets. Moreover, I am curious to know how the proposed model performs on larger-scale datasets, as the ones evaluated in the paper (e.g., CIFAR100 and Tiny-ImageNet) may not be sufficiently large-scale.

2. Furthermore, when it comes to out-of-distribution (OOD) detection results, it would be valuable for the authors to compare their proposed methods with commonly used approaches such as MSP, MaxLogit, and others.

The proposed method is commendably simple and straightforward. However, I believe that a more comprehensive evaluation is necessary to demonstrate the effectiveness of the proposed approach.

QUESTIONS:
See cons above

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new loss, AUCOC loss, to improve the networks accuracy and prediction confidence. The loss aims to reduce the number of errors made by the algorithm and thus also the number of delegated samples to domain experts.  The proposed loss focuses on maximizing the area under the COC curve during training in a differentiable manner. The AUCOC loss is used complementary to the original network’s training loss, resulting in increased classification accuracy, better OOD sample detections, and on par calibration performance.\n\nSTRENGTHS:\nThe usage of the ''area under the confidence operating characteristics'' curve as an additional loss (being differentiable) is a novel idea. \n\nThe paper is well written, the core method is well explained and the experimental results back the claims of the paper.\n\nThe proposed AUCOC loss is outperforming other loss functions.\n\nThe possible weakness of only using AUCOC loss is clearly described.\n\nWEAKNESSES:\nThe title is somewhat misleading. \n\nA clear reference of which CNNs are used for the experiments is needed, especially since different CNNs were used for the experiments. \n\nHow does ResNet-50 behave for CIFAR100 using the AUCOC loss and how does Wide-Resnet-28-10 behave on TinyImageNet? Was the latter Wide-Resnet also used for table 4? \n\nWhy was only the corruption type Gaussian noise evaluated for the CIFAR 100-C experiments? \n\nThe paper wants to minimize the expert load and to be able to predict OOD samples correctly. So far, the paper shows that the accuracy increases and the amount of delegated samples decreases in the OOD setting. While in its current state, the paper has important contributions, it would be interesting to explore if the proposed method also influences the prediction behavior of the CNN. [1] analyses the prediction behavior of CNNs and finds a shape-bias cue conflict: CNNs tend to recognize texture rather than shape, which is in contrast to human vision behavior, which recognizes objects based on their shapes. Therefore, an obvious step would be to investigate how the networks predict objects (i.e., shape or texture), and if the AUCOC can also influence that behavior. \n\nLine 149-153: This or/and is confusing. When does a higher AUCOC indicate lower number of samples delegated to humans *and* also a higher accuracy, and when is it one of these results? \n\nTypos: \n\nReferences to Eq. 3 are sometimes misleading, while a reference to the Eq. in line 153 would sometimes be more appropriate. \n\nLine 173: “estimate”: choose\n\nLine 178: Leibniz integral rule\n\n[1] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR 2019\n\nQUESTIONS:\nSee Weaknesses.\n\nIt would be interesting to see if the proposed AUCOC loss can influence the CNN prediction behavior to be more in line with humans resulting in a lower need for experts.\n\nLIMITATIONS:\nThe paper addressed the limitations of why the AUCOC loss is rather used in conjunction with classical CNN loss functions, like cross-entropy.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}