SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sara Sangalli
sara.sangalli@vision.ee.ethz.ch
Ertunc Erdil
Computer Vision Lab, ETH Zürich
The LOOP Zurich - Medical Research Center, Zurich, Switzerland
In human-AI collaboration systems for critical applications, in order to ensure minimal error, users should set an operating point based on model confidence to determine when the decision should be delegated to human experts. Samples for which model confidence is lower than the operating point would be manually analysed by experts to avoid mistakes. Such systems can become truly useful only if they consider two aspects: models should be confident only for samples for which they are accurate, and the number of samples delegated to experts should be minimized. The latter aspect is especially crucial for applications where available expert time is limited and expensive, such as healthcare. The trade-off between the model accuracy and the number of samples delegated to experts can be represented by a curve that is similar to an ROC curve, which we refer to as confidence operating characteristic (COC) curve. In this paper, we argue that deep neural networks should be trained by taking into account both accuracy and expert load and, to that end, propose a new complementary loss function for classification that maximizes the area under this COC curve. This promotes simultaneously the increase in network accuracy and the reduction in number of samples delegated to humans. We perform experiments on multiple computer vision and medical image datasets for classification. Our results demonstrate that the proposed loss improves classification accuracy and delegates less number of decisions to experts, achieves better out-of-distribution samples detection and on par calibration performance compared to existing loss functions.<sup>1</sup>
Artificial intelligence (AI) systems based on deep neural networks have achieved state-of-the-art results by reaching or even surpassing human-level performance in many predictive tasks [6; 33; 2; 36]. Despite the great potential of neural networks for automation, there are pitfalls when using them in a fully automated setting, especially pertinent for safety-critical applications, such as healthcare [17; 32; 34]. Human-AI collaboration aims at remedying such issues by keeping humans in the loop and building systems that take advantage of both [29]. An example of such human-AI collaboration is hate speech detection for social media [3], where neural networks could reduce the load of manual analysis of contents required by humans. Healthcare is another relevant application [5; 22], . For
$^{1}$ Code is available at: https://github.com/salusanga/aucoc_loss.
example, a neural network trained to predict whether a lesion is benign or malignant should leave the decision to human doctors if it is likely to make an error [15]. The doctors' domain knowledge and experience could be exploited to assess such, possibly, ambiguous cases and avoid mistakes.
A simple way of building collaboration between a network and a human expert is delegating the decisions to the expert when the network's confidence score for a prediction is lower than a threshold, which we refer to as "operating point". It is clear that the choice of the operating point can only be done through a trade-off between the network performance on the automatically analysed samples, i.e., the number of errors expected by the algorithm, and the number of delegated samples, i.e., experts' workload. The latter is crucial especially for applications where expert time is limited and expensive. For example, in medical imaging, the interpretation of more complex data requires clinical expertise and the number of available experts is limited, especially in low income countries [17]. Hence, predictive models that can analyse a large portion of the samples at high accuracy and identify the few samples that should be delegated to human experts would naturally be more useful with respect to this trade-off.
It is possible to evaluate the performance of a predictive model taking simultaneously into account the accuracy and the number of samples that requires manual assessment from a human expert with a performance curve reminiscent of Receiver Operating Characteristic (ROC) curves, as illustrated in Fig. 1a. We will refer to this performance curve as Confidence Operating Characteristics (COC) as it is similar to the ROC curve. A COC curve plots for a varying threshold on algorithm confidence, i.e., operating point, the accuracy of a model on the samples on which the algorithm is more confident than the threshold versus the number of samples remaining below the threshold. The former corresponds to the accuracy of the model on automatically analysed samples while the latter corresponds to the amount of data delegated to the human expert for analysis. In an ROC curve a balance is sought after between Sensitivity and Specificity of a predictive model, while a COC curve can be used by domain experts, such as doctors, to identify the most suitable balance between the accuracy on the samples that are automatically analysed and the amount of data delegated to be re-examined by a human for the specific task. Variations of this curve have been used to evaluate the performance of automated industrial systems [7].
In this paper, we focus on the trade-off between model accuracy and the amount of samples delegated to domain experts based on operating points on model confidence. Specifically, our goal is to obtain better trade-off conditions to improve the interaction between the AI and the human expert. To this end, we propose a new loss function for multi-class classification, that takes into account both of the aspects by maximizing the area under COC (AUCOC) curve. This enforces the simultaneous increase in neural network accuracy on the samples not analysed by the expert and the reduction in human workload. To the best of our knowledge, this is the first paper to include the optimization of such curve during the training of a neural network, formulating it in a differentiable way. We perform experiments on two computer vision and three medical image datasets for multi-class classification. We compare the proposed complementary AUCOC loss with the conventional loss functions for training neural networks as well as network calibration methods. The results demonstrate that our loss function complements other losses and improved both accuracy and AUCOC. Additionally, we evaluate network calibration and out-of-distribution (OOD) samples detection performance of networks trained with different losses. The proposed approach was also able to consistently achieve better OOD samples detection and on par network calibration performance.
For the performance analysis of human-AI collaborative systems, confidence operating characteristics (COC) curves can be employed, which plot network accuracy on accepted samples against manual workload of a human expert, e.g as in [7]. While such curves have been used, to the best of our knowledge, we present the first work that defines a differentiable loss based on COc curve, in order to optimize neural networks to take into account simultaneously accuracy and experts' load for a human-AI collaborative system. Thus, there is no direct literature with which we can compare.
The underlying assumption in deciding which sample to delegate to human expert based on a threshold on confidence scores is that these scores provided by deep learning models indicate how much the predictions are likely to be correct or incorrect. However, the final softmax layer of a network does not necessarily provide real probabilities of correct class assignments. In fact, modern deep neural
networks that achieve state-of-the-art results are known to be overconfident even in their wrong predictions. This leads to networks that are not well-calibrated, i.e., the confidence scores do not properly indicate the likelihood of the correctness of the predictions [8]. Network calibration methods mitigate this problem by calibrating the output confidence scores of the model, making the above mentioned assumption hold true. Thus, we believe that the literature on network calibration methods is the closest to our setting because they also aim at improving the interaction between human and AI, by enforcing correlation between network confidence and accuracy, so that confidence can be used to separate samples where networks' predictions are not reliable and should be delegated to human experts. Calibration methods aim to get models that are highly accurate in the samples they are confident, but not the problem of minimising the number of samples delegated to human experts, contrary to the loss proposed here.
Guo et al. [8] defines the calibration error as the difference in expectation between accuracy and confidence in each confidence bin. One category of calibration methods augments or replaces the conventional training losses with another loss to explicitly encourage reducing the calibration error. Kumar et al. [21] propose the MMCE loss by replacing the bins with kernels to obtain a continuous distribution and a differentiable measure of calibration. Karandikar et al. [16] propose two losses for calibration, called Soft-AvUC and Soft-ECE, by replacing the hard confidence thresholding in AvUC [18] and binning in ECE [8] with smooth functions, respectively. All these three functions are used as a secondary loss along with conventional losses such as cross-entropy. Mukhoti et al. [26] find that Focal Loss (FL) [23] provides inherently more calibrated models, even if it was not originally designed for this, as it adds implicit weight regularisation. The authors further propose Adaptive Focal Loss (AdaFL) with a sample-dependent schedule for the choice of the hyperparameter  $\gamma$ . The second category of methods are post-hoc calibration approaches, which rescale model predictions after training. Platt scaling [31] and histogram binning [40] fall into this class. Temperature scaling (TS) [8] is the most popular approach of this group. TS scales the logits of a neural network, dividing them by a positive scalar, such that they do not saturate after the subsequent softmax activation. TS can be used as a complementary method and it does not affect model accuracy, while significantly improving calibration. A recent work by Gupta et al. [9] fits a spline function to the empirical cumulative distribution to re-calibrate post-hoc the network outputs. They also present a binning-free calibration measure inspired by the Kolmogorov-Smirnov (KS) statistical test. Lin et al. [24] propose a Kernel-based method on the penultimate-layer latent embedding using a calibration set.
In this section, we illustrate in detail the Confidence Operating Characteristics (COC) curve. Then, we describe the proposed complementary cost function to train neural networks for classification: the area under COC (AUCOC) loss (AUCOCLoss).
Let  $D = \langle (x_{n},y_{n})\rangle_{n = 1}^{N}$  denote a dataset composed of  $N$  samples from a joint distribution  $\mathcal{D}(\mathcal{X},\mathcal{Y})$ , where  $x_{n}\in \mathcal{X}$  and  $y_{n}\in \mathcal{Y} = \{1,2,\dots,K\}$  are the input data and the corresponding class label, respectively. Let  $f_{\theta}(y|x)$  be the probability distribution predicted by a classification neural network  $f$  parameterized by  $\theta$  for an input  $x$ . For each data point  $x_{n}$ ,  $\hat{y}_n = \operatorname{argmax}_{y\in \mathcal{Y}}f_\theta (y|x_n)$  denotes the predicted class label, associated to a correctness score  $c_{n} = \mathbb{1}(\hat{y}_{n} = y_{n})$  and to a confidence score  $r_n = \max_{y\in \mathcal{Y}}f_\theta (y|x_n)$ , where  $r_n\in [0,1]$  and  $\mathbb{1}(.)$  is an indicator function.  $\mathbf{r} = [r_1,\ldots r_N]$  represents the vector containing all the predicted confidence for a set of data points, e.g., a batch.  $p(r)$  denotes the probability distribution over  $r$  values (confidence space). We assume a human-AI collaboration system where samples with confidence  $r$  lower than a threshold  $r_0$  would be delegated to a human expert for assessment.
3.2 Confidence Operating Characteristics (COC) curve
Our first goal is to introduce an appropriate evaluation method to assess the trade-off between a neural network's prediction accuracy and the number of samples that requires manual analysis from a domain expert. We focus on the COC curve, as it provides practitioners with flexibility in the choice of the operating point, similarly to the ROC curve.
3.2.1 x-y axes of the COC curve
To construct the COC curve, first, we define a sliding threshold  $r_0$  over the space of predicted confidences  $r$ . Then, for each threshold  $r_0$ , we calculate the portion of samples that are delegated to human expert and the accuracy of the network on the remaining samples for the threshold  $r_0$ , which form the x-axis and y-axis of a COC curve, respectively. These axes are formulated as follows
$$
x - \text {a x i s}: \tau_ {0} = p (r <   r _ {0}) = \int_ {0} ^ {r _ {0}} p (r) d r, \quad y - \text {a x i s}: \mathbb {E} [ c | r \geq r _ {0} ] \tag {1}
$$
For each threshold level  $r_0$ ,  $\tau_0$  represents the portion of samples whose confidence is lower than that threshold, i.e., the amount of the samples that are delegated to the expert.  $\mathbb{E}[c|r \geq r_0]$  corresponds to the expected value of the correctness score  $c$  for all the samples for which the network's confidence is equal or larger than  $r_0$ , i.e., accuracy among the samples for which network prediction will be used. This expected value, i.e., y-axis, can be computed as
$$
\mathbb {E} [ c | r \geq r _ {0} ] = \frac {1}{1 - \tau_ {0}} \int_ {r _ {0}} ^ {1} \mathbb {E} [ c | r ] p (r) d r. \tag {2}
$$
We provide the derivation of Eq. 2 in the Appendix A.
3.2.2 Area under COC curve
Like the area under ROC curve, area under COC curve (AUCOC) is a global indicator of the performance of a system. Higher AUCOC indicates lower number of samples delegated to human experts or/and higher accuracy for the samples that are not delegated to human experts but analysed only by the network. Lower AUCOC on the other hand, indicates higher number of delegations to human experts or/and lower accuracy on the samples analysed only by the network. Further explanation is reported in Appendix I. It can be computed by integrating the COC curve over the whole range of  $\tau_0\in [0,1]$ :
$$
A U C O C = \int_ {0} ^ {1} \mathbb {E} [ c | r \geq r _ {0} ] d \tau_ {0} = \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \mathbb {E} [ c | r ] p (r) d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}} \tag {3}
$$
Further details are provided in Section 3.5
3.3 AUCOCLoss: Maximizing AUCOC for training neural networks
As mentioned, in human-AI collaboration systems, higher AUCOC means a better model. Therefore, in this section we introduce a new loss function called AUCOCLoss that maximizes AUCOC for training classification neural networks.
AUCOC's explicit maximization would enforce the reduction of the number of samples delegated to human expert while maintaining the accuracy level on the samples assessed only by the algorithm (i.e., keeping  $\mathbb{E}[c|r\geq r_0]$  constant) and/or the improvement in the prediction accuracy of the samples analysed only by the algorithm while maintaining a particular amount of data to be delegated to the human (i.e., keeping  $\tau_0$  constant), as illustrated in Figure 1a.
We define our loss function to maximize AUCOC as
$$
A U C O C L o s s = - \log (A U C O C). \tag {4}
$$
We use the negative logarithm as AUCOC lies in the interval  $[0, 1]$ , which corresponds to AUCOCLoss  $\in [0, \inf]$  which is suitable for minimizing cost functions. The dependence of AUCOC on the network parameters may not be obvious from the formulation given in Eq. 3. Indeed, this dependence is hidden in how  $p(r)$  and  $\mathbb{E}[c|r]$  are estimated as we show next.
3.3.1 Kernel density estimation for AUCOC
We need to formulate AUCOC in a differentiable way in order for it to be incorporated in a cost function for the training of a neural network. For this purpose, we use kernel density estimation (KDE) on confidence predictions  $r_n$  for training samples to estimate  $p(r)$  used in Eq. 3
$$
p (r) \approx \frac {1}{N} \sum_ {n = 1} ^ {N} K \left(r - r _ {n}\right) \tag {5}
$$
where  $K$  is a Gaussian kernel and we choose its bandwidth using Scott's rule of thumb [35]. Then, the other terms in Eq. 3, namely  $\mathbb{E}[c|r]p(r)$  and  $\tau_0$ , are estimated as
$$
\mathbb {E} [ c | r ] p (r) \approx \frac {1}{N} \sum_ {n = 1} ^ {N} c _ {n} K (r - r _ {n}), \quad \tau_ {0} \approx \frac {1}{N} \int_ {0} ^ {r _ {0}} \sum_ {n = 1} ^ {N} K (r - r _ {n}) d r. \tag {6}
$$
Note that  $r_n = \max_{y\in \mathcal{Y}}f_\theta (y|x_n)$ , from where the dependency of AUCOC on  $\theta$  stems. Further, note that  $\tau_0$  is the x-axis of the COC curve. The AUCOC is defined by integrating over the  $\tau_0$  values, and therefore  $\tau_0$  should not depend on the network parameters, i.e., its derivative with respect to  $\theta$  should be equal to zero. We can write the derivative using Leibniz integral rule as follows:
$$
\frac {d \tau_ {0}}{d \theta} = \int_ {0} ^ {r _ {0}} \frac {d p (r)}{d \theta} d r + p \left(r _ {0}\right) \frac {d r _ {0}}{d \theta} = 0 \tag {7}
$$
Then, the constraint that  $\tau_0$  should not depend on  $\theta$  can be enforced explicitly by deriving the derivative  $dr_0 / d\theta$  from Eq. 7 as follows:
$$
\frac {d r _ {0}}{d \theta} = - \frac {\int_ {0} ^ {r _ {0}} \frac {d p (r)}{d \theta} d r}{p \left(r _ {0}\right)} \tag {8}
$$
where  $p(r)$  is implemented as in Eq. 5. Derivations are provided in the Appendix.
(b)
Figure 1: (a) shows how to improve AUCOC, 1) increasing the accuracy of the network and/or 2) decreasing the amount of data to be analysed by the domain expert. The pink curve has higher AUCOC than the blue one. (b) illustrates a toy example where two models have the same accuracy, ECE with 5 bins and KS. However, they have different AUCOC values due to different ordering of correctly and incorrectly classified samples according to the assigned confidence by the network.
3.4 Toy example: added value by AUCOC
In this section, we demonstrate the added value of assessing the performance of a predictive model using COC curve and AUCOC through a toy example. We particularly compare with the widely used expected calibration error (ECE) [8], a binning-free calibration metric called Kolmogorov-Smirnov (KS) [9] and classification accuracy.
Assume we have two classification models  $f_{\theta_1}$  and  $f_{\theta_2}$  and they yield confidence scores and predictions for 5 samples as shown in Figure 1b. The green circles denote the predicted confidence for correctly classified samples, while the red crosses the confidence of the misclassified ones.
ECE: ECE divides the confidence space into bins, computes the difference between the average accuracy and confidence for each bin, and returns the average of the differences as final measure of calibration error. If we divide the confidence space into 5-bins, as indicated with the gray dotted lines in the confidence spaces of  $f_{\theta_1}$  and  $f_{\theta_2}$ , ECEs computed for the both models will be identical. Furthermore, a similar situation can be constructed for any number of bins.
KS: KS is a binning-free metric, so it is less prone to binning errors than ECE. However, one can prove that there exist some confidence configurations for which the two models also report the same
KS, in spite of it being a binning-free metric. This happens, for example, if the confidence values are (from left to right): 0.45, 0.55, 0.65, 0.70, 0.75.
Accuracy: These models have equal classification accuracy, 3/5 for both.
Therefore, looking at these three performance metrics, it is not possible to choose one model over the other since  $f_{\theta_1}$  and  $f_{\theta_2}$  perform identically.
AUCOC: On the contrary, the AUCOC is larger for  $f_{\theta_1}$  than for  $f_{\theta_2}$ , as shown in Figure 1b. The difference in AUCOC is due to the different ranking of correctly and incorrectly classified samples with respect to confidence values. It does not depend on a particular binning nor exact confidence values, but only on the ranking. By looking at the AUCOC results, one would prefer  $f_{\theta_1}$  compared to  $f_{\theta_2}$ . Indeed,  $f_{\theta_1}$  is a better model than  $f_{\theta_2}$  because it achieves either equal or better accuracy than  $f_{\theta_2}$ , for the same amount of data to be manually examined. Changing point of view,  $f_{\theta_1}$  delegates either equal or lower number of samples to experts for the same accuracy level.
Comparison of AUCOC and calibration: As also demonstrated with the toy example, AUCOC and calibration assess different aspects of a model. The former is a rank-based metric with respect to the ordering of the predictive confidences of correctly and incorrectly classified samples. The latter instead assess the consistency between the accuracy of the network and the predictive confidences, i.e. it is sensitive to the numerical values of the confidences themselves.
3.5 Implementation Details
Construction of the COC curve, thresholds  $r_0$  and operating points: To provide flexibility in the selection of COC operating points, we need to cover the entire range [0, 1] of  $\tau_0$  values. As a consequence, the thresholds  $r_0$  need to span the confidence range of the predictions  $r_n$  of the neural network. A natural choice for such thresholds is employing the predictions  $r_n$  themselves. This spares us from exploring the confidence space with arbitrarily fine-grained levels. First, we sort the confidences  $\mathbf{r}$  of the whole dataset (or batch) in ascending order. Each predicted confidence is then selected as threshold level  $r_0$  for the vector  $\mathbf{r}$ , corresponding to a certain  $\tau_0$  (x-value). Subsequently,  $\mathbb{E}[c|r \geq r_0]$  (y-value) is computed. Note that setting the threshold to each  $r_n$  in the sorted array corresponds to going through  $\tau_0 = [1/N, 2/N, \ldots, (N-1)/N, 1]$  for  $N$  samples one by one in order.
Modelling correctness: Instead of using  $\mathbb{E}[c|r]p(r)\approx \frac{1}{N}\sum_{n = 1}^{N}c_nK(\| r - r_n\|)$  as given in Eq. 6, we approximate it as  $\mathbb{E}[c|r]p(r)\approx \frac{1}{N}\sum_{n = 1}^{N}r^{*}K(\| r - r_{n}\|)$  where  $r_n^* = f_\theta (y_n|x_n)$  is the confidence of the correct class for a sample  $n$ . The main reason is that the gradient of the misclassified samples becomes zero because  $c_{n}$  is zero when a sample  $x_{n}$  is not classified correctly. To deal with this issue we replace the correctness score  $c_{n}$ , which can be either 0 or 1, with  $r_n^*$  which can take continuous values between 0 and 1, following Yin et al. [39]. With this new approximation, we can back-propagate through misclassified samples and we found that this leads to better results.
Use as secondary loss: We observed in our experiments that using AUCOCLoss alone to train a network leads to very slow convergence with the existing optimization techniques. We believe this is due to the fact that the AUCOCLoss is a  $-\log(\sum_{n} z_{n})$  with  $z_{n} \in [0,1]$ . Optimization of such a form with gradient descent is slow because the contribution of increasing low  $z_{n}$ 's to the loss function is small. In contrast, cross-entropy is a  $-\sum_{n} \log z_{n}$ , where contribution of increasing low  $z_{n}$ 's to the loss is much larger, hence gradient-based optimization is faster. One can in theory create an upper bound to AUCOC loss by pulling the log inside the sum, as we show in the Appendix. However, when  $z_{n} \in [0,1]$  this upper bound is very loose and minimizing the upper bound not necessarily correspond to minimizing the AUCOC loss, hence does not maximize AUCOC. On the contrary, when AUCOCLoss is complements a primary cost that is faster to optimize, such as cross-entropy, it is improves over the primary loss and lead to the desired improved AUCOC while preserving the accuracy. This is obtained within the same amount of epochs and without ad-hoc fine-tuning the training hyper-parameters for the secondary loss.