SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we present our experimental evaluations on multi-class image classification tasks. We performed experiments on five datasets. We experimented with CIFAR100 [19] and Tiny-ImageNet, a subset of ImageNet [4], following the literature on network calibration. Further, we used two publicly
available medical imaging datasets, DermaMNIST and RetinaMNIST [38], since medical imaging is an application area where expert time is limited and expensive. Due to space reasons we report results on a third medical dataset, TissueMNIST [38], in Appendix D, as well as information about the datasets in Appendix C.
Loss functions: We compared AUCOCLoss (referred to as AUCOCL in the tables) with different loss functions, most of which are designed to improve calibration performance while preserving accuracy: cross-entropy (CE), focal-loss (FL) [23], adaptive focal-loss (AdaFL) [26], maximum mean calibration error loss (MMCE) [21], soft binning calibration objective (S-ECE) and soft accuracy versus uncertainty calibration (S-AvUC) [16]. We optimized MMCE, S-ECE, and S-AvUC losses jointly with a primary loss for which we used either CE or FL, consistently with the literature [21; 16]. The same is done for AUCOCLoss, applying KDE batch-wise during the training.
Evaluation metrics: To evaluate the performance of the methods, we used classification accuracy and AUCOC. Classification accuracy is simply the ratio between the number of correct samples over the total number of samples, and AUCOC is computed using Eq. 3. We also report some examples of operating points of COC curve - given a certain accuracy, we show the corresponding expert load  $(\tau_0@acc)$ , i.e., percentage of samples that need to be analyzed manually, on the COC curves.
Since we compared AUCOCLoss mostly with losses for network calibration, we also assessed the calibration performance of the networks, even though calibration is not a direct goal of this work. To this end, we used the following metrics: the widely employed equal-mass expected calibration error (ECE) [28] with 15 bins, the binning-free Brier score [1] and Kolmogorov-Smirnov (KS) score [9] and the class-wise ECE (cwECE) [20]. The evaluation is carried out post temperature scaling (TS) [8], as it has been proved to be always beneficial for calibration.
Hyperparameters: In addition to the common hyperparameters for all losses, selected as specified in Appendix C, there are also specific ones that need to be tuned for some of them. In this case, we used the best hyperparameter settings reported in the original papers. In cases where the original paper did not report the specific values, we carried out cross-validation and selected the setup that provided the best performance on the validation set. We also selected the weighting factor for AUCOCLoss in the same way. We found that optimal weighting values for AUCOCLoss all fell between 1 and 10. We found empirically that models check-pointed using ECE provided very poor results. Networks check-pointed using either accuracy or AUCOC provided comparable outcome with respect to accuracy, therefore we reported results on AUCOC as they provided the best overall performance.
OOD experiments: We evaluate the out-of-distribution (OOD) samples detection performance of all methods since OOD detection is crucial for a reliable AI system and it is a common experiment in the network calibration literature. The commonly used experimental setting in this literature is using CIFAR100-C [12] (we report results for CIFAR100 with Gaussian noise in the main paper and the average over all the perturbations in the Appendix) and SVHN [27] as OOD datasets, while the network is trained on CIFAR100 (in-distribution). We evaluated the OOD detection performance of all methods using Area Under the Receiver Operating Characteristics (AUROC) curve, with MSP [13], ODIN [14], MaxLogit [11] and EBM [25].
Class imbalance experiments: Finally, in Appendix E we report results for accuracy and AUCOC on imbalanced datasets, being class imbalance present in many domains, such as medical imaging. We report results on the widely used Long-Tailed CIFAR100 (CIFAR100-LT) [37; 42] with controllable degrees of data imbalance ratio to control the distribution of training set. We trained with three levels of imbalance ratio, namely 100, 50, 10.
Further details on hyper-parameter settings are provided in the Appendix.
First, we report the results for accuracy and COC-related metrics. Then, we report results for calibration, even though this is not an explicit goal of this work. Bold results indicate the methods that performed best for each metric, underlined results are the second best.  $\uparrow$  means the higher the better for a metric, while  $\downarrow$  the lower the better. The experiments results are averaged over three runs.
Table 1: Test results on the natural datasets CIFAR100 and Tiny-Imagenet. We report AUCOC, accuracy for both,  $\tau_0$  at  $90\%$  and  $95\%$  accuracy for CIFAR100 and  $\tau_0$  at  $65\%$  and  $75\%$  accuracy for Tiny-ImageNet, as the initial accuracy is also lower. In bold the best result for each metric, underlined the second best. AUCOCL improves, to varying degrees, the baselines in all metrics.
Dataset	CIFAR100	Tiny-ImageNet
Loss funct.	AUCOC ↑	Acc. ↑	τ0 ↓ @ acc.	AUCOC ↑	Acc. ↑	τ0 ↓ @ acc.
90%	95%	65%	75%
CE	91,43	75,71	29,03	44,61	72,56	47,39	39,88	56,29
FL (γ=3)	93,91	77,83	24,71	40,48	73,12	47,71	38,40	55,08
AdaFL53	93,89	77,64	25,08	40,74	73,19	47,81	38,56	55,20
CE+MMCE	92,42	75,35	30,01	44,71	72,47	47,11	39,94	56,70
FL+MMCE	93,90	77,78	25,62	40,90	73,51	48,03	37,83	55,15
CE+S-AvUC	93,99	77,65	24,62	45,00	72,92	47,89	38,28	54,91
FL+S-AvUC	93,97	77,64	25,97	40,82	74,30	48,69	35,83	53,21
CE+S-ECE	93,88	77,57	24,76	40,14	72,94	47,65	38,81	55,84
FL+S-ECE	93,41	76,69	28,13	43,29	72,61	47,40	39,94	56,71
CE+AUCOCL	94,49	78,94	21,60	36,73	74,56	49,10	34,78	52,01
FL+AUCOCL	94,18	78,31	23,50	36,68	74,30	49,19	34,85	53,15
Table 2: Test results on the medical datasets DermaMNIST and RetinaMNIST. We report AUCOC, accuracy for both,  $\tau_0$  at  $90\%$  and  $95\%$  accuracy for DermaMNIST and  $\tau_0$  at  $65\%$  and  $75\%$  accuracy for RetinaMNIST, as the initial accuracy is also lower. In bold the best result for each metric, underlined the second best. AUCOCL improves, to varying degrees, the baselines in all metrics.
Dataset	DermaMNIST	RetinaMNIST
Loss funct.	AUCOC↑	Acc. ↑	τ0↓@acc.	AUCOC↑	Acc. ↑	τ0↓@acc.
90%	95%	65%	75%
CE	89,84	71,59	43,51	56,21	71,45	52,10	39,53	68,58
FL (γ=3)	90,50	72,64	40,63	53,90	68,57	52,25	44,25	58,25
AdaFL53	90,11	73,10	40,78	55,86	68,85	48,58	48,67	62,00
CE+MMCE	89,71	70,99	45,82	58,07	69,18	48,50	42,92	69,13
FL+MMCE	89,34	71,72	46,81	59,10	67,08	50,33	47,50	78,91
CE+S-AvUC	89,67	71,51	43,04	57,09	68,15	51,42	42,71	62,29
FL+S-AvUC	89,42	71,04	45,47	58,69	66,80	52,00	45,58	70,54
CE+S-ECE	89,54	71,46	43,48	57,82	71,40	52,05	39,45	55,83
FL+S-ECE	90,22	72,62	40,95	57,46	70,49	51,33	42,42	79,91
CE+AUCOCL	90,87	74,30	39,01	52,70	72,47	53,10	38,33	53,81
FL+AUCOCL	91,35	74,80	37,30	53,90	72,31	53,58	39,42	56,12
In Tables 1, 2 we present our results based on accuracy and AUCOC. The results for TissueMNIST can be found in the Appendix. We observed that complementing CE and FL with our loss, i.e.,  $\mathrm{CE + AUCOCL}$  and  $\mathrm{FL + AUCOCL}$ , they were consistently better than the other losses in all experiments. To evaluate the significance of the accuracy and AUCOC results, we performed permutation test [30] between the best AUCOCLoss and the best baseline with 1000 rounds. In all the cases it provided a p score  $\ll 1\%$ , therefore the differences in the models are steadily significant. The advantage of our model was even more apparent in amount of expert loads corresponding to specific accuracy levels. In particular, we measured the percentage of samples delegated to expert  $(\tau_0)$  at  $90\%$  and  $95\%$  accuracy for CIFAR100, DermaMNIST and TissueMNIST, and at  $65\%$  and  $75\%$  accuracy for Tiny-Imagenet and RetinaMNIST (as the initial accuracy is also much lower on these datasets). In all the experiments, to varying degrees, AUCOCLoss provided lower delegated samples than the baselines. Noticeably, while some of the baselines may be close to the results of AUCOCLoss for certain metrics, none of them were consistently close to AUCOCLoss across all the datasets. For example, when looking at the last two columns for CIFAR100 and Tiny-ImageNet respectively of Table 1, CE+MMCE is worse by  $4\%$  than AUCOCLoss on Tiny-ImageNet, but by around  $9\%$  in
Table 3: Test results on DermaMNIST and RetinaMNIST for calibration: expected calibration error (ECE), KS score, Brier score and class-wise ECE (cwECE), post TS. In bold and underlined respectively are the best and second best results for each metric. Noticeably, AUCOCL performs comparably to the baselines, even though it does not aim at improving calibration explicitly.
Dataset	DermaMNIST	RetinaMNIST
Loss funct.	ECE↓	KS↓	Brier↓	cwECE↓	ECE↓	KS↓	Brier↓	cwECE↓
CE	3,07	2,15	38,11	2,22	8,42	6,15	59,75	6,01
FL (γ=3)	4,24	1,77	36,55	2,03	13,61	10,87	63,32	8,19
AdaFL53	3,88	1,95	36,19	1,66	13,63	11,12	65,64	8,47
CE+MMCE	3,59	2,73	38,96	2,20	11,58	9,75	62,18	6,87
FL+MMCE	3,65	2,29	38,91	2,34	12,49	10,54	65,72	7,81
CE+S-AvUC	3,79	2,69	38,28	2,21	8,37	7,01	64,36	6,91
FL+S-AvUC	3,48	2,13	37,98	1,82	11,60	5,86	64,06	7,12
CE+S-ECE	3,23	2,73	38,39	1,92	9,44	5,88	59,84	5,37
FL+S-ECE	4,5	2,67	36,91	2,08	12,52	10,05	61,16	6,08
CE+AUCOCL	5,70	1,56	36,12	1,78	8,15	4,47	58,69	4,46
FL+AUCOCL	5,10	2,04	35,46	2,04	10,77	8,84	60,52	5,32
Table 4: Test AUROC(%) on OOD detection, training on CIFAR100 and testing on CIFAR100-C (Gaussian noise) and SVHN, using MSP, ODIN MaxLogit and EBM. Best and second best results are in bold and underlined.
Dataset	C100-C AUROC↑	SVHN AUROC↑
Loss funct.	MSP	ODIN	MaxLogit	EBM	MSP	ODIN	MaxLogit	EBM
CE	74,37	75,22	74,51	66,18	77,42	79,42	67,71	67,71
FL (γ=3)	75,12	75,35	73,43	72,83	76,61	77,08	66,69	66,42
AdaFL53	74,53	74,68	71,29	70,58	80,3	81,28	66,94	66,73
CE+MMCE	74,67	74,53	70,67	70,17	75,79	77,39	66,34	66,23
FL+MMCE	74,42	74,39	70,01	68,70	77,57	77,41	66,97	66,60
CE+S-AvUC	73,63	73,62	72,16	72,05	78,04	78,90	67,93	67,96
FL+S-AvUC	72,78	76,72	69,53	68,51	79,68	79,68	67,38	67,24
CE+S-ECE	73,29	73,18	71,59	71,36	77,02	77,69	67,98	68,01
FL+S-ECE	74,29	73,97	71,51	70,26	79,68	81,17	67,38	67,78
CE+AUCOCL	76,03	76,90	78,02	78,30	82,03	83,50	69,51	69,69
FL+AUCOCL	76,51	76,82	75,14	74,68	80,51	79,35	69,52	69,46
CIFAR100. Figure 1a shows examples of COC curves. Overall, the plot of AUCOCLoss lies above all the baselines, which is a desirable behavior as it corresponds to better operating points.
Even though the proposed loss was not designed to improve calibration, it provided on par performance compared to the other cost functions particularly designed for network calibration, as reported in Table 3 for DermaMNIST and RetinaMNIST and in the Appendix for the other datasets.
In OOD experiments reported in Table 4, we used the model trained on CIFAR100 and evaluated the OOD detection performance on CIFAR100-C (with Gaussian noise) and SVHN dataset. Results for CIFAR100-C averaged over all the perturbations are reported in the Appendix. We employed state-of-the-art OOD detectors, namely MSP, ODIN, MaxLogit and EBM to determine whether a sample is OOD or in-distribution. The bold results highlight the best results in terms of AUROC. On both OOD datasets, AUCOCLoss always provided the highest AUROC and in almost all the cases also the second best.
Class imbalance experiments on CIFAR100-LT are reported in the Appendix. AUCOCLoss obtains best results for both accuracy and AUCOC, with higher benefits at increasing imbalance.
Crucially, CE+AUCOCL, where the proposed loss is used jointly with CE, outperformed every baseline in accuracy, AUCOC and  $\tau_0$  @acc. in all the experiments. It further outperformed all the baselines in OOD detection and class imbalance experiments, presented in the Appendix. Even
though the metric is not geared towards calibration,  $\mathbf{CE} + \mathbf{AUCOCL}$  yielded either the best or the second best calibration performance in the majority of the cases compared to the baselines.
In the Appendix, we explore how the performance of a model trained with AUCOCLoss varies, when changing batch size as it can be crucial for KDE-based methods. Even substantially lower batsch sizes do not have a considerable effect on the performance of the proposed method.
In this paper we proposed a new cost function for multi-class classification that takes into account the trade-off between a neural network's accuracy and the amount of data that requires manual analysis from a domain expert, by maximizing the area under COC (AUCOC) curve. Experiments on multiple computer vision and medical image datasets suggest that our approach improves the other methods in terms of both accuracy and AUCOC, where the latter was expected by design, provides comparable calibration metrics, even though the loss does not aim to improve calibration explicitly and outperforms the baselines in OOD detection.
While we presented COC and AUCOCLoss for multi-class classification, extensions to other tasks are possible future work as well as investigating different performance metrics to embed in the  $y$ -axis of COC. Moreover, aware of potential problems with KDE at the boundaries, i.e., boundary bias, we explored corrections like reflection method, which did not provide major improvements, but we will further investigate. We believe that this new direction of considering expert load in human-AI system is important and AUCOCLoss will serve as a baseline for future work.
Limitations: As described in Section 3.5, AUCOCLoss alone empirically leads to slow convergence, due to the small contribution of  $-\log(\sum_{n} z_{n})$  with  $z_{n} \in [0,1]$  in its formulation. Therefore, we recommend to use it as a secondary loss, to successfully complement existing cost functions.
This study was financially supported by: 1. The LOOP Zürich - Medical Research Center, Zurich, Switzerland, 2. Personalized Health and Related Technologies (PHRT), project number 222, ETH domain and 3. Clinical Research Priority Program (CRPP) Grant on Artificial Intelligence in Oncological Imaging Network, University of Zürich.
[1] Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review 78, 1-3.
[2] Chen, L.-C., G. Papandreou, F. Schroff, and H. Adam (2017). Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587.
[3] Davidson, T., D. Warmsley, M. Macy, and I. Weber (2017). Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM '17, pp. 512-515.
[4] Deng, J., W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255.
[5] Dvijotham, K., J. Winkens, M. Barsbey, S. Ghaisas, R. Stanforth, N. Pawlowski, P. Strachan, Z. Ahmed, S. Azizi, Y. Bachrach, et al. (2023). Enhancing the reliability and accuracy of ai-enabled diagnosis via complementarity-driven deferral to clinicians. Nature Medicine, 1-7.
[6] Esteva, A., B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun (2017). Dermatologist-level classification of skin cancer with deep neural networks. nature 542(7639), 115-118.
[7] Gorski, N., V. Anisimov, E. Augustin, O. Baret, and S. Maximov (2001). Industrial bank check processing: the a2ia checkreadertm. International Journal on Document Analysis and Recognition 3(4), 196-206.
In this Section we provide the derivation of AUCOC and its gradient formulation.
A.1 Derivation of AUCOC
AUCOC is defined as:
$$
A U C O C = \int_ {0} ^ {1} \mathbb {E} [ c | r \geq r _ {0} ] d \tau_ {0} = \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \mathbb {E} [ c | r ] p (r) d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}}
$$
The y-axis in COC curve is expressed mathematically by:
$$
\begin{array}{l} \mathbb {E} [ c | r \geq r _ {0} ] = \sum p (c | r \geq r _ {0}) c = \sum \frac {p (c , r \geq r _ {0})}{p (r \geq r _ {0})} c = \sum \frac {\int_ {r _ {0}} ^ {1} p (c , r) d r}{\int_ {r _ {0}} ^ {1} p (r) d r} c \\ = \sum \frac {\int_ {r _ {0}} ^ {1} p (c | r) p (r) d r}{\int_ {r _ {0}} ^ {1} p (r) d r} c = \frac {\int_ {r _ {0}} ^ {1} \sum p (c | r) c p (r) d r}{\int_ {r _ {0}} ^ {1} p (r) d r} = \frac {\int_ {r _ {0}} ^ {1} \mathbb {E} [ c | r ] p (r) d r}{\int_ {r _ {0}} ^ {1} p (r) d r} = \\ = \frac {\int_ {r _ {0}} ^ {1} \mathbb {E} [ c | r ] p (r) d r}{1 - \tau_ {0}} \\ \end{array}
$$
The  $x$ -axis in COC curve is expressed mathematically by:
$$
\tau_ {0} = p (r <   r _ {0}) = \int_ {0} ^ {r _ {0}} p (r) d r
$$
Using the  $\tau_0$  formulation, we can rewrite the y-axis as
$$
\mathbb {E} [ c | r \geq r _ {0} ] = \frac {\int_ {r _ {0}} ^ {1} \mathbb {E} [ c | r ] p (r) d r}{1 - \tau_ {0}}
$$
Maximizing the area under this curve, over  $\tau_0\in [0,1]$  corresponds to
$$
\max  \mathcal {A} = \max  \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \mathbb {E} [ c | r ] p (r) d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}}
$$
Let us assume to use a Gaussian kernel with this expression:
$$
K \left(\left\| r - r _ {n} \right\|\right) = \frac {1}{\sqrt {2 \pi} \alpha} \cdot \exp \left(- \frac {\left(r - r _ {n}\right) ^ {2}}{2 \alpha^ {2}}\right) \tag {9}
$$
Developing the equation, the area calculation becomes:
$$
\begin{array}{l} \mathcal {A} = \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \mathbb {E} [ c | r ] p (r) d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}} = \\ = \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbf {1} \left(c _ {n}\right) K \left(\| r - r _ {n} \|\right) d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}} = \\ = \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbf {1} \left(c _ {n}\right) \frac {1}{\sqrt {2 \pi} \alpha} \cdot \exp \left(- \frac {\left(r - r _ {n}\right) ^ {2}}{2 \alpha^ {2}}\right) d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}} = \tag {10} \\ = \int_ {0} ^ {1} \left\{\frac {1}{N} \sum_ {n = 1} ^ {N} \mathbf {1} \left(c _ {n}\right) \int_ {r _ {0}} ^ {1} \frac {1}{\sqrt {2 \pi} \alpha} \cdot \exp \left(- \frac {\left(r - r _ {n}\right) ^ {2}}{2 \alpha^ {2}}\right) d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}} = \\ = \int_ {0} ^ {1} \left\{\frac {1}{N} \sum_ {n = 1} ^ {N} \mathbf {1} \left(c _ {n}\right) \left(n d t r \left(\frac {1 - r _ {n}}{\sqrt {c o v}}\right) - n d t r \left(\frac {r _ {0} - r _ {n}}{\sqrt {c o v}}\right)\right) \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}} = \\ = \sum_ {k = 1} ^ {\# t h r e s h} \frac {f \left(\tau_ {0 , k - 1} , r _ {0 , k - 1}\right) + f \left(\tau_ {0 , k} , r _ {0 , k}\right)}{2} \left(\tau_ {0, k} - \tau_ {0, k - 1}\right) \\ \end{array}
$$
$$
\begin{array}{l} \tau_ {0, k} = \int_ {0} ^ {r _ {0, k}} p (r) d r \approx \frac {1}{N} \int_ {0} ^ {r _ {0, k}} \sum_ {n = 1} ^ {N} K (\| r - r _ {n} \|) d r = \tag {11} \\ = \frac {1}{N} \sum_ {n = 1} ^ {N} \left(n d t r \left(\frac {r _ {0 , k} - r _ {n}}{\sqrt {c o v}}\right) - n d t r \left(\frac {- r _ {n}}{\sqrt {c o v}}\right)\right) \\ \end{array}
$$
Where  $ndtr$  expresses the Gaussian cumulative distribution function and the last row in Equation 10 exploits the trapezoidal rule for integrals computation.
A.2 Derivations of the gradients of AUCOC
$$
\frac {d}{d \theta} \mathcal {A} = \int_ {0} ^ {1} \frac {d}{d \theta} \left\{\int_ {r _ {0}} ^ {1} \mathbb {E} [ c | r ] p (r) d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}}
$$
Here, we use the assumption discussed in Section Methods that  $\tau_0$  does not depend on any parameter, thus allowing us to apply Leibnitz's integration rule, obtaining:
$$
\begin{array}{l} \frac {d}{d \theta} \mathcal {A} = \int_ {0} ^ {1} \frac {d}{d \theta} \left\{\int_ {r _ {0}} ^ {1} \mathbb {E} [ c | r ] p (r) d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}} (12) \\ = \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \frac {d}{d \theta} \mathbb {E} [ c | r ] p (r) d r - \mathbb {E} [ c | r _ {0} ] p \left(r _ {0}\right) \frac {d r _ {0}}{d \theta} \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}} (13) \\ \end{array}
$$
$\tau_0$  can be expressed as:
$$
\tau_ {0} = p (r \leq r _ {0}) = \int_ {0} ^ {r _ {0}} p (r) d r \tag {14}
$$
$$
\frac {d \tau_ {0}}{d \theta} = \int_ {0} ^ {r _ {0}} \frac {d p (r)}{d \theta} d r + p \left(r _ {0}\right) \frac {d r _ {0}}{d \theta} = 0 \tag {15}
$$
$$
\frac {d r _ {0}}{d \theta} = - \frac {\int_ {0} ^ {r _ {0}} \frac {d p (r)}{d \theta} d r}{p (r _ {0})}
$$
Plugging this expression back into Equation 12 we obtain:
$$
\frac {d \mathcal {A}}{d \theta} = \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \frac {d}{d \theta} \mathbb {E} [ c | r ] p (r) d r + \mathbb {E} [ c | r _ {0} ] \int_ {0} ^ {r _ {0}} \frac {d p (r)}{d \theta} d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}} \tag {16}
$$
Assuming the use of a Gaussian kernel:
$$
K (| | r - r _ {n} | |) = \frac {1}{\sqrt {2 \pi} \alpha} \cdot \exp \left(- \frac {(r - r _ {n}) ^ {2}}{2 \alpha^ {2}}\right) \tag {17}
$$
And re-writing:
$$
\frac {\mathbb {E} [ c | r _ {0} ] p (r _ {0})}{p (r _ {0})} \approx \frac {\frac {1}{N} \sum_ {n = 1} ^ {N} \mathbf {1} \left(c _ {n}\right) K \left(\left\| r _ {0} - r _ {n} \right\|\right)}{\frac {1}{N} \sum_ {n = 1} ^ {N} K \left(\left\| r _ {0} - r _ {n} \right\|\right)} \tag {18}
$$
The gradient of the area becomes:
$$
\begin{array}{l} \frac {d \mathcal {A}}{d r _ {n}} = \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \frac {d}{d r _ {n}} \mathbb {E} [ c | r ] p (r) d r + \mathbb {E} [ c | r _ {0} ] \int_ {0} ^ {r _ {0}} \frac {d p (r)}{d r _ {n}} d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}} = \\ = \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \frac {d}{d r _ {n}} \frac {1}{\sqrt {2 \pi} \alpha} \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbf {1} \left(c _ {n}\right) \cdot \exp \left(- \frac {(r - r _ {n}) ^ {2}}{2 \alpha^ {2}}\right) d r + \right. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ c | r _ {0} ] \int_ {0} ^ {r _ {0}} \frac {d}{d r _ {n}} \frac {1}{\sqrt {2 \pi} \alpha} \frac {1}{N} \sum_ {n = 1} ^ {N} \exp \left(- \frac {(r - r _ {n}) ^ {2}}{2 \alpha^ {2}}\right) d r \} \frac {d \tau_ {0}}{1 - \tau_ {0}} = \\ = \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \frac {1}{\sqrt {2 \pi} \alpha^ {3} N} \mathbf {1} \left(c _ {n}\right) \cdot \left(r - r _ {n}\right) \cdot \exp \left(- \frac {\left(r - r _ {n}\right) ^ {2}}{2 \alpha^ {2}}\right) d r + \right. \tag {19} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ c | r _ {0} ] \int_ {0} ^ {r _ {0}} \frac {1}{\sqrt {2 \pi} \alpha^ {3} N} (r - r _ {n}) \cdot \exp \left(- \frac {(r - r _ {n}) ^ {2}}{2 \alpha^ {2}}\right) d r \} \frac {d \tau_ {0}}{1 - \tau_ {0}} = \\ = \int_ {0} ^ {1} \left\{- \frac {1}{\sqrt {2 \pi} \alpha N} \mathbf {1} \left(c _ {n}\right) \cdot \left[ \exp \left(- \frac {\left(1 - r _ {n}\right) ^ {2}}{2 \alpha^ {2}}\right) - \exp \left(- \frac {\left(r _ {0} - r _ {n}\right) ^ {2}}{2 \alpha^ {2}}\right) \right] - \right. \\ \end{array}
$$
$$
\mathbb {E} [ c | r _ {0} ] \frac {1}{\sqrt {2 \pi} \alpha N} \left[ \exp \left(- \frac {(r _ {0} - r _ {n}) ^ {2}}{2 \alpha^ {2}}\right) - \exp \left(- \frac {(- r _ {n}) ^ {2}}{2 \alpha^ {2}}\right) \right] d r \} \frac {d \tau_ {0}}{1 - \tau_ {0}}
$$
Also for the gradients in the code implementation we exploited the trapezoidal rule for the computation of the external integral between [0,1].
B Potential upper bound to AUCOCLoss
AUCOC has a formulation of the kind  $-\log (\sum_{n}z_{n})$  with  $z_{n}\in [0,1]$ . In contrast, cross-entropy is a  $-\sum_{n}\log z_{n}$ , where contribution of increasing low  $z_{n}$ 's to the loss is much larger, hence gradient-based optimization is faster. Exploiting Jensen's inequality, one could find the following upper bound of AUCOCLoss to minimise. However, from thorough experiments it has been proven not to be a tight enough bound for the optimisation to be successful. In fact, trying to optimise the rightmost term of the following equation, instead of the correct definition on the left, does not lead to a satisfactory optimisation of AUCOC.