SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} - \log \left(\int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} r _ {n} ^ {*} K (| | r - r _ {n} | |) d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}}\right) \leq \\ - \int_ {0} ^ {1} \left\{\int_ {r _ {0}} ^ {1} \log \left(r _ {n} ^ {*}\right) K \left(\left| \left| r - r _ {n} \right| \right|\right) d r \right\} \frac {d \tau_ {0}}{1 - \tau_ {0}} \\ \end{array}
$$
In CIFAR100 experiments, we followed Karandikar et al. [16] and used Wide-Resnet-28-10 [41] as the network architecture. We trained the models for 200 epochs, using Stochastic Gradient Descent (SGD), with batch of 512, momentum of 0.9 and an initial learning rate of 0.1, decreased after 60, 120, 160 epochs by a factor of 0.1. We set these parameters based on the best validation performance of CE and we keep it for all the losses. In Tiny/ImageNet experiments, we used ResNet-50 [10] as backbone architecture, SGD as optimiser with a batch size of 512, momentum of 0.9 and base learning rate of 0.1, divided by 0.1 at 40th and 60th epochs as in Mukhoti et al. [26] In DermaMNIST, RetinaMNIST and TissueMNIST [38] experiments, we followed the training procedures of the original paper, employing a ResNet-50 He et al. [10], Adam optimizer. The batch size is set to 128 for DermaMNIST and RetinaMNIST and to 512 for the larger TissueMNIST. We used the initial learning rate 0.0001 for DermaMNIST and 0.001 for RetinaMNIST and TissueMNIST, and trained the models for 100 epochs by reducing the learning rate by 0.1 after epochs 50 and 75.
All the models have been trained using either the NVIDIA GeForce RTX 2080 Ti or NVIDIA GeForce RTX 3090. The datasets have been split as follows.
For CIFAR100 we used 45000/5000/10000 images respectively as training/validation/test sets.
Tiny-ImageNet is a subset of ImageNet with  $64 \times 64$  images and 200 classes. We employed 90000/10000/10000 images as training/validation/test set, respectively.
DermaMNIST is composed of dermatoscopic images with 7007/1003/2005 samples for training, validation and test set, respectively, categorised in 7 different diseases.
RetinaMNIST is based on the DeepDriD24 challenge, which provides a dataset of 1,600 retina fundus images. The task is ordinal regression for 5-level grading of diabetic retinopathy severity. Consistently with the original paper, we split the source training set with a ratio of  $9:1$  into training and validation set, and use the source validation set as the test set.
D Results on a third medical dataset, TissueMNIST
Tables 5 and 6 report results on an additional medical dataset, TissueMNIST. The dataset contains 236,386 human kidney cortex cells, segmented from 3 reference tissue specimens and organized into 8 categories. We split the source dataset with a ratio of  $7:1:2$  into training, validation and test set. The results are consistent with those reported in the main paper.
Table 5: Test results on TissueMNIST for AUCOC and accuracy. The last two columns report  $\tau_0$  corresponding to  $90\%$  and  $95\%$  accuracy. In bold and underlined respectively the best and second best results for each metric.
Loss funct.	AUCOC ↑	Acc. ↑	τ0 ↓ @ acc.
90%	95%
CE	65,91	83,34	68,60	81,95
FL (γ=3)	66,07	82,77	79,41	87,82
AdaFL53	66,15	83,15	75,01	89,90
CE+MMCE	66,52	84,12	68,81	86,84
FL+MMCE	66,30	82,85	68,96	81,70
CE+S-AvUC	62,42	80,88	73,00	86,03
FL+S-AvUC	62,75	79,75	74,79	84,74
CE+S-ECE	65,58	83,53	70,67	83,84
FL+S-ECE	65,20	82,59	70,90	84,06
CE+AUCOCL	67,16	84,54	68,60	77,80
FL+AUCOCL	67,04	83,46	67,21	81,55
Table 6: Test results on TissueMNIST for calibration: expected calibration error (ECE), KS score, Brier score and class-wise ECE (cwECE), post TS. In bold and underlined respectively the best and second best results for each metric. Noticeably, AUCOCL performs comparably to the baselines, even though it does not aim at improving calibration explicitly.
Loss funct.	ECE↓	KS↓	Brier↓	cwECE↓
CE	0,89	0,36	46,11	1,65
FL (γ=3)	2,06	0,67	46,50	1,58
AdaFL53	2,10	1,11	46,58	1,79
CE+MMCE	0,98	0,53	45,27	1,83
FL+MMCE	1,55	0,57	45,87	1,21
CE+S-AvUC	2,28	1,87	50,32	1,62
FL+ S-AvUC	2,49	1,83	50,99	2,00
CE+S-ECE	2,38	0,98	52,88	2,71
FL+S-ECE	2,45	1,40	47,29	1,83
CE+AUCOCL	1,07	0,71	44,41	1,29
FL+AUCOCL	3,56	0,68	45,54	1,47
E Class imbalance experiments
In Table 7 we report the results for accuracy and AUCOC on the widely employed Long-Tailed CIFAR100. We employed a tunable data imbalance ratio (Nmax / Nmin, where N is number of samples in each class), which controls the class distribution in the training set and trained the models with three levels of imbalance ratio, namely 100, 50, 10. Noticeably, when AUCOCLoss complements cross-entropy it obtains the best results for both metrics in all the three settings, with higher benefits at increasing imbalance, while with FL it obtains always the second best AUCOC and comparable accuracy.
Table 7: Accuracy and AUCOC results on CIFAR100 Long-Tailed for 3 degrees of class imbalance in the training set, namely 100, 50 and 10. In bold and underlined respectively the best and second best results for each metric.
Imbalance ratio	100	50	10
Loss funct.	AUCOC ↑	Acc. ↑	AUCOC ↑	Acc. ↑	AUCOC ↑	Acc. ↑
CE	72,33	47,27	78,02	53,83	89,59	71,15
FL (γ=3)	73,66	47,00	79,53	53,50	91,91	71,59
AdaFL53	73,72	47,07	79,55	53,55	92,05	71,61
CE+MMCE	72,92	47,80	78,54	53,45	90,66	71,32
FL+MMCE	73,49	46,78	79,50	53,32	91,70	71,71
CE+S-AvUC	73,08	46,62	76,15	54,45	91,72	71,60
FL+S-AvUC	72,68	46,62	78,51	53,11	89,75	72,11
CE+S-ECE	73,53	47,58	79,31	53,84	91,63	71,67
FL+S-ECE	72,30	46,13	78,56	52,84	91,29	70,6
CE+AUCOCL	75,85	49,63	81,38	55,89	92,59	72,40
FL+AUCOCL	74,51	46,91	79,72	53,64	92,21	71,92
F Calibration results for Tiny-ImageNet and CIFAR100
In Table 8 we report calibration results respectively for CIFAR100 and Tiny-ImageNet. For all the metrics and datasets, AUCOCLoss provides comparable results with respect to the baselines.
Table 8: Test results on CIFAR100 and Tiny-Imagenet for calibration metrics, namely expected calibration error (ECE), KS score, Brier score and class-wise ECE (cwECE), post temperature scaling. In bold and underlined respectively the best and second best results for each metric. Noticeably, AUCOCL performs comparably to the baselines, even though it does not aim at improving calibration explicitly.
Dataset	CIFAR100	Tiny-Imagenet
Loss funct.	ECE↓	KS↓	Brier↓	cwECE↓	ECE↓	KS↓	Brier↓	cwECE↓
CE	2,41	1,1	33,91	0,211	1,54	0,74	66,25	0,163
FL (γ=3)	1,92	1,46	31,16	0,184	1,46	1,14	65,91	0,152
AdaFL53	1,47	1,22	31,3	0,183	1,35	0,85	65,71	0,159
CE+MMCE	2,32	1,23	34,6	0,209	1,79	0,93	66,42	0,157
FL+MMCE	1,92	1,82	31,24	0,181	1,94	1,63	65,49	0,151
CE+S-AvUC	3,23	0,53	31,71	0,199	2,13	1,15	65,20	0,146
FL+S-AvUC	1,83	0,53	31,74	0,177	1,51	1,13	64,62	0,15
CE+S-ECE	3,85	1,3	31,85	0,197	1,66	0,81	66,14	0,15
FL+S-ECE	1,74	1,56	32,19	0,180	2,6	2,37	66,40	0,161
CE+AUCOCL	1,65	0,75	29,78	0,184	1,65	1,29	64,23	0,157
FL+AUCOCL	1,34	0,95	30,24	0,175	1,86	1,29	64,45	0,155
G Results for varying batch size
In Table 9 we report the results on DermaMNIST with reduced batch sizeon which KDE is applied on, i.e. 64 and 28 samples per batch. The results of AUCOCLoss are not significantly affected by it, when complementing cross-entropy and focal loss.
Table 9: Test results on DermaMNIST for accuracy, AUCOC and ECE for batch sizes 64 and 32. Reducing the batch size does not affect significantly the performance of AUCOCL.
Batch	Loss funct.	AUCOC ↑	Acc. ↑	ECE ↓
64	CE+AUCOCL	91,16	75,18	11,00
FL+AUCOCL	91,13	74,71	9,35
32	CE+AUCOCL	90,14	73,80	15,50
FL+AUCOCL	91,10	73,47	6,69
H Additional results for CIFAR100 and Tiny-ImageNet
In Table 10 we report test results on CIFAR100 using ResNet-50 and Tiny-Imagenet using WideResnet-28-10, run for one seed. Best and second best results are in bold and underlined. They show consistency with the main experiments reported in the paper.
Table 10: Test results on CIFAR100 using ResNet-50 and Tiny-Imagenet using WideResnet-28-10, run for one seed. Best and second best results are in bold and underlined.
Dataset	CIFAR100	Tiny-ImageNet
Loss funct.	AUCOC ↑	Acc. ↑	τ0 ↓ @ acc.	AUCOC ↑	Acc. ↑	τ0 ↓ @ acc.
90%	95%	65%	75%
CE	91,10	73,69	34,72	49,42	74,03	49,29	34,74	52,28
FL (γ=3)	92,68	75,13	32,30	46,40	75,05	49,56	34,17	51,72
AdaFL53	92,65	74,5	32,01	45,78	75,36	49,49	33,77	50,76
CE+MMCE	92,55	74,75	31,08	44,53	74,12	48,56	35,83	52,48
FL+MMCE	92,46	74,47	31,69	46,63	74,79	49,32	34,57	52,12
CE+S-AvUC	92,63	74,12	31,02	44,83	74,14	49,13	35,00	53,17
FL+S-AvUC	92,71	74,82	31,11	44,51	74,21	49,02	35,22	53,12
CE+S-ECE	92,57	74,27	31,38	45,4	75,62	49,91	31,88	49,55
FL+S-ECE	92,63	75,00	31,35	47,42	74,57	49,16	35,60	52,36
CE+AUCOCL	93,81	75,94	28,39	42,61	76,64	51,40	29,97	47,65
FL+AUCOCL	93,19	75,90	29,45	43,11	75,86	50,84	32,00	50,12
I Further explanation on how improve AUCOC
There are two factors which contribute to an increase in AUCOC: decrease in the number of samples delegated to human experts (given the same network accuracy) and increase in the accuracy for the samples that are not delegated but analysed only by the network (given the same human workload).
These two aspects could manifest either individually, if the AUCOC improvement is generated by just a shift "up" or "left" of COC, or in a combined way. The example provided in Figure 1a shows an improvement in both axes ("and" case) and the proposed loss function does not favour one specific behaviour. Figure 2 provides an example of shifts "up" and "left" ("or" cases). From the AUCOC metrics alone, it is not possible to infer which mechanism is taking place.
Figure 2: Toy example of possible AUCOC increase by shifting "up" or "left" the blue COC curve.
J OOD results on all the perturbations on CIFAR100-C
In table 11 we report the average results on CIFAR100-C for all the perturbations available from Hendrycks and Dietterich [12].
Table 11: Test AUROC(%) on OOD detection, training on CIFAR100 and testing on CIFAR100-C (all 15 perturbations) for MSP, ODIN, MaxLogit and EBM. Best and second best results are in bold and underlined.
Dataset Loss funct.	C100-c all corruptions AUROC↑
MSP	ODIN	MaxLogit	EBM
CE 66,18	66,42	67,71	67,71
FL (γ=3)	66,57	66,36	66,69	66,42
AdaFL53	66,58	66,33	66,94	66,73
CE+MMCE	66,12	66,22	66,34	66,23
FL+MMCE	66,92	66,14	66,97	66,60
CE+S-AvUC	67,29	67,45	67,93	67,96
FL+S-AvUC	66,78	66,00	67,38	67,24
CE+S-ECE	67,32	67,50	67,98	68,01
FL+S-ECE	67,34	65,38	67,38	67,78
CE+AUCOCL	67,77	67,77	69,51	69,69
FL+AUCOCL	68,00	66,83	69,52	69,46