SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To evaluate the performance of our method under various circumstances, we selected three datasets: CIFAR10, CIFAR100[31], and Imagenet[1]. We employed different models for each dataset to reflect our approach's calibration capability across various model accuracy levels<sup>2</sup>. The models used in our experiments include Resnet[2], VGG[32], Densenet[33], SWIN[34], and ShuffleNet[35].
We randomly partitioned a validation set  $D_{val}$  from the standard training set: CIFAR10 and CIFAR100 adopted  $10\%$  of the data for validation, while Imagenet utilized  $5\%$ . Additionally, we randomly sampled  $10\%$  of the hold-out data  $D_{ho}$  from the standard test set for calibration. We performed 100 different test set splits for each dataset-model combination to obtain reliable results and reported
$^{2}$ Code and Appendix are available at https://github.com/ThyrixYang/group_calibration
the average performance over 100 trials for each method. We conduct paired t-test[36] to measure the static significance of the improvements. The hyperparameters of the comparative methods were tuned based on the corresponding literature with 5-fold cross-validation on the CIFAR10-Resnet152 dataset. We fixed the number of groups at  $K = 2$  and the number of partitions at  $U = 20$ , although 20 is not necessarily the optimal value. The strength of regularization was set to  $\lambda = 0.1$ , following a similar tuning approach as the comparative methods.
3.1 Experimental comparison
Table 1: Comparison of accuracy-preserving calibration methods. We utilized bold font to highlight the statistically superior  $(p < 0.01)$  results.
Dataset	Model	Uncal	TS	ETS	IRM(AP)	GC+TS(ours)	GC+ETS(ours)
CIFAR10	Resnet152	0.0249	0.0086	0.0101	0.0115	0.0079	0.0089
CIFAR10	Shufflingenet	0.0464	0.0107	0.0103	0.0146	0.0099	0.0093
CIFAR10	VGG11	0.0473	0.0125	0.0135	0.0157	0.0120	0.0122
CIFAR100	Densenet121	0.0558	0.0418	0.0289	0.0415	0.0411	0.0280
CIFAR100	Resnet50	0.0580	0.0435	0.0292	0.0424	0.0427	0.0269
CIFAR100	VGG19	0.1668	0.0485	0.0472	0.0476	0.0414	0.0360
Imagenet	Resnet18	0.0279	0.0178	0.0104	0.0188	0.0173	0.0100
Imagenet	Resnet50	0.0382	0.0182	0.0102	0.0218	0.0174	0.0103
Imagenet	Swin	0.0266	0.0367	0.0218	0.0140	0.0366	0.0193
Average improvements	-5.03%	-8.92%
Accuracy preserving methods. We compared our approach with methods that offer accuracy assurance, including uncalibrated (Uncal), temperature scaling (TS)[7], Ensemble Temperature Scaling (ETS)[23], and multi-class isotonic regression with accuracy-preserving $^3$  (IRM(AP))[23] methods. We report the top-label ECE[25]. From Table. (1), it can be observed that our method achieves the best performance on the majority of datasets, and the improvement is statistically significant. Another noteworthy aspect is the enhancement our method demonstrates when compared to the base methods without partitioning, namely GC+TS compared to TS, and GS+ETS compared to ETS. We have provided the average improvement relative to the corresponding base methods in the last row of Table 1. Specifically, GC+TS shows an average improvement of  $5.03\%$  over TS, while GC+ETS demonstrates an average improvement of  $8.92\%$  over ETS. This indicates that employing better partitioning strategies can enhance calibration performance.
Table 2: Comparison of non-accuracy-preserving calibration methods.
Dataset	Model	Hist.B	Beta	BBQ	DirODIR	GPC	IRM(NAP)	Acc. Dec.	GC+ETS (ours)
CIFAR10	Resnet152	0.0172	0.0095	0.0097	0.0101	0.0189	0.0087	-0.079%	0.0089
CIFAR10	Shufflingenet	0.0322	0.0137	0.0139	0.0158	0.0341	0.0119	-0.050%	0.0093
CIFAR10	VGG11	0.0279	0.0150	0.0137	0.0156	0.0376	0.0119	-0.129%	0.0122
CIFAR100	Densenet121	0.0632	0.0520	0.0307	0.0533	0.0306	0.0203	-0.149%	0.0280
CIFAR100	Resnet50	0.0618	0.0550	0.0334	0.0553	0.0346	0.0422	-3.686%	0.0269
CIFAR100	VGG19	0.0453	0.0642	0.0446	0.0932	0.1406	0.0470	-5.046%	0.0360
Imagenet	Resnet18	0.0714	0.0690	0.0483	0.0386	-*	0.0119	-0.027%	0.0100
Imagenet	Resnet50	0.0502	0.0707	0.0482	0.0326	-	0.0107	-0.006%	0.0103
Imagenet	Swin	0.0335	0.0629	0.0478	0.0148	-	0.0110	-0.060%	0.0193
* GPC failed to converge within a day on Imagenet datasets.
Non-accuracy preserving methods. We also compared our method with calibration approaches that do not guarantee accuracy, including Histogram Binning(Hist.B)[7], Beta Calibration (Beta)[10], Bayesian Binning into Quantiles (BBQ)[11], Dirichlet calibration with ODIR regularisation (DirODIR)[12], multi-class isotonic regression without accuracy-preserving (IRM(NAP))[23] and Gaussian process calibration(GPC)[24]. The comparative results are presented in Table. (2). It can be observed that our method achieves improved calibration performance while maintaining the
<sup>3</sup>We found that IRM with  $\epsilon \ll 10^{-3}$  is indeed not accuracy preserving, so we use  $\epsilon = 10^{-3}$  in IRM(AP) and  $\epsilon = 10^{-9}$  in IRM(NAP). We discuss this problem in the supplementary material in detail.
same predictive accuracy. On the majority of datasets, our method either meets or surpasses the best non-accuracy-preserving methods. We have also included in Table. (2) the influence of the non-accuracy-preserving method with the lowest ECE on predictive accuracy. It is evident that these methods generally have a noticeable negative impact on model accuracy. In contrast, our proposed method preserves the predictive results, ensuring that the predictive accuracy remains unchanged. We provide all the details and codes of the experiments in the supplementary material, as well as a few additional evaluation metrics (NLL, Birer score) and methods (vector scaling, isotonic regression, etc.) [7], which also supports that our method performs best.
Figure 2: Visualization of learned grouping function on CIFAR10.
3.2 Verifying effects of grouping functions
Quantitative analysis. We conduct experiments on the constructed setting described in Figure 1 with TS and our GC+TS method. TS has ECE=0.0157, and GC+TS has ECE=0.0118. We calculate the PCE with the partitions learned by our GC method, and PCE of TS=0.0174, PCE of GC+TS=0.0141, which indicates that GC+TS does improve the overall calibration performance by minimizing PCE.
Visualization of learned partitions. To explore the partition functions learned by our method, we visualized the output results of the grouping function on the CIFAR10-Resnet152 dataset. Specifically, we calculated the proportion of each class within each group and visualized two representative partitions in Figure. 2. In the left figure, group 1 primarily consists of airplanes, cars, birds, frogs, ships, and trucks. These classes may share certain visual characteristics like blue sky and water, which implies that the model's predictions might exhibit systematic overconfidence or underconfidence within these categories. Applying a group-specific calibration function to this group can help mitigate miscalibration issues. In the right figure, group 1 mainly comprises cats, deer, and dogs, which also share some visual features. To conclude, our proposed method can implicitly uncover meaningful partitions from the data despite not directly defining a partitioning approach.
3.3 Ablation study
We conducted experiments on the CIFAR10-Resnet152 dataset to investigate the impact of three hyperparameters in our method: the number of partitions, the number of groups within each partition, and the regularization strength used in the grouping function. The results are presented in Figure. 3. We observed that as the number of groups decreases, both NLL (negative log-likelihood) and ECE generally exhibit a monotonic increase. We attribute this trend to the fact that increasing the number of groups leads to fewer data points within each group, exacerbating model overfitting. On the other hand, as the number of partitions increases, the calibration performance consistently improves. This finding suggests that training on multiple distinct partitions can enhance the model's performance across various partitions, thereby improving the model's overall performance. This can be viewed as incorporating prior knowledge into the model,
(a) Number of groups
(b) Number of partitions
indicating that the model's predictions should be well-calibrated across any partition. The influence of the regularization strength hyperparameter in the grouping function is depicted in Figure 4. It can be observed that when the regularization strength is too high (i.e., a value of 1), the calibration performance is poorer. This suggests that in such cases, the expressive power of the grouping function is insufficient to learn meaningful groups that aid in calibration. Conversely, when the regularization strength is too low, the performance also deteriorates because the grouping function's expressive power is excessive, leading to the learning of partition functions that are specific to the training set and lack generalization.
Figure 3: The influence of the number of partitions and the number of groups in each partition.
Figure 4: Influence of  $\lambda$
We provide a concise overview of recent academic research on the definition, evaluation, and advancements in calibration methods.
Measures of calibration. In safety-critical applications[3-6] and tasks involving probability estimation[37-42], it is crucial to ensure that models not only deliver high accuracy but also provide predicted probabilities that accurately represent the genuine likelihood of correctness. As discussed in Section. 2.1, calibration, being a collective concept without direct labels, is typically evaluated by binning the predicted probabilities using different binning approaches. Various studies have proposed different binning methods, each focusing on a different aspect. For example, Kumar et al. [25] proposed binning based on the maximum predicted probability, Kull et al. [12] introduced binning using probabilities for all classes, and Zhang et al. [23], Popordanoska et al. [43], Kumar et al. [44] proposed kernel methods, which can be seen as a smoothed binning approach. In our framework, these methods can be considered as applying different grouping functions, where the grouping functions solely utilize the predicted probabilities. Additionally, other studies have explored alternative characteristics of calibration. For instance, Vaicenavicius et al. [22] and Gupta et al. [14] proposed hypothesis testing methods to assess whether the predicted probabilities approximate calibration, which is still defined solely on the predicted probabilities. Proper scoring rules such as likelihoods and Brier scores are also used to evaluate calibration[26, 45], which are typically used as a training loss to improve calibration.
The calibration considering fairness[46, 47] is also highly related to this work, where the calibration within different populations is considered. Perez-Lebel et al. [48] established an explained component as a metric that lower-bounds the grouping loss in the proper scoring rule theory. However, our work concentrates on learning a better grouping function to improve holistic calibration performance, rather than calibrating some given groups[47].
Improving calibration of deep models. The calibration of deep models can be categorized into two main directions. The first direction involves calibrating the model during the training phase by modifying the training process itself[23, 26, 49-51]. This may include using specialized loss functions tailored for calibration[52], employing ensemble methods[26, 53], or data augmentation[54], etc.
However, such approaches require modifications to the training phase and may entail relatively higher computational costs.
The second direction focuses on calibrating the model during the prediction phase. These methods are highly related to the definition of calibration metrics, which involves minimizing the discrepancy between model predictions and aggregated statistics within each bin. The simplest method is histogram binning[7, 37], where predicted probabilities are directly adjusted to match the actual class proportions within each bin. BBQ[11] further proposes an ensemble approach that calibrates results obtained with different numbers of bins. The transformation functions applied to predictions can also be improved beyond averaging within bins. For example, Beta distribution can fit the probability transformation for binary classification[10], while Dirichlet distribution can be employed for multi-class transformation[12]. New families of transformation functions have been proposed to preserve the ordering of predicted probabilities while improving calibration performance[13, 23]. Our method is orthogonal to these calibration methods since the underlying motivation of existing methods is to minimize the calibration error within a single group, while our method offers greater flexibility as we can employ different calibration functions for different groups.
Some recent work also proposed calibration with given groups. Multicalibration[55] proposes an algorithm for learning a multi-calibrated predictor with respect to any given subpopulation class. Durfee et al. [56] proposed to group the feature space with decision trees, then apply Platt scaling within each group, which concentrates on calibration ranking metrics (AUC), while our work concentrates on PCE metrics and proper scoring rules (Log-Likelihood). Yuksekgönül et al. [57] and Xiong et al. [58] propose heuristic grouping functions and apply a specific calibration method within each group, while we aim to propose a learning-based method that can generate partitions in an end-to-end manner.
5 Conclusion and limitations
This paper proposes a novel approach to define model uncertainty calibration from the perspective of partitioning the input space, thereby unifying previous binning-based calibration metrics within a single framework. Additionally, we extend the notion of calibration beyond output probabilities, allowing it to be defined on semantically relevant partitions. We establish a connection between calibration and accuracy metrics using semantically relevant partitions. Furthermore, our analysis inspires introducing a method based on learning grouping functions to discover improved partitioning patterns, thus aiding in learning the calibration function. Our framework holds substantial potential for further development, such as optimizing subset selection methods and refining model design, which will be our future work.
Limitations. Increasing the number of partitions or employing more complex grouping models will result in higher computational complexity, which is a limitation of our method. Nevertheless, when compared to other Bayesian methods[11, 12, 24], our method demonstrates notably faster speed. We present analysis and experimental comparisons of computational complexities in the supplementary material. Due to the reliance of our method on the features extracted from deep models, it is not applicable for calibrating non-deep models such as tree models[59] and SVMs[12, 60].
This work is supported by the National Key R&D Program of China (2022ZD0114805).
[1] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211–252, 2015.
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[42] Da-Wei Zhou, Yang Yang, and De-Chuan Zhan. Learning to classify with incremental new class. IEEE Trans. Neural Networks Learn. Syst., 33(6):2429-2443, 2022.
[43] Teodora Popordanoska, Raphael Sayer, and Matthew B. Blaschko. A consistent and differentiable lp canonical calibration error estimator. In NeurIPS, 2022.
[44] Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In ICML, 2018.
[45] Sebastian Gruber and Florian Buettner. Better uncertainty calibration via proper scores for classification and beyond. In NeurIPS, 2022.
[46] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Comput. Surv., 54(6):115:1-115:35, 2022.
[47] Geoff Pleiss, Manish Raghavan, Felix Wu, Jon M. Kleinberg, and Kilian Q. Weinberger. On fairness and calibration. In NeurIPS, 2017.
[48] Alexandre Perez-Lebel, Marine Le Morvan, and Gaeil Varoquaux. Beyond calibration: estimating the grouping loss of modern neural networks. In ICLR, 2023.
[49] Jiacheng Cheng and Nuno Vasconcelos. Calibrating deep neural networks by pairwise constraints. In CVPR, 2022.
[50] Wesley J. Maddox, Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In NeurIPS, 2019.
[51] Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncertainty optimization. In NeurIPS, 2020.
[52] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H. S. Torr, and Puneet K. Dokania. Calibrating deep neural networks using focal loss. In NeurIPS, 2020.
[53] Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In NeurIPS, 2020.
[54] Sunil Thulasidasan, Gopinath Chennupati, Jeff A. Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In NeurIPS, 2019.
[55] Ursula Hébert-Johnson, Michael P. Kim, Omer Reingold, and Guy N. Rothblum. Multicalibration: Calibration for the (computationally-identifiable) masses. In ICML, volume 80, pages 1944–1953, 2018.
[56] David Durfee, Aman Gupta, and Kinjal Basu. Heterogeneous calibration: A post-hoc model-agnostic framework for improved generalization. CoRR, abs/2202.04837, 2022.
[57] Mert Yüksekgonül, Linjun Zhang, James Zou, and Carlos Guestrin. Beyond confidence: Reliable models should also consider atypicality. CoRR, abs/2305.18262, 2023.
[58] Miao Xiong, Ailin Deng, Pang Wei Koh, Jiaying Wu, Shen Li, Jianqing Xu, and Bryan Hooi. Proximity-informed calibration for deep neural networks. CoRR, abs/2306.04590, 2023.
[59] Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In ICML, 2001.
[60] John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.