SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Comparison of calibration errors in  $10^{-2}$  between existing calibration methods ('base') and our proximity-informed framework ('ours'), on ImageNet dataset. (* means p = 0.01)
Model	Method	ECE ↓	ACE ↓	MCE ↓	PIECE ↓
base	+ours	base	+ours	base	+ours	base	+ours
BeiT	Conf	3.6137	0.8573*	3.5464	0.7205*	1.5801	0.2866*	4.2348	1.5379*
ETS	2.1318	0.9930*	2.1862	0.9023*	1.2155	0.4333*	2.9592	1.5872*
HB	4.8765*	5.5631	6.1728	5.9747	1.8383*	4.1239	7.2174	6.2886*
MIR	0.4509	0.536	0.5376	0.5455	0.1065	0.1395	1.8039	1.2645*
PTS	1.2685	0.9787	1.2744	0.8106	0.4858	0.4240	1.9782	1.6890
PTSK	1.8861	1.0288	1.9150	0.8582*	0.7934	0.5022	2.7093	1.6168*
TS	2.9894	1.3277*	3.1132	1.2493*	0.7388	0.7046	3.5366	1.9264*
Mixer	Conf	10.9366	2.8498*	10.9337	2.7162*	5.1519	1.0944*	11.0164	3.6485*
ETS	1.9381	1.3586*	2.1859	1.2210*	0.3204	0.2986	4.1034	2.2010*
HB	9.1774	6.7207*	9.7965	7.4825*	3.5964*	4.3997	12.9240	7.8672*
MIR	1.1128	0.9272	1.2190	0.9360*	0.2628	0.1430*	3.3912	2.2112*
PTS	5.7741	2.2208	5.8027	2.1298	2.8498	0.6940	9.3215	3.0623*
PTSK	6.6610	1.9466*	6.6173	1.7905*	3.1011	0.6131*	8.3148	2.7503*
TS	5.1937	1.6499*	5.0234	1.4455*	2.0189	0.4255*	5.8958	2.4809*
ResNet50	Conf	8.7246	2.7752*	8.6852	2.6344*	4.6122	1.3005*	8.9113	3.4224*
ETS	2.7620	1.6548*	3.6581	1.6627*	0.6624	0.5676	3.5750	2.4579*
HB	7.6311	6.1812*	9.3289	7.5380*	2.6377*	4.3484	10.1849	7.7372*
MIR	1.0281	0.9533	0.9643	0.8436	0.2062	0.2095	1.9751	1.8776*
PTS	2.2196	1.0138*	2.2098	1.0278*	0.6319	0.2529	4.0331	2.0445*
PTSK	4.4100	1.6015*	4.3761	1.5015*	1.8375	0.4980	5.4413	2.5003*
TS	5.1181	1.7964*	5.0864	1.7300*	2.5503	0.6881*	5.4640	2.5721*
ViT	Conf	1.1815	0.9016*	1.1839	0.7554*	0.3489	0.2543	1.9984	1.7540*
ETS	1.1080	0.9074	1.1791	0.7732*	0.2745	0.2684	1.9273	1.7533
HB	4.6920*	6.8776	7.1771	7.5824	2.4332*	4.8925	7.3239*	7.7552
MIR	0.8934	0.8638	0.8180	0.8537	0.1893	0.2182	1.7695	1.6818
PTS	1.0609	0.7940	1.0257	0.7124	0.3937	0.2600	2.0929	1.7510*
PTSK	1.6493	0.8966	1.5874	0.8139	0.6035	0.2957	2.5798	1.8211
TS	1.4905	0.9047*	1.4465	0.7880*	0.5069	0.2806*	2.1462	1.7562*
Third, we present the outcomes of our approach on the Natural Language Understanding task, specifically on the MultiNLI Match dataset, as displayed in Table 5. The results demonstrate that our method can improve confidence calibration performance in balanced datasets and achieve comparable performance in addressing proximity bias.
Table 5: Results of MultiNLI Match dataset on RoBERTa-base Model that is fine-tuned on MultiNLI Match. 'Base' refers to existing calibration methods and 'Ours' refers to our method applied to existing calibration methods. Calibration error is given by  $\times  {10}^{-2}$  .
Method	ECE ↓	ACE ↓	MCE ↓	PIECE ↓
base	+ours	base	+ours	base	+ours	base	+ours
Conf	2.39	1.71	2.68	2.03	0.54	0.36	4.07	3.70
TS	1.68	1.53	1.96	1.86	0.49	0.40	4.19	3.45
ETS	1.88	1.57	2.08	1.76	0.70	0.44	3.97	3.61
PTS	9.28	3.48	9.25	3.49	5.86	1.13	9.44	5.29
PTSK	11.94	6.10	11.91	6.15	10.24	4.03	12.23	6.80
HB	2.07	1.80	2.10	2.10	1.03	0.67	4.63	3.67
IR	1.30	1.23	1.71	1.54	0.30	0.45	3.70	3.66
MIR	1.02	1.04	1.22	1.22	0.34	0.32	3.73	3.35
E.3 Effectiveness on Datasets with the Long-tail Class Distribution.
To evaluate our method in large-scale long-tail datasets, we conduct experiments on long-tail datasets ImageNet-LT, and iNaturalist 2021. Table 6 shows our method's performance on ImageNet-LT, showing that our algorithm improves upon the original calibration algorithms in most cases under all four evaluation metrics, particularly on ECE, ACE, and PIECE. This suggests that our algorithm
Table 6: Performance of our proposed framework against base methods on ImageNet-LT using the pretrained ResNet50 model with classifier re-training techniques [20]. The symbol * denotes that the method is significantly better than the other one with a confidence level of at least  $90\%$ .
Method	ECE↓	MCE↓	ACE↓	PIECE↓
base	+ours	base	+ours	base	+ours	base	+ours
conf	7.4933	1.8724*	0.8672	0.3380*	7.4610	2.0168*	7.9229	3.9776*
TS	2.1965	1.8541	0.3100	0.3956	2.0580	1.7472	3.8407	3.8478
ETS	2.2704	1.8642	0.3264	0.3926	2.0648	1.7622	4.0024	3.8520
PTS	5.0023	1.7300*	0.6512	0.3160	4.9927	1.7401*	7.6502	3.9018*
PTSK	9.1862	1.8558	1.7478	0.3475	9.2080	1.9875	11.4242	3.9175*
HB	13.4644	12.9257*	7.5672	7.5670	13.5931	13.4697	16.2915	15.3259*
IR	8.0541	8.5778	1.0642	1.0508	7.9947	8.5546	8.8631	9.3678
MIR	1.6207	1.5063	0.3205	0.3217	1.6562	1.5310	3.7481	3.6285
can effectively mitigate the bias towards low proximity samples (i.e. tail classes), highlighting its practicality in real-world scenarios where data is often imbalanced.
E.4 Effectiveness on Datasets with Distribution Shift
To further evaluate the effectiveness of our proposed method in handling distribution shifts, we conduct experiments on the ImageNet Corruption dataset [14]. The base models are trained on the ImageNet dataset without any distribution shifts and are subsequently tested on the corrupted dataset. The results, shown in Figure 12, indicate that our method is able to consistently improve the performance of the original model even in the presence of distribution shifts. This is particularly notable in the case of data with distribution shifts that tend to fall in the low proximity region, where current algorithms tend to be poorly calibrated, but our method effectively addresses this problem.
Figure 12: The calibration error reduction in PIECE aachieved by integrating our method with existing calibration algorithms. Different colors indicate different base calibration algorithms. Each color represents a different base calibration algorithm. The bar indicates the difference in calibration error between the base algorithm and the one enhanced by our approach.
To explore the issue of domain shift, where test data has a shifted distribution not seen in training, we conduct experiments using ImageNet (training set) and ImageNet-Sketch (test set). The datasets are chosen because all ImageNet images are real-world photos, while all ImageNet-Sketch images are sketches, collected using Google Search with class label keywords and "sketch", similar to the case of skin color example provided in the introduction. We employ a Vision Transformer backbone from TIMM, trained on the ImageNet. Then we train our ProCal using the validation set from ImageNet and tested it on the 50,000 images from ImageNet-Sketch. The result shown in Table 7 shows that ProCal effectively improves upon existing algorithm in many cases. While we observe a slight increase in ECE, ACE, and MCE when ProCal is paired with with PTS and PTSK, this is probably attributed to the original methods suffering from the cancellation effect, where positive and negative calibration errors within the same confidence bin cancel out each other (see section 6.1). Under the PIECE metric that captures the cancellation effect, our method consistently outperforms all methods by large margins and effectively mitigates their proximity bias.
Table 7: Calibration performance of ViT-Large (vit_large_batch14 Clip_336) from timm [43] on ImageNet-Sketch dataset. Calibrators are trained using ImageNet validation set and tested on ImageNet-Sketch. 'base' refers to the methods, '+ours' shows the performance after integrating our method. Note that 'Conf+Ours' shows the result of our method applied directly to model confidence. Calibration error is given by  $\times  {10}^{-2}$  .
Method	ECE ↓	ACE ↓	MCE ↓	PIECE ↓
base	+ours	base	+ours	base	+ours	base	+ours
Conf	3.91	1.96	3.92	1.92	0.95	0.35	6.33	2.92
TS	7.60	2.38	7.57	2.38	1.42	0.57	7.95	3.27
ETS	3.16	2.07	3.22	2.03	0.37	0.39	4.92	2.97
PTS	1.62	1.60	1.65	1.66	0.34	0.25	3.89	2.74
PTSK	3.14	1.15	3.16	1.11	0.96	0.13	5.71	2.58
MIR	0.28	1.22	0.23	1.19	0.12	0.22	4.24	2.73
F.1 Comparison between Density-Ratio and Bin-Mean-shift.
Our comparison of Density-Ratio and Bin-Mean-shift reveals that Density-Ratio performs better in scaling-based methods while Bin-Mean-Shift demonstrates general robustness and adaptability in both continuous and discrete settings. This can be attributed to the fact that Density-Ratio can be thought of as an infinite binning-based method, enjoying good expressiveness, but it relies on density estimation which may not be as accurate when dealing with discrete outputs. In contrast, Bin-Mean-shift does not make any assumptions about the output. Therefore, it is important to note that both techniques have their own strengths and weaknesses, and the choice of which one to use should be based on the specific task at hand.
F.2 Hyperparameter Sensitivity
In this section, we evaluate the sensitivity of our method to various hyperparameter choices. Specifically, we examine the impact of the choice of distance metric and the number of neighbors in the local neighborhood. To evaluate the performance of our method under different hyperparameter settings, we use ResNet50 [12] as our base model and compare its calibration performance of the integration of our method and existing popular claibration algorithms under different hyperparameters. This study aims to provide a guideline for selecting appropriate hyperparameters when using our method.
Effect of Neighbor Size  $K$  In this study, we examine the influence of the number of neighbors ( $K$ ) on the performance of our proposed method. We assess the performance by varying the number of neighbors from  $K = 1$  to  $K = 1000$  and comparing the results. Figure 13 illustrates the impact of neighbor size  $K$  on performance. The results reveal a V-shaped relationship between the number of neighbors and performance. Initially, an increase in the number of neighbors from 1 yields increasing performance improvement. However, when the number of neighbors exceeds a certain threshold (e.g.  $K > 50$ ), performance begins to deteriorate as the neighborhood becomes more global rather than local, eventually reaching a saturation point. Notably, a small neighborhood size of  $K = 10$  is sufficient to capture the local neighborhood, and further increasing the neighborhood size does not yield additional benefits. This finding aligns with previous works [17, 45] that demonstrate the insensitivity of proximity to the choice of  $K$ . Based on these findings, we recommend employing a moderate range of neighbors, specifically between 10 and 50, to achieve optimal performance.
Effect of Distance Measure In this study, we explore the impact of different distance measures on the performance of our method. We compare the performance under four distance measures: L2 (Euclidean distance), cosine similarity, IVFFlat, and IVFPQ. L2 distance and cosine similarity are widely used measure in machine learning. IVFFlat and IVFPQ are approximate nearest-neighbor search methods implemented using the faiss library [18]. IVFFlat is memory-efficient and suitable for high-dimensional datasets, while IVFPQ is optimized for datasets with a large number of points and high-dimensional features. The results are depicted in Figure 14. Overall, we observe minimal performance differences across the various distance measures. Specifically, the use of cosine similarity
Figure 13: Hyperparameter sensitivity of the number of neighbors used in computing proximity. The results reveal a V-shaped relationship between the number of neighbors and performance. Initially, an increase in the number of neighbors from 1 yields increasing performance improvement. However, when the number of neighbors exceeds a certain threshold (e.g.  $K > 50$ ), performance begins to deteriorate.
and L2 distance yields comparable performance across the four calibration metrics. Additionally, employing IVFFlat results in slightly smaller calibration errors. However, the performance disparity is not significant. Different distance measures, including the approximation methods, offer similar performance improvements. This is because the density function for proximity values is smooth, and small measurement noise does not significantly impact the final density estimation. Considering efficiency, we recommend utilizing IVFFlat due to its favorable efficiency characteristics. However, it is important to note that the choice of the best distance measure depends on the specific problem and dataset, as each measure may exhibit varying performance in different scenarios.
G Pseudo-codes
We present the procedural steps of our approach in the form of a pseudocode. Algorithm 1 encompasses the general inference phase:
Algorithm 1 Inference procedure.
Require: Test sample  $\mathbf{x} \in \mathbb{R}^n$ , held-out embeddings  $E \in \mathbb{R}^{N \times d}$ , classifier  $f$  (calibrated or un-calibrated), number of nearest neighbors  $K$ , nearest neighbor search algorithm  $S$  using Faisss library, PROCAL calibrator  $C$
1: procedure INFERENCE(x)
2:  $\mathbf{e}_x, \hat{p}, \hat{y} \gets f(\mathbf{x})$
3:  $\mathbf{d} \gets S.\text{search}(\mathbf{e}_x, K)$
4:  $d_{x}\gets exp\{-mean(\mathbf{d})\}$
5: return  $C(\hat{p},d_x)$
6: end procedure
$\triangleright$  get feature embedding, prediction and confidence
$\triangleright$  proximity as the average distance to  $K$  nearest neighbors
$\triangleright$  return calibrated confidence
Algorithm 2 encapsulates the Density-Ratio Calibration algorithm.
Figure 14: Hyperparameter sensitivity of several proximity measure under four evaluation metrics.
Algorithm 2 Density-Ratio Calibration.
Require: Pre-trained model  $M$  , validation set with pre-computed proximity  $\mathcal{D}_{val} = \{\mathcal{X},\mathcal{Y},\mathcal{D}\}$  , test set with pre-computed proximity  $\mathcal{D}_{test} = \{\mathcal{X}_{test},\mathcal{Y}_{test},\mathcal{D}_{test}\}$    
1: procedure DENSITYRATIOCALIB(x)   
2:  $\mathcal{D}_{val}^{+} = \emptyset ,\mathcal{D}_{val}^{-} = \emptyset$    
3: for  $i = 1,\dots ,|\mathcal{X}|$  do   
4:  $\hat{y}_i,p_i\gets M(x_i),x_i\in \mathcal{X}$  ▷ get predicted class label and confidence   
5: if  $\hat{y}_i = y$  then ▷ split  $\mathcal{D}_{val}$  based on prediction correctness   
6:  $D_{val}^{+}\leftarrow \mathcal{D}_{val}^{+}\cup \{< p_{i},d_{i}>\}$    
7: else   
8:  $\mathcal{D}_{val}^{-}\gets \mathcal{D}_{val}^{-}\cup \{< p_{i},d_{i}>\}$    
9: end if   
10: end for   
11: KDE  $^+$  -KDE(D)+ ▷ 2-dimension KDE given confidence and proximity   
12: KDE  $^-$  -KDE(D)-   
13:  $\gamma \leftarrow \frac{|D_{val}|}{|D_{val}|}$    
14: for  $j = 1,\ldots ,|\mathcal{X}_{test}|$  do   
15:  $\hat{y}_j,p_j\gets M(x_j),x_j\in \mathcal{X}_{test}$    
16:  $s_j = \frac{\mathrm{KDE}^+(D_j,p_j)}{\mathrm{KDE}^+(D_j,p_j) + \gamma\times\mathrm{KDE}^-(D_j,p_j)}$  ▷ compute re-calibrated score for test sample   
17: end for   
18: return  $s_j,j = 1,\dots ,|\mathcal{X}_{test}|$    
19: end procedure