SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Miao Xiong $^{1*}$  Ailin Deng $^{1}$  Pang Wei Koh $^{23}$  Jiaying Wu $^{1}$
Shen Li $^{1}$  Jianqing Xu  $\mathbf{B}$  Bryan Hooi $^{1}$
<sup>1</sup> National University of Singapore <sup>2</sup> University of Washington <sup>3</sup> Google
Confidence calibration is central to providing accurate and interpretable uncertainty estimates, especially under safety-critical scenarios. However, we find that existing calibration algorithms often overlook the issue of proximity bias, a phenomenon where models tend to be more overconfident in low proximity data (i.e., data lying in the sparse region of the data distribution) compared to high proximity samples, and thus suffer from inconsistent miscalibration across different proximity samples. We examine the problem over 504 pretrained ImageNet models and observe that: 1) Proximity bias exists across a wide variety of model architectures and sizes; 2) Transformer-based models are relatively more susceptible to proximity bias than CNN-based models; 3) Proximity bias persists even after performing popular calibration algorithms like temperature scaling; 4) Models tend to overfit more heavily on low proximity samples than on high proximity samples. Motivated by the empirical findings, we propose PROCAL, a plug-and-play algorithm with a theoretical guarantee to adjust sample confidence based on proximity. To further quantify the effectiveness of calibration algorithms in mitigating proximity bias, we introduce proximity-informed expected calibration error (PIECE) with theoretical analysis. We show that PROCAL is effective in addressing proximity bias and improving calibration on balanced, long-tail, and distribution-shift settings under four metrics over various model architectures. We believe our findings on proximity bias will guide the development of fairer and better-calibrated models, contributing to the broader pursuit of trustworthy AI.
Machine learning systems are increasingly deployed in high-stakes applications such as medical diagnosis [30, 36, 9, 6], where incorrect decisions can have severe human health consequences. To ensure safe and reliable deployment, confidence calibration approaches [10, 23, 28] are employed to produce more accurate uncertainty estimates, which allow models to establish trust by communicating their level of uncertainty, and to defer to human decision-making when the models are uncertain.
In this paper, we present a calibration-related phenomenon termed proximity bias, which refers to the tendency of current deep classifiers to exhibit higher levels of overconfidence on samples of low proximity, i.e., samples in sparse areas within the data distribution (see Figure 1 for an illustrative example). In this study, we quantify the proximity of a sample (Eq. 1) using the average distance to its  $K$  (e.g.  $K = 10$ ) nearest neighbor samples in the data distribution, and we observe that proximity bias holds for various choices of  $K$ . Importantly, the phenomenon persists even after applying existing popular calibration methods, leading to different levels of miscalibration across proximities.
The proximity bias issue raises safety concerns in real-world applications, particularly for underrepresented populations (i.e. low proximity samples) [30, 35]. A recent skin cancer analysis highlights
* Miao Xiong (miao.xiong@u.nus.edu).
Figure 1: Samples with lower (higher) proximity tend to be more overconfident (underconfident). The results are conducted using XCiT, an Image Transformer, on the ImageNet validation set (All Samples). The sample's proximity is measured using the average distance to its nearest neighbors ( $K = 10$ ) in the validation set. We split samples into 10 equal-size bins based on proximity and choose the bin with the highest proximity (High Proximity Samples) and lowest proximity (Low Proximity Samples).
this concern by revealing that AI-powered models demonstrate high performance for light-skinned individuals but struggle with dark-skinned individuals due to their underrepresentation [11]. This issue can also manifest in the form of proximity bias: suppose a dark-skinned individual has a high risk of  $40\%$  of having the cancer. However, due to their underrepresentation within the data distribution, the model overconfidently assigns them  $98\%$  confidence of not having cancer. As a result, these low proximity individuals may be deprived of timely intervention.
To study the ubiquity of this problem, we examine 504 ImageNet pretrained models from the timm library [43] and make the following key observations: 1) Proximity bias exists generally across a wide variety of model architectures and sizes; 2) Transformer-based models are relatively more susceptible to proximity bias than CNN-based models; 3) Proximity bias persists even after performing popular calibration algorithms including temperature scaling; 4) Low proximity samples are more prone to model overfitting while high proximity samples are less susceptible to this issue.
Besides, we argue that proximity bias is overlooked by confidence calibration. Revisiting its definition,  $\mathbb{P}(Y = \hat{Y} \mid \hat{P} = p) = p$  for all  $p \in [0,1]$ , we find that its primary goal is to match confidence with the accuracy of samples sharing the same confidence level. However, Figure 1a reveals that although the model seems well-calibrated within each confidence group, there still exists miscalibration errors among these groups (e.g. low and high proximity samples) due to proximity bias.
Motivated by this, we propose a debiased variant of the expected calibration error (ECE) metric, called proximity-informed expected calibration error (PIECE) to further capture the miscalibration error due to proximity bias. The effectiveness is supported by our theoretical analysis that PIECE is at least as large as ECE and this equality holds when there is no cancellation effect with respect to proximity bias.
To tackle proximity bias and further improve confidence calibration, we propose a plug-and-play method, PROCAL. Intuitively, PROCAL learns a joint distribution of proximity and confidence to adjust probability estimates. To fully leverage the characteristics of the input information, we develop two separate algorithms tailored for continuous and discrete inputs. We evaluate the algorithms on large-scale datasets: balanced datasets including ImageNet [7] and Yahoo-Topics [49], long-tail datasets iNaturalist 2021 [3] and ImageNet-LT [27] and distribution-shift datasets MultiNLI [44] and ImageNet-C [15]. The results show that our algorithm consistently improves the performance of existing algorithms under four metrics with  $90\%$  significance (p-value  $< 0.1$ ).
Findings: We discover the proximity bias issue and show its prevalence over large-scale analysis (504 ImageNet pretrained models).
- Metrics: To quantify the effectiveness of mitigating proximity bias, we introduce proximity-informed expected calibration error (PIECE) with theoretical analysis.
- Method Effectiveness: We propose a plug-and-play method PROCAL with theoretical guarantee and verify its effectiveness on various image and text settings.
Confidence Calibration Confidence calibration aims to yield uncertainty estimates via aligning a model's confidence with the accuracy of samples with the same confidence level [10, 25, 28].
To achieve this, Scaling-based methods, such as temperature scaling [10], adjust the predicted probabilities by learning a temperature scalar for all samples. Similarly, parameterized temperature scaling [39] offers improved expressiveness via input-dependent temperature parameterization, and Mix-n-Match [48] adopts ensemble and composition strategies to yield data-efficient and accuracy-preserving estimates. Binning-based methods divide samples into multiple bins based on confidence and calibrate each bin. Popular methods include classic histogram binning [46], mutual-information-maximization-based binning [32], and isotonic regression [47]. However, existing calibration methods overlook the proximity bias issue, which fundamentally limits the methods' capabilities in delivering reliable and interpretable uncertainty estimates.
Multicalibration Multicalibration algorithms [13, 21] aim to achieve a certain level of fairness by ensuring that a predictor is well-calibrated for the overall population as well as different computationally-identifiable subgroups. [33] proposes a grouping loss to evaluate subgroup calibration error while we propose a metric to integrate the group cancellation effect into existing calibration loss. [21] focuses on understanding the fundamental trade-offs between group calibration and other fairness criteria, and [13] proposes a conceptual iterative algorithm to learn a multi-calibrated predictor. In this regard, our proposed framework can be considered a specific implementation of the fairness objectives outlined in [13], with a particular focus on proximity-based subgroups. This approach offers easier interpretation and implementation compared to subgroups discussed in [13, 21].
3 What is Proximity Bias?
In this section, we study the following questions: What is proximity bias? When and why does proximity bias occur?
Background We consider a supervised multi-class classification problem, where input  $X \in \mathcal{X}$  and its label  $Y \in \mathcal{Y} = \{1, 2, \dots, C\}$  follow a joint distribution  $\pi(X, Y)$ . Let  $f$  be a classifier with  $f(X) = (\hat{Y}, \hat{P})$ , where  $\hat{Y}$  represents the predicted label, and  $\hat{P}$  is the model's confidence, i.e. the estimate of the probability of correctness [10]. For simplicity, we use  $\hat{P}$  to denote both the model's confidence and the confidence calibrated using existing calibration algorithms.
Proximity We define proximity as a function of the average distance between a sample  $X$  and its  $K$  nearest neighbors  $\mathcal{N}_K(X)$  in the data distribution:
$$
D (X) = \exp \left(- \frac {1}{K} \sum_ {X _ {i} \in \mathcal {N} _ {K} (X)} \operatorname {d i s t} (X, X _ {i})\right), \tag {1}
$$
where  $\mathrm{dist}(X,X_i)$  denotes the distance between sample  $X$  and its  $i$ -th nearest neighbor  $X_{i}$ , estimated using Euclidean distance between the features of  $X$  and  $X_{i}$  from the model's penultimate layer.  $K$  is a hyperparameter (we set  $K = 10$  in this paper). We use the validation set as a proxy to estimate the data distribution. That is, we compute any point's proximity by finding its nearest neighbors in the held-out validation set. Although the training set can also be employed to compute proximity, we utilize the validation set because it is readily accessible during the calibration process.
The exponential function is used to normalize the distance measure from a range of  $[0, \inf]$  to  $[0, 1]$ , making the approach more robust to the effects of distance scaling since the absolute distance in Euclidean distance can cause instability and difficulty in modeling. This definition allows us to capture the local density of a sample and its relationship to its neighborhood. For instance, a sample situated in a sparse region of the training distribution would receive a low proximity value, while a sample located in a dense region would receive a high proximity value. Samples with low proximity values represent underrepresented samples in the data distribution that merit attention, such as rare ("long-tail") diseases, minority populations, and samples with distribution shift.
Proximity Bias To investigate the relationship between proximity and model miscalibration, we define proximity bias as follows:
Definition 3.1. Given any confidence level  $p$ , the model suffers from proximity bias if the following condition does not hold:
$$
\mathbb {P} \left(\hat {Y} = Y \mid \hat {P} = p, D = d _ {1}\right) = \mathbb {P} \left(\hat {Y} = Y \mid \hat {P} = p, D = d _ {2}\right) \quad \forall   d _ {1}, d _ {2} \in (0, 1 ], d _ {1} \neq d _ {2}.
$$
Figure 2: Proximity bias analysis on 504 public models. Each marker represents a model, where marker sizes indicate model parameter numbers and different colors/shapes represent different architectures. The bias index is computed using Equation (3) (0 indicates no proximity bias). Left: We observed the following: 1) Models with higher accuracy tend to have a larger bias index. 2) Proximity bias exists across a wide range of model architectures. 3) Transformer variants (e.g. DEiT, XCiT, CaiT, and SwinV2) have a relatively larger bias compared to convolution-based networks (e.g. VGG and ResNet variants). Right: Confidence calibrated by temperature scaling (Upper Right) is similar to the original model confidence w.r.t proximity bias. Our PROCAL (Bottom Right) is effective in reducing proximity bias. Analysis of other existing calibration algorithms can be found in Appendix D.
The intuition behind this definition is that, ideally, a sample with a confidence level of  $p$  should have a probability of being correct equal to  $p$ , regardless of proximity. However, if low proximity samples consistently display higher confidence than high proximity samples (as shown in Figure 1), it can lead to unreliable and unjust decision-making, particularly for underrepresented populations.
3.1 Main Empirical Findings
To showcase the ubiquity of this problem, we examine the proximity bias phenomenon on 504 ImageNet pretrained models from the timm library [43] and show the results in Figure 2 (see Appendix D for additional figures and analysis). We use statistical hypothesis testing to investigate the presence of proximity bias. The null hypothesis  $H_0$  is that proximity bias does not exist, formally, for any confidence  $p$  and proximities  $d_1 > d_2$ :
$$
\mathbb {P} \left(\hat {Y} = Y \mid \hat {P} = p, D = d _ {1}\right) = \mathbb {P} \left(\hat {Y} = Y \mid \hat {P} = p, D = d _ {2}\right). \tag {2}
$$
To test the above null hypothesis, we first split the samples into 5 equal-sized proximity groups and select the highest and lowest proximity groups. From the high proximity group, we randomly select 10,000 points and find corresponding points in the low proximity group that have similar confidence levels. Next, we reverse this process, randomly selecting 10,000 points from the low proximity group and find corresponding points in the high proximity group with matched confidence. We then merge all the points from the high proximity group into  $B_H$  and those from the low proximity group into  $B_L$ , with the  $B_H$  and  $B_L$  having similar average confidence. Finally, we apply the Wilcoxon rank-sum test [24] to evaluate whether there is a significant difference in the sample means (i.e. accuracy) of  $B_H$  and  $B_L$ . More implementation details can be found in Appendix C.
Inspired by the hypothesis testing, we define Bias Index as the accuracy drop between the confidence-matched high proximity group  $B_{H}$  and low proximity group  $B_{L}$  to reflect the degree of bias:
$$
\text {B i a s I n d e x} = \frac {\sum_ {(X , Y) \in B _ {H}} \mathbb {1} \{\hat {Y} = Y \}}{| B _ {H} |} - \frac {\sum_ {(X , Y) \in B _ {L}} \mathbb {1} \{\hat {Y} = Y \}}{| B _ {L} |} = \operatorname {A c c} \left(B _ {H}\right) - \operatorname {A c c} \left(B _ {L}\right). \tag {3}
$$
Note that  $B_H, B_L$  are obtained from the hypothesis testing process and hence have the same mean confidence.
The hypothesis testing results indicate that over  $80\%$  of 504 models have a p-value less than 0.05 (72% after Bonferroni correction [4]), i.e., the null hypothesis is rejected with a confidence level of at least  $95\%$ , indicating that proximity bias plagues most of the models in timm.
We show the bias index of 504 models in Figure 2 and make the following findings:
1. Proximity bias exists generally across a wide variety of model architecture and sizes. Figure 2 shows that most models (80% of the models as supported by hypothesis testing) have a bias index larger than 0, indicating the existence of proximity bias.
2. Transformer-based methods are relatively more susceptible to proximity bias than CNN-based methods. In Figure 2, models with lower accuracy (primarily CNN-based models such as VGG, EfficientNet, MobileNet, and ResNet [12] variants) tend to have lower bias index. On the other hand, among models with higher accuracy, Transformer variants (e.g., DEiT, XCiT [1], CaiT [40], and SwinV2) demonstrate relatively higher bias compared to convolution-based networks (e.g., ResNet variants). This is concerning given the increasing popularity of Transformer-based models in recent years and highlights the need for further research to study and address this issue.
3. Popular calibration methods such as temperature scaling do not noticeably alleviate proximity bias. Figure 2 (upper right) shows that the proximity bias index remains large even after applying temperature scaling, indicating that this method does not noticeably alleviate the problem. In contrast, Figure 2c demonstrates that our proposed approach successfully shifts the models to a much closer distribution around the line  $y = 0$  (indicating no proximity bias). The bias index figures for more existing calibration methods are provided in Appendix D.
4. Low proximity samples are more prone to model overfitting. Figure 4 in Appendix D shows that the model's accuracy difference between the training and validation set is more significant on low proximity samples (31.67%) compared to high proximity samples (0.6%). This indicates that the model generalizes well on samples of high proximity but tends to overfit on samples of low proximity. The overconfidence of low proximity samples can be a consequence of the overfitting tendency, as the overfitting gap also reflects the mismatch between the model's confidence and its actual accuracy.
4 Proximity-Informed ECE
As depicted in Figure 1, existing evaluation metrics underestimate the true miscalibration level, as proximity bias causes certain errors in the model to cancel out. As an example, consider a scenario:
Example 4.1. All samples are only drawn from two proximity groups of equal probability mass,  $d = 0.2$  and  $d = 0.8$ , with true probabilities of  $\mathbb{P}(Y = \hat{Y} | X, f)$  being 0.5 and 0.9, respectively. The model outputs the confidence score  $p = 0.7$  to all samples.
We consider the most commonly used metric, expected calibration error (ECE) that is defined as  $\mathrm{ECE} = \mathbb{E}_{\hat{P}}\left[\left|\mathbb{P}(\hat{Y} = Y\mid \hat{P}) - \hat{P}\right|\right]$ . In Example 4.1, the ECE is 0, suggesting that the model is perfectly calibrated in terms of confidence calibration. In fact, the model has significant miscalibration issues: it is heavily overconfident in one proximity group while heavily underconfident in the other, highlighting the limitations of existing calibration metrics. The miscalibration errors within the same confidence group are canceled out by samples with both high and low proximity, resulting in a phenomenon we term cancellation effect.
To further evaluate the miscalibration canceled out by proximity bias, we propose the proximity-informed expected calibration error (PIECE). PIECE is defined in an analogous fashion as ECE, yet it further examines information about the proximity of the input sample,  $D(X)$ , in the calibration evaluation:
$$
\operatorname {P I E C E} = \mathbb {E} _ {\hat {P}, D} \left[ \left| \mathbb {P} (\hat {Y} = Y \mid \hat {P}, D) - \hat {P} \right| \right]. \tag {4}
$$
Back to Example 4.1 where  $\mathrm{ECE} = 0$ , we have  $\mathrm{PICE} = 0.2$ , revealing its miscalibration level in the subpopulations of different proximities, i.e., the calibration error regarding proximity bias. Additionally, we demonstrate in Theorem 4.2 that PIECE is always at least as large as ECE, with the equality holding only when there is no cancellation effect w.r.t proximity. The detailed proof is relegated to Appendix B.
Theorem 4.2 (PIECE captures cancellation effect.). Given any joint distribution  $\pi(X, Y)$  and any classifier  $f$  that outputs model confidence  $\hat{P}$  for sample  $X$ , we have the following inequality, where equality holds only when there is no cancellation effect with respect to proximity:
$$
\underbrace {\mathbb {E} _ {\hat {P}} \left[ \left| \mathbb {P} (\hat {Y} = Y \mid \hat {P}) - \hat {P} \right| \right]} _ {\mathrm {E C E}} \leq \underbrace {\mathbb {E} _ {\hat {P} , D} \left[ \left| \mathbb {P} (\hat {Y} = Y \mid \hat {P} , D) - \hat {P} \right| \right]} _ {\mathrm {P I E C E}}.
$$
5 How to Mitigate Proximity Bias?
In this section, we propose PROCAL to achieve three goals: 1) mitigate proximity bias, i.e., ensure samples with the same confidence level have the same miscalibration gap across all proximity levels, 2) improve confidence calibration by reducing overconfidence and underconfidence, and 3) provide a plug-and-play method that can combine the strengths of existing approaches with our proximity-informed approach.
The high-level intuition is to explicitly incorporate proximity when estimating the underlying probability of the model prediction being correct. In addition, existing calibration algorithms can be classified into 2 types: 1) those producing continuous outputs, exemplified by scaling-based methods [10]; 2) those producing discrete outputs, such as binning-based methods, which group the samples into bins and assign the same scores to samples within the same bin. To fully leverage the distinct properties of the input information, we develop two separate algorithms tailored for continuous and discrete inputs. This differentiation is based on the observation that continuous outputs (e.g., those produced by scaling-based methods) contain rich distributional information suitable for density estimation. On the other hand, discrete inputs (e.g., those generated by binning-based methods) allow for robust binning-based adjustments. By treating these inputs separately, we can effectively harness the characteristics of each type.
In summary, Density-Ratio Calibration (§5.1) estimates continuous density functions, and aligns well with type 1) methods that produce continuous confidence scores. In contrast, Bin-Mean-Shift (§5.2) does not rely on densities, making it more compatible with type 2) calibration methods that yield discrete outputs. Together, these two calibration techniques constitute a versatile plug-and-play framework PROCAL, applicable to confidence scores of both continuous and discrete types.
5.1 Continuous Confidence: Density-Ratio Calibration
The common interpretation of confidence is the likelihood of a model prediction  $\hat{Y}$  being identical to the ground truth label  $Y$  for every sample  $X$ . Computing this probability directly with density estimation methods can be computationally demanding, particularly in high-dimensional spaces. To circumvent the curse of dimensionality and address proximity bias, we incorporate the model confidence  $\hat{P}$  and proximity information  $D(X)$  to estimate the posterior probability of correctness, i.e.,  $\mathbb{P}(\hat{Y} = Y \mid \hat{P}, D)$ . This approach is data-efficient since it conducts density estimation in a two-dimensional space only, rather than in the higher dimensional feature or prediction simplex space.
Consider a test sample  $X$  with proximity  $D = D(X)$  and uncalibrated confidence score  $\hat{P}$ , which can be the standard Maximum Softmax Probability (MSP), or the output of any calibration method.  $\mathbb{P}(\hat{Y} = Y\mid \hat{P},D)$  can be computed via Bayes' rule:
$$
\mathbb {P} \left(\hat {Y} = Y \mid \hat {P}, D\right) = \frac {\mathbb {P} \left(\hat {P} , D \mid \hat {Y} = Y\right) \mathbb {P} \left(\hat {Y} = Y\right)}{\mathbb {P} (\hat {P} , D)},
$$
where  $\hat{Y}$  is the model prediction and  $Y$  is the ground truth label. This can be re-expressed as follows by using the law of total probability:
$$
\frac {\mathbb {P} (\hat {P} , D \mid \hat {Y} = Y)}{\mathbb {P} (\hat {P} , D \mid \hat {Y} = Y) + \mathbb {P} (\hat {P} , D \mid \hat {Y} \neq Y) \cdot \frac {\mathbb {P} (\hat {Y} \neq Y)}{\mathbb {P} (\hat {Y} = Y)}}.
$$