Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors

## Abstract

Abstract Calibration is crucial in deep learning applications, especially in fields like healthcare and autonomous driving, where accurate confidence estimates are vital for decision-making. However, deep neural networks often suffer from miscalibration, with reliability diagrams and Expected Calibration Error (ECE) being the only standard perspective for evaluating calibration performance. In this paper, we introduce the concept of consistency as an alternative perspective on model calibration, inspired by uncertainty estimation literature in large language models (LLMs). We highlight its advantages over the traditional reliability-based view. Building on this concept, we propose a post-hoc calibration method called Consistency Calibration (CC), which adjusts confidence based on the model’s consistency across perturbed inputs. CC is particularly effective in locally uncertainty estimation, as it requires no additional data samples or label information, instead generating input perturbations directly from the source data. Moreover, we show that performing perturbations at the logit level significantly improves computational efficiency. We validate the effectiveness of CC through extensive comparisons with various post-hoc and training-time calibration methods, demonstrating state-of-the-art performance on standard datasets such as CIFAR-10, CIFAR-100, and ImageNet, as well as on long-tailed datasets like ImageNet-LT.

## 1 Introduction

Calibration is essential in many deep learning applications where accurate confidence estimates are as important as the predictions themselves. In fields like healthcare Chen et al. () and autonomous driving Feng et al. (), decisions often rely not only on the model’s output but also on how confident the model is in its predictions. A well-calibrated model should reflect the ground truth uncertainty. In healthcare, for instance, a model that accurately reflects uncertainty can help doctors trust the system’s confidence when diagnosing critical conditions.

However, current deep learning models are often found to be miscalibrated (Guo et al., ). To evaluate calibration performance, Naeini et al. () introduced ECE, which has become the gold standard, based on the reliability diagram (DeGroot & Fienberg, ). Although several improved metrics have since been proposed, such as AdaptiveECE (AdaECE) (Nixon et al., ) and ClasswiseECE (CECE) (Kull et al., ), they all adopt the same fundamental perspective on calibration: if a model assigns 80% confidence to its predictions, then, ideally, 80% of those predictions should be correct. We refer to this classical approach as the reliability view, which seeks to align predicted confidence levels with actual model accuracy.

The concept of consistency has gained increasing importance in black-box uncertainty estimation, particularly in recent developments in large language models (LLMs) (Wang et al., ; Tam et al., ; Xiong et al., ; Geng et al., ). If an LLM is confident in its answer, it should provide consistent responses to similar questions. For instance, if an LLM confidently answers the question “What is the answer to 5 + 3?” with “8”, it should also consistently provide “8” for the similar question “What is the result of five plus three?” In this paper, we extend this concept of consistency to model calibration, proposing a new perspective of calibration called consistency.

Specifically, in a classification task, if a model is confident in its prediction, it should consistently provide the same output across multiple perturbed versions of the input. Consistency measures how often a model’s prediction remains unchanged when the input is perturbed within a small neighborhood. A high consistency score implies that the model’s predictions are stable and confident. In this view, a perfectly calibrated model should have its predicted confidence levels align with the consistency observed across these perturbed inputs.

In the following sections, we discuss the differences between calibration from the perspectives of reliability and consistency in Sections [2.1](#S2.SS1) and [2.2](#S2.SS2). Section [2.3](#S2.SS3) highlights the advantages of the consistency approach over the reliability view through a toy example. In Section [2.4](#S2.SS4), CC is introduced, which involves perturbing the logits. We provide empirical evidence to explain its effectiveness in Section [2.6](#S2.SS6). Finally, in Section [2.5](#S2.SS5), we demonstrate that consistency can serve as a reliable method for local uncertainty estimation.

Our contributions can be summarized as follows:

- •
We introduce a novel perspective on calibration based on consistency and highlight its advantages over traditional reliability view represented by ECE.
- •
We propose an easy-to-implement and computationally efficient post-hoc calibration method called Consistency Calibration, which replaces the original confidence score with a consistency measure calculated from perturbed logits using data neighbors.
- •
CC serves as a reliable and effective method for local uncertainty estimation, as it does not require additional data samples or label information. Instead, it generates data neighborhoods based on the source data.
- •
We conduct comparisons with multiple post-hoc and training-time calibration methods, demonstrating state-of-the-art performance on standard datasets, including CIFAR-10, CIFAR-100, and ImageNet, as well as in long-tailed scenarios like ImageNet-LT.

## 2 Methodology

In a classification task, let $\mathcal{X}$ represent the input space and $\mathcal{Y}$ the label space. The neural network $f(\cdot)$ and projection head $g(\cdot)$ maps $x\in\mathcal{X}$ to a vector of logits $z=g(f(x))\in\mathbb{R}^{K}$, where each $z_{k}$ is the logit for class $k$. These logits are then transformed into a probability distribution $\hat{p}=\text{softmax}(z)$ over $K$ classes using the softmax function:

$$ $\hat{p}_{k}=\frac{e^{z_{k}}}{\sum_{i=1}^{K}e^{z_{i}}},\quad k=1,\dots,K,$ (1) $$

where $\mathbf{k}=\arg\max_{i}\hat{p}_{i}$ denotes the predicted label index. The ground-truth label $y\in\mathcal{Y}$ represents the true class, and $\hat{y}\in\mathcal{Y}$ is the predicted label. The confidence score $\hat{p}_{\mathbf{k}}$ represents the predicted probability assigned to the predicted label $\mathbf{k}$.

### 2.1 Calibration in the View of Reliability

Calibration in the view of reliability has been widely accepted since the introduction of the reliability diagram by DeGroot & Fienberg (). In this view, a classifier is considered perfectly calibrated if its predicted confidence $\hat{p}$ accurately represents the true probability of correctness. Formally, this is expressed as:

$$ $\mathbb{P}(\hat{y}=y\mid\hat{p}=p)=p\quad\text{for all}\,p\in[0,1].$ (2) $$

In other words, if a model assigns a confidence score of 80%, the prediction $\hat{y}$ should be correct 80% of the time.
To move beyond visual inspection of reliability diagram, Naeini et al. () developed a quantitative metric from the reliability diagram called the Expected Calibration Error (ECE). ECE provides a more precise measurement of miscalibration by calculating the average discrepancy between a model’s predicted confidence and the actual accuracy of predictions at the same confidence level. ECE is defined as:

$$ $\text{ECE}=\mathbb{E}_{\hat{p}}\left[\left|\mathbb{P}(\hat{y}=y\mid\hat{p})-\hat{p}\right|\right].$ (3) $$

In practice, due to finite sample sizes, an approximation is used by binning predictions into $M$ equally spaced confidence intervals, $\{B_{m}\}_{m=1}^{M}$. Each bin $B_{m}$ contains predictions with confidence scores $\hat{p}\in\left[\frac{m}{M},\frac{m+1}{M}\right)$. For each bin, the average confidence $C_{m}$ and accuracy $A_{m}$ are computed as:

$$ $C_{m}=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}\hat{p}_{i},\quad A_{m}=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}\mathbbm{1}(\hat{y}_{i}=y_{i}),$ (4) $$

where $\mathbbm{1}$ is the indicator function, and $|B_{m}|$ is the number of samples in bin $B_{m}$.
The approximate ECE is then computed as the weighted average of the absolute difference between bin accuracy and bin confidence:

$$ $\text{ECE}=\sum_{m=1}^{M}\frac{|B_{m}|}{N}\left|A_{m}-C_{m}\right|,$ (5) $$

where $N$ is the total number of samples.
Several variants of ECE exist. For instance, AdaECE uses adaptive binning to ensure equal sample sizes in each bin and avoid the issue of uneven confidence distribution in ECE, while CECE computes ECE on a per-class basis, enabling better detection of class-specific calibration errors.

### 2.2 Calibration in the view of Consistency

We offer an alternative perspective on calibration by examining it through the concept of consistency. In a real-world scenario, an individual confident in their answer tends to maintain that answer, even when faced with external doubts or minor alterations to the question. On the other hand, someone who is uncertain might change their response when presented with slightly misleading information or variations in the question. We define this adherence to the original answer as consistency.

Recent advances in LLMs, particularly black-box models utilize factual consistency to enhance performance (Wang et al., ; Tam et al., ; Xiong et al., ; Geng et al., ). These studies frame the consistency of a model’s responses as an indicator of its uncertainty. In the context of classification tasks, calibration can also be described in terms of consistency. Specifically, for classification models, we can formalize this relationship as follows:

###### Proposition 1 .

If a model is confident in its prediction, it should consistently output the same prediction when the input is slightly perturbed. The consistency $c$ of a sample $x$ is defined as

$$ $c_{k}(x)=\frac{1}{T}\sum_{t=1}^{T}\mathbbm{1}(\hat{y}(\tilde{x}_{t})=k),\text{ where }d(\tilde{x}_{t},x)<\epsilon^{*},\quad\text{for}\,k=1,\dots,K$ (6) $$

where $T$ is the number of perturbed neighbors, $\hat{y}(\tilde{x}_{t})$ is the predicted label for the perturbed input $\tilde{x}_{t}$, and the distance between the original sample $x$ and its perturbed version $\tilde{x}_{t}$ is smaller than a constant $\epsilon^{*}$, according to some distance metric $d$.
A model is said to be perfectly calibrated if, for all samples $x$, given a suitable set of perturbed neighbors $\{\tilde{x}_{t}\mid t=1,\dots,T\}$, the predicted confidence score $\hat{p}(x)$ satisfies:

$$ $\hat{p}_{k}(x)=c_{k}(x),\quad\text{for}\,k=1,\dots,K$ (7) $$

However, identifying a suitable perturbed neighborhood is non-trivial—it is challenging to determine an appropriate constant $\epsilon^{*}$ and distance metric $d$. Fortunately, in image classification tasks, a perturbed neighbor is often considered a data-augmented version of the original image. Thus, we begin our exploration by using image data augmentation.

To evaluate the effectiveness of consistency-based confidence, we design an experimental setting using a ResNet-50 model trained on CIFAR-10 with data augmentation (RandomCrop and RandomHorizontalFlip). We generate perturbed neighbors by applying various levels of data augmentation to the entire CIFAR-10 test set, creating 100 perturbed neighbors for each test sample. The calibration performance of consistency is assessed on the test set in the following settings:

- •
Baseline: Confidence score is extracted on the original test set, serving as the baseline.
- •
Weak Augmentation (Train Augmentation): The confidence score is replaced with consistency derived from perturbed neighbors generated using train-time augmentation (RandomCrop and RandomHorizontalFlip), denoted by the yellow star.
- •
Moderate Augmentation (Train Augmentation + ColorJitter): The confidence score is replaced with consistency measured from perturbed neighbors generated using train-time augmentation and varying strengths of ColorJitter, as indicated by the x-axis values.
- •
Stronger Augmentation (Train Augmentation + ColorJitter + Blur): The confidence score is replaced with consistency measured from perturbed neighbors generated using train-time augmentation, ColorJitter, and Blur, represented by the red triangle.

The evaluation results are shown in Figure [1(a)](#S2.F1.sf1). Consistency using neighbors generated with weak augmentation significantly reduces calibration error compared to the baseline. As we increase the perturbation strength with moderate augmentation, as shown by the x-axis values, the calibration error continues to decrease with minimal impact on accuracy, outperforming the commonly used calibration method, Temperature Scaling, up to a certain perturbation threshold.

However, when moderate augmentation with strength exceeds 0.1, accuracy begins to decline, and ECE increases sharply. With neighbors generated from Stronger Augmentation, both calibration and prediction accuracy deteriorate. This likely occurs because stronger perturbations distort the input to the extent that the model can no longer recognize the data, leading to degraded performance. This suggests that consistency has the potential to provide accurate uncertainty estimates when a suitable perturbed neighborhood is identified.

Figure: (a) Perturbation applied to images with different augmentations. Consistency calculated using weak or moderate augmentation neighbors significantly reduces calibration error.
Refer to caption: /html/2410.12295/assets/x1.png

###### Proposition 1 .

### 2.3 Consistency as a Representation of Ground Truth Uncertainty

On one hand, the reliability approach estimates calibration error by comparing the prediction confidence with the average correctness of samples that have similar confidence levels. In this view, the average correctness of such sample neighborhood is treated as an approximation of the ground truth uncertainty. On the other hand, the consistency approach directly uses consistency as a measure of ground truth uncertainty. Thus, we are interested in determining which of these two approaches more accurately approximates this uncertainty.

To explore this, we constructed a toy dataset consisting of two two-dimensional Gaussian distributions representing two groups of data:
$\mathcal{N}(\mu_{0},\Sigma),\mathcal{N}(\mu_{1},\Sigma)$
where $\mu_{0}$ and $\mu_{1}$ are the mean vectors, and $\Sigma$ is the shared covariance matrix for both groups, labeled 0 and 1, respectively. We generated 1,000,000 data points from each group to form the training dataset, which was used to train a CNN model. An additional 50,000 samples from each group were used to create the test dataset. The input space is $\mathcal{X}=\mathbb{R}^{2}$, and the label space is $\mathcal{Y}=\{0,1\}$, as illustrated in Figure [1(b)](#S2.F1.sf2).

The ground truth uncertainty, $\eta(x)$, is calculated from the probability density function (PDF) of each distribution:

$$ $\eta(x)=\frac{p^{0}(x)}{p^{0}(x)+p^{1}(x)}$ (8) $$

where $p^{0}(\cdot)$ and $p^{1}(\cdot)$ are the PDFs of the two distributions. The ground truth uncertainty is illustrated in Figure [1(c)](#S2.F1.sf3). For each label, the ground truth confidence can be expressed as $(\eta(x),1-\eta(x))$.

In Figure [1(c)](#S2.F1.sf3), for a point of interest (marked by a star), the reliability-based approach estimates ground truth uncertainty by calculating the average correctness $A=\frac{1}{|B|}\sum_{i\in B}\mathbbm{1}(\hat{y}_{i}=y_{i}),$ over a “confidence neighborhood” $B$ (i.e., samples with similar confidence, enclosed by the white boxes), similar to the definition Eq. [4](#S2.E4) in ECE. In contrast, the consistency approach estimates uncertainty by considering “data neighborhood,” as illustrated by the orange circle. While the reliability approach relies on the availability of multiple data samples within the confidence neighborhood, the consistency approach generates data neighborhoods by perturbing the data.

The key differences between the reliability and consistency views lie in their neighborhood selection criteria $S$ and aggregation methods. The reliability view selects a neighborhood $B$ based on confidence similarity and aggregates the correctness of the samples, while the consistency view selects a neighborhood based on data perturbations and computes consistency, as described in Eq. [9](#S2.E9). To compare the two approaches, we evaluate them under three neighborhood selection criteria:

- •
Figure [2(a)](#S2.F2.sf1): Reliability view (ECE): $B=\{\tilde{x}\mid|\hat{p}(x)-\hat{p}(\tilde{x})|<\epsilon\}$
- •
Figure [2(b)](#S2.F2.sf2): Reliability view (AdaECE): $B=\{\tilde{x}\mid\text{Top K closest confidence neighbors}\}$
- •
Figure [2(c)](#S2.F2.sf3): Consistency view (perturbing data): $B=\{\tilde{x}\mid\tilde{x}=x+\epsilon\}$

In Figures [2(a)](#S2.F2.sf1) and [2(b)](#S2.F2.sf2), we use the reliability approach to approximate ground truth uncertainty based on two confidence-neighbor selection criteria. In Figure [2(a)](#S2.F2.sf1), we replicate the standard ECE (Guo et al., ) approach by selecting confidence neighbors solely based on confidence differences. The x-axis represents the allowed confidence difference between neighbors and the point of interest, while the y-axis shows the average error between estimated and ground truth uncertainty across the test set. In Figure [2(b)](#S2.F2.sf2), we replicate the AdaECE (Nixon et al., ) approach by selecting the top-K nearest confidence neighbors to estimate uncertainty, with the lowest error (0.57%) achieved by selecting the top 9 nearest neighbors.

In Figure [2(c)](#S2.F2.sf3), we apply Gaussian noise $\epsilon$ to perturb the data samples and compute consistency across 100 generated neighbors, with the x-axis representing the noise strength. We compare the uncertainty estimates from the consistency approach with those from the reliability approach. The dashed lines indicate the minimal error achieved by each method. Within a certain range of perturbation strengths, the consistency approach outperforms, yielding a ground truth uncertainty estimation with an overall error as low as 0.3%.

Figure: (a) Reliability view: Estimating ground truth uncertainty using neighbors with confidence differences indicated on the x-axis.
Refer to caption: /html/2410.12295/assets/x4.png

It is important to note that the reliability approach using confidence neighborhoods is essentially equivalent to the ECE measurement, where the allowed confidence gap functions similarly to the hyperparameter “number of bins” in ECE. As shown in Figure [2](#S2.F2), the estimation error is sensitive to the allowed confidence gap—meaning that the choice of “number of bins” can significantly impact the ability of ECE to estimate the ground truth uncertainty. Similarly, this sensitivity is also observed in the consistency method, where the strength of perturbation noise affects the uncertainty approximation. Despite this sensitivity, the consistency approach achieves a lower overall estimation error, suggesting its potential as a robust alternative calibration metric.

### 2.4 More Efficient Consistency Calibration

Due to numerous types of data augmentations, determining the optimal perturbation strength using a continuous variable is challenging. To address this, we extend the perturbation process to the feature and logit levels by introducing noise with varying intensities. This approach yields effects similar to those observed with image-level perturbations, as demonstrated in Figure [3(a)](#S2.F3.sf1) and Figure [3(b)](#S2.F3.sf2).

Interestingly, feature- and logit-level perturbations maintain significant calibration performance while offer huge computational advantages. With image-level perturbations, inference must be performed on the entire model $T$ times. In contrast, feature-level require evaluating only the classification head, while logit-level only compute the argmax operation $T$ times. This results in substantial reductions in computational costs. Experiments on other layers can be found in Appendix [B](#A2).

###### Proposition 2 .

We propose a unified definition of our calibration methods, termed Consistency Calibration (CC), which identifies perturbed neighbors at different levels. The calibrated prediction confidence score $\hat{p}^{\prime}$ is formally defined as:

$$ $\hat{p}^{\prime}_{k}=\frac{1}{T}\sum_{t=1}^{T}\mathbbm{1}\left(\operatorname*{arg\,max}q\left(\widetilde{h(x)}^{t}\right)=k\right),\quad\text{for}\,k=1,\dots,K,$ (9) $$

where $h(x)$ is the representation of data $x$, $\widetilde{h(x)}^{t}$ is the perturbed representation, and $q$ is the pipeline to extract the logits $z$.

Specifically, for data-level perturbations: $h(\cdot)=I(\cdot)$, $\widetilde{h(x)}$ is the augmented data, $q=g(f(\cdot))$.
For feature-level perturbations: $h(\cdot)=f(\cdot)$, $\widetilde{h(x)}^{t}=h(x)+\epsilon_{t}$, $q=g(\cdot)$.
For logit-level perturbations: $h(\cdot)=g(f(\cdot))$, $\widetilde{h(x)}^{t}=h(x)+\epsilon_{t}$, $q=I(\cdot)$.
Here, $I(\cdot)$ is the identity function, $\epsilon_{t}$ represents the noise added to features or logits, with its strength determined by minimizing the ECE on a validation set. Given the strong calibration performance and computational efficiency of logit-level perturbations, we refer to logit-level consistency calibration as CC when no specification is provided.

Figure: (a) Performance of consistency calibration using data neighbors with feature perturbations at varying noise levels.
Refer to caption: /html/2410.12295/assets/x7.png

###### Proposition 2 .

### 2.5 Consistency as a Local Uncertainty Estimation

Consistency-based methods do not rely on label information or additional data, as they generate their own neighborhood by perturbing the input data. This property allows consistency to serve as a criterion for instance-level uncertainty measurement. As illustrated in Figure [3(c)](#S2.F3.sf3), we examine a miscalibrated (incorrect prediction with high confidence) CIFAR-10 test sample, where a ResNet-50 model trained with Cross-Entropy (CE) shows overconfidence, assigning a confidence score of 0.997 despite being incorrect. Using optimal temperature, determined via a validation set, the confidence after temperature scaling decreases slightly, but the model remains overconfident at 0.903.

For comparison, we apply CC by perturbing the logits (“CC (logits)”), applying train time data augmentation (“CC (Train Aug)”), and using a moderate augmentation method (“CC (Train Aug + Jitter)”). The confidence significantly decreases with these approaches. However, too strong augmentations may negatively impact model accuracy, which requires the need for a validation set to tune the augmentation strength, so we recommend using training-time augmentation to avoid the use of validation set while keeping the prediction accuracy.

Unlike many post-hoc calibration methods that require a large validation set to fine-tune hyperparameters, consistency-based confidence with train-time augmentation can directly provide calibrated confidence scores while maintaining recognizable by models. This approach is particularly valuable in data-limited scenarios, allowing consistency to produce an accurate local uncertainty estimation.

### 2.6 Why consistency calibration works?

Perturbing images results in straightforward and intuitive image neighborhoods, but the effectiveness of perturbations at the logit level requires further explanation. To understand why logit perturbations work, we examined the differences between highly confident correct predictions and overconfident incorrect ones. These represent well-calibrated and poorly calibrated samples, respectively. During logit disturbance, the label with second-largest logit most likely to become the prediction label. To investigate this, we plotted box plots for both the maximum and second-largest logits for correct and incorrect predictions, as shown in Figure [4(a)](#S2.F4.sf1).

For CIFAR-10 test samples, we selected predictions with confidence higher than 99%. We refer to the maximum logit of correct predictions as “Corr. Max” and that of incorrect predictions as “Incorr. Max.” Similarly, “Corr. 2nd” represents the second-largest logit of correct predictions, while “Incorr. 2nd” refers to the second-largest logit of incorrect predictions. As shown in Figure [4(a)](#S2.F4.sf1), the maximum logit for correct predictions is significantly higher than for incorrect predictions. Additionally, the second-largest logit in correct predictions is much lower than that in incorrect predictions. This indicates that the gap between the maximum and second-largest logits is much larger for correct predictions than for incorrect ones. Despite large difference, due to softmax saturation, the model assigns abnormally high confidence (greater than 99%) to both correct and incorrect predictions, leading to overconfident miscalibration.

Interestingly, we can leverage this difference in the logit gaps between correct and incorrect predictions. Perturbations can easily alter the predictions of overconfident, miscalibrated samples, while having minimal effect on well-calibrated, correct predictions. This different response to perturbations explains why consistency calibration is effective at the logit level. We observed similar patterns in experiments with CIFAR-100 and ImageNet, as shown in Figure [4(b)](#S2.F4.sf2) and Figure [4(c)](#S2.F4.sf3).

Figure: (a) CIFAR-10
Refer to caption: /html/2410.12295/assets/x10.png

## 3 Experiments

### 3.1 Experimental Setup

#### Datasets

We conduct experiments on several benchmark datasets, including CIFAR-10, CIFAR-100 (Krizhevsky et al., ), and ImageNet (Deng et al., ). To assess calibration performance in data-imbalance scenarios, we also include ImageNet-LT (Liu et al., ), characterized by its long-tailed class distribution. CIFAR-10 and CIFAR-100 contain 60,000 images of size $32\times 32$ pixels, with 10 and 100 classes, respectively, split into 45,000 training, 5,000 for validation and 10,000 test images. For ImageNet-1K, we split 20% of the original validation set as the new validation set, with the remainder used as the test set. We use the $\epsilon$ searched on ImageNet-1k validation set to calibrate ImageNet-LT test set. The testing batch size for all datasets is set to 128.

#### Models

We evaluate our approach across various neural network architectures, including ResNet-50 and ResNet-110 (He et al., ), Wide ResNet (Zagoruyko & Komodakis, ), DenseNet-121 (Huang et al., ), and Vision Transformers (ViT-B/16 and ViT-B/32) (Dosovitskiy et al., ). These models represent a diverse range of architectures and complexities, allowing us to assess the robustness of our method in different settings. For CIFAR-10 and CIFAR-100, we use pretrained weights from prior work (Mukhoti et al., ). All models are trained using stochastic gradient descent (SGD) with a momentum of 0.9 and weight decay of $5\times 10^{-4}$ for 350 epochs.The learning rate is initialized at 0.1 for the first 150 epochs, reduced to 0.01 for the next 100, and further decreased to 0.001 for the final 100 epochs. For ImageNet, we use pretrained models from PyTorch (Paszke et al., ), following the training recipe available on PyTorch’s model page.

#### Evaluation Metrics and Other Settings

Calibration performance is primarily evaluated using ECE, with additional metrics including AdaECE, CECE, Negative Log-Likelihood (NLL), and top-1 accuracy. All experiments are conducted on an NVIDIA 4090 GPU, with results averaged over five runs to ensure fairness. For all experiments, we set the number of perturbations to $T=1000$ and search the perturbation strength $\epsilon$ and noise type by minimizing ECE on the validation set.

**Table 1: Comparison of Post-Hoc Calibration Methods Using ECE$\downarrow$ Across Various Datasets and Models. ECE values are reported with 15 bins. The best-performing method for each dataset-model combination is in bold, and our method (CC) is highlighted. Results are averaged over 5 runs.**
| Dataset | Model | Vanilla | TS | ETS | PTS | CTS | GC | CC (ours) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR-10 | ResNet-50 | 4.34 | 1.38 | 1.37 | 1.36 | 1.46 | 1.04 | 0.78 |
| Wide-ResNet | 3.24 | 0.93 | 0.93 | 0.93 | 0.93 | 1.33 | 0.36 |  |
| CIFAR-100 | ResNet-50 | 17.52 | 5.71 | 5.68 | 5.64 | 6.05 | 3.55 | 1.25 |
| Wide-ResNet | 15.34 | 4.63 | 4.58 | 4.52 | 4.86 | 2.14 | 1.61 |  |
| ImageNet-1K | ResNet-50 | 3.76 | 2.09 | 2.09 | 2.08 | 3.14 | 2.54 | 1.53 |
| DenseNet-121 | 6.59 | 1.64 | 1.66 | 1.68 | 1.94 | 2.51 | 1.48 |  |
| Wide-ResNet-50 | 5.49 | 3.03 | 3.04 | 3.04 | 4.13 | 2.16 | 1.33 |  |
| Swin-B | 5.02 | 3.90 | 3.90 | 3.93 | 5.43 | 1.61 | 1.58 |  |
| ViT-B-16 | 5.61 | 3.61 | 3.62 | 3.64 | 5.50 | 1.75 | 1.66 |  |
| ViT-B-32 | 6.40 | 3.76 | 3.78 | 3.84 | 5.74 | 1.39 | 1.72 |  |
| ImageNet-LT | ResNet-50 | 3.67 | 2.00 | 1.99 | 2.00 | 2.21 | 1.4 | 1.24 |
| DenseNet-121 | 6.65 | 1.65 | 1.64 | 1.66 | 1.59 | 1.81 | 1.23 |  |
| Wide-ResNet-50 | 5.39 | 2.97 | 2.96 | 2.96 | 3.52 | 1.49 | 1.27 |  |
| Swin-B | 4.66 | 4.02 | 4.03 | 4.08 | 5.02 | 1.66 | 1.44 |  |
| ViT-B-16 | 5.57 | 3.61 | 3.62 | 3.64 | 4.94 | 1.76 | 1.61 |  |
| ViT-B-32 | 5.15 | 5.67 | 5.67 | 5.68 | 5.71 | 1.48 | 1.74 |  |

### 3.2 Comparison with Post-Hoc Calibration Methods

We compare our proposed CC with widely used post-hoc calibration techniques, including Temperature Scaling (TS) (Guo et al., ), Ensemble Temperature Scaling (ETS) (Zhang et al., ), Parameterized Temperature Scaling (PTS) (Tomani et al., ), Class-based Temperature Scaling (CTS) (Frenkel et al., ), and Group Calibration (GC) (Yang et al., ), as well as uncalibrated models (Vanilla). Our evaluation covers CIFAR-10, CIFAR-100, ImageNet-1K, and ImageNet-LT, using various CNNs and transformers.

#### Calibration on Standard Datasets

CC consistently outperforms these methods across CIFAR-10, CIFAR-100, and ImageNet-1K, significantly reducing calibration error. The most notable improvement is seen in CIFAR-100, where CC excels while GC, despite its strong performance on other datasets, struggles. This highlights CC’s robustness across datasets with varying complexities. CNNs, which often suffer from overconfidence, are generally well-calibrated with TS-based methods. However, transformers see limited calibration improvements from TS-based methods, with CC outperforming them by a large margin. On larger datasets like ImageNet-1K, CC maintains its advantage. Although GC slightly outperforms CC on ViT-B/32, it is computationally expensive due to the additional grouping process, whereas CC balances both efficiency and effectiveness.

#### Calibration on Long-Tail Datasets

On long-tail datasets like ImageNet-LT, TS-based models struggle to provide effective calibration, especially for transformers. For example, on ViT-B/32, TS-based methods fail to calibrate effectively, as they apply uniform adjustments across the dataset, smoothing or sharpening probabilities globally. In contrast, CC and GC perform well on long-tail datasets, particularly with transformers. GC excels due to its multicalibration (Hébert-Johnson et al., ), offering sample-wise adjustments, though it comes at a high computational cost. By leveraging local uncertainty estimation through input perturbations, CC better captures uncertainties in underrepresented tail classes, making it especially useful for handling imbalanced data scenarios.

### 3.3 Calibration Performance on Other Metrics

Figure: Figure 5: Calibration performance of ResNet-50 on ImageNet-1K using AdaECE$\downarrow$, CECE$\downarrow$, NLL$\downarrow$, and Accuracy$\uparrow$. ECE, AdaECE, and CECE are reported with 15 bins. Colors in the legend represent different methods. Results are averaged over 5 runs.
Refer to caption: /html/2410.12295/assets/x13.png

We also evaluate CC using additional metrics: AdaECE, CECE, NLL, and accuracy to provide a comprehensive view of its performance. Results for ResNet-50 on ImageNet are shown here, with results for other models and datasets available in Appendix [C](#A3).

#### AdaECE and CECE

CC demonstrates superior performance on both AdaECE and CECE compared to traditional methods. AdaECE accounts for uneven confidence distributions, improving the reliability of ECE, while CECE gives detailed insights into classwise calibration. CC’s strong results on both metrics show its effectiveness from different perspectives.

#### Accuracy Maintained

CC preserves the accuracy of the base models, showing no significant reduction in classification performance. As a post-hoc method, it does not require retraining, maintaining predictive capabilities, making it practical for real-world applications.

#### Increase in NLL

Interestingly, CC results in higher NLL values compared to other methods, reflecting a trade-off between calibration and the sharpness of probability estimates. This suggests that while CC reduces overconfidence in incorrect predictions, it also moderates overconfidence in correct predictions, leading to improved calibration without affecting accuracy.

**Table 2: Comparison of Train-time Calibration Methods Using ECE$\downarrow$ Across Various Datasets and Models. ECE values are reported with 15 bins. The best-performing method for each dataset-model combination is in bold, and our method (CC) is highlighted. Results are averaged over 5 runs.**
| Dataset | Model | Cross-Entropy | Brier Loss | MMCE | LS-0.05 | FLSD-53 | FL-3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | base | ours | base | ours | base | ours | base | ours | base | ours | base | ours |
| CIFAR-10 | ResNet-50 | 4.34 | 0.78 | 1.80 | 1.07 | 4.56 | 0.83 | 2.97 | 1.24 | 1.55 | 0.49 | 1.48 | 0.66 |
| ResNet-110 | 4.41 | 0.98 | 2.57 | 0.48 | 5.08 | 1.17 | 2.09 | 2.30 | 1.88 | 0.67 | 1.54 | 0.48 |  |
| DenseNet-121 | 4.51 | 1.07 | 1.52 | 0.78 | 5.10 | 1.18 | 1.87 | 1.39 | 1.23 | 0.68 | 1.31 | 0.98 |  |
| Wide-ResNet | 3.24 | 0.36 | 1.24 | 0.58 | 3.29 | 0.39 | 4.25 | 1.15 | 1.58 | 0.49 | 1.68 | 0.53 |  |
| CIFAR-100 | ResNet-50 | 17.52 | 1.25 | 6.57 | 1.57 | 15.32 | 1.98 | 7.82 | 5.08 | 4.49 | 1.43 | 5.16 | 1.52 |
| ResNet-110 | 19.05 | 4.57 | 7.88 | 3.24 | 19.14 | 4.41 | 11.04 | 4.58 | 8.55 | 3.47 | 8.64 | 3.67 |  |
| DenseNet-121 | 20.99 | 5.40 | 5.22 | 1.82 | 19.10 | 3.76 | 12.87 | 4.99 | 3.70 | 1.41 | 4.14 | 1.94 |  |
| Wide-ResNet | 15.34 | 1.61 | 4.34 | 1.87 | 13.17 | 2.17 | 4.89 | 4.21 | 3.02 | 1.64 | 2.14 | 1.78 |  |

### 3.4 Comparison with Training-Time Calibration Methods

We evaluate CC alongside training-time calibration techniques, including Brier Loss (Brier, ), Maximum Mean Calibration Error (MMCE) (Kumar et al., ), Label Smoothing (LS-0.05) (Szegedy et al., ), and Focal Loss variants (FLSD-53 and FL-3) (Mukhoti et al., ), as shown in Table [2](#S3.T2). Our analysis shows that combining CC with these methods consistently enhances calibration performance across various models and datasets, further validating CC’s effectiveness alongside training-time approaches.

Moreover, as seen in Table [1](#S3.T1), CC alone, as a post-hoc calibration method, already outperforms these train-time techniques with minimal computational overhead, while train-time methods require significantly more resources. Additional results for other settings are available in Appendix [D](#A4).

### 3.5 Ablation Study

#### Aggregation Methods

In our ablation study, we compare two aggregation methods for refining confidence estimates: the mean of softmax probabilities (Mean), defined as:

$$ $\hat{p}_{k}=\frac{1}{T}\sum_{t=1}^{T}\mathrm{softmax}\left(q\left(\widetilde{h(x)}^{t}\right)\right),\quad\text{for}\,k=1,\dots,K,$ (10) $$

and consistency-based aggregation (Consis.) as shown in Eq. [9](#S2.E9). Both methods leverage predictions over perturbed logits. The mean of softmax probabilities treats the perturbation process like an ensemble method, interpreting uncertainty as a distribution. We show the evaluation results on CIFAR-10 and CIFAR-100 in Table [4](#S3.T4). On smaller datasets like CIFAR-10, both methods perform similarly. However, on larger datasets with more classes, such as CIFAR-100 and ImageNet, consistency-based aggregation slightly outperforms softmax averaging. This suggests that consistency-based aggregation captures uncertainty better than the view of ensemble.

#### Choice of Noise

We investigate the impact of different noise types for input perturbations, comparing uniform noise (U) and Gaussian noise (G), as shown in Table [4](#S3.T4). Uniform noise performs better on datasets with fewer classes, such as CIFAR-10 and CIFAR-100. However, on larger datasets like ImageNet, Gaussian noise yields better results, likely due to variations in the gap between the maximum and second maximum logits across datasets as shown in Figure [4](#S2.F4). The choice of noise is treated as a hyperparameter, offering flexibility to adapt to different datasets and models.

#### Number of Perturbations

We also assess the impact of the number of perturbations. As shown in Figure [4](#S3.T4), our experiments indicate that CC achieves strong calibration performance with as few as $2^{4}=16$ perturbations. Although increasing the number of perturbations slightly improves results, the diminishing returns suggest that CC provides robust calibration with a moderate number of perturbations, ensuring both efficiency and accuracy.

**Table 3: Comparison of Aggregation Methods and Noise Types Using ECE$\downarrow$ Across Various Datasets and Models. ECE values are reported using 15 bins. The best-performing method for each dataset-model combination is highlighted in bold.**
| Dataset | Model | Mean U | Mean G | Consis. U | Consis. G |
| --- | --- | --- | --- | --- | --- |
| CIFAR-10 | ResNet-50 | 0.72 | 1.34 | 0.78 | 1.33 |
| Wide-ResNet | 0.37 | 0.80 | 0.36 | 0.83 |  |
| CIFAR-100 | ResNet-50 | 1.52 | 2.70 | 1.25 | 2.49 |
| Wide-ResNet | 1.86 | 2.08 | 1.61 | 1.88 |  |
| ImageNet | ResNet-50 | 2.37 | 1.41 | 2.29 | 1.27 |
| Wide-ResNet-50 | 2.17 | 1.7 | 2.23 | 1.57 |  |

## 4 Conclusion

Consistency offers an alternative perspective on calibration by focusing on prediction stability under perturbations as an indicator of confidence. CC has proven highly effective in reducing calibration errors across various datasets. However, CC has limitations, such as the need for tuning perturbation strength and noise type, and its current focus on classification tasks, with its application to regression remaining unexplored. Future work can aim to develop a new, more universal consistency-based metric to complement existing metrics like ECE. This would provide a more comprehensive evaluation to calibration, ultimately leading to more reliable deep learning models.

## Appendix A Related Works

Numerous studies have explored the phenomenon of overconfidence in modern neural networks and investigated their calibration properties  (Guo et al., ; Minderer et al., ; Wang et al., ; Tao et al., ). Calibration methods can generally be categorized into two main approaches: post-hoc methods and train-time calibration methods.

#### Calibration Methods

Post-hoc calibration methods adjust model outputs after training to improve calibration. A widely used technique is Temperature Scaling (TS) (Guo et al., ), which smooths softmax probabilities by search a temperature factor on a validation set. Enhanced variants of TS include Parameterized Temperature Scaling (PTS) (Tomani et al., ), which uses a neural network to learn the temperature, and Class-based Temperature Scaling (CTS) (Frenkel et al., ), which applies adjustments on a class-wise basis. Group Calibration (GC) (Yang et al., ) and ProCal (Xiong et al., ) aim for multi-calibration (Hébert-Johnson et al., ) by splitting data samples by proximity and grouping. Another stream of work is train-time calibration such as Brier Loss (Brier, ), Dirichlet Scaling (Kull et al., ), Maximum Mean Calibration Error (MMCE) (Kumar et al., ), Label Smoothing (Szegedy et al., ), and Focal Loss (Mukhoti et al., ) and Dual Focal Loss (Tao et al., ).  Tao et al. () propose to use a new training framework to improve calibration. However, these methods often require substantial higher computational overhead.

#### Ensemble-Based Calibration

Ensemble-based methods ensemble multiple outputs in different ways. They use models or samples to approximate Bayesian Inference. Lakshminarayanan et al. () propose deep ensembles as a scalable alternative to Bayesian Neural Networks (BNNs) for uncertainty estimation. Similarly, Gal & Ghahramani () treat dropout as approximate Bayesian inference. Data-centric ensemble techniques using test-time augmentation, as described by Conde et al. (), also help improve calibration. Zhang et al. () resort to the power of Bayesian inference and proposed a Ensemble-based TS (ETS). However, these methods typically require significant computational resources to train multiple models or perform repeated inferences. In contrast, our approach relies on consistency rather than probability distribution modeling.

#### Consistency in LLMs

Consistency has emerged as a key approach for black-box uncertainty estimation and hallucination detection in large language models (LLMs). These methods evaluate uncertainty by measuring variability in outputs across slight changes, such as different sampling techniques or rephrased prompts. Confident models produce stable outputs, while variability indicates uncertainty. For instance, SelfCheckGPT (Manakul et al., ) uses sampling and similarity metrics like BERTScore and NLI to detect hallucinations, while Lin et al. () analyze a similarity matrix to estimate uncertainty. Xiong et al. () further break down uncertainty estimation into prompting, sampling, and consistency-based aggregation. These methods, which rely on output stability, are efficient alternatives to probabilistic approaches.

## Appendix B Perturbation of different layer

This section presents a detailed analysis of the impact of perturbations applied at various levels of a ResNet50 model, trained on CIFAR-10. The experiments were conducted using 32 samples, and the effects on ECE, accuracy, and optimal perturbation values were evaluated.

**Table 5: Comparison of perturbations at different layers with number of samples set to 32 using ECE$\downarrow$ and Accuracy$\uparrow$, evaluated on ResNet50 with CIFAR-10. ECE values are reported with 15 bins. Optimal Perturbations for logits and features are represented in $\epsilon$ value**
| Perturbation Level | ECE (%) | Accuracy (%) | Optimal Perturbation |
| --- | --- | --- | --- |
| Image | 1.1 | 95.25 | train aug jitter0.1 |
| Logits | 0.73 | 95.04 | 8.2 |
| Feature (Last Layer) | 2.06 | 95.06 | 3.0 |
| Feature (Layer 4) | 0.53 | 95.29 | 13.28 |
| Feature (Layer 3) | 53.12 | 10.03 | 20.12 |
| Feature (Layer 2) | 56.28 | 10.02 | 20.21 |
| Feature (Layer 1) | 49.53 | 10.11 | 20.75 |

From Table [5](#A2.T5), we observe a clear trend in the performance of perturbations applied at different layers of the model. Perturbation at the logits level achieves a favorable trade-off between calibration and efficiency. Although the perturbation applied to the fourth layer’s feature space slightly improves the ECE to 0.53%, the associated computational cost is significantly higher, with the optimal perturbation value of 13.28.

On the other hand, perturbations applied at lower feature levels (Layer 1 to Layer 3) result in severe degradation of both accuracy and calibration. Specifically, the ECE increases drastically to above 50%, and accuracy drops to approximately 10%, with a significant increase in computing time and memory use. This suggests that perturbing the features at these lower layers disrupts the model’s ability to recognize patterns and correctly classify the input data. We hypothesize that this is due to the higher sensitivity of lower layers to the raw data structure, where perturbations may significantly distort the features necessary for effective recognition.

## Appendix C Comparison of post-hoc calibration methods on other metrics

As shown in table [6](#A3.T6), The proposed CC method consistently achieves the lowest AdaECE values, outperforming the other methods. This indicates better calibration performance, in line with our discussion in the main text. For instance, in CIFAR-10, Wide-ResNet has an AdaECE of 0.40 with CC compared to 3.24 for Vanilla, showing a significant improvement. Similar results are observed across other models and datasets. The formula for Adaptive-ECE is as follows:

$$ $\text{Adaptive-ECE}=\sum_{i=1}^{B}\frac{|B_{i}|}{N}\left|I_{i}-C_{i}\right|\text{ s.t. }\forall i,j\cdot|B_{i}|=|B_{j}|$ (11) $$

**Table 6: Comparison of Post-Hoc Calibration Methods Using AdaECE$\downarrow$ Across Various Datasets and Models. AdaECE values are reported with 15 bins. The best results for each combination is in bold, and our method (CC) is highlighted. Results are averaged over 5 runs.**
| Dataset | Model | Vanilla | TS | ETS | PTS | CTS | GC | CC (ours) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR-10 | ResNet-50 | 4.33 | 2.14 | 2.14 | 2.14 | 1.71 | 1.24 | 0.64 |
| ResNet-110 | 4.40 | 1.89 | 1.89 | 1.90 | 1.31 | 0.94 | 0.96 |  |
| DenseNet-121 | 4.49 | 2.12 | 2.12 | 2.12 | 1.71 | 1.28 | 1.20 |  |
| Wide-ResNet | 3.24 | 1.71 | 1.71 | 1.71 | 1.42 | 1.17 | 0.40 |  |
| CIFAR-100 | ResNet-50 | 17.52 | 5.76 | 5.72 | 5.66 | 5.79 | 3.43 | 1.61 |
| Wide-ResNet | 15.34 | 4.48 | 4.45 | 4.41 | 4.69 | 2.24 | 1.73 |  |
| ImageNet | ResNet-50 | 3.73 | 2.07 | 2.07 | 2.06 | 3.22 | 2.56 | 1.47 |
| DenseNet-121 | 6.59 | 1.67 | 1.68 | 1.69 | 1.89 | 2.49 | 1.36 |  |
| Wide-ResNet-50 | 5.32 | 2.97 | 2.97 | 2.95 | 4.13 | 2.18 | 1.27 |  |
| ViT-B-16 | 5.59 | 4.05 | 4.06 | 4.08 | 5.50 | 1.86 | 1.76 |  |
| ViT-B-32 | 6.40 | 3.83 | 3.85 | 3.91 | 5.73 | 1.33 | 1.77 |  |

As shown in table [7](#A3.T7), The CC method also performs the best in terms of class-wise calibration, with consistently lower CECE values. This confirms that CC provides better calibration across individual classes, as discussed in the main body. For example, for ResNet-50 on CIFAR-100, CC achieves a CECE of 0.20, which is the lowest among the methods.
CECE is another measure of calibration performance that addresses the deficiency of ECE in only measuring the calibration performance of the single predicted class.
It can be formulated as:

$$ $\text{Classwise-ECE}=\frac{1}{\mathcal{K}}\sum_{i=1}^{B}\sum_{j=1}^{\mathcal{K}}\frac{|B_{i,j}|}{N}\left|I_{i,j}-C_{i,j}\right|$ (12) $$

**Table 7: Comparison of Post-Hoc Calibration Methods Using CECE$\downarrow$ Across Various Datasets and Models. CECE values are reported with 15 bins. The best-performing method for each dataset-model combination is in bold, and our method (CC) is highlighted. Results are averaged over 5 runs.**
| Dataset | Model | Vanilla | TS | ETS | PTS | CTS | GC | CC (ours) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR-10 | ResNet-50 | 0.91 | 0.45 | 0.45 | 0.45 | 0.41 | 0.46 | 0.39 |
| ResNet-110 | 0.92 | 0.48 | 0.48 | 0.48 | 0.42 | 0.52 | 0.41 |  |
| DenseNet-121 | 0.92 | 0.48 | 0.48 | 0.48 | 0.41 | 0.54 | 0.43 |  |
| Wide-ResNet | 0.68 | 0.37 | 0.37 | 0.37 | 0.37 | 0.48 | 0.32 |  |
| CIFAR-100 | ResNet-50 | 0.38 | 0.21 | 0.21 | 0.21 | 0.22 | 0.21 | 0.20 |
| Wide-ResNet | 0.34 | 0.19 | 0.19 | 0.19 | 0.20 | 0.20 | 0.18 |  |
| ImageNet | ResNet-50 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 |
| DenseNet-121 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 |  |
| Wide-ResNet-50 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.02 |  |
| ViT-B-16 | 0.03 | 0.02 | 0.02 | 0.02 | 0.03 | 0.02 | 0.02 |  |
| ViT-B-32 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 |  |

As shown in table [8](#A3.T8), interestingly, the NLL values are generally higher with the CC method compared to some other calibration methods, despite its superior calibration performance in AdaECE and CECE. This suggests that while CC improves calibration, it may come at the cost of slightly higher NLL values. For instance, for CIFAR-100 on ResNet-50, CC has a higher NLL than TS, but it remains competitive overall.

**Table 8: Comparison of Post-Hoc Calibration Methods Using NLL$\downarrow$ Across Various Datasets and Models. The best-performing method for each dataset-model combination is in bold, and our method (CC) is highlighted. Results are averaged over 5 runs.**
| Dataset | Model | Vanilla | TS | ETS | PTS | CTS | GC | CC (ours) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR-10 | ResNet-50 | 41.21 | 20.39 | 20.39 | 20.38 | 20.15 | 19.97 | 20.39 |
| ResNet-110 | 47.52 | 21.52 | 21.52 | 21.52 | 20.84 | 20.68 | 23.33 |  |
| DenseNet-121 | 42.93 | 21.78 | 21.78 | 21.78 | 21.01 | 20.30 | 22.19 |  |
| Wide-ResNet | 26.75 | 15.33 | 15.33 | 15.33 | 15.13 | 15.32 | 17.10 |  |
| CIFAR-100 | ResNet-50 | 153.67 | 106.07 | 106.07 | 106.07 | 106.25 | 107.80 | 108.40 |
| Wide-ResNet | 140.11 | 95.71 | 95.71 | 95.71 | 96.38 | 96.92 | 99.30 |  |
| ImageNet | ResNet-50 | 96.12 | 94.82 | 94.82 | 94.81 | 99.58 | 99.07 | 140.57 |
| DenseNet-121 | 109.52 | 103.90 | 103.90 | 103.91 | 106.13 | 108.14 | 162.02 |  |
| Wide-ResNet-50 | 88.56 | 86.46 | 86.46 | 86.46 | 91.68 | nan | 120.59 |  |
| ViT-B-16 | 83.71 | 78.63 | 78.63 | 78.63 | 85.19 | 82.14 | 106.89 |  |
| ViT-B-32 | 107.76 | 101.67 | 101.67 | 101.66 | 107.53 | 105.45 | 141.71 |  |

[9](#A3.T9) indicates that there is little to no change in accuracy across the calibration methods, with all methods performing similarly in terms of classification accuracy. This patter is consistent with the main section, showing CC improves calibration without sacrificing accuracy. For example, on CIFAR-10, Wide-ResNet achieves almost identical accuracy for all methods, with CC slightly outperforming others in specific cases.

**Table 9: Comparison of Post-Hoc Calibration Methods Using Accuracy$\uparrow$ Across Various Datasets and Models. Top-1 accuracy values are reported. The best results for each combination is in bold, and our method (CC) is highlighted. Results are averaged over 5 runs.**
| Dataset | Model | Vanilla | TS | ETS | PTS | CTS | GC | CC (ours) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR-10 | ResNet-50 | 95.05 | 95.05 | 95.05 | 95.05 | 94.98 | 95.05 | 95.06 |
| ResNet-110 | 95.11 | 95.11 | 95.11 | 95.11 | 95.18 | 95.11 | 95.16 |  |
| DenseNet-121 | 95.02 | 95.02 | 95.02 | 95.02 | 95.01 | 95.02 | 95.04 |  |
| Wide-ResNet | 96.13 | 96.13 | 96.13 | 96.13 | 96.06 | 96.13 | 96.13 |  |
| CIFAR-100 | ResNet-50 | 76.70 | 76.70 | 76.70 | 76.70 | 76.72 | 76.70 | 76.71 |
| Wide-ResNet | 79.29 | 79.29 | 79.29 | 79.29 | 79.17 | 79.29 | 79.31 |  |
| ImageNet | ResNet-50 | 76.08 | 76.08 | 76.08 | 76.08 | 74.62 | 76.08 | 76.08 |
| DenseNet-121 | 74.16 | 74.16 | 74.16 | 74.16 | 73.08 | 74.16 | 74.37 |  |
| Wide-ResNet-50 | 78.40 | 78.40 | 78.40 | 78.40 | 77.07 | 78.40 | 78.48 |  |
| ViT-B-16 | 81.09 | 81.09 | 81.09 | 81.09 | 80.01 | 81.09 | 81.06 |  |
| ViT-B-32 | 75.94 | 75.94 | 75.94 | 75.94 | 74.90 | 75.94 | 75.90 |  |

In figure [6](#A3.F6), we see that the proposed CC method significantly reduces both AdaECE and CECE values compared to other calibration methods, indicating better calibration for Wide-ResNet on CIFAR-10. The accuracy remains mostly unchanged across all methods, while NLL is slightly higher for CC compared to other methods like TS and ETS. This behavior is consistent with our findings in the main text.

Figure: Figure 6: Calibration performance of ResNet-50 on Cifar-10 using AdaECE$\downarrow$, CECE$\downarrow$, NLL$\downarrow$, and Accuracy$\uparrow$. ECE, AdaECE, and CECE are reported with 15 bins. Colors in the legend represent different methods. Results are averaged over 5 runs.
Refer to caption: /html/2410.12295/assets/x15.png

In Figure [6](#A3.F6), for ResNet-50 on CIFAR-10, the CC method demonstrates excellent performance with the lowest AdaECE and CECE values, further supporting its effectiveness in calibration. NLL is higher for CC, which is interesting given its superior performance in other metrics. However, accuracy remains largely unchanged, consistent with the overall findings discussed in the text.

Figure: Figure 7: Calibration performance of Wide-ResNet on CIFAR-10 using AdaECE$\downarrow$, CECE$\downarrow$, NLL$\downarrow$, and Accuracy$\uparrow$. ECE, AdaECE, and CECE are reported with 15 bins. Colors in the legend represent different methods. Results are averaged over 5 runs.
Refer to caption: /html/2410.12295/assets/x16.png

Figure [8](#A3.F8) illustrates the performance of ResNet-50 on CIFAR-100 across different calibration methods. The proposed CC method again shows the lowest AdaECE and CECE, confirming its superior calibration performance. NLL for CC is slightly higher compared to TS, but accuracy shows minimal changes across methods. These results align with our overall conclusions that CC improves calibration without sacrificing accuracy.

Figure: Figure 8: Calibration performance of ResNet-50 on CIFAR-100 using AdaECE$\downarrow$, CECE$\downarrow$, NLL$\downarrow$, and Accuracy$\uparrow$. ECE, AdaECE, and CECE are reported with 15 bins. Colors in the legend represent different methods. Results are averaged over 5 runs.
Refer to caption: /html/2410.12295/assets/x17.png

## Appendix D Comparison of various training-time calibration methods on other metrics

As shown in Table [10](#A4.T10), CC consistently outperforms baseline models across all metrics and datasets. Specifically, on CIFAR-10 and CIFAR-100, CC achieves significantly lower AdaECE scores for ResNet-50, ResNet-110, DenseNet-121, and Wide-ResNet compared to traditional methods such as Brier Loss, and MMCE. For instance, on CIFAR-100 with ResNet-110, CC reduces the AdaECE from 19.05 (baseline) to 5.28, showing superior calibration performance.

**Table 10: Comparison of Train-time Calibration Methods Using AdaECE$\downarrow$ Across Various Datasets and Models. AdaECE values are reported with 15 bins. The best results for each combination is in bold, and our method (CC) is highlighted. Results are averaged over 5 runs.**
| Dataset | Model | Cross-Entropy | Brier Loss | MMCE | LS-0.05 | FLSD-53 | FL-3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | base | ours | base | ours | base | ours | base | ours | base | ours | base | ours |
| CIFAR-10 | ResNet-50 | 4.33 | 0.64 | 1.75 | 0.99 | 4.55 | 1.06 | 3.88 | 1.74 | 1.56 | 0.36 | 1.95 | 0.71 |
| ResNet-110 | 4.40 | 0.96 | 2.60 | 0.30 | 5.07 | 1.80 | 4.48 | 2.43 | 2.08 | 0.73 | 1.64 | 0.38 |  |
| DenseNet-121 | 4.49 | 1.20 | 2.02 | 0.64 | 5.10 | 1.76 | 4.40 | 1.94 | 1.38 | 0.53 | 1.23 | 0.69 |  |
| Wide-ResNet | 3.24 | 0.40 | 1.70 | 0.57 | 3.29 | 0.63 | 4.27 | 1.54 | 1.52 | 0.42 | 1.84 | 0.42 |  |
| CIFAR-100 | ResNet-50 | 17.52 | 1.61 | 6.55 | 1.90 | 15.32 | 1.88 | 7.66 | 6.17 | 4.39 | 1.48 | 5.09 | 1.70 |
| ResNet-110 | 19.05 | 5.28 | 7.72 | 3.54 | 19.14 | 5.14 | 11.14 | 8.00 | 8.56 | 3.50 | 8.64 | 3.98 |  |
| DenseNet-121 | 20.99 | 5.85 | 5.04 | 2.02 | 19.10 | 3.90 | 12.83 | 7.06 | 3.54 | 1.52 | 4.14 | 2.03 |  |
| Wide-ResNet | 15.34 | 1.73 | 4.28 | 1.92 | 13.16 | 2.06 | 5.14 | 4.75 | 2.77 | 1.79 | 2.07 | 1.58 |  |

In Table [11](#A4.T11), the CECE results further reinforce the effectiveness of CC across all metrics. For CIFAR-10, CC improves CECE for all models compared to baseline methods. For instance, with ResNet-50, the CECE decreases from 0.91 to 0.39. Similar trends are observed on CIFAR-100, with Wide-ResNet showing a reduction in CECE from 0.34 (baseline) to 0.18 when using CC, demonstrating enhanced class-wise calibration.

**Table 11: Comparison of Train-time Calibration Methods Using CECE$\downarrow$ Across Various Datasets and Models. CECE values are reported with 15 bins. The best results for each combination is in bold, and our method (CC) is highlighted. Results are averaged over 5 runs.**
| Dataset | Model | Cross-Entropy | Brier Loss | MMCE | LS-0.05 | FLSD-53 | FL-3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | base | ours | base | ours | base | ours | base | ours | base | ours | base | ours |
| CIFAR-10 | ResNet-50 | 0.91 | 0.39 | 0.46 | 0.35 | 0.94 | 0.47 | 0.71 | 0.53 | 0.42 | 0.35 | 0.43 | 0.39 |
| ResNet-110 | 0.92 | 0.41 | 0.59 | 0.41 | 1.04 | 0.50 | 0.66 | 0.67 | 0.48 | 0.39 | 0.43 | 0.37 |  |
| DenseNet-121 | 0.92 | 0.43 | 0.46 | 0.37 | 1.04 | 0.59 | 0.60 | 0.48 | 0.41 | 0.35 | 0.42 | 0.35 |  |
| Wide-ResNet | 0.68 | 0.32 | 0.44 | 0.32 | 0.70 | 0.38 | 0.79 | 0.41 | 0.41 | 0.28 | 0.44 | 0.30 |  |
| CIFAR-100 | ResNet-50 | 0.38 | 0.20 | 0.22 | 0.19 | 0.34 | 0.18 | 0.23 | 0.22 | 0.20 | 0.19 | 0.20 | 0.19 |
| ResNet-110 | 0.41 | 0.21 | 0.24 | 0.19 | 0.42 | 0.20 | 0.26 | 0.22 | 0.24 | 0.19 | 0.24 | 0.20 |  |
| DenseNet-121 | 0.45 | 0.23 | 0.20 | 0.20 | 0.42 | 0.23 | 0.29 | 0.22 | 0.19 | 0.19 | 0.20 | 0.19 |  |
| Wide-ResNet | 0.34 | 0.18 | 0.19 | 0.18 | 0.30 | 0.17 | 0.21 | 0.19 | 0.18 | 0.17 | 0.18 | 0.17 |  |

Table [12](#A4.T12) presents the NLL comparison. It is interesting as mentioned in the main section, the CC method sometimes produces higher NLL values.

**Table 12: Comparison of Train-time Calibration Methods Using NLL$\downarrow$ Across Various Datasets and Models. The best-performing method for each dataset-model combination is in bold, and our method (CC) is highlighted. Results are averaged over 5 runs.**
| Dataset | Model | Cross-Entropy | Brier Loss | MMCE | LS-0.05 | FLSD-53 | FL-3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Base | Ours | Base | Ours | Base | Ours | Base | Ours | Base | Ours | Base | Ours |
| CIFAR-10 | ResNet-50 | 41.2 | 20.4 | 18.7 | 22.3 | 44.8 | 20.9 | 27.7 | 29.3 | 17.6 | 22.7 | 18.4 | 24.2 |
| ResNet-110 | 47.5 | 25.5 | 20.4 | 22.5 | 55.7 | 25.5 | 29.9 | 29.4 | 18.5 | 21.9 | 17.8 | 23.1 |  |
| DenseNet-121 | 42.9 | 24.0 | 19.1 | 21.2 | 52.1 | 31.2 | 28.7 | 28.5 | 18.4 | 27.2 | 18.0 | 28.3 |  |
| Wide-ResNet | 26.8 | 17.1 | 15.9 | 16.2 | 28.5 | 18.2 | 21.7 | 24.5 | 14.6 | 17.6 | 15.2 | 19.9 |  |
| CIFAR-100 | ResNet-50 | 153.7 | 113.0 | 99.6 | 133.5 | 125.3 | 116.7 | 121.0 | 133.9 | 88.0 | 128.8 | 87.5 | 128.1 |
| ResNet-110 | 179.2 | 122.3 | 110.7 | 146.9 | 180.6 | 125.3 | 133.1 | 141.4 | 89.9 | 126.9 | 90.9 | 132.0 |  |
| DenseNet-121 | 205.6 | 163.1 | 98.3 | 139.9 | 166.6 | 146.8 | 142.0 | 185.8 | 85.5 | 129.0 | 87.1 | 130.8 |  |
| Wide-ResNet | 140.1 | 102.5 | 84.6 | 98.7 | 119.6 | 109.3 | 108.1 | 136.6 | 76.9 | 108.7 | 74.7 | 106.8 |  |

Table [13](#A4.T13) presents a comparison of classification accuracies. While achieving superior calibration performance by CC, the accuracy remains unaffected across all metrics.

**Table 13: Comparison of Train-time Calibration Methods Using Accuracy$\uparrow$ Across Various Datasets and Models. Top-1 Accuracy values are reported. The best results for each combination is in bold, and our method (CC) is highlighted. Results are averaged over 5 runs.**
| Dataset | Model | Cross-Entropy | Brier Loss | MMCE | LS-0.05 | FLSD-53 | FL-3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | base | ours | base | ours | base | ours | base | ours | base | ours | base | ours |
| CIFAR-10 | ResNet-50 | 95.05 | 95.06 | 94.99 | 95.01 | 95.01 | 94.99 | 94.71 | 94.68 | 95.02 | 94.95 | 94.75 | 94.75 |
| ResNet-110 | 95.11 | 95.16 | 94.52 | 94.48 | 94.60 | 94.63 | 94.48 | 94.49 | 94.57 | 94.63 | 94.92 | 94.94 |  |
| DenseNet-121 | 95.02 | 95.01 | 94.90 | 94.86 | 94.59 | 94.60 | 94.91 | 94.91 | 94.58 | 94.51 | 94.66 | 94.66 |  |
| Wide-ResNet | 96.13 | 96.12 | 95.92 | 95.90 | 96.09 | 96.05 | 95.80 | 95.83 | 95.99 | 96.01 | 95.87 | 95.87 |  |
| CIFAR-100 | ResNet-50 | 76.70 | 76.71 | 76.60 | 76.58 | 76.80 | 76.80 | 76.56 | 76.65 | 76.79 | 76.73 | 77.24 | 77.34 |
| ResNet-110 | 77.27 | 77.17 | 74.91 | 74.79 | 76.93 | 76.96 | 76.57 | 76.64 | 77.48 | 77.49 | 77.08 | 77.04 |  |
| DenseNet-121 | 75.47 | 75.49 | 76.27 | 76.30 | 76.03 | 76.03 | 75.94 | 75.96 | 77.34 | 77.34 | 76.76 | 76.85 |  |
| Wide-ResNet | 79.29 | 79.25 | 79.43 | 79.29 | 79.27 | 79.23 | 78.83 | 78.88 | 79.91 | 79.92 | 80.30 | 80.34 |  |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: As it was mentioned in the Weaknesses section, the paper: 

1) misses a significant body of important related work and lacks novelty (which I find to be a core issue), as sensitivity-based approaches have been around for a long time.

2) lacks thorough experimental validation to support the idea that the method is indeed extendable and effective.

3) the approach has important theoretical issues that were not addressed in the method discussion.

To name a few questions: 

1. How does the proposed method distinguish itself from prior perturbation-based uncertainty methods? Could its unique contributions be clarified?

2. Why were ensembling methods not included as baselines? Would a comparison, especially with efficient ensembling variants, provide a fuller performance picture?

3. How generalizable is the method beyond image classification? Could adaptations for language models or GNNs improve its applicability?

4. Have the theoretical limitations of the method been considered, and could further analysis address potential weaknesses in the approach’s foundations?
soundness: 2
strengths: * Topic model calibration is important within the domains of robustness, safety, and uncertainty, especially considering recent developments in deep learning.

* The paper is clearly written and easy to follow, with an intuitive and easily understandable idea. The related work section provides sufficient discussion of existing calibration metrics.

* The approach does not require architectural changes to the model.
summary: In the paper, the authors propose a method to calibrate image classifier probabilities using perturbations in the input image or hidden features, instead of other popular calibration methods such as various versions of Temperature Scaling. On several image classification benchmarks (e.g., CIFAR and ImageNet), the authors demonstrate that the final recalibrated predicted probabilities—generated by sampling labels from predicted class distributions for multiple perturbed image versions and then averaging them to recompute new class probabilities—can achieve low calibration errors (e.g., ECE) and outperform some calibration baselines.
weaknesses: * **the paper overlooks a significant body of relevant literature.** For example, the idea developed in [1] to generate uncertainty by running several inferences with perturbed inputs is very similar to the approach proposed in this paper, as are other sensitivity-based methods such as [2], which shares the same motivation as the proposed method, significantly undermining the paper's novelty. Additionally, [3], mentioned in the appendix, bears a strong resemblance to the proposed method. Given this, positioning the method as a novel approach to calibration or uncertainty estimation seems overstated.


* **the experiments lack relevant baselines.** Although the authors propose a version of the method where noise is added to intermediate hidden representations, the original formulation—where noise is applied to the inputs—requires multiple inferences. In this context, ensembling-based approaches (some of which are mentioned in the appendix) should be considered as relevant baselines and should not be omitted from the experiments. Moreover, it is unclear how the inference-time vs. calibration-quality trade-off is addressed in the proposed method; some existing ensembling approaches, such as Deep Ensembles, MC-Dropout, and BatchEnsembles, may require additional computation but consistently demonstrate the highest uncertainty quality [4] (in terms of both calibration and epistemic uncertainty). Additionally, recently introduced efficient ensembling approaches, such as PackedEnsembles [5], Depth Uncertainty [6], or MixMo [7], require only a single inference to generate multiple predictions and estimate ensembling-based uncertainty, thereby mitigating computational overhead while maintaining the high uncertainty quality of ensembles.


* related to the previous point, **the method is specifically designed with image classification in mind**, making both its design and the conducted experiments quite limited. The original formulation, which involves adding perturbations to the input, is not feasible for other popular domains, such as language modeling, where adding Gaussian noise is not feasible. Additionally, certain architectures, such as GNNs, are also not suitable for this approach. Using hidden representations to overcome these limitations appears more like an ad hoc solution to issues of computational overhead and broader applicability rather than a well-designed solution. A potential drawback of this version is the potential degradation of model performance (e.g., accuracy), as it resembles Dropout (or Gaussian Dropout [8]), which is known to degrade model performance [4].

* the following example illustrates the **potential ineffectiveness of the proposed approach**: consider the setup in Figure 1(b) and a point, such as (2, -4), which lies deep within the predicted class (class 0) but should have higher predicted uncertainty, as it is not near the center of the red class. Adding Gaussian noise to this point won’t change the predicted class, so the recalibrated predictions would remain highly miscalibrated. A similar issue could occur when the predicted softmax vector changes under different perturbations, but the argmax class remains nearly the same, resulting in a degenerate distribution and miscalibrated predictions. By contrast, ensembling approaches rarely exhibit this behavior.

[1] Mi, Lu, et al. "Training-free uncertainty estimation for dense regression: Sensitivity as a surrogate." AAAI 2022.

[2] Durasov, Nikita, et al. "Zigzag: Universal sampling-free uncertainty estimation through two-step inference." TMLR 2024.

[3] Conde, Pedro, et al. "Approaching test time augmentation in the context of uncertainty calibration for deep neural networks." arXiv 2023.

[4] Ashukha, Arsenii, et al. "Pitfalls of in-domain uncertainty estimation and ensembling in deep learning." ICLR 2020.

[5] Laurent, Olivier, et al. "Packed-ensembles for efficient uncertainty estimation." ICLR 2023.

[6] Antorán, Javier, et al. "Depth uncertainty in neural networks." NeurIPS 2020.

[7] Ramé, Alexandre, et al. "Mixmo: Mixing multiple inputs for multiple outputs via deep subnetworks." ICCV 2021.

[8] Gal, Yarin, and Zoubin Ghahramani. "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." ICML 2016.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - Are ECE and AdaECE calculated on the original test set or the perturbed set? Lines 229-231 suggest it’s the perturbed set.
- The sensitivity comparison in Lines 266-269 is hard to assess since the scales differ. The error behavior outside the range 0.5-1 is also unclear. Plus, it’s uncertain if this analysis on 2D data would apply to high-dimensional data like images.
soundness: 3
strengths: - The proposed consistency calibration method is interesting.
- The writing is generally clear.
- The experiments are conducted across CNN and ViTs over cifar10, 100, ImageNet and ImageNet-LT.
summary: This paper proposes a method called consistency calibration, which uses perturbed logits for calibration. Consistency under perturbation is treated as an indicator of confidence, aiming to improve the model's calibration.
weaknesses: - Some results or interpretations are unclear
   - Figure 1(a) is missing "Moderate Augmentation" in the legend. Does the accuracy of Train Augmentation go below 92%? Adding detailed numerical results would help clarify.
   - The final results use CC as a post-hoc method, but Sec 2.5 recommends training-time augmentation, which seems inconsistent.
   - Figure 3(c) only shows a case study for comparing CC, CC (Train Aug), and CC (Train Aug + Jitter). It would help to include more performance comparisons between these methods.

- The analysis of why consistency calibration work is not sufficient. The larger logit gap between correct and incorrect samples might help maintain accuracy under logit perturbation, but it doesn’t fully explain the better calibration. A more rigorous or theoretical analysis is needed.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 3
questions: see weakness
soundness: 3
strengths: Introduction of a New Calibration Perspective: This paper introduces consistency as a new criterion for calibration, providing a potentially more reliable alternative to traditional methods of evaluating model confidence.
Efficient Implementation: Consistency Calibration uses input perturbations to measure uncertainty, requiring no additional data or label information, which allows for a simple and computationally efficient post-hoc calibration process.
summary: This paper proposes a new approach called Consistency Calibration (CC) to improve uncertainty calibration in deep learning models. Instead of the traditional reliability-based calibration methods like Expected Calibration Error (ECE), the authors introduce a novel perspective called consistency, which evaluates the model's confidence by measuring whether predictions remain stable across perturbed inputs. The effectiveness of CC is validated on various datasets, including CIFAR-10, CIFAR-100, ImageNet, and ImageNet-LT.
weaknesses: Lack of Scalability Validation for Large Models: While the proposed method demonstrates superior performance on several datasets, there is a lack of validation on larger models, such as massive language models or more complex real-world scenarios. Future work should include experiments to assess the scalability of CC in such settings.
Need for Optimization of Perturbation Strength and Noise Type: Consistency Calibration requires optimization of the perturbation strength and noise type, which adds hyperparameter tuning complexity. To address this issue, automated hyperparameter tuning techniques should be explored to enhance practical usability.
Limited Experimental Scope for Calibration Performance: The experiments are focused primarily on key image datasets, with no validation in other domains (e.g., natural language processing, speech recognition). Further research should evaluate the performance of Consistency Calibration across various domains to establish its broader applicability.