Optimizing Calibration by Gaining Aware of Prediction Correctness

## Abstract

Abstract Model calibration aims to align confidence with prediction correctness.
The Cross-Entropy (CE) loss is widely used for calibrator training, which enforces the model to increase confidence on the ground truth class. However, we find the CE loss has intrinsic limitations. For example, for a narrow misclassification, a calibrator trained by the CE loss often produces high confidence on the wrongly predicted class ( e.g . , a test sample is wrongly classified and its softmax score on the ground truth class is around 0.4), which is undesirable. In this paper, we propose a new post-hoc calibration objective derived from the aim of calibration. Intuitively, the proposed objective function asks that the calibrator decrease model confidence on wrongly predicted samples and increase confidence on correctly predicted samples.
Because a sample itself has insufficient ability to indicate correctness, we use its transformed versions ( e.g . , rotated, greyscaled and color-jittered) during calibrator training.
Trained on an in-distribution validation set and tested with isolated, individual test samples,
our method achieves competitive calibration performance on both in-distribution and out-of-distribution test sets compared with the state of the art.
Further, our analysis points out the difference between our method and commonly used objectives such as CE loss and mean square error loss, where the latters sometimes deviates from the calibration aim.

## 1 Introduction

Model calibration is an important technique to enhance the reliability of machine learning systems.
Generally, it aims to align predictive uncertainty (a.k.a. confidence) with prediction accuracy.
We are interested in post-hoc accuracy preserving calibrators that scale the model output to make it calibrated (Guo et al., ; Zhang et al., ; Tomani et al., ; Gupta et al., ; Kull et al., ).

Figure: Figure 1: A failure example for calibrators trained by the Cross-Entropy (CE) or Mean Square Error (MSE) loss. A classifier makes a wrong prediction of a cat image. Before calibration, the classifier gives probabilities of 0.412 and 0.455 on the ground-truth and predicted classes, respectively. The calibrator trained with the CE loss assigns even higher confidence 0.538 to the wrong class, making things worse, and that trained by MSE maintains a similar confidence 0.456. In comparison, calibrator trained with the proposed Correctness-Aware (CA) loss effectively decreases confidence of this wrong prediction to 0.292, improving calibration.
Refer to caption: x1.png

Existing methods typically use Maximum Likelihood Estimation (MLE) to train a calibrator for the classification task, such as the Mean Square Error (MSE) loss (Tomani et al., , ; Zhang et al., ) and the Cross-Entropy (CE) loss (Guo et al., ; Zou et al., ). Although these approaches demonstrate efficacy in reducing calibration errors such as expected calibration error and Brier scores, they lack theoretical guarantee that the calibration error is minimized when MLE converges. In Fig. 1, for an image which is incorrectly classified and has a relatively high probability on the ground truth class, calibrators trained by the CE or MSE loss would give high confidence on the wrongly predicted class. It means a user may trust this prediction to be true. As to be revealed in Sec. 3.4, the inherent problem of CE and MSE loss functions limits them in calibrating such test cases, but our method effectively reduces the confidence.

In this paper, we derive a concrete interpretation of the goal of calibration which is then directly translated into a novel loss function that aligns with the newly interpreted goal.
Specifically, we start from the general definition of calibration and its error (Guo et al., ) and then, under a finite test set, represent the error in a discretized form. Minimizing this discretized error gives us a very interesting and intuitive calibration goal: A correct prediction should have possibly high confidence and a wrong prediction should have possibly low confidence.
Theoretically, this optimization goal can naturally reduce the overlap of confidence values between correct and incorrect predictions.

We translate this goal into a loss function that enforces high confidence (*i.e*., 1) for correctly classified samples, and low confidence (*i.e*., $\frac{1}{C}$, where $C$ is the number of classes) for wrongly classified ones. We name it as the Correctness-Aware (CA) loss. Yet for this method, it is non-trivial to identify classification correctness. To address this, we propose to use transformed versions of original images as the calibrator input: consistency between their prediction results suggests prediction correctness of the original sample.

Our method allows a calibrator to be trained on the in-distribution validation set and directly applied to individual test samples during inference, which is consistent with practical usage.
We demonstrate the effectiveness of the proposed strategy on various test sets. In both in-distribution (IND) and out-of-distribution (OOD) test sets, our method is clearly superior to uncalibrated models and competitive compared with state-of-the-art calibrators. Moreover, our method demonstrates the potential to better separate correct and incorrect test samples using their calibrated confidence. Below we summarize the main contributions.

- •
Theoretically, we derive the concrete goal of model calibration that has clear semantic meaning. This allows us to design a new calibration loss function: correctly classified samples should have high confidence, while incorrectly classified ones low confidence.
- •
To indicate prediction correctness, we use the softmax prediction scores of transformed versions of an original image as calibrator inputs.
- •
Our method achieves competitive calibration performance on various IND and OOD datasets.
- •
We diagnose commonly used calibration loss functions including the CE and MSE loss: they are often limited under test samples of certain characteristics.

## 2 Related Work

Loss functions used in post-hoc calibration. Existing post-hoc calibrators typically use the Maximum Likelihood Estimation (MLE) for optimization (Jung et al., ; Tao et al., ). For example, Mukhoti et al. () use the CE loss, while Kumar et al. () use the MSE loss. Additionally, Mukhoti et al. () uses the focal loss, a variant to CE, which enhances learning on wrong predictions. Guo et al. () directly optimizes ECE. This paper identifies inherent problems with MLE for calibration and derives a new loss function that better aligns with calibration goal.

OOD calibration deals with distribution shifts in test sets (Tomani et al., ).
A useful practice is modifying the calibration set to cover OOD scenarios (Tomani et al., ; Krishnan & Tickoo, ), but these methods
are usually designed for specific OOD scenarios which may lead to compromised in-distribution (IND) calibration performance.
Another line of works adapt to test data distribution that can be either OOD or IND, using domain adaptation (Wang et al., ), calibration sets from multiple domains (Gong et al., ), or improving calibration sets by estimating test set difficulty (Zou et al., ). In comparison, our calibrator is trained on the IND validation set only, and does not need test batches for update, but still demonstrates improved and competitive OOD calibration performance.

Predicting classification correctness has not been widely studied in the community. Among the few, Xia & Bouganis () investigate a three-way classification problem: classify a test sample into correct prediction, wrong prediction, or an out-of-distribution sample (its category is outside the training label space). We find the task of predicting classification correctness closely connected to model calibration.

## 3 Approach

Figure: Figure 2: Calibration pipeline. For a given image sample $\mathbf{X}$, we first obtain its logit vector $\mathbf{z}$ and softmax vector $\mathbf{v}$. We then apply $M$ different transformations (*e.g*., rotation, greyscale, colorjitter, *etc*.) on $\mathbf{X}$ to get its transformed versions as well as their related softmax vectors as $\mathbf{v}_{i}$ ($i\in\mathcal{I}_{M}$). Indices $\mathbf{q}\in\mathbb{R}^{k}$ of the top-$k$ largest probabilities (softmax scores) in $\mathbf{v}$ are used to acquire top-$k$ scores from $\mathbf{v}_{i}$ to form the concatenated input $\oplus_{i\in\mathcal{I}_{M}}\mathbf{v}_{i}[\mathbf{q}]$ to the calibrator. The calibrator outputs a temperature $\tau$, then being used to update the logit vector $\mathbf{z}$ to produce the calibrated softmax vector. We use our proposed Correctness-Aware (CA) loss.
Refer to caption: x2.png

### 3.1 Defining calibration error from i ts goal

Notations. We study calibration under the multi-way classification problem. Regular fonts are scalars, *e.g*., $\tau$; vectors are denoted by lowercase boldface letters, *e.g*., $\mathbf{x}$; matrices by the uppercase boldface, *e.g*., X for an image (^1^11For simplicity, we omit three color channels.). $\mathcal{I}_{C}$ denotes an index set of integers $\{1,...,C\}$, operator ‘;’ and $\oplus$ concatenate vectors, *e.g*., $\oplus_{i\in\mathcal{I}_{M}}\mathbf{v}_{i}\!=[\mathbf{v}_{1};...;\mathbf{v}_{M}]$.
A classifier $f$ takes a $d$-dimensional input $\mathbf{x}\in\mathbb{R}^{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}
{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}$ and its corresponding label $y\in\mathcal{I}_{C}$ with $C$ classes which are sampled from the joint distribution $p(\mathbf{x},y)=p(y|\mathbf{x})p(\mathbf{x})$. We use $\equiv$ to denote the equivalence.
The output of $f$ is denoted as $f(\mathbf{X})=(\hat{y},\hat{c})$, where $\hat{y}$ and $\hat{c}$ denote the predicted class label and maximum confidence score, respectively.

Calibration goal. According to Guo et al. (), the goal of model calibration is to ‘align confidence with the accuracy of samples’. Based on this, existing literature define perfect calibration as:

$$ $\mathbb{P}(\hat{y}=y|\hat{c}=c)=c,\forall c\in[0,1].$ (1) $$

Our calibration error formulation. We interpret Eq. (1) as: for any predicted confidence $\hat{c}$, the expected classification accuracy $\mathbb{E}^{\text{acc}}_{\hat{c}}$ of model $f$ on the conditional distribution $p(\mathbf{x}|\hat{c})$ should equal $\hat{c}$. Based on this interpretation, we write the calibration error of classifier $f$ as a function of $\hat{c}$:

$$ $l_{f}(\hat{c})={\|\hat{c}-\mathbb{E}^{\text{acc}}_{\hat{c}}\|}=\|\hat{c}-\int \mathbb{I}\{y_{\mathbf{x}}=\hat{y}_{\mathbf{x}}\}dp(\mathbf{x}|\hat{c})\|,$ (2) $$

where $\|\cdot\|$ denotes discrepancy measurement such as the $\ell_{2}$- distance. The indicator function $\mathbb{I}$$\{\cdot\}$ returns $1$ if the given condition ( the prediction matches the ground truth label accurately) is true; otherwise, it returns 0.

Denoting the distribution of predicted confidence as $p(\hat{c})$ and its probability density function as $dp({\hat{c}})$, the expectation of calibration error(^2^22This differs from the expected calibration error (ECE) metric. The ECE metric uses discretized histogram bins, whereas our calibration goal employs the continuous form.) of $f$ on $p(\hat{c})$ can be expressed as:

$$ $\mathbb{E}_{f}=\int l_{f}(\hat{c})dp(\hat{c}),$ (3) $$

where $l_{f}(\hat{c})$ is defined in Eq. (2). A model $f$ is considered to be perfectly calibrated if $E_{f}=0$. Model calibration is to optimize a calibrator which reduces $\mathbb{E}_{f}$ as much as possible.

### 3.2 Correctness-aware loss

In practice, the distribution $p(\hat{c})$ is unknown, hence, Eq. (3) cannot be directly computed. To solve this, we approximate the calibration error $\mathbb{E}_{f}$ by replacing $p(\hat{c})$ with an empirical distribution, formed by assembling Dirac delta functions (Dirac, ) centered at each predicted sample confidence $\hat{c}$ computed from a given dataset $\mathcal{D}=\{(\mathbf{x}_{i},y_{i})\}^{n}_{i=1}$, where $n$ is the number of samples:

$$ $dp{(\hat{c})}=\frac{1}{n}\sum_{i=1}^{n}\delta_{\hat{c}_{i}}(\hat{c}).$ (4) $$

Substituting Eq. (2) and Eq. (4) into Eq. (3), the empirical calibration loss can be written as:

$$ $\begin{split}\mathbb{E}_{f}^{\text{emp}}=\frac{1}{n}\sum_{i=1}^{n}\|\hat{c_{i} }-\int\mathbb{I}\{y_{\mathbf{x}_{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}i}}=\hat{y}_{\mathbf{x}_{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}i}}\}dp(\mathbf{x}_{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}i}|\hat{c_{i} })\|.\end{split}$ (5) $$

However, Eq. (5) is still hard to compute because the probability density function $dp(\mathbf{x}_{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}i}|\hat{c}_{i})$ is not accessible in practice. As such, we further discretize Eq. (5), where we assume a finite number of samples $\{\mathbf{x}_{ij}\}^{m}_{j=1}$ in each distribution $p(\mathbf{x}_{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0
}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}i}|\hat{c}_{i})$. Consequently, $\mathbb{E}_{f}^{\text{emp}}$ is reformulated as:

$$ $\begin{split}\mathbb{E}_{f}^{\text{emp}}=\frac{1}{nm}\sum_{i=1}^{n}\sum_{j=1}^ {m}\|\hat{c_{i}}-\mathbb{I}\{y_{\mathbf{x}_{ij}}=\hat{y}_{\mathbf{x}_{ij}}\}\| .\end{split}$ (6) $$

Note that in practice, $m=1$, because there is only one test sample for each $p(\mathbf{x}_{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0
}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}i}|\hat{c}_{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}i})$, *i.e*., a dataset only has one test sample with confidence 0.52893(^3^33We have a mild assumption that a test set has few, if any, duplicated images, where Eq. (6) will also approximately hold.). Therefore, the empirical calibration loss of classifier $f$ on a test set with $n$ samples should be:

$$ $\mathbb{E}_{f}^{\text{emp}}=\frac{1}{n}\sum_{i=1}^{n}\|\hat{c}_{i}-\mathbb{I} \{y_{\mathbf{x}_{i}}=\hat{y}_{\mathbf{x}_{i}}\}\|.$ (7) $$

Lower and upper bounds of CA loss.
Let us derive the CA loss range of Eq. (7) for a given test sample $\mathbf{x}_{i}$:
(i) If the sample is correctly classified, the indicator function $\mathbb{I}^{+}\{y_{\mathbf{x}_{i}}=\hat{y}_{\mathbf{x}_{i}}\}=1$. The maximum confidence for a correctly classified sample is $\hat{c}_{i}^{+}\in(\frac{1}{C},1]$, and the loss range would be $\|\hat{c}_{i}^{+}-\mathbb{I}^{+}\{y_{\mathbf{x}_{i}}=\hat{y}_{\mathbf{x}_{i}}
\}\|\in[0,1-\frac{1}{C})$.
(ii) If the sample is wrongly classified, $\mathbb{I}^{-}\{y_{\mathbf{x}_{i}}=\hat{y}_{\mathbf{x}_{i}}\}=0$ and $\hat{c}_{i}^{-}\in(\frac{1}{C},1]$, the loss range would be $(\frac{1}{C},1]$.
Let $\rho\in[0,1]$ be the ratio of correctly classified samples in $\mathcal{D}$.
For the correctly classified samples, we have:

$$ $0\leq\sum_{i=1}^{\rho\times n}\|\hat{c}_{i}^{+}-\mathbb{I}^{+}\{y_{\mathbf{x}_ {i}}=\hat{y}_{\mathbf{x}_{i}}\}\|<\rho\times n\times\left(1\!-\!\frac{1}{C} \right),\\$ (8) $$

and for the wrongly classified samples, we have:

$$ $\displaystyle(1-\rho)\times n\times\frac{1}{C}<\sum_{i=1}^{(1-\rho)\times n}\| \hat{c}_{i}^{-}-\mathbb{I}^{-}\{y_{\mathbf{x}_{i}}=\hat{y}_{\mathbf{x}_{i}}\}\|$ $\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq(1-\rho)\times n.$ (9) $$

By combining Eq. (8) and (9) with Equation (7), we can now derive the lower and upper bounds of our CA loss:

$$ $\displaystyle\quad\quad\!\mathbb{E}_{f}^{\text{emp}}=\frac{1}{n}\sum_{i=1}^{n} \|\hat{c}_{i}-\mathbb{I}\{y_{\mathbf{x}_{i}}=\hat{y}_{\mathbf{x}_{i}}\}\|$ $\displaystyle n\times\mathbb{E}_{f}^{\text{emp}}=\sum_{i=1}^{n}\|\hat{c}_{i}- \mathbb{I}\{y_{\mathbf{x}_{i}}=\hat{y}_{\mathbf{x}_{i}}\}\|$ $\displaystyle\quad\quad\quad\quad\!\!=\Bigg{(}\sum_{j=1}^{\rho\times n}\|\hat{ c}_{j}^{+}-\mathbb{I}^{+}\{y_{\mathbf{x}_{j}}=\hat{y}_{\mathbf{x}_{j}}\}\|$ $\displaystyle\quad\quad\quad\quad\!\!+\sum_{k=1}^{(1-\rho)\times n}\|\hat{c}_{ k}^{-}-\mathbb{I}^{-}\{y_{\mathbf{x}_{k}}=\hat{y}_{\mathbf{x}_{k}}\}\|\Bigg{)}$ (10) $\displaystyle n\times\left(\frac{1-\rho}{C}\right)\leq n\times\mathbb{E}_{f}^{ \text{emp}}\leq n\times\rho\times\left(1\!-\!\frac{1}{C}\right)$ $\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad+n \times(1-\rho)$ $\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad=n \times\left(\frac{1-\rho}{C}+\frac{C-1}{C}\right)$ $\displaystyle\quad\quad\quad\quad\quad\frac{1-\rho}{C}\leq\mathbb{E}_{f}^{ \text{emp}}\leq\frac{1-\rho}{C}+\frac{C-1}{C},$ (11) $$

where $C$ is the number of classes.
Hence, the CA loss has a lower bound of $\frac{1-\rho}{C}$ and an upper bound of $\frac{1-\rho}{C}+\frac{C-1}{C}$. We observe that both lower and upper bounds are closely tied to $\frac{1-\rho}{C}$, representing the fraction of misclassified samples per class in whole $\mathcal{D}$.

Theoretical insights of CA loss.
First, we rewrite Eq. (7) as a sum of two CA loss components, $\mathbb{I}\{y_{\mathbf{x}_{i}}=\hat{y}_{\mathbf{x}_{i}}\}$ equals $1$ and 0 0 for correctly and incorrectly classified samples, respectively, and we use $\ell_{1}$ as discrepancy measurement for simplicity:

$$ $\mathbb{E}_{f}^{\text{emp}}=\frac{1}{n}\Bigg{(}\sum_{j=1}^{\rho\times n}(1- \hat{c}_{j}^{+})\!+\!\sum_{k=1}^{(1-\rho)\times n}\hat{c}_{k}^{-}\Bigg{)},$ (12) $$

where $\hat{c}_{j}^{+}$ and $\hat{c}_{k}^{-}$ denote, respectively, the maximum confidence scores for correctly and incorrectly classified samples. Suppose $\rho\geq 50\%$, we have:

$$ $\displaystyle\mathbb{E}_{f}^{\text{emp}}=\frac{1}{n}\Bigg{(}\sum_{j=1}^{(1- \rho)\times n}(1-\hat{c}_{j}^{+})+\sum_{j=(1-\rho)n+1}^{\rho\times n}(1-\hat{c }_{j}^{+})$ $\displaystyle\qquad\qquad\qquad+\sum_{k=1}^{(1-\rho)\times n}\hat{c}_{k}^{-} \Bigg{)}$ $\displaystyle\qquad=1-\rho+\frac{1}{n}\Bigg{(}\sum_{k=1}^{(1-\rho)\times n} \hat{c}_{k}^{-}-\sum_{j=1}^{(1-\rho)\times n}\hat{c}_{j}^{+}\Bigg{)}$ $\displaystyle\qquad\qquad\qquad+\frac{1}{n}\sum_{j=(1-\rho)n+1}^{\rho\times n} (1-\hat{c}_{j}^{+}).$ (13) $$

Now using $\mathbb{E}^{\text{diff}}\!\!=\!\!\frac{1}{(1-\rho)\times n}\Big{(}\!\sum_{k=1}
^{(1-\rho)\times n}\hat{c}_{k}^{-}\!-\!\sum_{j=1}^{(1-\rho)\times n}\hat{c}_{j
}^{+}\!\Big{)}$ to denote the expectation of the difference in maximum confidence values between correct and incorrect predictions, and $\mathbb{E}^{\text{+}}=\frac{1}{(2\rho-1)n}\sum_{j=(1-\rho)n+1}^{\rho\times n}(
1-\hat{c}_{j}^{+})$ to denote the expectation of the maximum softmax scores for the rest $(2\rho-1)n$ correctly classified samples, Eq. (13) can be written as:

$$ $\displaystyle\mathbb{E}_{f}^{\text{emp}}=(1-\rho)\mathbb{E}^{\text{diff}}+(2 \rho-1)\mathbb{E}^{\text{+}}+(1-\rho)$ $\displaystyle\qquad\equiv(1-\rho)\mathbb{E}^{\text{diff}}+(2\rho-1)\mathbb{E}^ {\text{+}}$ (14) $$

We notice that minimizing our CA loss is equivalent to minimizing either $\mathbb{E}^{\text{diff}}$ or $\mathbb{E}^{\text{+}}$ (omitting the constant $(1-\rho)$): (i) minimizing $\mathbb{E}^{\text{diff}}$ aims to maximize the expectation of the difference in maximum confidence scores between correct and incorrect predictions, and (ii) minimizing $\mathbb{E}^{\text{+}}$ aims to push the maximum confidence score of correctly classified samples to 1. These two objectives align well with the model calibration goal. Below we take a close look at $\mathbb{E}^{\text{diff}}$:

$$ $\displaystyle\qquad\mathbb{E}^{\text{diff}}=\frac{1}{(1-\rho)\times n}\Bigg{(} \sum_{k=1}^{(1-\rho)\times n}\hat{c}_{k}^{-}-\sum_{j=1}^{(1-\rho)\times n}\hat {c}_{j}^{+}\Bigg{)}$ $\displaystyle=\frac{1}{1-\rho}\Bigg{(}\mathbb{E}_{f}^{\text{emp}}-\frac{1}{n} \sum_{j=(1-\rho)n+1}^{\rho\times n}\!\!\!(1-\hat{c}_{j}^{+})+\rho-1\Bigg{)}$ $\displaystyle\qquad\mathbb{E}_{f}^{\text{emp}}\rightarrow\frac{1-\rho}{C} \equiv\mathbb{E}^{\text{diff}}\rightarrow\frac{\Big{(}\frac{1}{C}-1\Big{)}\rho }{1-\rho}<0.$ (15) $$

Eq. (15) demonstrates that minimizing our CA loss $\mathbb{E}_{f}^{\text{emp}}$ during training toward the lower bound $\frac{1-\rho}{C}$ is equivalent to pushing $\mathbb{E}^{\text{diff}}$ toward $\frac{\big{(}\frac{1}{C}-1\big{)}\rho}{1-\rho}<0$. This means pushing the average maximum softmax scores of wrongly classified samples away from those of correctly classified samples, thereby reducing the overlap of confidence values between correct and incorrect predictions.

### 3.3 Gaining correctness awareness

From the intuition of the CA loss in Eq. (7), its optimization requires the post-hoc calibrator to be aware of the correctness of each test sample. Empirically, we find the test sample itself offer limited help to distinguish correctness, which leads to undesirable calibration performance.

Our solution to gaining correctness awareness is motivated by (Deng et al., ). They find consistency of model predictions of transformed (*e.g*., rotated, gray-scaled, *etc*.) images is highly correlated with model accuracy. While the insight from (Deng et al., ) is on the dataset level, our assumption is on the individual sample level: model behavior on transformed samples is useful to inform the correctness of model predictions.

The pipeline of our calibrator is presented in Fig. 2, we aim to calibrate a classification ‘model’ $f$. To do so, we compute the logit vector $\mathbf{z}$, softmax vectors of an original image $\mathbf{X}$ and its transformed versions $\mathbf{v}$ and $\mathbf{v}_{i}$ ($i\in\mathcal{I}_{M}$, assuming $M$ types of transforms), respectively. We determine the indices $\mathbf{q}\in\mathbb{R}^{k}$ of the $k$ largest softmax scores of $\mathbf{v}$, and use these indices $\mathbf{q}$ to locate and select the corresponding values from $\mathbf{v}_{i}$, forming new vector with $k$ dimensions. These $k$-dim vectors $\mathbf{v}_{i}[\mathbf{q}]$ of the transformed images are concatenated as $\oplus_{i\in\mathcal{I}_{M}}\mathbf{v}_{i}[\mathbf{q}]$, and used as the calibrator input.

Calibrator, $g$ parameterized by $\theta$, which is trained by the proposed CA loss. Here, the calibrator consists of two fully connected layers with a ReLU activation function in between. Each hidden layer comprises 5 nodes. The calibrator is optimized on the calibration (a.k.a., validation) set, and the calibration output is temperature $\tau$ to be used to scale the model logit vector $\mathbf{z}$ of original image. Below we show these steps in equations.

$$ $\displaystyle\mathbf{q}=\underset{\mathbf{q}}{\text{argmax}}\mathbf{v}\in \mathbb{R}^{k}$ (16) $\displaystyle\tau=g_{\theta}(\oplus_{i\in\mathcal{I}_{M}}\mathbf{v}_{i}[ \mathbf{q}])$ (17) $\displaystyle\hat{c}=\underset{c}{\text{max}}\sigma(\mathbf{z}/\tau)^{(c)}$ (18) $$

where $\sigma(\cdot)$ denotes the softmax function. Eq. (18) retrieves the maximum softmax prediction score $\hat{c}$.
Based on the CA loss in Eq. (7), the optimization goal now becomes:

$$ $\arg\min_{\theta}\mathbb{E}_{f}^{\text{emp}}(\hat{c},y_{\mathbf{x}},\hat{y}_{ \mathbf{x}}),$ (19) $$

where $y_{\mathbf{x}}$ and $\hat{y}_{\mathbf{x}}$ denote respectively the ground truth and predicted labels. Algorithm 1 summarises this calibration process.

Figure: Algorithm 1 Calibrator with our proposed Correctness-Aware Loss

In practice, transformations can be grayscale, rotation, color jitter, adding Gaussian noise, random erasing, *etc*. They are applied to the original image during training and inference. We do not assume access to test batches, which are used in some previous works (Guo et al., ; Wang et al., ). Nevertheless, if we assume such access, we can retrieve from test batches images that are similar to the original one and use softmax vectors of the retrieved images as calibrator inputs. As to be shown in Sec. 4.4, we find that grayscale, rotation, and colorjitter are effective ones, and that using four transformations give a good trade-off between calibration performance and computational cost.

During inference, using the $k$-dim vectors from the transformed images, we obtain an adjusted temperature from the calibrator. This temperature is used to scale the logits of the original image, the softmax vector of which is then updated.

Figure: Figure 3: Comparison of different loss functions w.r.t. temperature and softmax probability of the ground truth (GT) class. In a four-way classification task, we examine a wrongly predicted sample with logit vector $[a,2.0,0.1,0.05]$, where $a<2$ is the value on the ground truth class. We use $c_{\text{gt}}$ to donate the softmax score of the GT class. Top: The loss surface plots for varying temperatures and $c_{\text{gt}}$, with red and blue arrows representing positive and negative temperature gradients, respectively. Bottom: Shows 2D loss curves for varying $c_{\text{gt}}$. The lines in the bottom charts correspond to the lines of the same color in the top charts. Compared with traditional Maximum Likelihood Estimation (MLE) based functions (*e.g*., Cross-Entropy, Mean Squared Error), our Correctness-Aware loss minimization does not favor temperatures below 1 for incorrect predictions, while sometimes MLE does.
Refer to caption: x3.png

### 3.4 Comparison between CA loss and MLE

Maximum Likelihood Estimate ( MLE) is widely used for calibration training (Kumar et al., ), under concrete formats such as the Cross-Entropy (CE) or Mean Square Error (MSE) losses. This section goes through their connections and differences with the CA loss.

For correct predictions, MLE (*e.g*., CE or MSE loss) has similar effect with the CA loss. MLE enforces the softmax probability of the ground-truth class to be close to 1. For correct predictions, the softmax probability of the ground-truth class equals the sample confidence (maximum probability in the softmax vector). Under this scenario, MLE aligns with both the calibration objective and the CA loss: the confidence of correct predictions should be possibly high.

For wrong predictions, MLE sometimes deviates from the calibration goal while CA is theoretically consistent. In Fig. 3, we visualize the loss surface of the CE, MSE, and CA loss w.r.t. temperature and softmax probability on the ground truth class (first row), from which we use examples of two typical calibration training samples for more intuitive illustration (both first and second rows). Particularly, optimal temperature ($x$-axis) is achieved when the respective loss ($y$ axis) is minimum (the second row is easier to read).

For an absolutely wrong sample (green curves in both rows), whose probability of the wrongly predicted class is far greater than that on the ground-truth class, the optimization direction of MLE is similar to CA: the loss curve keeps decreasing and finally a large temperature or a low confidence is obtained. In fact, under this scenario, calibration objective requires the probability of the ground-truth class to increase and probability of the wrongly predicted class to decrease. This is consistent with the objective of MLE: to increase probability on the ground-truth class.

For a narrowly wrong sample (yellow curves in both rows), whose probability on the wrongly predicted class is much closer to that on the ground-truth class, the optimization direction of MLE is very different from or even opposite to CA. Take the yellow curves in the second row of Fig. 3 as example. The CE loss, to become smaller, leads to a small temperature, meaning a large confidence, which is undesirable for this wrongly predicted sample. For MSE, its minimum is achieved when temperature is around 1.0, which does not change the temperature and confidence much. This again is undesirable. In comparison, the CA loss keep decreasing when temperature increases so will eventually give a large temperature or a small confidence for this type of samples. This is consistent with the calibration objective.

Empirically, we find that such narrowly wrong predictions take up 2%-8% of the calibration set (ImageNet validation)(^4^44We first compute the ratio of the probability on the ground-truth class to that on the wrongly predicted class. We define a sample is narrowly wrong prediction if this ratio is higher than 0.5.). This would negatively impact training efficacy of MLE. Moreover, during inference, if a test set has many such narrowly wrong predictions, MLE will also be negatively impacted because of its unsuitable in dealing with such samples during training. This would explain why our system is superior to and on par with state of the art on OOD and IND test sets, respectively (refer to Sec. 4). But in both IND and OOD scenarios, our calibrated models are much better than uncalibrated ones.

**Table 1: Calibrator comparison under the ImageNet setup. Each reported number is averaged over 10 classifiers, described in Section 4.1. We use five test sets: ImageNet-Val, ImageNet-A, ImageNet-R, ImageNet-S, and ObjectNet, and four metrics: ECE (bin=25), BS, KS and AUC (AUROC). Best results in each column are in bold. When comparing CA and CE, better results are highlighted in blue.**
| Method | ImageNet-Val | ImageNet-A | ImageNet-R | ImageNet-S | ObjectNet |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ECE$\downarrow$ | BS$\downarrow$ | KS$\downarrow$ | AUC$\uparrow$ | ECE$\downarrow$ | BS$\downarrow$ | KS$\downarrow$ | AUC$\uparrow$ | ECE$\downarrow$ | BS$\downarrow$ | KS$\downarrow$ | AUC$\uparrow$ | ECE$\downarrow$ | BS$\downarrow$ | KS$\downarrow$ | AUC$\uparrow$ | ECE$\downarrow$ | BS$\downarrow$ | KS$\downarrow$ | AUC$\uparrow$ |  |
| Uncal | 8.71 | 14.01 | 14.39 | 85.96 | 39.44 | 32.90 | 43.47 | 61.87 | 13.97 | 16.89 | 19.90 | 88.06 | 20.92 | 21.67 | 29.01 | 82.57 | 31.21 | 25.20 | 36.48 | 78.05 |
| TS | 9.10 | 13.85 | 13.26 | 85.41 | 29.24 | 23.24 | 32.50 | 62.86 | 6.28 | 13.80 | 14.51 | 88.27 | 8.92 | 16.11 | 19.18 | 83.22 | 19.70 | 17.97 | 26.92 | 78.35 |
| ETS | 3.22 | 12.61 | 12.43 | 85.90 | 32.40 | 26.21 | 36.98 | 62.68 | 8.29 | 14.21 | 17.09 | 88.22 | 14.79 | 17.47 | 23.72 | 83.17 | 25.13 | 20.61 | 31.11 | 78.28 |
| MIR | 2.36 | 12.51 | 12.93 | 85.92 | 34.70 | 26.92 | 39.37 | 61.87 | 10.33 | 14.71 | 19.43 | 88.01 | 17.39 | 18.28 | 26.41 | 82.55 | 27.56 | 21.29 | 33.48 | 78.02 |
| SPL | 2.38 | 12.50 | 12.75 | 85.94 | 33.61 | 26.71 | 38.28 | 61.87 | 9.41 | 14.58 | 18.33 | 88.02 | 16.48 | 18.09 | 25.45 | 82.56 | 26.44 | 21.09 | 32.27 | 78.37 |
| Adaptive TS | 6.35 | 14.20 | 11.04 | 82.94 | 29.30 | 23.61 | 32.99 | 61.43 | 5.65 | 14.29 | 14.68 | 86.99 | 9.65 | 16.96 | 20.01 | 81.01 | 20.77 | 18.99 | 27.79 | 76.95 |
| TCP | 8.30 | 17.38 | 17.45 | 72.15 | 28.07 | 19.62 | 32.05 | 47.57 | 8.81 | 22.59 | 21.16 | 62.77 | 9.79 | 21.69 | 25.56 | 56.95 | 21.79 | 18.58 | 31.25 | 72.57 |
| ProCal | 2.89 | 12.60 | 13.33 | 86.08 | 38.82 | 30.99 | 42.35 | 61.75 | 11.84 | 16.29 | 19.13 | 86.22 | 19.85 | 19.63 | 28.10 | 82.30 | 25.22 | 22.61 | 31.58 | 75.15 |
| CE only (PTS) | 5.02 | 12.50 | 11.40 | 86.69 | 41.23 | 32.02 | 45.60 | 60.85 | 19.14 | 18.70 | 26.44 | 87.05 | 14.39 | 17.01 | 24.82 | 82.75 | 32.89 | 24.85 | 38.18 | 77.77 |
| CA only | 2.22 | 12.25 | 12.63 | 86.74 | 32.14 | 25.28 | 36.47 | 61.08 | 11.62 | 15.79 | 20.44 | 86.86 | 5.49 | 15.29 | 14.97 | 82.50 | 22.09 | 18.59 | 28.99 | 77.89 |
| CE+trans. | 3.42 | 12.87 | 11.77 | 85.36 | 28.06 | 22.01 | 32.38 | 63.47 | 6.22 | 13.28 | 15.30 | 88.64 | 11.52 | 15.94 | 21.08 | 83.84 | 20.89 | 18.28 | 27.69 | 78.50 |
| CA+trans. (ours) | 4.63 | 11.85 | 11.55 | 87.44 | 20.65 | 16.79 | 22.50 | 63.74 | 4.91 | 12.21 | 10.12 | 90.22 | 4.00 | 13.83 | 13.12 | 84.87 | 10.33 | 14.59 | 18.72 | 79.25 |
| CA + CE + trans | 4.24 | 13.15 | 11.53 | 84.91 | 26.54 | 20.85 | 30.80 | 63.85 | 5.63 | 12.86 | 14.57 | 88.96 | 9.50 | 15.15 | 19.52 | 84.31 | 21.42 | 18.79 | 28.02 | 78.14 |

### 3.5 Discussion

How could the CA loss improve the ECE metric? ECE bins confidence and calculates the difference between confidence and accuracy of samples in each bin. In the extreme case where bin size is infinitely small, each bin will contain only one sample (assuming no image duplicates), meaning accuracy of each bin is either 100% or 0%. In this scenario, the CA loss will push correct (wrong) predictions to the high (low) confidence, which always reduces ECE. When bin size gradually becomes larger, improvement brought by CA loss will be less definite but still visible.

Sample-adaptive temperature used in (Joy et al., ; Balanya et al., ; Wang et al., ) and our method has different properties from global temperature (Guo et al., ). Because global temperature does not change the order of samples ranked by their confidence, it cannot improve the ability of confidence to separate correct and wrong predictions. Sample-adaptive temperature at least has such potential (but under specific design). On the other hand, trained with the CE loss, the sample-adaptive temperature is demonstrated to produce a competitive calibrator for IND test sets (Joy et al., ; Balanya et al., ). But issues with CE and the lack of using additional information limit its effectiveness for OOD data. In comparison, our method is competitive on both IND and OOD test sets.

Why the CA loss sometimes still have empirical failures? A calibrator perfectly optimized by the CA loss will give 0 ECE, because all the correctly (wrongly) classified samples will have confidence of 1 (0). In practice, however, the bottleneck is to tell prediction correctness. We use augmented images but it might not be an optimal solution. In future we will explore new methods for correctness prediction.

**Table 2: Calibrator comparison under the CIFAR-10 setup. Each number is averaged over 10 classifiers (see Section 4.1). We use one IND test set (CIFAR10.1) and three OOD test sets (CIFAR-10.1 , CINIC, and CIFAR-10-C). Other notations are the same as Table 1.**
| Method | CIFAR-10.1 | Gaussian Blur | Defocus Blur | CINIC |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ECE$\downarrow$ | BS$\downarrow$ | KS$\downarrow$ | AUC$\uparrow$ | ECE$\downarrow$ | BS$\downarrow$ | KS$\downarrow$ | AUC$\uparrow$ | ECE$\downarrow$ | BS$\downarrow$ | KS$\downarrow$ | AUC$\uparrow$ | ECE$\downarrow$ | BS$\downarrow$ | KS$\downarrow$ | AUC$\uparrow$ |  |
| Uncal | 10.22 | 12.59 | 13.80 | 85.08 | 45.72 | 43.48 | 50.01 | 66.7 | 34.79 | 34.79 | 39.73 | 71.49 | 24.25 | 25.29 | 28.42 | 78.76 |
| TS | 4.84 | 11.06 | 11.78 | 85.15 | 35.25 | 34.54 | 42.81 | 66.78 | 24.99 | 28.17 | 33.93 | 71.40 | 16.04 | 20.77 | 24.17 | 79.15 |
| ETS | 2.77 | 10.78 | 10.87 | 85.08 | 30.04 | 30.84 | 39.41 | 66.69 | 19.98 | 25.69 | 39.73 | 71.23 | 11.48 | 19.13 | 22.15 | 79.25 |
| MIR | 2.29 | 10.83 | 11.01 | 84.98 | 30.39 | 30.83 | 39.73 | 66.67 | 20.31 | 25.63 | 31.51 | 71.41 | 12.58 | 19.59 | 22.86 | 78.68 |
| SPL | 3.04 | 10.89 | 10.61 | 85.04 | 29.12 | 30.57 | 38.49 | 66.72 | 19.64 | 25.53 | 30.46 | 71.48 | 11.88 | 19.51 | 22.03 | 78.72 |
| Adaptive TS | 4.17 | 11.29 | 11.74 | 83.40 | 22.07 | 26.58 | 34.44 | 65.97 | 12.15 | 23.31 | 27.27 | 70.27 | 10.73 | 19.48 | 21.99 | 77.76 |
| TCP | 11.40 | 13.83 | 12.78 | 76.09 | 12.38 | 24.64 | 30.48 | 54.83 | 6.71 | 24.66 | 24.35 | 58.25 | 15.35 | 23.62 | 17.47 | 73.16 |
| ProCal | 2.96 | 11.27 | 11.60 | 82.85 | 37.26 | 36.83 | 44.28 | 64.50 | 25.68 | 28.99 | 34.36 | 69.39 | 18.04 | 22.19 | 25.14 | 76.22 |
| CE Only (PTS) | 2.92 | 10.85 | 11.26 | 85.12 | 31.01 | 31.24 | 40.07 | 67.28 | 21.01 | 25.86 | 31.80 | 71.95 | 13.69 | 19.92 | 23.27 | 78.93 |
| CA Only | 2.42 | 10.87 | 11.06 | 84.82 | 29.41 | 30.19 | 39.05 | 67.11 | 19.45 | 25.17 | 30.99 | 72.04 | 12.65 | 19.68 | 22.93 | 78.55 |
| CE+trans. | 2.87 | 10.90 | 11.22 | 84.84 | 18.49 | 24.51 | 32.22 | 67.18 | 9.09 | 22.04 | 25.79 | 71.52 | 7.05 | 18.31 | 20.65 | 79.03 |
| CA+trans. (ours) | 2.76 | 10.79 | 10.72 | 85.15 | 11.45 | 22.18 | 27.91 | 67.33 | 4.65 | 21.30 | 22.42 | 71.57 | 7.27 | 18.34 | 15.90 | 78.91 |

Correctness prediction performance as a potential calibration metric. Given a confidence value, better separability between correct and wrong predictions leads to safer decision making for users, because less mistakes are made. This paper uses area under the ROC curve (AUROC) to measure the performance of predicting classification correctness. As shown in Section B.1 of Appendix, a small ECE does not always mean user makes less mistakes during decision making. Considering the strong tie between this 2-way classification problem and model calibration (Eq. (1)), we think AUROC can be an additional evaluation metric for model calibration. In Section B.1 of Appendix, more comparisons between AUROC and ECE are presented.

## 4 Experiments

### 4.1 Models and datasets

ImageNet-1k setup.
1. Models. We use 10 models trained or fine-tuned on the ImageNet-1k training set (Deng et al., ). We source these models from the model zoo Timm (Wightman, ).
2. Calibration sets. We use ImageNet-Val (Deng et al., ) to train calibrator.
3. Test sets.
(1) *ImageNet-A(dversarial)* (Hendrycks et al., ) comprises natural adversarial examples that are unmodified and occur in the real world.
(2) *ImageNet-S(ketch)* (Wang et al., ) contains images with a sketch-like style.
(3) *ImageNet-R(endition)* (Hendrycks et al., ) comprises of 30,000 images that exhibit diverse styles.
(4) *ObjectNet* (Barbu et al., ) is a real-world test set for object recognition where illumination, backgrounds and imaging viewpoints are very challenging.
(5) *ImageNet-Val*. We train the calibrator on half of the ImageNet validation set and test it on the remaining half.

CIFAR-10 setup.
1. Models.
We use 10 different models trained on the training split of CIFAR-10 (Krizhevsky et al., ) in this setup. We follow the practice in (Deng et al., ) to access the model weights.
2. Calibration set. Calibrators are trained on the test set of CIFAR-10.
3. Test sets
(1) CINIC-10 (Darlow et al., ) is a fusion of both CIFAR-10 and ImageNet-C (Hendrycks & Dietterich, ) image classification datasets. It contains the same 10 classes as CIFAR-10. (2) CIFAR-10-C(orruptions) contains subsets from CIFAR-10 modified by perturbations such as blur, pixelation, and compression artifacts at various severities.

iWildCam setup.
1. Model.
We use 10 models trained on the iWildCam(Beery et al., ) training set. They are downloaded from the official dataset website.
2. Calibration set. We train the calibrator on the iWildCam validation set.
3. Test set. We use the iWildCam test set containing animal pictures captured in the wild. Further details of the three setups are provided in Sec. A of Appendix.

### 4.2 Calibration methods and evaluation metrics

Methods. We compare our method with six popular calibration methods. They include scaling-based methods such as temperature scaling (TS) (Guo et al., ), ensemble temperature scaling (ETS) (Zhang et al., ), adaptive temperature scaling (Adaptive TS) (Joy et al., ), and parameterized temperature scaling (PTS) (Tomani et al., ). We also compare with a binning method multi-isotonic regression (MIR) (Zhang et al., ), True
Class Probability (TCP) (Corbiere et al., ), a spline-based re-calibration method (Spline) (Gupta et al., ), and Proximity-Informed Calibration (ProCal) (Xiong et al., ).

Metrics. Apart from the expected calibration error (ECE) (Guo et al., ), we report the Brier score (BS), adaptive calibration error (ACE) (Nixon et al., ), and Kolmogorov-Smirnov (KS) error (Gupta et al., ). In addition, we use area under the ROC curve (AUROC) to evaluate how well the calibrators separate correct predictions from wrong predictions, which, as mentioned in Sec. 3.5, might also be a good metric for calibration. All the numbers in Table 1, 2, and 3 are averaged over results of calibrating 10 different models introduced in Sec. 4.1.

### 4.3 Main observations

Comparison of calibration performance with the state of the art. We summarize calibration results under the ImageNet, CIFAR-10, and iWildCam setups in Table 1, Table 2, and Table 3, respectively. We have two observations. First, on OOD test sets, our method is very competitive across various metrics. For example, when compared with the second-best method on ObjectNet, the ECE, BS, and KS metrics of our method are 10.56%, 3.69%, and 8.97% lower, respectively. Second, on near IND or IND test sets such as ImageNet-Val and CIFAR-10.1 , our method is less advantageous but is still competitive. The reason for our method being more effective on OOD test sets is that there are more narrowly wrong predictions, mentioned in the last paragraph in Section 3.4. Besides, as explained in Section 3.3, the use of transformed images might not be an optimal way to inform classification correctness.

Comparing CA with MLE. In Table 1, Table 2, and Table 3, we compare ‘CA only’ with ‘CE only’, and ‘CA+trans’ with ‘CE+trans’. First, ‘CA only’ consistently outperforms ‘MSE only’ in 18 out of 20 scenarios under the ImageNet setup, 13 out of 16 scenarios under the CIFAR-10 setup, and 4 out of 4 scenarios under the iWildCam setup. Second, in most cases (*e.g*., 19 out of 20 scenarios under ImageNet setup), ‘CA+trans’ is better than ‘CE+trans’.
In addition, in Table 1, we observe that the combination of CE and CA does not yield better results compared to using CA alone in the OOD test set. The superiority of the CA loss is more evident on OOD datasets as discussed in Sec. 3.5.

Potential of CA in allowing confidence to better separate correct and wrong predictions. In Tables 1, 2, and 3, we compare separability and have two observations. First, existing methods typically do not have improvement in AUROC. This is not surprising, because their working mechanisms are not relevant to the separation of correct and wrong predictions. Second, our method improves AUROC under the ImageNet and iWildCam setups and in on par with existing methods on CIFAR-10. In fact, we find predictions of transformed images offer much less diversity under CIFAR-10 classifiers, losing their efficacy in telling prediction correctness. This could be addressed with a better method than transformed images, and we leave it for future work. These results, especially those under the challenging ImageNet and iWildCam setups, suggest our method has the potential to better distinguish between correct and wrong predictions by confidence scores, which could lead to improved decision making. A closer look at the ROC curves is provided in Fig. 5 and Sec. B.1 of Appendix.

**Table 3: Calibrator comparison under iWildCam setup. Each number is averaged over 10 classifiers (see Section 4.1). We use the iWildCam test set. Other notations are the same as Table 1.**
| Method | ECE$\downarrow$ | BS$\downarrow$ | KS$\downarrow$ | AUC $\uparrow$ |
| --- | --- | --- | --- | --- |
| Uncal | 16.04 | 17.77 | 22.11 | 86.59 |
| TS | 6.63 | 14.06 | 14.90 | 86.22 |
| ETS | 6.83 | 14.05 | 13.84 | 86.17 |
| MIR | 5.01 | 13.53 | 12.57 | 86.61 |
| SPL | 5.98 | 13.70 | 12.44 | 86.63 |
| ProCal | 7.09 | 14.68 | 16.39 | 84.70 |
| Adaptive TS | 7.17 | 14.25 | 12.88 | 86.33 |
| TCP | 13.30 | 13.57 | 13.73 | 90.51 |
| CE only (PTS) | 8.73 | 17.90 | 17.45 | 78.65 |
| CA only | 8.07 | 17.54 | 16.95 | 79.15 |
| CE+trans. | 6.78 | 12.88 | 12.07 | 88.70 |
| CA+trans. (ours) | 7.21 | 11.81 | 10.24 | 90.52 |

Figure: Figure 4: Impact of narrowly wrong and absolutely wrong predictions on calibrator performance. (Left:) we craft test sets containing 500 wrongly predicted samples with various degrees of being wrong. For example, the leftmost test set contains narrowly wrong samples, while the rightmost one contains absolutely wrong sample. Calibrator is trained on ImageNet-Val. (Right:) we craft training sets containing 1,000 wrong predictions and 1,000 correct predictions. The wrongly predicted samples also have different degrees of being wrong. We use ImageNet-A as test set. For both subfigures, we use ‘beit_base’ as the classifier and compare CA with CE and no calibration. Our method is more superior when training/test sets contain more narrowly wrong predictions.
Refer to caption: x4.png

Figure: Figure 5: (Left:) Comparing various combinations of image transformations, including rotation (R), grayscale (S), colorjitter (C), random erasing (E) and Gaussian noise (G). Different colors means different numbers of transformations. Dashed lines denote performance of no calibration and retrieval-based augmentation that accesses test batches. (Right:) Visualization of ROC curves of various calibrators. Existing methods typically do not improve AUC, while our method effectively does. All results in this figure are reported for ObjectNet using the model ‘beit_base_patch16_384’, as introduced in Appendix A.
Refer to caption: x5.png

Figure: Figure 6: Impact of $k$ in top-$k$ index selection (Sec. 3.3). We use ObjectNet test set. Under various $k$ our method is better (lower KS and higher AUROC) than uncalibrated models and TS. We choose $k\!=\!4$ as trade-off between performance and computational cost.
Refer to caption: x6.png

Effectiveness of using transformed images as calibrator input. We compare
‘CA+trans’ and ‘CA only’ in Table 1, Table 2, and Table 3. It is very clearly that ‘CA+trans’ gives consistently better calibration performance than ‘CA only’. It indicates the necessity of using transformed images.

### 4.4 Further analysis

Impact of narrowly wrong predictions in training and testing.
We construct various calibration (training) sets and test sets with samples of controlled degrees of being wrongly predicted. From Fig. 4 (left), if a test set is dominantly filled with narrowly wrong predictions, our method will have a huge improvement over CE and no calibration: in fact, CE has the same performance as no calibration in this scenario. As more absolutely wrong samples are included, the gap between smaller, but our method is still superior. This is because the calibration set also has various degrees of wrong predictions, so CE is not as well trained as CA and actually has similar performance as no calibration.

On the other hand, from Fig. 4 (right), when training set contains lots of narrowly wrong predictions, CE is very poor and even worse then no calibration. When more absolutely wrong samples are included, CE becomes gradually better and even close to our method. These results empirically verifies our analysis in Section 3.4.

Comparing different image transformations. We try different combinations of image transformations (including retrieval-based augmentation in test batches) as calibrator input. Results are summarized in Fig. 5. We observe that using rotation, gray-scaling, and color-jittering generally give good calibration results. Retrieval-based augmentation is also competitive, but it requires access to test batches which might not be practical. Moreover, we find that using only one transformation is not ideal. While using more transformations is effective, three is a good number to balance between calibration performance and computational cost.

Impact of $k$. We use the indices of top-$k$ confidences to locate and select the $k$-dim Softmax vectors from the transformed images. In Fig. 6, we find that for various values of $k$ our method improves over uncalibrated models. Moreover, $k\!>\!5$ does not bring much improvement. Considering computational cost, we use $k\!=\!4$. Note $k$ is chosen on the ImageNet-A test set and applied on all the other test sets.

Computational cost. On a server with 1 GeForce RTX 3090 GPU, it takes our method 583 seconds to train a calibrator on ImageNet-Val;
in comparison, it take PTS and temperature scaling 987 seconds and 32 seconds respectively in training. Because temperature scaling only learns a single parameter (*i.e*., temperature), it is the quickest to train. The inference time for ours, temperature scaling, and PTS is similar: 2.33, 1.63, and 2.8 milliseconds per image, respectively. The time complexity of our method is the same as that of PTS.

## 5 Conclusion

This paper starts from the general goal of calibration, mathematically interprets it, and derives a concrete loss function for calibration. Name as correctness-aware (CA) loss, in training it requires correct (wrong) predictions to have high (low) confidence, where such correctness is informed by transformed versions of original images. During inference, our calibrator also takes transformed images as input and tends to give high (low) confidence to likely correctly (wrongly) predicted images. We show our method is very competitive compared with the state of the art and potentially benefits decision making with plausible results on better separability of correct and wrong predictions. Moreover, we reveal the limitations of the CE and MSE losses for certain type of samples in the calibration set. Rich insights are given w.r.t how our method deals with such samples. In future we will study more effective correctness prediction methods to improve our system and how our method can be used for training large vision language models.

## Impact statement

This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

## Appendix A Experimental Setup

In this section, we introduce the benchmarking datasets and classification models used in our paper.

### A.1 ImageNet Setup

Models.
We employ the ImageNet models from the PyTorch Image Models (timm) library (Wightman, ), which offers models trained or fine-tuned on the ImageNet-1k training set (Deng et al., ). The models utilized in our paper are listed below:

{
‘beit_base_patch16_384’, ‘tv_resnet152’, ‘tv_resnet50’, ‘tv_resnet101’, ‘densenet121’, ‘inception_v4’, ‘densenet201’, ‘vit_base_patch16_384’, ‘deit_base_patch16_224’, ‘inception_v3’
}

Datasets.
We present the test sets employed in the main paper to evaluate the aforementioned ImageNet models. Datasets mentioned below can be accessed publicly via the provided links.

*ImageNet-A(dversarial)* (Hendrycks et al., ):
https://github.com/hendrycks/natural-adv-examples.
*ImageNet-S(ketch)* (Wang et al., ):
https://github.com/HaohanWang/ImageNet-Sketch.
*ImageNet-R(endition)* (Hendrycks et al., ):
https://github.com/hendrycks/imagenet-r.
*ImageNet-Blur* (Hendrycks & Dietterich, ):
https://github.com/hendrycks/robustness.
*ObjectNet* (Barbu et al., ):
https://objectnet.dev/download.html.

### A.2 CIFAR- 10 10 10 10 Setup

Models.
We employ the CIFAR-$10$ models from the
open source library (https://github.com/kuangliu/pytorch-cifar)
which offers models trained or fine-tuned on the CIFAR-$10$ training set (Krizhevsky et al., ). The models utilized in our paper are listed below:

{
‘VGG19’, ‘DenseNet121’, ‘DenseNet201’, ‘ResNet18’, ‘ResNet50’, ‘ShuffleNetV2’, ‘MobileNet’, ‘PreActResNet101’, ‘RegNetX_200MF’, ‘ResNeXt29_2x64d’
}

Datasets. Datasets used in the CIFAR10 setup can be found through the following links.
*CINIC* (Darlow et al., ):
https://github.com/BayesWatch/cinic-10.
*CIFAR10-C* (Hendrycks & Dietterich, )(https://github.com/hendrycks/robustness);

### A.3 iWildCam Setup

Models.
We employ the iWildCam models from the
open source library (https://worksheets.codalab.org/worksheets/
0x52cea64d1d3f4fa89de326b4e31aa50a)
which offers models trained or fine-tuned on the iWildCam training set (Beery et al., ). The models utilized in our paper are listed below:

{
‘iwildcam_erm_seed1’, ‘iwildcam_deepCORAL_seed0’, ‘iwildcam_groupDRO_seed0’, ‘iwildcam_irm_seed0’,‘iwildcam_erm_tune0’, ‘iwildcam_ermaugment_tune0’, ‘iwildcam_ermoracle_extraunlabeled_tune0’, ‘iwildcam_swav30_ermaugment_seed0’, ‘iwildcam_dann_coarse_extraunlabeled_tune0’, ‘iwildcam_afn_extraunlabeled_tune0’
}

Dataset.
*iWildCam-OOD* (Beery et al., ) can be download from the the official guidence: https://github.com/p-lambda/wilds/.

## Appendix B Additional Visualisations

### B.1 AUROC curves

Fig. 7 shows the comparison of Receiver Operating Characteristic (ROC) curves across different calibration methods.

Figure: Figure 7: Comparison of Receiver Operating Characteristic (ROC) Curves Across Different Calibration Methods. Each figure’s title specifies the classifier and the test set used. It is evident that our methods (green curves) yield a higher area under ROC curve (AUROC) compared to other calibration methods, signifying an enhanced ability of our model to distinguish between correct and incorrect predictions based on calibrated confidence.
Refer to caption: x7.png

### B.2 Distributions of predictions

Visualizations of the distributions for correct and incorrect predictions on four datasets are given in Fig. LABEL:fig:vis.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Why does the paper initially emphasize using a continuous calibration error instead of the Expected Calibration Error (ECE)?

What is the intended synergy between the CA loss and the transformation component, given their distinct purposes of reducing ECE and enhancing cross-domain robustness?

Could the proposed algorithm’s effectiveness be validated further by testing it on alternative baselines?
soundness: 3
strengths: This paper presents a range of validation scenarios to assess the effectiveness of the proposed framework. In numerous cases, the framework achieves state-of-the-art performance, validating the impact of its two novel schemes. The experimental setup and comparisons are thoughtfully designed, with detailed descriptions that enhance clarity and reproducibility. Mathematical derivations are presented comprehensively, and the overall narrative is organized in a way that makes the framework easy to follow and understand, emphasizing key components effectively.
summary: The paper introduces two innovative methods to address calibration errors in deep learning predictions: a correctness-aware loss function and a sample transformation technique. The correctness-aware loss function aims to directly minimize calibration error, effectively improving the calibration of misclassified samples by narrowing discrepancies across all classes. Additionally, to boost cross-domain performance, an augmentation-based transformation is applied to calibration samples, enhancing robustness across varied domains. Both methods are implemented in a post-hoc calibration framework, and the proposed algorithm demonstrates state-of-the-art performance, particularly in cross-domain settings.
weaknesses: The paper has several strengths, yet I have some specific concerns that warrant attention:

1. Definition of "Narrow Misclassification":
   The term "narrow misclassification" appears in the abstract, and the correctness-aware (CA) loss is presented as targeting this condition by adjusting predictions across different classes rather than solely reducing confidence in the incorrect class. However, a clear definition of "narrow misclassification" is missing, and it’s challenging to discern how it differs from absolutely wrong samples even after reviewing the derivations. Clear definitions and empirical analysis based on outcomes would help clarify this distinction.

2. Limitations from Augmentation Types Used:
   The transformation component uses augmentations, but it lacks an analysis of how different types of augmentation affect performance across domains. Depending on the augmentation type, the efficacy in cross-domain scenarios may vary. Experimental validation or analysis is needed to determine the diversity of augmentation types required or which specific augmentations are essential.

3. Similarity with Temperature Scaling:
   If the framework were designed with temperature scaling, where the temperature parameter is shared across all classes, it could similarly distribute confidence across classes rather than reducing only the incorrect class's confidence. This raises questions about the uniqueness of the proposed algorithm’s approach in addressing "narrow misclassification."

4. Derivation for the CA Loss Function:
   The derivation of the CA loss function appears to be unnecessarily complex. Initially, the paper emphasizes the use of continuous calibration error rather than Expected Calibration Error (ECE), suggesting a different approach. However, the final derivation seems equivalent to ECE-based loss, assuming discrete samples and small sample sizes, which undermines the rationale for a continuous assumption. Clarification is needed on why continuous assumptions were initially made if the final derivation closely resembles an ECE-based approach.

5. Bounds of the CA Loss:
   Bounds for the CA loss are derived based on assumptions that the sample sizes and accuracy across classes are similar. However, the significance of these bounds remains unclear, as they appear merely descriptive of the assumed conditions. Additional insights or generalized bounds demonstrating reduced CE loss could improve understanding.

6. Unclear Derivation in Equation 15:
   The derivation in Equation 15 is ambiguous due to an unexplained arrow, which might imply a limit. Clarification on which parameter converges to produce this outcome is necessary to improve the transparency of this mathematical derivation.

7. Parameter \theta in Equation 19:
   It is unclear if \theta in Equation 19 exclusively refers to the fully connected layers added for post-hoc calibration. This specification is important for clarity.

8. Synergy between CA Loss and Transformation Component:
   The CA loss reduces ECE, while the transformation improves cross-domain robustness. However, the synergy between these components is unclear, as seen in experimental results: applying CA loss significantly reduces ECE, while the transformation tends to increase ECE, showing a trade-off rather than synergy. Clarification is needed on why these mechanisms must be combined rather than sequentially applied as separate approaches.

9. Baseline (CE Only + PTS) Already Achieving State-of-the-Art Performance:
   In the result tables, the baseline (CE Only + PTS) already achieves state-of-the-art ECE and accuracy in multiple scenarios. While adding CA and transformation components improves performance further, it seems that these improvements are achieved largely because of the baseline's strong performance. To mitigate this concern, I recommend testing the proposed algorithm on alternative baselines.

10. Minor Points:
    - The text in figures is too small, making them hard to read.
    - Typo: Line 136, “samples'.” should be “samples.'” 

These concerns, if addressed, could enhance the clarity and impact of the proposed framework.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - What is the difference between CA only and CA trans.? Is CA only the calibration strategy that estimates the calibrator $g$ from the calibration set using the loss of Eq. (7) and no data augmentation? This is not clear.

- The approach focuses on calibration of the maximum confidence: can the strategy be adapted to calibrate the whole confidence vector (multiclass calibration)?
soundness: 2
strengths: - The idea of using test-time augmentation to predict a sample based temperature scaling factor and learning a network for predicting such temperature is novel, as far as I know.

- The justification of the loss on a toy example pointing out its behavior on so-called narrowly wrong samples is intuitive.

- Rather extensive experiments on several types of image datasets show the benefit of the approach over standard calibration methods and other optimization losses.
summary: The paper describes a method for post-hoc calibration of a classifier based on estimating  for each sample a scaling temperature on the output logits(sample-adaptive calibration strategy). Test time data augmentation is used to predict the scaling temperature and relies on a complementary network taking as input the softmax of selected transformed images and minimizes what is called a correctness-aware loss. The loss is justified by a better management of narrowly wrong predictions. The strategy is evaluated on several small to mid-size datasets and 10 networks per dataset.
weaknesses: - The goal of the formal development (Section 3.2) is not clear: what is it supposed to show? Is it to prove that the empirical criterion (7) is a good proxy for optimizing (3), given that $\hat{c}$ is produced by the calibration pipeline of Figure 2? If so, I am not convinced that the formal developments of Section 3.2 actually prove this.

- The writing lacks precision (see my first question, same symbol $E_f^{emp}$ but different concepts for instance). 

- The data augmentation is justified by the fact "that consistency in model predictions for transformed images correlates strongly with accuracy" (l. 261): if I can agree with this law, I don't clearly see where it applies in your framework. Or is it that by introducing some local perturbation of the data through transformations and measuring the variation in confidence scores, one can infer accuracy?  Then why not directly predict the confidence instead of a temperature?

- In general, I have difficulty understanding the conceptual connections between the test time data augmentation, the formal development, and the narrowly wrong sample analysis. The global logic of the writing is hard to follow.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. See above weakness

2. What is the core difference between calibration and misclassification (e.g. [R1]), both of them seem to be focusing on the incorrect predictions.

3. Fig. 6  illustrates the impact of ablating the top-k selection on the CA loss. The figure suggests that increasing k beyond 4 leads to a significant decline in performance. This trend raises questions about the potential effects of even higher values of k, such as 100 or 200, particularly in datasets like ImageNet. Additionally, since the authors have chosen k=4 as the default setting, it is important to consider how the model manages scenarios where the correct prediction is not included among the top-4 predictions.

4. The method involves training a calibrator with a new loss function and using transformed images, which could be more complex to implement compared to simpler calibration techniques.

 [R1]  Zhu, Fei, et al. "Openmix: Exploring outlier samples for misclassification detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
soundness: 3
strengths: **Novel Calibration Objective:** The paper introduces a new loss function, CA loss, which is a significant contribution to the field of model calibration. This loss function is intuitively designed to align with the goal of calibration, which is to ensure high confidence for correct predictions and low confidence for incorrect ones.

**Empirical Evidence:** The authors provide extensive experimental results demonstrating the effectiveness of their proposed method across various datasets, including IND and OOD test sets. The consistent performance improvement over uncalibrated models and other calibration techniques is a strong point.

**Theoretical Insights:** The paper not only proposes a new method but also provides theoretical insights into why existing methods like CE and MSE losses are limited, particularly for certain types of samples in the calibration set.
summary: The paper addresses the issue of model calibration in machine learning, specifically aiming to align a model's confidence with its prediction correctness. The authors identify limitations with the commonly used Cross-Entropy loss for calibrator training and propose a new post-hoc calibration objective, the Correctness-Aware loss. This objective function is designed to decrease model confidence on wrongly predicted samples and increase it on correctly predicted ones. The method utilizes transformed versions of samples to train the calibrator and is tested on both IND and OOD datasets. The paper claims that their method achieves competitive calibration performance compared to state-of-the-art techniques and provides a better separation of correct and incorrect test samples based on calibrated confidence.
weaknesses: **Dependency on Transformations:** The effectiveness of the CA loss relies on the use of transformed images to infer correctness. If these transformations do not adequately capture the characteristics of correct and incorrect predictions, the calibration might be less effective.

**Transfomations lack of theoretics:** While the use of transformations such as rotation, grayscale, color jittering, and others has proven to be effective in practice; however, the choice of transformations and their number in Fig. 4 are currently guided more by empirical results rather than a theoretical framework that explains why these five transformations should correlate with prediction correctness as so many transformation exists. And the paper also does not provide a theoretical basis for which transformations are the most informative for calibration or how to select the optimal set of transformations. The current approach might be seen as somewhat arbitrary, and the effectiveness could be dependent on the specific characteristics of the dataset and the model architecture. And There is a risk that the calibrator might overfit to the specific transformations used during training, which may not generalize well to real-world variations in data that were not captured by the training transformations

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: **1**)  What are the key differences with MDCA loss [C] and DCA loss [G] ? I would like to see concrete differences between them.

**2**)  Can MDCA loss and/or DCA loss be used in place of correctness-aware loss to obtain optimal temperature value? Beyond, CE and MSE losses, I believe it would be an interesting comparison between the effectiveness of proposed CA loss and these losses

**3**)  Is the post-hoc calibrator capable of calibrating non-ground truth classes as well?

**4**) What is the performance of the method under the SCE metric [H] compared to other post-hoc calibration methods? 

**5**) The intuition behind learning a mapping through g network from top-K softmax scores (corresponding to transformed versions) to temperature value is not very clear. 

**6**) L499: The paper mentions that existing methods do not improve AuC compared to proposed one. Will require more explanation.

**7**) How good is the method in overcoming under confidence of the model?

**8**) Can this post-hoc calibrator be used after a train-time calibration method? It would be interesting to observe the complementary strengths of the proposed post-hoc calibration method.
soundness: 3
strengths: **1**) Calibrating deep neural networks is an important step towards making AI models reliable and trustworthy, especially in safety-critical applications.

**2**) The proposed post-hoc calibrator is simple as it also learns to identify a per-sample temperature value that can be used to scale the logits.

**3**) The paper also mentions some theoretical insights into the proposed correctness-aware loss term by comparing and contrasting it with CE and MSE losses.

**4**) Results show that proposed idea is competitive against other post-hoc calibration methods.
summary: The paper undertakes the problem of calibrating deep neural networks for the task of classification. At the core of the method is a past-hoc calibrator which proposes a correctness-aware loss to search for the optimal temperature which is then used to scale the logits for a given sample. To determine the correctness of a sample, the method uses the well-known concept of consistency across different augmentations. A simple network is used to map top-K softmax predictions across augmentations to the temperature value. The correctness-aware loss optimizes this network to obtain the best temperature. The paper also shows mathematical insights on the proposed loss. The experiments have been conducted on different datasets to validate the effectiveness of the post-hoc calibrator. Results claim to achieve competitive performance against other post-hoc calibration methods, such as naive temperature scaling, ensemble temperature scaling, adaptive temperature scaling and isotonic regression.
weaknesses: **1**) The related work section completely misses an emerging direction of train-time calibration methods such as [A], [B], [C],  [D],  [E] and [F]. 

**2**) The paper lacks reliability diagrams to better understand the potential of proposed post-hoc calibrator in overcoming overconfidence and under confidence over the full spectrum of model confidence.

**3**) Why the proposed post-hoc calibrator is able to improve OOD calibration performance? There is no analyses that supports these results.

**4**) How the proposed post-hoc calibrator would perform under class-imbalanced scenarios?

**5**) The proposed correctness-aware loss appears similar to MDCA loss [C]. What are the key differences?


[A] Liu, B., Ben Ayed, I., Galdran, A. and Dolz, J., 2022. The devil is in the margin: Margin-based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 80-88).

[B] Patra, R., Hebbalaguppe, R., Dash, T., Shroff, G. and Vig, L., 2023. Calibrating deep neural networks using explicit regularisation and dynamic data pruning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1541-1549)

[C] Hebbalaguppe, R., Prakash, J., Madan, N. and Arora, C., 2022. A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16081-16090).

[D] Wei, H., Xie, R., Cheng, H., Feng, L., An, B. and Li, Y., 2022, June. Mitigating neural network overconfidence with logit normalization. In International conference on machine learning (pp. 23631-23644). PMLR.

[E] Liu, B., Rony, J., Galdran, A., Dolz, J. and Ben Ayed, I., 2023. Class adaptive network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16070-16079).

[F] Park, H., Noh, J., Oh, Y., Baek, D. and Ham, B., 2023. Acls: Adaptive and conditional label smoothing for network calibration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3936-3945).

[G] Liang, G., Zhang, Y., Wang, X. and Jacobs, N., Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classification BMVC 2020.

[H] Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in
deep learning. In CVPR Workshops, volume 2, 201

Meta-review: This paper received mixed reviews. The reviewers recognized the novel calibration loss and its sound motivation, extensive experiments, and competitive performance the paper achieved. They also raised various concerns, some of which are crucial; to name a few: weak theoretical justification (bjTY), presentation issues (e.g., notations and terms not defined properly, unnecessarily complex derivation of the loss, issues on global logic) (pvJo, bjTY), the use of input transformations (e.g., dependency on a set of transformations, lack of justification and theoretical foundation) (pvJo, bjTY, iPzo), potential trade-off between the two proposed components (pvJo), the use of an overly strong baseline (pvJo), and marginal difference from previous work (LQKt). 

The authors' rebuttal and subsequent responses in the discussion period address some of these concerns but failed to fully assuage all of them: two reviewers (pvJo, bjTY) still pointed out the weak theoretical justification and presentation issues. As a result, the two reviewers voted down after the discussion period. The AC found that these remaining concerns are not trivial and should be addressed appropriately before publication. In particular, Reviewer bjTY noted that the theoretical justification is weak since the derivation of the loss relies on an assumption that does not hold in general; this is a critical limitation since the derivation is central to the theoretical arguments made by the authors and, according to them, one of the main contributions of the paper. Moreover, the AC found that many critical concerns of the reviewers have not been well addressed in the rebuttal and revision. For example, although a couple of reviewers pointed out presentation issues ranging from unclear definitions of notations and terms to weak global logic, the manuscript has not been revised to address these concerns. Also, although some concerns like impact of input transformations and trade-off between the CA loss and the transformations can be addressed by simple and straightforward experiments, the authors did not directly address the concern, but reiterated their arguments using some of the reported results. 

Putting these together, the AC considers that the remaining concerns outweigh the positive comments and the rebuttal, and thus regrets to recommend rejection. The authors are encouraged to revise the paper with the comments by the reviewers, and submit to an upcoming conference.

Decision: Reject