A Data-Driven Measure of Relative Uncertainty for Misclassification Detection
Authors: Eduardo Dadalto Câmara Gomes
Marco Romanelli
Georg Pichler
Pablo Piantanida
Venue: ICLR 2024 poster (2024)
Keywords: Misclassification detection
Uncertainty estimation
Trustworthy AI
Safety
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We introduce a data-driven measure of relative uncertainty as a new method for detecting samples misclassified by machine learning classification models.

Abstract: Misclassification detection is an important problem in machine learning, as it allows for the identification of instances where the model's predictions are unreliable. However, conventional uncertainty measures such as Shannon entropy do not provide an effective way to infer the real uncertainty associated with the model's predictions. In this paper, we introduce a novel data-driven measure of uncertainty relative to an observer for misclassification detection. By learning patterns in the distribution of soft-predictions, our uncertainty measure can identify misclassified samples based on the predicted class probabilities.  Interestingly, according to the proposed measure, soft-predictions corresponding to misclassified instances can carry a large amount of uncertainty, even though they may have low Shannon entropy. We demonstrate empirical improvements over multiple image classification tasks, outperforming state-of-the-art misclassification detection methods.

Published as a conference paper at ICLR 2024
A DATA-DRIVEN MEASURE OF RELATIVE UNCERTAINTY FOR MISCLASSIFICATION DETECTION
Eduardo Dadalto'
Marco Romanelli New York University New York, NY, USA mr 68652@nyu..ed edu
Laboratoire des signaux et systèmes (L2S) Université Paris-Saclay CNRS CentraleSupélec
Gif-sur-Yvent France
eduardo. dewalat@eeetaaaeeeueeeeeeeoe
Georg Pichler*
Pablo Piantanida
Institute of Telecommunications
International Laboratory on Learning Systems (ILLS)
TU Wien
Quebec Al Institute (MILAA)
1040 Vienna, Austria
CNRS CentraleSupéllc- Université Paris-Saclay Montreal, Canada
georg. peorggpichleeriieee.cco org
ABSTRACT
Misclassification detection is an important problem machine learning, as it allows for the identification of instances where the model's predictions are unreli- able. However, conventional uncertainty measures such as Shannon entropy do not provide effective way to infer the real uncertainty associated with the model's 's predictions. In this paper, we introduce novel data-driven measure of uncertainty relative an observer for misclassification detection. By learning patterns in the distribution of soft- soft-predictions, our uncertainty measure can identify misclassified samples based on the predicted class probabilities. Interestingly, according to the proposed measure, sefeprdddicions corresponding misclassified instances can carry a large amount of uncertainty, even though they may have low Shannon en- tropy. We demonstrate empirical improvements over multiple image classification tasks, outperforming state-of-the-as misclassification detection methods.
INTRODUCTION
Critical applications, such as autonomous driving and automatic tumor segmentation, have benefited greatly from machine learning algorithms. This motivates the importance of understanding their limitations and urges the need for methods that can detect patterns on which the model uncertainty may lead to dangerous consequences (Amodei et al.] 2016).. In recent years, considerable efforts have been dedicated to uncovering methods that can deceive deep learning models, causing them to make classification mistakes. While these findings have highlighted the vulnerabilities of deep learning models, it is important to acknowledge that erroneous classifications can also oceur naturally. The likelihood of such incorrect classifications is strongly influenced by the characteristics of the data being analyzed and the specific model being used. Even small changes in the distribution the training and evaluation samples can significantly impact the occurrence of these misclassifications,
recent thread of research has shown that issues related to misclassifications might be addressed by augmenting the training data for better representation (Zhu al. [023al Zhang al. 2017] Pinto et al. 2022). However, in order to build misclassification detectors, all these approaches rely on statistics derived from the soft- poft-prediction output by the model, such the entropy related notions, interpreting it as an expression of the model's confidence. We argue that relying on the assumption that the model's output distribution is good representation of the uncertainty of the model is inadequate. For example, a model may be very confident on sample that is far from the training distribution and, therefore, is likely to be misclassifidd, which undermines the effective use of the Shannon entropy as measure of the real uncertainty associated with the model's prediction.
"qqual contribution.

---

Published as a conference paper at ICLR 2024
In this work, we propose data-driven measure of relative uncertainty inspired by Rao (9822 that relies on negative and positive instances to capture meaningful patterns in the distribution of soed-prrdictions. For example, positive instances can be correctly classified samples for which the uncertainty is expected to be low while negative instances (misclassified samples) are expected to have high uncertainty. Thus, the goal is to yield high and low uncertainty values for negative and positive instances, respectively. Our measure is "relative", as it is not characterized axiomatically, but only serves the purpose of measuring uncertainty of positive instances relative to negative ones from the point of view of a subjective observer d. We employ relative uncertainty to measure the overall uncertainty of a model, encompassing both aleatoric and epistemic uncertainty components. By learning to minimize the uncertainty in positive instances and to maximize it in negative instances, our metric can effectively capture meaningful information to differentiate between the underlying structure of distributions corresponding to two categories of data. Interestingly, this notion can be expanded to any binary detection tasks in which both positive and negative samples are available.
Our contributions are three-fold: 1) We leverage a novel statistical framework for categorical distributions to devise a learnable measure of relative uncertainty (REL-U) for a model's predictions, which induces large uncertainty for negative instances, even if they may lead to low Shannon entropy (cf. Section 33: 2) We propose a closed- foosed-form solution for training REL-U in the presence of positive and negative instances (ef. Section 4 3) We report favorable and consistent results over different models and datasets, considering both natural misclassifications within the same statistical population, and in case of distribution shift, or mismatch, between training and testing distributions (cf.. Section 5)
2 RELATED WORKS
Misclassification detection aims to evaluate the reliability of decisions made by classifiers and determine whether they can be trusted or not. A simple baseline relies on the maximum predicted probability (Hendryeks & Gimpel 2017). but state-of-the-arr classifiers have shown to be overconfi- dent in their predictions, even when they fail (Cobb & Looveren, 2022). Liang et al. (2017) proposes applying temperature scaling (Guo et al. (201]) and perturbing the input samples to the direction of the decision boundary to detect miselassifications better. A line of research trains auxiliary parameters to estimate a detection score (Corbière et al.) (019) directly, following the idea of learning to reject (Chow) 1970; Geifman & Yl-ian 2017). Exposing the model to outliers severe augmentations during training has been explored in previous work (Zhu et al. 2023a to evaluate if these heuristics are beneficial for this particular task apart from improving robustness to outliers. Granese al.) (2021) proposes a mathematical framework and a simple detection method based on the estimated probability of error. We show that their proposed detection metric is a special case of ours. Zhu et al.. (2023)) study the phenomenon that calibration methods are most often useless or harmful for failure prediction and provide insights into why. [en et al.] (2023) discusses how training settings such as pre-training or outlier exposure impact misclassification and open-set recognition performance. Related sub-fields are -ittrf-distribution detection (Lee et al. 2018), open set recognition (Geng et al. 202I). novelty detection (Pimentel et al.) 2014), anomaly detection (Chalapathy & Chawla 2019). ). adversarial attacks detection (Akhtar & Mian (218)), and predictive uncertainty estimation
via Bayesian Neural Networks estimation (Gal & Ghahramani 2016 Lakshminarayanan et al. 2016 Mukhoti et al.
2021 Finbinder et al.) 2022
Snoek et 2019). We refer the reader to Vadera et al.
(2020) as a survey in the topic.
A different take on the problem of uncertainty in AI is conformal learning Angelopoulos et al. T2 Romano et al. (2020): in addition to estimating the most likely outcome, a conformal predictor provides a "prediction set" that provably contains the ground truth with high probability. Recently, considerable effort has been invested into quantifying uncertainty by disentangling and estimating two quantities: epistemic uncertainty, i.c. the uncertainty that can be decreased by adding new observation to the training set available to a model, and aleatoric uncertainty, which is fundamentally present in the data and cannot be reduced by adding training data. In general, these works rely on inducing higher sensitivity at the level of the network's 's internal representation or on modifications to the training procedure or auxiliary models Liu et al. (2020) proposes to embed distance-ard awareness ability in a given neural network by adding spectral normalization to the weights during training as to translate the distance in the data manifold into dissimilarity the hidden representation level. Van Amersfoort et al. (2020) proposes a new loss function and centroid updating scheme to speed up the computation of RBF-based nets Kotelevskii et al. (2022) proposes approach
2

---

Published as a conference paper at ICLR 2024
that provides disentanglement of epistemic and aleatoric uncertainty by computing the agreement between a given model and the Bayes classifier based on their kernel estimate and measuring the point-wise Bayes risk. Finally, Mukhoti et al. (2023) utilizes spectral normalization during training and feature-space density estimator after training to quantify the epistemic uncertainty disentangling it from the aleatoric one.
3 A DATA-DRIVEN MEASURE OF UNCERTAINTY
D Matrix
Class-Wise Confusion Matrix with True Labels
Doctor Matrix
sirlers
6
1
D
automobile
0.4
kird
c3E
Tue Libe
deer
20g
0.2
horse
4
1
<tip
0.0
sipuase automatdee
And
cat
dieer
deg
fros
froee
vhis
mork
aimitere
auteolale
deg
ship
trick
Arpiaate
natemoble
Bird
cat
dlerr
Frog
force
Predicted Label
Figure 1: Intuitive example illustrating the advantage of REL-U compared entropy- basedd methods:
REL-UU (left-end side heatmap) captures the real uncertainty (central heatmap) much better than Doctor (Granese al. 202T;; detailed analysis is provided in Section 5.3
Before introduce our method, start by stating basic definitions and notations. Then, describe our statistical model and some useful properties of the underlying detection problem.
Let X Rd be a (possibly continuous) feature space and let lety= = { .., C} denote the label space related to some task of interest. Moreover, we denote by Pxy be the underlying joint probability distribution on X' x V. We assume that a machine learning model is trained on some training data, which ultimately yields a model that, given samples E X, outputs a probability mass function (pmf) on y, which we denote as vector p(x). This may result from a soft-max output layer, for example. A predictor f: X + ) is then constructed, which yields fx) arg maxyey p(x)). We note that we may also interpret (xx the probability distribution of Ý on V.i. given xx distributed according p. ())x) p(x)y.
In statistics and information theory, many measures of uncertainty were introduced, and some were utilized in machine learning to great effect. Among these are Shannon entropy (Shannon 1948 Sec. 6). Rényi entropy (Rényi 1961). q-entropy (Tsallis 1988), as well as several divergence measures, capturing notion of distance between probability distributions, such as Kullback-Leibler divergence (Kullback & Leibler 195T), f diivergeeece (Csiszár) 1964), and Rényi divergence (Rényi) 1961). These definitions are well motivated, axiomatically and/or by their use in coding theorems. While some measures of uncertainty offer flcxibility by choosing parameters, c.g., a for Rényi o-entropy, they invariant w.r.t. relabeling of the underlying label space. In case, however, this semantic meaning of specific labels can be important and we do not expect a useful measure of "relative" uncertainty to satisfy this invariance property.
Recall that the quantity p(x) is the soft-prediction output by the model given the input x. The entropy measure of Shannon (Shannon 1948 Sec. 6)
H( Σ f(x), log (f(x)y)
vEY
and the concentration measure of Gini (Gini 1912)
Sgini (x) e 1
(p(x),)²
vEy
have commonly been used to measure the dispersion of a categorical random variable Ý given a sample x. It worth emphasize that either measure may be used carry out analysis of
3

---

Published as a conference paper at ICLR 2024
dispersion for random variable predicting discrete value (e.g., label). This is comparable to the analysis of variance for the prediction of continuous random values.
Regrettably, these measures suffer from two major inconveniences: they are invariant to relabeling of the underlying label space, and, more importantly they lead to very low values for overconfident predictions, even if they are wrong. These observations make both Shannon entropy and the Gini coefficient unfit for our purpose, i.e., the detection of misclassification instances. Evidently, we need a novel measure of uncertainty that can operate on probability distributions p(x) and that allows us to identify meaningful patterns in the distribution from which uncertainty can be inferred from data. To overcome the aforementioned difficulties, we propose construct a class of uncertainty measures that is inspired by the measure of diversity investigated in Rao (1982), defined as
sa(x) Bid(Y, X'IX x = Σ Σ VEY V'EY
d(y, ((x))'))))))())))')
(3)
where d E is a class of distance measures and, given X = x, the random variables P, 8 p(x) independently and identically distributed according to p(x). The statistical framework introducing here offers great flexibility by allowing for an arbitrary function d that can be learned from data, as opposed to fixing predetermined distance in Rao (1982). In essence, the uncertainty in equation as relative to a given observer d, which appears as a parameter in the definition. To the best of our knowledge, this is fundamentally novel concept of uncertainty.
regard
4 FROM UNCERTAINTY TO MISCLASSIFFIATTIN DETECTION
We wish to perform misclassification detection based on the statistical properties of soft soft-predictions of machine learning systems. In essence, the resulting problem requires binary hypothesis test, which, given probability distribution over the class labels (the soft-prediction), decides whether misclassification event likely occurred. We follow the intuition that by examining the ooft-prediction of categories corresponding to given sample, the patterns present in this distribution provide meaningful information to detect misclassified samples. For example, if a sample is misclassified, this can cause significant shift in the soft- noft-prediction even if the classifier is still overconfident. From a broad conceptual standpoint, examining the structure of the population of predicted distributions very different from the Shannon entropy of a categorical variable. We are primarily interested in the different distributions that we can distinguish from each other by means of positive (correctly classified) and negative (incorrectly classified) instances.
4.1 MISCLASSIFICATION DETECTION BACKGROUND
We define the indicator of the misclassification event as E(X) & l[((X) + Y]. The occurrence of the "misclassification" event is then characterized by E = 1. Misclassification detection is standard binary classification problem, where E needs to be estimated from X. We will denote the misclassification detector g: X -> {0, 1). The underlying pdf px can be expressed as mixture of two random variables: X ~ (x (positive instances) and X_ ~ (x] (negative instances), where Px = and (x represent the pdfs conditioned on the error event and the event of correct classification, respectively.
Let s: X + IR be the uncertainty measure in 3) that assigns a score s(x) to every sample x in the input space X. We can derive misclassification detector g by fixing a threshold 7 R, g(x; 7) = 1 (x) S 7]. where g(x) = means that the input sample x is detected as being E = 1. In Granese et al. (2021), the authors propose to use the Gini coefficient (2) as a measure of uncertainty, which is equivalent to the Rényi entropy of order two, i.e., H2 (x) log Eyer (f(x),)²
4.2 A DATA-DRIVEN MEASURE OF RELATIVE UNCERTAINTY FOR MODEL'S PREDICTIONS
We first rewrite sx(xx) 3 in order to make it amenable to learning the metric d. By defining the
C. x C matrix D a (day) using d, d(i, j), we have sx(x) p(x) D For sx(x) to yield good detector g. we design a contrastive objective, where we would like (X, )]. which is the
expectation over the positive samples, to be small compared to the expectation over negative samples,
4

---

Published as a conference paper at ICLR 2024
Le., ((sss(_))) ). This naturally yields the following objective function, where assume the usual properties distance function d(y: y) = and d(y'. y) = d(y.y')) 2 0 for all y, y E y
Definition 1. Let us introduce our objective function with hyperparameter E [0, 1].
C(D) (1 - X) [(X+) ( (D - A
(4)
for a fixed E Rt,, define our optimization problem as follows:
minimizeneeexe L(D)
subject to
d; = 0,
ViEy
d,j 2 0,
Vi.j E
(5)
dy = dai Tr(DDT) S
Vi,j E
The first constraint in equation that the elements along the diagonal are zeros, which ensures that the uncertainty measure is zero when the distribution is concentrated at single point. The second constraint ensures that all elements are non-negative, which natural condition, so measure of uncertainty is non-negative. The natural symmetry between two elements stems from the third constraint, while the last constraint imposes a constant upper-boung on the Frobenius norm of the matrix D, guaranteeing that solution for the underlying optimization problem exists.
Proposition (Closed form solution). The constrained optimization problem defined in admits closed form solution D = 7 (dig), where
ReLU (A. [(X_ (Tx()_)((_))
i i i=
d'j
(6)
The multiplicative constant Z is chosen such that D' satisfies the condition Tr(D"(D*))) K.
The proof is based Lagrangian approach and relegated to Appendix A.1 Algorithm | in
Appendix AA.2 summarizes all the main steps for the empirical evaluation, including the data preparation and the computation of the matrix D
Note that, apart from the zero diagonal and up to normalization.
D' ReLU (A [p(X_) ) )] (1 A) EE ( p)X.)))
Finally, we define the Relative Uncertainty (REL-U) score for given sample as
SREL-U (x) - p(x) D D" p
(8)
Remark. ( is a special case of when de = i + j and du = 0. Thus, 81 a--a(xx 8gini m (x)
when choosing to be the Hamming distance, which was also pointed out in (Rao) 1982 Note 1).
EXPERIMENTS AND DISCUSSION
In this section, validate measure of uncertainty in the context of misclassification detection, considering both the case when the training and test distributions match (cf. Section 5.11.. and the case in which the distributions mismatch (cf. Section 5.2). Although our method requires additional positive and negative instances, we show that lower amounts are needed (hundreds or few thousands) compared methods that involve re- re-training or fine-tunin finetuniig (hundreds of thousands).
5.1 MISCLASSIFICATTION DETECTION MATCHED DATA
We designed our experiments as follows: for given model architecture and dataset, trained the model on the training dataset. We split the test set into two sets: one portion for tuning the detector (held out validation set) and the other for evaluating it. Consequently, Inperparameteess in an unbiased way and cross-ver validate performance over many splits generated from random seeds. For ODIN (Liang al. 2017) and Doctor (Granese al. (021), found
compute all
5

---

Published as a conference paper at ICLR 2024
the best temperature (T) and imput pre-processiie magnitude perturbation (c). For method, tuned the best lambda parameter (A), T, and e. For details on temperature and input pre- pre-processing equations, see Appendix [A.6] As of evaluation metric, consider the false positive rate (fraction of misclassification. detected as being correct classifications) when 95% of data is true positive (fraction of correctly classified samples detected as being correct classifications), denoted FPR at 95% TPR (lower is better). AUROC results are similar among methods (see Figure 6 in the appendix).
Table 1 showcases the misclassification detection performance in terms of FPR at 95% TPR of our method and the strongest baselines (MSP (Hendrycks & Gimpel (017). ODIN Liang et al. 2017). Doctor (Granese et al. 202I)) on different neural network architectures (DenseNet-121 (Huagg
et 2017). ResNet-34 (He et al. 2016)) trained on different datasets (CIFAR-10) 10, CIFAR- 100
(Krizhevsky 2009)) with different learning objectives (Cross- entropy loss, LogitNorm (Wei et al. 2022), MixUp (ZZang et al. 2017), Rp (Pinto et al., (022), OpenMix (Zhu et al. ) Please refer to Appendix A.. for details on the baseline methods. We observe that, on average, our method performs best 1/20 experiments and is equal to the second best in 4/9 out of the remaining experiments. works consistently better on all the models trained with cross-entropy loss and the models trained with Rp objective, which achieved the best accuracy among them. We observed some negative results when training with logit normalization, but also, the accuracy of the base model decreased. Results for Bayesian methods for uncertainty estimation such Deep Ensembles Taakshminaraynnan et al. 2016) and MCDropout (Gal & Ghahramani (016)), as well as results for an MLP directly trained on the tuning data are reported in Table in the Appendix A.6 We report superior detection capabilities for the task at hand.
Table 1: Misclassification detection results across two different architectures trained on CIFAR-10 and CIFAR- 100 with five different training losses. We report the average accuracy these models and the detection performance in terms of average FPR at 95% TPR (lower is better) in percentage with one standard deviation over ten different seeds in parenthesis.
Model
Training
Accuracy
MSP
ODIN
Doctor
REL-U
CrossEntropy
(4.7)
(0.7)
(0.2)
18.3 (0.2)
DenseNet-122 121 (CIFAR-10)
LogitNorm
92.4
(1.2)
32,7 (1.0)
37.4 (0.5)
(7.0
Mixup
95.1
54.1 (13.4)
38.8 (1.2)
24.5 (1.9)
37.6 (0.9)
OpenMix
94.5
57.5 (0.0)
53.7 (0.2)
33.6 (0.1)
31.6 (0.4)
RegMixUp
95.9
(8.0)
30.4 (0.4)
23.3 (0.4)
22.0 (0.2)
CrossEntropy
73.8
45.1
41.7 (0.4)
41.5 (0.2)
41.5 (0.2)
DenseNet- 121
LogitNorm
73.7
(2.4)
60.8 (0.2)
(0.4)
(0.4)
Mixup
77.5
48.7 (2.3)
41.4 (1.4)
37.7 (0.6)
37.7 (0.6)
(CIFAR-100)
OpenMix
72.5
52.7 (0.0)
51.3(
48.1 (0.3)
45.0 (0.2)
RegMixUp
78.4
49.7 (2.0)
45.5
43.3 (0.4)
40.0 (0.2)
CrossEntropy
95.4
25.8 (4.8)
(1.0)
14.3 (0.2)
14.1 (0.1)
LogitNorm
94.3
30.5 (1.6)
26.0 (0.6)
31.5 (0.5)
31.3 (0.6)
ResNet-34 (CIFAR-10)
Mixup
60.1 (10.7)
38.2 (2.0)
26.8 (0.6)
19.0 (0.3)
OpenMix
94.0
40.4 (0.0)
39,5 (1.3)
28.3 (0.7)
28.5 (0.2)
RegMixUp
97.1
34.0 (5.2)
26.7 (0.1)
21.8 (0.2)
18.2 (0.2)
CrossEntropy
79.0
(2.5)
38.3 (0.2)
(0.9)
32.7 (0.3)
LogitNorm
76.7
(1.0
55.7 (0.1)
(0.2)
65.4 (0.2)
ResNet-34
Mixup
78.1
53.5 (6.3)
43.5 (1.6)
37.5 (0.4)
37.5 (0,3)
(C1FAR-100)
OpenMix
77.2
46.0 (0.0)
43.0 (0.9)
41.6 (0.3)
39.0 (0.2)
RegMisUp
80.8
50.5 (2.8)
45.6 (0.9)
40.9 (0.8)
37.7
Figure 2 displays how the amount of data reserved for the tuning split impacts the performance
of the best two detection methods. We demonstrate how our data-driven uncertainty estimation metric generally improves with the amount of data fed to it in the tuning phase, especially on more challenging setup such as on the CIFAR-100 model.
Training losses or regularization is independent of detection. Previous work highlights the independence of training objectives from detection methods, which challenges the meaningfulness of evaluations. In particular, we identify three major limitations in (Zhu et al. 2023a): The evaluation of post-hoe methods, such Doctor and ODIN, lacks consideration of perturbation and temperature hyperparameters. Despite variations in accuracy and the absence of measures for coverage and risk, different training methods are evaluated collectively. Furthermore, the post-hoc methods assessed on these models. The primary flaw in their analysis stems from evaluating different detectors on distinct models, leading to comparisons between (models, detectors) tuples that have different
not

---

Published as a conference paper at ICLR 2024
FPR at 85% TPR
30.0
ReH-
FPR 95% TPR
42.0
Rel-U
Doctor
25.0
40.0
20.0
38.0
20
30
Split (%)
Split
(a) CIFAR-10
(b) CIFAR-100
Figure 2: Impact of the tuning split size on the misclassification performance ResNet-34 trained with CE loss for our method and the Doctor baseline. Hyperparameters are set to their default values (T 1.0, 0.0, and 0.5), ie., only the impact of the validation split size is observed.
misclassification rates. As result, such an analysis may fail to determine the most performant detection method in real-world scenarios.
Does calibration improve detection? There has been growing interest in developing machine learning algorithms that are not only accurate but also well-calibrated, especially in applications where reliable probability estimates are desirable. In this section, we investigate whether models with calibrated probability predictions help improve the detection capabilities of our method or not. Previous work (Zhu et al., 2023b) has shown that calibration does not particularly help or impact misclassification detection models with similar accuracies, however, they focused only calibration methods and overlooked detection methods.
To assess this problem in the optics of misclassification detectors, we calibrated the soft-probabilities of the models with a temperature parameter (Guo et al. 2017). Note that this temperature is not necessarily the same value the detection hyperparameter temperature. This calibration method simple and effective, achieving performance close to state-of-the-art (Minderer et al.] (2021). To measure how calibrated the model is before and after temperature scaling, we measured the expected calibration error (ECE) (Guo et al. 2017) before, with 7' = 1, and after calibration. We obtained the optimal temperature after cross- validation procedure on the tuning set and measured the detection performance of the detection methods over the calibrated model on the test set. For the detection methods, we use the optimal temperature obtained from calibration, and no input pre- pre-processing is conducted (c = 0), to observe precisely what is the effect of calibration. We set À = 0.5.
Table shows the detection performance over the calibrated models. We cannot conclude much from the CIFAR benchmark as the models are already well-calibrated out of the training, with ECE of around 0.03. In general, calibrating the models slightly improved performance on this benchmark. However, for the ImageNet benchmark, calibration, while REL-U U remained more or less invariant to calibration on ImageNet, suggesting that the performance of REL-U is robust under the model's 's calibration.
observe that Doctor gained lot from the
Table 2: Impact of model probability calibration on misclassification detection methods. The uncalibrated and the calibrated performances are in terms of average FPR at 95% TPR (lower is better) and one standard deviation in parenthesis.
Architecture
Dataset
ECE;
ECEr
Uncal. Doctor
Cal. Doctor
Uncal. REL-U
Cal. REL-U
DenseNet- 121
CIFAR-10
0.03
0.01
31.1 (2.4)
(3.8
32.7 (1.7)
(2.1)
CIFAR-100
0.03
0.01
(1.1)
45.9 (0.9)
45.7 (0.9)
46.6 (0.6) (0.6)
R4sNer-34
CIFAR-10
0.03
0.01
(0.0)
(3.0(
26.2 (0.0)
(4.1
CIFAR 100
0.06
0.04
40.0 (0.3)
38.7 (1.0)
40.6 (0.7)
38.9 (0.9)
ResNet-50
ImageNet
0.41
0.03
76.0 (0.0)
55.4 (0.7)
51.7 (0.0)
53.0
5.2
MISMATCHED DATA
So far, we have evaluated methods for misclassification detection under the assumption that the data available learn the uncertainty measure and that during testing are drawn from the same distribution.
7

---

Published as a conference paper at ICLR 2024
In this section, consider cases in which this assumption does not hold true, leading to mismatch between the generative distributions of the data. Specifically, we investigate two sources of mismatch: Datasets with different label domains, where the symbol sets and symbols cardinality in each dataset; it) Perturbation of the feature space domain generated using popular distortion filters. Understanding how machine learning models and misclassification detectors perform under such conditions can help us gauge and evaluate their robustness.
different
Mismatch from different label domains. We considered pre-trained classifiers on the CIFAR-10 dataset and evaluated their performance in detecting samples in CIFAR-10 and distinguishing them from samples in CIFAR 100, which has a different label domain. Similar experiments have been conducted in Ren et al. (202I); Fort et al. (202T); Zhu et al. (2023a). The test splits were divided into validation set and an evaluation set, with the validation set consisting of 10%, 20%, 33%, or 50% of the total test split and samples used for training were not reused. For each split, we
0.910
0,928
Docter
0.33
Rsa-U
01.30
Rea.
0.008
0,928
38%
0.34
01.30
at
0,006
0.34
0,024
0.30
0.904
0.37
10
20
50
10
20
50
10
20
50
10
20
33
50
Spit(%)
Spie(%))
Spit(()
Spit(()
(a) DenseNet-121
(b) ResNet-34
Figure 3: Impact of different validation set sizes (in percentage of test split) for mismatch detection.
combine the number of validation samples from CIFAR-10 with equal number of samples from 100. In order to assess the validity of our results, each split has been randomly selected 10 times, and the results are reported in terms of mean and standard deviation in Figure We observe how our proposed data-driven method performs when samples are provided to accurately describe the two groups. In order to reduce the overlap between the two datasets, and in line with previous work (Fort et al. 202T). we removed the classes in CIFAR- 100 that most closely resemble the classes in CIFAR- For the detailed list of the removed labels, we refer the reader Appendix(A..
Mismatch from feature space corruption. We trained model on the CIFAR-10 10 dataset and evaluated its ability to detect misclassification on the popular CIFAR-10C corrupted dataset, which contains a version of the classic CIFAR- 10 test set perturbed according to 19 different types of corruption and levels of intensities. With this experiment, we aim to investigate if our proposed detector is able to spot misclassifications that arise from input perturbation, based on the sole knowledge of the misclassified patterns within the CIFAR- 10 test split.
Consistent with previous experiments, we ensure that no samples from the training split are reused during validation and evaluation. To explore the effect of varying split sizes, we divide the test splits into validation and evaluation sets, with validation the total test split. Each split has been produced 10 times with 10 different seeds and the average of the results has been reported in the spider plots in Figure In the case of datasets with perturbed feature spaces, we solely utilize information from the validation samples in CIFAR-10 10 to detect misclassifications in the perturbed instances of the evaluation datasets, without using corrupted data during validation. We present visual plots that demonstrate the superior performance achieved by our proposed method compared to other methods. Additionally, for the case of perturbed feature spaces, we introduce radar plots, in which each vertex corresponds to a specific perturbation type, and report results for intensity 5. This particular choice of intensity is motivated by the fact that it creates the most relevant divergence between the accuracy of the model the original test split and the accuracy of the model on the perturbed test split. Indeed the average gap in accuracy between the original test split and the perturbed test split is reported in Table 5 in Appendix A.8
consisting of 10%, 20%, 33%, 50% of
We observe that our proposed method outperforms Doctor in terms of AUROC and FPR, as demon- strated by the radar plots. As (Figure 4 show how the area covered by the AUROC values achieves similar or larger values for the proposed method, indeed confirming that it is able better detect misclassification in the
see, in the case of CIFAR- 10 CIFAR- 10C, the radar plots
8

---

Published as a conference paper at ICLR 2024
mismatched data. Moreover, the FPR values lower for the proposed method. For completeness. we report the eπror bar tables in Tables and Appendix A.8 Additionally, as particular case of mismatch from feature space corruption, have considered the task of detecting mismatch between
MNIST and SVHN, the results are reported Figure Appendix A.8
fox
elastic
elattic
defocus
defocus
glane
glass
impulao
impulso
bright
bright
logg
josk
200m
motias
monion
Nowrkle
specile
poalats
plse ats
Doctor
Gaturate
Spotter
Doctor
saturote
Spatter
Rur.
SNOW
REc U
SNOW
(a) AUROC
(b) FPR at 95% TPR
Figure 4: CIFAR-10 vs CIFAR-IOC, ResNet-34, using 10% of the test split for validation.
5.3 EMPIRICAL INTERPRETATION OF THE RELATIVE UNCERTAINTY MATRIX.
F exxmplifies the advantage of our method over the entropy-based methods in and (2). In particular, the left-end side heatmap represents the D matrix learned by optimizing on CIFAR- 10. Darker shades of blue indicate higher uncertainty, while lighter shades of blue indicate lower uncertainty. The central heatmap is the predictor's 's class-wise true confusion matrix. The vertical axis represents the true class, while the horizontal axis represents the predicted class. For each combination of two classes i., the corresponding cell reports the count samples class j that were predicted class i. The correct matches along the diagonal are dashed for better visualization of the mistakes. The confusion matrix is computed on the same validation set used to compute the D matrix. Crucially, our uncertainty matrix can express different degrees of uncertainty depending on the specific combination of classes at hand. Let us focus for instance on the fact that most of the incorrectly classified dogs predicted cats, and vice-versa. The matrix D fully captures this by assigning high uncertainty to the cells at the intersection between these two classes. This not the case for the entropy-based methods, which cannot capture such a fine-gratd fine-grained uncertainty, and assign the same uncertainty to all the cells, regardless of the specific combination of classes at hand.
SUMMARY AND CONCLUDING REMARKS
To the best of our knowledge, we are the first to propose REL.UU, a method for uncertainty assessment that departs from the conventional practice of directly measuring uncertainty through the entropy of the output distribution. REL-U uses a metric that leverages higher uncertainty score for negative data w.r.t. positive data, e.g., incorrectly and correctly classified samples in the context of misclassification detection, and attains favorable results on matched and mismatched data. In addition, our method stands out for flexibility and simplicity, relies on closed form solution to optimization problem. Extensions to diverse problems present both an exciting and promising avenue for future research.
Limitations. We presented machine learning researchers with a fresh methodological outlook and provided machine learning practitioners with a user-friendly tool that promotes safety in real-world scenarios. Some considerations should be put forward, such the importance of cross-validating the hyperparameters of the detection methods to ensure their robustness on the targeted data and model. As data-driven measure of uncertainty, to achieve the best performance, it is important to have enough samples at the disposal to learn the metric from as discussed on Section 5.1 As every detection method, method may be vulnerable targeted attacks from malicious users.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work has been supported by the project PSPC AIDA: 2019-PSPC-09 funded by BPI-France. This work was granted access to the HPC/AI resources of IDRIS under the allocation 2023 AD01101280322 made by GENCI.
REFERENCES
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. CoRR, 0bs//800100055: 2018.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, arXiv preprint arXiv:1606. 06566, 2016.
Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. CoRR, abs/2107.075 2021.
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael I. Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Jun Cen, Di Luan, Shiwei Zhang, Yixuan Pei, Yingya Zhang, Deli Zhao, Shaojie Shen, and Qifeng Chen. The devil is in the wrongly-classfied samples: Towards unified open-set recognition. ArXiv, abs/2302.04002, 2023.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey. CoRR, 0bs//190.03440)) 2019.
C. K. Chow. On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theory, 16(1):41-44 46, 1970. doi: 10. 109//TT.1970 197.0.44460.
Oliver Cobb and Arnau Van Looveren. Context-aware drift detection. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, PMLR, 2022.
volume 162 of Proceedings of Machine Learning Research, pp.
Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Ach---Bu. Emily B. Fox, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canuda, pp. 2898-2909, 2019.
Imre Csiszár. Eine informationsthhorrtischhe ungleichung und ihre anwendung auf den beweis der ergodizität markoffschen ketten. Magyer Tud. Akad. Mat. Kutato Int. Koezl., 8:85-108, 1964.
Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertainty aware classifiers with conformalized deep learning. CoRR, 005072205505878, 2022. doi: 10.48550/arXiv. 2205.05878.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of diiti--distiiittion detection. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6- 14, 2021, virtual, pp.
2021.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1050-1059. JMLR.org org, 2016.
10

---

Published as a conference paper at ICLR 2024
Yonatan Geifman and Ran El- Yaniv.. Selective classification for deep neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4878 4887, 2017.
Chuanxing Geng. Sheng-Jud Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE Transactions Puttern Analysis and Machine Intelligence, 43(10))311-3661 oct 2021. doi: 10. 10/1000mmmm..200... 29016994444
Corrado Gini. Variabilità mutabilità; contributo allo studio delle distribuzioni delle relazioni statistiche. In [Fasc. I.J. Tipogr. Di Cuppini 1912., 1912.
Federica Granese, Marco Romanelli, Daniele Gorla, Catuscia Palamidessi, and Pablo Piantanida. DOCTOR: A simple method for detecting misclassification errors. Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 5669-5681 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings the 34th International Con- ference on Machine Learning, ICML 2017, Sydney, NSW. Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1 20.10.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassífied and out-of-distribttion examples in neural networks. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2017, Honolulu. HI, USA, July 21-26, 2017, pp. 2261-2269 IEEE Computer Society, 2017. doi: 10.1100/CVP..2207.2224
Nikita Yurevich Kotelevskii, Aleksandr Artemenkov, Kirill Fedyanin, Fedor Noskov, Alexander Fishkov, Artem Shelmanov, Artem Vazhentsev, Aleksandr Petiushko, and Maxim Panov. Non- parametric uncertainty quantification for single deterministic neural network. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The Annals of Mathemati- cal Statistics, 22(1))79-88) 86, March 1951. doi: 10. 1AM2/////0
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2016.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting outtof-distributton samples and adversarial attacks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianch Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp- 7167-7177. Curran Associates, Inc., 2018
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distributiio image detection in neural networks. arXiv preprint arXiv:1706.02690. 2017.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of duittf-diittibuutin image detection in neural networks. In International Conference Learning Representations, 2018.
11

---

Published as a conference paper at ICLR 2024
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance aware- ness. In H. Larochelle, M. Ranzato, R. Hadsell. M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 7498-7512. Curran Associates, Inc., 2020.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Ann Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Jishnu Mukhoti, Joost van Amersfoort, Philip H. S. Torr, and Yarin Gal. Deep deterministic uncer- tainty for semantic segmentation. In CoRR, volume abs/21 1.00079, 2021.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H.S. Torr, and Yarin Gal. Deep deterministic uncertainty: A new simple baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24384-24394, June 2023.
Marco AF Pimentel, David A Clifton, Lei Clifton, and Lionel Tarassenko. A review of novelty detection. Signal processing. 99:215-249, 2014.
Francesco Pinto, Harry Yang. Ser-Nam Lim, Philip Torr, and Puneet K. Dokania. Regmixup: Mixup as a regularizer can surprisingly improve accuracy and out distribution robustness. In Advances in Neural Information Processing Systems, 2022.
C Radhakrishna Rao. Diversity and dissimilarity coefficients: unified approach. Theoretical population biology, 21(1):24-43, 1982.
Jie Ren, Stanislav Fort, Jeremiah Z. Liu. Abhijit Guha Roy, Shreyas Padhy, and Balaji Laksh- minarayanan. A simple fix to mahalanobis distance for improving near-ood detection. CoRR, abs/2106.09022, 2021.
Alfréd Rényi. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability. Berkeley, California, USA, 1961.
Yaniv Romano, Matteo Sesia, and Emmanuel Candès. Classification with valid and adaptive coverage. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florin, Balcan, and Hsuan- Mieunn-Tienaae Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3): 479-423, July 1948. ISSN 0005-8580. doi: 10.
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D. Sculley, Joshua V. Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncertainty? evalu- ating predictive uncertainty under dataset shift. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13969-13988 2019.
Constantino Tsallis. Possible generalization of boltzmann-gibbs statistics. Journal of statistical physics, 52:479-488 1988.
Meet P. Vadera, Adam D. Cobb, Brian Jalaian, and Benjamin M. Marlin. Ursabench: Comprchensive benchmarking of approximate bayesian inference methods for deep neural networks. CoRR, Abs/2000.04446. 2020.
Joost Van Amersfoort, Lewis Smith, Yee Whye Tch. and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In Hal Daumé and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9690-9700 PMLR, 13-18 Jul 2020.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International Conference Machine Learning, 2022.
12

---

Published as a conference paper at ICLR 2024
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. CoRR, abs/1710.09412, 2017.
Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Cheng-Lin Liu. Openmix: Exploring outlier samples for misclassification detection. ArXiv, (bbs/2303.10993 2023a.
Fei Zhu, Zhen Cheng, Yao Zhang, and Cheng-Lin Liu. Rethinking confidence calibration for failure prediction. ArXiv, abs/2303.02970, 2023b.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 PROOF OF PROPOSITION 1
We have the optimization problem
uniiiiizepeoos L(D) subject to
d = 0, vi fl....,, C}
di - d3. = 0, Vi.j (1....,c) Tr(DDT) KK
(9)
di 0, Vi,j ( (1)
in standard form (Boyd & Vandenberghe, 2004, eq. (4.1)) and can thus apply the KKT conditions
(Boyd& & Vandenberghe/ 2004 eq. (5.49)). We find
VL(D") - Σ
Si Vd +
Σ
14 Vd
Σ
Vidly d(. + +~ V(V(D(())))) - K) =
(10)
as well as the constraints
di
Gi dz.
(11)
dip <
E/ 2
(12)
Eidy =
K² 20
(13)
<*(""D"(""""> - K) = 0
(14)
We have
VL(D")= = (1-)) A)- E - A [p(X_)
(15)
(DT) K) 2D
(16)
and thus
0
(1 E (x(+)) +) - - -E" diag(u")
+ - -v-') +K22D"
(17)
1
D
- (1 - 2) [b(x.) (((x)) (D(_ [ + diag(u")
2x
vz (U+)T)
(18)
As VL(D") in 15 is already symmetric, we can choose 2² = 0. We choose 14 = diag( (VL(D))
to ensures di 0. The non-negativity constraint can be satisfied by appropriately choosing 0 E- = ReLU(-VC(D') satisfied. In total, this yields D
Finally, x² is chosen such that the constraint "r((D(DD)T)) K is
Z (e)) where
A) E [ D )b)
A BxX P(X_),
( i=j"
(19)
The multiplicative constant 2x > is chosen such that D satisfies the condition Tr(D" (D=)
Remark. technical problem may occur when di as defined in 19 equal zero for all i.j E (1,2...., C.}. In this case, D cannot be normalized to satisfy Tr(D=(D*)T) = K and the solution to the optimization problem in (9) is the all-zero matrix D = 0. Le., learning is
performed in this case. We deal with this problem by falling back to the Gini coefficient 2 where
similarly no learning required.
Equivalently, one may also add small numerical correction E to the definition of the ReLU function, i.e., ReLU(x) max(x, e). Using this slightly udapted definition when defining D 2 (e). naturally yields the Gini coefficient in this case.
We use X = diag(x) for a vector x to obtain a matrix X with x on the diagonal and zero otherwise. Slightly abusing notation, also write diag(X () obtain the diagonal of the matrix X as vector
14

---

Published as a conference paper at ICLR 2024
A.2 ALGORITHM
In this section, we introduce a comprehensive algorithm to clarify the computation of the relative uncertainty matrix D
Algorithm 1 Offline relative uncertainty matrix computation.
Require: p: X' + RC trained on training set with C' classes, validation set Dm = {(xx, iid
PXY 1y and hyperparameter À E [0,
Dt +- 2. D + Ø for (x. y) E Dm do
Initialize empty positive and negative sets respective sets with positive or negative samples
if arg m p(x), y then m t D U [f(x)}
else
m t Dm U (f(x)}
end if end for
a
+
E DE p. p. #* + [aml
EpEdi. PTp
DDt
D
+ Ocxc
D Cy C square matrix with zeroed out elements
for
+ Lii i
C,
Build D* according to (6)
for j 1, I C.j f do
if + j then
d max (Au4) (1 - A) plj. 0)
end if end for end for return D"
At test time, it suffices compute
obtain the relative uncertainty of the prediction.
A.3 DETAILS ON BASELINES AND BENCIIMARKS
In this section, we provide a comprehensive review of the baselines used on our benchmarks. We state the definitions using notation introduced in Section
A.3.1 MSP
The Maximum Softmax Probability (MSP) baseline Hendrycks & Gimpel (017) proposes to use the confidence of the network as detection score:
SMpP(x) = max pxx),
(20)
yEy
A.3.2 ODIN
Liang et al. (2177) improve upon Hendrycks & Gimpel (2017) by introducing temperature scaling and input pre- processsing techniques as described in Appendix A.4 and then compute (20) as the detection score. We tune hyperparameters T and validation set for each pair of network and training procedure.
A.3.3 DOCTOR
Granese et al. (2021) propose (2) as the detection score and applies temperature scaling and input pre-processing as described in Appendix A..4] Likewise, we tune hyperparameters T' and e on validation set for each pair of network and training procedure.
A.3.4 MLP
We trained an MLP with two hidden layers of 128 units with ReLU activation function and dropout with p 0.2 on top of the hidden representations with binary cross entropy objective the
15

---

Published as a conference paper at ICLR 2024
validation set with Adam optimizer and learning rate equal to 10-3 until convergence. Results misclassification are presented in Table
A.3.5 MCDROPOUT
Gal & Ghahramani (2016) propose to approximate Bayesian NNs by performing multiple forward passes with dropout enabled. To compute the confidence score, we averaged the logits and computed the Shannon entropy defined in [ We set the number of inferences hyperparameter
and we set the dropout probability to p 0.2. Results on misclassification are presented in Table 3
A.3.6 DEEP ENSEMBLES
Lakshminarayanan et al. (016) propose to approximate Bayesian NNs by averaging the forward pass of multiple models trained on different initializations. We experiments with k = 5 different random seeds. To compute the confidence score, we averaged logits and computed the MSP response (2 Results misclassification are presented in Table
A.3.7 CONFORMAL PREDICTIONS
According to conformal learning Angelopoulos & Bates (2021); Angelopoulos et al. (2021); Romano jtt al. (2020) the presence of uncertainty in predictions is dealt by providing, addition to estimating the most likely autcome-actionall "covers" the ground truth with a high probability. This means that the predictor implements an uncertainty set function, i.e., function that returns set of labels and guarantees the presence of the right label within the set with a high probability for a given distribution.
uncertainty quantification, "prediction set" that provably
A.3.8 LOGITNORM
Wei et 2022 observe that the norm of the logit keeps increasing during training, leading to overconfident predictions. So, they propose Training neural networks with logit normalization to hopefully produce more distinguishable confidence scores between in- and out-of-distribttion data. They propose normalizing the logits of the cross entropy loss, resulting in the following loss function:
exp (C)])
((f(x). = -- loog
(21)
Gi=l Ei= exp (X)))
A.3.9 MIXUP
Zhang et al. (2017) propose to train a neural network on convex combinations of pairs of examples and their label to minimize the empirical vicinal risk. The mixup data is defined
x Ax;
A)x; and Ay (1 (1-X)u; A)yy for i, (1....,),
(22)
where À is sampled according Beta(a, a:) distribution. We used 1.0 train the models. Observe a slight abuse of notation here, where y is actually an one-hot encoding of the labels
y Mygl,..., c=
A.
REGMIXUP
Pinto et al. 2022 use the cross entropy of the mixup data as in (22) with À sampled according to
Beta 10) distribution as a regularizer of the classic cross entropy loss for training a network. The objective is balanced with hyperparameter 7, usually set to 0.5.
A. OPENMIX
Zhu et al. 2023a) explicitly add an extra class for outlier samples and uses mixup as a regularizer for the cross entropy loss, but mixing between inlier training samples and outlier samples collected from the wild. It yields the objective
L EDa. [e([(x): y)] + 7ED [E([(x), (x). 5)].
(23)
where y E R is hyperparameter, x = Axintier + (1 - X)xontliers and ÿ = Ayaller + (1 - A) + 1)
with slight abuse of notation. The parameter À is sampled according to Beta 10) distribution.
16

---

Published as a conference paper at ICLR 2024
A.4 TEMPERATURE SCALING AND INPUT PRE-PROCESSING
Temperature scaling involves the use of scalar coefficient T IR+ that divides the logits of the network before computing the softmax. This affects the network confidence and the posterior output probability distribution. Larger values of T induce more flat posterior distribution and smaller values, peakier responses. The final sumperature-ccale--sotite
function is given by:
exp (z/T)
a(2) -(3)=E
Σ, exp (z
Moreover, the perturbation is applied to the input image in order to increase the network "sensitivity" to the input. In particular, the perturbation is given by:
= x x sign
log (x))).((x)],
for O. Note that (-) replaced by the scoring functions of ODIN (20) and Doctor to compute input pre- pre-processing in their respective experiments.
A.5 ADDITIONAL COMMENTS ON THE ABLATION STUDY FOR HYPER-PARAMETER SELECTION
Ablation study. We conducted ablation studies on all relevant parameters: T, e, and A. It is crucial emphasize that T intrinsic to the network architecture and, therefore, must not be considered hyper--paree parameter for REL-U. U. Figure 5 illlusrates three ablation studies conducted to analyze and comprehend the effects of different factors on the experimental results. Fach subplot represents each hyperparameter ablation study, showcasing the outcomes obtained under specific conditions. We observe that À 2 0.5, low temperatures, and low noise magnitude achieve better performance. Overall, the method is shown to be robust to the choices of hyperparameters under reasonable ranges.
Additionally, the introduction additive noise serves the purpose of ensuring a fair comparison with Doctor/ODIN, where the noise was utilized to enhance detection performance. Nevertheless, as indicated by the results in the ablation study illustrated in Figure 5 0 seems to be close to optimal most of the time, thereby positioning REL-U as effective algorithm that relies only on the soft-probabilin output, therefore comparable to Granese et al. ); Liang et al. (018) in their version with perturbation, and Hendrycks & Gimpel (2017). Furthermore, REL- exhibits considerable degree of insensitivity to various values of X, as evident from Figure 5 This suggests that a potential selection for À could have been À / NN), aiming balance the ratio between the number of positive (N+) and negative (N-) examples. In such a scenario, there are no hyper-parametert at all.
ResNet-34 (C- 10)
ManseNat-1211 (C-10)
ResNot-34 (C- 100)
DenseNet:121 (G-100)
RestVtt 50 (ImageNot)
FPR 85% TPR
40
20
0.00
0.25
0.50
0.75
1.00
0.50
0.75
1.00
1.25
1.50
1.75
2.00
1.0,
r=
T,0
0.5, =.0)
=
T=
Figure 5: Ablation studies for temperature, lambda, and noise magnitude effects. The x-axis represents the experimental conditions, while the y-axis shows the performance metric.
A.6 ADDITIONAL RESULTS ON MISCLASSIFICATION DETECTION
Bayesian methods. In this paragraph, we compare our method to additional uncertainty estimation methods, such Deep Ensembles (Lakshminarayanan al.) 22166), MCDropout (Gal & Ghahramani 2016), and a MLP directly trained on the validation data used to tune the relative uncertainty matrix. The results available in Table
17

---

Published as a conference paper at ICLR 2024
Table Misclassification detection results across two different architectures trained on CIFAR- 10 and CIFAR-IH with CrossEntropy loss. We report the detection performance in terms of average FPR at 95% TPR (lower better) percentage with one standard deviation over ten different seeds in parenthesis.
Model
Dataset
MCDropout
Deep Ensembles
MLP
REL-U
DenseNet- 121
CIFAR-I0
30.3 (3.8)
25.5 (0.8)
37.3 (5.8)
18.3 (0.2)
DenseNet-122 121
CIFAR-100
47.6 (1.2)
45.9 (0.7)
78.4 (1.4)
41.5 (0.2)
ResNet-34 ResNet-34
CIFAR-10
25.8 (4.9)
14.8 (1.4)
33.6 (2.7)
14.1 (0.1)
CIFAR-I00
42.3 (1.0)
37.4 (1.9)
63.3 (1.0)
32.7 (0.3)
ROC and Risk-Coverage curves. We also display the ROC and the risk-coverage curves for main benchmark on models trained on CIFAR-10 with cross entropy loss. We observe that the performance of REL-U is comparable other methods in terms of AUROC while outperforming them in high-TPR regions and reducing the risk of classification errors when abstention is desired (coverage) as observed in Figure
DenseNet-121 (CIFAR-10) (CE)
DenseNet-121 (CIFAR-10) (CE)
1.00
MSP (AUC-0,78)
ODIN (AUC=0.77)
True Positive Rate
0.95
DOCTOR (AUC-0.60)
100 Risk
MIPP (AUC-0.92)
(AUC=0.59)
0.90
MSP (AUC-0.99)
ODIN (AUC=0.93)
0.85
DOCTOR (AUC-
MLP (AUC=0.91)
Ours (AUC=0,93)
0.80
0.0
0.2
0.6
0.8
1.0
0.5
0.6
0.7
0.8
False Positive Rate
Coverage
(a) DenseNet- 121 ROC curve.
(b) DenseNet- 121 RC curve.
ResNet-34 (CIFAR-10) (CE)
ResNet-34 (CIFAR-10) (CE)
1.00
MSP (AUC-0.35)
(AUC=0.34)
True Positive Rate
DOCTOR (AUÇ=0.34)
0.95
Risk
MLP (AUC-0.A) -3)
Ours (AUC=0.30)
MSP /AUC=0.95)
100
ODIN (AUC=0.95)
0.85
DOCTOR (AUC+0.95)
(AUC-0.93)
Ours (AUC=0,95)
0.80
0.0
0.2
0.6
0.8
1.0
0.5
0.6
0.7
10
False Positive Rate
Coverage
(c) ResNet-34 ROC curve.
(d) ResNet-34 RC curve.
Figure 6: Equivalent performance the detectors in terms of ROC demonstrating lower FPR for our method for high TPR regime. The risk and coverage (RC) curves also looks similar between methods, with small advantage to our method in terms of AURC.
Performance of conformal prediction. We take into account the application of conformal pre- dictors applied to the problem of misclassification. In particular, consider the excellent work in Angelopoulos & Bates (202I), but most importantly (Agellopoulss importatty/ngglooollls et al. (202T), which, in turn, builds upon Romano al. (2020). Conformal predictors, stark contrast with standard prediction
18

---

Published as a conference paper at ICLR 2024
models, learn "prediction set function". i.e. they return set of labels, which should contain the correct value with high probability, for given data distribution. In particular, Angelopoulos et al. (2021) proposed revision of Romano al. (020), with the main objective of preserving the guarantees of conformal prediction, while, at the same time, minimizing the prediction set cardinality sample basis: samples that "harder" to classify can produce larger than samples that casier to correctly classify. The models are "conformalized" (cf. Angeloppulos et al. (2021)) using the same validation samples, that also available the other methods. We reject the decision if the second largest probability within the prediction set exceeds a given threshold, as then, the prediction set would contain more than one label, indicating possible misclassification event. The experiments are run on 2 models, 2 datasets and 3 training techniques for a total of 12 additional numerical results reported in Table For the model trained with cross entropy in Table 4 the area
Table Misclassification detection results across two different architectures trained on CIFAR-10 10 and CIFAR-100 with five different training losses. We report the average accuracy of these models and the detection performance in terms average FPR at 95% TPR (lower is better) percent with one standard deviation ten different seeds in parenthesis. The values for the conformalized models are reported in the right-most column.
Model
Training
Accuracy
MSP
ODIN
Doctor
REL-U
Conf.
DenseNet-121 (CIFAR-I0)
CrossEntropy
94.0
32.7 (4.7)
24.5 (0.7)
21.5 (0.2)
18.3 (0.2)
(3.3)
Mixup
95.1
54.1 (13.4)
38.8 (1.2)
24.5 (1.9)
37.6 (0.9)
57.6 (6.9)
RegMixUp
95.9
41.3 (8.0)
30.4 (0.4)
23.3 (0.4)
22.0 (0.2)
30.3 (5.1)
CrossEntropy
73.8
45.1 (2.0)
41.7 (0.4)
41.5 (0.2)
41.5 (0.2)
46.5 (1.3)
DenseNet-121 (CIFAR-100)
Mixup
77.5
48.7 (2.3)
41.4 (1.4)
37.7 (0.6)
37.7 (0.6)
47.0 (1.3)
RegMixUp
78.4
49.7 (2.0)
45.5 (L.1)
43.3 (0.4)
40.0 (0.2)
46.0 (1.3)
CrossEntropy
95.4
25.8 (4.8)
19.4 (1.0)
14.3 (0.2)
14.1 (0.1)
26.8 (4.6)
ResNet-34
Mixup
96.1
60.1 (10:7)
38.2 (2.0)
26.8 (0.6)
19.0 (0.3)
58.1 (5.6)
(CIFAR-10)
RegMixUp
97.1
34.0 (5.2)
26.7 (0.1)
21.8 (0.2)
18.2 (0.2)
41.9 (7.0)
CrossEntropy
79.0
42.9
38.3
34.9
32.7 (0.3)
38.7 (1.5)
ResNet-34
Mixup
53.5 (6.3)
43.5 (1.6)
37.5 (0.4)
37.5 (0.3)
(0.9)
(CIFAR-100)
RegMixUp 80.8
50.5 (2.8)
45.6 (0.9)
40,9 (0.8)
37.7 (0.4)
47.7 (1.5)
under the ROC curve, averaged over 10 seeds, is 0.92 (0.7) for the DenseNet-122 121 conformalized model on CIFAR- 10, and 0.93 (0.7) for the ResNet- conformalized model on CIFAR-10 showing comparable results w.r.t. the results in Figures 6a and 6b
19

---

Published as a conference paper at ICLR 2024
A.7 MISMATCH FROM DIFFERENT LABEL DOMAINS
In order to reduce the overlap between the label domain of CIFAR-I0 and CIFAR-100, in this experimental setup we have ignored the samples corresponding to the following classes in CIFAR- 100: bus, camel, cattle, fox, leopard, lion, pickup truck, streetcar, tank, tiger, tractor, train, and wolf.
A.8 MISMATCH FROM DIFFERENT FEATURE SPACE CORRUPTION
Table 5: We report the gap in accuracy between the original and the corrupted test set for the considered model. The gap is reported and average and standard deviation over the 19 different types of corruptions for corruption intensity equal to 5. The maximum and minimum gap are also reported, with the relative corruption type.
Architecture
Average gap
Max gap
Min gap
DenseNet121
0.36 + 0.18
0.66 (Gaussian Blur)
0.04 (Brightness)
ResNet34
0.35 + 0.20
0.72 (Impulse Noise)
0.03 (Brightness)
Table 6: DenseNet- 121, error bar table, mismatch from different feature space corruption
Doctor
REL-U
Corruption
Split (%)
AUC
FPR
AUC
FPR
10
0.90 = 0.00
0.31 = 0.00
0.90 + 0.01
0.35 + 0.03
20
0.90 = 0.00
0.31 0.00
0.90 + 0.00
0.32 0.01
Brightness
33
0.90 + 0.00
0.31=0000
+.00
0.32 + 0.01
0.90 0.00
0.31 + 0.00
0.90 0.00
0.32 0.00
0.66 + 0.02
0.77 + 0.03 0.03
0.73 + 0.02
0.70 0.02
20
0.66 + 0.02
0.77 177:000
0.73 + 0.01
0.69 0.02
Contrast
33
0.67 0.01
0.76
0.74 0.01
0.68 + 0.01
50
0.66 + 0.01
0.77
0.74 0.01
0.67 0.01
0.70 + 0.01
0.75 0.00
0.72 + 0.03
0.71 0.05
20
0.70 0.01
0.75
0.73 - 0.01
0.69 0.01
Defocus blur
33
0.70 = 0.00
0.75
0.73 0.01
0.70
50
0.70 + 0.00
0.75
0.73 + 0.01
0.71 0.01
0.80 0.01
0.56
0.81 0.01
0.55 = 0.02
20
0.80 + 0.01
0.56
0.82 + 0.00
0.53 - 0.02
Elastic transform
33
0.80 0.00
0.50
0.82 - 0.00
0.53 0.01
50
0.80 + 0.00
0.56 0.00
0.82 + 0.00
0.53 + 0.01
10
0.76 + 0.01
0.63 0.01
0.79 + 0.01
0.56 - 0.03
20
0.76 + 0.01
0.63 + 0.01
0.79 + 0.01
0.55 + 0.02
Fog
33
0.77 0.00
0.63
0.80 + 0.00
0.56 0.02
50
0.77 - 0.00
0.63 0.00
0.80 + 0.00
0.55 - 0.01
10
0.78 + 0.00
0.62 0.00
0.79 + 0.01
0.61 + 0.02
20
0.78 + 0.00
0.62 0.00
0.79 - 0.01
0.59 0.02
Frost
33
0.78 + 0.00
0.62
0.80 - 0.00
0.59 - 0.01 0.01
50
0.78 + 0.00
0.62 062-0.00 0.00
0.80 + 0.00
0.59 - 0.01
10
0.60 + 0.00
0.84 + 0.00
0.61 + 0.05
0.82 + 0.05
0.60 + 0.00
0.84 + 0.00
0.63 0.03
0.82 0.02
Gaussian blur
33
0.60 + 0.00
0.84 + 0.00
0.62 + 0.02
0.82 + 0.01
50
0.60 = 0.00
0.84+..00 0.00
0.61 + 0.02
0.70 + 0.00
0.72 + 0.00 0.00
0.69 + 0.02
0.73 0.02
20
0.70 + 0.00
0.72
0.71 + 0.01
0.72 0.01
Gaussian noise
20

---

Published as a conference paper at ICLR 2024
33
0.70
0.72 0.00
0.70 0.01
0.73 =
50
0.70 0.00
0.72 0.00
0.70 0.01
0.73 0.01
10
0.72 0.00
0.73
0.71
0.73
20
0.72 0.00
0.73:0000
0.72 0.01
0.72
Glass blur
33
0.72 + 0.00
0.01
0.73 - 0.00
50
0.72 0.00
0.73 = 0.00
0.72 0.00
0.73 0.00
0.62 + 0.00
0.85 = 0.00
0.61 + 0.03
0.84 0.01
20
0.62 + 0.00
0.85 + 0.00
0.63 + 0.02
0.83 0.01
Impulse noise
33
0.62 + 0.00
0.85 - 0.00
0.62 + 0.01
0.84 - 0.01
50
0.62 + 0.00
0.85 + 0.00
0.62 + 0.01
0.84 = 0.01
0.81 0.00
0.58 + 0.00
0.80 0.01
0.56 0.02
20
0,81 = 0.00
0.58 = 0.00
0.80 = 0.00
0.55 = 0.01
Jpeg compression
33
0,81 = 0.00
0
0.81 0.00
0.55 0.01
50
0.81 + 0.00
0.58 = 0.00
0.81 + 0.00
0.55 0.01
0.78 + 0.01
0.63 = 0.00
0.81 0.01
0.56 0.02
20
0.78 + 0.01
0.63 0.00
0.82 + 0.01
0.53 = 0.02
Motion blur
33
0.78 = 0.00
0.00
0.82 - 0.00
0.54 = 0.02
50
0.78 + 0.00
0.63 + 0.00
0.82 + 0.00
0.54 + 0.01
10
0.68 + 0.00
0.82 + 0.00
0.68 + 0.03
0.80 0.01
20
0.68 + 0.00
0.82 + 0.00
0.67 + 0.03
0.81 0.01
Pixelate
33
0.68 0.00
0.82 0.00
0.66 + 0.02
0.81 0.01
50
0.68 + 0.00
0.82 0.00
0.67 + 0.02
0.81 = 0.01
10
0,89 + 0.00
0.37 - 0.01
0.88 + 0.01
0.39 - 0.03
20
0.89 + 0.00
0.37 0.01
0.88 + 0.00
0.36 - 0.01
Saturate
33
0.37 0.89 = 0.00
0.88 = 0.00
0.37 = 0.01
50
0.89 = 0.00
0.00
0.88 + 0.00
0.36=000
10
0.71 + 0.00
0.00
0.72 + 0.02
0.72 - 0.02
20
0.71 0.00
0.72 0.00
0.73 0.01
0.70 0.02
Shot noise
33
0.71 + 0.00
0.00
0.73 + 0.01
0.70 + 0.01
50
0.71 = 0.00
0.00
0.73 + 0.01
0.71
0.81 + 0.00
0.60 + 0.00
0.81 0.01
0.57 0.01
20
0.81 + 0.00
0.60 0.00
0.81 0.01
0.57 0.02
Snow
33
0.81 + 0.00
0.60 0.00
0.81 + 0.00
0.57 0.01
50
0.81 0.00
0.60 = 0.00
0.81 + 0.00
0.57 0.00
0.78 + 0.00
0.80 = 0.00
0.77 + 0.02
0.80 0.04
20
0.78 + 0.00
0.80 = 0.00
0.77 - 0.01
0.79 0.03
Spatter
33
0.78 + 0.00
0.80 + 0.00
0.77 + 0.01
0.80 0.02
50
0.78 + 0.00
0.80 + 0.00
0.77 + 0.00
0.80 0.02
0.73 + 0.00
0.68 + 0.00
0.74 0.02
0.67 0.03
20
0.73 + 0.00
0.68 = 0.00
0.75 + 0.01
0.65 = 0.02
Speckle noise
33
0.73 + 0.00
0.68 + 0.00
0.75 0.01
0.65 0.01
50
0.73 + 0.00
0.68 - 0.00
0.75 + 0.01
0.66 + 0.01
10
0.73 + 0.01
0.72 0.01
0.76 + 0.01
0.67 0.04
20
0.73 + 0.01
0.71
0.76 0.01
0.65 0.02
Zoom blur
33
0.73 + 0.00
0.72
0.77 0.01
0.66 0.02
50
0.73 + 0.00
0.72 0.00
0.77 + 0.01
0.67 = 0.01
Table 7: ResNet-34, error bar table, mismatch from different feature space corruption
Doctor
REL-U
Corruption
Split (%)
AUC
FPR
AUC
FPR
21

---

Published as a conference paper at ICLR 2024
10
0.91 0.00
0.30
000000000
0.33 =:3=0000
20
0.91 0.00
0.00
0.92 0.00
0.30
Brightness
33
0.91 = 0.00
0.30
0.92
0.30
50
0.92 + 0.00
+330+0001
0.92 0.00
0.31 0.01
10
0.66 + 0.03
0.76
0.70 + 0.02
0.68 + 0.03
20
0.66 0.02
0.71 0.01
0.67 0.02
Contrast
33
0.66 0.02
0.75
0.72 = 0.01
0.66 = 0.02
50
0.66 0.01
0.75
0.72 + 0.01
0.66 0.01
10
0.75 0.02
0.60 + 0.01
0.82 + 0.01
0.49 + 0.01
20
0.75 0.01
0.60
0.82 # 0.01
0.49 = 0.01
Defocus blur
33
0.76 0.76:000 0.01
0.00
0.82 0.00
0.50 = 0.01
50
0.76 0.01
0.00
0.82 0.00
0.50 0.01
0,81 = 0.02
0.01
0.84 0.01
0.45 0.01
20
0.81 + 0.01
0.01
0.85 + 0.00
0.44 0.01
Elastic transform
33
0.81 + 0.01
0.52
0.85 + 0.00
0.44 0.01
50
0.81 + 0.01
0.00
0.85 + 0.00
0.44 = 0.00
0.73 0.02
0.08
0.81 0.01
0.56 0.02
20
0.73 + 0.01
0.77
0.81 + 0.01
0.57 = 0.03
Fog
33
0.74 0.01
0.77 + 0.03
0.81 0.01
0.59 0.02
50
0.74
0.77
0.82 = 0.00
0.59 = 0.03
10
0.80 0.00
0.65 0.02
0.81 0.01
0.60 0.05
20
0,80 0.00
0.82 - 0.00
0.59 - 0.02
Frost
33
0.80 0.00
0.65 :
0.82 + 0.00
0.59 0.01
50
0.80 + 0.00
0.65 = 0.01
0.82 - 0.00
0.58 = 0.01
10
0.71 + 0.01
0.00
0.75 = 0.01
0.65 0.01
20
0.71 = 0.00
0.72 0.00
0.75 = 0.01
0.66 0.01
Gaussian blur
33
0.71 + 0.00
0.72 0.00
0.75 + 0.00
0.66 0.01
50
0.71 + 0.00
0.72
0.00
0.75 + 0.00
0.67 0.01
10
0.60 + 0.00
0.85 + 0.01
0.60 + 0.03
0.87 + 0.02
20
0.60 + 0.00
0.61 + 0.01
0.87 0.01
Gaussian noise
33
0.60 0.00
0.85
0.61 161 0.01
0.87 0.01
50
0.60 = 0.00
0.85 0.00
0.61 0.01
0.87 0.00
10
0.72 0.00
0.72 0.00
0.73 + 0.01
0.70 + 0.03
20
0.72 0.00
0.72
0.74 0.01
0.69 0.01
Glass blur
33
0.72 + 0.00
0.72
7.00
0.70 = 0.01
0.72 + 0.00
0.71+00 0.00
0.74 0.00
0.69 0.00
10
0.63 + 0.00
0.82 + 0.00
0.66 0.02
0.80 0.03
20
0.63 + 0.00
0.82 0.00
0.66 + 0.01
0.80 0.01
Impulse noise
33
0.63 + 0.00
0.82 + 0.00
0.66 + 0.01
0.80 0.01
50
0.63 + 0.00
0.82 0.00
0.67 + 0.01
0.80 0.00
0.81 + 0.01
0.57 + 0.02 0.02
0.82 + 0.01
0.51 0.03
20
0.81 + 0.01
0.56
0.83 + 0.00
0.50 - 0.01
Jpeg compression
33
0.81 = 0.00
0.07
0.83 0.00
0.51 0.01
50
0.81 + 0.00
0.50
0.83 0.00
0.51 0.01
0.78 0.01
0.59
0.83 + 0.01
0.47 0.01
20
0.78 0.01
0.01
0.84 + 0.01
0.47 - 0.01
Motion blur
33
0.78 0.01
0.01
0.84 0.00
0.48 0.01
50
0.78 + 0.00
0.57
0.84 0.00
0.48 + 0.01
10
0.73 0.00
+.00
0.73 + 0.02
0.69 0.04
20
0.73 0.00
0.70 0.01
174+0000
0.69 + 0.03
Pixelate
33
0.73 0.00
0.00
1.74+0001
0.69 0.02
50
0.73 + 0.00
0.70
0.74 0.01
0.68 -
10
0.90 + 0.00
0.31 + 0.01
0.90 + 0.01
0.32 + 0.08
Saturate
22

---

Published as a conference paper at ICLR 2024
20
0.90
0.00
0.91
0.30 0:30=000
33
0.90 0.90 0.00
0.01
0.91 0.00
0.30
50
0.90
0.91 0.30
0.29 0.01
0.63 0.00
0.86
0.65 0.03
0.86 0.04
20
0.63 + 0.00
0.85 0.01
0.65 + 0.01
0.86 = 0.01
Shot noise
33
0.63 0.00
0.86 0.00
0.65 0.01
0.86 0.02
50
0.63 + 0.00
0.86 = 0.00
0.65 0.01
0.86 = 0.00
10
0.84 + 0.00
0.55 0.03
0.85 + 0.01
0.49 0.03
20
0.84 + 0.00
0.53 0.02
0.85 + 0.00
0.48 + 0.02
Snow
33
0.84 + 0.00
0.05
0.85 $ 0.00
0.48 0.02
50
0.84 = 0.00
0.56
0.85 = 0.00
0.48 = 0.01
10
0,83 = 0.00
0.59 0.02
0.82 = 0.01
0.60 0.06
20
0,83 = 0.00
0.83 0.01
0.58 0.04
Spatter
33
0.83 + 0.00
0
0.83 + 0.01
0.58 0.02
50
0.83 + 0.00
0.59 0.00
0.83 + 0.00
0.58 0.01
10
0.68 + 0.00
0.81 = 0.01
0.70 + 0.03
0.79 = 0.06
20
0.68 0.00
0.81 = 0.01
0.70 0.01
0.78 0.03
Speckle noise
33
0.68 + 0.00
0.81 + 0.00
0.70 +.01
0.79 + 0.02
50
0.68 + 0.00
0.81 + 0.00
0.70 + 0.01
0.79 0.01
0.79 + 0.01
0.58 + 0.01
0.84 + 0.01
0.47 0.02
20
0.79 - 0.01
0.58 0.00
0.84 + 0.00
0.48 0.01
Zoom blur
33
0.79 + 0.01
0.58 0.00
0.84 + 0.00
50
0.79 0.00
0.58 + 0.00
0.84 + 0.00
0.49 0.01
0.53
0.68
50% TRR
0.52
0.850
0.870
0.51
Doctor
0.30
0.843
0.64
0.865
0.835
FRR
0.49
0.48
0.60
0.825
10
20
50
10
20
50
10
20
35
50
15
20
Spiit81
Spiiu(4)
Spi(1h)
(a) DenseNet- 121.
(b) ResNet-34.
Figure 7: SVHN versus MNIST mismatch analysis.
23


Reviews:
[{"review_id": "KpOx3Uyvc3", "reviewer": "Reviewer_9edT", "summary": "In this paper, the authors present a technique to learn their proposed uncertainty measure for misclassification detection. The learned uncertainty measure has a closed form solution which makes it easy to implement. The authors present a wide range of empirical results comparing against baselines trained with different loss functions, and looking at the ablation of various hyperparameters of their approach.", "questions": "- I think the paper would benefit from having an algorithm block just to clarify the exact learning procedure. For instance, information like the detector is tuned on a held-out set which the model wasn't trained out should be highlighted much clearly in the paper. \n\n- A high-level question here is: how is this different from some logit/feature transformation on held-out set? There has been some work in the literature (although around OOD detection) that looks at using latent representations from deep neural network backbone and training models that use them (see for e.g. [1]). I see that you have taken a slightly different approach here that looks at using closed form solution, but the idea seems very similar. Can you describe how your paper differentiates from this high-level approach of taking features from a deep net backbone and then applying some transformation to it to generate an uncertainty metric? \n\n- Building upon the prev point, would it make a lot of difference if you were to use a simple MLP architecture rather than the closed form solution you have right now to transform the soft outputs into an uncertainty measure that's being fine-tuned on held-out tuning set? The MLP can also be trained on held-out tuning set. \n\n- How do you think your approach compares with Bayesian neural networks which also introduce other forms of uncertainty such as model uncertainty? (see [2] for e.g.)\n\n\n\nReferences\n\n[1] Dinari, O. &amp; Freifeld, O.. (2022). Variational- and metric-based deep latent space for out-of-distribution detection. <i>Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence</i>, in <i>Proceedings of Machine Learning Research</i> 180:569-578 Available from https://proceedings.mlr.press/v180/dinari22a.html.\n\n[2] Vadera, M., Li, J., Cobb, A., Jalaian, B., Abdelzaher, T., & Marlin, B. (2022). URSABench: A system for comprehensive benchmarking of Bayesian deep neural network models and inference methods. Proceedings of Machine Learning and Systems, 4, 217-237.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is well motivated and looks at the useful problem of identifying potential errors that models can make to avoid downstream problems. \n\n- The paper provides a theoretically sound approach to learning the uncertainty metric. \n\n- The ablation study provided by the authors is important in understanding the approach better given the # of hyperparameters in the approach.", "weaknesses": "- I think the paper can improve on its presentation. Most importantly, the authors should provide a clear description of how the learn their metric function $s_d$ because that's the crux of their paper. \n\n- While the ablation results provided by the authors are useful, this approach still requires a lot of knobs to be tuned, especially if someone is interested in applying it to a completely different application domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iUPAIpY8L6", "reviewer": "Reviewer_Pj5j", "summary": "In the paper, the authors propose a novel approach to learn a measure of \"relative\" uncertainty based on data. \nBy utilising both positive (correctly classified) and negative (misclassified) instances, they construct a CxC matrix, which is needed for the proposed method.\nThis is an interesting approach, as typically to estimate uncertainty we use some ready-to-use formulas (like entropy or maxprob). And to the best of my knowledge, this is the first idea when misclassifications are directly used to derive a measure of uncertainty.", "questions": "Here I have not only questions, but also some comments.\n\n1. Choice of lambda: can we choose it based on the data? Like $\\lambda \\sim \\frac{1}{\\text{Npos.examples}}$\n2. In page 9: \"no class labels required\", but it is not really true. You need class labels to make positive/negative pairs. Saying this, I mean that we cannot use the approach in semi-supervised scenario, when objects without labels are available. We still need to know labels.\n3. The confusion matrix in Figure 1, as I understood, was computed using the validation data. Shouldn't it be more fair to build it on test set?\n4. In scenarios where there's a very good classifier and a high-quality dataset (like MNIST or CIFAR10, with low aleatoric uncertainty), there might be a significant imbalance between positive and negative samples. This could lead to the situation that the trained uncertainty measure is unreliable. How should one address this situation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and I find it easy to follow.\n2. The concept of learning a \"relative\" uncertainty measure from correctly and incorrectly classified data is innovative and novel.\n3. The authors have provided a comprehensive experimental evaluation.\n4. Authors provided code.", "weaknesses": "There are no major drawbacks that I see. But I think the following could improve the paper even further:\n1. A more detailed discussion / description of the baselines, such as Doctor, would be beneficial. The same applies to different objectives like Mixup, RegMixUp etc.\n2. I think it also should be mentioned that the paper is actually not the first one introducing a data-driven measure of uncertainty. There is a whole batch of works (see for example [1, 2, 3, 4]) that use data to derive estimates of epistemic or/and aleatoric uncertainties. I think it could help readers to correctly place the current paper in the right position in the broader research landscape.\n3. A discussion on how \"relative uncertainty\" compares to well-known epistemic and aleatoric uncertainties would be insightful.\n\n---\n\n[1] Kotelevskii N. et al. Nonparametric Uncertainty Quantification for Single Deterministic Neural Network //Advances in Neural Information Processing Systems. \n\n[2] Van Amersfoort J. et al. Uncertainty estimation using a single deep deterministic neural network //International conference on machine learning.\n\n[3] Liu J. et al. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness //Advances in Neural Information Processing Systems.\n\n[4] Mukhoti J. et al. Deep deterministic uncertainty: A simple baseline //arXiv preprint arXiv:2102.11582.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pTIyAF9FTK", "reviewer": "Reviewer_YaSF", "summary": "This paper proposes to evaluate how classification model predictions tend to be wrong without knowing the groundtruth label. It proposes to calculate a distance matrix of different classes, and use it to calculate a measure for the task. Empirical evaluations using different model structures and data are conducted to compare the proposed method with existing entropy measures.", "questions": "- How to prepare positive and negative data for calculating the distance matrix at the first place?\n- What does match and mismatch mean in experiments?\n- Why use auc instead of the metric mentioned in Granese et al. 2021?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper considers a practically important problem.\n- It proposes a principled method and shows how to calculate with clear formulas.\n- The proposed idea has a certain degree of originality.\n- Thorough experiments are conducted and the source code is provided for reproduction.", "weaknesses": "- The paper is overall not easy to follow. For example, the related work section comes at a random place. Also, I encourage authors to conduct wider and more structured literature reviews, to cover related topics such as deep model calibration and classification with rejection, and present in a more structured way.\n  - Please also give reivew on the measure of diversity investigated in Rao (1982), and how the proposed method is inspired. This accounts for the core contribution for the paper.\n- Terminology is not unifed. For example, \"soft-prediction\", \"the posterior distribution\", \"the predicted distrbution\" is used, which seem to mean one same thing.\n- Some words are used without clear definition or in a confusing way.\n  - \"feature\" is used in cases where it seems to mean a data instance, such as an image.\n  - lambda parameter for the proposed method is mentioned in experiments, but there is no definition.\n- Presentation overall needs to be polished. See questions below.\n  - At first, the presentation of \"positive\" and \"negative\" is confusing. I was even thinking about binary classification for a second.\n  - In contribution 2, \"closed-form solution for training REL-U\" is confusing. In my opinion, if it is closed-form, usually is not considered as a traning process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vZayiNYlPY", "reviewer": "Reviewer_tQ2Y", "summary": "The paper proposes a new measure to determine predictions’ uncertainty based on a distance between different labels that is found using additional data. Such distance can be found solving an optimization problem in closed form and the authors experimentally show the methods introduced obtain good results", "questions": "Can you articulate the meaning of the metric in (3) in more precise terms? refer the distance d as an observer is a bit strange, what does the data-based distance d(y,y') describe?\n\nIt is not correct to refer to p_x,y as “probability density function” since Y has a finite range of values", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The addressed topic is very relevant since the probabilistic assessment provided by supervised learning algorithms are often unreliable. The uncertainty metric introduced can be of interest and the fact that the distance between labels can be found in closed form is also valuable", "weaknesses": "In order to assess the paper contribution, the authors need to compare their methods with stronger techniques to assess predictive uncertainty, such as conformal prediction. For instance the authors could compare their method with a conformal predicition algorithm that instead of using data to obtain the uncertainty metric s_d, uses such data as calibration data to obtain prediction intervals. Such comparison is also important in that the assessment of prediction uncertainty in terms of prediction sets seems to be more useful in practice than that in terms of the metric s_d(x)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)