Effective Uncertainty Estimation with Evidential Models for Open-World Recognition
Authors: Charles Corbière
Marc Lafon
Nicolas THOME
Matthieu Cord
Patrick Perez
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
uncertainty estimation
evidential models
misclassification detection
out-of-distribution detection
confidence learning

Abstract: Reliable uncertainty estimation is crucial when deploying a classifier in the wild. In this paper, we tackle the challenge of jointly quantifying in-distribution and out-of-distribution (OOD) uncertainties. To this end, we leverage the second-order uncertainty representation provided by evidential models and we introduce KLoS, a Kullback–Leibler divergence criterion defined on the class-probability simplex. By keeping the full distributional information, KLoS captures class confusion and lack of evidence in a single score. A crucial property of KLoS is to be a class-wise divergence measure built from in-distribution samples and to not require OOD training data, in contrast to current second-order uncertainty measures. We further design an auxiliary neural network, KLoSNet, to learn a refined criterion directly aligned with the evidential training objective. In the realistic context where no OOD data is available during training, our experiments show that KLoSNet outperforms first-order and second-order uncertainty measures to simultaneously detect misclassifications and OOD samples. When training with OOD samples, we also observe that existing measures are brittle to the choice of the OOD dataset, whereas KLoS remains more robust.

Under review as a conference paper at ICLR 2022
EFFECTIVE UNCERTAINTY ESTIMATION WITH EVI- DENTIAL MODELS FOR OPEN- WORLD RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
Reliable uncertainty estimation is crucial when deploying classifier in the wild. In this paper, we tackle the challenge of jointly quantifying in-distribution and out-of-distribtioon (OOD) uncertainties. To this end, leverage the second- order uncertainty representation provided by evidential models and we introduce KLoS, a Kullback-Leibler divergence criterion defined on the class-brobayilityy simplex. By keeping the full distributional information, KLoS captures class con- fusion and lack of evidence in single score. A crucial property of KLoS is be class-wise divergence measure built from in-distribution samples and to not require OOD training data, in contrast to current second-order uncertainty mea- sures. We further design an auxiliary neural network. KLoSNet, to learn refined criterion directly aligned with the evidential training objective. In the realistic context where no OOD data is available during training, our experiments show that KLoSNet outperforms first-order and second-order uncertainty measures to simultaneously detect misclassifications and OOD samples. When training with OOD samples, we also observe that existing measures are brittle to the choice the OOD dataset, whereas KLoS remains more robust.
INTRODUCTION
Safety is a major concern in visual-recognition applications such as autonomous driving (McAI- lister et al.] 2017) and medical imaging (Heckerman et al., 1992). However, modern neural net- works (NNs) struggle to detect their own misclassification (Hendrycks & Gimpell 2017). In addi- tion, when exposed to outtof-distribution (OOD) samples, NNs have been shown to provide over- confident predictions instead of abstaining (Hein al. 2019; Nguyen et al., (2015). Obtaining reli- able estimates of the predictive uncertainty is thus necessary to safely deploy models in open-world
conditions (Bendale & Boult 2015).
Notable progress has been made in NN uncertainty estimation with the renewal of Bayesian neu- ral networks (Gal & Ghahramani 2016 Maddox et al.) (219)) and ensembling (Lakshminaray
et al. 2017: Ovadia et al., 2019b). These techniques produce a probability density over the predic- tive categorical distribution obtained from sampling. recent class of models, coined evidential (Malinin & Gales 2018 Sensoy et al. 2018), proposes instead to explicitly Iearn the concentration parameters of a Dirichlet distribution over first-order probabilities. They have been shown im- prove generalisation (Joo et al. 2020). OOD detection (Nandy et al. 2020) and adversarial attack
detection MMllnnin Gales] 2019).
Based on the subjective logic framework (Josang 2016), evidential models enrich uncertainty rep- resentation with evidence information and enable represent different sources of uncertainty, Con- flicting evidence, e.g., class confusion, is characterized by the expectation of the second-order Dirichlet distribution while the distribution spread on the simplex expresses the amount of evidence in prediction (Shi et al.) 2020). These sources of uncertainty also known as data and model uncertainty in the machine learning literature (Malinin Gales] 2019).
For open-world recognition, model should be equipped with uncertainty measure that accounts both for class confusion and lack of evidence to detect misclassifications and OOD samples. The pre- dictive entropy and the maximum class probability (MCP) targeting total uncertainty actually reduce probability distributions on the simplex to their expected value and compute first-order uncertainty measures Joo al. 2020 Sensoy al. 2018). This causes significant loss of information, as

---

Under review as a conference paper at ICLR 2022
Dog
Dog
ENN
ENIN
Cononiration
(a) In distribbution image
(b) Outlier with same class confusion
MCP = 0.50, entropy = 0.97, KLoS = 97.85
MCP = 0.50, entropy = 0.97, KLoS = 104.71
Figure 1: Limitations of first-order uncertainty measures and their handling with KLoS. (a) in-distribution image with conflicting evidence between dog and wolf. (b) An outlier with same class confusion but lower amount of evidence. An evidential neural network (ENN) outputs class-wise evidence information as concentration parameters of Dirichlet density (visualized on the simplex) over 3-class distributions. Although this density flatter for the second imput, the predictive entropy and MCP, only based first-order statistics, are equal for both inputs. In contrast, the proposed measure, KLoS, captures both class confusion and lack of evidence, hence correctly reflecting the larger uncertainty for the latter sample.
shown in Fig. T the resulting measures are invariant to the spread of the distribution, whereas un- certainty caused by class confusion and lack of evidence should be cumulative, property naturally fulfilled by the predictive variance in Bayesian regression (Murphy 2012).
Existing OOD detection methods with evidential models (Malinin & Gales 2018 2019, Nandy et al. (020 use second-order un- certainty measures assuming that the Dirichlet distribution spread is larger for OOD than for in-distribution (ID) samples, e.g., pre- cision (ep or mutual information. They also rely on using auxil- iary OOD data during training to enforce higher distribution spread on OOD inputs. However, this assumption is not always fulfilled in absence of OOD training data, noted by Charpentier et al. 2020;; Sensoy et al. (2020). As shown in Fig. 2 for a model trained on CIFAR-10. 10, au values largely overlap between IDs and OODs when no OOD training data is used, limiting the effective- ness of existing second-order uncertainty measures. Consequently, neither current first-order nor second-order uncertainty measures appear to be suited for open-world settings.
Invoistritutioon Cuilaffelistiilaiion
Danaity
241
241
240
541
340
2
60
Precesican
Figure 2: Precision densities for ID) (CIFAR-10) and OOD (TinyImageNet) samples when no OOD training data is used.
Contributions. In this paper, we introduce KLoS, a measure that accounts for both in-distribution and out-of-distribution sources of uncertainty and that is effective even without having access to auxiliary OOD data at train time. KLoS computes the Kullback-Leibe (KL) divergence between model's 's predicted Dirichlet distribution and specifically designed class-wise prototype Dirichlet distribution. By leveraging the second-order uncertainty representation that evidential models pro- vide, KLoS captures both class confusion and lack of evidence in a single score. Prototype distri- butions are designed with concentration parameters shared with in-distribution training data, which enables to detect OOD samples without assuming any restrictive behavior, c.g., having low preci- sion ag. KLoS naturally reflects the training objective used in evidential models and we propose to leam an auxiliary model, named KLoSNet, to regress the values of a refined objective for training samples and to improve uncertainty estimation. To assess the quality of uncertainty estimates in open-world recognition, we design the new task of simultaneous detection of misclassifications and OOD samples. Extensive experiments show the benefits of KLoSNet various image datasets and model architectures. In presence of OOD training data, we also found that our proposed measure is more robust to the choice of OOD samples while previous measures may perform poorly. Finally, we show that KLoS can be successfully combined with ensembling to improve performance.
CAPTURING IN-DISTRIBUTION AND OOD UNCERTAINTIES
Section 2.2 presents our measure to capture class confusion and lack of evidence with evidential models. We propose confidence learning approach to enhance in-distribution uncertainty estima- tion Sec Beforehand, we review evidential models and their learning in Section 2. to put in perspective the benefits of the proposed approach.
2

---

Under review as a conference paper at ICLR 2022
2.1 BACKGROUND: EVIDENTIAL NEURAL NETWORKS
Let us consider a training dataset D of N i.dd. samples æ with label E [[1, C| drawn from an un- known joint distribution p(x, y) Bayesian models and ensembling methods approximate the poste- rior predictive distribution P(y = cl=", D) by marginalizing over the network's parameters thanks sampling. But this comes the cost of multiple forward passes. Evidential Neural Networks (ENNs) propose instead to model explicitly the posterior distribution over categorical probabilities p( [e y) by variational Dirichlet distribution,
(a 0))
go π ax) Dir(Dii(araaa (T Dir(T)/a((,,,) 0))
II c=1
(2.
(1)
II ((a 0))
whose concentration parameters ax 0) exp f(x 0) are output by network f with parameters 0: I is the Gamma function and (rg (z 0) Σe (a 0). Precision og controls the sharpness of the density with more mass concentrating around the mean as ag grows. By conjugate property, the exp fe .0)
predictive distribution for new point x" is P(y c|æ² D) x (m|aa) which is the usual output of a network f with softmax activation.
Eiz exp fx(=" 0)
The concentration parameters evidence in each class. For instance, in Fig. Ta the a's output by the ENN indicates that the image almost equally likely to be classified wolf or as dog. More interestingly, also distinguishes this in-distribution images from the OOD sample in Fig. 1b via the total amount of evidence do-
be interpreted
pseudo-counts representing the amount of
Training Objective. The ENN training is formulated
variational approximation to minimize the KL divergence between go (T| and the true posterior distribution p( π x. x);)):
L (0:20 D) = E ((x (axx)))((x.))) KL (10 (T a)) p(Tr
x
Σ
- (v(a, -(o 0))) + (L((0(T))a) |l p(([)))
(3)
N
(ED
where ei is the digamma function. Following Joo et al. (2020), we use the non-informative uniform prior p( = -Dir( [1)), where 1 is the all-one vector, and weigh the KL divergence term with
Lear (0: D)
1
Σ
-Vb(ay vd(o) (DI((Di)())aa) Dir(#|11))
N
In particular, minimizing loss Eq. enforce training sample's precision og to remain close to C' + X-1 value (Malinin & Gales 2019
2.2 KULLBACK-LEIBLER DIVERGENCE MEASURE ON THE SIMPLEX
By explicitly learning distribution of the categorical probabilities æ, evidential models provide second-order uncertainty representation where the expectation of the Dirichlet distribution relates class confusion and its spread to the amount of evidence. While originally used measure the total uncertainty, the predictive entropy H H[ylæ, le, 0) and the maximum class probability MCP(s, 0) max. P(y c|a, 0) only account for the position on the simplex. These measures are invariant to the dispersion of the Dirichlet distribution that generates the categorical probabilities. This can be problematic, as illustrated in Fig. I To capture uncertainties due to class confusion and lack of evidence, an effective measure should account for the sharpness of the Dirichlet distribution and its location on the simplex.
We introduce a novel measure, named KLoS for "KL on Simplex", that computes the KL divergence between the model's output and a class-wise prototype Dirichlet distribution with concentrations 7 focused on the predicted class ÿ):
KLoS(x) A (Di 0)) Dir Dir(m(/1))
where (1x 0) exp (a. 0) model's output and 7g (1 .., 1, ,,...,1 1) the uniform concentration parameters except for the predicted class with 1+ -
For conciseness, denote ae (ar 0), Vo hereafter.
3

---

Under review as a conference paper at ICLR 2022
Dir(wirr)) Dir(xja(xx, B))
(a) Correct prediction
(b) Class confusion
(c) Lack of evidence
(d) Higher amount of evidence
Figure 3: KLoS and KLoS' on the probability simplex. Given the input sample, the blue region represents the distribution predicted by the evidential model and the orange region represents the prototype Dirichlet distribution with parameters 7a = (I, , I1,TT II...,1] focused on the pre- dicted class y. Illustration of the behavior of KLoS in absence of uncertainty (a), in of class confusion (b) and in case of a different amount of evidence, either lower (c) or higher (d).
The lower KLoS is, the more certain the prediction is. Correct predictions will have Dirich- let distributions similar to the prototype Dirichlet distribution 7 and will thus be associated with a low uncertainty score (Fig. Bh). Samples with high class confusion will present an ex- pected probability distributions closer simplex's center than the expected class-wise prototype K-1+7 K -)) resulting in a higher KLoS score (Fig. 3p). Similarly, KL.S also penalizes samples having a different precision ag than the precision og ++C-1 of the prototype Yij. Samples with smaller (Fig. 3b) and higher (Fig. B1l) amount of evidence than oj receive a larger KLoS score.
P -
Effective measure without OOD training data. Since in-distribution samples are enforced to have precision close to ag during training, the class-wise prototypes are fine estimates of the concentra- tion parameters of training data for each class. Hence, KLoS is divergence-basee metric, which only needs in-distribution data during training to compute its prototypes. This behavior is illustrated in Section [4. The proposed measure will be effective detect various types of OOD samples whose precision is far from az- In contrast, second-order uncertainty measures, e.g., mutual infor- mation, assume that OOD samples have smaller ag, a property difficult fulfill for models trained only with in-distribution samples (see Fig. 2 In Section 4.3 we explore more in-depth the impact of the choice of OOD training data on the actual ag values for OOD samples.
By approximating the digamma function vb (see Appendix B KLoS can also be decomposed as:
(LoS(x)) -r 1) log
C ag
(- 1)(
1
(L((Ir(T))a) |ir((r)))) (6)
200
20g
The first term is the standard log-lik ligeliicl and relates only to expected probabilities, hence to the class confusion. The ratio kg/ag makes invariant to any scaling of the concentration parameters vector a. The other terms take into account the spread of the distribution by measuring how close og is to (T+C-1), and measure the amount of evidence.
2.3 IMPROVING UNCERTAINTY ESTIMATION WITH CONFIDENCE LEARNING
When the model misclassifies an example, i.c., the predicted class differs from the ground truth y. KLoS measures the distance between the ENN's output and the wrongly estimated posterior p( |ac, g). This may result in an arbitrarily high confidence / low KL. divergence value. Measuring instead the distance to the true posterior distribution p( ( (z 3) (Fig. 4 would more likely yield a greater value, reflecting the fact that the classifier made an error. Thus, a better measure for misclassification detection would be:
Dirirty,)
Figure 4: KLoS*
KLoS* (x. y) A KL (Dir(T (xx, Dir((T/w)))
where Yy corresponds to the uniform concentrations except for the true class y with 7 = 1 1-1
Connecting KLoS" with Evidential Training Objective. Choosing such value for results in KLoS" matching the objective function in Eq. (4). This means that KLoS* explicitly minimized by the evidential model during training for in-distribution samples. By mimicking the evidential
4

---

Under review as a conference paper at ICLR 2022
training objective, we reflect the fact that the model is confident about its prediction if KLoS" is close to zero. addition, minimizing the KL divergence between the variational distribution qo |()) and the posterior p( a y) is equivalent to maximizing the evidence lower bound (ELBO). Hence, a small KLoS" value corresponds to a high ELBO, which coherent with the common assumption in variational inference that higher ELBO corresponds "better" models (Gal 2016).
Obviously, the true class of an output is not available when estimating confidence on test samples. We propose to learn KLoS* by introducing an auxiliary confidence neural network, KLoSNet, with parameters w, which outputs confidence prediction Ca w). KLoSNet consists of small decoder, composed of several dense layers attached the penultimate layer of the original classification network. During training, we seek such that C(z,w) w) is close to KLoS* (x, y), by minimizing
[xLoSNa(w>; D)
Σ ||C(x, - -L.Ss (a. y) y)
(..y))
KLoSNet can be further improved by endowing it with its own feature extractor. Initialized with the encoder of the classification network, which must remain untouched for not affecting its perfor- mance, the encoder of KLoSNet can be fine-tuned along with its regression head. This amounts to minimizing Eq. 8 w.r.t. to both sets of parameters.
The training set for confidence learning is the one used for classification training. In the experiments, we observe a slight performance drop when using a validation set instead. Indeed, when dealing with models with high predictive performance and small validation sets, end up with fewer misclassification examples than in the train set. At test time, we now directly use KLoSNet's 's scalar output (ae, w²) as our uncertainty estimate. As previously, the lower the output value, the more confident the prediction.
RELATED WORK
Misclassification Detection. Several works (Jiang al. 2018 Corbière al. 2019 Moon al., 2020) aim to improve the standard MCP baseline (Hendrycks. & Gimpel (017)) in misclassine misclassification detection with NNs. In particular, Corbière al. (2019) design an auxiliary NN to predict a con- fidence criterion on training points. We adapt their high-level idea for evidential models to tackle jointly the detection of misclassified in-distribution samples and of OOD samples. With evidential models, Shi et al. (2020) use dissonance in their active learning framework. In contrast to other second-uncertann uncertannye measures, dissonance relates to class confusion, but does not acknowledge for the amount of evidence.
Out-of-Distribute Detection. Bayesian neural networks (Neal 1996) and ensembling (Laksh- minarayanan et al. 2017) offer a principled approach for uncertainty estimation in which second- order measure can be derived by measuring the dispersion between individual probabilities vectors. But this comes the expense of an increased computational cost. Evidential models emulate semble of models using a single network, but usually require OOD samples during training (Malinin) Gales] 2018: 2019 Nandy al.] (2020), which may be unrealistic in many applications. KLoS alleviates this constraint and remains effective without OOD in training. Another range of methods proposes to improve OOD detection on any pre-trained model. ODIN (Liang al., 2018) mitigates over-confidence by post- post-processing logits with temperature scaling and by adding inverse adver- sarial perturbations. Lee et al. (2018) proposes confidence score based on the cass-conditional Mahalanobis distance, with the assumption of tied covariance. Although effective, both approaches need OOD data to tune hyperparameters, which might not generalize to other OOD datasets (Shafaei eet al.) 2019). Finally, Liu et al. (2020) interpret pre-trained NN as energy-basee model and compute the energy score to detect OOD samples. Interestingly, this score corresponds to the log precision log ag, which is similar the EPKL measure Malinin & Gales (2019) used in ENNs.
4 EXPERIMENTS
In this section, we assess our method against existing baseline uncertainty measures on synthetic data and conduct extensive experiments across various image datasets, architectures and settings.
Experimental Setup. Uncertainty measures are derived from an evidential model (Eq. (4) with
10-2 2 in (Joo et al. 2020 Malinin & Gales. 2019), except for second-order metrics where
5

---

Under review as a conference paper at ICLR 2022
(a) Toy dataset
(b) Com1/Er.
(c) Entropy
(d) Mut. Inf.
(c) Mahalanobis
(f) KLoS
Figure 5: Comparison of various uncertainty measures for given evidential elassifier toy dataset. (a) Training samples from 3 input Gaussian distributions with large overlap (hence class confusion) and OOD test samples (blue); (b) Correct (yellow) and erroneous (red) class predictions on in-domain test samples; (c-f) Visualisation of different uncertainty measures derived from the evidential model trained the toy dataset. Yellow (resp. purple) indicates high (resp. low) certainty.
we found that setting À = 10-3 improves performance. We rely on the learned classifier to train our auxiliary confidence model KLoSNet, using the same training set and following loss Eq. All details about architectures, training algorithms and datasets are available in Appendix C
Baselines. We evaluate our approach against: first-order uncertainty metrics (Maximum Class prob-
ability (MCP) and predictive entropy (Entropy)), second-ord metrics (mutual information (Mut. Inf.) and dissonance), post-training methods for OOD detection (ODIN and Mahalanobis) and for misclassification detection (ConfidNet). Except in Section 4.3] we consider setups where no OOD data is available for training. Consequently, the results reported for ODIN and Mahalanobis are obtained without adversarial perturbations, which is also the best configuration for the considered tasks. We indeed show Appendix that these perturbations degrade misclassification detection.
4.1 SYNTHETIC EXPERIMENT
We analyse the behavior of the KLoS measure and the limitations of existing first- and second-order uncertainty metrics on a 2D synthetic dataset composed of three Gaussian-distribtted classes with equidistant means and identical isotropic variance (Fig. 5). This constitutes scenario with high first-order uncertainty due to class overlap. OOD samples are drawn from a ring around the in- distribution dataset and are only used for evaluation. Fig. 5c shows that Entropy correctly assigns large uncertainty along decision boundaries, which is convenient to detect misclassifications but yields low uncertainty for points far from the distribution. Mut. Inf. (Fig.5dl have the opposite behavior than desired by decreasing when moving away from the training data. This due to the linear nature of the toy dataset where models assign higher concentration parameters far from decision boundaries, hence smaller spread on the simplex, as also noted in (Charpentier et al. 2020). Additionally, Mut. Inf. does not reflect the uncertainty caused by class confusion along decision boundaries. Neither Entropy nor Mut. Inf. is suitable to detect OOD samples in this synthetic experiment. In contrast, KLoS allows discriminating both misclassifications and OOD samples from correct predictions as uncertainty increases far from in-distribution samples for each class (Fig. 51) KLoS measures a distance between the model's output and class-wise prototype distribution. Here, can observe that it acts divergence-basse measure for each class.
We extend the comparison to include Mahalanobis (Fig. 5e which a distance-based measure by assuming Gaussian class conditionals on latent representations, here in the input space. However, Mahalanobis does not discriminate points close to the decision boundaries from points with a similar distance to the origin. Hence, it may be less suited to detect misclassifications than KLoS. Addi- tionally, KLoS does not assume Gaussian distributions in the latent space nor tied covariance, which may be strong assumption when dealing with high-dimension latent space. In Appendix [D.]] complementary quantitative evaluation on this toy problem confirms our findings regarding the inad- equacy of first-order uncertainty measures such MCP and Entropy, and the improvement provided by KLoS over Mahalanobis on misclassification detection.
4.2 COMPARATIVE EXPERIMENTS
The task of detecting both in-distribution misclassifications and OOD samples gives the opportunity to jointly evaluate di-dstiiibutinn and outtof-distribution uncertainty representations of a method. In this binary classification problem, correct predictions are considered positive samples while

---

Under review as a conference paper at ICLR 2022
Table 1: Comparative experiments on CIFAR-10 and CIFAR-100. Misclassification (Mis.), out- of-distribution (OOD) and simultaneous (Mis+OOD) detection results (mean % AUROC and std. over runs). Bold type indicates significantly best performance (p<0..5) according to paired t-test.
LSUN
TinyImageNe
STL.-10
Method
Mis.
OOD
Mis+OOD
OOD
Mis+OOD
OOD
Mis+OOD
MCP
87.6 +1.6
79.7 +1.1
84.9 +1.1
80.3 +1.5
85.2 1.5
60.3 +1.2
75.2 1.4
Entropy
83.5 42.4
83.8 +0.3
87.9
+0.2
82.3 40.4
87.2 =0.4
60.1 +1.2
75.0 114
CIEAR-10
VGG-16 16
ConfidNet
Dissonance
90.2 40.8
91.9 +0.2
82.1 +1.5
84.8 +0.3
87.6 +1.1
90.1 10.1
83.5 4.0.6
84.2 4.0.
89.7 #0.1
88.3 +0.7
61.5 +1.6
64.1 +0.1
79.6 =0.1
77.2 #1.1
Mut. Inf.
81.1
84.6 10.6
+1.0
80.6 10.8
83.4
61.3 40.8
65.0 =2.5
Diss. +ur. Inf.
92.0
86.5 +0.3
89.8 +0.2
83.6 +0.33
89.5 +03
63.6 +0.5
79.4 +0.4
ODIN
86.0 +2.0
79.5 #1.2
83.8 1155
79.6 +1.9
84.0 #2.0
54.7 +1.5
65.0 #2.6
Mahalanobis
91.3
88.9 +002
91.3 #0.1
86.4 +:0.2
90.2 #0.1
63.4 +0.2
78.8 +0.3
KLoSNet (Ours)
92.5 10..6
87.6 +0.9
91.7 #0.9
86.6 1:0.9
91.2 00.8
114
=0.9
MCP
84,9 10.8
79.6 #1.0
83.0 10.9
77.2 0.0.
81.8 +0.7
58.5 11.2
72.5 =0.4
Entropy
84.6 40.8
79.6 #1.1
82.8 +0.9
77.2 0.07
81.6 +0.7
58.4
72.2 20.4
CIFAR-10
ResNet-13 18
ConfidNet
90.7 +0.4
84.6 +1.1
88.6 +0.6
83.5 +1.1
88.0 +0.6
63.2 +1.2
77.9 00.5
Dissonance
92.9
90.3 +0.4
92.7 10.4
87.7 40.3
91.4
67.3 40.5
81.2 #0.4
Mut. Inf
80.6
:1.0
79.4
74.3 40.8
78.0
56.4
#0.2
Mist. Inf.
92.4
86.7 #1.0
90.1
84.3
80.6
65.2 +0.7
80.3 #0.4
ODIN
83.7
78.9
81.9 +0.9
76.5 40.7
+0.7
57.9 +1.2
71.5
Mahalanobis
:0.4
90.7 10.4
91.8
87.6 10.4
90.3 =0.4
66.8 10.5
80.0 =0.3
KLoSNet (Ours)
93.9 +:044
93.1 #1.1
94.4 +0.3
90.6 +:06
93.2 +002
68.5 +0.3
82.3 0.22
MCP
82.9
62.8 +1.3
77.6 +0.9
72.0 +0.5
81.8 -0.7
69.7 +0.7
80.9 =0.7
Entropy
82.2
63.2
77.2
72.5 4006
81.5 +0.8
70.1 +0.8
80.6
ConfidNet
84.4
65.3 +2.0
80.0 11.
73.8 1006
83.7
71.5 10.6
82.7 20.3
CIFAR-100
VGG-16
Dissonance
80.1
#1.4
78.7 10.8
70.3 +0.4
82.5
69.3 +0.4
82.2 20.4
Mut. Inf.
78.9 +:0.8
65.6 +0.7
76,2 +:0.9
71.8 4:0.2
79.1 +0044
70.1 +006
78.5 0.6
Diss.+Mur. Inf.
84.2 40.6
65.1 +0.3
80.1 +0.4
70.1 +.0.3
82.5 +0.5
69.5 +0.3
82.3 0.5
ODIN
82.1 #0.8
62.9 #14
77.1 110
71.9 1006
81.3 =0.8
69.6 +0.8
80.3 =0.7
Mahalanobis
84.0 +0.2
71.1 +1.0
82,4 +0.5
77.0 +:0.5
84.9 +0.3
75.4 +0.3
84.3 00.5
KLoSNet (Ours)
86.7 +:0.4
68.4 #1.1
83.0 +006
76.4 4:0.4
86.4 #0.4
75.0 +0.5
86.0 0.4
MCP
84.0 +:044
70.4 +0.9
81.0 +0.3
76.6 +.0.5
83.6 +00.4
75.4 +0.5
83.1 #.22
Entropy
83.7 +0.4
70.4 +0.9
80.8 +0.3
76.9 +0.5
83.5 +033
75.7 +0.5
83.0 =0.3
CIFAR- 100
ResNec-11 18
ConfidNet
87.1 +002
73.0 +14
84.5 -0.6
79.1 40.3
86.8 -0.3
78.5 +0.8
86.6 =0.5
Dissonance
86.7 +0.4
72.3 +0.4
84.0 +0.2
75.0 +0.4
85.3 +0.4
74.7 +0.3
85.2 +0.2
Mut. Inf
40.4
70,2 #11
80.0 10.4
76.4 10.6
82.6 -0.3
75.1 40.5
82.1 =0.3
Diss. Mut. Inf.
86.5 40.4
71.8 10.8
83.6 10.5
76.1 40.3
84.7 20.4
75.2 40.5
84.6 =0.3
ODIN
83.7 10.1
70,3 10.9
80.8 10.3
76.6 40.5
20.3
75.4 40.5
83.0 =0.3
Mahalanobis
85.9 +-0.4
75.2 10.6
84.5 +0.1
78.4 +0.55
85.9 +0.3
77.5 +0.4
85.6 +0.3
KLoSNet (Ours)
86.9 10.3
73.1 40.4
84.4 10.1
80.8 10.2
87.3 -.2
79.0
40.2
86.7 20.3
misclassified inputs and OOD examples constitute negative samples. Following standard practices (Hendrycks & Gimpel 2017), use the area under the ROC curve (AUROC) to evaluate threshold- independent performance. Results for other relevant metrics are available in Appendix D
The models used in the experiments present high predictive performances. Most often, there much fewer misclassifications the test set than considered OOD samples. Hence, joint detection performances might be dominated by the evaluation of the quality of OOD detection. To mitigate this unbalance, we propose to consider the following scheme based on oversampling. Let AM be the subset of in-distribution test examples that are misclassified by the observed model and Ao the set of OOD test samples. We randomly sample x| k|AAol points in AM. with == 1. Supposing |Aol 2 |ANil this corresponds oversampling the set of misclassifications. This samplsd to the OOD set to form the negative examples for detection training. The set of correct predictions remains the same. We observed that the variance in AUROC due this sampling is negligible and report only the mean hereafter.
is then added
Experiments are conducted with VGG-16 (Simonyan & Zisserman 2015) and ResNet-18 (He et al.) 2016) architectures on CIFAR-I0 and CIFAR- datasets (Krizhevsky 2009). The OOD datasets used for evaluation are LSUN Yu et al., 2015). TinylmageNet and STL-10 (Coates et al. 2011). Along with simultaneous detection results, provide separate results for misclassification detec- tion and OOD detection respectively in Table | On OOD detection, Mahalanobis and KLoSNet outperform other methods, including second-order measures. ODIN also fails deliver here as

---

Under review as a conference paper at ICLR 2022
Table 2: Impact of confidence learning. Comparison of detection performances between KLoS and KLoSNet for CIFAR-10 and CIFAR-100 experiments with VGG-16 architecture.
LSUN
ImagemageNet Mis+OOD
STL-10
Method
Mis.
0OD
Mis+OOD
0OD
0OD
Mis+OOD
CIFAR-10
KLeS
92.1 +0.3
86.5 +0.3
91.2 +0.2
85,4 =0.3
90.4 #0.2
64.1 10.3
79.6 10.3
VGG-16
KLoSNet
92.5 +0.6
87.6 +0.9
91.7 +0.9
86.6 =0.9
91.2 +0.8
67.7 #1.4
81.8 40.9
CIFAR-I00 VGG-16
KLoS
85.4 +0.2
65.1 +111
81.3 +0.6
74.5 =0.4
85.4 #0.4
72.7 #0.3
84.8 +0.4
KLoSNet
86.7 #0.4
68.4 +1.1
83.0 -0.6
76.4 =0.4
86.4 #0.4
75.0 10.5
86.0 10.4
(a) SVHN
(b) LSUN
(c) CIFAR-100
SUNO
Ogns
Figure 6: Effect of OOD training data on precision Crg. Den- sity plots for CIFAR- CUFAR-I0/Tiyylmai 10miiiimaeeeeet benchmark: (a.b) with in- appropriate OOD samples (SVHN, LSUN); (c) with close OOD samples (CIFAR-100).
Figure 7: Impact of the over- sampling factor (CIFAR- 10/TinylmaeNNet.
logits small due regularization in the evidential training objective. Mut. Inf. and other spread- based second-ordes uncertainty measures (see Appendix (D2) fall short to detect correctly OOD. Indeed, for settings where OOD training data is not available, there is guarantee that every OOD sample will result in lower predicted concentration parameters as previously shown by the density plot of precision ogg in Fig. 2 This stresses the importance of class-wise divergence-basee measure. While Mahalanobis may sometimes be slightly better than KLoSNet for OOD detection, it performs significantly less well in misclassification detection, in line with the behavior shown in synthetic experiments. As result, KLoSNet appears to be the best measure in every simultaneous detection benchmark. For instance, for CIFAR- with VGG-16, KLoSNet achieves 81.8% AUROC while the second best, Mahalanobis, scores 78.8%. We also observe that KLoSNet improves sig- nificantly misclassification detection, even compared to dedicated methods such as ConfidNet or second-order measures related class confusion, e.g., dissonance. Another baseline could be to combine two measures specialized respectively for class confusion and lack of evidence, such as Dissonance+Mut.IN Inf. But it still performs less well than KLoSNet. In Appendix E evaluate our approach on the task of selective classification when inputs are subject to corruptions (which can be seen as OOD samples close to the input distribution, hence particularly challenging). and observe similar results.
Impact of Confidence Learning. To evaluate the effect of the uncertainty measure KLoS and of the auxiliary confidence network KLoSNet, we report detailed ablation study in Table 2 We can notice that KLoSNet improves misclassification over KLoS but also OOD detection in every benchmark. We intuit that learning to improve misclassification detection also helps to spot some OOD inputs that share similar characteristics.
Oversampling Factor. When deploying model in the wild, it is difficult to know beforehand the proportions of misclassifications and OOD samples will have to handle. We vary the oversampling factor in [0.01; 100] for CIFAR 10 10/TnylmaagNNe in Fig. to assess the robustness of our approach. KLoSNet consistently outperforms all other measures, with larger gain when increases.
Combining KLoS with Ensembling. Aggregating predictions from an ensemble of neural net- works not only improves generalization Lakshminarayanan et al., 2017 Rame Cord 2021] but also uncertainty estimation (Ovadia et al. (019a). We train an ensemble of ten evidential models on CIFAR-10 10 and evaluate the performance of KLoS obtained from averaged concentration param- eters. On CIFAR- MARR--O/Tiin I0/TinylmageNe benchmark, misclassification and OOD detection performances are improved by respectively +1.8 points and +1.9 points, resulting in a +1.1 points gain on the joint detection task.
8

---

Under review as a conference paper at ICLR 2022
2 1 S
AU AU-RDC (88)
A1 RDC (%)
AU-ROC
68
8
1D0 LSUN
CIARR110
LSUN
(a) Mis. detection
(b) OOD detection
(c) Mis.+OOD detection
Comparative detection results with different OOD training datasets. While using OOD samples training improves performance in general, the gain value varies widely, sometimes even being negative for inappropriate OOD samples e.g., SVHN. KLoS remains the best measure in every setting. Experiment with VGG- architecture on CIFAR-10 dataset.
Figure 8:
4.3 EFFECT OF TRAINING WITH OUT-OR-DISTRIBUTTI SAMPLES
Previous results demonstrate that simultaneous detection of misclassifications and OOD samples can be significantly improved by KLoSNet. We now investigate settings where OOD samples are available. We train evidential model to minimize the reverse KL-divergence (Malinin & Gales) 2019) between the model output and sharp Dirichlet distribution focused on the predictive class for in-distribution samples, and between the model output and uniform Dirichlet distribution for OOD samples. This loss induces low concentration parameters for OOD data and improves second-order uncertainty measures such Mut. Inf
The literature on evidential models only deals with an OOD training set somewhat related to the in- distribution dataset, e.g. CIFAR-100 for models trained on CIFAR-10. 10. Despite semantic differences, CIFAR-10 and CIFAR- 100 images were collected the same way. which might explain the general- isation other OOD samples in evaluation. Contrarily, CIFAR-10 objects and SVHN street-view numbers differ more for instance. In Fig. [] we vary the OOD training set and compare the uncer- tainty metries taken from the resulting models. As expected, using CIFAR-100
training OOD data improves performance for every measure (MCP, Mut. Inf. and KLoS). However, the boost provided by training with OOD samples depends highly on the chosen dataset: The performance of Mut. Inf. decreases from 92.6% 5.6% AUC with CIFAR- 100 to 82.9% when switching to LSUN, and even becomes worse with SVHN (78.5% compared using no OOD data (80.6%). Indeed, Fig. Glshows that only the CIFAR- 100 dataset seems to be effective to enforce low oo unseen OOD samples.
We also note that KLoS outperforms or is on par with MCP and Mut. Inf. every setting. These results confirm the adequateness of KLoS for simultaneous detection and extend our findings to settings where OOD data is available at train time. Most importantly, using KLoS on models with- out OOD training data yields better detection performance than other measures taken from models trained with inappropriate OOD samples, here being every OOD dataset other than CIFAR-100.
5 DISCUSSION
We propose KLoSNet, an auxiliary model to estimate the uncertainty of classifier for both in- domain and out-of-domain inputs. Based on evidential models, KLoSNet is trained predict the KLoS* value of prediction. By design, KLoSNet encompasses both class confusion and evidence information, which necessary for open-wort recognition. Our experiments extensively demon- strate its effectiveness across various architectures, datasets and configurations, and reveal its class- wise divergence-baser behavior. Many real-world applications, e.g., based on active learning or domain adaptation, necessitate to correctly detect both sources of uncertainty. It will be interesting to apply KLoSNet in these contexts. We also show that, far from being the panacea, using training OOD samples depends critically on the choice of these samples for existing uncertainty measures. KLoS, on the other hand, is more robust to this choice and can alleviate their use altogether. While finding suitable OOD samples may be easy for some academic datasets. it may turn problem- atic in real-worth applications, with the risk degrading performance with an inappropriate choice. How build suitable OOD training sets is important open problem to attack.

---

Under review as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
To better understand the link between the evidential training objective and KLoS* criterion stated in Appendix A.2 provide a detailed derivation in Appendix A We also elaborate on the the- oretical decomposition of KLoS (Eq. (6) in Appendix [ Regarding experiments reproducibility, Appendix describes extensively how we generate data in the synthetic experiment (Section 441 the image classification datasets used to evaluate our approach including in-distribution and OOD datasets, the implementation and the hyperparameters the experiments shown in Section [.2 In particular, we took care in our experiments to perform multiple runs and present mean and std re- sults. Giving access to the code has yet to authorized internally, but we plan to release the code in the future.
REFERENCES
Abhijit Bendale and Terrance Boult. Towards open world recognition. In IEEE/CVE Conference Computer Vision and Pattern Recognition, 2015. [
Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. Posterior network: Uncertainty esti-
mation without ood samples density- -based pseudo-counts. In Advances Neural Information
Processing Systems, 2020. 6 16
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2011.
Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. In Advances in Neural Information Processing Systems, 2019. 5
Ran El- Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 2010. 21
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
S
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model un- certainty in deep learning. In Proceedings of the International Conference Machine Learning,
2016. 1
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
David Heckerman, Eric Horvitz, and Bharat Nathwani. Toward normative expert systems: Part i the pathfinder project. Methods of information in medicine, 31, 07 1992. doi: 10.1055/
s- s-0038-11 1634867. [
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In Proceedings of International Conference on Learning Representa- tions, 2019. 21
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and outtof-distribttion examples in neural networks. In Proceedings of International Conference Learning Represen- tations, 2017. 7
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust classifier. In Advances in Neural Information Processing Systems, 2018.
Taejong Joo, Uijung Chung, and Min-Gwan Sco. Being bayesian about categorical probability, In Proceedings of the International Conference Machine Learning, 2020.
13
10

---

Under review as a conference paper at ICLR 2022
Audun Josang, Subjective Logic: A Formalism for Reasoning Under Uncertainty. Springer, 2016.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A bench- mark of in-the-wild distribution shifts. arxiv, 2020. 21
A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
16
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.
5
S
Kimin Lee, Kihok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Process- ing Systems, 2018. 19
Shiyu Liang, Yixuan Li. and R. Srikant. Enhancing the reliability of out- out-of-distributiin image detec- tion neural networks. In Proceedings of International Conference on Learning Representations, 2018.
5 19
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detec- tion. In Advances Neural Information Processing Systems, 2020.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems 32, 2019. [
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, 2018.
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved uncer- tainty and adversarial robustness. In Advances in Neural Information Processing Systems, 2019.
5 16
Rowan McAllister, Yarin Gal, Alex Kendall, Mark van der Wilk, Amar Shah, Roberto Cipolla, and Adrian Weller. Concrete problems for autonomous vehicle safety: Advantages of Bayesian deep learning. In Proceedings of the International Joint Conference on Artificial Intelligence, 2017.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. In Proceedings of the International Conference on Machine Learning, 2020. 5
Kevin P. Murphy, Machine learning A Probabilistic Perspective. MIT Press, 2012.
Jay Nandy, Wynne Hsu, and Mong-Li Li Lee. Towards maximizing the representation gap between in- domain and out-of-distribution examples. In Advances in Neural Information Processing Systems, 2020. 2 16
Radford M. Neal. Bayesian Learning for Neural Networks. Peringer-Verlage 1996.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
16 18
A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In IEEF/CVF Conference on Computer Vision and Patlern Recognition, 2015. T
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019a. g
11

---

Under review as a conference paper at ICLR 2022
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019. T6
Alexandre Rame and Matthieu Cord. Dice: Diversity in deep ensembles via conditional redundancy adversarial estimation. In International Conference on Learning Representations, 2021.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshmi- narayanan. A simple fix to mahalanobis distance for improving near-ood detection, 2021. 19
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classifica- tion uncertainty. In Advances in Neural Information Processing Systems, 2018.
Murat Sensoy, Lance Kaplan, Federico Cerutti, and Maryam Saleki. Uncertainty-awarc deep classi- fiers using generative models. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Alireza Shafaei, Mark Schmidt, and James J. Little. A less biased evaluation of out-of-distribution sample detectors. In BMVC, 2019. 5
W. Shi, Xujiang Zhao, Feng Chen, and Qi Yu. Multifaceted uncertainty estimation for label-efficient deep learning. Advances in Neural Information Processing Systems, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations, 2015.
16
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 17X3v:1506.033655, 2015. 16
12

---

Under review as a conference paper at ICLR 2022
A LINK BETWEEN KLOS AND EVIDENTIAL TRAINING OBJECTIVE
In this section, we detail the connection between KLoS* and the evidential training objective. For conciseness, we denote ag a 0), Vc [[0, C] hereafter.
A.1 DERIVING THE EVIDENTIAL TRAINING OBJECTIVE
We first show the equivalence between Eq. and Eq. (4. Starting from Eq. (2)
ya (0; D)
(z (lk.y))p((xx))
[KL π p (az v))] (0(TT) ax)
(9)
/
Σ
90
(x) log
(10)
N
p(T ax, y)
(x
[
10 ( (()()) 40 (T] log
Σ
(11)
N
p(y , x p))p((
(m.y)ED
loylppp//......... x)) KL (10((T(()) (T/22)
(((((())))
log 1ogp(v))))
(
(12)
where N card(D). As the log-likelihood log p(y|æ) does not depend on parameters 0
min Lvar (0; = min
Σ
[an(mlan log x)] + (Lo || p( [z)) (13)
0
N
(w.))E
For sample (xx, y). the reverse cross-entropy term amounts to Em log p(y , x)] -vv(ay)) -((a)) where & is the digamma function. Hence, we recover (q. of the main paper:
min Cvar (0: D)
Σ -(W(a)))v(aa)) (g p(T[))). (0,
(14)
Considering that most of the training inputs x associated with only observation y in D, should choose small concentrations parameters & for the prior p( xx) = Dir( (((1)) to prevent the resulting posterior distribution p( (x) Dir(m ( Sy + L..., 3c) from being dominated by the prior. However, this causes exploding gradients in the small-value regimes due to the digamma function, e.g. v0(00) 10-4.
To stabilize the optimization, we follow Joo et al. (2020) and use the non-informative uniform prior distribution p( (< (a) D1I( where 1 is the all-one vector, and weight the KL-divergence
term with X > 0:
(0; = N
Σ (V((a,) - (b(o)) + AKL(Dir(T|a(a,,)) 0)) || Dir(=|1)). (15)
(x.y)ED
While L 0 D) slightly differs from Lvar (0: D), both functions lead to the same optima. Indeed, by considering their gradient, we can show that a local optimum of (0; D) is achieved for a sample when a = ($1...., 1,... Bc) 1/X,..,, 1). Hence, their respective ratio between each element is equal:
11......e,
and local optimum of (0; D) is a (1, 1+
of
a
(16)
A.2 LINK WITH K
Let us remind the definition of KLoS* as KL. divergence between the model's output and class- wise prototype Dirichlet distribution with concentrations 7j focused on the true class y::
KLoS* (xx, u) A (Dir( ax (D),( 0)) Dir (ir(T
(17)
13

---

Under review as a conference paper at ICLR 2022
where a( 0) exp faz 0) is the model's output and Yy (I,..., 1) the vector of uniform concentration parameters except for the true class with = 1/X+1
The KL divergence between two Dirichlet distributions can be obtained in closed form and KLoS* can be calculated as:
KLoS* (a: v) = KI. (Dr((TT))))) | Dir(TT//))
(18)
= log logoT((e [(o)) - log T(C - tog ((1+1/A)
Σ log ((ao)
+ ((a--))(((a))
- w(oo)) + (ay- (1+ 1/ (v(a,) - w(oo)) (19)
GF
On the other hand, the Kl.-divergence between the model's output and an uniform Dirichler distri- bution Di)( reads:
(Dr((((T)))) || Dir(T(/1) = log lloo) ((C) E log [(.))+ (oc-1)(((aa)
(20)
Hence, KLoS* can be written as:
1 KLoS (x. v) (w(ag) (((o)) + (I((Di((T)))) || Dir(a(1))
+ (log [11 1/A) log [(C + 1/A) log [(C)).
(21)
Let us decompose a N EEEDEE L xx y. 0) We retrieve that KLoS* (æ) & la (a, y, 0) where log 1og 1o on the model parameters
log I'(C)) does not depend
In summary, minimizing the evidential training objective C (0; D) is equivalent to minimizing the KLoS value each training point æ.
B DECOMPOSITION OF KLOS
This section provides details of the derivation of Eq. 66) Let us remind the definition of KLoS as KI. divergence between the model's output and class-wise prototype Dirichlet distribution with concentrations Yy focused on the predicted class ÿ.
(L) & (L((Dr(TT)(( 0)) Dir(T(?s))
(22)
where axx 0) cxp (x, 0) the model's output and Y/ (1,..., T, T,... 1) the vector of uniform concentration parameters except for the predicted class with r > 1. For conciseness, we denote a, (x, 0), Ve E [0. C| hereafter.
Similar to the derivation done in Appendix A.2 KLoS can be derived as: KLoS(x)) = (7 (bb((a)) v((a)) (I.(Dir(TT)))) Dir(x 1)),
(23)
where 1 is the all-one vector.
By considering the asymptotic series approximation to the digamma function, vb(x) log x O 22 ), the previous expression can be approximated by:
2z +
KLoS(z) - 1) log(
Ogg Ok
+ (-
1 1)) )) + (L(((I())) Dir(T(|1)) (24)
200 20g
To gain further intuition about the decomposition, we provide illustrations of the first term (negative ligeliiel,hood, NLL)
Method
Mis. (t)
OOD (1)
and the second term in Fig. 9 similar to those shown in Fig. of the paper. We also present quantitative results Table We observe that the NLL term, which is equivalent MCP
NL. term
80.2 +1.1
15,9 +0.7
Second term
48.2 4-4.3
98.4 40.1
KLoS
79.4 41.2
98.8 40.3
Table 3: Results of KLoS decompo- sition on synth. data.
14

---

Under review as a conference paper at ICLR 2022
measure, helps to detect misclassifications while the second term denotes increasing uncertainty as we move either far away from training data. Hence, by using either the NLL term or the second term, one could distinguish the source uncertainty if needed.
In partieular, the following lemma.
show that the second term reaches its minimum for og C thanks to
(a) KLoS
(b) NLL term
(c) Second term
Figure 9: Visualisation of the different terms of KLoS, derived from the evidential model trained on the same toy dataset than in the main paper.
Lemma B.1. local minimum of g ag + 11) 2ax 2ag)) KT.(Dir(TT)a) (ir(=(])) is reached for 0o T C
Proof. Let us denote la TRC such as Ve, ac = Me Og. between two Dirichlet distributions, g reads:
By the definition of the KI, divergence
C +
9: 7 11 1
1
Σ c=1
+ log Log [(av)
log (() (4.00 (1 ((oo))
2pg ao
c=1
(25)
By taking the derivative:
g(0o) (r 1--11 -1)
1&g
+ +(ao) Σ HcV(Haao) + - -
doro
2 og
(4:00
- (ao)). (26)
where (x)) simplifies to:
1 (x) is the trigamma function. As Ee He = 1, the previous equation
g(00) = 11 11 don
d
1 1- Pg 2pg 0
[m.00 1) ((oo)(()()oo)) ((ll)))).
(27)
We consider the asymptotic series approximation to the trigamma function (x)) ( Given this approximation, the derivative reads:
1
x
g(ao) No (T- 1 Hg 1
+
Σ
(p - 1)(He( +
(28)
doo
244 aj
16000
2 2
20
1=1
(T 1) 1 - 1bg
1
(M00000 -1) 1- Me 1
+
(29)
211g
211e
og
The derivative vanishes when Vc # 9. 0:000 = and Mga0 = т. hence a local minimum of g is reached for og 1.
15

---

Under review as a conference paper at ICLR 2022
C EXPERIMENTAL SETUP
In this section, we provide comprehensive details about the datasets, the implementation and the hyperparameters of the experiments shown in Section 4 of the main paper.
Synthetic Data. The training dataset (Fig. 3 of the paper) consists of 1,000 samples (ær, v) from distribution p(x, y) over TR22 21, defined as:
1 N(x 2: o 12222 p(x y y) 3
(30)
where p41 (0. v3/2), 12 = (-1.-v3/2) Ms = (1, (1,-13/22 and a = 4. The marginal distribu- tion of x is Gaussian mixture with three equally weighted components having equidistant centers and equal spherical covariance matrices. The test dataset consists of 1,000 other samples from this distribution. Finally, we construct an out-of-distribution (OOD) dataset following Malinin & Gales) (2019), by sampling 100 points in R2 such that they form 'ring' with large noise around the train- ing points. Some OOD samples will be close to the in ii-distribution while others will be very far (see Fig. 3 of the paper). The number of OOD samples has been chosen so that it amounts approximately to the number of test points misclassified by the classifier. The classification is performed by a simple logistic regression.
A set of five models is trained for 200 epochs using the evidential training objective (Eq, 6 of the paper) with regularization parameter À 5e-2 and Adam optimizer with learning rate 0.02. Un- certainty metrics MCP, Entropy, Mut. Inf., Malahanobis and KLoS are computed from these models.
Image Classification Datasets. In Sections 4.2 and 4.3 of the main paper, the experiments are conducted using CIFAR-I0 and CIFAR-I00 datasets Krizhevsky (2009). They consist 32 32 natural images featuring 10 object classes for CIFAR-IO and 100 classes for CIFAR-100. Both datasets are composed with 50,000 training samples and 10,000 test samples. We further randomly split the training set to create validation set of 10,000 images.
OOD datasets are TinyImageNe LSUN Yu et al. (2015) a scene classification dataset (10,000 test images of 10 scenes) -, STL-110 dataset similar to CIFAR-I0 but with different classes, and SVHN SVHN[Netzer et al. (2011) an RGB dataset of 28 28 house-number images (73,257 training and 26,032 test images with 10 digits)- -. We downsample each image of TinylmageNet, LSUN and 100 to size 32 x 32.
subset of ImageNet (10,000 test images with 200 classes)
Training Details. We implemented in PyTorch Paszke et al. (2019) a VGG-16 architecture Si- monyan & Zisserman (2015) in line with the previous works of Charpentier et al. (2020); Malinin & Gales (2019) Nandy et al. (2020), with fully-connected layers reduced to 512 units. Models are trained for 200 epochs with batch size of 128 images, using stochastic gradient descent with Nes- terov momentum of 0.9 and weight decay 5e-4. The learning rate initialized at 0.1 and reduced by factor of 10 50% and 75% of the training progress. Images randomly horizontally flipped and shifted by +-4 pixels as a form of data augmentation.
Balancing Misclassification and OOD Detection. Most neural networks used in our experiments tend to overfit, which leaves very few training errors available. We provide accuracies on training, val- idation and test sets in Table With such high predictive perfor- mances, the number of misclassifications is usually lower than the number of OOD samples (~10,000). Hence, the oversampling ap- proach proposed in the paper helps to better balance misclassifica- tion detection performances and OOD detection performances in the reported metrics.
CIFAR-10
CIFAR-100
Train Val Test
99.0 +0.1
91.2 +0.2
93.6 10.1
70.6 10.3
93.0 10.3
70.1 10.4
Table 4: Mean accuracies (%) and std. over five runs.
KLoSNet. We start from the pre-trained evidential model described above. As detailed in Section 3.2 of the main paper, KLoSNet consists of small decoder attached to the penultimate layer of
16

---

Under review as a conference paper at ICLR 2022
Table 6: Other baselines results on comparative experiments on CIFAR-10 and CIFAR-100.
Bold type indicates significantly best performance (p<20.05) according to paired t-test.
LSUN
TinyImageNet Mis+OOD
STL-10
Method
Mis.
OOD
Mis+OOD
OOD
OOD
Mis+OOD
Diff. Ent.
86.8 +1.0
85.6 +0.5
87.2 +0.7
82.7 +0.7
85,8 +0.8
62.0 #1.0
75.4 +1.3
CIFAR-10
VGG-16
EPKL
83.9 11.5
84,5 1.0.7
85.1 1:1.0
80.4 10.88
83.2 11.
61.3 1.0.8
73.8 1:1.1
KLoSNet (Ours)
92.5
87.6 +-0.
91.7 +-0.9
+-0.9
91.2 +0.8
67.7 +1.4
81.8
CIEAR-10
Diff.
82.7
78.3 11.2
81.1
75.9 1.0.8
79.9 100.7
$7.5
70.8
ResNet-18
EPKI.
80.2 +0.6
76.8 +1.3
79.0 +-0.9
74.1 +-088
77.7 +-0.7
56.2 +1.0
68.9 +-0.3
KLoSNet (Ours)
93.9 10.4
93.1 11.1
94.4 1.0.3
90.6 1.0.6
93.2 10.2
68.5 1..33
82.3 1.0.2
Diff. Ent.
80.2 +00.8
65.6 +-0.9
77.2 +-0.8
72.7 +-0.3
80.4 +-0.4
71.0 +-0.5
79.7 +00.5
CIFAR-100
VGG-16
EPKL
78.8 10..8
65.2 11100
76.1 1.0.9
71.6 1.0.2
78.9 1.0.4
70.0 1.0.6
78.3 1.0.6
KLoSNet (Ours)
86.7
68.4 1.111
83.0
76.4 1.0.4
86.4 1:0.4
75.0 1:0.5
86.0
Diff.
83.0 10..4
70.1 11.1
80.2
76.8 1.0.5
83.0 1:0.3
75.6 1.0.5
82.5
CIFAR-100
ResNet-188
EPKL
82.5 10.4
70.2 11.1
80.0 1.0.4
76.3 1.0.6
82.5 1.0.3
75.0 1.0.5
82.0 1.0.2
KLoSNet (Ours)
86.9 +0.3
73.1 +0.4
84.4 4-0.1
80.8 +-0.2
87.3 +-0.2
79.0 +-0.2
86.7 +-0.3
the main network. In CIFAR experiments, this corresponds to VGG-16's fel layer of size 512. This auxiliary neural network is composed of five fully-connected layers of size 400, except for the last layer obviously. KLoSNet decoder's weights are trained for 100 epochs with & loss (Eq. in the main paper) and with Adam optimizer with learning rate le-4. As KLoS" ranges from zero to large positive values (> 1000), one may encounter some issues when training KLoSNet. Consequently, we apply a sigmoid function, azz) = Ite after computing the KL-divergence between the NN's output and Yg. To prevent over-fitting, training is stopped when validation AUC metric for misclassification detection starts decreasing. Then, a second training step is performed by initializing new encoder E' such that OF = 0F initially and by optimizing weights (0g),u) w) for 30 epochs with Adam optimizer with learning rate 1e-6. We stop training once again based on the validation AUC metric.
D ADDITIONAL RESULTS
D.1 DETAILED RESULTS FOR SYNTHETIC EXPERIMENTS
We detail in Table 5 the quantitative results for the task of simultaneous detection of misclas- sifications and of OOD samples for the syn- thetic experiment presented in Section 4.1 of the paper. First-order uncertainty measures such as MCP and Entropy perform obviously well on the first task with 80.2% AUC for MCP. However, their OOD performance drops to 15% AUC on this dataset. On the other hand, Mahalanobis is adapted detect OOD samples but not as good for misclassifications. KLoS achieves comparable performances to best methods in misclassification detection and in OOD detection (79.4% for Mis. and 98.8% for OOD). As a result, when detecting both in- puts simultaneously. KLoS improves all baselines, reaching 89.2% AUC.
Method
Mis. (1)
OOD (t)
Mis+OOD (*)
MCP
80.2 :1.1
15.9 #0.7
48.6
Entropy
78.4 =15
40.3
41.7=10
Mut. Inf.
75.0 +2.3
2.2 10.2
38.8 =1.
Diff. Ent.
42.7
1.9
38.0 1133
Mahalanobis
+1.8
98.5 +0.3
75.0 1.4
KLoS
79.4
98.8 40.3
=0.5
Table 5: Synthetic experiment: misclassifica-
tion (Mis.), out-of-distribution detection (OOD) and simultancous detection (Mis+OOD) (mean % AUC and std. over runs). Bold type indicates significant top performance (p < 0.05) according paired t-test.
D.2 RESULTS WITH OTHER SECOND-UNCERTAINTT MEASURES RESULTS
In T we presents results with other second-nncertainty measures results: differential entropy
(Diff. Ent.) and EPKI.. As observed with Mut. Inf. their performances remains below KLoSNet.
17

---

Under review as a conference paper at ICLR 2022
Table 7: AUPR results for experiments with VGG-16 architecture on CIFAR-10 and CI- FAR100: Misclassification (Mis.), out-of-distributin (OOD) and simultaneous (Mis.+OOD) de-
tection results (% mean and std. over runs).
CIFAR-10
TinyImageNett Mis+OOD(t))
STL-110
Method
Mis. (t)
OOD
Mis+OOD (f)
OOD (f)
OOD (t)
Mis+0ODX1)
MCP
44.8 12.8
80.8 10.6
92.7 #0.4
80.6 20.8
92.7 10.6
57.3 10.5
85.6 10.0
Entropy
44.2 1331
82.1
81.6
57.5
NenfidNet
45.1 13.22
81.2 11.3
93.4 20.4
82.1 20.4
93.7 10.3
58.3 10.8
86.4 10.5
Mut
85.9
82.4
84.4
CIFAR-10
VGG-16
Diff. Ent
40.8 +0.3
86.5 +0.3
93.7 +0.3
83.6 +0.4
93.0 +:03
58.4 #0.4
85.3 +0.5
39.0
40.4
92,9
82.2 +0.5
+0.6
ODIN
43,4
82.1 40.8
92.5 #0.6
81.3 11.0
92.4 +0.7
57.2=20. #0.6
85.0 +0.8
Mahalanobis
43.7 433.5
+0.4
94,9 +0.3
84.9 +0.3
59.5 +0.2
KLoSNer (Ours)
4:2.5
82.2
94,8 202
83.1 20.7
94.7 +:.2
61.7 20.6
$8.4 10.6
MCP
68.2 #1.6
614
90.6 #0.6
70.3 +0.7
92.5 #0.4
62.5 :0.6
90.3 +0.4
Entropy
67.1 ::16
90.4
71.2
63.1
NenfidNet
68.5 +1.8
61.7 20.6
91.0 =0.4
71.2 20.4
92.9 #0.4
63.7 =0.5
90.8 +0.3
CIFAR-100
Mut. Inf
63.1
63.9
70.8
63.1
VGG 16
Diff Ent
64.6
63.8 10.7
90.3 20.4
71.5 20.4
91.8 1:0.
63.8 10.5
89.6 10.3
70.7 40.4
+0.6
ODIN
67.6++1.77
+1.6
90.4 +0.6
70.5 +0.7
92.3 +0.4
62.6 #0.6
90.1 +044
Mahalanobis
65.7
+1.5
91.8 +0.5
73.1 +0.6
#0.6
KLoSNer (Ours)
69.9.:114
64.3 +1.3
92.2 +0.5
72.6 +0.5
93.7 +0.3
65.6 +0.7
92.0 +0.3
CIFAR- 10 with VGG-16
(b) CIFAR- 10 with ResNet18
CIFAR-10
SVHN
CIFAR-100
SVHN
Method
Mis. (t)
(0)
Mis+OOD (f)
Method
Mis. (?)
0OD (t)
Mis+OOD (f)
MCP
#1.6
87.3 +2222
88.9 #0.5
MCP
84.9 10.8
79.6
83.0
Entropy
83.5 12.
85.5 12.3
86.9 11.9
Entropy
84.6 +0.8
79.6 +1.1
82.8 =0.9
ConfdNeet
90.2 10.8
89.0 13.1
91.0
ConfidNet
90.7 10.4
84.6 11.1
88.6 20.6
84.1 +1.5
80.0 +3.9
77.0 +1.2
79.4
Dift Ent.
86.8 +1.0
86.0 +2.00
87.6 #0.9
Dif. Lint
82.7 #0.6
78.3 +1.2
81.1 =0.9
EPKI.
76.8
79.0
ODIN
86.0 12.0
86.8 12.2
87.7 11.0
ODIN
83.7 +00.7
78.9 +1.0
81.9 =0.9
Mahalanobis
91.2
89.1
91.5
Mahalanobis
91.2
90.7
91.8 =0.3
KLoSNet (Ous)
92.5 10.6
89.8 1:300
92,7 11.2
KLoSNet (Ours)
93.9 1:04
93.1 #1.1
94.4 =0.3
(c) CIFAR 100 with VGG- 16
(d) CIFAR- 100 with ResNet18
SVTN
CIFAR-100
SVIIN
Method
Mis. (1)
OOD (?)
Mis+OOD (t)
Method
Mis. (?)
0OD (t)
Mis+OOD (f)
82,9
+0.8
79.6 +1.0
83.0
Entropy
82.2 10.8
72,9 13.9
81.5 12.0
Entropy
84.6 +0.8
79.6 +1.1
82.8 =0.9
ConfidNet
84.4
68.0
80.8
ContidNett
Mut. Inf.
78.9 10.8
72.7 1:1.9
79.5 12.5
Mus. Inf
80.6 +0.6
77.0 +1.2
79.4 =0.9
Diff. Eint.
80.2
72.4
80.2
Dift. Ent
82.7 10.6
78.3
81.1
EPKI.
78.8 +0.8
72.7 +4.88
79.4 +2.4
EPKL
80,2 10.6
76.8 11.3
79.0 10.9
ODIN
82.1 10.88
72.0 133.8
81.3 11.9
ODIN
83.7 =0.7
78.9 #1.0
81.9 =009
Mahalanobis
84.0 10.2
73.4
15.0
83.2 12.5
Mahalanobis
KLoSNet (Ouns
86.7 +0.4
70.4 +50.7
83,5 +2.8
KLoSNet (Ours)
93.9 10.4
93.1 11.1
94.4 10.3
Figure 10: Results with SVHN as OOD dataset (% mean AUROC and std. over 5 runs).
D.3 AUPR RESULTS WITH VGG-16 ARCHITECTURE
Along with the AUROC metric, we also present results with the area under the precision-recal curve (AUPR) metric in Table [ The precision-recall (PR) curve is the graph of the precision TP/(TP FP) as a function of the recall = TP/ HTP//TP- + FNN. This metric is threshold-inde indeeeenene but in contrast AUROC, the AUPR adjusts for the different positive and negative base rates. In our experiments, misclassifications-OO0 results confirm that KLoSNet the best suited measure detect simultancously misclassifications and OOD samples.
samples are used as the positive detection class. These
D.4 RESULTS WITH SVHN AS OOD TEST DATASET
We report in Fig. 10 the results when evaluating with SVHN Netzer et al, (2011) as OOD dataset. Along with simultaneous detection results, also provide separate results for misclassifications
18

---

Under review as a conference paper at ICLR 2022
Table 8: Comparative experiments including relative Mahalanobis (Rel Maha) and Maha- lanobis (Maha) baselines on CIFAR-10 and CIFAR-100. Misclassification (Mis.), out-of- distribution (OOD) and simultaneous (Mis+OOD) detection results (mean % AUROC and std. over 5 runs). Bold type indicates significantly best performance (p<00.05) according to paired t-test.
LSUN
TinyImageNet
STL,110
Method
Mis.
OOD
Mis+OOD
OOD
Mis+OOD
OOD
Mis+OOD
Maha
91.2 +0.3
88.9 =0.2
91.3 10.1
86.4 :0.2
90.2 #0.1
63.4 1022
78.8 10.3
C-FAR-10
VGG-16
Rel_Maha
91.3
88.8
91.4
86.4
90.2 +0.1
63.3
78.8
KLoSNet
92.5
87.6
91.7 1009
86.6 10.9
91.2 10.8
67.7 +1.4
81.8 10.9
Maha
84.0
71.1 #1.0
82.4 82.4 10.5
77.0 10.5
84.9 10.3
75.4 10.3
84.3 10.5
CIFAR-100
VGG-16
Rel_Maha
83.5
70.5
81.9
77.0
84.6
75.4
84.1 +0.5
KLoSNet
86.7
68.4 #11
83.0 1:06
76.4 10.4
86.4 1:04
75.0 1:0.5
86.0 10.4
detection and OOD detection respectively. Similarly to the comparative results in the main paper, KLoSNet outperforms all the baselines in every simultaneous detection benchmark, with Maha- lanobis being second.
D.5 COMPARISON WITH RELATIVE MATIALANOBIS
In T 8
we include results with the relative Mahalanobis distance proposed in |en al. (2021)
Results show no improvement with the simultaneous detection performances, and even slight de- crease for the task of misclassification detection. In the original workshop paper, the authors evaluate their method with wear-OOl) samples, such CIFAR-10 vs. CIFAR-100, while we evaluate here with far OOD datasets c.g. TinyImageNet.
D.6 IMPACT OF ADVERSARIAL PERTURBATIONS
In the original papers, ODIN and Mahalanobis preprocess inputs by adding small inverse adver- sarial perturbations to reinforce networks in their prediction; this has also the observed benefit to make in-distribution and out-of-distriutiion samples more separable. The tuning of the adversarial noise's magnitude depends on the evaluated OOD data. In Figure Tla we plot the AUC of each de- tection task with different values of perturbation magnitude with ODIN, Mahalanobis and KLoS, using SVHN as OOD dataset. Even though there exists particular noise value for improved OOD detection (Fig. 11a middle), increasing noise magnitude deteriorates performances in misclassifica- tion detection (Fig 1a left) for each method. The best results on the simultaneous detection task (Fig. Iaa right) correspond to 0, as done in experiments presented the main paper.
Worse, except with SVHN, adversarial perturbations detrimental even OOD detection. We report the AUC results varying adversarial perturbations on CIFAR-10 dataset when using LSUN (Fig. [1b TinyImageNet (Fig. 11c and STL-10 (Fig. 11d OOD datasets. The best results each considered task correspond to = O and KLoS outperforms both Mahalanobis and ODIN. As opposed to results with SVHN OOD dataset, did not observe improvements
any method (ODIN, Mahalanobis and KLoS) when using inverse adversarial perturbations for OOD detection with LSUN, TinylmageNet and STL-10 datasets. Similar results are observed in Liang et al. (2018) (Appendix B, Fig. 8) when using WideResNet architectures. Regarding Mahalanobis in Lee et al. (2018), the authors only provided ablation for SVHN dataset.
19

---

Under review as a conference paper at ICLR 2022
(a) Mis. detection
(b) OOD detection
(c) Mis.+00D detection
90
Mahalanobic
ODIN
90
85
KLoS
90
AUJROC (%)
AU-ROC (%)
80
85
AU-ROC
85
75
80
65
60
75
0.00
0.01
0.02 0.03 0.04 0.05
0.00
0.01 0.02 0.03 0.04 0.05
0.00 0.01 0.02
0.04 0.05
Noise magnitude
Noise magnitude
Noise magnitude
(a) CIFAR-10 1 SVHN
(a) Mis. detection
(b) OOD detection
(c) Mis. +OOD detection
90
90
Mahalaanbis
KLoS
80
AU-ROC (%)
AU-ROC 9
80
AU-ROC
70
60
70
50
55
0.00 0.01 0.02 0.03 0.04 0.05
0.00 0.01 0.02 0.03 0.04 0.05
0.00
0.01
0.02 0.03 0.04 0.05
Noise magnitude
Noise magnitude E
Noise magnitude
(b) CIFAR-IO LSUN
(a) Mis. detection
(b) OOD detection
(c) Mis.+0OD detection
CDIN
85
90
Mahalanobis
KLo5
80
85
75
AU-ROC (%)
80
70
65
70
60
55
0.00 0.01 0.02 0.03 0.04 0.05
0.00 0.01 0.02 0.03 0.04 0.05
0.00 0.01 0.02 0.03 0.04 0.05
Noise magnitude
Noise magnitude e
Noise magnitude
(c) CIFAR-10/ TinylmageNet
(a) Mis. detection
(b) OOD detection
(c) Mis.+0OD detection
90
ODIN
64
80
Mahalanobis
85
KLoS
62
AU-ROC (%)
AU-ROC (%)
AU-ROC
80
70
75
58
70
56
65
65
54
60
52
0.00
0.01 0.03 0.04 0.05
0.00
0.02 0.03 0.04 0.05
0.00 0.01 0.02 0.04 0.05
Noise magnitude
Noise magnitude E
Noise magnitude
(d) CIFAR-10 STL-10
Figure 11: Effect of inverse adversarial perturbations OOD-designed measures and KLoS for misclassification detection, OOD detection and simultaneous detection with VGG-16 architecture.
20

---

Under review as a conference paper at ICLR 2022
E SELECTIVE CLASSIFICATION IN PRESENCE OF DISTRIBUTION SHIFTS
Classification with a reject option, also known as selective classification El-Yaniv & Wiener 2010) consists scenario where classifier can abstain on points where its confidence is below a certain threshold. This is appropriate for applications where the system can hand over to human experts or users. Performance can be measured on risk-coverage curves. The coverage is the probability mass of the non non-rejected region in X' and can be empirically estimated by the percentage of the non- rejected samples. The risk of a selective classifier is the average loss on the accepted samples. Given chosen coverage, good selective classifiers correlate with low risk. Averaged performances are evaluated on risk-coverage curves with wwrshold--ndependenn area-under-curve metrie, denoted here AURC. The lower the AURC, the better the selective classifier.
Previous works evaluate the performance on in-distribution data. However, a classifier may en- counter data drawn from different distribution when deployed in the wild. Following Koh et al. (2020), we extend selective classification by penalizing non-rejected OOD samples. If sample is drawn from the in-distribution, apply the maximum cost of 1, whatever the prediction. As for simultaneous detection, we rely on oversampling to mitigate the unbalance between misclassifications and OOD) samples.
compute the 0/1 cost function usual. For OOD samples,
Experiments are conducted with previously trained VGG-16 networks on CIFAR-100. We mea- sure their selective classification when subject to distribution shifts by considering CIFAR-I00C Hendryeks & Dietterich (2019) as OOD dataset. This dataset is constructed by corrupting the orig- inal CIFAR-IOO test set. There a total of 15 types of corruptions, which can be grouped into five families, namely noise, blur, weather and digital. Each corruption comes with five different levels of severity. While this dataset commonly used to measure robustness distribution shift, we focus here on models' ability to reject these samples along with misclassifications made on the original CIFAR- test set.
The results are reported by corruption family (noise, blur, weather and digital) in Fig. 13a and de- tailed in Tables 13b and 12b] One common observation regardless the criterion that selective classification is harder when subject to noise perturbations than other types of perturbation. In each case, KLoSNet and ConfidNet obtain the best performances. For instance, for weather perturbations on CIFAR- 10-C, KloSNet achieves 42.7% AURC and ConfidNet 43.4% AURC. In particular, KloS- Net outperforms every other method for blur, weather and digital perturbations of CIFAR-100-C. Hence, when subject more accurate uncertainty estimates without sacrificing predictive performances. Note that for noise corruptions, the results depend widely on the run, which makes interpretation more difficult.
unforeseen distribution shift, model equipped with KLoSNet provides
We present the results of a similar experiment on CIFAR10-C in Figure 12a and its detailed results in Table 12b Here, KLoSNet obtains oate-of-theeard results in every corruption and also on average with 41.0% AURC.
21

---

Under review as a conference paper at ICLR 2022
00
56
Entropy
ODIM
Machalaanaa
AL RC (%)
KLNett
8
noise
blur (a) Aggregated results by corruption type
weather
digital
Noise Shot
Blur
Westher
Digital
Methou
Gassdan
Impale
Defiens
Gloss
Mittion
Zoem
Stow
Frint
Figg
lhrigh
Contras
Etntic
Piezl
JMKG
65.4%%
66.5%
48
65
44.7%
42.716
06.955
43.2%
CondulNes
47.4%
45.8%
47.1%
42.9%
48.1%
45.2%
44.9%
42.5%
43.5%
41.6%
39.1%
45.9%
42.6%
42.0%
42.1%
1.3%
41.45
Mor Inf.
13.03
92.19
51.2%
88 135
53.05
19 99
19:5%
08.8%
49
47.8%5
6..9%
51.555
47.9%
09:8
48.7%
4.8%
87.115
ODIN
40.3%
47.7%
48.%
64.3%
49.5%
46.3%
460%
64.1%
45.2%
43.2%
41.2%
47.36
43.9%
44.3%
43.7%
2.2%
42,99
Mahulanokis
(b) Detailed results
Figure 12: Selective classification on CIFARIO-C. Comparative performance in AURC (%) of classification with the option to reject misclassified test samples and samples from shifted distribu- tions. Results are average on 5 runs (mean + std.).
MCP
ODIN
Entropy
Mahalanobis
ConfidNet
KLoSNet
AU-RC (%)
Mut. Inf.
52
50
noise
blur Aggregated results by corruption type
weather
digital
Weather
Digital
Impuiie
Bright
MCP
53.3%
52.7%
55.2%
50.8%
54.0%
52.3%
$2.5%
$1.3%
52.0%
50.3%
41.9%
52.1%
50.7%
30.4%
51.3%
13.15
49.45
Komopy
$3.99
$3.05
55.00
11.99
54.80
53.15
$3.15
51 81.8%
12.69
51.05
48.215
$2.7%
$1.15
51.95
$1.7%
13.55
49.915
CouddNee
53.5%
52.8%
55,0%
50.1%
53.7%
52.3%
51.9%
30.3%
31.6%
30.0%
47.2%
51.7%
50.3%
49.3%
50.9%
11.7%
49.055
$410%
Mahulanobis
KLasNa
54.05
51.16
55.95
69.5%
51A4
51.6%
$1.4%
58.7%
51.1%
49.2%
66.5%
51.3%
00.9
49.7%
90.7%
9.7%
4006%
(b) Detailed results
Figure 13: Selective classification on CIFAR100-C. Comparative performance in AURC (%) of classification with the option to reject misclassified test samples and samples from shifted distribu- tions. Results are average on 5 runs (mean + std.).
22


Reviews:
[{"review_id": "sRd3jCfrytN", "reviewer": "Reviewer_ucNF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ves: \n\n+ The idea of using a class-wise divergence measure based on evidential models is interesting. \n\n+ The paper has a nice presentation overall. The figures are effective in conveying the key idea and intuition. \n\n+ The results section is well structured. It's nice to see both OOD detection and ID misclassification performance. \n\nConcerns: \n\n- Authors highlight that the uncertainty estimation should consider both class confusion and lack of evidence. However, this may not be always desirable for OOD uncertainty estimation. Take an extreme example, in the classification task of \"bird\" vs \"car\", given images with \"A: bird sitting on the car\" and \"B: computer\", the model would have more class confusion in the former case. Yet A is an in-distribution image and B is an OOD image. \n\n- The above concern brings to my next question: have the authors investigated the efficacy of the two individual terms in Equation (6) for OOD uncertainty estimation?\n\n- \"KLoS also penalizes samples having a different precision α_0. \" I understand that lower precision should be penalized but why a higher amount of evidence is not preferred?\n\n- The experimental results are based on relatively small model capacity (ResNet-18 at the largest). It is unclear how well the method scale for common capacity (say ResNet-101). This is particularly concerning, as Mahalanobis (the best baseline considered) is relatively disadvantaged in this small-capacity setting where the representation might not be as optimal as larger models. Even in this setting, the results are not conclusively in favor of the proposed method, and only is marginally better than Mahalanobis. With that being said, the results would have been more complete if a consistent trend is shown at a larger model capacity. \n\n- Missing  OOD detection baselines Gram Matrix [1],  Generalized ODIN [2]. \n\n- Lastly, it'd be great if the authors can comment on the low performance of common baselines in Table 1. For example, the reviewer uses a standard ResNet-18 trained on CIFAR-100, using the same learning rate and schedule as detailed on Page 16. The OOD uncertainty estimation performance (on LSUN) is not matchable to Table 1. For example, using MCP yields AUROC 75.73, using ODIN and Mahalanobis yields 82.13 and 82.98 respectively. In contrast, the performance for the baselines methods seems low across the board (~70-75%). \n\n\nReferences\n\n[1] C. Sastry, S. Oore. Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices. ICML 2020.\n\n[2] Y. Hsu, Y. Shen, H. Jin, Z. Kira. Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data. CVPR 2020. \n", "summary_of_the_paper": "This paper proposes an uncertainty measurement KLoS by computing the KL divergence between the Dirichlet distribution from the Evidential Neural Networks and a class-wise prototype concentrated on the predicted class. An extra network KLoSNet is further trained to align with the evidential training objective. Empirical studies are performed to show the efficacy of both OOD uncertainty estimation and ID misclassification detection. Results are shown on the task of (1) synthetic dataset with three Gaussian-distributed classes and (2) CIFAR datasets. \n\n", "summary_of_the_review": "In summary, this paper presents an interesting idea however the experimental evaluations are not sufficiently convincing in the current state. I would like to hear the authors address my concerns raised above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oybDINJSp_", "reviewer": "Reviewer_rode", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides nicely drawn illustrative figures that explain the impact of first-order and second-order uncertainties on uncertainty quantification profiles of classification models.\n\nThe paper suffers from multiple fundamental weaknesses detailed below:\n\ni) The paper lacks a concrete hypothesis. Which particular problem of the \"existing\" evidential learning problem is being targeted? Figure 1 only motivates the evidential learning framework, but does not point to any weakness of it. The claim in Page 2 that the evidential models require auxiliary OOD data holds for the Prior Networks approach (Malinin & Gales, 2018, 2019) but not for EDL (Sensoy et al., 2018). As I understand, the only novelty claim is to use Eq 5 as an OOD and misclassification detection criterion instead of the well-established mutual information or entropy criteria. However, I am having hard time to decipher what the claimed limitation of the mutual information criterion is. All the argumentation in the paper is valid only against MCP, which is obvious for the uncertainty quantification community, but what is the problem with mutual information, which in essense has mechanisms for handling both in-domain and out-of-domain uncertainties? For example, the claim:\n\n\"To capture uncertainties due to class confusion and lack of evidence, an effective measure should account for the sharpness of the Dirichlet distribution and its location on the simplex.\" \n\nholds also for the mutual information criterion.\n\nii) The only section in the paper where there is a claim about a methodological novelty is Section 2.2. I am having hard time to make sense of the criterion proposed in Eq 5. It first calculates the Dirichlet density corresponding to an input pattern that captures both first and second-order uncertainties, then makes a heuristic choice of comparing it to a spiky Dirichlet distribution. Why should it be better than decomposing the uncertainty of the predictive distribution according to the law of total variance or any other decomposition criterion that has an information-theoretic meaning? I also do not understand why divergence from the spiky (winner-take-all) version of the own prediction of a model should correlate with the probability of an input sample coming from a different domain.\n\niii) The paper does not report experimental results that give a decisive outcome about the central claims of the paper. As I understand, all models in Table 1 are trained using Eq 6 first, which corresponds to the Reverse KL design of Malinin et al., 2019, just without OOD data. It is then blurry what exactly happens. Is then only KLoSNet trained further on Eq 8? If so, the comparison is unfair, as the proposed solution gains more training resources than its counterparts. If all other models are also trained further with Eq 8, the comparison is still unfair, because they are trained further with a loss that is designed for another OOD criterion than they are designed for. A decisive comparison could have been as follows:\n\na) Take EDL loss \"literally\". Train it for X epochs.\n\nb) Take Reverse KL loss \"literally\". Train it for X epochs.\n\nc) Take Eq 8 and train KLoSNet \"without\" any pretraining for X epochs. \n\nEvaluate all three models with respect to both mutual information and KLoS scores. Now compare which score and which model gives best OOD performance and draw the scientific claim accordingly.\n\nThe results in favor of KLoSNet reported in Table 1 may have something to do with post-training KLoSNet with respect to a criterion that is later used as a performance score. This would of course make both the trained model and the chosen criterion advantageous over others, but it would be the same if any other criterion was chosen.", "summary_of_the_paper": "The paper studies the problem of doing out-of-domain and misclassification detection using the evidential learning framework, which suggests modelling the second-order uncertainties of class assignment probabilities with a Dirichlet-Multinomial hierarchical model. The model maps an input observation, be it an image, to the concentration parameters of the Dirichlet prior on the class probabilities by a neural net with an arbitrary architecture. On top of this existing framework (Sensoy et al., 2018, Malinin & Gales 2018), the paper proposes to use a KL divergence based decision criterion, called \"KL on Simplex (KLoS)\",  which fits a reverse KL divergence between the model output and a spike on the predicted class label in order to smoothen the predictions.", "summary_of_the_review": "Overall, the paper lacks a clear hypothesis statement (Item i)), it does not propose a novel solution (Item ii)), and it does not conduct a set of experiments that give conclusive results in favor of a solid scientific hypothesis (Item iii)). Hence my initial grade is a clear reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "UdGjsiWkLSf", "reviewer": "Reviewer_oef3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The \\lambda term seems to be an important hyper-parameter but it was unclear (unless I missed it!) how it was tuned or how sensitive it is.\n\nIt would have been helpful to frame the proposal in terms of standard notions of epistemic (model) and aleatoric (irreducible) uncertainty. If the proposed measure is meant to capture epistemic uncertainty, would we expect this value to be greater on average on held-out data, for smaller training datasets? If so, it'd be nice to show that via experiments training with different amounts of data and showing this value exhibits the desired trends.\n\nOverall, while the experiments suggest that the proposed method is doing something reasonable, it's not clear to me that it's doing something reasonable for the purported reasons. Besides the other suggestions above, some additional model ablations may be helpful to demonstrate the benefits of the proposed approach, such as auxiliary regression of simplified versions of the proposed objective.\n\nSeparate from my concerns about the justification for the method, I also found there to be a lack of references to related recent work and I think the paper would be more convincing with consideration of stronger baselines. Experimental comparisons aren't strictly necessary in all cases, but at minimum further discussion would be help position this work relative to other deterministic approaches, such as:\n\n- The Mahalanobis baseline is competitive with the proposed approach, but more recent, improved versions of Mahalanobis have been proposed for OOD: https://arxiv.org/pdf/2106.09022.pdf\n- Deep deterministic uncertainty (DDU): https://arxiv.org/pdf/2102.11582.pdf\n- MIR, see: https://arxiv.org/pdf/2107.00649.pdf\n- Deterministic uncertainty quantification (DUQ): https://arxiv.org/pdf/2003.02037.pdf\n\n", "summary_of_the_paper": "This paper suggests training an auxiliary DNN to predict the uncertainty associated with a given example. The auxiliary model uses the predictions of a model fit to the predict the concentration parameters of a Dirichlet (an \"evidential neural network\") to measure the KL divergence between the predicted posterior and one based on the ground truth labels. (The auxiliary model is necessary because the ground truth labels are not available at test time.) The primary claim is that \"[...] leveraging the second-order uncertainty representation that evidential models provide, KLoS captures both class confusion and lack of evidence in a single score.\" In a series of experiments, it is found that the proposed model is competitive with or outperforms selected alternate models across several benchmarks for OOD detection and misclassification detection.", "summary_of_the_review": "Overall, this paper proposes an interesting and to my knowledge novel approach to uncertainty quantification. However, I believe it has two main weaknesses: (1) there's a lack of discussion and experimental comparisons to recent work in uncertainty estimation and (2) there isn't enough introspection of the proposed approach to confirm it's working for the purported reasons.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7qPoLEW89-", "reviewer": "Reviewer_FYgq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows good results and an extensive evaluation on various datasets and two (smaller) network architectures. Additionally, a toy dataset showcases the differences between typical measures and the proposed KLoS well.\nThere are three main weaknesses:\n\n#### W1 - Baseline Comparisons\nTo my understanding from the appendix, KLoSNet is trained in three stages. First a standard classifier is trained based on (Joo et al., 2020) which corresponds to Eq 6. Then, the weights are frozen and a 5 layer MLP is trained to predict KLoS* from the final layer of the original network. Finally, the whole network is finetuned.\nThis procedure makes it difficult to compare the performance in Tab. 1, as the baselines do not receive the extra training, which potentially conflates the performance gain through longer training with the benefits of the proposed method.\n\n#### W2 - Joint Metric\nKLoS is designed as a joint metric for classification confidence and OOD detection. This requires combining the two concepts into one score by implicitly weighing them. It is not clear that this step is meaningful or desired. It might be beneficial to know whether a prediction is uncertain because of class uncertainty or because the sample is far from the distribution. \n\n#### W3 - Ablation\nWhile some hyperparameters such as the oversampling rate $\\kappa$ are ablated, some important experiments are missing: \n\nThe choice of the prototype Dirichlet distribution $\\gamma_{y}$ is potentially an important hyper parameter. It would be interesting to understand the influence of the choice of precision $\\alpha$ as this controls the ‘smoothing’ of the actual target.\n\nThe proposed training scheme relies on initialization of KLoSNet with the original classifier. Is this important to learn a proper measure or does this simply speed up convergence?\n\n#### W4 -  Minor Concerns\nSecon 4.1 states that the Mahalanobis metric ``may have trouble detecting misclassifications’’, however the results in Tab.1 do not seem to confirm this explanation. \n", "summary_of_the_paper": "The paper proposes a method for uncertainty estimation of a classifier network called “KLoSNet”. The KLoS metric is designed to measure both the uncertainty between classes and the probability of out-of-domain samples. For a given sample, the estimated KLoS value is predicted from a secondary network that is trained to predict the true KLoS score of the first since ground truth class information is not available during test time.\nThe method is evaluated on a standard set of benchmarks and shows good performance with respect to recent work.", "summary_of_the_review": "The design of the evaluation makes it difficult to understand exactly where the performance improvement comes from (W1) as the proposed method seems to be trained ~50% longer than the baselines. Additionally, I am not fully convinced that a joint metric for OOD and misclassification is desired (W2). Finally, some ablation experiments are needed to understand the effect of certain design choices (W3). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received 4 diverging ratings: 6, 5, 5, 3. On the positive side, reviewers appreciated the central idea and a quality manuscript. At the same time, they have raised important concerns around unfair comparisons with baselines, experiments not fully supporting the claims and lack of comparisons with some prior methods. After discussions with the authors most reviewers stayed with their original ratings. 
The AC agrees that the weaknesses in this case outweigh the strengths. The final recommendation is to reject.
Decision: Reject