R-EDL: Relaxing Nonessential Settings of Evidential Deep Learning
Authors: Mengyuan Chen
Junyu Gao
Changsheng Xu
Venue: ICLR 2024 spotlight (2024)
Keywords: uncertainty quantification
evidential deep learning
subjective logic
single-forward-pass uncertainty method
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We propose a relaxed version of evidential deep learning by deprecating two nonessential settings not mandated by subjective logic theory.

Abstract: A newly-arising uncertainty estimation method named Evidential Deep Learning (EDL), which can obtain reliable predictive uncertainty in a single forward pass, has garnered increasing interest. Guided by the subjective logic theory, EDL obtains Dirichlet concentration parameters from deep neural networks, thus constructing a Dirichlet probability density function (PDF) to model the distribution of class probabilities. Despite its great success, we argue that EDL keeps nonessential settings in both stages of model construction and optimization.
In this work, our analysis indicates that (1) in the construction of the Dirichlet PDF, a commonly ignored parameter termed prior weight governs the balance between leveraging the proportion of evidence and its magnitude in deriving predictive scores, and (2) in model optimization, a variance-minimized regularization term adopted by traditional EDL encourages the Dirichlet PDF to approach a Dirac delta function, potentially exacerbating overconfidence. Therefore, we propose the R-EDL (Relaxed-EDL) method by relaxing these nonessential settings. Specifically, R-EDL treats the prior weight as an adjustable hyper-parameter instead of a fixed scalar, and directly optimizes the expectation of the Dirichlet PDF provided to deprecate the variance-minimized regularization term. Extensive experiments and SOTA performances demonstrate the effectiveness of our method. Source codes are provided in Appendix E.

Published as a conference paper at ICLR 2024
R-EDL: RELAXING NONESSENTIAL SETTINGS OF Ev- IDENTIAL DEEP LEARNING
Mengyuan Chen Junyy Gao Changsheng Xu 12,33
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS),
Institute of Automation, Chinese Academy of Sciences (CASIA) Schooo of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS) *eng Cheng Laboratory, ShenZhen, China thenmengyuan222115 a...aacccooo cn; {junyu. - gao, csxu) ac. cn
ABSTRACT
A newly-arising uncertainty estimation method named Evidential Deep Learn- ing (EDL), which can obtain reliable predictive uncertainty in a single forward pass, has garnered increasing interest. Guided by the subjective logic theory, EDL obtains Dirichlet concentration parameters from deep neural networks, thus constructing a Dirichlet probability density function (PDF) to model the distribu- tion of class probabilities. Despite its great success, we argue that EDL. keeps nonessential settings in both stages of model construction and optimization. In constructing the Dirichlet PDF, commonly ignored prior weight parameter gov- erns the balance between leveraging the proportion of evidence and its magni- tude in deriving predictive scores. In model optimization, variance- minimized regularization term adopted by traditional EDL encourages the Dirichlet PDF to approach Dirac delta function, potentially exacerbating overconfidence. There- fore, propose the R-EDL (Relaxed-EDL) method by relaxing these nonessen- tial settings. Specifically, R-EDL treats the prior weight as an adjustable hyper- parameter instead of fixed scalar, and directly optimizes the expectation of the Dirichlet PDF provided to deprecate the variance-minimized regularization term. Extensive experiments and SOTA performances demonstrate the effectiveness of our method. Source codes are provided in Appendix E.
INTRODUCTION
In high-risk domains such autonomous driving and medical analysis, is imperative for models to reliably convey the confidence level of their predictions (Choi et al., 2019; Abdar et al., 2022). However, previous research suggests that most modern deep neural networks (DNNs), especially when trained for classification via supervised learning, exhibit poor calibration, tending predomi- nantly towards over-confidence (Guo al., 2017). Despite effective uncertainty methods based on Bayesian theory and ensemble techniques have been developed, these mainstream methods of un- certainty quantification necessitate multiple forward passes the inference phase (Blundell al., 2015; Dusenberry et al., 2020; Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017; Wen et al., 2020; Egele et al., 2022), imposing substantial computational burdens that hamper their widespread industrial adoption. This limitation drives the interest of researchers in exploring how to achieve high-quality uncertainty estimation with minimal additional cost.
Evidential deep learning (EDL) (Sensoy al., 2018) is such newly arising single-forward-parr uncertainty estimation method, which has attracted increasing attention for its success in various pattern recognition tasks (Amini et al., 2020; Bao et al., 2021; Qin et al., 2022; Chen et al., 2022: Oh & Shin, 2022: Sun et al., 2022; Park et al., 2022: Sapkota & Yu, 2022; Gao et al., 2023a). Drawing upon the theory of subjective logic (Josang, 2001: 2016), EDL harnesses both the proportion of col- lected evidence among classes and their magnitude value to achieve high-quality uncertainty estima- tion. effectively mitigating model over-confidence on misclassified and ouf-o--distribution samples. Specifically, in C'class classification task, EDL constructs a Dirichlet distribution Dir(px, axx
"Corresponding author

---

Published as a conference paper at ICLR 2024
to model the distribution of class probability px under the guidance of subjective logic, and the concentration parameter vector ax (z) is given by (1,
x (x)x) = ex(x) + ax(x),
where the base rate ax typically set as a uniform distribution over X. and its scalar coefficient C serves as parameter termed as prior weight. Note that to keep the notation uncluttered, we use ax (2) as simplified expression of x (X = x), and similarly for ex(x) and a (a)) The random variable X denotes the class index of the input sample, and ex(x) signifies the amassed evidence for the sample's s association with class I. Thereafter, for model optimization, the traditional EDL method integrates the mean square error (MSE) loss over the class probability px, which is assumed to follow the above Dirichlet distribution, thus deriving its optimization goal as
1
Ledl
Dixpxixpxx.ax) Ill px (z.D)ED
Σ
D
(2)
1
Σ
[|ج] - (xxx + (xxx
D
(
where the training set 'D consists of sample features and their one-hot labels denoted (z, y) and y [x]] refers to the z-th element of y. A rigorous mathematical exposition of subjective logic and a more detailed introduction of EDL will be provided in section 2 and section 3.1.
Despite the remarkable success of EDL, we argue that in the existing EDL-based methodology, there exists nonessential settings in both model construction and model optimization. These settings have been widely accepted by deep learning researchers, however, they are not intrinsically mandated in the mathematical framework of subjective logic. Specifically, (1) in model construction, the com- monly ignored prior weight parameter in Eqn. 1 actually governs the balance between capitalizing on the proportion of evidence and its magnitude when deriving predictive scores. However, EDL prescribes this parameter's ' value to be equivalent to the number of classes, potentially resulting in highly counter-intuitive outcomes. Therefore, we advocate for setting the prior weight parameter as a free optimization, the EDL loss function given by Eqn. 2 includes variance- minimized regularization term, which encourages the Dirichlet PDF modeling the distribution of probabilities to approach a Dirac delta function which is infinitely high and infinitesimally thin, or other words, requires an infinite amount of evidence of the target class, thus further intensifying the over-confidence is- sue. Contrarily, we advocate for directly optimizing the expectation the Dirichlet distribution towards the given one-hot label, thus deprecating this regularization to obtain more reliable pre- dictive scores. Note that both the above relaxations strictly adhere to the subjective logic theory. Theoretical analysis in section 3 and experiment results in section 5 both demonstrate that relax- ing the above nonessential settings contributes to alleviating the over-confidence issue and bringing more accurate uncertainty estimation. Our contributions include:
in the neural network to adapt to complex application cases. (2) in model
An analysis of the significance of the commonly ignored parameter termed prior weight on balancing the tradc-off relationship between leveraging the proportion and magnitude of evidence to compute predictive scores in the subjective logic framework. Relaxing the rigid setting of fixing the parameter to the number of classes has been shown to enhance the quality of uncertainty estimation.
An analysis of the advantages of directly optimizing the expected value of the constructed Dirichlet distribution, instead of minimizing the integration of MSE loss over the class probability px which follows the above Dirichlet distribution. Relaxing the EDL optimiza- regularization term has been shown
tion objective by deprecating the variance- mitigate the issue of over-confidence.
Extensive experiments confidence estimation and out-of-distribution detection, which comprehensively demon- strate the effectiveness of our proposed R-EDL with remarkable performances under the classical, few-shot, noisy, and video-modality settings.
multiple benchmarks for uncertainty estimation tasks, including
Derivations, proofs, additional experiment results and details are given in Appendix.
'In deep learning. the sample feature is usually denoted by the symbol æ. However, to preclude ambiguity with the symbol denoting the value of the random variable X, we employ instead of a to denote the sample feature. The random variable X, the label y, and the feature pertain the same input sample.
2

---

Published as a conference paper at ICLR 2024
2 SUBJECTIVE LOGIC THEORY
Just as the names of binary logic and probabilistic logic imply, an argument in binary logic must be either true or false, and one probabilistic logic can take its probability in the range [0, to express the meaning of partially true. Furthermore, subjective logic (Jøsang, 2001; 2016) extends probabilistic logic by explicitly including uncertainty about probabilities in the formalism. Specifi- cally, argument in subjective logic, also called a subjective opinion, is formalized as follows:
Definition 1 (Subjective opinion). Let X be a categorical random variable on the domain X. A subjective opinion over the random variable X is defined as the ordered triplet w = (bx, ux: ax), where bx a belief mass distribution over X, ux is a uncertainty mass, ax is a base rate, aka prior probability distribution over X, and the additivity requirements Drex (x(x) + ux 1 and Exex ax (z) 1 are satisfied.
Belief mass assigned to a singleton value x X expresses support for being TRUE, and uncer- tainty mass can be interpreted as belief mass assigned to the entire domain. Therefore, subjective logic also provides a well-defined projected probability, which follows the additivity requirement of traditional probability theory, by reassigning the uncertainty mass into each singleton of domain X according to the base rate ax as follows:
) be Definition 2 (Projected probability of a subjective opinion). Let wx = (bx, uxx ax)
subjective opinion. The projected probability Px of the opinion wx is defined by Px(x) = (x + (x)uxx Vz E X. Note that the additivity requirement Σxex Px(x)) 1 is satisfied.
Furthermore, the subjective logic theory points out that, if the base rate ax and a parameter termed prior weight, denoted W, is given, there exists bijection between multinomial opinion and Dirichlet probabilistic density function (PDF). This relationship emerges from interpreting second- order uncertainty by probability density, and plays an important role in the formalism of subjective logic since it provides a calculus reasoning with PDFs. The proof is provided in Appendix A.I.
Theorem 1 (Bijection between subjective opinions and Dirichlet PDFs). Let X be random variable defined in domain X. and wxx = (bx. ux ax) be subjective opinion. Px is proba- bility distribution over X, and a Dirichlet PDF with the concentration parameter ax is denoted by Px axx) where x ax(x) (I) 2 O, and (x * 0 if a (r) < 1. Then, given the base rate ax, there exists bijection between the opinion wxx and the Dirichlet PDF Dir(px, ax))
( (x(x))
F: (bx, ux ax) + Dir(px, ox)
II pxx(x))x(()) (3)
ILEE (x)
zEX
where I' denotes the Gamma function, ox satisfies the following identity that xx (z)
bx(r: + (x) Vo E eX,
(4)
ux
and W IR,, a scalar called a prior weight, whose setting will be further discussed in section 3.2.
R-EDL: ALLEVIATING OVER CON-ICONF BY RELAXING NONESSENTIAL
SETTINGS OF EDL
Despite the significant success achieved by EDL and its related works, argue that the existing EDL-based methodology (section 3.1) keeps rigid settings on the construction of the Dirichlet dis- tributions specified in Theorem and the design of optimization objectives, which, while widely accepted, are not intrinsically mandated within the subjective logic framework (section 2). Theoret- ical analysis in this section and comprehensive experiments in section 5 both demonstrate that those nonessential settings hinder this line of methods from quantifying more accurate uncertainty. Specif- ically, in this section, rigorously analyze and relax two nonessential settings in EDL, including: (1) in model construction, the prior weight parameter is prescribed to be equivalent to the number of classes (section 3.2); (2) model optimization, the traditional optimization objective includes variance-minimizes regularization term, which potentially intensifies over-confidence (section 3.3). Note that our relaxations the above EDL. settings strictly adhere to subjective logic.
3.1 PRELIMINARY: EVIDENTIAL DEEP LEARNING
Based on the subjective logic theory, Sensoy et al. (2018) proposes a single-forward forward-pass uncertainty estimation method named Evidential Deep Learning (EDL), which lets deep neural networks play
3

---

Published as a conference paper at ICLR 2024
the role of analysts give belief mass and uncertainty mass of samples. For example, in the case of C-class classification, the belief mass bx and uncertainty mass ux of the input sample, whose category index is random variable X taking values r from the domain X [L...., C), are given by
ex(x) bx Exexx ex(')++
C
"x "x= Exex ex(x)+
Xx==xx
Vx X.
(5)
Specifically, ex(2) (z), which denotes the evidence of the random variable X taking the value x, is the æ-th element of the evidence vector ex = [[((z)) E R4, where is the feature of the input sample, y is a deep neural network, ʃ is a non-negative activation function, e.g., softplus, sometimes also called the evidence function, and the scalar C' in this equation serves as the prior weight.
According to Theorem 1, there exists a bijection between the Dirichlet PDF denoted Dir x (Px, ox) and the opinion wx (bx, ux, ax) if the requirement in Eqn. 4 satisfied. Substituting Eqn. into Eqn. 4 and setting the prior weight W in Eqn. 4 as C, we obtain the relationship between the parameter vector of the Dirichlet PDF and the collected evidence in EDL., as expressed by Eqn. 1. Moreover, since EDL sets the base rate a (I)) as uniform distribution, the relationship given by Eqn. can further simplified into x (x) = ex (x) + 1, Vx E X.
To perform model optimization, EDL integrates the conventional MSE loss function over the class probability px which is assumed to follow the Dirichlet PDF specified in the bijection, thus derives the optimization objective given by Eqn. 2. The detailed derivation is provided in Appendix A.2. In inference, EDL utilizes the projected probability Px (refer to Definition 2) as the predictive scores, and uses Eqn. 5 to calculate the uncertainty mass ux the uncertainty of classification,
ex(x) +1
(x(x)
C
C. Sx
Px(x) ex('))
ux ExEx ex(x)
Vz e (6)
Sx
where Sx is the sum of (x(x) over x E X.
3.2 RELAXING RIGID SETTING OF PRIOR WEIGHT IN MODEL CONSTRUCTION
In this subsection, we elucidate how W orchestrates the equilibrium between leveraging the propor- tion and magnitude of evidence to compute predictive scores. Conclusively, we argue against the rigidity of fixing W to the class number and propose viewing it as an adjustable hyper-parameter.
The nomenclature of prior weight comes from the expression of Eqn. 1. Here, the scalar coefficient C, functioning the prior weight W, denotes the weight of the base rate ax, which is alternatively termed the prior distribution. In Theorem 1, it should be noted that the existence of the bijection is contingent upon certain prerequisites; specifically, the base rate ax and the prior weight W must be provided. Typically, in the absence of prior information, we default to setting the base rate as a uniform distribution over the domain X, i.e., ax (x) 1/C, Vo E X, and [ C. However, the setting of the prior weight W is worth further discussion.
We argue that fixing the prior weight to the cardinality of the domain, which is widely adopted by EDL researchers, is not intrinsically mandated by subjective logic and may result in counter counter-intuitive intuitive results. For example, a 100-classes classification task forces W 100. Even though the neural net gives an extreme evidence distribution e = [100, O, O, .......... 0] E R 100 EDL will reach the prediction that the probability of the sample belonging to Class is P (100 + 1)/(100 + 100) 0.5 by Eqn. 6, which is highly counter-intuitive
The underlying reason for the above phenomenon is that the value of W dictates the degree to which the projected probability is influenced by the magnitude of the evidence or contrarily the proportion of the evidence. To elucidate this point more clearly, we first revisit Eqn. 5 and Eqn. 6 without fixing the prior weight W to C. In this way, we can obtain a generalized form of the projected probability Px as
ex(x)+
Px(x) = (x) + axx()) Ex W W
VI e X.
When the prior weight W is set to zero, the projected probability Px in Eqn. 7 degenerates to conventional probability form, which solely relies on the proportion of evidence among classes and is unaffected by their magnitude, as scaling the evidence by a constant coefficient has no impact on Px. However, when W is not zero, we have
e (x) + C
Px(x) <
ex(x) + =1-
ex(x)/W+ +1' VI E X,
(8)
4

---

Published as a conference paper at ICLR 2024
where the equlity holds if Ex L''Ex'' ex(x') = 0. Eqn. 8 indicates that, in scenarios of extreme evidence distributions, i.e., when the evidence for all classes except class x is zero, the upper bound of Px (2) is governed by the ratio of the evidence for class x to the prior weight W. In other words, the upper bound of Px (I)) purely relies on the magnitude of ex (z) when the prior weight W is given, and a lower magnitude results in a larger gap between the upper bound of Px (a) and 1.
From the two cases presented above, it becomes evident that the value of W determines the extent to which the projected probability (x( is influenced by the magnitude and proportion of evidence respectively. Specifically, small W implies that Px (xx is predominantly influenced by the propor- tion of evidence distribution, whereas large W leads Px (z) to mainly considering the magnitude of the evidence while overlooking the evidence proportion.
Intuitively speaking. for any specific case, there should exist an optimal value for W which can bal- ance the inherent trade-off between leveraging the proportion of evidence and its magnitude to ob- tain predictive scores minimizing the model over-confidence on misclassified and out-of-distribution samples. However, it is unlikely that such an optimal value is universally applicable to all scenar- ios, given the myriad of complex factors influencing the network's output. Hence, we advocate for relinquishing the rigidity of assigning the number of classes W, but instead, treating W as an adjustable hyper-parameter within the neural network. Therefore, we revisit Eqn. 4 to derive generalized form of the concentration parameter a of the constructed Dirichlet PDF as
(ex(x) ex (x)
ax(x)=
W = ex(xx)+ Va X,
(9)
W
X
where À = W/C E R. is a hyper-parametee Note that both the projected probability and the uncertainty mass retain the same form as in Eqn. 6, i.e., Px(x) = ax (x(xx)Sx and ux = C/Sx. when represented by ax (z) and Sx.
3.3 DEPRECATING VARIANCE- -MNIMIIZED REGULARIZATION IN MODEL OPTIMIZATION
In the preceding subsection, we underscore the imperative of treating the prior weight W as an adjustable hyper-parameter, which enables the projected probability Px to effectively balance the trade-off between leveraging the proportion and the magnitude of collected evidence. Consequently, in this subsection, elucidate the reasoning underlying our optimization objective, which focuses on directly optimizing the projected probability Px. Upon comparison with the traditional loss func- tion employed in EDL, it becomes evident that our method deprecates commonly used variance- minimizing regularization term. We undertake a meticulous examination of the motivations for relaxing the EDL optimization objective by excluding this term.
With the generalized setting of xx in Eqn. 9, the projected probability Px has the following variant:
(x = Ox (x)
ex(x) +
Vx x.
(10)
Sx
Ex + CX
Consequently, by substituting the class probability in traditional MSE loss with the projected prob- ability Px in Eqn. 10, we seamlessly derive an appropriate optimization objective denoted Creal within our relaxed-EDL. framework in the following form:
Breat
D
E [ylz) - ?x (EE
Σ
(11)
Regarding the reason for adopting the above optimization objective, we contend that the projected probability Px has the unique property of alleviating the overconfidence typically arising from optimization toward the hard one-hot labels y. As previously noted, the projected probability Px harnesses both the magnitude and proportion of collected evidence to more accurately represent the actual likelihood of given output. From an optimization perspective, compared to the proportion of evidence among classes, i.e., e (x(x)/2 E, ex (x). or the belief mass bx the projected probability Px has more tolerance towards the existence of the uncertainty mass ux, since ux also contributes to the projected probability Px according to the base rate ax- In other words, the item a ux alleviates the urgency of the projected probability Px tending the one-hot label y when the model has not collected enough evidence, since the uncertainty mass u is inversely proportional to the total amount of evidence, thus mitigating the over-confidence issue some extent.
5

---

Published as a conference paper at ICLR 2024
Meanwhile, the optimization goal in Equ. 11 can also be interpreted as encouraging the expectation of the Dirichlet distribution to converge to the provided label, since the bijection introduced in Theorem I has been established on the following identity:
Px (a) Exx Px (x)),
(12)
which can be easily derived from Eqn. 10 and the property of Dirichlet distributions. Therefore, by substituting Eqn. 12 into Eqn. 11 and then comparing it with Equ. 2. can find that the essential difference hetween the two optimization goals is that, EDL optimizes the expectation of the tradi- tional MSE loss over the constructed Dirichlet PDF, while our proposed R-EDI. directly optimizes the expectation of the constructed Dirichlet PDF with MSE loss. As a result, regularization term, denoted Lvar, which attempts minimize the variance of the Dirichlet distribution given by the following equation is deprecated:
Lver
1 D
Σ
Var Dx (px
S
E XEX ax(x) (Sx +1)
(13)
D
(E.))ED
(F.U)ED
Let us delve deeper into this variance-minimizee regularization term. When the variance of a Dirich- let distribution is close to zero, the Dirichlet probability density function is in the form of Dirac delta function which is infinitely high and infinitesimally thin. Consequently, in the entire training phase, the regularization term Lear keeps requiring an infinite amount of evidence of the target class, which further intensifies the serious over-confidence issue we seek to mitigate. From another per- spective, the Dirichlet distribution which models the distribution of first-order probabilities would gradually degenerate to a traditional point estimation of first-order probabilities when its variance approaches zero, thus losing the advantage of subjective logic in modeling second-order uncer- tainty. Therefore, we posit that omitting this regularization term contributes to alleviating the over- confidence issue which commonly results in suboptimal uncertainty estimation, while preserving the merits of subjective logic. Our ablation study further corroborates this assertion. Moreover, follow- ing previous works (Sensoy et al., 2018; Deng et al., 2023), we adopt an additional KL-divergence based regularization for optimization, and its detailed introduction can be found in Appendix A.2.
RELATED WORK
Extensions and applications of EDL. A detailed introduction of EDL can be found in section 3.1, and here we briefly introduce the follow-up works of EDL. After Sensoy al. (2018) proposes EDL, Deep Evidential Regression (DER) (Amini et al., 2020: Soleimany et al., 2021) extend this paradigm by incorporating evidential priors into the conventional Gaussian likelihood function, thereby en- hancing the modeling of uncertainty within regression networks. Kandemir et al. (2022) combines EDL, neural processes, and neural Turing machines to propose the Evidential Tuning Process, which shows stronger performances than EDL but requires a rather complex memory mechanism. Meinert et al. (2023) offers further insights into the empirical effectiveness of DER, even in the presence of pwrrpparameterize representations of uncertainty. Recently, I I-EDL proposed by Deng et al. (2023) largely outperforms EDL by incorporating Fisher information matrix to measure the informative- ness of evidence carried by samples. For application, DEAR (Bao et al., 2021) achieves impressive performances on open-set action recognition by proposing a novel model calibration method to regularize the EDL training. Moreover, EDL has achieved great success in other applications of computer vision (Qin et al., 2022; Oh & Shin, 2022; Sun et al., 2022; Park et al., 2022: Sapkota & Yu, 2022: Chen et al., 2023a;b; Gao al., 2023b). Compared with previous efforts, our method is the first to consider relaxing the nonessential settings of the traditional EDL while strictly adhering to the subjective logic theory.
Other single-model uncertainty methods based on DNNs. In addition to EDL-related works, var- ious single-model methods exist for estimating predictive uncertainties. Efficient ensemble meth- ods (Wen et al., 2020; Dusenberry et al., 2020), which cast set of models under single one, show state-of-the-arr performances on large-scale datesets. While these methods are parameter-efficient they necessitate multiple forward passes during inference. Bayesian Neural Networks (Ritt) et al., 2018; Izmailov et al., 2021) model network parameters as random variables and quantify un- certainty through posterior estimation while suffering from significant computational cost. A widely-recognizer method Monte Carlo Dropout (Gal & Ghahramani, 2016), which interprets the dropout layer as random variable following Bernoulli distribution, and training neural network

---

Published as a conference paper at ICLR 2024
with such dropout layers can be considered an approximation to variational inference. Two other notable single-forward forward-aasss pass methods, DUQ (Van Amersfoort et al., 2020) and SNGP (Liu et al,, 2020), introduce distance-aware output layers using radial basis functions or Gaussian processes. Although nearly competitive with deep ensembles in OOD benchmarks, these methods entail ex- tensive modifications the training procedure and lack easy integration with existing classifiers. Another group of efficient uncertainty methods are Dirichlet-based uncertainty (DBU) methods, to which EDL also belongs. Prominent DBU methods encompass KL-PN (Malinin & Gales, 2018), RKL-PN (Malinin & Gales, 2019), and Posterior Network (Charpentier et al., 2020), which vary in both the parameterization and the training strategy of the Dirichlet distribution. Compared these preceding methods, our approach combines the benefits of exhibiting favorable performances, being parameter-effiiient,, and easily integrable.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Baselines. Following Deng et al. (2023), we focus on comparing with other Dirichlet-based uncer- tainty methods, including the traditional EDL (Sensoy et al., 2018), I-EDL (Deng et al., 2023), KL-PN (Malinin & Gales, 2018), RKL-PN (Malinin & Gales, 2019), and PostN (Charpentier al., 2020). Additionally, we present the results of the representative DUQ (Van Amersfoort et al., 2020) and the popular Bayesian uncertainty method MC Dropout (Gal & Ghahramani, 2016) for reference. For experiments concerning modal-mo data, follow- ing Bao et al. (2021), we compare our methods with: OpenMax (Bendale & Boult, 2016), MC Dropout, BNN SVI (Krishnan et al., 2018), RPL (Chen et al., 2020), and DEAR (Bao et al., 2021).
angle-forward-pass method
Datasets, Implementation details, Hyper-parameter settings. Refer to Appendix C.
5.2 CLASSICAL SETTING
A classifier with reliable uncertainty estimation abilities should exhibit following characteristics: (1) Assign higher uncertainties to out-of-distribution (OOD) than in-distribution (ID) samples; (2) Assign higher uncertainties to misclassified than to correctly classified samples; (3) maintain com- parable classification accuracy. Therefore, We first evaluate our method by OOD detection and confidence estimation image classification, measured by the area under the precision-recit curve (AUPR) with labels for ID) / correctly classified data, and labels O for OOD/ / misclassified data. For the Dirichlet-base uncertainty methods, we use the max probability (MP) and the sum of Dirichlet concentration parameters, aka the scaled reciprocal of uncertainty mass (UM) of subjective opin- ions, as the confidence scores. For MC Dropout and DUQ, we only report their MP performances since they do not involve Dirichlet PDFs. As Table shows, our R-EDL method shows consistently favorable performances on most metrics. In particular, comparing with the traditional EDL method and the SOTA method Z-EDI,, our R-EDI. obtains absolute gains of 6.13% and 1.74% when eval- uated by MP on the OOD detection setting of CIFAR- 10 against SVHN. Besides, our method also achieves superior performances on confidence estimation while maintaining a satisfactory classifi- cation accuracy. All results are averaged from 5 runs, and the relatively small standard deviations indicate that R-EDL exhibits stable performances.
5.3 FEW-SHOT SETTING
Next, we conduct more challenging few-shot experiments on mini-ImageNet to further demonstrate the effectiveness of our method. As shown in Table 2, we report the averaged top-1 accuracy of classification and the AUPR scores of confidence estimation and OOD detection over 10,000 few- shot episodes. We employ the N-way K-shot setting, with N {5, 10) and K (1,55 20}. Each episode comprises N random classes and K random samples per class for training, m1n(15, K) query samples per class for classification and confidence estimation, and an equivalent number of query samples from the CUB dataset for OOD detection. As depicted in Table 2, our R-EDL method achieves satisfactory performances on most N N-way K-shot settings. Specifically, comparing with the EDL and I-EDL methods, our R-EDL obtains absolute gains of 9.19% and 1.61% % when evalu- ated by MP on OOD detection of the 5-way 5-shot task.
5.4 NOISY SETTING
Thereafter, we employ noisy data to assess both the robustness of classification and the OOD detec- tion capability of our method in the presence of noise. Following Deng et al. (2023), we generate
7

---

Published as a conference paper at ICLR 2024
Table I: Accuracy of classification and AUPR scores of confidence estimation and OOD detection, averaged over 5 runs. On MNIST we adopt ConvNet consisting of 3 conventional layers and 3 dense layers as the backbone, while on CIFARIO we adopt VGG16. A-B denotes taking A/B as ID/OOD data. MP refers to max probability, and UM refers to uncertainty mass.
MNIST
MNIST FFMIST
Mothod
Classification Acc
Confidence Estimation
0OD Detection
OOD Detection
MP
UM
MP
UM
MP
UM
MC Dropout
99.26=0000
99.98=0.00
94,00=0.1
96,56=t0.3
DUQ
98.65-+0.1
99.97-40.0
98.52+0.1
97.92-40.6
KL-PN
99.01=0.00
99. 39.92++0.0
99.95=0000
92.97=1.2
93.39+ 1.0
98, 98.4440..1
98.16=t0.0
RKL-PN
99.21=000
99.67=00.0
99.57=0.0
60.76=2.9
53.76+3.4
78.45:33.1
32.18:3.6
PostN
99.34-10.0
99. 99.98++0.0
99.97.40.0
95.75--0.2
94.59+-0.3
97.78-40.2
97.24+0.3
EDL
98.22-0.31
99.99 09,09=00.00
99.9840.00 00
97. 97.02:1:0.76 76
96.31+22033
98. 10:00444
98 08:084444
EDL
99.21
99.98
99.08
98.34:10.24
98.33:10.24
98.89.10.22
90.86:0.299
R-EDL/Ours)
99.33-0.03
99.99:00.0
99.99-10.00
98.69-10.19
98.69-10.20
99.29-10.11
99.29-0.12
CIFARIO
CIFARIO
AARRIO-CIFARIOO
Method
Classification
Confidence Estimation
OOD Detection
OOD Detection
Acc
MP
UM
MP
UM
MP
UM
MC Dropout
82.84:0.11
97.15=00.0
51.39=0.1
45.57
DUQ
89.33:10.2
97.89-0.3
80.23=3.4
84,75#11
KL-PN
27.46++1.7
50.61+4.0
42.49+4.2
13,96=1.9
43.23+22.3
61,41+2.8
61,53++3.4
RKL-PN
64.76:+0.3
86.11=00.4
85. 0,3
53.61=1.1
49.37:00.8
55.4242.6
54.74
PostN
84.85::0.0
97.76::0.0
97.25=0.0
80.21
77.71:0.3
81,961:0.8
82.06:0.88
EDL
83.55=0.64
97,86=00.17
97.83++0.17 17
78.87:3.50 50
79.12433.69
84.30=20.67
84, 84.18:0.74
I-EDL
89.20=0.32
98.72=0012
98.6340.11
83,26::2.44
82.96:+2.17
85.35:10.69
84.84:0.64
R-EDL(Ours)
90.09=0.30
98.98=00.05
98.98:00.05
85.00-:1.222
85.00-
87.72:+0.31
87.73=0.31
Table 2: Results of the few-shot setting for WideResNet-28-10 10 on mini- Imtge,, with CUB as OOD data, averaged over 10000 episodes.
S.y 1-Shot
10-Way I-Shot
Mehhod
Classiication
Confidence Estimation
OOD Detection
Classtiiation
Contidence Estimation
OOD Detection
Ace
MP
UM
MP
UM
Ace
MP
LM
MP
UM
EDI.
61:00=0:11
72
75.34=0.12
66.78-0.12
63
46.55=0008
11.37-011
61 68:68
1::00000000
67 67.81:00112
Z EDL
63.8210,10
828310.11
80,3310.11
71.790.12
74.76.10.13
49,37
68 40.10.09
67,54...09
71,6010.10
10,95
(Bur(O
63,93.0.11
82.710.11
80,5010.11
729110.12
7484
50/02.0.077
68.68.0.09
67.1210.99
72,310010
73,0110.10
5.Way S-Shot
I0-Way S-Shot
Mehhod
Classtiication
Confidence Estimation
OOD Detection
Chastiication
Confidence Extimation
OOD Detection
Ace
MP
UM
MP
UM
Ace
MP
UM
MP
UM
EDL
58-0008
30:00044
02.00=0:05
74.46-0.10
76.53-0014
62.50=00:0
55.00000
84 35
71.66-0.10
76.28-0010
T-EDL
82.00:00007
94.4210.04
03.01
82:01:0.100
82 12.48:10.0 10
67 0.000
89.14:004
B5 55.522000
80661:0.11
82.28 82.29::0.10
R-uus(UUurs)
29:000444
93.65:0.155
83,65-0100
54.22:0.10
70.5
N4:0004
26.26:00055
02.0
03.37:00199
5 Way 20-Shox
W0-Way 20-Shot
Mahood
Chasifisation Ace
Confidence Estimation
DOD Detection
Classiication Ace
Confidence Estimation
OOD Detection
MP
UM
MP
UM
MP
UM
MP
UM
EDL
05.54
97,05 97.7710.02
79.75:0.12
0:13000009
94,5710.03
93. 13.1500000
74.5010.05
76.9910.09
Z-BDI.
88. 18.12=0.05
07.93=1.02
96 96.0000:022
34.29=0.19
85. 40
78. 97:00044
95.15=00022
93. 32+0000
81 34:40:077
#2.52=0.07
(-ur)(
88.74L.0.0
97.83.L0.02
97.10
$4,85
15.07
79.79.0.04
94.8110022
934411002
82,22.0,06
82,7210.07
the noisy data by introducing pero-mean isotropic Gaussian noise to the test split of the ID dataset. Fig. 1(a) clearly illustrates the superior performance of R-EDL in terms of the average of these two key metrics. More results and analysis are provided in Appendix D.3.
5.5 ABLATION STUDY AND PARAMETER ANALYSIS
We assess the performance impact of relaxing two aforementioned nonessential settings in EDL, as summarized in Table 3. In particular, we explore the effects of retaining the original value of X = 1, and of reintroducing the deprecated variance-minimiz original settings are restored, R-EDL reverts to traditional EDL. As evidenced in rows and 4 of Table 3, reverting to each of these original settings results in noticeable performance decline, or conversely, relaxing these settings leads to performance gains, particularly in OOD detection. For instance, measured by the AUPR score for OOD detection in the setting of CIFAR-10 vs SVHN
regularization term Cear. Note that if both
8

---

Published as a conference paper at ICLR 2024
930
68
ALU0A Scoree of 0S0
R05
60
650
88
[20
0cterre 10:00 Vilue of wwpphppppapmenttg
Vilue of pyperppmmeeeee
(a) Performance in noisy setting
(b) Parameter analysis
Figure 1: (a) The performance trends of EDI., I-EDI,, and R-EDL,, measured by the average of the classification accuracy and the AUPR score of OOD detection, across varying levels of Gaussian noise. (b) Parameter analysis of the hyper-parameter A, evaluated by accuracy of classification CIFAR-I0 10 and AUPR score OOD detection against CIFAR-100, respectively.
Table 3: Ablation study on the classical setting and the few-shot setting, with respect to the relax- ations about treating as syper-parameter and deprecating the regularization term Lvar.
Classical setting
Few-shof setting (1l-wway 5-shot)
Method
CIFAR-10
SVHN
CIFAR-100
mini- Imaieet
+ CUB
Cls
Conf
OOD Detect
OOD Detect
Cls
Conf
0OD Detect
EDL T-EDL
83.55-H0.64
97.83-+0.17
79.12:43.69
84, 18:40440774
62.50=0.08
84.35-+0.06
76.28=0.10
89.20:1:0.22
98.63:+0.11
82.96:22.17
84.84:10.64
67.89=0.06
85.52=00055
82.29=0.10
R-EDL w/
88.53-+-0.88
98.78-+0.05
83.24:+:.225
86.95-+0.37
0.07
85.82:#0.05
0.089:00.09
R-EDL Cour
90.03:10.15
98.96:10.07
1.604:1..66
87.59:10.39
69.97=0.05
85.58:0.05
82.25=00.99
R- R-EiL((Ours)
90.09:10.30
98.98:10.05
85.00::1.22
87.73:10.31
70.51=0.05
8.026::0055
83.37:00099
relaxing just one setting yields improvements of 4.12% and 4.92% respectively. Moreover, when both settings are relaxed, the performance of R-EDL. improves by 5.88%. Thus, we conclude that both relaxations are effective and their joint application yields further optimized performance.
Moreover, we further investigate the effect of the hyper-parameter A. Fig. 1(b) demonstrates the trend of variation in classification accuracy on CIFAR-10 and the AUPR score for OOD detection on CIFAR- as the hyper-parameter À varies from 0,01 to 1.5. In this setting, À is ultimately established at 0.I, selected from the range [0.1:0.1:1.00 based on the best classification accuracy on the validation set. More results and analysis can be found in Appendix D.5.
Due to space limitation, please refer Appendix D.4 for results of Video-modality Setting, and Appendix D.6 for Visualization of Uncertainty Distributions with different metrics.
CONCLUSION
Summary. We propose Relaxed-EDL., generalized version of EDL, which relaxes two tradition- ally adopted nonessential settings in the model construction and optimization stages. Our analysis reveals two key findings: (1) A commonly ignored parameter termed prior weight governs the bal- ance between leveraging the proportion of evidence and its magnitude in deriving predictive scores; (2) A variance-minimized regularization term adopted by the traditional EDL method encourages the Dirichlet PDF to approach Dirac delta function, thereby heightening the risk of model over- confidence. Based on the findings, R-EDL treats the prior weight as an adjustable hyper- parameeer instead of fixing to the class number, and directly optimizes the expectation of the Dirichlet PDF provided to deprecate the evaluations underscore the efficacy of our proposed methodology.
regularization term. Comprehensive experimental
Deficiencies and Future directions. This paper can be extended along two directions below. (1) Al- though the crucial role of the prior weight parameter in balancing the trade-off between leveraging the proportion and the magnitude of collected evidence has been elucidated, the underlying mecha- nism dictating its optimal value a topic worth further investigation. (2) The optimization objective of R-EDL can be interpreted PDF. While this approach is principled and effective, it is somewhat coarse. Future work could explore optimization goals considering other statistical properties of Dirichlet PDFs.
optimization of the expected value of the constructed Dirichlet

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grants 62036012, 62236008, U21B2044, 62102415, 62072286, and 62106262.
REFERENCES
Moloud Abdar, Abbas Khosravi, Sheikh Mohammed Shariful Islam, U Rajendra Acharya, and Athanasios V Vasilakos. The need for quantification of uncertainty in artificial intelligence for clinical data analysis: increasing the level of trust in the decision-making process. IEEE Systems, Man, and Cybernetics Magazine, 8(3):28-40, 2022.
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33:14927-14937, 2020.
Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 13349-13358, 2021.
Abhijit Bendale and Terrance E Boult, Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. Posterior network: Uncertainty es- timation without ood samples via density- baaed pseudo-counts. Advances in Neural Information Processing Systems, 33:1356-1367, 2020.
Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng. Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 507-522. Springer, 2020.
Mengyuan Chen, Junyu Gao, Shicai Yang, and Changsheng Xu. Dual-evidential learning for weakly-supervised temporal action localization. In European Conference on Computer Vision, pp. 192-208. Springer, 2022.
Mengyuan Chen, Junyu Gao, and Changsheng Xu. Cascade evidential learning for world supervs-upe temporal action localization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14741-14750, 2023a.
Mengyuan Chen, Junyu Gao, and Changsheng Xu. Uncertainty-awar dual-evidential learning for seaelvesupervised temporal action localization. IEEE transactions on pattern analysis and ma- chine intelligence, 2023b.
Jiwoong Choi, Dayoung Chun, Hyun Kim, and Hyuk-Jae Lee. Gaussian yolov3: An accurate and fast object detector using localization uncertainty for autonomous driving. In Proceedings of the IEEE/CVF International conference on computer vision, pp. 502-511, 2019.
Tarin Clanuwat, Mikel Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint 11200:181,.01718, 2018.
Danruo Deng, Guangyong Chen, Yang Yu, Furui Liu, and Pheng-Ann Heng. Uncertainty estimation by fisher information- evidential deep learning. arXiv preprint arXiv:2303.0204, 2023.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In International conference on machine learning, pp. 2782-2792. PMLR, 2020.
Romain Egele, Romit Maulik, Krishnan Raghavan, Bethany Lusch, Isabelle Guyon, and Prasanna Balaprakash. Autodeuq: Automated deep ensemble with uncertainty quantification. In 2022 26th International Conference Pattern Recognition (ICPR), pp. 1914-1914, IEER, 2022.
10

---

Published as a conference paper at ICLR 2024
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Junyu Gao, Tianzhu Zhang, and Changsheng Xu. Learning model relationships for zero-shot video classification. IEEE transactions on pattern analysis and machine intelligence, 43(10): 3476-3491, 2020.
Junyu Gao, Mengyuan Chen, and Changsheng Xu. Collecting cross-modal presence-absence ev- idence for weakly weakly-supervised audio-visual event perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18827-18836, 2023a.
Junyu Gao, Mengyuan Chen, and Changsheng Xu. Vectorized evidential learning for weakly- supervised temporal action localization. IEEE transactions on pattern analysis and machine intelligence, 2023b.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What bayesian neural network posteriors really like? In International conference machine learn- PMLR, 2021.
ing. pp.
Audun Jøsang. A logic for uncertain probabilities. International Journal of Uncertainty, Fuzziness and Knowledge-Boud Systems, (033):77--1 2001.
Audun Jøsang. Subjective logic, volume Springer, 2016.
Melih Kandemir, Abdullah Akgül, Manuel Haussmann, and Gozde Unal. Evidential turing pro- cesses. In International Conference on Learning Representations, 2022.
Ranganath Krishnan, Mahesh Subedar, and Omesh Tickoo. Bar: Bayesian activity recognition using variational inference. arXiv preprint arXiv: 1.03300 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 201 International conference on computer vision, pp. 2556-2563, IEEE, 2011.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30, 2017.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. con/exdb/mmist// 1998.
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshmi- narayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances Neural Information Processing Systems, 33:7498-755 7512, 2020.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved un- certainty and adversarial robustness. Advances in Neural Information Processing Systems, 32, 2019.
David McAllester. Some pac-bayesian theorems. Proceedings of the eleventh annual confer- ence on Computational learning theory, pp. 230-234, 1998.
Nis Meinert, Jakob Gawlikowski, and Alexander Lavin. The unreasonable effectiveness of deep idential regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9134-9142, 2023.
11

---

Published as a conference paper at ICLR 2024
Mathew Monfort, Bowen Pan, Kandan Ramakrishnan, Alex Andonian, Barry A McNamara, Alex Lascelles, Quanfu Fan, Dan Gutfreund, Rogério Schmidt Feris, and Aude Oliva. Multi-moments in time: Learning and interpreting models for multi-action video understanding. IEEE Transac- fions on Pattern Analysis and Machine Intelligence, 44(12):9434 (112):9444-445, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and A Ng. The street view house numbers (svhn) dataset. Technical report, Technical report, Accessed 2016-08-01.[Onnnnn 2018.
Dongpin Oh and Bonggun Shin. Improving evidential deep learning via multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7895-7903, 2022.
Younghyun Park, Wonjeong Choi, Soyeong Kim, Dong-Jun Han, and Jaekyun Moon. Active learn- ing for object detection with evidential deep learning and hierarchical uncertainty aggregation. In The Eleventh International Conference on Learning Representations, 2022.
Yang Qin, Dezhong Peng, Xi Peng, Xu Wang. and Peng Hu. Deep evidential learning with noisy cor- respondence for modal retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4948 4956, 2022.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR C018-Conference Track Proceedings, volume 6. International Conference Representation Learning, 2018.
Hitesh Sapkota and Qi Yu. Adaptive robust evidential optimization for open set detection from imbalanced data. In The Eleventh International Conference on Learning Representations, 2022.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classifica- tion uncertainty. Advances in neural information processing systems, 31, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.153: 2014.
Ava P Soleimany, Alexander Amini, Samuel Goldman, Daniela Rus, Sangeeta N Bhatia, and Con- nor W Coley. Evidential deep learning for guided molecular property prediction and discovery. ACS central science, 7(8):1356-1366 2021.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: /2 2012.
Shuzhou Sun, Shuaifeng Zhi, Janne Heikkilä, and Li Liu. Evidential uncertainty and diversity guided active learning for scene graph generation. The Eleventh International Conference on Learning Representations, 2022.
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In International conference on machine learning, pp- 9690-9700. PMLR, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-200 dataset. 2011.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint
2017.
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. arXiv preprint arXiv:2101. 06395,,,,........1... 2021.
12

---

Published as a conference paper at ICLR 2024
A
PROOF AND DERIVATION
This section provides the proof of Theorem 1 and the derivation of optimization objectives of EDI..
A.1 PROOF OF THEOREM 1
Theorem 1 (Bijection between subjective opinions and Dirichlet PDFs). Let X be random variable defined domain X. and wx = (bx. ux ax) be a subjective opinion. px is a proba- bility distribution over X, and Dirichlet PDF with the concentration parameter ax is denoted by Dir(Px xxx), where (x(x) 2 O, and px (x) * if ax (x) < 1. Then, given the base rate ax, there exists bijection F between the opinion wx and the Dirichlet PDF Dir(px, ax))
(Z (x(x))
F w = (bx,ux, ax) + Dir(px, ax) ILeex ((xx(x))
Pxx
(14)
FEX
where I denotes the Gamma function, ax satisfies the following identity that a (z)
bx bx(r
+ +ax(x)W,
(15)
ux
W E R., is a given scalar representing a non-informative prior weight.
Proof. The proof of the bijection will be performed in two steps. First, we will prove Dirich- let distribution Dir Dix(px. ax) uniquely specified by its parameters axx aka there exists a bijective mapping between Dir(px, xxx and x Then, we will prove the bijection between the Dirichlet pa- rameters x and the subjective opinion wx. Therefore, the bijection between wx and Dir(px, oxx) can be established due to the transitivity of bijection.
Step 1: To prove the mapping F1 : ax +> Dir(px, ax is bijective, we will prove it is both injective and surjective. The surjective property is obvious due to the mapping form. We use proof by contradiction to verify the injectivity as follows.
Assuming that there exists two Dirichlet distributions over the random variable X, which are param- eterized by two different concentration parameter vectors ax and ax respectively, sharing exactly the same probability density function, i.e., 3x X, ax (I) o âx (I), and Vr e X and vpx Spx,
T (ErEx xx (x) Irex (x)(
I ( Ox (x)) ILe [(xxx (z))
II px
II
px
(16)
zeX
ZEX
where Six/ is a [X] dimmennsio unit simplex. Taking the logarithm of both sides, we have - (oo((((xxx)))
>
((ax(=)-1)loglll
= loooo log(B(xx(x)))+ > ((xx(z)-1)) lox(px
(xx)))),
(xx
ZEX
xEx
(17)
where denotes a |X dimeessioo beta function. Therefore, we have
Σ (ax ãx (x)) log (x)( loog (B (ax((1))) B [ (2)) (B(ax(2))
Vpx E Spx
(18)
aeX
Since the above equation holds for any probability distribution Px, have
Σ (ax(x) (x)) log (px(x) - (x)) = 00 FEX
Vpx.p's E Spx--
(19)
The above equation can be regarded as a homogenous linear equation with (x - ãx (x) as variables and log (px(x) p'x (z)) as parameters. Due to the arbitrariness of px and p'x- and the property of homogencous systems of linear equations, we know that Eqn. 19 only has a particular solution, i.e., ax (z) (x) for any x E X, which violates our assumption.
Therefore, F, is both injective and surjective, thus bijective.
Step 2: To prove the bijection between wix and ax, we also need prove the mapping Fg w ax is both injective and surjective. Since the base rate ax and the non-informative prior weight W in Eqn. 15 are given, F32 can be simplified (bx, ux) + ax with the following formulation:
ax(x)= = bx (x(
Vr X.
(20)
ux
13

---

Published as a conference paper at ICLR 2024
First, proof by contradiction verify the injection. Assuming that there exists two different sets of belief mass and uncertainty mass which corresponds to the same set of Dirichlet concentration parameters, aka there exists (bx, ux), (bx, üx). ox, which satisfies
a (x) = (x) bx(x) üx
E
(21)
and Ea X. (x) o bx(x),, values of x: E X and utilize the additivity requirement Exex (x(x) +ux = 1, then we will have
or ux + ux. We take the summation of Eqn. 21 across all possible
[ax(x)= ZEX
ux úx
(22)
ux
Thus we reach ux üx and after using the relationship in Eqn. 21, we will have bx(x) = bx(x). Vr E X. Thereafter, our assumption violated and thus F2 is injective.
Second, we prove F2 is surjective, aka for any Dirichlet parameter set ax, there exists set of (bx, ux) satisfying Eqn. 20. By summing Eqn. 20 over all values of x X, we have
Sx - ux
(23)
ux
where Sx = ExEx (x). By reorganization and substituting ux into Eqn. 20, we have
bx(x)= = axx
ux Sx +1'
(24)
Sx +1
which satisfy all the requirements. Therefore, the mapping F2 is surjective.
Finally, since Fi and F22 both bijective, F = Fi F2 is also bijective.
Moreover, cases of no prior information available, we generally set the base rate ax (x) as uniform distribution, i.e., ax (z) x Vz X, and Eqn. 15 can be reorganized
ax(x)) = bx (xr)
+
W
Vo E X,
(25)
ux
X
or equivalently as
(x W bx(x) Exrex ax (x')
W
ux
Exex (x(x))
E X,
(26)
by utilizing the additivity requirement Exex bx (x) ux =1
Besides, noteworthy that comprehensive elaborations on the concepts within the Subjective Logic theory available in Jøsang (2001; 2016).
A.2 DERIVATION OF OPTIMIZATION OBJECTIVES IN EDL
As aforementioned in section 3.1, to perform model optimization, EDL integrates the conventional MSE loss function over the class probability px which is assumed follow the Dirichlet PDF specified in the bijection, thus derives the optimization objective as
Leat
Σ
EpxxDii(xx.ax) [lly
px
(*.))E
Σ EpxxDir(xx.ax)
Σ [1|2]) (x)))/xx//xxx+>>>>>>>>>> +px
(27)
(*.))ED
zEX
Σ
- 2y Bix.pp + E Du(px.)) b (x))))
(z.v)ED xex
Using the identity E[x=] E[x]] + Var[z], we know that
Ledl
Σ
Σ [ F (xxx + (pxx)
(28)
(EE
14

---

Published as a conference paper at ICLR 2024
Since the Dirichlet distribution has the following properties:
(x(xx)
Varlpx(x)]] ax(())(xx ax(x))
Bxpx(x)]
(29)
Sx
(5x +1)
where Sx =iin ax (z), we can explicitly express Ledl by ax (2) and Sx as
oxl1) 2 ox(x) - ax(()) Sx
Ledl
Σ
y(ai
(30)
(Sx
(z.y)ED ZEX
Furthermore, EDL introduces auxiliary regularization classes by minizing the Kullback-Leible (KL) divergence between a modified Dirichlet distribution and uniform distribution. This regularization term has demonstrated promising empirical results and has been elucidated by Deng et al. (2023) using the PAC- Bayesian theory (McAllester, 1998). Specifically, the regularization term has the following form:
suppress the evidence of non-target
Lu
Σ (2.y)ED
KL ax) 1))
[
T(Sx)
+ Σ -ax - (áx())) - (((x)) xEX
log T(C) ILEex T (áx(()))
D
(2.D)E
(31)
where I denotes the Gamma function, and ãx y (1 y) represents modified Dirichlet parameter vector whose value of the target class has been set 1.
B DERIVATION FOR UNCERTAINTY MEASURES
This section provides the derivation of several uncertainty measures, including expected entropy, mutual information, and differential entropy, Dirichlet-based uncertainty models. The following content is adapted from the Appendix of Malinin & Gales (2018) and Deng al. (2023).
B. EXPECTED ENTROPY
Let X be random variable defined in X, where X is domain consisting of multiple mutually dis- joint values. Let p be a probability distribution over X, and let Dir a) be a Dirichlet distribution parameterized by the concentration parameter vector input sample, x E X = {1,..., C} denotes the value of X, satisfying p(X = x) p(x), then the expected entropy of the random variable X over the Dirichlet distribution Dir(p, a can be derived as follows:
If X represents the category index of an
Dir(p,a) (- ( -E(o))np(s) dp
a)
PENG
DEX
- zEX PENo
Dir(p, a)) (-p(x) (-p())))((()) (npx)))
I(S)
Σ
II (())) (-pp)))))((((()))))))) (np()) dp
IIEE (x))
zEX PEN&
(32)
((x)
T(S)
Σ
Poppte In
p( (((()))())))))))
ZEX PENo S T(a(()) +1) T(a(x')
a(I) aa(1)
Σ
JPENo (p-DDitp..ct, (n
ZEX
a(z)
Σ
5
15

---

Published as a conference paper at ICLR 2024
where S == = alz). is C-dimensional unit simplex, y denotes the digamma function, and 1, denotes a one-hot vector with the æ-th element being set to 1 and other elements being set to O. The last third equation comes from the property of Gamma function that [(n) (n 1)!. In some literature, the expected entropy is used to measure the data uncertainty.
B.2 MUTUAL INFORMATION
In the Dirichlet-based uncertainty methods, the mutual information between the labels y and the class probability p. which can be regarded as the difference between the total amount of uncertainty and the data uncertainty, can be approximately computed as:
1ly. p
H. [ (ir(pppppp. p [p(x)]]
pi--Dip(p.a) [H[p(æ)]]
Distributional Uncertainty
Total Uncertainty
Expocted Data Uncertainty
(z)
a i
(v(a(() 11) +
(33)
S
S
xEX
FEX
(xx)) S
(((2) S
In
(a(()) +1) + +1)
REX
The calculation of the expected data uncertainty utilizes the result of Eqn. 32. The mutual informa- tion often used to measure the distributional uncertainty.
B.3 DIFFERENTIAL ENTROPY
The derivation of the differential entropy of the Dirichlet distribution is given by:
H(Dir(p, a)] = [
JPENo
Dir(p, o:) In Dir(p,
Dir(p,
ImI(()) - Er(a(z)) + "(a(z)) Imp(s)) dp
peNo
zEX
xEX
InI((()))) -n In InT( - Σ (a(()) () [np(()]
DEX
zEX
[n((a())) -I)())))))n((s))))) >< (a(x) 1))(((())) -((().
FEX
zEX
(34)
Differential entropy is also prevalent measure of distributional uncertainty. lower entropy indi- cates that the model yields sharper distribution, whereas higher value signifies more uniform Dirichlet distribution.
C EXPERIMENTAL SETTINGS
C.1 DATASETS
Following Deng et al. (2023), we conduct experiments on the following groups of image classi-
fication dataset: (1) MNIST (LeCun, 1998), FMNIST (Xiao et al., 2017), KMNIST (Clanuwat et al., 2018); (2) CIFAR-10 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2018). CIFAR- 100 (Krizhevsky et al., 2009): (3) mini-ImageNet (Vinyals et al., 2016), CUB (Wah et al., 2011). Within each group, we designate the first dataset in-distribution training data, while utilizing the subsequent ones as OOD data. Moreover, to evaluate the effectiveness of our method on video-modality data, also conduct open-set action recognition experiment by taking UCF- 101 (Soomro et al., 2012) as ID data and HMDB-51 (Kuchne et al., 2011) and MiT-v2 (Monfort al., 2021) OOD data following Bao al. (2021). Below the detailed introductions:
The MNIST (LeCun, 1998) database consists of handwritten digits ranging from 0 to Specifically, MNIST contains 60,000 training images and 10,000 testing images, which have been normalized to fit into 28 28 pixel bounding boxes. We use the proportion of [0.8, 0.2] to partition the training samples into training and validation sets.
16

---

Published as a conference paper at ICLR 2024
FashionMNIST (FMNIST) (Xiao al., 2017) is a dataset designed as more challenging replace- ment for MNIST. Created by Zalando Research, FMNIST features grayseale images of various clothing items such shirts, trousers, sneakers, and bags. The dataset is structured similarly to MNIST, containing 60,000 training images and 10,000 testing images, each of which is 28 x 28 pixels in size. We use FMNIST as OOD data when training models on MNIST.
Kuzushiji-MNN -NNIS (KMNIST) (Clanuwat et al., 2018) is another drop-in replacement for MNIST, consisting of training set with 60,000 handwritten Kuzushiji (cursive Japanese) Hiragana charac- ters and a testing set comprising 10,000 ones. Similar to MNIST, the handwritten characters have been processed to fit into 28 28 pixel resolution grayscale images. We also use KMNIST as OOD data when using MNIST as ID data.
CIFAR-10 (Krizhevsky al., 2009) comprises 60,000 32 x 32 color distributed across 10 distinct classes such as airplanes, birds, cats, ships, and more, with each class containing 6,000 images. Among them, 50,000 are designated for training and the remaining 10,000 for testing. We partition the training images into training and validation sets using a split ratio of [0.95, 0.05).
Street View House Numbers (SVHN) (Netzer et al., 2018) dataset consists of digit images of house numbers from Google Street View. Specifically, it contains 73257 digits for training and 26032 digits for testing. We use SVHN as OOD data when training models on CIFARIO.
SIFAR-100 100 (Krizhevsky et al., 2009) is just like the CIFAR-10, except it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class, We use CIFAR- 100 OOD data when using CIFAR-10 as ID data.
mini-ImageNNt (Vinyals et al., 2016) is designed for few-shot learning evaluation. mini-ImageNet comprises 50,000 84 x 84 color images for training and 10,000 ones for testing, evenly distributed across 100 classes, and these 100 classes are subdivided into sets of 64, 16, and 20 for meta-training meta-validation, and meta-testing tasks, respectively.
The Caltech-UCSD Birds (CUB) (Wah et al., 2011) dataset contains 1 11,788 images of 200 subcate- gories belonging to birds, 5,994 for training and 5,794 for testing. We use CUB as OOD data when using mini-ImageNet as ID data in the few-shot setting.
UCF-101 (Soomro et al., 2012) is action recognition data set of realistic action videos, collected from YouTube. Specifically, UCF-101 contains 13320 videos distributed across 101 action cate- gories. For experiments of video-modality setting, we train models on UCF-101 training split and take its testing set as known samples in inference. Following Bao et al. (2021), despite there exists a few overlapping classes between UCF-101 and the OOD datasets, HMDB-51 and MíT-v2, we do not manually clean the data for standardizing the evaluation.
(Kuehne et al., 2011) is collected mostly from movies, and a small proportion from Prelinger archive, YouTube and Google videos. Specifically, HMDB-51 contains 6,849 clips of 51 action categories, each containing a minimum of 101 clips. We use its testing set as unknown samples in the video-modality setting.
HMDB-51
Multi-Moments in Time (MiT-v2) (Monfort et al., 2021) has 305 classes and its testing split contains 30,500 video samples. We also use its testing set as unknown samples in the video-modality setting.
C.2 IMPLEMENTATION DETAILS
Classical setting. In alignment with Deng et al. (2023), a ConvNet with three convolutional and three dense layers is employed for MNIST, while VGG16 (Simonyan & Zisserman, 2014) serves as the backbone network for CIFAR-10. As Table shows, FMNIST and KMNIST are utilized as OOD data for MNIST, while SVHN and CIFAR- 100 are used for CIFAR-10, 10. We use Softplus as the activation function to keep the collected evidence non-negative. The Adam optimizer is employed with a learning rate of 1 X 10-3, decaying by 0.1 every 15 epochs for MNIST, and learning rate of 1 x 10-4 for CIFAR-10. 10. The
À is set to 0.1, which is selected from the range [0.1:0.1: 1:100 based on the optimal classification accuracy the validation set. The batch size is set to 64, and the maximum training epoch is set to 60 and 200 for MNIST and CIFAR- 10, respectively. Reported results are averaged over runs.
17

---

Published as a conference paper at ICLR 2024
Besides, for the baseline methods which require OOD data in the training phase, ie.,, KL-PN and RKL-PN, uniform noise instead of actual OOD test data is used as OOD training data to ensure a fair comparison as previous works did (Charpentier et al., 2020; Deng et al., 2023).
Table 4: Implementation details of experiments in the classical setting.
ID dataset
OOD dataset
Optimizer
Learning rate
(decay,step) for Ir
Max Epoch
MNIST FMNIST & KMNIST
Adam
x 10-3
(0.1, 15)
60
0.1
CIFAR-IO
SVHN & CIFAR-IO1 100
Adam
x 10
200
0.1
Few-shot setting. Following Deng et al. (2023), we adopt a pre-trained WideResseet28-111 10 network from Yang et al. (2021) to extract features and train a single dense layer for experiments under a challenging few-shot setting the mini- ImageNet dataset, with the testing set of CUB as OOD data. We employ the N-way K-shot setting, with N E {5, 10} and K E (1, 5, 203. Each few-shot episode comprises N random classes and K random samples per class for training, min(15, K) query samples per class from mini-ImageNNt for classification and confidence estimation, and an equivalent number of query samples from the CUB dataset for OOD detection. Reported results are averaged over 10,000 episodes. Note that in the few-shot setting, we perform setting relaxations on I-EDL achieve stronger performances. Softplus is used the activation function to keep evidence non-negative. The LBFGS optimizer is employed with the default learning rate 1.0 for 100 epochs. The hyper- peper-pprrmmter À is also selected on the meta-validation set, as shown Table
Table 5: List of the hyper-paramede À of experiments in the few-shot setting.
Setting
5-Way I-Shot
5-Way 5-Shot
5-Way 20-Shot
10-Way 1-Shot
10-Way 5-Shot
10-Way 20-Shot
0.7
0.2
0.3
0.6
0.7
Noisy setting. Noisy samples are generated by adding zero-mean Gaussian noises with standard deviations of 2..025:0.025:0.2000 to the testing samples of CIFAR-10. The hyper-parameter À is set 0.3, which is selected by the best AUPR score of OOD detection on the clean validation set of CIFAR-10 against the noisy validation set with zero-mean 0.1-SD Gaussian noise.
Video-modality setting. Following Bao al. (2021), explore the open-set action recognition task on UCF-101 101 with 13D as the backbone network. The HMDB-51 11 and MiT-v2 are used as sources of unknown samples. The hyper-paramenn
is set to 0.8, and the batch size set to 8.
Other details. Our model is implemented with Python 3.8 and Py PyTorch 1.12. All experiments are conducted on NVIDIA RTX 3090 GPUs. Source codes are provided in the supplementary material.
D ADDITIONAL RESULTS
D.1 CLASSICAL SETTING
In Table and Table 7, we provide the AUPR and AUROC scores of OOD detection in the classical setting, measured by MP (Max projected probability). UM (Uncertainty Mass), DE (Differential Entropy), and MI (Mutual Information), respectively. Table 8 compares EDL-related works with the temperature scaling method (Guo al., 2017) the classical setting, including results evaluated by the Expected Calibration Error (ECE) with 15 bins and the Brier score. Although temperature scal- ing achieves impressive results when evaluated by the ECE metric, there still exists performance gap with our method on OOD detection ability.
Besides, we believe that employing the AUPR scores for evaluation purposes aligns more closely with our objectives than using ECE or Brier score. As delineated in Section 5.2, our primary cri- terion for assessing confidence estimation is the model's ability in differentiating between correctly classified and misclassified samples, as well as between ID and OOD samples based on the pre- dicted confidence. Despite that ECE is frequently employed to assess the degree of correspondence between the model's is confidence and the true correctness likelihood, confidence distribution accom- panied by a low ECE does not inherently ensure a clear distinction between correct and incorrect predictions. For instance, in balanced two-class dataset scenario, if a binary classifier categorizes
18

---

Published as a conference paper at ICLR 2024
Table 6: AUPR scores of OOD detection in the classical setting, measured by MP (Max projected probability), UM (Uncertainty Mass), DE (Differential Entropy), and MI (Mutual Information).
MNIST KMNIST
MNIST- -MNIIST
Method
MP
UM
DE
MI
MP
UM
DE
MI
EDL
97.02.10.76
96.31:12.03
96.92:10911
96.41=1.85
98. 08.1010.44
98.08.00.42
98.10.10.3 98.09:10.42
I-EDL
98.24::0.44
98.33=+0.24
08.34::0.24
98.33=0.24
98.89=0.28
98.86+0.29
98.8940.29
98.87=0.29
R R-Eu)(OUu))
98.69:10.19
98.69-10.20
98.70:10.19
98169=0.19
99.29
99.29
99.31
99.29:10.12
CIFAR-10 -S-NHH
---R-----IIAA--0O
Method
MP
UM
DE
M1
MP
UM
DE
MI
EDL
78.87+3.500
79.12=33.0 69
78.91#3.54
79.11=3.68
84.30=0.67
14,1840,74
84.32+0.67
84.19=0.74
I-EDL
83.26:22.44
82.96:22.17
83.31#2.47
83.07=2.27
85,3520.69 0.69
84.8410.64
85.40:0.69
84.95:0.65
R (-ED)(Ours)
85.00:11.22
85,00:1..22
85.01:11.14
85.00 1.22
87.72.1.311
87.73
87.61
87.73
Table AUROC scores of OOD detection in the classical setting, measured by MP (Max projected probability), UM (Uncertainty Mass), DE (Differential Entropy), and MI (Mutual Information).
MNIST KMNIST
MNIST-
Method
MP
UM
DE
MI
MP
UM
DE
MI
EDL I-EDL
96.5911..59
1.35
96.4910.80
96.22:11.29
97.39-10.57
97. 57.40:10.54
97.48.1.53
97,43=0.53
08.00=0.26
97,97=0.26
97.99=0.26
97.97=0.26
98.49:0030
98.41=0.39
98.48=0.37
98.42=0.39
R-EDL(OOus)
0.140=0.118
98.39=00.18
98.42:10.18
98.40:10.18
98.99:10.14
98.98:10.14
99.05:1..14
98.98=0.14
CIFARIO-SVVHN
CIFARIO-CCIFARIU
Method
MP
UM
DE
MI
MP
UM
DE
MI
EDL I-EDL
80.64:4.22
81.06=4.52
80.724:4.33
81.05++.551
80.96:t0.81
80.63++.01
80.99=0.83
80.65=1.00
87.58=2.03
86.79=1.35
87.6922.09
87,01
83.55:0.67
82.15=0.500
83.69=0.68
82.44=0.50
R-EDL(OUrs)
87.47
87.47=1.224
87.5410.96
87.47:1.224
85.26:10.66
85.26:10.35
84.90.10.45
85.26
all samples into single class with a consistent confidence output of 50%, the ECE would be zero, yet this result lacks practical significance.
D.2 FEW-SHOT SETTING
Table shows few-shot results of OOD detection measured by more uncertainty metrics. Table 10 compares our method and label smoothing in the few-shot setting, All results consistently demon- strate the superior OOD detection performance of our proposed method.
D.3 NOISY SETTING
We also employ noisy data assess both the robustness of classification and the OOD detection capability of our method with the interference of noise. Following Deng et al. (2023), we generate the noisy data by introducing zero-mean isotropic Gaussian noise the test split of the ID dataset. Table 11 reports the classification accuracy and the AUPR scores for OOD detection across varying levels of Gaussian noise on CIFAR-10. essential to note that these two metrics are not mutually
Table 8: Comparison of temperature scaling method with EDL-related works in the classical setting, including results evaluated by the Expected Calibration Error (ECE) with 15 bins and the Brier score. Downward arrows (L) indicate that lower values correspond to better performance for these metries.
Confidence estimation
OOD detection
Method
ECE
Brier score
AUPR
AURP (SVHN)
AUPR (CIFAR100)
Temperature scaling
1.06+0.10
18.44=0.49
98.89-00.05
81.89+2.19
86.86=0.48
EDL
20.08.11.77
40.68 2.339
97.86.10.17
78.8711..50
84.30.10.67
I-EDL
39.96.10.37
55.3
98.72.10.12
83.26.12.44
85.35
R-EDL(OUur)
3.48
18.1410.51
98.98-10.05
85,00 1.22
87.72.10.31
19

---

Published as a conference paper at ICLR 2024
Table 9: AUPR scores of OOD detection in the few-shot setting, measured by MP (Max projected probability), UM (Uncertainty Mass), DE (Differential Entropy), and MI (Mutual Information).
5-Way I-Shot
10-Way I-Shot
Method
MP
UM
DE
MI
MP
UM
DE
MI
EDL I-EDL
66.7820.12
63.41=00.13
69.0010.12
66.11110.13 59.19:1.099
67.81
67.78-1.12
67.84-0.12
71.79=0.12
74.76=0.13
74.0440.13
74.7040.13
71.00:0.10
71.9540.10
7157=0010
71.95=0.10
R R-EDL(OUrs)
72.91:0.12
7..84=0.133
74.3410.13
74.76:10.13
72.83:10100
73.08:10.10
73.17:10.10
73.08-0.10
5- 5-Way 5-Shot
10-Way 5-Shot
Method
MP
UM
DE
MI
MP
UM
DE
MI
EDL I-EDL
74,46=0.10
76,53=0,14 14
77.40=0.12
16.69=0.13
71 16.00000100
76.28 10
75,74=0,10
76. 16-1000.10
82.04:0.10
82.48=0.10
02,3010.10
82.47:0.100
80.6340.11
$2.29:10110 10
81.06=00099
81,96=0.10
R R-EDL()))))
83.65
84.22 0.10
84.05
84.13.10.10
82.39-10.09
$3.37.10.00
82.98-10.09
83.28 0..28-0.09
5-Way 20-Shot
10-Way 20-Shot
Method
MP
UM
DE
MI
MP
UM
DE
MI
EDL T-EDL
80.01
79.78=0.12
80,35=0.11
79.83=0.12
74.5020.08
76.89=00.09
76.70=0.08
76.86=0.09
84.2920.09
85.40=0.09
85.12:0.09
85.3510.09
81.34:0.07
82.52:0..07
82. 12.0070.07
82.41=0.07
R-EDL(Ours)
84.85=0.09
85.57=0.09
85.43++0.09
85,53=00009
82.22-+0.08
82.72=10007
82.48=0.08
82.68=0.08
Table 10: Comparison label smoothing and R-EDL in the few-shot setting.
5-Way 1-Shot
5-Way 5-Shot
5-Way 20-Shot
Method
MP
En/UM
MP
En/UM
MP
Ent/UM
Label smoothing
72.03++0.2
73.00+0.2
77.17+0.2 83.6510.1
77.11+0.2
76.11+002
75.35-0.2
R R-EDL(Ours)
72.91=1.11
7..84-0.1
84.22:10.1
84.85=0.11
85.57=0.1
exclusive; a robust and reliable classifier should excel in both dimensions simultaneously. While both EDL and Z-EDL methods tend to excel in only one of the metrics, Table 11 and Fig, ((a) clearly present the superior performance of R-EDL in terms of the average of these two key metries.
D.4 VIDEO-MODALITY SETTING
We also assess approach using video-modality samples (Bao et al., 2021; Gao al., 2020), specifically on the open-set action recognition task. Following Bao et al. (2021), we train models on UCF-101 training split and use the testing splits of HMDB-51 and MiT-v2 datasets as unknown sources. Given that the state-of-the-art method DEAR is predicated on EDL, we substitute its EDL implementation with our own R-EDL, version. As evidenced by Table 12, this modification yields enhanced performance, further substantiating the efficacy of R-EDL.
Table 11: Results of the noisy setting for VGG16 on CIFAR-1O, with the generated noisy data as OOD data, averaged over 5 seeds. Noisy samples are generated by adding zero-mean Gaussian noises with standard deviations of [0.025:0.025:0.200 to the testing samples of CIFAR-10.
Noise
0.050
Method
Cls
00D
Avg
Cls
OOD
Avg
Cls
0OD
Arg
Cls
0OD
Avg
EDL
(#1.25)
81.36
(2:0344)
$2.27
66.81
(#2.85
72.18
(=1.03)
59.42
65.80
(#5.255
58.58
(+1.355
67.24
62.91
(=5,49)
46,00
(+:181)
73.08
59.54
T-EDL
(40039)
85.74
(+.055
54.55
70.15
(+1114)
71.24
(= 1.99)
67.00
69.12
(4:2.65)
52.14
(43.12)
77.33
64.74
(+:3.35)
36,80
(+3.71)
83.02
59.91
R- R-EDL((um))
(+0.355
87.54
(+0055)
53.48
70.51
(+1.56)
76.35
(=1.02)
64.19
70.27
(+:303)
60.19
(+1144)
75.33
67.76
(#3.67)
45,70
(+11544
82.97
64.34
SD of Noise
0.125
0.150
0.175
0.200
Method
Cls
00D
Cls
OOD
Avg
Cls
0OD
Avg
Cls
00D
Avg
37.19
77.28
31.55
80.66
27.19
83.75
23,93
86.58
EDL
(+44744)
(#:224)
57.23
(t:3.89)
(=2.86)
56.11
(#:2.77)
(4:3881)
55:47
(=2.21)
(tt4.67)
55.26
T-EDI.
(13,50)
26.86
(144000)
85.88
$6.37
(13336)
20.84
(..4220)
87.38
$4.11
(133044)
17.16
(114366
88.29
52.72
(122711)
15.10
(14.44)
88.96
$2.03
R-EDL(OUu))
(#2.97)
35.16
(=1.49)
87.62
61.39
(t:1.87)
28.47
(=155))
90.50
$9.48
(#:2.08)
24.07
(=:1.86)
92.32
58.20
(=22550)
20.83
(#:2.24)
93.50
57.16
20

---

Published as a conference paper at ICLR 2024
Table 12: Results of video-modality setting for I3D backbone UCF-101, with HMDB-51 and MīT-v2 as OOD data. Results of baselines are reported by Bao et al. (2021).
UCF-101 1CF-101-MMiT-2
Method
Open maF1
Open Set AUC
Open maF1
Open Set AUC
OpenMax
67.85+0.12
74.34
66.22:#0.16
77.76
MC Dropout
71.1310.15
75.07
68.11:10.20
79.14
BNN
71.57:10.17
74.66
68.65-10.21
79.50
SoftMax
73.19+0..17
75.68
68.84-+.223
79.94
RPI. 71.4++0.15
75.20
68. +00000
79.16
DEAR
77.2440.18
77.08
69.98-0.223
81.54
R- R-EDL(Ours)
78.73.10.15
77.94
70.85.10.28
82.26
Cessificttnn
02.0
1022
51
41
X0.00
64:07
AUPP OOD Detection
88.0
18.90
88:0
20.33
56.3
90.04
14.00
2011
87.1
87.63
07.72
27.0
58.7
26,70
87.0
5713
96.6
85.0
90.5
48:22
940
684
62.
96.3
03 11 88.5
305 61 03
11
648 8s 07 36
Value of hyper-parameeerrr A
Value of hype-garrmedee A
Valuee of hypor-parameter
(a) Classification
(b) Confidence estimation
(c) OOD detection
Figure 2: Parameter analysis the syper-parameter X, (a) evaluated by accuracy of classification on CIFAR-10, (b) AUPR score of confidence estimation CIFAR-10, and (c) AUPR score of OOD detection against CIFAR-100, respectively.
D.5 PARAMETER ANALYSIS
Moreover, we further investigate the effect of the hyper- apper-parameter X. As an expanded version of figure 1(b), figure 2 includes the trend of variation in classification accuracy on CIFAR-10, the AUPR score for confidence estimation on CIFAR-10, and the AUPR score for OOD detection on CIFAR-100 as the hyper-parameter À varies from 0.01 1.5. Observation reveals that a smaller generally outperforms the tradition EDL setting where À = 1, indicating that the EDL setting for the prior weight is excessively high, leading suboptimal results. Nonetheless, an excessively small À also has detrimental effects. For instance, setting À to 0.01 results in significant decrease classification accuracy to 83. and a drop in the AUPR score of OOD detection to 85.87%. In this setting, À is ultimately established at 0.1, selected from the range 1:010:1::100 based on the best classification accuracy on the validation set.
D.6 VISUALIZATION UNCERTAINTY DISTRIBUTIONS
Figs. 3,4,5, and 6 show density plots of the normalized uncertainty measures for CIFAR-10 against SVHN, and CIFAR- 10 against CIFAR- 100, while Figs. 7,8,9, and 10 show density plots for MNIST against FMNIST, and MNIST against KMNIST. The uncertainty measures include max projected probability, uncertainty mass, differential entropy, and mutual information. We apply min-max normalization on each uncertainty value и, i.e., Unorm = (u min u) (max u min u). It can be observed that method attaches higher confidence to ID data and makes uncertainty of OOD data more aggregated, exhilarating better separability.
The density plots of I-EDL show different shapes with those of EDL and R-EDL, since I-EDL utilizes the Fisher information matrix to measure the amount of information that the categorical probabilities carry about the concentration parameters of the corresponding Dirichlet distribution, thus allowing a certain class label with higher evidence to have a larger variance. Consequently. the predictions made by the I-EDL approach are typically less extreme, resulting in bimodal distribu-
21

---

Published as a conference paper at ICLR 2024
tion on the uncertainty density plot where the two peaks are closer to the center of the density axis compared to the EDL and R-EDL methods.
Additionally, deduce that the similarity in the shapes of the uncertainty density plots between EDL and R-EDL may stem from the fact that R-EDL's modifications to EDL only consist of relax- ations of non-essential settings, without introducing any additional mechanisms.
E
SOURCE CODE
Our code is available httph::////ee. //tp:///itt// http:///ggg..cccccc///////aa//// REDL.
22

---

Published as a conference paper at ICLR 2024
14
In-distriuuton (CIFAR1O) out-of-distribution (SVHIN)
14
In.distribution (CIFARIO) out-of but-of-ditrrbuudon (SWHN)
14
In-disrrbbution (CIFAR1O)
12-
12
12
out-ofdistributon (SVHN)
AUPR 78.87
AUPR 83.26
AUPR 85.00
Donsity
Density
Donsity
8
8
4-
4-
4
2
2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
max probabiiity
probability
max probability
(a) EDL
(b) Z-EDL
(c) R-EDL
14-
n-distribution (CIFARIO)
14-
In-distribution (CIFARIO) out of-distribution (CIFAR10O)
14
m.distribution
oub oub-ot-detribbttoo (CIFAR100)
12:
out disstrbbbbtnnnn (CIFAR100)
Density
10
Density
10
85.35
Density
10
4-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
max probability
max probability
max probability
(d) EDL
(e) Z-EDL
(I) R-EDL
Figure 3: Uncertainty distribution measured by max projected probability on CIFARIO.
in-distribution (CIFAR1O)
14
In-distrbuution (CIFARLO) oul-of-distributionn (SVHN)
14
in-distribution (CIIAR10)
12
out-ef-iistribuiion (SHHN)
12
12
out-ofdistribution (SVHN)
Density
AUPR 79.12
Density
10
AUPR 82.
Density
AUPR
8
8
4-
4-
4
2-
2
11000000
10L 1 Www www.
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
uncertainty mass
uncertainty
uncertainty mass
(a) EDL
(b) Z-EDL
(c) R-EDL
14.
didistribution (CIFAR10)
14
In nn-disribution (CIFAR1O) out out-of-ditributtion (CIFAR10O)
14
diddstributio FAR10) out out-of-distrbutiio (CIFAR100)
cut-ar-dicrrbbttoon (CIFAR100)
Density
10-
Density
10:
84.84
Density
10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
1.0
0
0.2
0.4
0.6
1.0
uncertainty mass
uncertainty
uncertainty mass
(d) EDL
(c) T-EDL
(f) R-EDL
Figure 4: Uncertainty distribution measured by uncertainty mass on CIFAR10 100.
23

---

Published as a conference paper at ICLR 2024
14
Inddistrbuuton (CIFAR10) out-o--distributio (SVHIN)
14
Indiistributon (CIFAR1O) out-o--bittrbuutionnSS (SVHN)
14
In-distributton (CIFARLO:
12-
12
12
Dut-of-iitributtonn (SVHIN
Donsity
AUPR 78.91
Demsity
10-
AUPR 83.31
Donsity
AUPR 85.01
8-
8-
8
4-
4-
4
2
2
6.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
differential entropy
differential entropy
differential entropy
(a) EDL
(b) Z-EDL
(c) R-EDL
14-
n-distribution (CIFARIO)
14-
indissrrbutionn (CIFAR1O)
14
nndistribution out disstrbuttnnntin (CIFAR100)
12-
oub oub-of-dierrbbutio (CIFAR100)
12-
out-ef-disstibuutoo (CIFAR100)
Density
10
Density
10
85.40
Density
10
4-
4-
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
differential entropy
differential entropy
differential entropy
(d) EDL
(e) T-EDL
(I) R-EDL
Figure 5: Uncertainty distribution measured by differential entropy on CIFAR 10.
in-distribution (CIFAR1O)
14
in-distributin (CIFAR1O) out-of-distribution (SVHN)
14
in-distribution (CIIAR10)
12.
out-ef-distribuiion (SVHN)
12
12
out-ofdistribution (SVHN)
10-
AUPR 79.11
10
AUPR 83.07
AUPR 85.00
Density
Density
Density
8
8
4-
4-
4
2
2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
mutual information
mutual information
mutual information
(a) EDL
(b) Z-EDL
(c) R-EDL
14.
distiiribution (CIFAR10)
14
In ni-distibution (CIFAR10) out out-of-disributtion (CIFAR100)
14
diddstibbutin FAR10) out out-ot-distriuttio ICIFAR100)
out out-of-distribution (CIFAR100)
12
Density
10-
AUPR
Density
10-
84.95
Density
10
4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
mutual information
mutual information
mutual information
(d) EDL
(c) T-EDL
(f) R-EDL
Figure 6: Uncertainty distribution measured by mutual information on CIFAR 10.
24

---

Published as a conference paper at ICLR 2024
14
ii-distribution (MNIST)
14
In-distribution (MNIST)) out out----dittriuuttn (FMNIST)
14
in.iittribution (MNIST)
12-
out-of-distribution (FMNIST)
12
12
out-of-dlstribution (FMMST)
AUPR 98.10
AUPR 98.89
AUPR 99.29
Donsity
Demsity
Donsity
8-
8
8
4-
4-
2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
max probability
probability
max probability
(a) EDL
(b) Z-EDL
(c) R-EDL
14-
In- didditribuuton out cot-tt-diitribution (KMNIST)
14-
In-distribution out suttbtioittiiuution (KMNIST)
14
in-distribution (MNNSS) out disstribbitiiin (KMNST)
12-
12-
Density
10
AUPR
Density
10
98.34
Density
10
AUPR
4-
4-
01
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
max probability
max probability
max probability
(d) EDL
(e) T-EDL
(I) R-EDL
Figure 7: Uncertainty distribution measured by max projected probability on MNIST.
14
In-distribution (MNIST) out-of-distribution (FMNIST)
14
Indiistribttion out-of-distribuuiio (FMNIST)
14
in-dstribution (MNIST)
12-
12
12
out-of-iistrbuttio (FMNIT)
Density
10-
AUPR 98.08
Density
10
AUPR 98.86
Density
AUPR 99.29
8
8
4-
4-
4
2-
2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
uncertainty mass
uncertainty
uncertainty mass
(a) EDL
(b) Z-EDL
(c) R-EDL
14.
Indisstribution (MNIST)
14
In In-distribution out-ot-d stributio (KMNIST)
14
In-distribution (MNIST) out- ous-ct-distribuuton (KMNST)
out-of-llstrbution (KMNIST)
12-
Density
10-
AUPR
Density
10-
98.33
Density
10
AUPR
4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
uncertainty mass
uncertainty
uncertainty mass
(d) EDL
(c) T-EDL
(f) R-EDL
Figure 8: Uncertainty distribution measured by uncertainty mass on MNIST.
25

---

Published as a conference paper at ICLR 2024
14
In-distribution (MNIST) out-of-dstribution (FMNIST)
14
In-distribution (MNIST))
14
indisiiribution (MNIST)
12-
12
out out-o--distriuuton (FMNIST)
12
oub-of-dlstributton (FMNST)
Donsity
AUPR 98.10
Demsity
10
AUPR 98.89
Donsity
AUPR 99.31
8-
8-
8
4-
4-
2
2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
differential entropy
differential entropy
differential entropy
(a) EDL
(b) Z-EDL
(c) R-EDL
14-
in.distribution (MNIST)
14-
in-distribution out out-otdiittibutiio (KMNIST)
14
In-distribution (MNNSS) out didiiiitbbbttion (KMNST))
12-
out-of-listrbution (KMNIST)
12-
Density
10
Density
10
98.34
Density
10
AUPR
6-
4-
4-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
differential entropy
differential entropy
differential entropy
(d) EDL
(e) I-EDL
(I) R-EDL
Figure 9: Uncertainty distribution measured by differential entropy on MNIST.
14
In-distribution (MNIST) out-of-distribution (FMNIST)
14
In-iistribution (MNIST)
14
in-dstribution (MNIST)
12-
12
out-of-distribution (FMNIST)
12
out-of-iistribution (FMNIT)
10-
AUPR 98.09
10-
AUPR 98.87
AUPR 99.29
Density
Demsity
Density
8
8
4-
4-
2-
1100000
www.e
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
mutual information
mutual information
mutual information
(a) EDL
(b) Z-EDL
(c) R-EDL
14.
Indisttibuttoon (MNIST)
14.
In Inddistribution out-ot-dl stribuutnnnno (KMNIST)
14
In-distribution (MNIST) out- ouf-ot-distributton (KMNST)
out-of-llstrbution (KMNIST)
12
Density
10-
AUPR 96.41
Density
10:
98.33
Density
10
AUPR
4
110
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
mutual information
mutual Information
mutual information
(d) EDL
(c) Z-EDL
(f) R-EDL
Figure 10: Uncertainty distribution measured by mutual information on MNIST.
26


Reviews:
[{"review_id": "Gt3UaKMMHv", "reviewer": "Reviewer_ErzN", "summary": "The authors present a relaxation of Evidential Deep Learning (EDL) methods that makes adjustments to the canonical model formulation and loss function. \nEDL has recently gained popularity as an alternative family of uncertainty quantification methods that provides confidence estimates in a single forward pass by having neural networks parameterize distributions over distributions.\nThe authors argue that two details of the standard formulation of EDL are misguided:\nFirstly, they argue that a scalar which gives weight to the prior compared to the evidence is often unnecessarily set to the number of classes in the classification problem, leading to sometimes unintuitive results.\nTo solve this, the authors propose to instead treat it as a task-specific hyperparameter.\nSecondly, they make a case that optimizing prior networks using the expected MSE loss between the predicted probabilities and a one-hot target vector creates over-confident networks due to the variance term in the objective's decomposition.\nHere, they propose to simply optimize the MSE loss directly, without the expectation.\nThey show in their experiments in a classical, few-shot setting for image classification and video action recognition datasets that their method, called R-EDL) able to better indicate the presence of OOD inputs and misclassified examples based on its uncertainty than their tested baselines, all while maintaining a high classification accuracy.", "questions": "My questions are the follows:\n\n* Was there any particular rational in not including Brier score / ECE in the evaluation?\n* What was the reason for including the Posterior Network as a baseline, but not the Natural Posterior Network? (see weaknesses section)\n* Even after consulting the appendix, it is unclear to me if and which regularization term has been used to trained the models (since e.g. Sensoy et al. [2] use an additional KL divergence term). Given the number of options for regularization (see e.g. [3] for an overview) it feels necessary to know which regularizer had been used. In any case, this part would deserve a short mention in the main text. If no regularizer had been used at all, I would find the good results for the OOD detection task surprising, since many other works lament that uncertainty on OOD behaves differently for EDL than desired.\n* Judging from Table 3, I find it a bit strange that either fixing lambda = 1 or re-adding the variance term to the loss function seem to have only minor effects on the results of R-EDL compared to the EDL baseline. This implies to me that a) there might be an interaction between the two components that might go beyond the sum of its parts or that b) the EDL baseline might be tuned suboptimally compared to the proposed R-EDL model. I would be interested in the authors commenting on this aspect.\n\nOther notes:\n* The link to the appendix after the contributions in section 1 seems to be missing.\n\n[2] Sensoy, Murat, Lance Kaplan, and Melih Kandemir. \"Evidential deep learning to quantify classification uncertainty.\" Advances in neural information processing systems 31 (2018).\n\n[3] Ulmer, Dennis, Christian Hardmeier, and Jes Frellsen. \"Prior and posterior networks: A survey on evidential deep learning methods for uncertainty estimation.\" Transactions on Machine Learning Research (2023).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper presents its ideas well: Despite the proposed changes being relatively minor in nature, it motivates them with illustrating examples and reiterates EDL's basis in subjective logic to motivate them.\n* The presentation of the paper is polished and it is generally well written, albeit that earlier sections can feel slightly verbose at times.\n* The paper compiles a number of relevant baselines that are used for comparison in the presented experiments.\n* The authors test their approach in a classical, few-shot and noisy setting along with an ablation study.\n* Experimental details for possible replications are documented well in the appendix.", "weaknesses": "* It would have felt more natural to me to include the Natural Posterior Network [1] instead of / besides the Posterior Network as a baseline, since it boasts better results than its predecessor.\n* Regarding the creation of a new hyperparameter, I do appreciate the authors' ablation study in 5.5. However, the investigation, given that this step is one of the key contributions of the paper, feels a bit shallow. I would be interested to understand better how to interpret the best values found in the context of the authors' original argument and to distill any trends. I found Figure 1b) particularly puzzling, since lambda seems to have major effect on classification performance, but less so in terms of actual OOD detection (~2 points AUPR). A similar graph about misclassification detection, which seems to align better with the authors original arguments, is not given.\n* Although the second contribution - effectively deprecating the variance term in the loss function - is supposed to help overconfidence, confidence is never directly evaluated. This might be due to the common conflation of the terms uncertainty and confidence, with the latter often referring to the actual probability assigned to predicted class - however works like the aforementioned Charpentier et al. [1] evaluate both the quality of uncertainty estimates obtained through different EDL metrics as well as the direct confidence of their model through the Brier score. The authors do try to assess confidence through measuring the AUPR for their misclassification task (including the MP value, i.e. the maximum probability across all classes), which I would argue is appropriate, but assesses confidence only in its ability to discriminate between correctly and incorrectly classified samples, not in its degree to correspond to the (in expectation) chance of classifying correctly, like in the instance of the expected calibration error.\n\n[1] Charpentier, Bertrand, et al. \"Natural Posterior Network: Deep Bayesian Predictive Uncertainty for Exponential Family Distributions.\" International Conference on Learning Representations. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EbPAzVbYlF", "reviewer": "Reviewer_yPVK", "summary": "The authors focus on improving the popular Evidential Deep Learning approach introduced by Sensoy et al. (2018). They identify two potential sources of improving it, (i) the prior weight by turning it into a hyperparameter, and (ii) the variance term in the objective loss, by removing it. The approach is evaluated in a series of experiments.", "questions": "- Can the authors report additional calibration results? E.g., by reporting the expected calibration error (Guo et al., 2017)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written with a clear line of thought and additional implementations.\n- Both modifications, prior weight, and removal of variance regularization, are motivated theoretically and their respective influence is validated via an ablation study.\n- The improvements are evaluated in a series of experiments", "weaknesses": "- The experiments lack an in-distribution calibration evaluation\n\n\n\n### Minor\n- Typo: The first paragraph on p7 KL-PN is mentioned twice", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4MLe3bwJae", "reviewer": "Reviewer_vQHP", "summary": "The paper improves the Evidential Deep Learning (EDL) approach, a method for performing classification with reliable uncertainty estimations, in two ways: i) adopting a more generic definition of evidence that also incorporates prior belief, ii) removing the Kullback-Leibler (KL) divergence based regularizer term and employing a prior weight term for regularization that emerges naturally from this generalized definition.", "questions": "* While the paper justifies all the steps up till Equation 10 rigorously, the choice of the loss in Equation 11 looks a bit arbitrary. Why should one choose the MSE between the class probabilities and one-hot labels instead of the simple and straightforward cross-entropy loss? How would it work numerically? If it works better or worse, what is the interpretation of the outcome?\n\n* Why were the MP and UM scores chosen as the performance metrics to measure in-distribution performance instead of the well-established Expected Calibration Error (ECE)?\n\n* How does the method compare against post-hoc calibration algorithms such as temperature scaling?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed approach for improving EDL is novel, creative, and innovative. It is also built on solid theoretical foundations. The resulting loss function is derived rigorously from first principles and clearly identifies the limitations of EDL.\n\n* This is a high-quality piece of work that not only develops a novel method, but also illustrates its development steps and the rationale of each step unequivocally, as well as demonstrating the practical benefit of the end outcome with a large body of experiments. The proposed method brings a clear performance improvement.", "weaknesses": "*  The paper below addresses the over/under regularization problem of EDL using an alternative approach, strictly speaking via Bayesian inference: **“Kandemir et al., Evidential Turing Processes, ICLR, 2022”**. It could be worthwhile to clarify the advantage of the proposed approach with respect to this alternative.\n\n* The phrase “multiple mutually disjoint values” in Definition 1 is a little confusion and potentially imprecise. A better alternative could be **“Let X be a categorical random variable on the domain $\\mathbb{X}$”**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rYb533StpZ", "reviewer": "Reviewer_5NaH", "summary": "The paper proposes new tweaks to the existing framework of evidential deep learning. These tweaks include treating a pre-set parameter as a tuning hyperparameter. The paper also uses an alternative objective to optimize for evidential deep learning.\n\nThe paper provides mathematical justification for the design choices. The paper also run experiments to demonstrate that the tweaks are effective.", "questions": "* \"Despite its great success\" sentence in the abstract is difficult to parse. Consider breaking it into a few sentences.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "originality: the paper provides incremental improvement toward the existing evidential deep learning framework.\nquality: the proposed modification seems solid both mathematically and empirically. I particularly like that the paper carried out fairly exhaustive experiments to measure the utility of the proposed method across various tasks, comparing the proposed method with various alternatives. The ablation study is also a nice addition.\nclarity: the paper provides a mathematical formula and associated explanation to justify the tweaks of the framework presented in the paper. I can follow the arguments made in the paper on the high level.\nsignificance: the proposed tweaks appeared to make the framework more pratical and effective.", "weaknesses": "* presentation can be improved. \n  * for example, more intuition about some of the key concepts of the paper can be provided such as subjective opinion, belief mass, uncertainty mass, and Dirichlet distribution.\n  * In the experiments, max probability and UM are used as confidence scores. However, it is not clear how to interpret them and what is a good score and a bad one.\n  * Table 1 caption mentioned AUPR score, but it does not seem to be actually reported.\n* Consider the prior weight as hyperparameter also means that the algorithm requires hyperparameter tunning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O46yJl0Hcb", "reviewer": "Reviewer_R4i1", "summary": "The authors propose a novel R-EDL method to recalibrate uncertainty estimation by relaxing nonessential settings of EDL.", "questions": "as aforementioned", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- An interesting perspective to explore uncertainty estimation\n\n- Comprehensive experiments", "weaknesses": "- Could you present the detailed derivation of Formulas 7,9 and 10?  These formulas are the essential steps, but I may not catch up with your idea.  \n\n- The authors are recommended to make necessary explanations on different shapes of I-EDL with EDL and R-EDL in Figure 2 and Figure 5.\nAdditionally, uncertainty representations for ID (MNIST) and OOD (FMNIST) are better to show the differences among EDL, I-EDL, and R-EDL, as presented in I-EDL (figure 4). \n\n- What role did W play in the process of uncertainty estimation or evidence collection?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)