Title: Hyper-opinion Evidential Deep Learning for Out-of-Distribution Detection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Out-of-distribution Detection

Evidential Deep Learning

Uncertainty Estimation.
Abstract: Evidential Deep Learning (EDL), grounded in Evidence Theory and Subjective Logic (SL), provides a robust framework to estimate uncertainty for out-of-distribution (OOD) detection alongside traditional classification probabilities.However, the EDL framework is constrained by its focus on evidence that supports only single categories, neglecting the other collective evidences that could corroborate multiple in-distribution categories. This limitation leads to a diminished estimation of uncertainty and a subsequent decline in OOD detection performance.Additionally, EDL encounters the vanishing gradient problem within its fully-connected layers, further degrading classification accuracy.To address these issues, we introduce hyper-domain and propose Hyper-opinion Evidential Deep Learning (HEDL). HEDL extends the evidence modeling paradigm by explicitly integrating sharp evidence, which supports a singular category, with vague evidence that accommodates multiple potential categories.Additionally, we propose a novel opinion projection mechanism that translates hyper-opinion into multinomial-opinion, which is then optimized within the EDL framework to ensure precise classification and refined uncertainty estimation.HEDL integrates evidences across various categories to yield a holistic evidentiary foundation for achieving superior OOD detection. Furthermore, our proposed opinion projection method effectively mitigates the vanishing gradient issue, ensuring classification accuracy without additional model complexity. Extensive experiments over many datasets demonstrate our proposed method outperforms existing OOD detection methods.

Hyper-opinion Evidential Deep Learning for Out-of-Distribution Detection

Abstract

Evidential Deep Learning (EDL), grounded in Evidence Theory and Subjective Logic (SL), provides a robust framework to estimate uncertainty for out-ofdistribution (OOD) detection alongside traditional classification probabilities. However, the EDL framework is constrained by its focus on evidence that supports only single categories, neglecting the other collective evidences that could corroborate multiple in-distribution categories. This limitation leads to a diminished estimation of uncertainty and a subsequent decline in OOD detection performance. Additionally, EDL encounters the vanishing gradient problem within its fullyconnected layers, further degrading classification accuracy. To address these issues, we introduce hyper-domain and propose Hyper-opinion Evidential Deep Learning (HEDL). HEDL extends the evidence modeling paradigm by explicitly integrating sharp evidence, which supports a singular category, with vague evidence that accommodates multiple potential categories. Additionally, we propose a novel opinion projection mechanism that translates hyper-opinion into multinomialopinion, which is then optimized within the EDL framework to ensure precise classification and refined uncertainty estimation. HEDL integrates evidences across various categories to yield a holistic evidentiary foundation for achieving superior OOD detection. Furthermore, our proposed opinion projection method effectively mitigates the vanishing gradient issue, ensuring classification accuracy without additional model complexity. Extensive experiments over many datasets demonstrate our proposed method outperforms existing OOD detection methods.

1 Introduction

Deep Learning (DL) models have been widely adopted in many real-world applications[25, 57, 64, 15]. However, these models are trained under the implicit assumption that the training and test data are

drawn from the same distribution[70], leading to overconfident predictions[45]. Thus when a DL model encounters an input that differs from its training data, it may be overconfident with wrong prediction, bringing rise to the out-of-distribution (OOD) problem. The resolution of the OOD problem is of utmost importance, and researchers have devoted significant attention to studying the intricacies of OOD detection[5, 16, 19, 30, 31, 43].

To address OOD problem, a variety of methods have been developed in DL[12, 4, 51]. Some researchers apply post-processors to the base classifier to generate an uncertainty score for OOD detection. These post-hoc methods only take effect at inference phase and are easy to use, but rely on the performance of the pretrained model. Others propose training methods that involve training-time regularization, which require more computational resources. To train an uncertainty-aware model without additional computation, a recent search leverages Evidence Theory and Subjective Logic (SL) with DNNs[54], called Evidential Deep Learning (EDL)[55, 24, 54, 7]. EDL offers uncertainty estimation in neural networks which represents the degree of 'unknown' in opinion. It modifies the existing DL structure slightly and allows neural network to quantify the uncertainty for OOD detection with a well-defined theory framework. Evidential models have been extended to many areas such as open set recognition[2], classification[35, 32, 22, 36, 33], multi-view learning[72, 68, 23, 34].

The EDL models face several challenges, with one primary issue arising from the theoretical framework. The evidence in multinomialopinion in EDL exclusively supports singleton sets, which contains only one category. In other words, EDL only captures the evidence which supports single category and rejects others. As a result, EDL is unable to effectively leverage vague evidence, such as features supporting a composite set containing multiple categories. As Figure 1 shows, EDL suffers from performance degradation in the face of ambiguous samples.

In addition, the parameters of fully-connected layer in EDL models are facing vanishing gradient problem when number of category in datasets rises[49]. Vanishing gradient in EDL leads to failure in classification of several categories. To mitigate this problem, Pandey et al.[49] introduce regularization techniques. However, these efforts yield unsatisfactory results in real world OOD detection tasks.

To train an evidential model maintaining classification accuracy and providing reliable uncertainty estimation for OOD detection, we incorporate EDL with hyper-opinion and propose Hyper-opinion Evidential Deep Learning (HEDL). While EDL is built upon multinomial-

Uncertainty of ID samples in EDL

Uncertainty of ID samples in HEDL

class A

classB

Traditional EDL

Hyper-opinion EDL

1.0

0.8

Mass

0.6

0.4

0.2

0.0

SharpforA<Vague->

Sharp for B

Sharp forA<Vague-->

Sharp for B

sharp belief for A

sharpbelieffor B

11

vaguebelieffor{A,B}

sum of belief

uncertainty

opinion in a basic domain, hyper-opinion represents the opinion in the hyper-domain, which includes the basic domain and the composite sets. Through the concepts of composite set, HEDL is able to learn from vague evidence ignored by EDL. HEDL provides an effective mechanism for quantifying evidence that supports composite sets, thereby enhancing the differentiation of OOD data and classification accuracy. Our major contributions can be summarized as follows:

We introduce an evidential representation within the hyper-domain, which integrates sharp evidence that supports a singular category, with vague evidence that accommodates multiple potential categories, to establish a more comprehensive and accurate evidentiary foundation.

We develop a hyper-opinion framework within the hyper-domain and propose a novel opinion projection. This method transfers hyper-opinion to multinomial-opinion, allocating evidence to each category precisely and mitigating the vanishing gradient problem, while preserving computational efficiency.

Data

feature extractor

ID data

opinion

hyper

一1

multinomial

opinion

probabilitiesP

classification

featurex

evidencey

beliefH

beliefk

>(

uncertaintyu

detection

(a)Hyper-opinionBelief

(b）Opinion Projection

OODdata

(a) Hyper-opinion Belief

(b) Opinion Projection

(c) Multinomial-opinion Optimization

f2

f

f3

activate

→

e

eH

build

b

b望

C,C3}

[C,C2}

[C2,C3}

bk=

>

H(Wxx）b

b→{C}

projected from hyper-opinion

Dir(Pla)

f4

fs

→

→

eH

H

b

b号

[C}

H()is Heaviside function

beliefH

beliefk

b→{C3}

b2→{C2}

u

a=（a，a,a)

b

bz

PP2P3u

>L(O)

Our proposed Hyper-opinion Evidential Deep Learning (HEDL) procures more exhaustive evidence, which refines the precision of uncertainty estimation, and consequently enhances the performance of OOD detection while maintaing ID classification accuracy.

We carry out experiments over multiple challenging datasets to validate the OOD detection in HEDL outperforms existing OOD detection methods.

2 Related Work

2.1 Uncertainty based OOD Detection

Accurately quantifying predictive uncertainty in DL models is crucial for recognizing out-ofdistribution (OOD) samples. Traditional softmax-based models provide confidence estimation through class posteriors, which are inversely correlated with predictive uncertainty[16]. Several methods applicable to pre-trained classifiers that output class posteriors using softmax have been proposed[3, 14, 53, 37, 60, 18], including Out-of-Distribution Detector for Neural Networks[31] and Mahalanobis Distance[30]. Besides, deep ensemble is a technique developed for uncertainty quantification[29], which constructs an ensemble of neural networks and measures uncertainty based on the agreement/disagreement across the ensemble components[13]. However, this approach significantly increases the scale of model parameters, leading to high computational and storage complexity. Alternatively, neural networks based on Bayesian statistics called Bayesian neural networks[12, 4, 42] is raised to to quantify different uncertainties in Bayesian formalism. Bayesian methods normally apply approximation to address the intractability issue in marginalization of latent variables. And as such methods require sampling for uncertainty quantification, leading to expensive computations. A recent research effort has summarized OOD detection methods and established an OOD benchmark [69].

2.2 Evidential Deep Learning

EDL introduces a conjugate higher-order evidential prior for the likelihood distribution that enables the model to capture the evidence vacuity as predictive uncertainty. The training of an EDL model can be regarded as an evidence-collecting process. Researches on multiple applications with EDL have been done, e.g., Dirichlet prior is introduced over the multinomial likelihood for evidential classification[2, 73, 11], evidential models for regression[1, 48], adversarial robustness[27] and calibration[63]. Most existing methods built upon EDL are trained on evidential losses conjunct with regularization of the evidence to guide the evidence vacuity, i.e. , uncertainty, behavior[47, 56]. Some EDL models combine with the idea of outlier exposure[17] that provides access of OOD data to guide the evidence learning process of EDL models[40, 41].

In this work, we focus on evidential models for classification and OOD detection, and consider settings where no extra regularization and OOD data are used during model training to make the proposed approach more broadly applicable to practical real-world situations.

3 Proposed Method

Our method's framework is depicted in Figure 2, which operates under the assumption of no prior information.

3.1 Hyper-opinion Belief

Subjective Logic (SL) is a theory of uncertain reasoning based on probability theory and belief theory in a domain X , which represents the set of exclusive possible states of a variable situation. It introduces the concepts of belief mass and uncertainty mass to describe the degree of belief and uncertainty about an event.

Traditional EDL is built upon multinomial-opinion within domain X in SL and domain X is a limited portion of hyper-domain R ( X ) , where P ( X ) is the powerset of X .

Let us consider a domain X with cardinality of K , SL provides a belief mass b k representing the belief degree and a base rate a k representing the prior information for each singleton k = 1 , ..., K and an overall uncertainty mass of u . The three compose a multinomial-opinion ω = ( b , u, a ) , belief mass and uncertainty mass sum up to one, eg.,

Our method models the evidence in hyper-domain R ( X ) with hyper-opinion, which provides a belief mass b H x , x ∈ R ( X ) , representing the belief degree of set x . Along with a H and u , the three compose a hyper-opinion ω H = ( b H , u, a H ) and the hypernomial belief mass distribution also follows the additivity requirement:

Hyper opinion allows belief mass to be divided into two types called sharp belief mass and vague belief mass. Belief mass that only supports a specific singleton is called sharp belief mass, eg., k ∈ X , it discriminates between this and other singletons. EDL built upon the multinomial-opinion only offers sharp belief mass estimation. Considers a domain X of K mutually exclusive singletons, for each singleton k = 1 , ..., K , sharp belief mass is

Belief mass assigned to a composite set x ∈ C ( X ) , where C ( X ) = R ( X ) / X , represents vague belief mass because it expresses cognitive vagueness. It supports the truth of multiple singletons in X simultaneously. Vague belief mass can be allocated to a singleton k as

where a ( k | x ) is relative base rate. When no prior information is available, a ( k | x ) can be simplified to

where | x | is the cardinality of x . Then in hyper-opinion, a belief mass b H x for a set x is computed using the evidence for the set. Let e H x ≥ 0 be the evidence derived for the set x , then the belief b H x and the uncertainty u are computed as

By introducing hyper-opinion, vague beliefs that assigned to composite sets can be take into consideration, which better measure comprehensive evidence and estimate uncertainty more accurately.

In practice, we activate the features extracted by the neural network as evidence in hyper-domain, and build them within hyper-opinion to distinguish sharp belief and vague belief. This allows the model to maintain its vagueness among similar in-distribution categories, thereby ensuring that the uncertainty remains low.

3.2 Opinion Projection

A projection from hyper-opinion to multinomial-opinion is needed to realize the projected probability of each singleton. Therefore we introduce a novel opinion projection implementation that projects belief mass from hyper-opinion into multinomial-opinion, with b V k and b S k that can be calculated by Eq. 4 and Eq. 5, following

We activate the features extracted by neural network for ascertaining non-negative evidence within the hyper-domain. After associate evidence with belief in hyper-opinion, we determine the set each belief mass supports as mentioned in section 3.1, and project the belief mass from hyper-opinion to multinomial-opinion.

Specifically, we apply a unit step activation function to the parameters of the fully connected layer, eg., Heaviside function

It offers an access to a matrix W S = H ( W ) , where W corresponds to the weight matrix of the fully connected layer. W S represents the information of set each belief mass supports.

Assume there are K singletons and N belief masses supporting different sets, it offers a matrix W S N,K . For a belief mass b H x supporting set x , W S x is a vector that contains information about which singletons belong to the set x .

Once the set each belief mass supports has been identified, projecting hyper-opinion to multinomialopinion is straightforward. For each belief mass within the hyper-opinion, we can compute its relative base rate to each singleton, and allocate belief mass accordingly. For a singleton k , its total projected multinomial-opinion belief mass is

where a is the base rate. Without any prior information, Eq. 11 can be simplied to

To date, we have successfully delineated the process of projecting belief mass from a hyper-opinion to a multinomial-opinion within a neural network framework. In practical terms, this projection is executed by applying a linear transformation to the output of the fully connected layer. This transformation facilitates the allocation of belief mass to the respective singletons in the multinomialopinion. Consequently, the incremental computational complexity associated with our method is constant as O(1).

where o is the output of fully-connected layer and W,W p , b H are all detached variables, making G ( W, b H ) a constant during one training epoch.

The output after opinion projection represents the projected multinomial-opinion in EDL, which has the equivalent meaning in EDL and can be optimized with the same techniques. We used an example to show why the uncertainty estimation of HEDL outperforms EDL in Appendix A.

3.3 Multinomial-opinion Optimization

By building evidence within hyper-domain and projecting hyper-opinion belief mass into multinomialopinion belief mass, we construct a flow that can be optimized in multinomial-opinion framework to obtain the comprehensive evidence and accurate uncertainty estimation for OOD detection, which is similar to traditional EDL.

As the sum of evidence ∑ x ∈R ( X ) e H x and uncertainty u remain the same during the projection, we can pass the belief mass in the form of evidence to simplify the calculation. Therefore the projected probability distribution derived from the projected multinomial-opinion can correspond to an expected probability distribution derived from a Dirichlet distribution parameterized by α

The Dirichlet distribution is a probability density function (pdf) for possible values of the probability mass function (pmf) P and is given by:

In projected multinomial-opinion, the expected probability for the k th singleton calculation is

which allows to be optimized by the loss function defined in EDL

̸

where ψ ( · ) is the digamma function, y i is a one-hot vector encoding the ground-truth class of observation x i with y ij = 1 and y ik = 0 for all k = j , and α i be the parameters of the Dirichlet density on the predictors.

At this point, we have established the complete framework of HEDL, spanning all stages ranging from input processing to classification and uncertainty estimation. Our method objective has the following proposition in the Appendix B.

By establishing the framework of HEDL, we comprehensively extract the sharp and vague evidence each sample contains and allocate preciously, thereby enabling accurate classification. Moreover, comprehensive evidence contributes to improved uncertainty estimation and subsequently enhances the performance of OOD detection.

4 Experiment

In this section, we describe our experimental setup and demonstrate the effectiveness of our method on a wide range of OOD evaluation benchmarks and the most widely used metric AUROC is adopted[52, 21, 10, 37]. We also conduct an ablation analysis that leads to an improved understanding of our approach.

4.1 Setup

In-distribution Datasets. We use the CIFAR-10[28], CIFAR-100[28], Flower-102[46] and CUB200-2011[65] as ID data.

Out-of-distribution Datasets. For the OOD test datasets, we use three common benchmarks[69]: SVHN[44], Textures[6], Places365[74], that are used in Openood-benchmark[69]. There is no overlapping between ID datasets and OOD datasets.

Evaluation Metrics. We measure the following metrics: 1) FPR95 measures the false positive rate (FPR) when the true positive rate (TPR) is equal to 95%. Lower scores indicate better performance. 2) AUROC measures the area under the Receiver Operating Characteristic (ROC) curve, which displays the relationship between TPR and FPR. The area under the ROC curve can be interpreted as the probability that a positive ID example will have a higher detection score than a negative OOD example. 3) AUPR measures the area under the Precision-Recall (PR) curve. The PR curve is created by plotting precision versus recall. AUROC is the most common metric[52, 21, 10, 37] and we use AUROC as the main metric for OOD detection performance while accuracy measures performance of detecting ID samples. Our goal is to detect more OOD samples while maintaining ID classification performance.

Implementation Details. We follow the experiment settings outlined in OpenOOD[69]. We use ResNet-18[15] for CIFAR-10 and CIFAR-100. For more intricate datasets that are not included in OpenOOD[69], such as fine-grained datasets Flower-102 and CUB-200-2011, we employ ResNet34[15] for enhanced feature representation. All experiments are implemented with PyTorch[50] and carried out with NVIDIA GeForce RTX 3090 GPU. We use the standard data split for all datasets, and the number of training epochs is 100, the initial learning rate is 0.0001 with AdamW[39], and the batch size is 128. At test time, all images are resized to 224 × 224. For HEDL model, we first train the feature extractor with softmax layer for 90 epochs and then train in HEDL framework for 10 epochs. HEDL does not introduce any additional hyperparameters, thereby eliminating the need for extensive hyperparameter tuning, and W prior is set to 1 for HEDL.

Baseline Methods. We compare our method with several classical and state-of-the-art OOD detection methods. Specifically, we compare our method with post-hoc inference methods and training methods. From MSP[16] to GEN[38] are post-hoc inference methods, which affect OOD detection performance only and do not change model accuracy. The others are training methods. We excluded methods that required auxiliary OOD data due to the practical real-world situations consideration. We leverage selected experimental results from OpenOOD[69] to demonstrate the effectiveness of our approach.

4.2 OODDetection Results

The comparative results on CIFAR-10 and CIFAR-100 are detailed in Table 1, and the results on Flower-102 and CUB-200-2011 are shown in Appendix C. For each model, we utilize three OOD datasets, thereby aiming to achieve more realistic and generalized outcomes. We reveal a common challenge: when confronted with more complex data scenarios, training methods struggle to maintain both accuracy and OOD detection capabilities simultaneously. However, HEDL consistently achieves better OOD detection performance than existing state-of-the-art OOD detection methods while preserving the accuracy of ID classification, even under complex data scenarios. Notably, HEDL accomplishes this enhancement without additional regularization strategies or hyperparameters, indicating strong generalization ability on different datasets, it also avoids incurring higher computational costs. The experimental training time analysis of HEDL can be found in Appendix D.

4.3 Gradient Analysis

The gradient norms of fully-connected layer parameters over EDL and HEDL during training is shown in Figure 3, alongside the final accuracy for each category. The sum of these gradient norms has been normalized for comparative analysis. It is observed that the gradient norms for several parameters within the fullyconnected layer of the EDL model remain zero throughout the training process, which correlates with a significantly lower final accuracy for these categories. This outcome is indicative of the vanishing gradient problem. Conversely, HEDL does not experience this issue, demonstrating that our proposed method effectively circumvents the challenge of vanishing gradients within the fully-connected layer.

4.4 Ablation Study

EDLsuffers from a notable decline in both ID accuracy and OOD detection when facing a proportional rise in the volume of vague evidence. In contrast, HEDL demonstrates the capability to consistently extract comprehensive evidence and maintain its performance regardless of the dataset scale.

We investigate the performance of our method with ablation experiments on two challenging finegrained datasets. The fine-grained datasets contain more vagueness among categories and can better prove the effectiveness of our methods. We conducte ablation experiments on the effects of hyperopinion and opinion projection, respectively. Note that opinion projection can only be built upon hyper-opinion.

Figure 4 illustrates the uncertainty distribution of ID and OOD samples across different datasets for EDL, HEDL without opinion projection, and HEDL itself. Notably, on the latter three more complex datasets, the approaches based on hyper-opinion exhibits a distinct performance advantage. It is also

EDL on Cifar100

1.0

0.8

0.6

0.4

0.2

0.0

20

40

Category

09

80

100

HEDL on Cifar100

1.0

0.8

0.6

0.4

0.2

0.0

D

20

40

Category

60

80

100

Accuracy (Gradient Vanishing Points)

Accuracy (Others)

Gradient Norm Sum

EDL

HEDL w/o projection

HEDL

30

OE

30

In-Distribution

In-Distribution

In-Distribution

25

Out-of-Distribution

25

Out-of-Distribution

25

Out-of-Distribution

20

20

20

Density

Density

Density

15

15

15

10

10

10

5

5

5

0

0.0

0.2

0.4

0.6

0.8

1.0

0

0.0

0.2

0.4

0.6

0.8

1.0

0

0.0

0.2

0.4

0.6

0.8

1.0

Uncertainty

Uncertainty

Uncertainty

EDL

HEDLw/oprojection

HEDL

30

30

30

In-Distribution

In-Distribution

In-Distribution

25

Out-of-Distribution

25

Out-of-Distribution

25

Out-of-Distribution

Density

20

Density

20

Density

20

15

15

15

10

10

10

5

5

5

Q

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

8'0

1.0

D

0.0

0.2

0.4

0.6

0.8

1.0

Uncertainty

Uncertainty

Uncertainty

EDL

HEDL w/o projection

HEDL

14

In-Distribution

14

In-Distribution

14

In-Distribution

12

Out-of-Distribution

12

Out-of-Distribution

12

Out-of-Distribution

10

10

10

Density

Density

Density

8

8

8

6

6

6

4

4

4

2

2

2

0

0.0

0.2

0.4

0.6

0.8

1.0

0

0.0

0.2

0.4

0.6

0.8

1.0

0

0.0

0.2

0.4

0.6

0.8

1.0

Uncertainty

Uncertainty

Uncertainty

EDL

HEDLw/oprojection

HEDL

14

In-Distribution

14

In-Distribution

14

In-Distribution

12

Out-of-Distribution

12

Out-of-Distribution

12

Out-of-Distribution

Density

10

Density

10

Density

10

8

8

8

6

6

6

4

4

2

2

2

Q

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Uncertainty

Uncertainty

Uncertainty

(d) CUB-200-2011, the overlap between ID and OOD is 50%, 20%, and 17% for EDL, HEDL w/o projection, and HEDL, respectively.

worth observing that, in these datasets, instances of ID data with maximum uncertainty are present in the EDL model. This phenomenon can be attributed to the failure of extracting evidence of those categories due to the vanishing gradient problem.

Table 2 shows that evidence built on hyper-opinion can be considered comprehensively, leading to accurate uncertainty estimation and above baseline OOD detection performance. But without the correct projection from hyper-opinion to multinomial-opinion, vague evidence can not be assigned precisely, leading to inaccurate classification.

5 Conclusion

In this paper, we propose Hyper-opinion Evidential Deep Learning (HEDL), a novel approach designed to generate precise uncertainty estimation for Out-of-Distribution (OOD) detection. Our method encapsulates a comprehensive representation of evidence within hyper-opinion, which allows model to preserve its vagueness among In-Distribution categories to reject OOD data.

Additionally, by projecting hyper-opinion to multinomial-opinion, HEDL circumvents the vanishing gradient problem encountered in the fully-connected layers of traditional EDL. This projection is optimized within an established framework, yielding accurate and reliable evidence. Notably, our method

accomplishes superior OOD detection performance while simultaneously upholding classification accuracy without incurring additional computational complexity. Extensive experimental results across numerous datasets substantiate the efficacy of the proposed Hyper-opinion Evidential Deep Learning.

Limitations and societal impact. Our proposed HEDL method achieves best performance by transfering learning on pre-trained models. In future work, it is necessary to reduce the dependence on pre-trained models and explore alternative approaches. This work aims to improve the safety of deep learning models, which tends to benefit a wide range of applications of AI in social life.

Acknowledgments and Disclosure of Funding

This work was supported by the National Natural Science Foundation of China (No. 62173252, 62476165).

References

Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems , 33:14927-14937, 2020.

Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1334913358, 2021.

Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1563-1572, 2016.

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning , pages 1613-1622. PMLR, 2015.

Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Informative outlier matters: Robustifying out-of-distribution detection using outlier mining. 2020.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606-3613, 2014.

Danruo Deng, Guangyong Chen, Yang Yu, Furui Liu, and Pheng-Ann Heng. Uncertainty estimation by fisher information-based evidential deep learning. arXiv preprint arXiv:2303.02045 , 2023.

Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In International Conference on Learning Representations , 2022.

Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. In International Conference on Learning Representations , 2021.

Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems , 34:7068-7081, 2021.

Wei Fu, Yufei Chen, Wei Liu, Xiaodong Yue, and Chao Ma. Evidence reconciled neural network for out-of-distribution detection in medical images. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 305-315. Springer, 2023.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

Mudasir A Ganaie, Minghui Hu, AK Malik, M Tanveer, and PN Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence , 115:105151, 2022.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2016.

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations , 2018.

Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for realworld settings. In International Conference on Machine Learning , pages 8759-8773. PMLR, 2022.

Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting outof-distribution image without learning from out-of-distribution data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10951-10960, 2020.

Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. 2021 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp , pages 8706-8715, 2021.

Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems , 34: 677-689, 2021.

Bingbing Jiang, Chenglong Zhang, Yan Zhong, Yi Liu, Yingwei Zhang, Xingyu Wu, and Weiguo Sheng. Adaptive collaborative fusion for multi-view semi-supervised classification. Information Fusion , 96:37-50, 2023.

Bingbing Jiang, Xingyu Wu, Xiren Zhou, Anthony G Cohn, Yi Liu, Weiguo Sheng, and Huanhuan Chen. Semi-supervised multi-view feature selection with adaptive graph learning. IEEE Transactions on Neural Networks and Learning Systems , 35(3):3615-3629, 2024.

Audun Jøsang. Subjective logic , volume 3. Springer, 2016.

Uday Kamath, John Liu, and James Whitaker. Deep learning for NLP and speech recognition , volume 84. Springer, 2019.

Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 813-822, 2021.

Anna-Kathrin Kopetzki, Bertrand Charpentier, Daniel Zügner, Sandhya Giri, and Stephan Günnemann. Evaluating robustness of predictive uncertainty estimation: Are dirichlet-based models reliable? In International Conference on Machine Learning , pages 5707-5718. PMLR, 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems , 31, 2018.

Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations , 2018.

Xinyan Liang, Yuhua Qian, Qian Guo, Honghong Cheng, and Jiye Liang. Af: An associationbased fusion method for multi-modal classification. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(12):9236-9254, 2021.

Xinyan Liang, Pinhan Fu, Qian Guo, Keyin Zheng, and Yuhua Qian. Dc-nas: Divide-andconquer neural architecture search for multi-modal classification. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 13754-13762, 2024.

Wei Liu, Xiaodong Yue, Yufei Chen, and Thierry Denoeux. Trusted multi-view deep learning with opinion aggregation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7585-7593, 2022.

Wei Liu, Yufei Chen, Xiaodong Yue, Changqing Zhang, and Shaorong Xie. Safe multi-view deep classification. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 8870-8878, 2023.

Wei Liu, Yufei Chen, and Xiaodong Yue. Building trust in decision with conformalized multi-view deep classification. In ACM Multimedia 2024 , 2024.

Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in neural information processing systems , 33:21464-21475, 2020.

Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax-based out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23946-23955, 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems , 31, 2018.

Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. Advances in Neural Information Processing Systems , 32, 2019.

Aryan Mobiny, Pengyu Yuan, Supratik K Moulik, Naveen Garg, Carol C Wu, and Hien Van Nguyen. Dropconnect is effective in modeling uncertainty of bayesian deep networks. Scientific reports , 11(1):5458, 2021.

Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5216-5223, 2020.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 427-436, 2015.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing , pages 722-729. IEEE, 2008.

Deep Shankar Pandey and Qi Yu. Multidimensional belief quantification for label-efficient meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14391-14400, 2022.

Deep Shankar Pandey and Qi Yu. Evidential conditional neural processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 9389-9397, 2023.

Deep Shankar Pandey and Qi Yu. Learn to accumulate evidence from all training samples: theory and practice. In International Conference on Machine Learning , pages 26963-26989. PMLR, 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Tim Pearce, Felix Leibfried, and Alexandra Brintrup. Uncertainty in neural networks: Approximately bayesian ensembling. In International conference on artificial intelligence and statistics , pages 234-244. PMLR, 2020.

Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems , 32, 2019.

Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In International Conference on Machine Learning , pages 8491-8501. PMLR, 2020.

Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems , 31, 2018.

Glenn Shafer. A mathematical theory of evidence , volume 42. Princeton university press, 1976.

Weishi Shi, Xujiang Zhao, Feng Chen, and Qi Yu. Multifaceted uncertainty estimation for labelefficient deep learning. Advances in neural information processing systems , 33:17247-17257, 2020.

Shashi Pal Singh, Ajai Kumar, Hemant Darbari, Lenali Singh, Anshika Rastogi, and Shikha Jain. Machine translation using deep learning: An overview. In 2017 international conference on computer, communications and electronics (comptelix) , pages 162-167. IEEE, 2017.

Yue Song, Nicu Sebe, and Wei Wang. Rankfeat: Rank-1 feature removal for out-of-distribution detection. Advances in Neural Information Processing Systems , 35:17885-17898, 2022.

Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In European Conference on Computer Vision , pages 691-708. Springer, 2022.

Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems , 34:144-157, 2021.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning , pages 20827-20840. PMLR, 2022.

Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems , 33:11839-11852, 2020.

Christian Tomani and Florian Buettner. Towards trustworthy predictions from deep neural networks with fast adversarial calibration. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9886-9896, 2021.

Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, Eftychios Protopapadakis, et al. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience , 2018, 2018.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4921-4930, 2022.

Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International Conference on Machine Learning , pages 23631-23644. PMLR, 2022.

Cai Xu, Jiajun Si, Ziyu Guan, Wei Zhao, Yue Wu, and Xiyue Gao. Reliable conflictive multiview learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 16129-16137, 2024.

Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking generalized out-ofdistribution detection. Advances in Neural Information Processing Systems , 35:32598-32611, 2022.

Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7947-7958, 2022.

Jinsong Zhang, Qiang Fu, Xu Chen, Lun Du, Zelin Li, Gang Wang, Shi Han, Dongmei Zhang, et al. Out-of-distribution detection based on in-distribution data patterns memorization with modern hopfield energy. In The Eleventh International Conference on Learning Representations , 2022.

Qingyang Zhang, Yake Wei, Zongbo Han, Huazhu Fu, Xi Peng, Cheng Deng, Qinghua Hu, Cai Xu, Jie Wen, Di Hu, et al. Multimodal fusion on low-quality data: A comprehensive survey. arXiv preprint arXiv:2404.18947 , 2024.

Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning on graph data. Advances in Neural Information Processing Systems , 33:12827-12836, 2020.

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence , 40(6):1452-1464, 2017.

A An Example within EDL and HEDL

W=(camel,cattle,kangaroo,plain)

HEDL(Ours)

evidence

hyper-opinion

multinomial-

f=4

W = (0.3,0.3,0, 0.1)

e

(camel.cattle

bH

(camel.cattle =0.154

opinion

correct &lower uncertainty

P( camel ) = 0.154

label:cattle

H

(camel.cattle,

=3

b[camel }=0.115

P( cattle }= 0.385

f=3

W2 = (0.2,0.2,0.2,0.1)

kangaroo

kangaroo

cattle,

cattle:

2

=5

build

bH

bH

kangaroo

kangaroo

cattle=0.192

cattle,

=0.077

Projection

Opinion

b(kangaroo) = 0.077

b[ cattle }= 0.346

b plain 1 = 0.308

P(kangaroo) = 0.115

P( plain  = 0.346

f=2

W3 = (0,0.5,0.5,0.5)

[uod

8=

bH

plain)

=0.308

=0.154

= 0.154↓

W=1K=4

1=0.154

f=5

EDL

incorrect&higheruncertainty

W= (0,1.1,0,0)7

evidence

multinomial-opinion

P( camel } = 0.091

fs = 8

Ws = (0.1, 0.1,0,1.2)

Connected

Layer

Fully

activate

e( camel } = 1.0

e(cattle}=7.5

e(kangaroo) = 1.6

e(plain}=7.9

Dir(Pla)

b[ camel } = 0.045

b(kangaroo) = 0.073

b( plain )= 0.359

b( catte }= 0.341

P( cattle } = 0.386

P(kangaroo) = 0.118

P( plain }= 0.405X

W = 1K= 4

=0.182

u = 0.182

ignore/diminish

feature

enhance

evidence

hyper-opinion

multinomial-opinion

activate

A sample displaying the classification process of EDL and HEDL is shown in Figure 5, EDL tends to ignore or diminish the amount of vague evidence to get sharper belief mass. The loss of evidence leads to increased uncertainty and potential misclassification. In contrast, HEDL framework reserves the vague evidence, thereby achieving improved estimations of uncertainty and more accurate classification results. Notice that when there is no evidence supporting a set x , then e H x = 0 .

B Gradient Vanishing Analysis

Proposition 1. By building evidence on hyper-opinion and then projecting to multinomial-opinion, we avoid the vanishing gradient problem in fully-connected layer in traditional EDL.

Proof 1. Consider the neural network forward propagation in EDL

where bias stands for the bias of the fully-connected layer, z represents the feature extracted by the neural network. We can write expressions for all partial derivatives as follows:

Therefore by the chain rule, we can calculatethe the gradient w.r.t. W as:

Obviously when exists o k ≤ 0 , ∀ k ∈ X , vanishing gradient problem is unavoidable in traditional EDL. To ensure that proposed HEDL is not associate with similar problem, considering the forward propagation of HEDL:

where G ( W, b H ) can be calculated by Eq. 7 and Eq. 13, and the gradient w.r.t. W is calculated by chain rule:

where ∂ L ∂α k , ∂α k ∂e k , ∂o k ∂W are known items that won't cause vanishing gradient problem. Consider

where W, b H are all detached variables that are irrelevant variables in this partial derivative item, implying that G ( W, b H ) remains constant during the backward process.

Consequently, the opinion projection successfully circumvents the vanishing gradient problem in the fully-connected layer.

C Experiment Results on Flower-102 and CUB-200-2011

Table 3 details the comparative results on two fine-grained datasets Flower-102 and CUB-200-2011. On more complex fine-grained datasets, HEDL consistently demonstrates superior performance in OOD detection.

D Experiment Analysis of Computational Complexity

Table 4 presents the average training time per epoch of EDL and HEDL compared with MSP on different datasets, all under identical training conditions. The results indicate that the implementation of HEDL does not incur additional computational complexity.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction provide a comprehensive overview of the contributions, the scope, and the method of our paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation of the proposed method in the paper, namely that the best performance of our method depends on the performance of the pre-trained model we adapted.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a complete and correct proof for the theoretical results in the paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper fully discloses all the information needed to reproduce the main experimental results, including the theoretical and practical implementation, and training details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code of this paper is included in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: This paper specifies all the training and test details necessary to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: This paper presents a series of replicable experiments conducted across multiple datasets, proving the statistical significance of the experiments of our proposed method.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This paper provides sufficient information on the computer resources needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper discusses the positive societal impacts of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators of the data and models we used in this paper are properly credited and are the license and terms of use explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the details of our code as part of our submissions via structured templates, along with documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript takes a significant step forward in the realm of evidential deep learning by considering a more holistic hyper-opinion evidence framework.  The approach presented offers a novel perspective for optimizing evidential deep learning models, notably enhancing their ability to detect out-of-distribution instances without additional settings. This enhancement is particularly evident when the models are applied to complex datasets.\n\nSTRENGTHS:\nThe manuscript is commendable for presenting a fresh and innovative approach to the framework of evidential deep learning. It is noteworthy that while previous research has mainly concentrated on enhancing the settings in evidential deep learning, there has been an oversight in addressing the structural deficiencies of the framework itself.\n\nThe authors have introduced an interesting concept by treating the features extracted by the network as evidence when applying the theory to practical scenarios. The authors also give full mathematical proof on the problems they solved. This approach grounds in a solid theoretical foundation based on subjective logic, demonstrates simplicity in its application and shows impressive results in experimental validation.\n\nWEAKNESSES:\nThe manuscript exhibits some shortcomings in the handling of certain details. The hyper-domain does not encompass the set itself and the empty set; however, in practice, due to the treatment of features as evidence, the possibility of evidence representing these two sets cannot be ruled out. Furthermore, the subjective logic stipulates that evidence from the same subset should appear at most once, yet the approach presented in this paper does not guarantee full compliance with this condition.\n\nQUESTIONS:\nI had some concerns on the setup of the ablation experiment. What implement the authors take to transform from hyper-opinion to multinomial-opinion without opinion projection?\n\nLIMITATIONS:\nThe limitation is understandable and the method does not present any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of out-of-distribution (OOD) detection. Traditional Evidential Deep Learning framework collects sharp evidence that supports a single category while ignoring vague evidence that supports multiple categories, leading to inaccurate uncertainty estimation and decreased OOD detection performance. The authors introduce hyper-domain and propose Hyper-opinion Evidential Deep Learning. With hyper-opinion, HEDL explicitly models evidences as sharp and vague evidences that support single categories and multiple categories respectively. HEDL extends the framework of EDL and establishes more accurate uncertainty estimation for OOD detection. Experiments on several datasets demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n-The proposed method is sound in theory. HEDL extends the framework of EDL, establishes a more accurate uncertainty estimation framework for OOD detection. It’s quite novel and enlightening.\n-Without additional regularization terms nor computational complexity, it maintains the simplicity and generalizability.\n-This paper has sufficient experiments to demonstrate the effectiveness of the proposed method and each module. It achieves better performance compared with SOTA methods.\n\nWEAKNESSES:\n-To ensure fairness, the value of W_{prior} should be explicitly stated and consistent with EDL, cause the value of W_{prior} has been proved to be crucial to the effectiveness of EDL model [1].\n-The authors may consider incorporating the KL divergence as a loss term to enhance the model's performance. \n-In the section when proofing the existence of the gradient vanishing problem in EDL. It would be beneficial to delineate the conditions where o_k is less than zero occurs.\n-The paper could benefit from a wider analysis of the impact of the other two loss functions mentioned in EDL, e.g., the MSE loss and the Log loss, when applied to the HEDL model. \n\n[1] Chen M, Gao J, Xu C. R-EDL: Relaxing Nonessential Settings of Evidential Deep Learning[C]//The Twelfth International Conference on Learning Representations. 2023.\n\nQUESTIONS:\n1.Whether the opinion projection process that translates hyper-opinion into standard multinomial-opinion, leads to a loss of uncertainty information? Please provide a detailed discussion on this matter.\n2.Whether HEDL encounters the issue of exponential explosion, a challenge mentioned in subjective logic as the number of categories expands. Could the authors address this potential concern and discuss any mechanisms to manage such an increase in complexity?\n\nLIMITATIONS:\nThe limitations discussed in the manuscript do not appear to significantly constrain the applicability of the method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an out-of-distribution detection method based on evidential deep learning. This method models the evidence in hyper-domain, and the hyper-opinion in Subjective Logic is used to replace the multinomial-opinion in traditional evidential deep learning. The Hyper-opinion Evidential Deep Learning considers the vague evidence ignored in traditional Evidential Deep Learning, so as to achieve better uncertainty estimation effect. The OOD detection performance of proposed method exceeds the current SOTA OOD detection methods while maintaining the classification accuracy.\n\nSTRENGTHS:\n1.\tHEDL achieves SOTA OOD detection performance, while maintaining classification accuracy.\n2.\tAs the number of categories in a dataset increases, the traditional EDL framework's performance significantly deteriorates. In contrast, HEDL can consistently extract comprehensive evidence and maintain its performance, regardless of the dataset's scale.\n3.\tHEDL does not introduce additional computational complexity of the model.\n4.\tHEDL can mitigate the vanishing gradient problem in EDL theoretically and practically.\n\nWEAKNESSES:\n1.\tThe origin of the sample uncertainty depicted in the upper portion of Figure 1 is not clearly defined. It is imperative to clarify whether the data presented comes from actual experimental outcomes or if it is merely illustrative example. If the data is based on real-world results, please provide an explanation of how to quantify the samples vagueness.\n2.\tThe paper does not specify the value of $W_{prior} $ nor does it analyze its role as a hyperparameter. It is essential to elucidate whether $W_{prior} $ is equivalent to EDL. Please provide a detailed numerical definition of $W_{prior} $.\n3.\tFigure 2 can be described more clearly. In the second module ‘Opinion Projection’ of the lower part, the positioning of the right bar appears to be more logically associated with the 'Multinomial-Opinion Optimization' section. To improve the figure's comprehensibility and accuracy, it is recommended to reposition the right bar accordingly.\n\nQUESTIONS:\n1.\tIn Figure 4, the uncertainty distribution of In-Distribution and Out-of-Distribution data is similar between HEDL w/o projection and HEDL. This observation raises the question of the contribution of the opinion projection. Could the authors further explain the specific role and impact of the opinion projection in the whole method?\n2.\tCould the authors clarify the conceptual difference between a 'Dirichlet hyper distribution' and a standard 'Dirichlet distribution' in Equation 8? From my understanding, it seems that the two distributions equal. If there is a nuanced difference, please provide an explanation to delineate the unique characteristics of the 'Dirichlet hyper distribution' as employed in your method.\n\nLIMITATIONS:\nYes, limitations and social impact are discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a method for Out-of-Distribution detection called Hyper-opinion Evidential Deep Learning which is based on Evidential Deep Learning. It models the evidence on the hyper-domain, considering the extra vague evidence containing multiple possible categories. The measurement of vague evidence makes HEDL have more accurate uncertainty estimation. This paper also proposes an opinion projection method to mitigate the gradient vanishing problem in EDL. Experiments on several datasets show that the proposed method outperforms the current Out-of-Distribution detection methods.\n\nSTRENGTHS:\n(1)The paper has clear motivation for each part of the proposed method and the method has a solid theoretical foundation.\n\n(2)The proposed method achieves SOTA results compared with other methods.\n\nWEAKNESSES:\n1.Eqs. 13, 14, and 15 are hard to understand. The authors can detail on the meanings between the different W's.\n\n2.Why HEDL is trained on a pre-trained feature extractor and how to ensure the fairness of comparative evaluations with other methods?\n\n3.Theoretical proof of how HEDL solves vanishing gradient problem is missing. How is the EDL gradient vanishing point determined in Figure 3 and does this phenomenon consistently result in the vanishing gradients for certain fixed classes during each iteration?\n\n4.Adding experiments on model complexity comparison can be beneficial to illustrate the computation complexity of HEDL.\n\n5.More state-of-the-art methods (CIDER[1] and NECO[2]) can better demonstrate the effectiveness of proposed method. \n\n[1] https://openreview.net/forum?id=aEFaE0W5pAd\n\n[2] https://openreview.net/forum?id=9ROuKblmi7\n\n I’m mostly concerned about 2 and 3. If the authors can address these questions, I may consider raising my score.\n\nQUESTIONS:\nPlease answer the questions in Weakness.\n\nLIMITATIONS:\nPlease see Weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}