Hyper-opinion Evidential Deep Learning for Out-of-Distribution Detection

Authors: jingen  qu
yufei  chen
xiaodong  yue
wei  fu
qiguang  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 08:58:50

Abstract: Evidential Deep Learning (EDL), grounded in Evidence Theory and Subjective Logic (SL), provides a robust framework to estimate uncertainty for out-of-distribution (OOD) detection alongside traditional classification probabilities.However, the EDL framework is constrained by its focus on evidence that supports only single categories, neglecting the other collective evidences that could corroborate multiple in-distribution categories. This limitation leads to a diminished estimation of uncertainty and a subsequent decline in OOD detection performance.Additionally, EDL encounters the vanishing gradient problem within its fully-connected layers, further degrading classification accuracy.To address these issues, we introduce hyper-domain and propose Hyper-opinion Evidential Deep Learning (HEDL). HEDL extends the evidence modeling paradigm by explicitly integrating sharp evidence, which supports a singular category, with vague evidence that accommodates multiple potential categories.Additionally, we propose a novel opinion projection mechanism that translates hyper-opinion into multinomial-opinion, which is then optimized within the EDL framework to ensure precise classification and refined uncertainty estimation.HEDL integrates evidences across various categories to yield a holistic evidentiary foundation for achieving superior OOD detection. Furthermore, our proposed opinion projection method effectively mitigates the vanishing gradient issue, ensuring classification accuracy without additional model complexity. Extensive experiments over many datasets demonstrate our proposed method outperforms existing OOD detection methods.

Main paper: 
 introduction: 
 Deep Learning (DL) models have been widely adopted in many real-world applications [25, 57, 64, 15] . However, these models are trained under the implicit assumption that the training and test data are As sample gets vaguer, EDL tends to extract a minimal quantity of sharp evidence, results in elevated uncertainty estimation. HEDL demonstrates the capability to extract vague evidence as sample vagueness increases, thereby maintaining lower uncertainty levels.The EDL models face several challenges, with one primary issue arising from the theoretical framework. The evidence in multinomialopinion in EDL exclusively supports singleton sets, which contains only one category. In other words, EDL only captures the evidence which supports single category and rejects others. As a result, EDL is unable to effectively leverage vague evidence, such as features supporting a composite set containing multiple categories. As Figure 1 shows, EDL suffers from performance degradation in the face of ambiguous samples.In addition, the parameters of fully-connected layer in EDL models are facing vanishing gradient problem when number of category in datasets rises [49] . Vanishing gradient in EDL leads to failure in classification of several categories. To mitigate this problem, Pandey et al. [49] introduce regularization techniques. However, these efforts yield unsatisfactory results in real world OOD detection tasks.To train an evidential model maintaining classification accuracy and providing reliable uncertainty estimation for OOD detection, we incorporate EDL with hyper-opinion and propose Hyper-opinion Evidential Deep Learning (HEDL). While EDL is built upon multinomialopinion in a basic domain, hyper-opinion represents the opinion in the hyper-domain, which includes the basic domain and the composite sets. Through the concepts of composite set, HEDL is able to learn from vague evidence ignored by EDL. HEDL provides an effective mechanism for quantifying evidence that supports composite sets, thereby enhancing the differentiation of OOD data and classification accuracy. Our major contributions can be summarized as follows:• We introduce an evidential representation within the hyper-domain, which integrates sharp evidence that supports a singular category, with vague evidence that accommodates multiple potential categories, to establish a more comprehensive and accurate evidentiary foundation.• We develop a hyper-opinion framework within the hyper-domain and propose a novel opinion projection. This method transfers hyper-opinion to multinomial-opinion, allocating evidence to each category precisely and mitigating the vanishing gradient problem, while preserving computational efficiency. Subsequently, the second component projects the hyper-opinion to multinomial-opinion. Ultimately, the framework optimizes the opinion to attain precise classification and to furnish robust uncertainty estimations for OOD detection.• Our proposed Hyper-opinion Evidential Deep Learning (HEDL) procures more exhaustive evidence, which refines the precision of uncertainty estimation, and consequently enhances the performance of OOD detection while maintaing ID classification accuracy.• We carry out experiments over multiple challenging datasets to validate the OOD detection in HEDL outperforms existing OOD detection methods.2 Related Work 
uncertainty based ood detection: 
 Accurately quantifying predictive uncertainty in DL models is crucial for recognizing out-ofdistribution (OOD) samples. Traditional softmax-based models provide confidence estimation through class posteriors, which are inversely correlated with predictive uncertainty [16] . Several methods applicable to pre-trained classifiers that output class posteriors using softmax have been proposed [3, 14, 53, 37, 60, 18] , including Out-of-Distribution Detector for Neural Networks [31] and Mahalanobis Distance [30] . Besides, deep ensemble is a technique developed for uncertainty quantification [29] , which constructs an ensemble of neural networks and measures uncertainty based on the agreement/disagreement across the ensemble components [13] . However, this approach significantly increases the scale of model parameters, leading to high computational and storage complexity. Alternatively, neural networks based on Bayesian statistics called Bayesian neural networks [12, 4, 42] is raised to to quantify different uncertainties in Bayesian formalism. Bayesian methods normally apply approximation to address the intractability issue in marginalization of latent variables. And as such methods require sampling for uncertainty quantification, leading to expensive computations. A recent research effort has summarized OOD detection methods and established an OOD benchmark [69] .EDL introduces a conjugate higher-order evidential prior for the likelihood distribution that enables the model to capture the evidence vacuity as predictive uncertainty. The training of an EDL model can be regarded as an evidence-collecting process. Researches on multiple applications with EDL have been done, e.g., Dirichlet prior is introduced over the multinomial likelihood for evidential classification [2, 73, 11] , evidential models for regression [1, 48] , adversarial robustness [27] and calibration [63] . Most existing methods built upon EDL are trained on evidential losses conjunct with regularization of the evidence to guide the evidence vacuity, i.e., uncertainty, behavior [47, 56] . Some EDL models combine with the idea of outlier exposure [17] that provides access of OOD data to guide the evidence learning process of EDL models [40, 41] .In this work, we focus on evidential models for classification and OOD detection, and consider settings where no extra regularization and OOD data are used during model training to make the proposed approach more broadly applicable to practical real-world situations.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitation of the proposed method in the paper, namely that the best performance of our method depends on the performance of the pre-trained model we adapted.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
proposed method: 
 Our method's framework is depicted in Figure 2 , which operates under the assumption of no prior information.Subjective Logic (SL) is a theory of uncertain reasoning based on probability theory and belief theory in a domain X, which represents the set of exclusive possible states of a variable situation. It introduces the concepts of belief mass and uncertainty mass to describe the degree of belief and uncertainty about an event.Traditional EDL is built upon multinomial-opinion within domain X in SL and domain X is a limited portion of hyper-domain R(X), where P(X) is the powerset of X. R(X) = P(X)/{{X}, {∅}}.(1) Let us consider a domain X with cardinality of K , SL provides a belief mass b k representing the belief degree and a base rate a k representing the prior information for each singleton k = 1, ..., K and an overall uncertainty mass of u. The three compose a multinomial-opinion ω = (b, u, a), belief mass and uncertainty mass sum up to one, eg.,EQUATIONOur method models the evidence in hyper-domain R(X) with hyper-opinion, which provides a belief mass b H x , x ∈ R(X), representing the belief degree of set x. Along with a H and u, the three compose a hyper-opinion ω H = (b H , u, a H ) and the hypernomial belief mass distribution also follows the additivity requirement:bEQUATIONHyper opinion allows belief mass to be divided into two types called sharp belief mass and vague belief mass. Belief mass that only supports a specific singleton is called sharp belief mass, eg., k ∈ X, it discriminates between this and other singletons. EDL built upon the multinomial-opinion only offers sharp belief mass estimation. Considers a domain X of K mutually exclusive singletons, for each singleton k = 1, ..., K, sharp belief mass isEQUATIONBelief mass assigned to a composite set x ∈ C(X), where C(X) = R(X)/X, represents vague belief mass because it expresses cognitive vagueness. It supports the truth of multiple singletons in X simultaneously. Vague belief mass can be allocated to a singleton k asEQUATIONwhere a(k|x) is relative base rate. When no prior information is available, a(k|x) can be simplified toEQUATIONwhere |x| is the cardinality of x. Then in hyper-opinion, a belief mass b H x for a set x is computed using the evidence for the set. Let e H x ≥ 0 be the evidence derived for the set x, then the belief b H By introducing hyper-opinion, vague beliefs that assigned to composite sets can be take into consideration, which better measure comprehensive evidence and estimate uncertainty more accurately.In practice, we activate the features extracted by the neural network as evidence in hyper-domain, and build them within hyper-opinion to distinguish sharp belief and vague belief. This allows the model to maintain its vagueness among similar in-distribution categories, thereby ensuring that the uncertainty remains low.A projection from hyper-opinion to multinomial-opinion is needed to realize the projected probability of each singleton. Therefore we introduce a novel opinion projection implementation that projects belief mass from hyper-opinion into multinomial-opinion, with b V k and b S k that can be calculated by Eq. 4 and Eq. 5, followingEQUATIONWe activate the features extracted by neural network for ascertaining non-negative evidence within the hyper-domain. After associate evidence with belief in hyper-opinion, we determine the set each belief mass supports as mentioned in section 3.1, and project the belief mass from hyper-opinion to multinomial-opinion.Specifically, we apply a unit step activation function to the parameters of the fully connected layer, eg., Heaviside functionEQUATIONIt offers an access to a matrix W S = H(W ), where W corresponds to the weight matrix of the fully connected layer. W S represents the information of set each belief mass supports.Assume there are K singletons and N belief masses supporting different sets, it offers a matrix W S N,K . For a belief mass b H x supporting set x, W S x is a vector that contains information about which singletons belong to the set x.Once the set each belief mass supports has been identified, projecting hyper-opinion to multinomialopinion is straightforward. For each belief mass within the hyper-opinion, we can compute its relative base rate to each singleton, and allocate belief mass accordingly. For a singleton k, its total projected multinomial-opinion belief mass isEQUATIONEQUATION)where a is the base rate. Without any prior information, Eq. 11 can be simplied toEQUATIONTo date, we have successfully delineated the process of projecting belief mass from a hyper-opinion to a multinomial-opinion within a neural network framework. In practical terms, this projection is executed by applying a linear transformation to the output of the fully connected layer. This transformation facilitates the allocation of belief mass to the respective singletons in the multinomialopinion. Consequently, the incremental computational complexity associated with our method is constant as O(1).EQUATION)where o is the output of fully-connected layer and W, W p , b H are all detached variables, making G(W, b H ) a constant during one training epoch.The output after opinion projection represents the projected multinomial-opinion in EDL, which has the equivalent meaning in EDL and can be optimized with the same techniques. We used an example to show why the uncertainty estimation of HEDL outperforms EDL in Appendix A.By building evidence within hyper-domain and projecting hyper-opinion belief mass into multinomialopinion belief mass, we construct a flow that can be optimized in multinomial-opinion framework to obtain the comprehensive evidence and accurate uncertainty estimation for OOD detection, which is similar to traditional EDL.As the sum of evidence x∈R(X) e H x and uncertainty u remain the same during the projection, we can pass the belief mass in the form of evidence to simplify the calculation. Therefore the projected probability distribution derived from the projected multinomial-opinion can correspond to an expected probability distribution derived from a Dirichlet distribution parameterized by αEQUATION)The Dirichlet distribution is a probability density function (pdf) for possible values of the probability mass function (pmf) P and is given by:EQUATIONIn projected multinomial-opinion, the expected probability for the k th singleton calculation isEQUATIONwhich allows to be optimized by the loss function defined in EDLEQUATIONwhere ψ(•) is the digamma function, y i is a one-hot vector encoding the ground-truth class of observation x i with y ij = 1 and y ik = 0 for all k ̸ = j, and α i be the parameters of the Dirichlet density on the predictors.At this point, we have established the complete framework of HEDL, spanning all stages ranging from input processing to classification and uncertainty estimation. Our method objective has the following proposition in the Appendix B.By establishing the framework of HEDL, we comprehensively extract the sharp and vague evidence each sample contains and allocate preciously, thereby enabling accurate classification. Moreover, comprehensive evidence contributes to improved uncertainty estimation and subsequently enhances the performance of OOD detection.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code of this paper is included in the supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment: 
 In this section, we describe our experimental setup and demonstrate the effectiveness of our method on a wide range of OOD evaluation benchmarks and the most widely used metric AUROC is adopted [52, 21, 10, 37] . We also conduct an ablation analysis that leads to an improved understanding of our approach.In-distribution Datasets. We use the CIFAR-10[28], CIFAR-100 [28] , Flower-102 [46] and CUB-200-2011 [65] as ID data.Out-of-distribution Datasets. For the OOD test datasets, we use three common benchmarks [69] : SVHN [44] , Textures [6] , Places365 [74] , that are used in Openood-benchmark [69] . There is no overlapping between ID datasets and OOD datasets.Evaluation Metrics. We measure the following metrics: 1) FPR95 measures the false positive rate (FPR) when the true positive rate (TPR) is equal to 95%. Lower scores indicate better performance. 2) AUROC measures the area under the Receiver Operating Characteristic (ROC) curve, which displays the relationship between TPR and FPR. The area under the ROC curve can be interpreted as the probability that a positive ID example will have a higher detection score than a negative OOD example.3) AUPR measures the area under the Precision-Recall (PR) curve. The PR curve is created by plotting precision versus recall. AUROC is the most common metric [52, 21, 10, 37] and we use AUROC as the main metric for OOD detection performance while accuracy measures performance of detecting ID samples. Our goal is to detect more OOD samples while maintaining ID classification performance. Implementation Details. We follow the experiment settings outlined in OpenOOD [69] . We use ResNet-18 [15] for CIFAR-10 and CIFAR-100. For more intricate datasets that are not included in OpenOOD [69] , such as fine-grained datasets Flower-102 and CUB-200-2011, we employ ResNet-34 [15] for enhanced feature representation. All experiments are implemented with PyTorch [50] and carried out with NVIDIA GeForce RTX 3090 GPU. We use the standard data split for all datasets, and the number of training epochs is 100, the initial learning rate is 0.0001 with AdamW [39] , and the batch size is 128. At test time, all images are resized to 224 × 224. For HEDL model, we first train the feature extractor with softmax layer for 90 epochs and then train in HEDL framework for 10 epochs. HEDL does not introduce any additional hyperparameters, thereby eliminating the need for extensive hyperparameter tuning, and W prior is set to 1 for HEDL.Baseline Methods. We compare our method with several classical and state-of-the-art OOD detection methods. Specifically, we compare our method with post-hoc inference methods and training methods.From MSP [16] to GEN [38] are post-hoc inference methods, which affect OOD detection performance only and do not change model accuracy. The others are training methods. We excluded methods that required auxiliary OOD data due to the practical real-world situations consideration. We leverage selected experimental results from OpenOOD [69] to demonstrate the effectiveness of our approach. The comparative results on CIFAR-10 and CIFAR-100 are detailed in Table 1 , and the results on Flower-102 and CUB-200-2011 are shown in Appendix C. For each model, we utilize three OOD datasets, thereby aiming to achieve more realistic and generalized outcomes. We reveal a common challenge: when confronted with more complex data scenarios, training methods struggle to maintain both accuracy and OOD detection capabilities simultaneously. However, HEDL consistently achieves better OOD detection performance than existing state-of-the-art OOD detection methods while preserving the accuracy of ID classification, even under complex data scenarios. Notably, HEDL accomplishes this enhancement without additional regularization strategies or hyperparameters, indicating strong generalization ability on different datasets, it also avoids incurring higher computational costs. The experimental training time analysis of HEDL can be found in Appendix D. The gradient norms of fully-connected layer parameters over EDL and HEDL during training is shown in Figure 3 , alongside the final accuracy for each category. The sum of these gradient norms has been normalized for comparative analysis. It is observed that the gradient norms for several parameters within the fullyconnected layer of the EDL model remain zero throughout the training process, which correlates with a significantly lower final accuracy for these categories. This outcome is indicative of the vanishing gradient problem. Conversely, HEDL does not experience this issue, demonstrating that our proposed method effectively circumvents the challenge of vanishing gradients within the fully-connected layer.EDL suffers from a notable decline in both ID accuracy and OOD detection when facing a proportional rise in the volume of vague evidence. In contrast, HEDL demonstrates the capability to consistently extract comprehensive evidence and maintain its performance regardless of the dataset scale.We investigate the performance of our method with ablation experiments on two challenging finegrained datasets. The fine-grained datasets contain more vagueness among categories and can better prove the effectiveness of our methods. We conducte ablation experiments on the effects of hyperopinion and opinion projection, respectively. Note that opinion projection can only be built upon hyper-opinion.Figure 4 illustrates the uncertainty distribution of ID and OOD samples across different datasets for EDL, HEDL without opinion projection, and HEDL itself. Notably, on the latter three more complex datasets, the approaches based on hyper-opinion exhibits a distinct performance advantage. It is also worth observing that, in these datasets, instances of ID data with maximum uncertainty are present in the EDL model. This phenomenon can be attributed to the failure of extracting evidence of those categories due to the vanishing gradient problem.Table 2 shows that evidence built on hyper-opinion can be considered comprehensively, leading to accurate uncertainty estimation and above baseline OOD detection performance. But without the correct projection from hyper-opinion to multinomial-opinion, vague evidence can not be assigned precisely, leading to inaccurate classification. 
conclusion: 
 In this paper, we propose Hyper-opinion Evidential Deep Learning (HEDL), a novel approach designed to generate precise uncertainty estimation for Out-of-Distribution (OOD) detection. Our method encapsulates a comprehensive representation of evidence within hyper-opinion, which allows model to preserve its vagueness among In-Distribution categories to reject OOD data.Additionally, by projecting hyper-opinion to multinomial-opinion, HEDL circumvents the vanishing gradient problem encountered in the fully-connected layers of traditional EDL. This projection is optimized within an established framework, yielding accurate and reliable evidence. Notably, our method accomplishes superior OOD detection performance while simultaneously upholding classification accuracy without incurring additional computational complexity. Extensive experimental results across numerous datasets substantiate the efficacy of the proposed Hyper-opinion Evidential Deep Learning.Limitations and societal impact. Our proposed HEDL method achieves best performance by transfering learning on pre-trained models. In future work, it is necessary to reduce the dependence on pre-trained models and explore alternative approaches. This work aims to improve the safety of deep learning models, which tends to benefit a wide range of applications of AI in social life. A sample displaying the classification process of EDL and HEDL is shown in Figure 5 , EDL tends to ignore or diminish the amount of vague evidence to get sharper belief mass. The loss of evidence leads to increased uncertainty and potential misclassification. In contrast, HEDL framework reserves the vague evidence, thereby achieving improved estimations of uncertainty and more accurate classification results. Notice that when there is no evidence supporting a set x, then e H x = 0. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: This paper specifies all the training and test details necessary to understand the results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: This paper presents a series of replicable experiments conducted across multiple datasets, proving the statistical significance of the experiments of our proposed method.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators of the data and models we used in this paper are properly credited and are the license and terms of use explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We provide the details of our code as part of our submissions via structured templates, along with documentation. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Proposition 1. By building evidence on hyper-opinion and then projecting to multinomial-opinion, we avoid the vanishing gradient problem in fully-connected layer in traditional EDL.Proof 1. Consider the neural network forward propagation in EDLEQUATIONEQUATIONEQUATIONEQUATIONwhere bias stands for the bias of the fully-connected layer, z represents the feature extracted by the neural network. We can write expressions for all partial derivatives as follows:EQUATIONEQUATION)EQUATION)Therefore by the chain rule, we can calculatethe the gradient w.r.t. W as:EQUATIONObviously when exists o k ≤ 0, ∀k ∈ X, vanishing gradient problem is unavoidable in traditional EDL. To ensure that proposed HEDL is not associate with similar problem, considering the forward propagation of HEDL:EQUATIONEQUATIONEQUATIONwhere G(W, b H ) can be calculated by Eq. 7 and Eq. 13, and the gradient w.r.t. W is calculated by chain rule:EQUATIONwhere ∂L ∂α k , ∂α k ∂e k, ∂o k ∂W known items that won't cause vanishing gradient problem. ConsiderEQUATIONwhere W, b H are all detached variables that are irrelevant variables in this partial derivative item, implying that G(W, b H ) remains constant during the backward process.EQUATIONConsequently, the opinion projection successfully circumvents the vanishing gradient problem in the fully-connected layer.Table 3 details the comparative results on two fine-grained datasets Flower-102 and CUB-200-2011. On more complex fine-grained datasets, HEDL consistently demonstrates superior performance in OOD detection. Justification: Our abstract and introduction provide a comprehensive overview of the contributions, the scope, and the method of our paper. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper introduces Hyper-opinion Evidential Deep Learning (HEDL), an extension of the Evidential Deep Learning (EDL) framework, for out-of-distribution detection. The method addresses the limitations of EDL by incorporating both sharp evidence (supporting single categories) and vague evidence (supporting multiple categories). The authors introduce an opinion projection mechanism that mitigates the vanishing gradient issue encountered in EDL. The method is shown to outperform existing OOD detection methods across various datasets.

All reviewers acknowledged its significant contributions, sound theoretical foundation, and solid experimental validation. They consistently rated the paper positively, with all ratings falling within the weak accept-to-accept range.

The main strengths of the paper:
The paper presents a novel extension to the EDL framework, enhancing its capability for OOD detection by effectively modeling both sharp and vague evidence. The introduction of the opinion projection mechanism is a meaningful contribution, addressing the vanishing gradient issue and improving classification accuracy without increasing model complexity. Extensive experiments demonstrate that HEDL outperforms state-of-the-art methods in OOD detection, confirming the practical relevance of the proposed approach.
The paper is grounded in solid theoretical foundations and includes rigorous mathematical proofs to support the proposed innovations.

Weaknesses:
Some equations were initially difficult to understand, and the paper could benefit from additional clarifications and improvements in the presentation of certain concepts before the camera ready. Concerns were raised about the fairness of comparative evaluations due to the use of a pre-trained feature extractor. The authors clarified that consistent experimental conditions were applied across all methods.
Minor concerns were noted about the clarity of some figures and the explanation of certain terms, which were addressed in the author rebuttal.

Discussion: The authors effectively addressed all concerns raised by the reviewers during the rebuttal process. They provided additional explanations, clarified ambiguous points, and presented further experimental results, which satisfied the reviewers and led to an overall positive reception of the paper. 

All in all, given the strong theoretical contributions, robust experimental validation, and positive feedback from reviewers, I recommend accepting this paper. The paper is expected to have a positive impact in the area of probabilistic methods and OOD detection, contributing valuable insights to the NeurIPS community.

Reviews: review: paper_summary: paper_summary: This paper provides a method for Out-of-Distribution detection called Hyper-opinion Evidential Deep Learning which is based on Evidential Deep Learning. It models the evidence on the hyper-domain, considering the extra vague evidence containing multiple possible categories. The measurement of vague evidence makes HEDL have more accurate uncertainty estimation. This paper also proposes an opinion projection method to mitigate the gradient vanishing problem in EDL. Experiments on several datasets show that the proposed method outperforms the current Out-of-Distribution detection methods.
strength_weakness: strength_weakness: 1.Eqs. 13, 14, and 15 are hard to understand. The authors can detail on the meanings between the different W's.

2.Why HEDL is trained on a pre-trained feature extractor and how to ensure the fairness of comparative evaluations with other methods?

3.Theoretical proof of how HEDL solves vanishing gradient problem is missing. How is the EDL gradient vanishing point determined in Figure 3 and does this phenomenon consistently result in the vanishing gradients for certain fixed classes during each iteration?

4.Adding experiments on model complexity comparison can be beneficial to illustrate the computation complexity of HEDL.

5.More state-of-the-art methods (CIDER[1] and NECO[2]) can better demonstrate the effectiveness of proposed method. 

[1] https://openreview.net/forum?id=aEFaE0W5pAd

[2] https://openreview.net/forum?id=9ROuKblmi7

 I’m mostly concerned about 2 and 3. If the authors can address these questions, I may consider raising my score.
questions: questions: Please answer the questions in Weakness.
limitations: limitations: Please see Weakness
paper_summary: paper_summary: This paper introduces an out-of-distribution detection method based on evidential deep learning. This method models the evidence in hyper-domain, and the hyper-opinion in Subjective Logic is used to replace the multinomial-opinion in traditional evidential deep learning. The Hyper-opinion Evidential Deep Learning considers the vague evidence ignored in traditional Evidential Deep Learning, so as to achieve better uncertainty estimation effect. The OOD detection performance of proposed method exceeds the current SOTA OOD detection methods while maintaining the classification accuracy.
strength_weakness: strength_weakness: 1.	The origin of the sample uncertainty depicted in the upper portion of Figure 1 is not clearly defined. It is imperative to clarify whether the data presented comes from actual experimental outcomes or if it is merely illustrative example. If the data is based on real-world results, please provide an explanation of how to quantify the samples vagueness.
2.	The paper does not specify the value of $W_{prior} $ nor does it analyze its role as a hyperparameter. It is essential to elucidate whether $W_{prior} $ is equivalent to EDL. Please provide a detailed numerical definition of $W_{prior} $.
3.	Figure 2 can be described more clearly. In the second module ‘Opinion Projection’ of the lower part, the positioning of the right bar appears to be more logically associated with the 'Multinomial-Opinion Optimization' section. To improve the figure's comprehensibility and accuracy, it is recommended to reposition the right bar accordingly.
questions: questions: 1.	In Figure 4, the uncertainty distribution of In-Distribution and Out-of-Distribution data is similar between HEDL w/o projection and HEDL. This observation raises the question of the contribution of the opinion projection. Could the authors further explain the specific role and impact of the opinion projection in the whole method?
2.	Could the authors clarify the conceptual difference between a 'Dirichlet hyper distribution' and a standard 'Dirichlet distribution' in Equation 8? From my understanding, it seems that the two distributions equal. If there is a nuanced difference, please provide an explanation to delineate the unique characteristics of the 'Dirichlet hyper distribution' as employed in your method.
limitations: limitations: Yes, limitations and social impact are discussed in the main paper.
paper_summary: paper_summary: This paper studies the problem of out-of-distribution (OOD) detection. Traditional Evidential Deep Learning framework collects sharp evidence that supports a single category while ignoring vague evidence that supports multiple categories, leading to inaccurate uncertainty estimation and decreased OOD detection performance. The authors introduce hyper-domain and propose Hyper-opinion Evidential Deep Learning. With hyper-opinion, HEDL explicitly models evidences as sharp and vague evidences that support single categories and multiple categories respectively. HEDL extends the framework of EDL and establishes more accurate uncertainty estimation for OOD detection. Experiments on several datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: -To ensure fairness, the value of W_{prior} should be explicitly stated and consistent with EDL, cause the value of W_{prior} has been proved to be crucial to the effectiveness of EDL model [1].
-The authors may consider incorporating the KL divergence as a loss term to enhance the model's performance. 
-In the section when proofing the existence of the gradient vanishing problem in EDL. It would be beneficial to delineate the conditions where o_k is less than zero occurs.
-The paper could benefit from a wider analysis of the impact of the other two loss functions mentioned in EDL, e.g., the MSE loss and the Log loss, when applied to the HEDL model. 

[1] Chen M, Gao J, Xu C. R-EDL: Relaxing Nonessential Settings of Evidential Deep Learning[C]//The Twelfth International Conference on Learning Representations. 2023.
questions: questions: 1.Whether the opinion projection process that translates hyper-opinion into standard multinomial-opinion, leads to a loss of uncertainty information? Please provide a detailed discussion on this matter.
2.Whether HEDL encounters the issue of exponential explosion, a challenge mentioned in subjective logic as the number of categories expands. Could the authors address this potential concern and discuss any mechanisms to manage such an increase in complexity?
limitations: limitations: The limitations discussed in the manuscript do not appear to significantly constrain the applicability of the method.
paper_summary: paper_summary: This manuscript takes a significant step forward in the realm of evidential deep learning by considering a more holistic hyper-opinion evidence framework.  The approach presented offers a novel perspective for optimizing evidential deep learning models, notably enhancing their ability to detect out-of-distribution instances without additional settings. This enhancement is particularly evident when the models are applied to complex datasets.
strength_weakness: strength_weakness: The manuscript exhibits some shortcomings in the handling of certain details. The hyper-domain does not encompass the set itself and the empty set; however, in practice, due to the treatment of features as evidence, the possibility of evidence representing these two sets cannot be ruled out. Furthermore, the subjective logic stipulates that evidence from the same subset should appear at most once, yet the approach presented in this paper does not guarantee full compliance with this condition.
questions: questions: I had some concerns on the setup of the ablation experiment. What implement the authors take to transform from hyper-opinion to multinomial-opinion without opinion projection?
limitations: limitations: The limitation is understandable and the method does not present any potential negative societal impact.

Comments: title: Retaining my positive score
comment: Thank you for helping us improve the paper, we will add the definition about $W_{prior} $ to the experimental setup in an updated version of this paper. We really appreciate your valuable comment!
Thank you for your rebuttal. I am glad that the authors validated the wider analysis I raised for KL divergence and other loss functions. The definition of W_{prior} addressed my concern about the fairness, and please include the definition in the paper. The discussion on uncertainty loss and exponential explosion is convincing. After reading other reviews and your responses, I am going to retain my score.
Thank you for helping us improve the paper and updating the score, we really appreciate your valuable comment!
I would like to thank the authors for the additional explanation and more experimental results. I look forwards to seeing an updated version of this paper with these results, and I have updated my score.