Title: Vicinal Label Supervision for Reliable Aleatoric and Epistemic Uncertainty Estimation
Venue: NeurIPS 2025
Accepted: yes
Keywords: evidentail deep learning; uncertainty estimation
Abstract: Uncertainty estimation is crucial for ensuring the reliability of machine learning models in safety-critical applications. Evidential Deep Learning (EDL) offers a principled framework by modeling predictive uncertainty through Dirichlet distributions over class probabilities. However, existing EDL methods predominantly rely on level-0 hard labels, which supervised a uncertainty-aware model with full certainty. We argue that hard labels not only fail to capture epistemic uncertainty but also obscure the aleatoric uncertainty arising from inherent data noise and label ambiguity.
  As a result, EDL models often produce degenerate Dirichlet distributions that collapse to near-deterministic outputs. 
  To overcome these limitations, we propose a vicinal risk minimization paradigm for EDL by incorporating level-1 supervision in the form of vicinally smoothed conditional label distributions. This richer supervision exposes the model to local label uncertainty, enhancing aleatoric uncertainty quantification, while also mitigating the degeneration of the Dirichlet distribution into a Dirac delta function, thereby improving epistemic uncertainty modeling.
	Extensive experiments show that our approach consistently outperforms standard EDL baselines across synthetic datasets, covariate-shifted out-of-distribution generalization tasks, and out-of-distribution detection benchmarks, providing more reliable uncertainty estimates.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper adresses a limitation in a class of deep learning uncertainty quantification (UQ) methods referred to as Evidential Deep Learning (EDL) methods, which rely on level-0 hard labels for supervision. The authors argue that hard labels do not reflect inherent data noise (aleatoric uncertainty). In practice, they observe that this mismatch encourages EDL models to produce degenerate Dirichlet distributions that collapse into near-deterministic outputs, leading to poor uncertainty quantification.\n\nSTRENGTHS AND WEAKNESSES:\nPros : \n- Avoiding EDL degenerate solutions seems to be achieved by the proposed method\n- The method is simple in its construction and thus does not add significant complexity on top of EDL algorithms.\n- The efficiency of the method is theoretically grounded\n\nCons : \n- The empirical comparisons in the experimental section omit important state-of-the-art works leveraging UQ methods to detect outliers/OOD inputs. \n- The experiments focus on OOD detection uses cases and do not examine whether quantified uncertainties align with incorrect predictions, as is commonly done\n- [Minor] The experiments involve training runs on two vision datasets and one toy dataset. The paper would gain in impact with a more solid evaluation involving more training datasets and other data modalities.\n\nQUESTIONS:\nA concern with this paper is that the experiments are not sufficiently convincing, as the authors seem eager to claim that their method establishes new standards in OOD input detection. Generally speaking, UQ methods are not explicitly designed for outlier/OOD input detection but may develop this capability as a byproduct. Specifically, the benchmark should include at least one UQ method known to exhibit strong outlier-detection performance, such as Liu et al. (2020). However, the experiments do validate the added value of their method compared to EDL for this use case.\n\n(Liu et al. 2020) Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T., & Lakshminarayanan, B. (2020). Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in neural information processing systems, 33, 7498-7512.\n\nAnother important aspect to be addressed in the experiments is evaluating other use cases for quantified uncertainties, particularly whether those uncertainties align with classification errors. Indeed, one generally expect higher levels of uncertainties in the regions of the input space where the model fails to predict the correct class labels. \n\nBesides, many competing methods in the benchmark exhibit terrible performances. Could the authors comment on why they fail to so dramatically ?\n\nThe main contribution of the proposed modification to EDL is its ability to capture the input distribution (as Fig. 1 suggests) and leverage it to assign high uncertainty in low-probability regions of the input space. Yet this emerging capability is neither fully explored nor theoretically explained. Based on the authors’ comments that they use a large value for β, I suspect that most mixed samples lie near class boundaries; thus, it is unclear why the model would also learn to assign high uncertainty to inputs far from the training samples.\n\nMinor: In Figure 1, unless I’m mistaken, the displayed uncertainty-scoring function is not specified.\n\nLIMITATIONS:\nThe limitations are addressed very briefly (one sentence - 2 lines) in the conclusion. Usually, one would expect at least a whole paragraph.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEvidential Deep Learning (EDL) is a widely used approach for uncertainty quantification (UQ) that models predictive uncertainty via a second-order distribution---specifically, a Dirichlet distribution over the probability simplex in classification tasks. Recent studies have pointed out that most existing EDL methods rely on level-0 supervision (i.e., hard labels), which can lead to unfaithful uncertainty representations. In particular, the learned Dirichlet distribution may become degenerate, collapsing to a Dirac delta function.\n\nThis work aims to address this issue by incorporating level-1 supervision, i.e., categorical label distributions. Since such ground-truth level-1 labels are typically expensive to obtain, the authors propose generating them using techniques inspired by MixUp. Two types of level-1 labels are constructed: one to model aleatoric uncertainty and another for epistemic uncertainty. The UQ model is then trained using a vicinal risk minimization (VRM) objective based on the constructed labels. The authors argue that this approach mitigates the degeneracy issue of Dirichlet distributions and enables more faithful uncertainty modeling. Empirical results demonstrate that the proposed method yields improved performance on downstream tasks such as out-of-distribution (OOD) detection.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- This work tackles an important research problem: overcoming the limitations of existing EDL methods that struggle to learn faithful representations of uncertainty.\n\n- The motivation and overall direction are intuitive. Specifically, the paper clearly identifies the key issue, EDL methods learning second-order distributions from level-0 (hard) labels, and proposes constructing level-1 (soft) labels as a principled solution.\n\n- The proposed method demonstrates strong empirical performance on downstream UQ tasks.\n\nWeaknesses:\n\n- The contribution of the paper is somewhat limited, as several of the key arguments have already been well-established in the existing uncertainty quantification (UQ) literature. For example, prior works [2, 3] have pointed out that a core issue with EDL lies in its reliance on level-0 hard labels, which can cause the learned Dirichlet distribution to collapse into a Dirac measure. Additionally, [39] has shown that existing EDL methods fail to faithfully capture both aleatoric and epistemic uncertainty. Furthermore, both [22] and [39] emphasize the importance of incorporating level-1 supervision into the EDL framework, and [39] advocates for distillation-based approaches that construct level-1 labels through ensemble or bootstrap-based techniques (see Section 6 of [39]).\n\n- The proposed method is heuristic and appears to be sensitive to hyperparameters. While level-0 hard labels are sampled from an underlying true distribution, the level-1 labels in this work are constructed heuristically via MixUp. As a result, the distribution of these crafted labels, and by extension, the behavior of the learned UQ model, depends on the specific MixUp parameters. The ablation studies in Section 5 further confirm that the proposed method performs well only under well-tuned hyperparameter settings.\n\n- Some of the paper’s claims are not entirely accurate. Specifically, the concern regarding the Dirac delta collapse applies only to EDL objectives without regularization. In practice, many EDL methods employ regularizers, which prevent the learned Dirichlet from collapsing into a delta function. As shown in [39], with regularization, the learned second-order distribution converges to a fixed-form Dirichlet (see Example 5.2), determined by the regularization hyperparameter. Thus, while the distribution may not collapse, it does not necessarily mean the model has learned faithful or meaningful uncertainty.\n\n- The justification for the proposed approach in Section 3.2 is unconvincing. The authors suggest that mixing in-distribution samples with random noise helps epistemic uncertainty estimation and argue that perturbed samples should exhibit greater epistemic uncertainty than the clean samples. However, this assumption is questionable. Epistemic uncertainty reflects the model's lack of knowledge and should not depend on the input distribution. A simple counterexample illustrates this: consider a human expert acting as a highly confident classifier with negligible epistemic uncertainty. If a clean image is perturbed with sufficient random noise, even a human may become uncertain, not due to lack of knowledge of a human expert (epistemic uncertainty), but because the image has become ambiguous or unrecognizable (aleatoric uncertainty). This suggests that adding noise primarily increases aleatoric uncertainty, not epistemic uncertainty, as the model would struggle to make reliable predictions regardless of additional training data.\n\nQUESTIONS:\n1.   Can you elaborate on the sensitivity to hyperparameters?  More generally, it is helpful to understand in what scenarios the method will fail or otherwise not be useful, and how it will fail when it does.\n\n2.  How can the approach of Section 3.2 be justified in light of the comments above?\n\n3.  How would the authors address the comment regarding one of the claims, as mentioned above?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEvidential Deep Learning aims to estimate both aleatoric and epistemic uncertainty using Dirichlet distributions, but existing approaches relying on hard labels (level-0 supervision) often produce degenerate outputs with unreliable uncertainty estimates, including both aleatoric and epistemic uncertainty. This paper proposes a vicinal risk minimization framework that constructs level-1 supervisory signals through local label interpolation, without requiring additional annotations. The authors provide a theoretical analysis showing improved generalization by reducing the lower bound of generalization error, and empirical results demonstrate consistent gains over baselines in out-of-distribution detection and generalization tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper is well-written and easy to follow; the motivation is clearly articulated, the methodology is well structured, and the appendix provides detailed derivations and theoretical proofs.\n2. The experimental evaluation includes a broad set of EDL-related baselines across both out-of-distribution (OOD) detection and OOD generalization tasks, showing consistent improvements of the proposed method.\n3. The proposed approach is simple to implement and can be readily integrated into other uncertainty-aware models and tasks without requiring additional annotation or architectural changes.\n\n**Weaknesses**\n1. The paper claims that standard EDL methods fail to capture reliable aleatoric uncertainty and that the proposed method improves this; however, the evaluation would benefit from more targeted experiments measuring aleatoric uncertainty quality, such as misclassification detection or selective prediction.\n\n2. The total loss consists of two components (Equation 14), i.e., vicinal supervision and noise augmentation. However, the paper lacks an ablation study isolating their individual contributions, which would help clarify the effectiveness of each.\n\n3. Since the proposed method involves mixup and additional data generation, a brief analysis or discussion of computational complexity and training overhead would strengthen the practical understanding of the method's scalability.\n\nQUESTIONS:\n1. The proposed method builds on mixup and label smoothing strategies, and Table 3 shows it improves ID accuracy even for MSP. How can we tell whether the performance gains in Tables 1 and 2 come from improved representation learning versus improved uncertainty modeling? For example, can we analyze latent representations (e.g., in PostNet) before and after applying the proposed method?\n\n2. The paper introduces two new uncertainty quantification measures—Dirichlet differential entropy and conditional entropy. What is the motivation for using these specific measures, and is there any experimental evidence showing their advantages over existing metrics?\n\n3. How does Theorem 2 theoretically support improved generalization and robustness? Does it also suggest an improvement in the quality of the uncertainty estimates?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to further improve the epistemic and aleatoric uncertainty estimation using vicinal risk minimization for evidential deep learning. The paper provides a three-level view: level-0 being hard labels, level-1 being categorical distributions, level-2 being Dirichlet distributions. The paper analyzed the limitations of EDL (level-2 learning against level-0 labels) and proposes to use vicinal supervision to enhance the aleatoric uncertainty estimation and noise-augmented vicinal risk minimization for epistemic uncertainty estimation. The proposed method is used for OOD detection and generalization on several common image datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well written and most information is clearly presented.\n\n2. Without checking all mathematical details, judging from the high level propositions and conclusions of the theoretical analysis, the theorems are of significance for the uncertainty estimation problem. The theoretical analysis is also essential to the proposed method, showing good consistency. \n\nThe main novelty with the theoretical framework is the detailed analysis on learning from level-0/level-1 supervision with level-2 models.  Although level-2 models have already been studied by many (e.g. [3],[2] from the references), Theorem 1 and Proposition 1 formally identifies the insufficiency of traditional risk minimization of level-2 models that also includes the aleatoric uncertainty perspective (rather than epstemic uncertainty alone). To my knowledge, this is a novel contribution and reveals a meaningful characteristic of evidential learning.  \n\nThe proposed method is not limited to a specific learning task, and the difference comes from the supervision side of things (X and Y). This makes the solution more applicable to various problems, and aligns with the trend of introducing noise into machine learning. This also shows the advantage of VRM in the uncertainty quantification aspect, in addition to generalization. \n\n3. The experimental results show good OOD detection and generalization performances. A sufficient number of recent EDL baselines are included in the comparison.\n\nWeaknesses:\n\n1. As mentioned in the paper, the evaluation is limited to CIFAR scale datasets and the OOD behavior on more realistic tasks is not clear.\n\n2. In the hyperparameter ablation study, the accuracy trend is not clear and the analysis is limited.\n\n3. The notation of $\\mathbb{P}_1$ and $\\mathbb{P}$ from lines 75-77 can be confusing.\n\nQUESTIONS:\n1. How realistic are the assumptions in Theorem 1?\n\n2. Can the analysis in Theorem 3 be generalized to other types of evidential losses?\n\n3. Is there more analysis on the ablation study in Figure 3?\n\nLIMITATIONS:\nThe limitation of datasets and tasks is briefly mentioned, but could be better discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}