Hyper Evidential Deep Learning to Quantify Composite Classification Uncertainty
Authors: Changbin Li
Kangshuo Li
Yuzhe Ou
Lance M. Kaplan
Audun Jøsang
Jin-Hee Cho
DONG HYUN JEONG
Feng Chen
Venue: ICLR 2024 poster (2024)
Keywords: Evidential Neural Network
hyperdomain
vagueness
Primary area: general machine learning (i.e., none of the above)
TL;DR: We propose a novel framework called Hyper-Evidential Neural Network (HENN) that explicitly models predictive uncertainty caused by composite set labels in training data using a belief theory called Subjective Logic (SL).

Abstract: Deep neural networks (DNNs) have been shown to perform well on exclusive, multi-class classification tasks. However, when different classes have similar visual features, it becomes challenging for human annotators to differentiate them. When an image is ambiguous, such as a blurry one where an annotator can't distinguish between a husky and a wolf, it may be labeled with both classes: {husky, wolf}. This scenario necessitates the use of composite set labels. 
In this paper, we propose a novel framework called Hyper-Evidential Neural Network (HENN) that explicitly models predictive uncertainty caused by composite set labels in training data in the context of the belief theory called Subjective Logic (SL).
By placing a Grouped Dirichlet distribution on the class probabilities, we treat predictions of a neural network as parameters of hyper-subjective opinions and learn the network that collects both single and composite evidence leading to these hyper-opinions by a deterministic DNN from data.
We introduce a new uncertainty type called vagueness originally designed for hyper-opinions in SL to quantify composite classification uncertainty for DNNs.
Our experiments prove that HENN outperforms its state-of-the-art counterparts based on four image datasets.
The code and datasets are available at: https://shorturl.at/dhoqx.

Published as a conference paper at ICLR 2024
HYPER EVIDENTIAL DEEP LEARNING TO QUANTIFY COMPOSITE CLASSIFICATION UNCERTAINTY
Changbin Li', Kangshuo Li', Yuzhe Ou' Lance M. Kaplan', Audun Josang', Jin-Hee Cho', Dong Hyun Jeong'. Feng Chen'
Department of Computer Science, The University of Texas at Dallas', US Army Research Laboratory². University of Oslo Virginia Tech". University of the District of Columbia
[changbin. 1i, kangshuo 1i, yuzhe. . ou, feng. cut dallas edu lance. m. kap ai vaarry................. mi 1, audun. josangemn uic. no, djeong@udc. edu, jicholvt.edd
ABSTRACT
Deep neural networks (DNNs) have been shown to perform well exclusive, multi-class classification tasks. However, when different classes have similar visual features, it becomes challenging for human annotators to differentiate them. This scenario necessitates the use composite class labels. this paper, we propose novel framework called Hyper-Eividentaal Neural Network (HENN) that explicitly models predictive uncertainty due to composite class labels training data in the context of the belief theory called Subjective Logic (SL). By placing a grouped Dirichlet distribution on the class probabilities, we treat predictions of a neural network as parameters of hyper-subjective opinions and learn the network that collects both single and composite evidence leading to these hyper-opinions by a deterministic DNN from data. We introduce new uncertainty type called vagueness originally designed for hyper-opinions in SL to quantify composite classification uncertainty for DNNs. Our results demonstrate that HENN outperforms its state-of-the-art counterparts based on four image datasets. The code and datasets are available at: https: /ttp//gttt/////// con/
INTRODUCTION
In various applications, particularly those dependent on data from low-quality sensors or high-quality data with insufficiently distinct features to separate some individual classes, the resulting data often exhibits significant vagueness and ambiguity (Allison, 2001: Ng et al., 2011). For example, in security surveillance, grainy images from store cameras may not provide clear enough resolution to accurately distinguish between different individuals activities, necessitating the use of composite class labels to address this uncertainty (Allison, 2001). Similarly, in the field of medical imaging, a radiograph displaying features suggestive multiple possible diagnoses may require composite labels to capture this uncertainty (Allison, 2001) effectively. When different classes have similar visual features in image datasets, it becomes challenging for human annotators differentiate them. An ambiguous image, such as blurry one where an annotator cannot distinguish between a husky and a wolf, may be labeled with both classes: (husky, wolf). The composite label implies that the image belongs to husky wolf, but not both. When training data consists of composite class labels, existing DNN methods face the following challenges: (a) how train a DNN model based on training set with composite labels; (b) how to train DNN to predict the composite labels that human annotators could provide; and (c) how to quantify the predictive uncertainty of a DNN due to the evidence of composite labels collected from the training set.
In current literature, partial label learning (Feng et al., 2020; Hong et al., 2023) has been proposed to address the first challenge. It aims to train DNN that disambiguate the partially- samples and predict singleton class labels for testing data. To address the second challenge, conformal prediction (Vovk al., 2005; Romano et al., 2020; Angelopoulos et al., 2021) is typically considered
training

---

Published as a conference paper at ICLR 2024
in safety-critic applications (e.g., computer vision based medical diagnostics) and aims to provide a composite set that covers the true class label (e.g.. the true diagnosis) with high probability (e.g.. 90%). A composite set generated by conformal prediction method is due to high entropy in the predicted class probabilities rather than composite class labels in the training set.
To the best of our knowledge, limited work has been conducted to address the third challenge. For predictive uncertainty quantification, several types/sources of predictive uncertainty have been studied in deep learning: model uncertainty (mutual information between model parameters and the predicted class probabilities (Depeweg et al., 2018; Malinin & Gales, ()), data uncertainty (entropy of the predicted class probabilities (Gal, 2016)), confidence (the largest predicted class probability (Hendrycks & Gimpel, 2017)), vacuity (uncertainty due to lack of evidence (Jøsang, 2016; Shi et al., 2020)), and dissonance (due to conflicting evidence (Zhao et al., 2020)). However, lacks an effective uncertainty measure (here we name vagueness) that can quantify the uncertainty associated with predictions of DNN due to composite class labels in the training set. For example, if the prediction (e.g., singleton class or a composite class) of DNN for a given input sample is based on evidence collected from training samples mostly labeled to composite sets, the vagueness should be high. If the collected evidence is from training samples mostly labeled to singleton classes, the vagueness should be low.
We propose new framework called Hyper Evidential Neural Network (HENN) that explicitly models the predictive uncertainty of a DNN due to composite class labels in the training set. HENN is designed based on the theory of Subjective Logic (Jøsang, 2016) and aims to predict the evidence parameters of a hyper-opinion regarding the classification of the input sample. A hyper-opinion defines a belief mass distribution on the composite sets of singleton classes and an uncertainty mass and can be equivalently represented by a grouped Dirichlet distribution (GDD). We introduce a new uncertainty measure based on hyper-opinions, originally designed in SL (Jøsang, 2016; Jøsang et al., 2018), to quantify the vagueness of a DNN. Our contributions are three-fold: (1) We propose a novel framework (HENN) that can quantify vagueness, a new uncertainty type for measuring the predictive uncertainty of a DNN due to composite class labels in the training set. (2) We propose new loss function, uncertainty partial cross entropy (UPCE), for HENN training. UPCE is a generalization of the well-known uncertainty cross-entropy (Snsoyy et al., 2018; Biloš et al., 2019) designed for singleton class labels. We provide a theoretical analysis of UPCE and propose a regularization term to future improve the effectiveness of UPCE for HEÑN learning. (3) We conduct extensive empirical analyses on four image datasets to demonstrate the effectiveness of the HENN in comparison with five competitive baselines.
RELATED WORK
Evidential Neural Networks (ENNs) (Sensoy et al., 2018; 2020; Ulmer et al., 2023) are deterministic neural networks that predict subjective opinions (Dirichlet distributions, equivalently) about the classifications of the input samples. The predicted subjective opinions can be used to quantify predictive uncertainties, such as vacuity (due to lack of evidence) and dissonance (due to confficting evidence). PriorNet (Malinin & Gales, 2018; 2019) and PosteriorNet (Charpentier et al., 2020) are in this category. While Bengs et al. (2022) investigated the flaw of second-order uncertainty estimation of ENNs because of the lack of ground truth of target distribution, many applications (Xie et al., 2023; Sun et al., 2023; Park et al., 2023; Sapkota & Yu, 2023) show the usefulness of ENNs in recent years.
Partial label learning aims to train a DNN that can disambiguate the partially-labeedd training samples and predict singleton class labels for testing data. Average-based methods (Cour et al., 201 treat each candidate label as equally important during training. Conversely, identification based approaches (Feng et al., 2020; Xu al., 2021; Wang et al., 2022a; Qiao al., 2023; Yan & Guo, 2023a;b; Hong et al., 2023) aim to disambiguate the effect of noisy labels and maximize outputs based on the most likely "ground-truth" label. Soft label learning aims to aggregate labels collected from multiple annotators to create probabilistic "soft" labels for training data and learn DNN for singleton class predictions based on the soft labels in the training data (Peterson et al., 2019; Collins et al., 2022). However, both partial and soft- learning methods are limited to singleton-class predictions but not composite set predictions. Their learned models do not provide uncertainties associated with singleton-class predictions due to composite class labels in the training set.
Compusite set prediction. E-CNN (Tong et al., 2021) could do set prediction for any possible combinations among all singleton classes based on Sempster-Shafer theory. RAPS (Angelopoulos
2

---

Published as a conference paper at ICLR 2024
et al., 2021) is state-of-the-arr conformal prediction method that gives more stable predictive sets by regularizing the small scores of unlikely classes after Platt scaling. However, these methods predict composite sets for data instances with large probabilities for multiple classes. This means that their locations in the representation space are near the decision boundary of the DNN. In contrast, HENN predicts composite sets for data instances near the training instances with composite labels. These two methods are considered baselines in our empirical study.
3 HYPER-OPINIONS AND EVIDENTIAL UNCERTAINTY MEASURES
3.1 HYPER-OPINIONS IN SUBJECTIVE LOGIC AND GDD
In the Dempster- Shafer Theory of Evidence (DST) (Shafer, 1976), class probabilities in the Bayesian theory are generalized to belief masses in subjective opinions. It assigns belief masses to subsets of ground set of exclusive possible states or classes (called "domain"). One can then express ' do not know' by assigning all belief masses to the whole domain as an opinion for the truth over possible classes. SL formalizes the DST's notion of belief assignments using a hyper-opinion. More specifically, let Y = (1...., K} denote the class domain with the cardinality K. Let (Y) denote the reduced power set of Y (called "hyper-domain)) which is the set of the power set of Y that excludes {K}}. A hyper-opinion w (b, u) assigns belief mass by each element (singleton class or composite set) S E (Y) and provides an uncertainty mass of u called vacuity. These mass values are all non-negative and sum up to one, i.c.,
{ and {0}. Let (()) denote the set of composite sets: '6( = ( \ {{I},
SE by = 1.
(1)
A belief mass by is computed using the evidence for each element S (Y).. S represents a singleton class if it has single class element (e.g.. S {1}); otherwise, it represents a composite set (e.g., S = (1,2)). Let es 2 0 be the evidence derived for S, then the belief by and the uncertainty mass u are computed as:
es T
K
by
and
(2)
T
where T = E Es es + K. The uncertainty mass is inversely proportional to the total evidence: Eseary) es. When the total evidence is O, the belief mass for each S needs to be O, and the uncertainty mass is 1. In contrast Bayesian modeling terms, we define "evidence"" a measure of the accumulated support from training samples, indicating that the input sample should be categorized into a particular singleton class or composite set. The accumulated support can be interpreted as the weighted aggregated number of training samples that support this class or composite set. Unlike a simple count of samples, evidence typically weighted. This means that not all samples contribute equally to the evidence. For instance, some samples might be more informative reliable than others, and the network learns to weigh their contribution to the evidence accordingly. Fig. shows examples of high uncertainties for different types and their corresponding probability density plots for 3-class classification.
A hyper-opinion can be equivalently represented by hyper-Dirichle, distribution of the class- probability vector p Ak. where A = {p>E Ps: = and pk E [0, 1]} is the K-dimensional simplex. It is characterized by class-specific concentration parameters a = [.,,-.., oxx] and set- specific concentration parameters e = [c)]sew(r) The probability density function (pdf) for possible
values of the class class-probability vector is given by
K
HyperDir(pla,c) c) 2
II
PE
II
Σ
Ph
"8
for p DEAK,
k=
SE
RES
where Zn is the normalization constant that has no analytical form. The concentration parameters of a Siree--Dricchett distribution HyperDir(pl c) can be mapped to the evidence parameters of a hyper-opinion as follows: ag = e + 1 for ie 1.... K and cs = es, VS E (V)).
This paper considers an important special instance of Hyper-Dirichler distribution: the grouped Dirichlet distribution (GDD), as it offers the practical appeal of an analytical normalization factor that can be easily calculated. GDD assumes that the composite sets in (Y) represent partition of the ground set of singleton classes, i.e., = (Si, .S., S, where UJ_, = and S, nS, = 0.
vi,j E {l,..., 17, and i * j. Let ej denote cs,. The pdf of GDD has the following form:
K
GDD(Pla, c) -
P ok
II
Σ pi) I
for P E Ak.
(4)
A=1
J=1
3

---

Published as a conference paper at ICLR 2024
aq + Cg. and B(-) is beta function. We aim to design and train evidential neural networks that can effectively predict the hyper-opinions about the uncertainty-aware classification of the input sample. As discussed in the following subsec- tion, the predicted hyper-opinions can quantify vagueness and other uncertainty types.
where -= = I['- (tathes)) ]
(,
Relations with multinomial opinions and Dirichlet distribution. A hyper-opinion is a generalized version of the multinomial opinion that assigns belief masses to singleton classes but not composite sets (Jøsang et al., 2018). In particular, if there is no evidence for the composite sets in ((Y), then and the resulting hyper-opinion only assigns belief
es 0, AS E (()). It follows that masses to singleton classes. and the corresponding Hyper Dirichlet distribution HyperDir(a, 0) is equivalent to the Dirichlet distribution Dir(a).
3.2 VAGUENESS AND OTHER EVIDENTIAL UNCERTAINTY MEASURES BY HYPER- OPINIONS
SL explicitly represents second-order probabilistic uncertainty through a hyper-opinion consisting of a belief mass distribution on (YY) and uncertainty mass. A hyper-opinion can be used to quantify different types of uncertainty, such as vagueness (due to composite evidence), vacuity (due to lack of evidence), and dissonance (due to conflicting evidence). The vagueness uncertainty measure (also named total vague belief mass) of hyper-opinion can be estimated as:
(ag(w) = Σ SE((E)) bg.
(5)
An opinion is totally vague when vag(w) 1, and is partially vague when 0 < (w) 1. An opinion has voguunaageessss when only a single composite set has (vague) belief mass assigned to it. On the other hand, an opinion has pluri-vagueness when several composite sets have (vague) belief' masses assigned to them.
The vacuity uncertainty corresponds to the uncertainty mass u in a hyper-opinion and is calculated as vac(w)= = ( in Eq. 2. The dissonance of a hyper-opinion can be derived from the same amount of conflicting evidence for different singleton classes or composite sets (see 5q.54 in App.) for its estimation based on the hyper-opinion). The vagueness vag(w) is different from the vacuity vac(w) in that vagueness results from existing evidence of composite sets that fail to discriminate between specific singleton classes, but vacuity reflects the lack of evidence for any singleton classes and composite sets. A totally vacuous opinion does not contain any vagueness by definition. The vagueness (w) is different from the dissonance diss(w) in that vagueness is due to evidence on composite sets, whereas dissonance reflects confliciigg evidence collected from different singleton classes or composite sets. It is possible that opinion has a high vagueness (e.g., vag(w) 1) but low dissonance (e.g., diss(w) = O). Hyper-opinions can contain vagueness, whereas multinomial opinions never contain vagueness. The ability to express vagueness is thus the main aspect that makes hyper-opinions different from multinomial opinions.
Table An example of hyper-opinion with low vacuity and dissonance but high vagueness.
S (X)
{1}
(2)
{3}
{1.2}
(1.3}
(2.3)
vac(w)
diss(w)
vag(w)
Evidence og
3
0
0
D.
24 0
Belief Mass be:
0.8
0.1
0.2
0.8
Evidenoo egg
0
0.1
0.744
Belief Mass big
0.1 0.4
0.4
Tab. provides two examples (Y (1, 2),, 3). The first example of hyper-opinion reflects high vagueness. There is high evidence observed on the composite set {2, 3). and it causes high vagueness. There also conflicting evidence between {1} and {2,3) 3) that contributes to dissonance. However, the evidence on {2, 3} dominates the evidence on {1}. the dissonance is low. The total evidence is large for K 3, and it results in low vacuity. The second example a hyper-opinion with low vagueness and high dissonance, where the evidence in classes 2 and 3 is equally distributed on singletons instead of composite set and becomes the conflicting evidence between the two classes
4 HYPER EVIDENTIAL NEURAL NETWORK
In this section, we will present novel hyper-evidential neural network (HENN) that predicts hyper-opinion about the classification of the input feature vector x. The predicted hyper-opinion can be used to quantify different types of predictive uncertainty, such as vagueness, vacuity, and
4

---

Published as a conference paper at ICLR 2024
Exilenoe
Uniar
puudyy Wart
0.
Confident
(b) Dissonance
Hone
2-151,,24,0044
Huddy
Hipply
HENN
Siudiy Warl
Hashh
Will
20.3
15.5
14.7
P 0-4.1.945,004
di 0.5
(c) Vacuity
(d) Vugneness
P: projected probabiilty
1og veguanane uncertt riy
diAi disecrsaion uncertrenty
Examples different uncertainties.
Different predictive uncertainties from HENN.
Figure 1: Left: Different prohability densities corresponding specific uncertainty type for 3-class classification (Brighter colors mean higher density). Fach corner represents a class. (a) confident prediction. (h) Conflicting evidence exists for two classes (dissonance or data uncertainty). (c) Uniform Dirichlet distribution with no evidence for known classes (i.... OOD imputs) (vacuity uncertainty). (d) There is enough evidence to exclude one class but still fail to determine the final prediction from the rest of the classes. Right: The first example shows a confident prediction w/o vagueness and low dissonance. The other two examples have the same projected probabilities but different sources uncertaintics. One is caused by conflicting evidence (dissonance), and the other one is caused by vague evidence only for the final decision from the set (Husky, Wolf (vagueness). (dg.(d) is drawn by grouped Dirichlet distribution, not ordinary Dirichlet distribution.
dissonance, as discussed in Section 3.2. We consider the scenario where the composite sets form a partition of the ground set of singleton classes, = {S1, ...,,,,.,,,. S, and the hyper-opinion can be equivalently represented by grouped Dirichlet distribution. Formally, the HENN is defined function f(,, RD + R x K-) mapping an imput x R the evidence vector e IR K** where O are the network parameters, e = [e1, ... ee,, ES. es,,] and e₁ and es, refer to the predicted evidence values of the singleton class and the composite set S, respectively. The architecture of HENNs for classification is similar to classical neural networks. The only difference is that the softmax layer is replaced with an activation layer (e.g.. Softplus or ReLU) ascertain non-nege non-negative and unbounded output that is considered as the evidence vector for the predicted hyper-opinion (or grouped Dirichlet distribution, equivalently). Based on the predicted evidence vector, we can then predict the singleton class or composite set that has the largest evidence:
(6) If m E (1...., K}. then the prediction is a singleton class; otherwise, it is the composite set Sm-K E S. We can also transform the evidence vector e to a grouped Dirichlet distribution GDD(p az c) based on the mapping between the parameters (a, c) and e: [er + I,... EEK + 1] and c [es,.... es,] The relations between this distribution GDD( c), the class probability vector p. and the class label y have the form:
arg (1..,....
y ~ Cat(p), p ~~ GDD(a, c), e ff;
(7)
where Cat(p) is categorical distribution on the class variable y. The expectation of the class- probability vector p has the form:
p Ep-aaotla....l)) Pa = EEpx] = ak Σ Bi
-lke E E.))) for k E (1.--,,K)
1
(8)
Bo as;
where Bg = Σラ= 3j. os, = Eies, an Bj as, + Cg. Then, use as the projected class probability vector, we can also predict the singleton class with the largest projected class probability:
y = arg max
(9)
Relations with ENNs: ENNs (a.k.a prior networks (Malinin & Gales, 2018)) and their variants (e.g.. posterior networks (Charpentier et al., 2020)) are deterministic neural networks that predict the multinomial opinion (Dirichlet distribution, equivalently) about the singleton class label of the input sample. In comparison, HENNs are deterministic neural networks that predict the hyper-opinion (GDD, equivalently about the classification of the input sample into a singleton class or composite class label. As discussed in Section 3.2, hyper-opinion has the ability express the vagueness uncertainty (due to evidence collected from composite labels in the training data). but multinomial opinions never contain vagueness. HENNs are designed to handle composite labels in the training set and can quantify the composite classification uncertainty using the vagueness measure, whereas ENNs can not. addition, as multinomial opinion a special instance of hyper-opinion, by setting the evidence on composite sets to zero, HENNs include ENNs as special instance.
5

---

Published as a conference paper at ICLR 2024
4.1 THE LOSS FUNCTION AND REGULARIZATION FOR HENN LEARNING
Let D = {(x(0, 5 denote a training set, where x refers to the input and y E {0, 1]. refers
label. For instance, 5 [0, 1,, 1,110) O, represents composite set label {2, for the sample i and y( [0, 0, 1,0] represents a singleton class label 3. In the related task of partial label learning (Cour et al., 201 1), the partial cross-entropy (PCE) is used loss function for learning softmax- -baee NN based on composite set (or called partial) labels:
to the binary vector representation of singleton class label or composite
PCE(p. - log Σ (cPe).
(10)
where p refers to the class probability vector predicted by the softmax layer of the NN. When ý is singleton class label, the PCE loss becomes equivalent to the standard cross-entropy (CE) loss:
CE(P. y) Ek=1 Ti Pk. Pk- As the output of a HENN function, namely, Uncertainty Partial Cross Entropy (UPCE), to learn the parameters of HENN:
GDD of p. we propose new loss
UPCE 5; 0) p0p a e (PCE(P.
(11)
where 0 refers to the network parameters of the HENN. We note that if is a singleton class label, and we replace GDD with the Dirichlet distribution, then the UPCE loss becomes equivalent to the default UCE loss used in learning ENNs: UCE(X, = E (p-rir(plaa) [CE(p, V)]. Our proposed UPCE loss has the following analytical form (see our Proposition Al in App. B.2 for derivations):
UPCE(X, 0:0)= = (0(86)) - ((((e)) "115")
|1
1)+
(12)
[m0(880) v(als))
Σ
(W((8)) - (0) 1 E (S)) {)))1(05 1),
where the first term corresponds to composite example and the second term refers to singleton example respectively. In particular, vi(-) is digamma function, a(1) = 1 = =1( + +||c[1
(8) (1) = Eies, aj is the sum of strength parameters corresponding all singleton classes in the partition S,. For simplicity, we let 91 represent IC-th B corresponding to one of a composite label in the list (S S,......,, which contains the singleton ground truth, and ays denote IS-th a corresponding to the singleton target.
denotes the sum of all positive strength parameters for the i-th sample, Ss
Our proposed UPCE loss function has the lower bound (see Proposition A2 in App. C.I):
UPCE(x, 5: 0) > PCE 5:
(13)
It follows that the minimization of the UPCE loss ensures the minimization of the PCE loss between the projected class-probabilit vector Eap((pia.c) [p] and the composite set label y. This result indicates a favorable property of UPCE: The HENN with the UPCE loss function is optimized to output high projected probabilities for the classes belonging to the composite set label but low projected probabilities for the other classes.
However, as shown in Proposition 1, we observed an issue with the UPCE loss function in differenti- ating the evidence values of singleton classes and composite sets. In particular, when ý a composite set label, the learned HENN based UPCE tends to predict large evidence values for both the composite set label y and for all the singleton classes belonging to the composite set. Similarly, when ý is a singleton class label, the learned HENN based UPCE tends predict large evidence values for this singleton class and all the composite set that contains this singleton class as an element.
Proposition (Properties of the empirical UPCE risk function). Assume that the universal ap- proximation property (UAP) holds for a HENN, iee, the network can learn arbitrary mapping function from the input feature vector x to the evidence vector e. Then, the empirical UPCE risk function R(f)= = > EN (PCE(x()) 50) approaches the infimum 0 if the solution 6" satisfies the following properties, with = fx:: )r): (1) v(x. E DD where ý denotes a singleton class label, & E [K] the predicted evidence values Gk + 400 and es, - +00, VS; E S, such that k S and (2) v(x, E DD where denotes composite set label S, the predicted evidence values es. + +oc and ex + +o0, Vk E S,
To address the previous issue, we propose the following KL-diiergencce regularization term (see App. B.3 for derivations) to make the evidence output more flat:
Reg(x, y: 0) kl. [DD( @)][GDD(p|] (0)).
(14)

---

Published as a conference paper at ICLR 2024
where KI(-) is KL-divergence, and ā = = + parameters after the removal of ground- -ounthtruth parameters from the predicted parameters a and c. This regularization term is designed to enforce misleading evidence from the false single and composite classes in a and c to be as small as possible. The regularized UPCE loss function has the form:
and è = (1 ÿ) o are the GDD
[()) 1 Σ N
(UPCE(X*) s 0) + À Reg(x!), 3 0)
(15)
where À is the tradeoff coefficient. As indicated in Proposition 2 below, the HENN, when trained using the aforementioned regularized UPCE loss, tends to predict high evidence for the ground-truth singleton class/composite set while predicting low evidence for other elements. Stochastic gradient descent (i.e., Adam) is adopted to optimize the regularized loss function. The pseudocode is shown in (Algo. 1).
Proposition 2 (Effectiveness of the regularization term Reg(x,) 5: 0)). Following the UAP assump- tion, the regularized empirical UPCE risk defined in Eq. (15) approaches the infimum 0 if the solution O* satisfies the following properties: 1) v(x,8) E DD where ÿ singleton class la- bel k E [K, the predicted evidence values ea + +00 and e + 0, VI E S U [K] \ k; and 2) v(x. y) E DD where y denotes a composite set label Sy, the predicted evidence values es, - +00 and e + 0, Vt E [U[K]
The proofs of the Propositions are shown in App. C.2 and C.3, respectively. This is consistent with the fact that the Dirichlet distribution is a special instance of GDD. As a generalized framework of ENN, the HENN learned based on UPCE will perform similarly to the ENN learned based UCE when trained based on the same dataset with only singleton class labels.
Limitations and discussions. In essence, the proposed HENN is the GDD extension of evidential deep learning (Ulmer et al., 2023) that is hased upon Dirichlet distributions. The propositions demonstrate the need for the KL regularization term in the cost function so that only evidence for the corresponding ground truth class can grow large. While the assumption of UAP in the above propositions may not hold in practice, the analysis does demonstrate how UPCE requires the KL regularization term to moderate the evidence. An ablation study is discussed in Section 5.2 to empirically demonstrate the need for the regularization term.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets & Preprocessing. TinyImageNet (Fei-Fei et al., 2015), Living17 (Santurkar et al., 2021), 1), Nonliving26 (Santurkar et al., 2021), and CIFARI00 (Krizhevsky & Hinton, 2009) are used in the experiments. Each dataset has class hierarchy relation. For example, CIFARI00 has 100 subclasses which are grouped into 20 disjointing superclasses. Superclasses are utilized to generate composite class labels because of their semantic and visual similarities. We first select a fixed number of composite class labels, denoted as M. Then several random subclasses for each selected superclass will be chosen. A subset of the selected images will be blurred by the Gaussian Blurring operation (RichardWebster al., 2018) generate vague images, and the corresponding set of categories/subclasses of these vague images will be the new label (composite instead of singleton) of these vague images build the dataset. Detail is presented in App. E.
Baselines. DNN is the traditional deep neural network model. ENN (Sensoy et al., 2018) is the evidential network that only deals with traditional singleton domains as DNN does. We also use UCE loss and KL regularizer for a fair comparison for ENN. In practice, it is necessary to set a threshold value of predicted conditional class probabilities to generate set predictions for DNNs and ENNs. (see App. E.3). E-CNN (Tong et al., 2021) 1) could do set prediction for any possible combinations among all singleton classes based on DST. RAPS (Angelopoulos et al., 2021) leverages conformal prediction to generate prediction set to ensure the size of the predicted set is as small possible. PiCO (Wang et al., 2022b) applies contrastive learning into partial label learning problem.
Implementation. Both HENN and ENN use Softplus as the activation layer. Since HENN is model- agnostic, we consider three pre-trained backbones: EfficientNet-b2 (Tan & Le, 2019), ResNet50 (He et al., 2015) and VGG16 (Simonyan & Zisserman, 2015) for HENN model and all other baselines for a fair comparison. Model agnoistic property experiments are represented in App. F.2 due to the space limit. To generate composite examples for baselines, create duplicate training examples with
7

---

Published as a conference paper at ICLR 2024
Table 2: Results (%) based on Gaussian kernel size: 3x3 on CIFARI00 and tinylmageNet. (The average of three runs is provided, and the confidence interval is included in the App. due to space limitations.)
LinyEmageNet CompIS
living17 ComplS
nondiving26 ComplS
M
Methods
OverIS
Acc
OverS
Acc
OverIS
Acc
DNN (Tan & Le, 2019)
83.4
66.9
79.8
88.1
81.0
83,3
85.6
62.0
82.9
ENN (Sensoy ot 2018)
75.9
63.5
80.7
88.0
72.3
84.5
85.0
52.9
10
E-CNN (Tong al., 2021)
33.4
31.1
68.2
30.5
36.8
65.7
28.3
35.8
60.6
RAPS (Angelopoulos et 2021)
73.1
43.6
79.8
86.4
61.3
83.3
82.7
46.3
82.9
PiCO (Wang al., 20226)
57.2
35.6
64.3
62.5
43.7
65.2
61.8
42.6
64.8
HENN (ours)
84.4
93.4
82.5
88.8
96.5
85.6
86.9
96.8
85.4
DNN (Tan & Le, 2019)
84.3
67.3
79.5
88.1
84.8
80.2
85.6
68.9
81.5
ENN (Sensoy et 2018)
83.5
60.7
81.2
88.0
78.3
82.4
85,4
62.6
82.9
E.CNN (Tong al., 2021)
32.5
33.3
68.4
31.6
37.3
65.5
29.8
35.1
60.1
RAPS (Angelopoulos et al., 202))
68.1
45.6
79.5
85.5
66.5
80.2
83.8
56.1
81.5
PICO (Wang et 20226)
35.3
64.6
61.4
43.1
42.5
64.6
HENN (ours)
84.6
90.6
81.6
88.8
96.6
85.7
86.9
96.2
84.1
different singleton labels in the composite set. We adopt grid search based on held-out validation set to select the best hyperparameters for each competitive method. Please refer to App. E,4 for details.
Evaluation Metric. Jaccard Similarity (JS) (Zaffalon al., 2012) is used evaluate model's [ouğن where ý is the predicted set of classes and y is ground-truth set of classes. Either 9. or both can be a single class or set of two or more classes. A model identifies datapoint as composite if two more classes are predicted and singleton otherwise. We compare HENN's performance with baselines in terms of different average JS. OverJS: averaged JSs of all test samples, Ei JS .0()). ComplS: averaged
performance in predicting set of classes: JS ú) = yyni|
ISs of composite samples the model identifies, 1 > (S( ()) g((),, where len(f()) 1. Here N. Ne E ((1en(g()) 1) denotes the number of examples which are predicted as composite sets. Accuracy is used to evaluate the projected singleton label prediction (Acc). The Area Under the Receiver Operating Characteristic (AUROC, the larger the better) is to measure the different uncertainties in discriminating between true composite and true singleton samples.
5.2 EXPERIMENTAL RESULTS
For each dataset, we consider different numbers of composite class labels during training: M= 10, 15. 20, and multiple Gaussian kernel sizes of blurring operation:3 x3, 5x5, 7x7. Due the space limit, some additional experiments including CIFAR 100 are presented in App. F.
Classification. Tab. 2 shows the results of composite predictions on IinylmageNet, living and nonliving26 in terms of OverJS, CompJS (for composite prediction) and accuracy (for singleton prediction) based on Gaussian kernel size 3x3. HENN outperforms other baselines in terms of OverJS (over 1% for tinylmageNet) and CompJS In particular, the improvement of CompJS is significant (over 15-20% for three datasets). This validates that HENN is not only able to recognize vague images, but also differentiate different vague images. In particular, both RAPS and E-CNN underperform HENN in terms compJS. This because RAPS and E-CNN are inclined to make set predictions if they be other difficult (not vague) images in which E-CNN and RAPS cannot make a single decision. Therefore, compared DNN, more examples will be wrongly predicted composite sets. On the other hand, Tab. 2 also demonstrate the efficacy of HENN in singleton prediction (Eq. 9) in terms of Acc. The improvement over 2-5% for three datasets. PiCO performs worse than HENN because the composite labels in its setting are randomly flipped, and it might not be able to deal with blurring images during training. In summary, HENN can generate high-quality composite prediction and accurate singleton label classification. It is practical to consider a limited number of composite sets due to the majority of clearly labeled data. So our experiments not encounter the combinatorial complexity issue (2*).
about the final prediction. In addition to the vague images, there might
Analysis of confusion between multiple classes. The ROC curves depicted in Fig. 2 show- case performances of various uncertainty indicators, namely vagueness of HENN, vacuity and dissonance of ENN, and entropy of DNN, in identifying confusion between multiple classes when some samples have composite labels, and some have singleton labels (see more in Fig. 4 and Fig. in App.). For both datasets, HENN's vagueness outperforms the other uncertainties, as indicated by its larger AUC score and smallest error region, making it highly effective dis-
8

---

Published as a conference paper at ICLR 2024
criminator between composite and singleton samples and successful indicator of confusion be- tween a set of classes when there is composite evidence. RAPS and E-CNN, however, do not provide any mcasurement to evaluate these uncertainties. ENN's vacuity, similar to epistemic uncertainty, which is more useful for OOD detection (Fig. 1) and is not suitable to our case. While dissonance and entropy are better than vacuity, they are still inferior to vagueness. A data point with high dissonance is usually lo-
cated in the decision boundary, and point with high vagueness can also be close the deci- sion boundary. However, the verse does not apply for vagueness, because vagueness could also be decided by the labeling bias of the an- notators, but not purely by their closeness to the decision boundary between the associated singleton classes. For instance, an annotator who has extensive knowledge about different cat breeds (i.e., Tabby, Egyptian, Persian), will still annotate them singletons, even if they are near decision boundaries. However, this annota- tor may give composite labels for other animal breeds, such as dog breeds (i.e., Husky, Malamute, Samoyed), that he may not be knowledgeable about. For this reason, a data point with high dissonance may likely have low vagueness.
True Pouitive Rate
0.8
Tue Positivee Rates
0.8
0.0
0.0
0.4
0.4
0.2
Veguevesa (MDC 1.01
0.2
Wegumees (MUC EEC
Evtropy 2UCCx0/011
0.2
0.4 0.6 0.8
0.2
0.4
0.6 0.8 1
False Positive Rate
Positive Rate
(a) CIFAR100 M=15
(b) minylmageNet M=15
Figure 2: ROC curves of separating composite and sin- gleton examples among different measurements: vague- ness of HENN, vacuity and dissonance of ENN, and entropy of DNN on based on kernel size 7x7.
Effect of regularization. To show the effec- tiveness of KL. divergence regularization, we compare different regularizations and UPCE loss without any regularization Tab. 3.HENN- KL refers to the HENN with the proposed regularization (Eq. 14). HENN-Ent refers to the HENN with the entropy of GDD the regularization Reg = -H H [CD((pla, c)] (see App. B,4). KL-Drr refers to the HENN using KL-divergence only for singleton classes Reg KL Dir(pla)) Generally, the compar- ison of their performances is HENN-KL KL-L-LEENNNE > HENN-KL-Dir > HENN-only-
Table 3: Model performance of different regular-
ization on M=15, and kernel size: 7 7x7 on nonliv- ing26.
Methods
OverIS
Comp15
Acc
HENNN only-UUCCC UPCE
78.25
62.89
84,96
HEEN--KL-E
86.66
94.15
82.13
HENN Ent
86.68
94.44
86.33
HENN-KI KI.
86.93
94.78
85.19
App. F.4 illustrates the coefficient effect of the KL regularizer.
Effect of varying numbers of composite la- bels. To investigate the effect of ratio of com- posite class labels during training, we vary M = {10, 15, 20} in experiments. Fig. shows OveralIJS and Accuracy regarding the number of composite sets. Regularized HENN outper- forms other baselines for these two metrics. In particular, with the increase of number of com- posite sets, the gap between HENN and base- lines is enlarging in terms of accuracy (Fig. 3b), which demonstrates the advantage of HENN.
ACC
22
HaNH
UNR
RAYS
Composili Suts
Composite
(a) OverJS
(b) Accuracy
Figure 3: OverIS and Accuracy trends vs. the number of composite labels in IinylmageNet.
CONCLUSION
In this work, we propose a novel hyper-evidential network framework (HENN) designed to predict hyper-opinions and quantify predictive classification uncertainty caused by composite class labels (introduced as vagueness) by utilizing composite training examples. This framework is capable of identifying either singleton class composite set with the highest belief, and it predict the singleton class with the greatest projected class probability, Extensive empirical findings show that HENN outperforms other competitive methods, demonstrating its effectiveness and potentiality.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
thank Raisaat Atifa Rashid for her contribution to initial dataset preprocessing. We thank anonymous reviewers for the stimulating discussion and for helping improve the paper. This work is supported by the National Science Foundation (NSF) under Grant No DMS 222055222774 FAI-2147375, IIS-2107450, IIS- IIS-2107451, and IIS-1107499.
REFERENCES
Russell Alan Hart, Linlin Yu, Yifei Lou, and Feng Chen. Improvements uncertainty quantification for node classification via distance-based regularization. Advances in Neural Information Processing Systems (2023), 2023. URL https: //tp:///aaivv//aa wttp://oo///...../00999999
Paul D Allison. Missing data. Sage publications, 2001.
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael I. Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net 2021. URL net Corummid==eNiii
Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. Pitfalls of epistemic uncertainty quantifi- cation through loss minimisation. In Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:/ wtwppppeeevvveeeeeeeeeeee/ppeeee net forum? Forum?id=eppjx. ARZW5.
Marin Biloš, Bertrand Charpentier, and Stephan Günnemann. Uncertainty on asynchronous time event prediction. In H. Wallach, H. Larochelle, A. Beygelzimer, F d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: http:///ccceeeeeeeeeeeeee neurips cc cc/paper. fi Files/papes 2019 http://feeeeeeeeeee2222222222262
Pwieeeeaappppppppp.eeeeee .dff.
Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. Posterior network: Uncer- tainty estimation without ood samples density-based pseudo-counts. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural In- formation Processing Systems, volume 33, pp. 1356-1367. Curran Associates, Inc., walle/paape htppp/pppp//0/// pdf.
neurips hcppppss.pe.rrrppppppppppp... 2020. URL ffle:00eccc00000aaeeee4000113333
Katherine M Collins, Umang Bhatt, and Adrian Weller. Eliciting and learning with soft labels from every annotator. In Proceedings of the AAAI Conference Human Computation and Crowdsourcing, volume 10, pp. 40-52, 2022.
Timothee Cour, Ben Sapp. and Ben Taskar. Learning from partial labels. J. Mach. Learn, Res., 12 10111) jul 2011. ISSN 1532-4435.
G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems (MCSS), 244):333-314, December 1989. ISSN 0932-4194, doi: 10.1007 BF02551274, URL htllt:/p//eee doi org /10 1007 /B/////22252222/
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 2,.1109/CVPR.200 200.56
Stefan Depeweg, Jose-Miguel Hemnandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decom- position of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pp. 1 1184-1193. PMLR, 2018.
Li Fei-Fei, Andrej Karpathy, and Justin Johnson. Tiny imagenet visual recognition challenge. URL http::/tiny-imageneee hww.://wwwwn....aaa....cmmm herokuapp.com, 2015.
10

---

Published as a conference paper at ICLR 2024
Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-labed learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10948-10960. Cur- ran Associates, Inc., 2020. URL https: httptttss////ccccpppttttt http://ooooeeeeeeeeee neur wttperrrrrppcccpprrreaeeae afleepeerrrs_/////eeeeeesepsssss htt.r/occeeennnneeee......c//c/e (eaprr22200/1111 htt.///eeee/dddffeee /ttp////11144446e466666ee66eee83
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and distrruittt examples in neural networks. In International Conference Learning Representations, 2017.
URL https: /ttp:////ppeee/eeeeeeeeeee
net/ / forum? id=Hkg4TI9x.
Feng Hong, Jiangchao Yao, Zhihan Zhou, Ya Zhang, and Yanfeng Wang. Long-tailed partial
label learning via dynamic rebalancing. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ooonpeerreeee ew ee. net. tent/foounm forum? d= fWoK4 4orK4KK SKsS.
Audun Jøsang. Subjective Logic: A Formalism for Reasoning Under Uncertainty. Springer Publishing Company, Incorporated, 1st edition, 2016. ISBN 3319423355.
Audun Jøsang, Jin-Hee Cho, and Feng Chen. Uncertainty characteristics of subjective opinions. In 2018 21 International Conference on Information Fusion (FUSION), pp. 1998-2005, 2018. doi: 0M.23919/ICIF/22 8.8554444
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprini arXiv:1412.6980, 2014.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken. Multilayer feed- forward networks with nonpolynomial activation function can approximate any func- tion. Neural Networks, 6(6):611-867 1993a. ISSN 0893-6080. doi: https://doi.or//10.
1016/80899-6080005800031333 URL https://www.scieett www.w/...eeeeeiitetttt.cc com/ rtt.c//sieeetttccccc/eeeeeeee science/ A.t.cclepp1//$88300005
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks, 6(6):861-867, 1993b.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: http:////cccceeenc.c/esccscsscss file/ ee 2 htts:///e22bbeeeeeeefceaffaaaaa3
wtwp:ooceeeiiinn.nenieeeeeeeeeee neurips wwwrips.cc/ppeeeeeeeeee pdff.
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved uncer- tainty and adversarial robustness. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus -uc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Cur- Associates, Inc., 2019. URL https: httpp//ppteeecseeeeeeppppo nwwrips......e cc/paper___p... files/ paper/2019/ (ttee////ee//// 7dd2ae 7 7 Bee B
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):
39 41, 1995.
Kevin P. Murphy. Probabilistic Machine Learning: An introduction. MIT Press, 2022. URL probml... ai.
Kai Wang Ng, Guo Liang Tian, and Man Lai TANG. Dirichlet and Related Distributions: Theory, Methods and Applications. Blackell,, United States, April 2011. ISBN 9780470688 199. doi: 10.1000 10.10002/7811111 11 19995784. Publisher Copyright: 2011 John Wiley & Sons, Ltd. All rights reserved. Copyright: Copyright 2018 Elsevier B.V., All rights reserved.
11

---

Published as a conference paper at ICLR 2024
Younghyun Park, Wonjeong Choi, Soyeong Kim, Dong-Jun Han, and Jackyun Moon. Active learning for object detection with evidential deep learning and hierarchical uncertainty aggregation. In The Eleventh International Conference on Learning Representations, 2023. URL https:
/ net. forum? d=MnE jsw-vjj jsw-vj-2
Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Feng Qi and Christian Berg. Complete monotonicity of a difference between the exponential and trigamma functions and properties related to modified bessel function. Mediterranean Journal of Mathematics, 10: 10:1685-1696, 2013.
Congyu Qiao, Ning Xu, and Xin Geng. Decompositional generation process for instance-dependent partial label learning. In The Eleventh International Conference on Learning Representations, 2023. URL /tpett///eeeeeeeeeeeeeee
Brandon RichardWebster, Samuel E Anthony, and Walter Scheirer. Psyphy: psychophysics driven evaluation framework for visual recognition. JEEE transactions on paftern analysis and machine intelligence, 41(9):2880-2286, 2018
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 3581-3591, Curran Asso- ciates, Inc., 2020. URL https: ppwoceddoeeeeeeeeooooooooc ings 2020/file 2etiF//1le/244e7788888522e
neuri i oc http:/ppprrfllleeeeeeeeeeeeeeeae satrr_fills/pppppe /ssp//pp/////////// .ddf
4eddd448ddded 60267 5edooe61117755555-5555
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. (BREEDS): Benchmarks for subpopulation shift. In International Conference on Learning Representations, 2021. URL
https:://pperev/ .ne.ee.ee////eeeeeeeee.eee.eeeee forum? Corrmmiid=m@PpByyAA
Hitesh Sapkota and Qi Yu. Adaptive robust evidential optimization for open set detection from imbalanced data. In The Eleventh International Conference on Learning Representations, 2023.
URL https:ttp///// //tp::///pppaae http:///ppeeeeeeeeeeeeeee net forum? 1d=3yU-hcJBqe.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning quantify classifi- cation uncertainty. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information P Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL hww.hhtttss//procceeeeeie 2018 2018/fFF a9t.ll/eee1/1100 2208///LLL.222222222200444466662
neuri ps cc wwtp/pape_ftte firf_iii /elleespppt pas/paper////
Murat Sensoy, Lance Kaplan, Federico Cerutti, and Maryam Saleki. Uncertainty-aware deep classifiers using generative models. Proceedings of the AAAI Conference on Artificial Intelligncce 34(04):5620 06000055777766 Apr. 2020.
Glenn Shafer. A Mathematical Theory of Evidence. Princeton University Press, 1976. ISBN 9780691 100425. URL http: www. istor.
Weishi Shi, Xujiang Zhao, Feng Chen, and Qi Yu. Multifaceted uncertainty estimation for label- efficient deep learning. Advances in neural information processing systems, 33:17247-17257, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: /ttp///aavvvv//////////eee/eawvv org/ abs 1409. 1556.
Shuzhou Sun, Shuaifeng Zhi, Janne Heikkilä, and Li Liu. Evidential uncertainty and diversity guided active learning for scene graph generation. In The Eleventh International Conference on Learning Representations, 2023. URL. http:::s////ppeevvvieeee net d= 10
12

---

Published as a conference paper at ICLR 2024
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Research, pp. 6105-6114, 14. PMLR, 09-15 Jun 2019. URL https: http://pooceeeeeee ngs ml
Machine Learning, volume 97 of Proceedings of Machine Learning
uress/v97/ee 9a html.
Zheng Tong, Philippe Xu, and Thierry Denoeux, An evidential classifier based on dempster-shafer theory and deep learning. Neurocomputing. 450:275-299 2021.
Dennis Thomas Ulmer, Christian Hardmeier, and Jes Frellsen. Prior and posterior networks: A survey on evidential deep learning methods for uncertainty estimation. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: http:::://opeerviewww net ferrrorumm
id=xq$8k9E75c.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 595 604, 2015.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29, Springer, 2005.
Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. Solar: Sinkhorn label refinery for imbalanced partial-label learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022a. URL httph:tt//// oppnrevieeeeeeeeeeeeeeeeeeeeeeee net/ / forum? ? i WUU.G
Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng. Gang Niu, Gang Chen, and Junbo Zhao. PiCO: Contrastive label disambiguation for partial label learning. In International Conference on Learning Representations, 2022b. URL https: //t///pprreee//e wwwie.eew..e...........w...w.w.. wwt.wewwwwwwwwewweeeeeeeeeeeeeee /ete/fooumt d-EhY EeeieEYyyyyyyee 22y gJ.
Mixue Xie, Shuang Li, Rui Zhang, and Chi Harold Liu. Dirichlet-based uncertainty calibration for active domain adaptation. In The Eleventh International Conference on Learning Representations, 2023. URL https: http////eeeeeeeeeeeeeeeeeeeeeeee net
Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https::/ /ttp////eeeeeiieeeeee/ee/e. net / forum?id- AnJUTpZiiWD.
Yan Yan and Yuhong Guo. Mutual partial label learning with competitive label noise. In The Eleventh International Conference on Learning Representations, 2023a. URL https::/
fortmform?ia - BUrxG8 3i IBCrC.
Yan Yan and Yuhong Guo. Partial label unsupervised domain adaptation with class-prototype
alignment. In The Eleventh International Conference on Learning Representations, 2023b. URL
https: httpss://pppeeeeeeee http///ppneeeeeeeeeeeeeeeeeeeeep .nt// / forum?id jpqCqgggw3t.
Marco Zaffalon, Giorgio Corani, and Denis Mauá. Evaluating credal classifiers by utility-dicouuntdd predictive accuracy. International Journal of Approximate Reasoning, 1282))1222-1311 2012 doi: http:://doi.org//10/ http://1..1011....aa2222...00...
URL https: http///wwww.../////// www.::/wceenee.eoo rect com/
Xujiang Zhao, Feng Chen, Shu Hu, and Jin- Cho. Uncertainty aware semi-supervised Iearn- ing on graph data. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12827-12836. Curran Asso- ciates, Inc., 2020, URL http::::///oocccinnntttt///////t neurips cc cctccc//ppaee sareerrffieeeeeeeeeeeeeeeeeeeree 2000E11eeBBebbao00000eeeeeeeeeee 1at.aa111-Paap.pppp.c
13

---

*8 Published as a conference paper at ICLR 2024
A NOTATIONS
For clear interpretation, we list main notations used in this paper and their corresponding explanation, as shown in Table 4.
Table Important Notations and Descriptions
Notation
Description
Y
Domain of singleton elements or classes
Hyper-domain of Y
8(V)
Domain of composite classes Training data with size NV Beta function, Gamma function, Digamma function, trigamma function Dirichlet distribution with strength
D= (xx() ((x())(())) )
B()) v())or)))
Dir(a)
CDD(a,
Grouped Dirichlet distribution with strength and composite evidence
K, MM
Total number of singleton (composite) classes
Normalizing constant of the Grouped Dirichlet distribution
A K
K-dimensional simplex, i.e., Ag = (plp = [pi, .Pk] [o, 1] and ||||| = 1) Singleton ground truth label Vague belief of value S in (V)
y
bs
Vacuity of evidence in hyper-opinion Total number of partitions
Total number of elements in (V), iee. the total no. of singleton and composite classes small error
w
Hyper-opinion a random hyper-variable y E (Y))
60
The feature vector the i-th sample
y
Binary vector over (0, 1)
b
[1, ...,...000........... bx. bs b 1
Belief mass distribution
[e,, 6x
Observed evidence vector over (VY), (e1, ,K, ex+1 ex]" Class probability vector over Y
[p PK
α [ai [kl
Strength vector of Dirichlet distribution or the singleton part in grouped Dirichlet distribution An element as a set in hyper-domain (singleton or composite)
(S ..., S
The set of partitions j-th composite set GDD
8
(el, cr
Evidence vector for the partitions in S
f(x00:) 0)
HENN parameterized by 0 that takes x input Singleton ground-trutt -ruth index Composite ground-truft -tuth index
IS
IC
(())
Uncertainty loss function w.r.t. parameters
(P)
UPCE loss
Re)(O)
KL-divergence regularizer
p(x)
Natural parameter based (only in Appendix)
((c)
Natural parameter based (only in Appendix)
u(p)
Sufficient statistic of natural parameter p(a) (only in Appendix)
v(p)
Sufficient statistic of natural parameter 7(c) (only in Appendix)
14

---

Published as a conference paper at ICLR 2024
B DERIVATIVES OF LOSS FUNCTION
B.1 EXPECTATION OF GDD
Theorem. Letx GDD 2 ( c) with 2 partitions, x An where D denotes the n-dimensional
simplex, a (a1,.... 0nn) J is the strength parameter, and c = (C1, (2), is the composite evidence parameter. Let St, S2 denote the partitions. The moment of x; is given by
B(x,) = 0i
(64 (.. 1 (
32 1
(16)
812 0si
082
where as, = Eies, ag as, Eies, ag, 31, 32, and B12 are defined as B1 = as, + c1, B2 = os, + c2, 912 = 81 + f2, and 1() denotes the indicator function.
According to the above Theorem which is from the book of Dirichlet and Related Distributions (Ng et al., 2011), analogy from two partitions to multiple partitions, we can get Eq. 8 in the main paper:
Elpx|
OCk Bo
Σ 9
1(k Si)
Us
where Bu E}= as, as, Eles, ay and 3, as, + Cgj
B.2 UPCE LOSS OF GDD
Proposition A1 (Analytical form of UPCE). Given the i-th sample (()) E D, and a HENN f(: O), the Uncertainty Partial Cross Entropy (UPCE) loss for this sample can be formulated as the following analytical form:
UP (1) 0) E (p-a,(ppa(()))00)) log Σ
j&Pk)
v /(0) cbB( ) > 1)+
" Σ
Bft0) - v(al2))
(v(3590)
Σ
(3)) 1(5 E ES,) + 1)
LES,
where B5 = Eles, + cj.
Proof. The formal formulation of UPCE loss is formulated as follows.
K
UP (O) -E ppp-000ppppp (1) ((),)(())) (- log Σ TkPk)
log
Σ 1
Pi
(115 > 1)+
log Pis
(11 I|1 1)
(17)
=1
Term 2
We need to get the log expectations Term and Term 2 above to calculate the UPCE loss. The following is to explain how derive these two terms.
Given the PDF of GDD GDD(p)e a. c) (Eq. 4) we can rewrite it in the form of exponential family:
p(x: >) h(x) exp (TTx(x) - A(y))
(18)
with natural parameters y, sufficient statistic u(x), and oog-partition A(y).
Construct the pdf of GDD as exponential family:
p(x; YY exp (log GDD(plox, c))-
(19)
15

---

Published as a conference paper at ICLR 2024
The logarithm term can be constructed as:
log GDD(p)a, c)
log
Pk
Pi
log Z
k=1
j=1
IES;
log Pk
Σ log
Σ lES,
pi
0j
log Z
(20)
aak
log
Σ
Cj
log
Σ
Pi)
loog Z.
IES,
Note that Gamma function has the transition relation between Beta function:
(ba)T(b) B(a, (a 6),
(21)
which can be generalized to multiple variables (Murphy, 2022) as follows,
Biai, (la1,a2..... a2..... ax) I I(a))
(22)
r K ax)
Therefore, the normalizing constant:
Z II B (larhEs)) j=1
)
(13))7-1)
(23)
Ihes r(as)
11 [(B;)
I j=1
r(Eies, an)
r(E)=1)) B,)
Now define the log-partition A(a, c) follows:
Ihs, I(au)
IT I(3,)
log Z =
log
+ log
r( o1)
r(>
j=1
log
I((a))
log I'(
an) log II [(3,) - log I'
3,)
(24)
j=1
IES,
j=1
IES}
j=1
j=1
log ((ai)
log T(
an)
log [(3,) - log r
83)
j=1 LES,
SES,
j=1
=A(a,c)) c).
Suppose Ps a₁ p and 7 -j. = (1, ..., Cy) v()) (log Eies, Pl, log Eies, Pt. .t.... log LiEs, pi] then
(a] 1, ..., ak 1JT, u(p) [log Pl, 11 log P2.... log pk]
the PDF of GDD would be in the form of exponential family as follows:
GDD(pla, exp
Σ (ax 1) log ps +
Σ cj log Σ Pi) A(a,c) j=1
LES;
(25)
eexp p(a)T u(() + 7(c)T v(p) A(a,c) c)
We can identify that (p(a), 7(c)} are natural parameters, (u(p), (p)) are corresponding sufficient statistics, respectively.
According to the property with respect to the exponential family, we can state that dA(a,c) c)
[[u(p))] = d'A(a,,) c) = dak
dA(ax, c) Ejo(p),] dA(aa, = de;
(26)
dpk
dni
16

---

Published as a conference paper at ICLR 2024
Since 8j Lies, ar + Cj, so day
0& = 1 Sj). In addition, since xb(x)
d dx
log [xx), the log
expectation (p)( in Eq. 26 would be:
E[log(ps)] = 0A(a.c) c)
8on
8 EiEs, log ((r) dax
2 log I(EiE,, ai)
a (lgg(T(3))
a (lgg((())=)) 83))
dan
doa.
ZiEs, OlogI(ar)
Eies, 8cu
05; deck
"
0ß;
(33))
+ >
31)
dax
deck
(E5)
(27)
Σ ((xx) 1k E ES)
Σ
at) 1kk E S,)+ + Σ (3,) 1 E 8,)
-4 Σ
3g)
Σ
1(k S,)
f=1
LES
1=1
-=1
f=1
"1
=(ox))
Σ
1
01) 1(k E {S))
+ Σ
(B,)) 1 E (s) 11 Σ
33)
1=1
1E5,
|=1
5=1
= -(v(an) - v Σ 81) + Σ (B,,) - 1 Σ anl) 1(k E S,).
ies,
Similarly, with the leverage of E [(v(p) in Eq. 26.
pi) aA(a,c) c)
Bloo
de, " 3Σ tes, ((r))))))))))))))))))()))))
LES,
" a log r(Ees,, oi)
" [log(T(3)) (lg(( 33))
dc, (log()
dc,
dc,
Oc,
1=1
0 (Og()(E) 3,))
dc,
(28)
i 33==1 Bg
$
3,)
Oc₁
dc²
08,
7
03;
= ((3,) 0cz
+0
8,)
Dcg
f=1
= (33) 10
83).
Thus, we successfully derive the essential component Term (Eq. 28) and Term (Fq. 27), which can be used calculate UPCE loss as follows,
UPCE( 0 = E Pp-
c0)) log Σ jkpa)
E
log
1 (1]5 > 1) + B
log Pis e (11 1)
= p 3(0) - v v(((3) 111520011 > 1)+
g v(a8)) - (v(0g()) - 10 15
a(9)) 105 E {, 111 =1)
where Bg Eles, an a1+ + Cj.
17

---

Published as a conference paper at ICLR 2024
B.3 KL DIVERGENCE AS REGULARIZATION
Let KL(.) denote the KL-divergence of two distributions. According to the Appendix C.3 in Ulmer et al. (2023), the KL-divergence of two GDD distributions can be written as:
(DD(p(ã,
XL GDD(plå, ()
log
EDD(p)ox,,) c)
(29)
log GDD
IE log GDD(pla,
Since we derived the entropy of GDD distribution Eq. 33, we have
"
Eloog GDD(p/a, c)] log Z(a, c)
Σ (ax
1)
log Σ
c
log Σ
Pt
(30)
LES,
By putting the above term into the Eq. 29, have:
KL cDD(pla, (cddpl.,))((DD((laa.
K
log Z(ā,c) +
Σ
06k 1)E
log
+ Σ
log
Σ Pt
k=
==1
LES,
K
(31)
log Z(a, c)
+
Σ
(an- 1)
log Pk
+
Σ
1 log
Σ LES,
pl
km
j=1
Z(a,c) c)
K
log
+
+I
(āx
ax)E
log
+ >
(2g
c,)EE
log
Σ
Pi
Z(ā C)
k=1
5=1
IES;
Therefore, we derive the following regularization based on GDD(p) 1 ,0"),
KL CDD(pla GDD 07))
K
" Σ j=1
(32)
log 7(1x,00) 07)) log Z( c)
Σ
[ik
1)
log
Pk
log pt
IES,
where E [og and E log Eies, Pt are derived in Eq. 27 and Eq. 28 respectively, ãg = Ux (1 - U2. @ ag the Dirichlet parameter after removal of the non-misleading evidence from the predicted parameters a, specifically, skip the comparison of og with 1 given for [K] C (1 ū5) @ ej as composite evidence parameter with the target class setting to be 0, for j E [x]
B.4 ENTROPY OF GDD
We can derive the entropy of a GDD distribution from its definition, and by using the component [lg pu] and E log Eies, Pt which are derived in Eq. 27 and Eq. 28 respectively, the full analytical
18

---

Published as a conference paper at ICLR 2024
form can be derived:
H[p] = E log GDD(p ax c)]
= log 7
log
K
PE
mk
1
log II
17 Σ
Pi
%/
1=
1=1
IES;
K
log Z -
E
log Pk
E
log
km1
==1
ies;
K
=log? %
(o== 1)
log
Cj
R
log
P1
k=1
1=1
IES;
K
7 (bb(an) -w( Σ B1))
イ + Σ
(33)
log Z -
Σ
(as
- 1)
(b()))
m(E ai) 1(k 5,)
k=1
,=1
_=1
ies;
cyl
log
IES,
7
71
= log Z
Σ (as: 1)v((ax) + - (cx - 1)
Σ 3,) Σ
cak - 1) Σ
(B3)) - - -02 ai)
1(k E ES))
5=1
155,
Σ cj (M(33)
$5 Σ B))
C THEORETICAL ANALYSIS OF LOSS FUNCTION
C.I CONVEXITY OF CE & PCE
To prove the convexity of CE and PCE loss with respect to class probabilities, only need to show that the second-order derivative of both losses is non-negative. For CE loss CE(p. =
-E
k= Wh log Pk. since pk 2 0 and in 2 0 for any k E {1,2, ...., K}:
d
d
CE
CE
[- Ük log psl
dpk
dpk
Pk:
(34)
d
d
ūk
ÿk
CE"K CE
2
0.
dpk
dpk.
Pk
Pk
By Eq. 34, we can know that the Hessian matrix is diagonal and positive semi-definite, Hence, the CE loss is convex.
For PCE loss PCE(p,))
= =oogg(> (hPk), we have:
d
PC = PCE
dpk
Σx DiP;
(35)
K
PCE
d
PCE
-
-ixe
)
JjP;)
-2
Uk
2²2 0,
dpk
Σ/=1 Ú,P;
where OR is the k-th element in the binary vector y representing classes in R Analogously, PCE loss is convex and thus follows Jensen's 's inequality.
Proposition A2 (Lower Bound of UPCE). Given any instance (x.5), and HENN f(: 0). the Uncertainty Partial Cross Entropy (UPCE) for this sample UPCE(x. ÿ: 0) has the following lower bound:
UPCE(X, 5;
(p).
(36)
19

---

Published as a conference paper at ICLR 2024
Proof. Since (p y; 0) is convex (proved earlier), it is straightforward to get the following inequality through Jensen's 's inequality:
UPC = B ppppllll PC 0:0)
(37)
2 PCE PCE[[ppaoppllaellD... (p),
C.2 PROOF OF PROPOSITION
Proposition (Properties of the empirical UPCE risk function). Assume that the universal ap- proximation property (UAP) holds for a HENN, ie., the network can learn arbitrary mapping function from the input feature vector x to the evidence vector e. Then, the empirical UPCE risk function R(f) = N/ Em= UPCE(x y0); approaches the infimum if the solution 0 satisfies the following properties, with [(x: )r): (1) v(x. y) E 'D where denotes a singleton class label, & E [K]. the predicted evidence values Ck + +00 and es, + +00, VS; E SS, such that l S and (2) v(x, E DD where denotes composite label Si the predicted evidence values es, - +00 and ek + +00, Vk E S,.
Proof. Given the HENN with empirical risk as R(J) N EE (UPCE(X(), 5,00 0)]. we can show that one of the optimal risk minimizers can always predict non-confident evidence while still maintain the property regarding the loss minimizer for arbitrary examples in the training set.
First, we show the properties hold for a UPCE loss minimizer. Since opinions in Subjective Logic rely on estimating evidence form subjective opinions and reflect structural knowledge, it is necessary to have accurate and consistent evidence output that supports subset of the hypothesis space. Based on this definition, the composite evidence should not be too large given only singleton training examples, and vice versa. Nonetheless, Proposition states that for a given data point (x, y), different minimizers trained with the same UPCE objective can end up with different evidence predictions.
As UPCE loss is convex in terms of evidence, we consider analyzing the impact of output evidence on UPCE loss. We start with partial derivatives because the UPCE loss is multivariate differentiable. For clarity, we've organized our proof into two parts: one dealing with single ground- scenario, and the other with a composite ground-truth.
Case 1: Under singleton ground-truth assumption.
Ideally, under the singleton ground-truth assumption, we anticipate that, as the singleton ground- truth evidence increases, the UPCE loss should decrease. When composite evidence increases, the UPCE loss should increase, i.c., do UPCE < 0, Vv E [K] and Be, rooted in the fact that the UPCE loss is multivariate differentiable. we explicitly write the partial derivative for composite evidence e, (v [n]) with singleton ground-truie -tuth,, will have
8 UPCE 2 0, Vv E [n]. This expectation is
0 dey
UPCR
11
B1)
(ois)
+
x an)
1l( Σ 04 c)) 1() 5.)
dey
1=1
j=1
LES,
IES,
0
K
n Σ
" Σ
1 E 8,) 2 rl an c,)
c))
(38)
ac
dec
km
1=1
j-1
IES,
K
9
1 E 8
Σ
a (v)=))
1=1
1=1
IES;
where (1) wh (x:) Aw(ar)
known as the trigamma function, which positive and monotonically decreasing on (0, +00) (Qi & Berg, 2013). Next, we will go through different composite set labels to
de
simplify the partial derivative.
If the partial derivative taken is not for the composite class label including the singleton ground-truth, then 2 UPCE ti (EK on + Σソ= (z). Since ar 1, Cg O. and ()) is positive (0, +00), 8ey
20

---

Published as a conference paper at ICLR 2024
0 UPCE 0. However, if the partial derivative taken is exactly for the composite set it follows that Sc,
label including the singleton ground-truth, then e De,
UPCE -/h (
-))-))>>>- (v e
0, and t/ (-) is monotonically decreasing on T +)) j) It follows that d UPCCE < O. This outcome, which is not desirable, reveals that the optimal HENN has the potential to increase the composite evidence output, even in cases where the singleton ground- truth does not apply. Remember that in Eq.2. Subjective Logic determines the subjective opinion based on evidence. Therefore, this approach to prediction can negatively impact the quantification of uncertainty and further affect the classification accuracy that relies on evidence-related projected class probabilities.
Sy) Eres, on ( j). Since Og 1, Cj
(0, +00), therefore 1/1 (E
K 04 1) Σー (v 1)) (EiEs, ar a
Under the same singleton ground-truth assumption, the partial derivative for singleton evidence au E [K]) is:
0 UPCE
① da,
[v0 Σ B,) - (ass) +
Σ
71
25
Σ au) - v 5e IES,
ai + c)) 1)) S,)
8aw
(ES,
(39)
th
B3) ti (ans) 1S)
Σ
(y S,) E (
Σ ai) (3;)). LES
Following the same - class, then (UPCE 80,, (- Σラ= [y E S2) E S2)( (Eies, ai) 1/1 (8,)). Since Bg 2 Eies, an the dereasing monotonicity of trigamma function gives t (Lies, an) (3,) > 0.
if the partial derivative taken is not for the singleton ground-truth
> 0. If the partial derivative for the singleton ground-truth, [tb (2)=11))))) Σ}= 1 S,) E ( Σー= 1(y { E S,) (Eies, Lies ai) - t (a = 15}]. Noting that B, <= E² B, and aTs <EiES, Og,
So the partial derivative SUPCE
Day
rewrite the equation as @UPCE das
know that dUPCE Sa,
< 0 by decreasing monotonicity of trigamma function.
Hence, for V(x, f) D. with fixed finite values of e and a except for either ground-trutt singleton evidence or for both the singleton ground-truth evidence and the composite evidence including the singleton ground-truth, the limits
lim UPCE lim
((3)) - (aus) Σ
(bb 1E5,
Σ
an) ((B)) E 5.)
+ +
ogs 4:00 os +-000
7
(40)
lim +0o, 4-00
UPCR
lim 4.00 1se8;
(b30) Maiss + Σ
(N) Σ
oi) +(3)))
8,))
+
1=1
IE8,
S,
hold.
Recall that og = 6 + 1, and trigamma function 1) (-) is also strictly convex. Therefore, rewrite the concentration parameters as evidence, for v(x, D, where ý is a singleton class label [K]. when eg + +oo and es, + +00, YS, S, such that k S,, we will have (P ((), 0) approaches the infimum 0. is worth noting that the infimum can also be approached when solely maximizing the singleton ground-truth evidence. Hence, with different evidence predictions causing the same loss for the same learning objective, hyper-opinions derived from the evidence will also become inconsistent.
Case 2: Under composite ground-truth assumption.
we assume the ground-truth is composite class label, we expect that as the composite ground-truth evidence increases, the UPCE loss decreases, in contrast, if any singleton evidence increases, the UPCE loss should increase. Mathmatically, our goal is Je, 0, Vv E [K]. For composite ground-truth, the partial derivative with respect to cz, v E [n] is known
2 R-UPCE < O, Vv E [n]. and aa,
a_ UPCE >
21

---

Published as a conference paper at ICLR 2024
UPCB = 0
10 Σ B,) - (3)c))
Joy
doy
1=1
(41)
= 1
3,) (f(c)
If the partial derivative taken for a composite class label that is not the ground-truth, de,
0 UPCE
2 O 33). Since 83 0, and w (-) is positive on (0, +oo), it follows that 0 UPCE > 0. This means the HENN will compress non-related composite evidence. Nonetheless, the partial derivative 8_
ac.
for the composite ground-truth Dc, UPCE = =b1 3,) (Bic). Since Σ3 33 > 9ic, and (-) is monotonically decreasing on (0, +oo), it follows that 4 O B3) < Φ (31c) (vv = IC),
0 UPCE indicating HENN will only enlarge the evidence of ground -truthhhruhhhhh among all composite
Be,
set classes during training.
Similarly, the partial derivative for singleton evidence Og (v E [K]) under composite ground-truth is:
0 dov
UPCE
R
Bg) (Bc))
dow
j=1
(42)
= thi
83) - 1v E Sic) (Stc);
3=1
If the partial derivative taken is not for the singleton class included in the composite ground-trute then da, UPCE = des,
a ( {C}= 8,)] > 0. In contrast, if the partial derivative is with respect to the
singleton class included in composite ground-truth, then a Day UPCE Sa. [>] (3) - s/a (Bic)
Since Σ}=1 3, Sic, then we have t (1) By) < th (3ic). 0 UPCE < O, which causes the confusion. In other words, the UPCE loss guides HENN to enlarge the evidence of composite ground-turth and the singleton classes included in the ground-truth.
Now given finite fixed values of a and except for either the ere ac with several other og included in composite ground-trume truth, we have the limits:
7 Σ
lim UPCE 4000
liu
(B)))
(3,)) = =i) 0,
t.00
i=1
(43)
(3,) = 1) 0.
lim
UPCE
lim
v(B6)
kESS
If we convert the parameters back to evidence space, then the limits show that for v(x, y) E D, where y is a composite class label S;, the (x(E(x()), (()) O) approaches the infimum 0 as the predicted evidence values es, + +00 and ex +oc, vk S.. The infimum value can also be approached by only maximizing the composite evidence for the ground truth. Again, with different evidence, predictions correspond the same empirical loss for the same composite learning objective, causing unreliable issues in subjective opinion modeling.
After proving cases of inconsistent evidence predictions regarding the optimal loss minimizer, we prove that the risk minimizer can be approximated by a UPCE loss minimizer for each training data point. This step is crucial for connecting the empirical risk minimizer with the loss minimizer and for highlighting the inconsistency in evidence predictions made by the empirical risk minimizer HENN. Under the assumption of universal approximation property (UAP) (Cybenko, 1989; Leshno et al., 1993a), suppose the HENN has the capability to produce sufficient non-linearity to estimate any functions in the evidential space, we can have at least one optimal HENN f(:;0) (or f for short) such that
N
N
R('") inf R(J) inf 0EG
UPCE(X (6)
s
(0,00)
inf UPCE(X .5 .000 (44) dee
N
N
22

---

Published as a conference paper at ICLR 2024
This equation connects the empirical risk minimizer and the loss minimizer each train- ing data, To prove Eq,(44) we apply the assumed UAP. Note that UPCF(x(0) g 0) UPCE(f(x() (x) (()). We abbreviate them by UPCE() for simplicity. Since UPCE y: 0) Z
inf UPCE x, y; 0), have Ee [UPCE(0] 2 v E [inl and trivial conclusion inf E [UPCE(]] 2 1 E inf UPCE
Now recall the universal approximation property demonstrates the existence of a function that can approximate any function within the same function space. Applying assumed UAP to our setting, states for any (x, V) E DD and arbitrary function g(x. ý: 0) inf UPC=(f(x; 0): V). there exists an optimal HENN can approximate g()) by mapping input features to evidence f(x; 0*). s.t.
sup (xl (') x.
< Ve 0.
(45)
Because of the relation Bu Bic and as, > ais, the form of UPCE loss based on digamma functions in (12) determines its positive value, according to the increasing monotonicity of digamma function on (0, +00).
Given the limits shown in (40) and Eq.(43), we know that the infimum of UPCE is 0, (4.) can be rewritten as:
sup (PC 0"), <, Ve > 0.
(46)
Based on the inequality
inf inf JPCE
<
inf [UPCE
N Σ e=
(47)
with both lower bound and upper bound as 0, according to the squeeze theorem, the exchangeability
of inf operators N N'' [in UPCE(X, y; 0)] inf Ein UPCEXX, 0) holds for each
training data point, and the Eq.(44) is proved.
Knowing the existence of empirical minimizer for all observations also works the loss minimizer on each training data point, the HENN should always predict evidence f(x; 0*) = (ā, ē), v(x, e
D.
UPCE( 0*).)) inf UPCE 0, v(x, E
(48)
We conclude that the properties derived from the analysis of the UPCE loss UPCE(X, ;: for arbitrary (x, ~ also holds for the HENN with empirical risk R(f) under the assumption of UAP.
C.3 PROOF OF PROPOSITION 2
Proposition 2 (Effectiveness of the regularization term Reg(x, 0))). Following the UAP assump-
tion, the regularized empirical UPCE risk defined in Eq. (15) approaches the infimum
the
solution 0* satisfies the following properties: 1) v(x, E D. where ý is singleton class la- bel E [], the predicted evidence values eg + too and e, + 0, Vt S UUTT [K] \ k; and 2) v(x. y) E D. where y denotes composite set label S,, the predicted evidence values es, + +00
and e, 0, VL E [K( St.
Proof. To address the inconsistent prediction issue of our evidence output, the KL-divergence between regularizer. All 0 evidence for each element in
the predicted GDD and flat GDD is introduced the hyper-domain composes a flat GDD as GDD(pI K 0%). ). In following section, for simplicity, we ab- breviate KL (DD(p) [ 1 0²)] by ( c()), (PCE(x (,0) 0) by (PC
and let [K] denote {1...., K}. [n] denote {1, ..., n) Now the optimal regularized generalization risk is
23

---

Published as a conference paper at ICLR 2024
R(S*) - inf
UPCE(X
,0000) ,000xx(),,(0000
kI(a")
inf
[UPCE + (AAKI(al), c(.)]
(49)
N
kL (0) c
inf
UPCE (8) +
N Σ
N
A=1
According to the partial derivatives and the convexity of UPCE loss proved in section C.2, we already have two special limits for singleton ground truth shown in Eq.(40). Correspondingly, to make UPCE loss approach its infimum value with composite ground-truth, there are also two special limits mentioned in (43) Multiple choices to minimize the UPCE loss imply different combinations of a and с can become the loss minimizer and output by HENN.
Consider the KL-divergence between predicted GDD and flat GDD
(L(a("), cc = /
a ( c(O)
GDD(pla
c(() log
dp.
(50)
GDD 1 07)
A
c e(0)
It is straightforward to have its minimizer when the value of log aDD[p] .0n is 0. This indicates that
we aim to minimize the difference between (+ and 1 K as well as between c( and 0", Predicting flat GDD except for the evidence of the ground-truth make the KL-divergence reach the minimum value of O. Therefore,
arg min ( (()
(51)
= =((a.c) = = L, IS [K], = u
0, IC,j E [v]].
Note that the feasible region of the output evidence for minimizers of UPCE loss and the KL- divergence overlaps, which illustrates that both infimums can be approached simultaneously. Specif-
ically, for singleton ground-truth, the intersection of feasible evidence between KL-divergence minimizer and UPCE loss minimizer is ((a.c) ag = 1, ais + +00 & * IS, k [K]. c 0²}. In contrast, the intersection for composite ground-trume -tutth can be written {(a.c) c) 9c > jo,,j * J [n]].
1 Cj
The overlap of feasible evidence towards the lower bound of the UPCE loss, along with its regularizer, also enables the application of the Uniform Approximation Property (UAP) the regularizer, with inf + Σn Reg] t > [inf Reg). Based on the assumed UAP, there exists configuration 0' such that HENN is empirical UPCE loss minimizer for each training data point. Within the feasible evidence region for minimizing the UPCE loss with 0', the learning objective is improved when considering the overlap in evidence outputs. This suggests the presence of an optimal configuration 0" within the feasible range of 0' that can attain the minimal KL-divergence at every training data point without hurting the optimality for empirical UPCE risk. By focusing learning 0*, we finally can get the optimal regularized HENN given UAP assumption holds.
Therefore, move the infimum operator into the empirical risk,
R(**) inf
U +X
(L
()) c()))
(52)
N
inf (PC A inf (L c(0)]
As proved in section C.2, based on the assumption of UAP, there exists regularized loss minimizer for each data point, which also works as the empirical regularized minimizer for D {(x(1) ()))
24

---

Published as a conference paper at ICLR 2024
Table 5: Dataset Statistic.
Dataset
CIFAR100
tinylmageNet
Living17
Nonliving26
Image Resolution
32x32
64x64
224x224
224x224
# superclasses
20
29
17
26
# subclasses
100
200
68
104
Training set size
45k
90k
79.56k
119.5k
Validation set size
5k
10k
8.84k
13.3k
Test set size
10k
10k
3.4k
5.2k
SELECTED composite classes
120,15,10)
120,15,10)
(15,10)
(20,15,10)
Algorithm 1 Pseudo-code of HENN (one epoch)
Require:
Training dataset {(x(() H HENN model f(, 0); tradeoff cocfficient X; learning rate Y; the number of sampling data N; Batch size: [B];
1: Initialize model parameters 0.
2: for iter = 1, do
3: Sample mini-batch from D
4: Generate the evidence vector e
i=1 (e E R (Ix)) e ((x(9),0)
5: for cach (x(1), (1)) E B do
5: //based Grouped Dirichlet Distribution (GDD)
6: Get the UPCE loss for this example (PCK(() (0) via Eq. 12
7:
Get the entropy regularization for this example Reg (0) via Eq. 14
8:
()) Get the loss for this example: g (0) = UPCE ( (0) + AReg (0)
9:
end for
10:
Σ i=1 ( (0). Get the loss L for all examples in this batch B: (()) [3]
11: Update model parameters 0 via gradient descent 0' 0 VV)
end for
By replacing the empirical risk minimizer with the regularized loss minimizer. We focus on loss minimizer that produces:
UPCE + X Reg + inf JPCE + X Reg] = inf UPCE + A inf Reg
(53)
Clearly, optimal regularized HENN f(x: 0) = (ā, é) will take the intersection of the feasible space for approaching minimal UPCE loss and regularizer, that is, (ā, ē) {(a, c) an = 1, + IS, e [K]. ans + +00 C = 0²) u {(a( c) : a = 1 C.- = O,j + IC, E [v]. qc - +00]- Convert parameter space back to evidence space, then we can say for {(x,)) class label & [K], the predicted evidence has the form of e₁ +00, + 0, Vt S U [K] k.
D where ý is singleton
For V(x, f) E D, where ý denotes a composite class label S, the predicted evidence should be es; +000 and e vt E [K] St
D RELATIONS WITH ALEATORIC AND EPISTEMIC UNCERTAINTIES
Epistemic and aleatoric uncertainties are two broad categories used to classify existing predictive uncertainty measures. Epistemic uncertainty is due a lack of evidence knowledge in the training data it is a known unknown. It is reducible by collecting more data. In comparison, aleatoric uncertainty due to the inherent complexity of the data (e.g., wrong labels, incomplete is unknown wnknown. It is irreducible by
or partial labels, and other data randomness) collecting more data (e.g., the stochasticity of a dice roll cannot reduced by observing more rolls), assuming the same measurement precision in the collected data (Gal, 2016). The aforementioned evidential uncertainties, including vacuity, vagueness, and dissonance, and other uncertainty measures, such model uncertainty (mutual information between model parameters and the predicted class probabilities), data uncertainty (entropy of the predicted class probabilities), and confidence (the largest predicted class probability) can be classified to epistemic and aleatorie uncertainties based
25

---

Published as a conference paper at ICLR 2024
whether they be reduced by collecting more data. In particular, the vacuity and model uncertainty fall into the category of epistemic uncertainty, and the dissonance and vagueness belong to the category of aleatoric uncertainty. The dissonance is irreducible by collecting more conflicting evidence. The vagueness is irreducible when we use the same measurement precision (sensor and annotator) collect extra training data due the invariant underlying distribution for getting composite labels. A recent work (Shi et al., 2020) demonstrates that the entropy of the predicted class probabilities can be decomposed into two distinct sources of uncertainty: vacuity and dissonance. As confidence is correlated with this entropy. both data uncertainty and confidence may involve mixture of epistemic and aleatoric uncertainties.
D.1 EXAMPLE ABOUT EVIDENCE
In medical diagnostics, the presence of 24 pieces of composite evidence could suggest that there are approximately 24 similar cases resulting in diseases 2, 3 based on the current observation. This implies that the cases identified as having either disease 3, but without specific information to distinguish between them. Conversely, 3 instances of class evidence indicate that similar cases have been identified as disease 1. In such scenarios, doetors might not have clear preference between diseases 2 and 3, while maintaining conflicting opinion between disease 1 and (2,3) for this observation.
D.2 DISSONANCE HYPER-OPINION
Given hyper-opinion with non-zero belief masses, the dissonance measure can be estimated as:
bs Es d(SAs Bal (bgr, bs)
diss(w)
Σ
(54)
Est d(SAS'')bs
(*)
where Bal(S", -) lbs bsl/(bs/ + bs), and d(SA8') is the size of the symmetric difference between S and S" (Jøsang et al., 2018).
E REPRODUCIBILITY
E. DATASET
Table 5 shows detailed statistics for four datasets we used. In particular, tinylmageNet has 29 superclasses because we keep all superclasses which have 2-3 subclasses only.
We use CIFAR100 (Krizhevsky & Hinton, 2009), tinylmageNet (Fei-Fei et al., 2015), Living17 (San- turkar et al., 2021), and Nonliving26 (Santurkar et al., 2021) our experiments. CIFAR 100 has 100 classes containing 600 images each (500 for training and 100 for testing, and the image size is 32)32). The 100 classes in this dataset are divided into 20 disjoint superclasses, each with 5 unique subclasses. Note that compose composite class labels within the same superclass. Dataset linylmageNet has 200 classes containing 550 images each (500 for training and 50 for testing, and the image size is 64 x 64). We generate the hierarchy information of tinylmageNet and generate superclasses according to the existing ImageNet class hierarchy WordNet (Miller, 1995). In addition, usually can be challenging to distinguish between different classes due their similar visual features. While WordNet is a hierarchy based on semantic relationships between words, rather than visual similarities. Therefore, Living17 and Nonliving26 are considered because their class hierarchy is generated based on visual and semantic similarities. Both of them are subsets of ImageNet dataset (Deng et al., 2009) with an image size x24. Refer to Table 5 and 6 in their paper for more information.
We split the original training set into training and validation set according to the ratio 9:1. Therefore, the number of images per class will 450/50/50 for training/validationnnes set for CIFAR100, similarly for other datasets.
E.2 DATASET PREPROCESSING
For each dataset, the first step is to select vague images. To achieve that, first, we select M superclasses randomly from all superclass candidates SELECTED composite classes in experiments. For
26

---

Published as a conference paper at ICLR 2024
each SELECTED composite class, 2 or more subelasses belonging to this superclass will be selected randomly as components of the composite class label. Given the designed composite class labels, we can further select a fraction of images under each of the singleton classes included in the domain of composite classes (V). The selected examples are therefore expected to be converted to composite examples by applying Gaussian-blurring and label replacement to introduce vagueness. When selecting images to blur, for each selected singleton class, we balanced the number of singleton images remaining and the number of composite examples converted. Please check our code for implementation"
The selected vague examples will be blurred by Gaussian Blurring operation. To apply the Gaussian blur operation, there are two parameters to set: kernel size and variance s sigm. We use three different kernel s zes (3 x 3, 5 x 5, 7 x 7), and s gma determined by the default relation between them in PyTorch: s sig = 0.3 * ((kernel __si ze 1) 0.5 1) + 0.8
We used 2 methods for data augmentation following a typical computer vision setting. First, each image is applied to a random horizontal flip with the flipping probability of 0.5. After that, a random corp is introduced for each image with a size of 32 x 32 and padding of 4. Then, resize images to x24 because the pretrained model is trained by ImageNet (Deng et al., 2009) which is 224x224, we need to match the input size for model predictions. We apply regular data augmentation approaches and normalization to the data. Data augmentation approaches are only applied to the training set. For validation and test sets, we only use resize and normalization.
E.3 IMPLEMENTATION
Baselines. DNN and ENN cannot predict set directly. In practice, it is necessary to set a threshold to make set prediction for DNN and ENN. The prediction set should consist of all classes with softmax probabilities larger than or equal to the pre-defined threshold.
In addition, DNN and ENN are only able to deal with singleton class labeled examples and cannot deal with composite class label during training. Note that there are vague images with composite class labels during training. To make baselines can handle these examples, and to avoid removing training examples, we duplicate composite examples and provide them singleton class labels which are from the subclasses of composite class labels. This ensures that all classes remain exclusive. For example, assuming there is an image with the composite class label A,B during training, we duplicate x and take image r with the singleton class label and the same image with the singleton class label B as input for model training.
HENN: Pseudo-code of HENN is shown in Algorithm 1.
E.4 HYPERPARAMETERS TUNING
We list all related methods and their corresponding hyperparameter settings below. For our method and all other baselines, we adopt Adam (Kingma & Ba, 2014) as optimizer with parameters Bn 0.9. 32 = 0.999, weight decay
= le 8 provided in (Kingma & Ba, 2014). The number of epochs for all experiments is set to 100. Other hyperparameters used in this paper mainly are learning rate and weight of entropy regularizer. Grid search is leveraged to determine the best hyperparameters based on held-out validation set for each specific experiment. Specifically, (1) DNN. the learning rate is chosen from { fle-5, 1e-4, 1e-3); the cutoff is chosen from (0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5). (2) ENN. the learning rate is chosen from (1e-5, 1c-4, 1c-3); the weight of entropy regularizer À is chosen from (1, le-1, le-2, 1e-3, le-4, le-5). the cutoff is chosen from [i for i in range (0, 0. 02, 0.011) ) J. (3) E-CNN. the learning rate is chosen from (le-5, le-4, 1e-3): the optimizer is Nadam. (4) RAPS. Kreg is chosen from {1, 2, 5, 10, 50]; À is chosen from {0, Ic-4, 1e-3, 0.01, 0.02, 0.05, 0.2, 0.5, 0.7, 1): a is chosen from (0.1, 0.2, 0.3, 0.4). (5) PiCO. learning rate is chosen from (1e-5, le-4, 1e-3). (6) HENN. the learning rate is chosen from (le-5, 1e-4, 1e-3) and the weight of regularizer À chosen from (1, 1e-1, 1e-2, 1e-3, 1e-4, le-5).
'Orr code: https: pettps:/ggihhinn http://gtttttcccoo http:///o///oooooo//g///e/////// 01 http//eervieeettiiii. Phtpps: / http://yttttcccccttt/ functions www.aal.aassiiinnllggaa html
27

---

Published as a conference paper at ICLR 2024
A fixed number of epochs is given, and the highest validation accuracy is used to determine the best epoch.
For HENN, validation accuracy means the classification accuracy including the additional composite class labels on hyperdomain, such as 215-class classification tiny ImageNet dataset. Even though report multiple metries, such OverJS, CompIS, and Acc, use validation accuracy to select the model. We use the Best validation accuracy to evaluate and determine which combination of hyperparameters to use.
For DNN, there are two sets of hyperparameters. The first set includes the hyperparameter of general DNN: learning rate (we only tune this hyperparmeter for now). The second set includes the hyperparameter used to generate set prediction: the cutoff on class probability. For example, if there only three classes and the prediction of the DNN for one test image is: {0.6, 0.3, 0.1}. the cutoff is 0.3, then the set is: (class 1, class 2).
For the first set, use the accuracy on the validation to tune. Note that is always 100-class classification after using duplicates for vague examples. Each duplicated image has its own class label. For example, for one training/validation image that has set label: class I, class 3. We will create two duplicates of this image labeled class and class 3, respectively. For the second set, use the overJS on the validation set to tune. Here, we will replace duplicates with the images with vague labels in the validation set, in order to calculate the overJS.
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 ADDITIONAL RESULTS
Table 6, 11, 12 show composite and singleton prediction results for different Gaussian kernel size 3x3, 5x5, 7x7 for CIFAR100 and tinylmageNet dataset, and Table 8, 9, 10 show composite and singleton prediction results for living 17 and nonliving26 dataset, which represents consistent observation as in main paper.
Table 6: Results (%) based on Gaussian kernel size: 3x3 on CIFARI00 and Imageee. (The average and 95% confidence interval of three runs are provided.)
CIFARI00 Camp1S
tiny ImageNet CompJS
Methods
OverJS
DNN (Tan & Le, 2019)
86.8-10.33
68.6411.42
84.311-051
83.4-1..38
66.9440993
79.8-4-.32
ENN (Sensay et al. 2018)
84.4:t-228
42.34:129
84.84:0222
75.9
63.5::126
80.7::0.77
10
E.CNN (Tong et al., 2021)
38.521074
34.242633
73.24:992
33.4:2033
31.1
68. 68.2::0.22
RAPS (Angelopoulos et 2021)
73.1+:037
79.8+0.32
PICO (Wang et al., 2022b)
59.6:2-.38
28.144441
63.64:0448
57.2:t0.99
35.64:355
64.34:063
(ours)
86.5:0047
90.411363
86.5::0.53
04.4:20.44
93.4::2.57
82.510072
DNN (Tan & Le, 2019)
86.64:035
71.64:145
82.24:039
84 34.3420.43
67. 1.33
79.5
ENN (Sensyy
2018)
84.2-107
47.8:1::5
83.8:10377
83.5:10.00
60.71:114
81 21.2:10.06
15
E-CNN (Tong et al., 2021)
33.2:100.4
31.3-1:3433
68.61:099
32.5:10.83
33.3:13.52
68.41:000
(Angelopoulos et 2021)
PiCO (Wang el al., 20226)
58.4+90.4
25.5++4332
61.3+:0.50
56.83
35.34:3.53
64.65:0.64
HENN (ours)
86.8:10.88
90.1444.36
85.8-10.119
84.6
90.6-:-261
81.6
DNN (Tan & Le, 2019)
86.8:1035
75.44::.65
80.3:1:35
84.0:10.33
$7 1..9
81.5:10366
(Sensoy et 2018)
58.94-0.36
E CNN (Tong et al,, 2021)
28.6+10.78
23.74:325
73.6+:0.77
23.3-20.66
22.4::251
67.8+40997
(Angelopouloss et al., 2021)
80.5::035
56.741.54
30.14:035
16.42
41.1-1147
81.54:336
PICO (Wang et al., 20226)
57.5:1071
29.1114-45
61.910056
57.5:10.11
39.6:1:366
65.31:071
HENN (ours)
86.7:10.77
90.2.11.36
86.3:1:034
84.9
90.7
81.7
F.2 MODEL-AGNOSTIC PROPERTY
Table shows model agnostic performance (%) on M=10, and kernel size: 5x5 on CIFARION including confidence interval for three different runs. It demonstrates different methods" performance based on ResNet50 and VGG16 on CIFARIO6 100. HENN outperforms other approaches, for example, the Acc of HENN surpasses that of DNN by 2% for CIFAR100. The consistent observation is demonstrated based on different backbones, which validates the model agnostic property of our proposed approach.
28

---

Published as conference paper at ICLR 2024
Table 7: Model agnoistic performance (%) on M=10, and kernel size: 5x5 on CIFAR100. (The average and 95% confidence interval of three runs provided.)
ResNet50 et 2015)
VGG16 (Simonyan OveralUS
201.5)
Methods
OvemilJS
ComplS
Acc
CompIS
Acc
DNN
82.0+:0.66
56.741129
80.6220.11
77.61:032
13.3
75.240.38
ENN (Sensoy et al., 2018)
80. 1:028
46.7=11.22
80.9=0.25
74.620.33
42.7=141
76.2:10:3
RAPS (Angelopoulos al., 2021)
40.1
80.6:10211
66.41.0.44
35.5
75.2:10.38
HENN (ours)
82.91:034
85.722.41
81.120.32
78.4
78.5=2.3
77.7#0047
Table 8: Results (%) of BREEDS-living based on two Gaussian kernel sizes. (The average and 95%
confidence interval three runs are provided.)
Gaussian kernel size: ComplS
3
Gaussian kemel size: CompUS
5x55
Methods
OverJS
DNN (Tin & Le, 2019)
88. 28
81.0+1.744
83.14b:.29
88 38.33
80.4::078
83.24:043
(Sensoy 2018)
72.34-0.1
88.0:00.16
84.61:0.01
10
ENC (Tong et al., 2021)
30.5++0.77
36.8+:1.4
65.7+:0.66
30.4# 1.34
35.8+-088
65.74:042
(Angelopoulos al., 2021)
86.420227
61.141.56
83.14:020
05
60.74:.89
83.24:043
HENN (ours)
88.811039
96.5:10022
85.6111.24
88.7:10:35
96.910881
85.910.33
DNN (Tan & Le, 2019)
88.11:0.9
84.8:1.66
80.2:1:034
88.42:0.33
84.5:
80.62:048
ENN (Sensoy et al., 2018)
88.0 18.0::0.03
78.3::0055
82.41:0.36
87 07.8:2
75 75:4:::.38
84.7:1:600
B-CNN (Tong al.,
31.6:1:.445
37.3441.58
35.111991
64.8
RAPS (Angelopoulos et al., 2021)
85.5 $5.5++0.35
66.5++0072
80.2++0.34
85 83.9-t0.42
67.6-t00.2
80.6-t:0448
HENN (ours)
88.8:10.17
96.6:1:0.65
85.7:11.27
88.9
97.5
1.74
Table 9: Results (%) of BREEDS-nonlivinge nonlivingg26 based on two different Gaussian kernel sizes, (The average and 95% confidence interval of three runs provided.)
kemel ComplS
Gaussian
M
Methods
OverIS
Acc
OverJS
ComplS
Acc
DNN (Tan & Le, 2019)
85 85.6:10032
62.01005
82.910035
86.0:10.06
64.0:1:60
83.0110.12
ENN (Sensay et al., 2018)
85.0++0.99
52.942274
84.5+:0.3
85.0 33.04:0.48
$2.8+:3799
84.24:0.6
(Tong al., 1)
28
60.61::97
37.1
60.81:076
RAPS (Angelopoulos et al., 2021)
82.7:10366
46.34:101
82.910033
83.4:10-22
49.5
83.01:0.12
HENN (ours)
86.9.10.13
96.8:10.7
85.4110335
87,0
96.2
85.3
DNN (Tan & Le, 2019)
85.6:10.39
68 30
81.51:016
85 5:10.055
67 07.3113341
81 1:10.55
(Sensoy
2018)
85,4-10.6
62.641:1.99
82.94:021
85.3
83.2
15
E ENNN (Tong et al., 2021)
29.8++1.22
35.14:4411
60.1 +:037
28 7379
35.1 35.11+44.7
60.3-4:084
RAPS (Angelopoulos al., 2021)
56.11:228
81.540
83.7:04
55.9:10.9
$1.450055
HENN (ours)
86.94-0.33
96.2411.14
84,14-030
86,9
95.6++1099
84.8::0.00
DNN & Le,
86.710334
74.5
80.3:10.15
86.511042
76.2.1041
79.8:10.23
ENN (Sensoy et al., 2018)
85.9-+0.43
68.34-2113
81.7++0.35
86.0++0.99
67.9++2.44
82.2-40073
(Tong al., 2021)
6::657755
36.8::3.66
60.940065
RAPS (Angelopoulos et ol., 2021)
82.41041
$7.74:022
80.3+:0.15
1::23
57.8::0445
79.82:0:3
HENN (ours)
87.4:1022
94.5:10:16
85.5:10.41
87.5
94.5
85.3
Table 10: Results (%) of Gaussian kernel size: 7x7 on Living17 and Nonliving26. (The average and 95% confidence interval of three provided.)
Living17 ComplS
Nonliving26 ComplS
M
Methods
OverIS
Acc
OverJS
Acc
Le,
88-4-4024
85.8-40.34
82.94-0.19
ENN (Sensay et al,, 2018)
87.9++029
71 71.04999
84.4-t:.23
85 0.
14.6
84.14:033
(Tong al., 2021)
30.4:i.98
36.74:165
65.5
28.24:774
35.5-1147
59.41:2.1
RAPS (Angelopoulos et al., 2021)
85.9-10.32
60.8:1:.2
83.31:049
03.5:11.37
13.6:1:173
82.9::0.9
HENN (ours)
88.74:0.6
96.0-1032
85.34-022
86.8
95.91:122
85.01:0.9
DNN (Tin & Le, 2019)
88.44-0.35
83.24:231
80.2++0.79
83.8 05.88
70.6++120
$1.24-06
(Sensny et 2018)
82.74:1000
62.1-1076
83.04..57
15
E-CNN (Tong al., 2021)
30.5:10:7
36.611:8
65 65.6112.99
1::099
35.6:1:197
6:2.6:2.6
RAPS (Angelopoulos et al., 2021)
85.7-10.38
66.9:11..3
80.2-1:0.79
83.7:10.46
56.0:10..4
81.2110.3
HENN (ours)
88.74-029
97.14-033
84,4-1-.71
86.9+0221
94,8-11442
85.2--0555
29

---

Published as a conference paper at ICLR 2024
Table 11: Results (%) of Gaussian kernel size: 5x5 on CIFARI00 and tinyImageNet (based on one run).
CIFAR100 ComplS
Liny LinyImageNee CompIS
M
Methods
OverdS
Acc
OverIS
Acc
DNN (Tan Le. 2019)
86.5
65,3
83.8
83,9
46.0
83.2
ENN (Sensny et al., 2018)
83.1
58.8
84.5
54.8
43.1
56.1
E-CNN (Tong 2021)
28.5
22.4
74.2
23.4
68.2
RAPS (Angclopoulos ot 2021)
80.5
49.8
83.8
72.5
43.7
83.2
(ours)
DNN (Tan Le, 2019)
86.2
69.9
82.4
83.2
50.4
82.3
ENN (Sensoy al., 2018)
82.8
58.4
84.5
52.8
55.7
15
E-CNN (Tong al., 2021)
28.7
23.4
70.2
23.3
21.2
68.5
(Angelopoulos al. 2021)
82.3
HENN (ours)
86.1
85.4
84.2
86.2
83.3
82.3
DNN (Jin Le. 2019)
86.2
73.1
80.6
83.2
53.8
81.7
ENN (Sensuy el al., 2018)
82.4
65.3
82.3
57.7
21.6
59.1
E-CNN (Tong 2021)
73.5
68.2
RAPS (Angelopoulos et al., 2021)
78.8
55.2
80.6
75.4
40.2
81.7
(ours)
Table 12: Results (%) of Gaussian kernel size: 7 7 CIFAR100 and tinylmageNet (based run).
CIFAR100 ComplS
tiny Imngeeeeee CompIS
M
Methods
OvertS
Acc:
OverIS
Acc
DNN (Tan & Le, 2019)
86.2
62.4
83.8
83.7
44.8
83.3
(Sensoy al., 2018)
10
E-CNN (Tong al., 2021)
28.5
22.4
74.2
23.6
21.8
68.0
RAPS (Angelopoulos al. 2021)
80.0
49.3
83.8
71.5
43.9
83.3
HENN (ours)
87.0
82.7
85.8
84.3
86.9
83.8
DNN (Jan 2019)
82.5
82.5
ENN (Sensoy ol., 201
82.5
39.6
83.6
48.0
42.3
82.4
E-CNN (Tong al., 2021)
28.7
23.4
70.2
23.5
21.9
68.2
RAPS (Angelopoulos al. 2021)
78.3
51.6
82.5
73.8
43.5
82.5
HENN (ours)
86.4
79.9
84.3
84.1
83.0
83.5
DNN (Tan & Le. 2019)
85,3
69.8
80.5
83.5
56.4
81.7
(Sensoy al.,
20
E-CNN (Tong al., 2021)
28.6
23.7
73.4
23.3
21.5
68.2
RAPS (Angelopoulos al, 2021)
53.2
80.5
74.1
81.7
HENN (ouns)
85.5
81.0
80.7
83.9
81.2
83.1
F.3 SEPERATION OF SINGLETON AND COMPOSITE EXAMPLES
Fig. 4 and 5 show comprehensive ROC curves for CIFARI00 and tinyImageNa based on different Ms and different Gaussian kernel sizes, which indicates that vagueness is the best indicator compared to other different uncertainty measurements.
30

---

Published as a conference paper at ICLR 2024
1
Positive Rate
True Positive Rate
True Positive Rate
0.6
0.6
0.6
0.4
Vagueness (AUC- 1.001
Vagueness (AUC-1.00
Vagueness (AUC- 1.001
0.2
Vecalty 28)
0.2
(AUC-0.32)
0.2
Veculty (AUC-0.35
Dissorance (AUC=0.90)
Dissonance (NUC=D 90)
Dissonance (NUC=0.81 0.87)
Entropy (AUC=0.87)
Entropy (AUC=0.88)
Entropy (AUC=0.89)
0.2 0.4 0.6 0.8
0
0.2
0.4 0.6 0.8
0
0.2
0.4 0.6 0.8
False Positive Rate
False Positive Rate
False Positive Rate
(a) M=10. Ker=3
(b) M=100, Ker=5
(c) M=10, Ker=7
True Positive Rate
True Positive Rate
0.8
Positive
0.6
0.6
0.6
0.4
0.4
Vegueness (AUC= 1.001
Vagueness (AUC= 1000
Vagueness (AUC> 001
0.2
INUC=0.25 25)
0.2
Vaculty (MUCH0
0.2
Vaculty (MUC=0.33 (AUC- Entropy (AUC-0.89)
(AUC-0.98)
Dissonance (AUC-D.39)
Entropy (AUC-0.86)
Entropy (AUC-0.88)
0.2 0.4 0.6 0.8
0.2 0.4
0.8
0.2 0.4 0.6 0.8
False Positive Rate
False Positive Rate
False Positive Rate
(d) M=15, Ker=3
(e) M=15, Ker=5
M=15, Ker=7
Positive Rate
True Positive Rate
True Positive Rate
0.8
0.8
0.8
0.6
0.4
0.4
0.4
Vagueness (AUC-1.00)
Vagueness (AUC-1.00 0.001
Vagueness (AUC-1.00)
Vaculty JAUC=0 28)
Varuity (AUC=0.3)) 0.32
Vacuity (AUC-0.31] 0.11
(AUC-0.98)
(AUC-0.97)
(AUC=0.97)
0
Entropy (AUC=0.87)
Entropy (AUC-0.88)
Entropy (AUC=0.89)
0.8
False Positive Rate
False Positive Rate
False Positive Rate (i) M=20, Ker=7
(g) M=20, Ker=3
(h) MM=00, Kor=5
Figure 4: ROC curves of separating composite examples and singleton examples among different measurements: vagueness of HENN, vacurity of ENN, dissonance of ENN, and entropy of DNN CIFARIOU for different numbers of selected composite classes and kernel sizes ("Ker" represents "kemel size").
31

---

Published as a conference paper at ICLR 2024
1
Positive Rate
True Positive Rate
True Positive Rate
0.8
0.6
0.6
0.6
0.4
Vagueness (AUC=1.00
Vagueness (AUC-1.00
Vagueness (AUC- 1.001
0.2
Vecalty (AUC-0.30 30)
0.2
0.2
Veculty (AUC-0.35
Dissorance (AUC=0.90)
Dissonance (NUC=D) 90)
Dissonance (NUG=0
Entropy (AUC=0.83)
Entropy (AUC=0.83)
Entropy (AUC=0.84)
0.2 0.4 0.6 0.8
0.2 0.4 0.6 0.8
0
0.2
0.4 0.6 0.8
False Positive Rate
False Positive Rate
False Positive Rate
(a) M=10. Ker=3
(b) M=10,, Ker=5
(c) M=10, Ker=7
1
True Positive Rate
True Positive Rate
0.8
Positive
0.6
0.6
0.6
0.4
0.4
Vegueness (AUD=1.000 1.001
Vagueness (AUC= 1000
Vagueness (AUD> 001
0.2
INUC=0.20
0.2
Vaculty (MUCH0
0.2
Vaculty (MUC=0.39) (AUC-D.99) Entropy (AUC-0.85)
(AUC-0.98)
Dissonance (AUC-D.39)
Entropy (AUC-0.84)
Entropy (AUC-0.84)
0.2 0.4 0.6 0.8
0.2 0.4 0.8
0.2 0.4 0.6 0.8
False Positive Rate
False Positive Rate
False Positive Rate
(d) M=15, Ker=3
(e) M=15, Ker=5
M=15, Ker=7
Positive Rate
True Positive Rate
True Positive Rate
0.8
0.8
0.8
0.6
0.4
0.4
0.4
Vogueness (AUC-1.00)
Vagueness (AUC-1.00 0.001
Vagueness (AUC-1.00) 1.001
Vaculty JAUC=0.31)
Varuity (AUC=0.3)) 0.32)
Vacuity (AUC=0.34 04)
(AUC-0.98)
(AUC-0.97)
(AUC=0.97)
0
Entropy (AUC=0.85)
Entropy (AUC-0.86)
Entropy (AUC=0.87)
0.8
False Positive Rate
False Positive Rate
False Positive Rate (i) M=20, Ker=7
(g) M=20, Ker=3
(h) MM=00, Kor=5
Figure 5: ROC curves of separating composite examples and singleton examples among different measurements: vagueness of HENN, vacurity of ENN, dissonance of ENN, and entropy of DNN tinylmageNet for different numbers of selected composite classes and kernel sizes. ("Ker" represents "kemel size"")
32

---

Published as a conference paper at ICLR 2024
F.4 ABLATION STUDY ON REGULARIZER
To explore the effect of Regularizer for singleton evidence, we try different tradeoff coefficients À for KL regularization in our HENN method. Experiments are conducted on CIFAR100 with a pre-trained EfficientNet-b3 model and fixed learning rate 1e-5. The results are represented in Table 13. Without this regularization term, the ComplS O, which means that the model does not predict composite prediction, and the Acc is 78.5%. The reason is minimizing UPCE loss solely cannot provide sufficient composite evidence output for those composite sets. In this way, the composite examples will have relatively low accuracy compared to singleton ones. If the coefficient À increases, demonstrating larger preference on flat GDDs instead of UPCE minimizer, the evidence for singleton classes will proved in Proposition 2. Therefore, the CompJS will no longer be zero since
reduce to be flat the model tend to replace confusing singleton evidence for composite examples. 0.1 gives us the both the composite prediction metries (OverJS, CompJS) well as singleton
best performance prediction accuracy (Acc). This means that HENN can predict composite class labels but also has good singleton class label prediction. This ablation study verifies the importance of the regularization term and shows fine-tuned tradeoff hyperparameter can provide reliable composite and singleton prediction simultaneously. In addition, the OveralIJS remains high across different choices of As, demonstrating the robustness of our method.
Table 13: Effect of Regularizer: Different trade-off coefficient À on CIFAR100 with pretrained EfficientNet-b3 model and le-5 learning rate.
OverIS
CompJS
Acc
0
76.4
0.0
78.5
0.01
83.6
76.3
85.1
0.1
87.9
87.7
85.3
1.0
81.8
72.0
79.7
F,5 ADDITIONAL RESULTS
Table 14: Results (%) of NAbirds based on the trained EfficientNet-b3 backbone. (The average 95% confidence interval three rums are provided based on three runs.)
Methods
Over1S
CompIS
Acc
DNN (Tan & Le, 2019)
77. 09.38::0.19
35.244:352
78.04=+0.27
(Sensoy et al., 2018)
76.7220056
37.4642.39
78.45:0031
HENN (ours)
$0.01
71.4211143
80.14:10.35
F.5.1 EXPERIMENTS FINE- GRAINED DATASET: NABIRDS
We also conduct experiments on one ane-grained dataset: NAbirds (Van Horn et al., 2015). It has 555 different categories of birds and each category has around 50 images for both training and test set. According to the provided class hierarchy information, these 555 subclasses can be divided into 404 groups (superclasses). After filtering out superclasses which has only single subclass, the same procedure as previous four datasets (TinylmageNet, Living17, Nonliving26, and CIFAR 100) is applied to randomly select 10 composite class labels. Tab. 14 shows results based on the fine-grained dataset NAbirds (Van Horn et al., 2015). Consistent with previous experiments on four datasets, HENN outperforms DNN and ENN for large margin in terms of CompJS. And HENN also performs better in terms of OverJS and Acc.
F.5.2 REAL-WORLD DATASET WITH COMPOSITE CLASS LABELS
We admit that the datasets with Gaussian blurring are semi-synthetic. From sizable pool of applicants, we selected 23 students from our department and tasked them with annotating images in the CIFAR10 dataset and one subset of tinylmageNet (renamed as innllmageNet-20), categorizing each as either a singleton class or a composite set. This effort successfully resulted a real-world dataset enriched with human-annotated singleton and composite labels. Tab. 15 shows results based
33

---

Published as a conference paper at ICLR 2024
Table 15: Results (%) on CIFARIO based on two backbones, (The average 95% confidence interval of three runs are provided based on five runs.).
ResNet18 CompāS
EfficientNet-h3
Methods
OverIS
CompI5
DNN (Tan Le, 2019)
79.73-#0..3
40. 700
82.17=0.54
92.53:a..11
53.59=3.15
96.49:00.11
(Sensoy et
46.80.-006
82.75-0.19
54.83-0059
E- CNN (Tong al., 2021)
59.68++0.62
31.84+081
66.23+1147
63 63.65-+0992
34.74=2.91
68 08.98+0..72
RAPS (Angelopoulos al., 2021)
62.60:20.45
33.8024466
82.17:0.54
65.70:0000
39.40=2.29
96.4920.21
HENN (ours)
80,74.0.117
51.44:102
83.03-00144
93,38110066
72.87
97.52::0.4
Table 16: Results (%) on tinylmageNet-20 based on the ResNet18 backbone. (The average and 95% confidence interval of three provided based five runs.).
ResNe118 ComplS
Methods
Overd8
Aoc
DNN (Tan & Le, 2019)
40.03-+00.9
24.70:4115
42.20-+1:24
ENN (Sensoy ot al., 2018)
36.44:1:.65
22.78:11.48
42.45:11.15
HENN (ours)
42.43--0.75
25.32+1377
1.0
1.0
1.0
True Positive Rate
True Positive Rate
Positive
0.6
0.4
Veg nece (MUC-5 28)
0.4
Veg ae neas (MUC- m5)
0.4
Vag meee (MUC- 63)
Vacurty (AUC=0.55
Vacurty (AUCHO
Vicutyy (AUC=0 52)
Dissomance JAUCHO, 75)
Cssomance (AUCHO
0.2
sssnnance (AUCH2
Enerpyy (AUC-0.75) 0575
Entropy (AUC-0.83)
Enjopy (AUC-057)
0.8.
0.7
0,4
0.6
0.8
L.0
08.6
022
044
0.6
03
T.0
0.8.0
0.2
0.4
0.8
0.
1.0
Faise Positive
False Positive
False Positive Rate
(a) CIFARIO (ResNet18)
(b) CIFARIO (EfficientNet-b3)
(c) tinylmageNet-20 (ResNet18)
Figure 6: AUC curves of different uncertainty types: Vagueness, Vacuity, Dissonance, and Entropy for two datasets. (a) CIFARIO based on ResNet18 training from scratch; (b) CIFAR10 fine-tuned on pre-trained EfficientNet-b3; (c) tinylmageNet-20 based on ResNet18 training from scratch.
on real-world dataset CIFAR10 Krizhevsky & Hinton (2009) based on two backbones. HENN outperforms DNN and ENN for a large margin. Fig. 6a and 6b show AUROC curves and scores for different metrics: vagueness, dissonance, vacuity, and entropy. It demonstrates that vagueness is good indicator to identify whether the image is singleton-labeled or composite-labeled, indicating HENN's advantage.
F.5.3 ANOTHER DATA CORRUPTION
Table 17:
Results (%) of BREEDS-Living-1 17 based the pre-trained EfficientNet-b3 backbone. (The average and confidence interval of three runs are provided based on three runs).
Methods
OverdS
CompIS
Ace
DNN (Tan Le, 2019)
87.28:10233
74.61 22.57
1.336
ENN (Sensuy al., 2018)
87.46:50.34
69.44:3325
85.38
RAPS
/Angelopoulas 2021)
85.38-2032
62. 00.10.0266
84.35 36
HENN (ours)
88.09-20.21
96.33
86.12:1037
Besides the Gaussian blurring we used, the Bicubic transformation was also examined as detailed in 17b. 17. The findings from this analysis align consistently with the result of the experiment based on Gaussian blurring.
F.5.4 CASE STUDY ON HENN TRAINED WITH EXCLUSIVE SINGLETON CLASS DATA
Tab. presents the accuracy results from the ENN and HENN methods trained and evaluated on CIFAR 100 with only singleton class data (without Gaussian blurring and label replacement) across
34

---

Published as a conference paper at ICLR 2024
Table 18: Case Study: HENN Trained with Exclusive Singleton Class Data.
Methods
ENN
HENN
Acc(%)
85.82 + 1.0 85.81 1 2.4
trials each. The mean accuracy and the standard deviation are reported. Under traditional singleton classification setting, HENN still shows comparable performance in terms of accuracy compared to ENN model. notable advantage HENN is its ability to quantify an additional type of uncertainty compared to ENN with minimal performance degradation observed even if the training data consists of exclusive singleton ones.
F.5.5 CASE STUDY ON EVIDENCE OUTPUT
In this section, we show the effect of the regularization coefficient À by demonstrating its impact on the output evidence throughout a case study. To verify our intuitions for Proposition 2, we set experiments to inspect the non-zero ratio of both singleton evidence and composite evidence among all testing examples. A small positive threshold value of y 10 is introduced to determine whether the mean singleton or composite evidence is non-zero while adapting to the computation precision in practice. In other words, for each testing data point, we calculate the predicted evidence fx; 0) = (a, c). and based on the given hyper-domain, it is feasible to get mean evidence in singleton domain â ক.. indicator function of non-zero singleton prediction
E k=1 Org, and the composite domain č
|X|
[ 2==1 y= Cj. Define the
(x))
[1,
if 2 7, otherwise,
Geomp (c) 1, if 2 7 0,
Isogi (
(55)
0,
otherwise,
and the non-zero ratios are ZZ Nisa Ei'r grngl (a) DZcomp Nuw Eder (comp(C) by taking the mean of all testing samples. The case study is carried out CIFAR 100 with EfficientNet-b3 backbone with the same setting as in previous sections. By controlling regularization coefficient different levels of value, the predicted evidence from HENN listed in Table 19, indicating that larger regularization can adjust the evidence distribution to be more balanced between singleton and composite parts. Oppositely, a lower regularization coefficient or without any regularization can result in concentrating predictive evidence only on the singleton part.
Table 19: Case Study: Effectiveness of regularization term on evidence distribution.
A
nZZagg
12Z.cmp
0.01
71.52%
71.72%
10
100.0%
0.04%
10
100.0%
0.2%
0
100.0%
0.3%
Empirical verification of Propositions 1 and Eq.14. Our case study CIFAR 100 in App. F.5.5 demonstrates observations consistent with our propositions: (1) HENN trained based on UPCE and a training set consisting of only singleton class labels predicts non-zero evidence composite class labels for 14.4% of the training samples even that the training set does not have evidence composite class labels to accumulate, and (2) The HENN trained based UPCE and training set consisting of only composite class labels predicts non-zero evidence on singleton class labels for 100.0% of the training samples even that the training set does not have evidence of singleton class labels to accumulate. Our proposed regularization can avoid these unexpected behaviors. The UAP for neural networks has been studied (Leshno et al., 1993b) and recently used in the theoretical analyses of ENN-related paper for graph data (Alan Hart et al., 2023).
35


Reviews:
[{"review_id": "oyH6N2bVVZ", "reviewer": "Reviewer_RL9p", "summary": "The paper suggests a variant of Dempster-Shafer Theory to arrive at decisions that allow for uncertainty quantification for composite classification. In the proposed HENN framework the uncertainty from composite annotations during training is leveraged for quantification of classification uncertainty. Experiments on image datasets are used to demonstrate the effectiveness of HENN which the authors support by a theoretical analysis as well.", "questions": "I have only two questions:\n- do you see any relation to Brust et al.: Making Every Label Count: Handling Semantic Imprecision by Integrating Domain Knowledge. ICPR 2020 and which concepts of this work is related to yours? \n- did you perform experiments on more relevant benchmark datasets, like NABirds (https://dl.allaboutbirds.org/nabirds). For this dataset, hierarchies exists and benchmarks in fine-grained recognition which most likely would benefit most from your suggested idea.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Undoubtedly, annotation uncertainty is a big challenge in today's machine-learning models. Thus, this paper tackles a highly relevant topic. In addition, uncertainty in the output of DNN models is the focus of many papers, e.g. to find the right calibration of the values to allow interpretation and decision-making. The authors suggest DST for solving this issue, which I did not see before for DNN. If the theoretical analysis (Sec. 4.1) can be confirmed to be correct and relevant in practice (which I could not!), I would see a relevant contribution by this work, at least to inspire others to look into hyper-opinions.", "weaknesses": "I see two significant shortcomings:\n1. composite classification is not new, and solutions exist in a different context and with different methods (i.e. no DST). See, for example, Brust et al.: Making Every Label Count: Handling Semantic Imprecision by Integrating Domain Knowledge. ICPR 2020. The authors did not make clear what the advantage of their framework is compared to such work. This referenced work might differ, but the overall modelling procedures look similar to the one in the submission: annotations are at a different level of a concept hierarchy (dog->husky, dog->wolf, might generate label \"dog\" if uncertain, or \"husky/wolf\" if certain). \n2. the selected datasets are not suitable to demonstrate a (at least for me) complex theory behind uncertainty and how to include it into deep learning loss functions and regularization. For example, I am not sure what Gaussian blurring will make with tiny-images, i.e. what remains as information after blurring. \n\nI am also not happy with the theoretical part of the paper, although I have to admit that I am not that familiar with DST, and this hindered me from diving deeper into the derivations done in Sec. 4.1 The notation and style of presentation is probably only proper for an expert in this area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y5Gwf9XJKC", "reviewer": "Reviewer_Gzpb", "summary": "A hyper-evidential neural network is presented in this paper, for classification of data, modelling predictive uncertainty based on training data with composite set labels. The uncertainty is measured by introducing the vagueness type of measure. Results are presented where HENN performance is compared favorably with other methods over four image datasets. Detailed analysis is included in Appendices.", "questions": "Following the above, would it be possible to apply the method in real world applications involving composite labels? \nMoreover, performance seems to heavily depend on regularization hyperparameter, which is selected in an ad-hoc manner. Is this a pitfall for method's robustness over different datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper deals with a significant problem, i.e., to train DNNs when (some) training data have composite labels, being able to predict these labels and quantify the predictive uncertainty due to these labels. It defines a related measure, vagueness, to do so and extends neural network structures to model this uncertainty for classification problems; it uses an uncertainty partial cross entropy loss function extending the normal UCE function.  An experimental study is presented which illustrates a good performance over four image datasets, when compared to five other methodologies that can be applied in this context.", "weaknesses": "The paper defines vagueness as a measure of the predictive uncertainty due to composite labels of training data. It then uses gaussian blurring in the experiments to create such data cases and perform the experimental verification. However, this is a rather specific synthetically generated experiment, which can not justify the significance of the results over real world applications. Such applications could, for example, include classification of facial images showing compound and primary emotions in-the-wild (where compound emotions two or more primary ones), or in image2image translation tasks. Moreover, the comparison shown in Table 2 does not seem fair, since - as also discussed in the 'Classification' results subsection (in 5.2) - the other methods are not designed to handle this type of vague images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v21uhCfSap", "reviewer": "Reviewer_AQu8", "summary": "The paper proposes a novel framework for deep learning to deal with composite labels, i.e. labels that couldn't be assigned to a single class due to quality of the input, for example, and were assigned to more than one class. The framework assumes a neural network to output, similarly to the training labels, both singleton classes and composite labels. The paper also proposes a novel uncertainty metric, called vagueness, that is measuring uncertainty related to composite label evidence in training. To train a novel neural network with composite output, the paper proposes a novel loss function.\n\n\n============\n\nUpdate after rebuttal: I have read the authors’ response and other reviews. I really appreciate the effort, information and novel results provided by the authors to all reviews. I am happy to see even my suggested small experiment on using singleton labels only implemented. \n\n\nHowever, a tiny negative note. I appreciate the authors’ answer and motivation about having composite labels during training, though if possible I would suggest using a slightly more modern reference. Also, the authors haven’t address my concern about why we need for the neural network to output composite labels. Though, it is tiny in comparison to strengths of the paper and none of the other reviewers saw that as a concern, so I am increasing my score.", "questions": "My main question to the authors, the answer to which hopefully will clarify any doubts from my side, is the question of motivation of the problem setup. I.e. how important is the problem of having composite labels during training and why do we need to also output composite labels by a NN?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* A very thorough work has been done with the new framework (and basically the new problem setup). Extensive theoretical evaluation, empirical evaluation including ablation study, a good number of baselines considered for comparison. \n* In addition to the new framework itself, the novel uncertainty metric is made as a proper contribution on its own with the thorough theoretical and empirical comparison with other uncertainty metrics. \n* The paper is mostly well-written and presents the context of prior work. The only drawbacks I can see are obviously due to the lack of space.\n\nOriginality: I am not familiar well with the related works, but both the framework and the uncertainty metric appears to be novel. \n\nQuality: Very well done and thought through piece of work. See above.\n\nClarity: Mostly well-written and easy to follow. \n\nSignificance: Empirical evaluation demonstrates that even in terms of single class classification (i.e. output of a NN is a single class) the proposed framework demonstrates improved accuracy in comparison to all considered baselines. Moreover, the proposed uncertainty metric demonstrates significantly superior performance in terms of distinguishing between composite and single labels examples.", "weaknesses": "The main weakness that I see in the paper is the lack of motivation how severe is the problem of having composite labels in practice for training and why we need a NN to output us a composite label in practice. The main conceptual motivation that I gathered from the paper is that the new framework allows to estimate uncertainty related to composite labels in training data. However, the argument is not too convincing. It seems that we don't have to produce a composite label in order to estimate the effect of composite labels in the training data (not that I know how to do it without). \nEmpirical evidence shows us the practical motivation of using the novel framework: it gives us the higher accuracy when we consider a single class prediction. However, again this is shown in the experiments with the composite labels in training, which is not well motivated how often in practice we would have these labels for training. \nTo this end, I think a small experiment, showing that the proposed framework still works with traditional setup of single labels for training only, would be beneficial. \n\nOriginality: I appreciate that the considered problem formulation is different in that a single class is expected to be true for each sample, but maybe lacking due to the quality of the sample, but still I find it is a related area. The area of research devoted to multiclass classification is missing in the related works (e.g., Augustin, A.; Venanzi, M.; Hare, J.; Rogers, A.; and Jennings, N. 2017. Bayesian aggregation of categorical distributions with applications in crowdsourcing. AAAI Press/International Joint Conferences on Artificial Intelligence, but there are lots of others).\n\nQuality: The code is provided which should elevate this weakness, but based on the text not all implementation details are provided sufficiently to reproduce the results. See details below. \nDifferent types of image corruption would be interesting to see in the experiments in addition to the Gaussian blur. \n\nClarity: A lot of theory is packed in a very small space with lack of illustrative examples. See details below. \n\nSignificance: As per the main drawback mentioned above, I am not sure about the significance of the paper due to very specific problem setup, which was not too convincing for me. \n\nSpecific comments/suggestions:\n1. First two sentence of the text are unclear how to connect to the rest of the text. The problem considered is not related to missing data but rather ambiguous data due to the quality of the input. \n2. Missing reference in third paragraph of Section 1. \n3. Section 3. Illustrative examples of what this means in practice would be much appreciated. Note that example from Table 1 is not sufficient. For example, what does evidence 24 mean in practice? 24 annotators voted for this category? \n4. Figure 1 right is not referred to in the text. \n5. UCE (page 6) is not defined.\n6. Propositions 1 and 2. How reasonable is the assumption of universal approximation property? \n7. Eq. 15 appears without any introduction, connection to the previous text. \n8. Preprocessing of the dataset to create composite labels is unclear (including the text in the appendix). \n* \"Several random subclasses for each selected superclass will be chosen\" - for what? How these subclasses are used further?\n* Which images are selected to be blurred? \n9. How CompJS is computed for baselines not producing composite output is unclear until explanation of what cutoff is in Appendix. Also mentioning of cutoffs in the main text is unclear until this explanation in Appendix (no reference in the main text). \n10. It is unclear exactly how superclasses are extracted for Living17 and Nonliving26 datasets.\n11. The full list of data augmentations is required for reproducibility.\n12. Up until the last paragraph in page 26 in Appendix, it is not clear how the process of duplicating samples with composite labels is working (there is no reference to this paragraph in the earlier text). \n13. Section F.4 There are SinglJS, SingleAcc and Acc mentioned in different places. What is the correct one?\n \n\nMinor:\n1. Page 7. The paragraph before Section 5. \"As a generalized framework of ENN, The HENN\" -> \"the\"\n2. Section F.4 title. \"regularier\" -> \"regulazier\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MPiDNh7xBs", "reviewer": "Reviewer_3vp1", "summary": "The DNN’s uncertainty due to composite set labels (i.e., an example might be labeled as a set of possible classes, but only one class is true) in training data is considered. This work introduces a new type of uncertainty termed vagueness, and propose a framework Hyper-Evidential Neural Network (HENN) to quantify this type of uncertainty. Further, a loss named uncertainty partial cross entropy (UPCE) is proposed. Finally, the proposed method is shown to be effective on four image datasets.", "questions": "Please see the comments in the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tA novel type of DNN uncertainty is introduced, and the uncertainty calibration is evaluated with the metric Jaccard Similarity.\n2.\tAn extension of the partial cross, uncertainty partial cross entropy (UPCE) is proposed.", "weaknesses": "1.\tAlthough the newly introduced uncertainty concept and problem setup sound interesting, I am not sure the empirical evaluation is realistic and convincing enough. Are all the composite labels used in the empirical evaluations synthetically generated? Could you provide more concrete examples in these synthetic datasets? I am curious how realistic these generated composite labels are. If so, is it possible to run evaluations on real-world composite set labels?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)