Random-Set Neural Networks

## Abstract

Abstract Machine learning is increasingly deployed in safety-critical domains where robustness against adversarial attacks is crucial and erroneous predictions could lead to potentially catastrophic consequences. This highlights the need for learning systems to be equipped with the means to determine a model’s confidence in its prediction and the epistemic uncertainty associated with it, ‘to know when a model does not know’. In this paper, we propose a novel Random-Set Neural Network (RS-NN) for classification. RS-NN predicts belief functions rather than probability vectors over a set of classes
using the mathematics of random sets , i.e., distributions over the power set of the sample space.
RS-NN
encodes
the ‘epistemic’ uncertainty induced in machine learning by limited training sets via the size of the credal sets associated with the predicted belief functions.
Our approach outperforms state-of-the-art Bayesian (LB-BNN, BNN-R) and Ensemble (ENN) methods in a classical evaluation setting in terms of performance, uncertainty estimation and out-of-distribution (OoD) detection on several benchmarks (CIFAR-10 vs SVHN/Intel-Image, MNIST vs FMNIST/KMNIST, ImageNet vs ImageNet-O) and scales effectively to large-scale architectures such as WideResNet-28-10, VGG16, Inception V3, EfficientNetB2, and ViT-Base.

## 1 Introduction

Despite its
success,
artificial intelligence, in its current form, struggles to
provide reliable predictions when confronted with unfamiliar data . This includes noisy samples deliberately designed to deceive models, or out-of-distribution data (OoD) beyond the model’s training distribution.
This emphasizes the need for learning systems to be able
to determine a model’s confidence in its own predictions, acknowledge the limits of its knowledge and gauge these limitations to make informed decisions by modelling the *epistemic uncertainty* associated with it, in essence, ‘to know when a model does not know’.
Epistemic uncertainty pertains to uncertainty associated with our ignorance about the underlying data generation process ; ; . Within machine learning, a major source of epistemic uncertainty arises from the
limited representativeness of the available training data, constrained in both quantity and quality.

Figure: Figure 1: Inference in a Bayesian Neural Network (top) as opposed to a Random-Set Neural Network (bottom), with corresponding measures of uncertainty and their sources.
Refer to caption: x1.png

In this paper, we propose to model epistemic uncertainty using a *random set* approach . As we focus on
classification involving a finite list of classes,
we model uncertainty using *belief functions* , the finite incarnation of random sets . The theory of belief functions, or Dempster-Shafer theory of evidence , is, in fact, a generalisation of Bayesian inference . Classical (discrete) probabilities are simply special belief functions, and Bayes’ rule is a special case of Dempster’s rule
originally proposed for the combination of belief functions.
Fig. 2 contrasts the inference processes of a Random-Set Neural Network (RS-NN) and a Bayesian Neural Network .
In hierarchical Bayesian inference, the predictive posterior distribution is a second-order probability distribution .
It involves sampling from predicted first-order distributions for a collection of possible models
within the Bayesian Neural Network.
A single (mean) prediction is derived by Bayesian Model Averaging (BMA), while uncertainty is measured by the entropy of the mean prediction and the variance of the posterior distributions.

In a Random-Set NN (RS-NN)
(Fig. 2, bottom), each test input at inference time is mapped to a belief function on the collection of classes $\mathcal{C}$.
This belief function is encoded by a mass value $m(A)\in[0,1]$
assigned to a finite budget $\mathcal{O}=\{A_{k}\subset\mathcal{C}\}$ of
*sets* of classes. The
most relevant
such sets
are identified from training data by fitting Gaussian Mixture Models (GMMs) to
the labelled data
and computing the overlap among the resulting clusters .
Such a predicted belief function is mathematically equivalent to a convex set of probability vectors (a *credal set* ; ) on the
class list $\mathcal{C}$.
A pointwise prediction can then be obtained by computing the centre of mass of this credal set, termed *pignistic probability* . In
RS-NN,
uncertainty can be expressed either via the entropy of the pignistic prediction, or via the width of the credal set prediction .
As we show in this paper, the entropy of a
pignistic prediction is more diverse than that of a Bayesian prediction across in-distribution and out-of-distribution data (see Tab. 1),
and the width of the credal prediction, as it encodes epistemic uncertainty about the prediction itself, is empirically much less correlated with the confidence score than entropy (see §D.5.1).

Using a random-set representation prevents the need to discard information, a challenge observed in Bayesian Model Averaging ; .
While
Bayesian inference requires
defining prior distributions for model parameters even in the absence of relevant information,
in belief theory,
priors are not required for the inference process, thus avoiding the selection bias risks that can seriously condition Bayesian reasoning .
Based on our extensive experiments on multiple datasets, including large-scale ones, RS-NNs not only demonstrate superior accuracy compared to state-of-the-art Bayesian models (Sec. 4.2),
but also arguably better encode the epistemic uncertainty associated with the predictions (Sec. 4.4) and
better distinguish between in-distribution and out-of-distribution data (Sec. 4.3).
Furthermore,
RS-NNs effectively circumvent the tendency of standard neural networks to generate overconfident incorrect predictions, as illustrated in Fig. 2 in an experiment on Fast Gradient Sign Method (FGSM) adversarial attacks on CIFAR-10.
CNN misclassifies with high confidence scores of 92.42% and 97.78%, while RS-NN shows lower confidence at 35.92% and 39.58%.

Contributions. Firstly,
we propose a novel *Random-Set Neural Network (RS-NN)* based on the principle that a deep neural network
predicting belief values for *sets* of classes, rather than individual classes, has the potential to be
a more faithful representation of the epistemic uncertainty induced by
the limited quantity and quality of the training data. As they assign probability values to sets of outcomes directly, random sets ; naturally model the fact that observations almost invariably come in the form of sets, and can accommodate
ambiguity, incomplete data, and non-probabilistic uncertainties.
Secondly, we develop a novel *budgeting* method for efficiently selecting from the training data a fixed budget of
relevant sets of classes for the task at hand,
by fitting Gaussian Mixture Models to classes and computing clusters of classes,
thus avoiding the
exponential complexity of
vanilla random-set implementations
and ensuring scalability.
A random-set representation serves as a foundational framework: when integrated with a budgeting of sets, this method acts as a versatile wrapper applicable to any model architecture.
Thirdly,
we illustrate the advantages of two new methods for assessing the uncertainty associated with a random-set prediction: the Shannon entropy of the pignistic prediction (the centre of mass of the corresponding credal set) and the width of the credal prediction itself.
Finally,
we present a large body of experimental results
demonstrating how RS-NN outperforms both state-of-the-art Bayesian (BNN-R, LB-BNN) and Ensemble (ENN) methods in terms of: (i) performance (test accuracy, inference time) (Sec. 4.2); (ii)
results
on various out-of-distribution (OoD) benchmarks (Sec. 4.3), including CIFAR-10 vs. SVHN/Intel-Image, MNIST vs. FMNIST/KMNIST, and ImageNet vs. ImageNet-O; (iii)
ability to provide reliable measures of uncertainty quantification (Sec. 4.4) in the form of pignistic entropy and credal set width, verified on OoD benchmarks;
(iv) scalability to large-scale architectures (WideResNet-28-10, Inception V3, EfficientNet B2, ViT-Base) and datasets (e.g. ImageNet) (Sec. 4.5).
Additionally, we provide statistical guarantees on RS-NN using non-conformity scores (§A), we show how RS-NN is robust to adversarial attacks (§D.2) and noisy data (§D.1),
and circumvents the overconfidence problem in CNNs
(§D.3).
A qualitative assessment of entropy vs credal set width is given in §D.5.1.

Paper outline. We first recall the notion of random sets, the mathematics of belief functions, and the associated credal and pignistic predictions (Sec. 2). We then explain the architecture, loss function and uncertainty representation of RS-NN (Sec. 3). In Sec. 4, we provide a large body of evidence supporting our approach. Sec. 4.6 details the limitations our method, including the budgeting time and fixed number of focal sets. Finally, Sec. 5 concludes and outlines future work. Appendix §A includes a discussion on statistical guarantees, §B, §C and §D
contain additional evidence.

Related Work. The machine learning community has recognised the challenge of estimating uncertainty in model predictions, leading to the development of various methods such as Bayesian approximations , evidential Dirichlet distributions and conformal prediction ; ; ; ; .
Some methods rely on prior knowledge , whereas others require setting a desired threshold on the metrics .
Some
; argued that classical probability theory is, in fact, not equipped to model “second-level" uncertainty on the probabilities themselves. This has led to the formulation of numerous uncertainty calculi ,
including
possibility theory , probability intervals
, credal sets , random sets and imprecise probability .
Bayesian approaches, pioneered by Buntine and others ; , have recently become dominant in uncertainty estimation. Notable techniques include R-BNN (Reparameterisation) , variational inference with reparameterisation trick
and Laplace Bridge Bayesian approximation (LB-BNN) which uses the Laplace Bridge to map efficiently between Gaussian and Dirichlet distributions and enhance computational efficiency. Various approximations of full Bayesian inference exist, such as Markov chain Monte Carlo (MCMC), variational inference (VI), and Dropout Variational Inference . Recent work addresses challenges in computational cost, model priors, and tractable function-space variational inference ; ; ; . Despite their advantages, Bayesian models face challenges when the model prior is misspecified. Our approach mitigates these challenges by eliminating the need for sampling during inference and prior selection, reducing computational complexity compared to Bayesian inference, as demonstrated in our experiments (see Tab. 2).
Ensemble-based approaches, such as Deep Ensembles and Epistemic Neural Networks (ENN) , efficiently estimate uncertainty by leveraging multiple models. However, the computational cost of training ensembles, especially for large models, is often impractical
(see Appendix §B).

## 2 Random sets and belief functions

Random sets.
A die is
a simple example
of a (discrete) random variable. Its probability space is defined on the sample space $\Theta=\{\text{face1},\text{face 2},\ldots,\text{face 6}\}$, where elements are mapped to the real numbers $1,2,\ldots,6$, respectively.
Now, imagine that faces 1 and 2 are cloaked, and we roll the die. How do we model this new experiment, mathematically? Actually, the probability space has not changed (as the physical die has not been altered, its faces still have the same probabilities). What has changed is the mapping: since we cannot observe the outcome when a cloaked face is shown (we assume that only the top face is observable), both face 1 and face 2 (as elements of $\Theta$) are mapped to the set of possible values $\{1,2\}$ on the real line $\mathbb{R}$ (Fig. 4).

Figure: Figure 3: The random set (set-valued random variable) associated with a cloaked die in which faces 1 and 2 are not visible.
Refer to caption: x3.png

Mathematically, this is a *random set* ; ; ; , i.e., a set-valued
random variable, modelling random experiments in which observations come in the form of sets.

Belief functions. Random sets have been proposed by Dempster and later Shafer as a mathematical model for subjective belief, alternative to Bayesian probability. For this reason,
on finite domains (classification) they assume the name of *belief functions*.
While classical discrete mass functions assign normalised, non-negative values to *elements* $\theta\in\Theta$ of their sample space, a belief function independently assigns normalised, non-negative mass values to *subsets* of the sample space:

$$ $m(A)\geq 0,\forall A\subset\Theta,\sum_{A}m(A)=1.$ (1) $$

The belief function associated with a mass function $m$ then measures the total mass of the subsets of each focal set $A$. Mass functions can be recovered from belief functions via Moebius inversion :

$$ $Bel(A)=\sum_{B\subseteq A}m(B),\>\>\>\>\>m(A)=\sum_{B\subseteq A}{(-1)^{|A \setminus B|}{Bel}(B)}.$ (2) $$

It follows that classical (discrete) probability measures
are a special case of belief functions.
Consider for instance a
sample space (class list)
$\Theta=\mathcal{C}=\{\text{$c_{1}$},\text{$c_{2}$},\text{$c_{3}$}\}$ and let $2^{\Theta}$ be its power set (collection of all subsets). Define a mass function $m:2^{\Theta}\rightarrow[0,1]$ as: $m(\{\text{$c_{1}$}\})=0.3$, $m(\{\text{$c_{2}$}\})=0.2$, $m(\{\text{$c_{3}$}\})=0.1$, $m(\{\text{$c_{1}$},\text{$c_{2}$}\})=0.4$, whereas
$m(\{\text{$c_{1}$},\text{$c_{3}$}\})=0$, $m(\{\text{$c_{2}$},\text{$c_{3}$}\})=0$ and $m(\{\text{$c_{1}$},\text{$c_{2}$},\text{$c_{3}$}\})=0$ (Fig. 4, top).
Note that
$m$
is normalised: $\sum_{B\subseteq\Theta}m(B)=1$.
By (2), the belief value of the composite class $\{\text{$c_{2}$},\text{$c_{3}$}\}$ is:
$\text{Bel}(\{\text{$c_{2}$},\text{$c_{3}$}\})=m(\{\text{$c_{2}$}\})+m(\{\text{
$c_{3}$}\})=0.2+0.1=0.3$ (Fig. 4, left). Similarly, the belief value of composite class $\{\text{$c_{1}$},\text{$c_{2}$}\}$ can by computed as $\text{Bel}(\{\text{$c_{1}$},\text{$c_{2}$}\})=m(\{\text{$c_{1}$}\})+m(\{\text{
$c_{2}$}\})+m(\{\text{$c_{1}$},\text{$c_{2}$}\})=0.3+0.2+0.4=0.9$.

Pignistic probability.
Given a belief function $Bel$, its *pignistic probability* is the precise probability distribution obtained by re-distributing the mass of its focal sets $A$ to its constituent elements, $c\in A$:

$$ $BetP(c)=\sum_{A\ni c}\frac{m(A)}{|A|}.$ (3) $$

Smets originally proposed to use the pignistic probability for decision making using belief functions, by applying expected utility to it. Notably, the pignistic probability is geometrically the centre of mass of the credal set associated with a belief function (, Fig. 4).

Credal prediction. As anticipated, RS-NN is designed to predict a belief function (a finite random set) on the set of classes.
A belief function, in turn, is associated with a convex set of probability distributions (a *credal set* ; ; ; ; ; ) on the same domain.
This is the set:

$$ $Cre=\left\{P:\mathcal{C}\rightarrow[0,1]|Bel(A)\leq P(A)\right\},$ (4) $$

where
$P$ is any probability distribution on
the collection of classes $\mathcal{C}$. As they have belief values as lower bounds, credal sets induced by belief functions have a peculiar shape (see Fig. 4, right, for a case with 3 classes). Their vertices are induced by the permutations of the elements of the sample space (for us, the set of classes) ; . Given one such permutation (e.g., $(c_{2},c_{4},c_{1},c_{3})$ for a set of 4 classes)
the probability vector representing the corresponding vertex of the credal prediction assigns to each specific class
the mass of all the focal sets containing that class, *but* not containing any class preceding it in the permutation order .

The size of the resulting credal prediction measures the extent of the related epistemic uncertainty (see Sec. 3.3), arising
from lack of evidence. A credal prediction, by encompassing multiple potential distributions, reflects the model’s acknowledgment of this uncertainty. A wider credal set indicates higher uncertainty, as the model refrains from committing to a specific probability distribution due to limited or conflicting evidence. In contrast, a narrower credal set implies lower uncertainty, signifying a more confident prediction based on substantial, consistent evidence.

## 3 Random-Set Neural Network

### 3.1 Architecture

Representation.
A classifier $e$ (e.g., a neural network) is a mapping from an input space $X$ to a
target space
$\mathcal{C}$ (the set of classes),
i.e.,
$e:X\rightarrow\mathcal{C}$.
In our set-valued classification setting, on the other hand, $e$ is a mapping from $X$ to the set of all *subsets* of $\mathcal{C}$, the powerset $\mathbb{P}(\mathcal{C})$, namely: $e:X\rightarrow\mathbb{P}(\mathcal{C})$.

As shown in Fig. 5, RS-NN predicts for each input data point a
belief function,
rather than a vector of softmax probabilities as in a traditional CNN.
For $N$ classes, a ‘vanilla’ RS-NN would have $2^{N}$ outputs (as $2^{N}$ is the cardinality of $\mathbb{P}(\mathcal{C})$), each being the belief value (2) of the focal set of classes $A\in\mathbb{P}(\mathcal{C})$ corresponding to that output neuron.
Given a training datapoint with attached a true class, its
ground truth is encoded by the vector $\mathbf{bel}=\{Bel(A),A\in\mathbb{P}(\mathcal{C})\}$
of belief values for each focal set of classes $A\in\mathbb{P}(\mathcal{C})$, where
$Bel(A)$ is set to 1 iff the true class is contained in the subset $A$, 0 otherwise.(^1^11Note that the belief encoding of ground-truth is not related to label smoothing. It maps ground-truth labels from the original class space to a set space without adding noise, thus preserving label ”hardness”.)
This corresponds to full certainty that the element belongs to that set and there is complete confidence in this proposition.

Figure: Figure 5: RS-NN model architecture. Given a collection $\mathcal{C}$ of $N$ classes, the network selects the top $K$ relevant (focal) sets of classes $A_{1},\ldots,A_{K}$ from the powerset $\mathbb{P}(\mathcal{C})$, encodes the ground truth training classes as belief functions $Bel$, and uses them to predict a belief function $\mathbf{\hat{bel}}$ for each input test data point, using the loss (7). Mass values $\hat{m}$ and pignistic probability estimates $BetP$ are computed from the predicted belief function. Uncertainty is estimated as described in Sec. 3.3
Refer to caption: x5.png

Budgeting. To overcome the exponential complexity of using $2^{N}$ sets of classes (especially for large $N$), a fixed budget of $K$ relevant non-singleton (of cardinality $>1$) focal sets are used.
These focal sets are obtained by clustering the original classes $\mathcal{C}$ and selecting the top $K$ focal sets of classes with the highest overlap ratio, computed as the intersection over union for each subset $A$ in $\mathbb{P}(\mathcal{C})$: $overlap(A)={\cap_{c\in A}A^{c}}/{\cup_{c\in A}A^{c}},\>A_{1},\ldots,A_{K}\in
\mathbb{P}(\mathcal{C})$.
The clustering is performed on feature vectors of images of each class generated by a standard CNN trained on the original classes $\mathcal{C}$.
The feature vectors are further reduced to 3 dimensions using t-SNE (t-Distributed Stochastic Neighbor Embedding) before applying a Gaussian Mixture Model (GMM) to them.
Ellipsoids , covering $95\%$ of data, are generated using eigenvectors and eigenvalues of the covariance matrix $\Sigma_{c}$ and the mean vector $\mu_{c}$, $\forall c\in\mathcal{C},P_{c}\sim\mathcal{N}(x_{c};\mu_{c},\Sigma_{c})$ obtained from the GMM to calculate the overlaps.
To avoid computing a degree of overlap for all $2^{N}$ subsets, the algorithm is early stopped when increasing the cardinality does not alter the list of
most-overlapping sets of classes.

The $K$ non-singleton focal sets so obtained, along with the $N$ original (singleton)
classes, form our network outputs
$\mathcal{O}=\mathcal{C}\cup\{A_{1},\ldots,A_{K}\}$.
E.g., in a $100$-class scenario, the powerset contains $2^{100}$ subsets ($10^{30}$ possibilities).
Setting a budget of $K=200$, for instance, results in
$100+K=300$ outputs, a far more manageable number.

The budgeting step is a one-time procedure, applied to any given dataset before training. It requires
just 2 minutes for CIFAR-10, 7 minutes for CIFAR-100, and
2 hours for ImageNet ($\approx$ 1.1M images, 1000 classes).
In return, it greatly helps overcome the computational challenge of dealing with a large number of classes. Budgeting can be performed on features obtained from pre-trained classifiers.

### 3.2 Loss function

A Random-Set prediction problem is mathematically similar to the multi-label classification problem, for in both cases
the ground truth vector contains several 1s. In the former case, these correspond to sets all containing the true class; in the latter, to all the class labels attached to same data point.
Despite the different semantics, we can thus adopt as loss Binary Cross-Entropy (BCE) (5)
with sigmoid activation,
to drive the prediction of a belief value
for each
focal set in the identified budget.

$$ $\begin{array}[]{lll}\mathcal{L}_{BCE}=\displaystyle-\sum_{i=1}^{b_{size}}\frac {1}{|\mathcal{O}|}\sum_{A\in\mathcal{O}}\Big{[}Bel_{i}(A)\log(\hat{Bel}_{i}(A) )+(1-Bel_{i}(A))\log(1-\hat{Bel}_{i}(A))\Big{]},\end{array}$ (5) $$

where $i$ is the index of the training point within a batch of cardinality $b_{size}$,
$A$ is a focal set of classes in the budget $\mathcal{O}$,
$Bel_{i}(A)$ is the $A$-th component of the vector $\mathbf{bel}_{i}$ encoding the ground truth belief values
for the $i$-th training point,
and $\hat{Bel}_{i}(A)$ is the corresponding belief value in the predicted vector $\hat{\mathbf{bel}}_{i}$ for the same training point.
Both $\mathbf{bel}_{i}$ and $\hat{\mathbf{bel}}_{i}$ are vectors of cardinality $|\mathcal{O}|$ for all $i$.

To ensure that the predicted scores amount to a valid belief function, we incorporate
a mass regularization term $M_{r}$ in the loss function that encourages non-negativity (1) of the (predicted) mass values
$\hat{m}(A)$, $A\in\mathcal{O}$, which can be obtained from the predicted belief function $\hat{\mathbf{bel}}$ via the Moebius inversion formula (2).
For it to be valid, the sum of the masses of the predicted belief function must be equal to 1 (1), therefore, we also add a mass sum term $M_{s}$, namely

$$ $\begin{array}[]{l}\displaystyle M_{s}=\max\left(0,\frac{1}{b_{size}}\sum_{i=1} ^{b_{size}}\sum_{A\in\mathcal{O}}\hat{m}_{i}(A)-1\right),\hskip 5.0pt \displaystyle M_{r}=\frac{1}{b_{size}}\sum_{i=1}^{b_{size}}\sum_{A\in\mathcal{ O}}\max(0,-\hat{m}_{i}(A)).\end{array}$ (6) $$

All the components $\mathcal{L}_{BCE}$, $M_{r}$ and $M_{s}$ are computed during batch training.
Two hyperparameters, $\alpha$ and $\beta$, control the relative importance of the two mass terms, yielding as the total loss for RS-NN:

$$ $\mathcal{L}_{RS}=\mathcal{L}_{BCE}+\alpha M_{r}+\beta M_{s}.$ (7) $$

The regularization terms aim to penalize deviations from valid belief functions. When the values of $\alpha$ and $\beta$ are too small, the regularization terms may not be sufficient to ensure that predictions are valid belief functions. This occurs because we do not enforce the model to predict valid belief functions during training; rather, we merely encourage it – just like, e.g., training time regularisation in neurosymbolic learning encourages predictions to be commonsense .
In such cases, (^2^22At any rate, improper belief functions are employed in the literature, e.g. for conditioning)
post-training, we fix all negative masses (obtained from predicted belief functions) to zero and introduce a final subset, often referred to as the ‘universal set’ or ‘full set,’ containing all the classes in the powerset, to the final budget $\mathcal{O}$. This subset is assigned all the remaining mass, ensuring that the sum of masses across all focal sets in $\mathcal{O}$ equals 1. This adjustment compensates for the fact that, during clustering, only a few focal sets of classes $\{A_{1},\ldots,A_{K}\}$ from the power set $\mathbb{P}(\mathcal{C})$ are selected, rather than
considering
any arbitrary set of classes.

### 3.3 Accuracy and uncertainty estimation

Pignistic prediction.
The pignistic probability (3) is the central prediction associated with a belief function (seen as a credal set): standard performance metrics such as accuracy can then be calculated after extracting the most likely class according to the pignistic prediction
(e.g. in §D.5.3).

Entropy of the pignistic prediction.
The Shannon entropy of the pignistic
prediction
$BetP$
can then be used as a measure of the uncertainty associated with the predictions,
in some way analogous to the entropy of a Bayesian mean average prediction:

$$ $H_{RS}=-\sum_{c\in\mathcal{C}}{BetP}(c)\log{BetP}(c).$ (8) $$

A higher entropy value (8) indicates greater uncertainty in the model’s predictions.

Size of the credal prediction.
A sensible measure of the epistemic uncertainty attached to a random-set prediction $\mathbf{\hat{bel}}$ is the size of the corresponding credal set (4).
There are several ways of measuring the size of a convex polytope like a credal set .
Given the upper and lower bounds to the probability assigned to each class $c$ by
distributions within the predicted credal set $\hat{Cre}$,

$$ $\overline{P}(c)=\max_{P\in\hat{Cre}}P(c),\quad\underline{P}(c)=\min_{P\in\hat{ Cre}}P(c),$ (9) $$

we propose a simple way to measure the size of the credal set as the difference between the lower and upper bounds (9) associated with the most likely class according to the pignistic prediction.
For each class $c$, not just $\hat{c}$, the predicted pignistic
estimate $BetP(c)$ falls within the interval $[\underline{P}(c),\overline{P}(c)]$, whose width $\overline{P}(c)-\underline{P}(c)$
indicates the epistemic uncertainty associated with the prediction (§A.4).

## 4 Experiments

### 4.1 Implementation

Experiments, datasets and baselines.
Firstly, we assess the test accuracy (%) and inference time (ms) on multi-class image classification datasets, including MNIST , CIFAR-10 , Intel Image , CIFAR-100 , and ImageNet . Tab. 2 provides a comparison of test accuracies and inference times of RS-NN against state-of-the-art Bayesian methods, R-BNN ,
LB-BNN , Ensemble classifier ENN
and standard CNN as detailed in Sec. 4.2.

Our second set of experiments (Sec. 4.3) concerns out-of-distribution (OoD) detection.
Tab. 1 shows our results on OoD detection metrics AUROC (Area Under Receiver Operating Characteristic curve) and AUPRC (Area Under Precision-Recall curve) for RS-NN, LB-BNN, ENN and standard CNN. We provide quantitative assessment on several iD vs OoD datasets: CIFAR-10 vs SVHN /Intel-Image , MNIST vs F-MNIST /K-MNIST , and ImageNet vs ImageNet-O . We also report the Expected Calibration Error (ECE) for RS-NN, LB-BNN, ENN and standard CNN on datasets CIFAR-10, MNIST and ImageNet (see Tab. 1).

In the third set of experiments (Sec. 4.4), we test the uncertainty estimation capabilities of RS-NN using both the entropy of the pignistic prediction and the size of the predicted credal set. Tab. 1 presents the predicted pignistic entropy of RS-NN alongside entropies of LB-BNN, ENN, and CNN on in-distribution (iD) datasets, namely CIFAR-10, MNIST, ImageNet, and out-of-distribution (OoD) datasets CIFAR-10 vs SVHN/Intel Image, MNIST vs F-MNIST/K-MNIST, and ImageNet vs ImageNet-O.
Tab. 3 shows credal set widths for RS-NN predictions across the same
datasets.

In our final set of experiments, we explore the adaptability of RS-NN (Sec. 4.5) to large-scale architectures, employing it on models such as WideResNet-28-10, VGG16, Inception V3, EfficientNetB2 and ViT-Base. Further, to underscore the
model’s ability to leverage
transfer learning, we conduct training and testing on a pre-trained ResNet50 model initialised with ImageNet weights (Tab. 4).

Additional experiments include
providing statistical guarantees on RS-NN using non-conformity scores (§A),
evaluating the robustness of RS-NN to adversarial attacks (§D.2),
noisy and rotated in-distribution samples in Appendix §D.1, how RS-NN circumvents the overconfidence problem in CNNs is given in §D.3.
Results showing how credal set width is less correlated with confidence scores than entropy are discussed in
in §D.5.1. Ablation studies on $\alpha$ and $\beta$ are shown in §D.5.5 and number of non-singleton focal sets ($K$) in §D.5.6. The ablation study demonstrates that the best accuracy is obtained at small values (0 to 1e-2) of $\alpha$ and $\beta$, and $K=20$ for CIFAR-10 dataset.

Baselines and backbones.
All models, including RS-NN, R-BNN,
LB-BNN, ENN and standard CNN, are trained on ResNet50 (on 3 NVIDIA A100 80GB GPUs) with a learning rate scheduler initialized at 1e-3 with 0.1 decrease at epochs 80, 120, 160, and 180. Standard data augmentation including random horizontal/vertical shifts with a magnitude of 0.1 and horizontal flips are applied to all models.
ResNet50, excluding the top classification layer, serves as the common architecture.
ResNet was originally designed for classification on the ImageNet dataset (1000 classes). To accommodate a reduced number of classes in smaller datasets (e.g., CIFAR-10/MNIST with 10 classes, Intel Image with 7 classes, and CIFAR-100 with 100 classes), two additional dense layers (1024 and 512 neurons, ReLU activation) are added to ResNet50.
Similar techniques are commonly applied in deep learning to adapt model architectures to different datasets ; .
The output layer of RS-NN on ResNet50 has the same number of units as the number of (selected) focal sets $|\mathcal{O}|$, and uses sigmoid activation since the ground truth encoding resembles a multi-label classification problem
(see Sec. 3.2).
For all other models, the final output layer simply consists of a softmax activation for multi-class classification.

Training the RS-NN. We first train a CNN using Adam optimizer on categorical cross-entropy loss over 100 epochs with learning rate scheduler and batch size of 128. The feature vector $f$ is extracted from the penultimate layer of this trained model for clustering. Alternatively, features can be obtained directly from a pre-trained classifier.
In the clustering phase, we use
50 CPU cores (all available),
setting
a budget $K$ of 20 focal sets for CIFAR-10/ MNIST/ Intel Image, 200 for CIFAR-100 and 3000 for ImageNet.
RS-NN is trained from scratch on ground-truth belief encoding of sets using the $\mathcal{L}_{RS}$ loss function (7) over 200 epochs, with a batch size ($b_{size}$) of 128. The data is split into 40000:10000:10000 samples for training, testing, and validation respectively for CIFAR-10 and CIFAR-100, 50000:10000:10000 samples for MNIST, 13934:3000:100 samples for Intel Image, 1172498:50000:108669 for ImageNet. For all OoD datasets, we use 10,000 testing samples except for Intel Image (CIFAR-10 vs Intel Image), we use 3000 samples. All datasets are resized to $224\times 224$.

**Table 1: OoD detection performance and uncertainty estimation for models trained on ResNet50 on CIFAR-10 vs SVHN/Intel Image, MNIST vs F-MNIST/K-MNIST and ImageNet vs ImageNet-O. Evaluation metrics include AUROC/AUPRC (OoD); Entropy of predictions (uncertainty) and Expected Calibration Error (ECE).**
| In-distribution (iD) | Out-of-distribution (OoD) |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Model | Test accuracy<br>(%) ($\uparrow$) | Uncertainty measure | In-distribution<br>Entropy ($\downarrow$) | ECE ($\downarrow$) | SVHN | Intel Image |  |  |  |  |
| AUROC ($\uparrow$) | AUPRC ($\uparrow$) | Entropy ($\uparrow$) | AUROC ($\uparrow$) | AUPRC ($\uparrow$) | Entropy ($\uparrow$) |  |  |  |  |  |  |
| CIFAR-10 | RS-NN | $\mathbf{93.53}$ | Pignistic entropy | $\mathbf{0.088}\pm\mathbf{0.308}$ | $\mathbf{0.0484}$ | $\mathbf{94.91}$ | $\mathbf{93.72}$ | $\mathbf{1.132}\pm\mathbf{0.855}$ | $\mathbf{97.39}$ | $\mathbf{90.27}$ | $\mathbf{1.517}\pm\mathbf{0.740}$ |
| LB-BNN | $89.95$ | Predictive Entropy | $0.191\pm 0.412$ | $0.0585$ | $88.14$ | $81.96$ | $0.828\pm 0.243$ | $82.21$ | $55.17$ | $0.763\pm 0.722$ |  |
| ENN | $91.55$ | Mean Entropy | $0.126\pm 0.323$ | $0.0556$ | $92.76$ | $89.05$ | $0.887\pm 0.514$ | $85.67$ | $58.09$ | $0.600\pm 0.578$ |  |
|  | CNN | $92.08$ | Softmax Entropy | $0.114\pm 0.304$ | $0.0669$ | $93.11$ | $91.0$ | $0.930\pm 0.610$ | $87.75$ | $65.54$ | $0.719\pm 0.673$ |
| MNIST |  |  |  |  |  | F-MNIST | K-MNIST |  |  |  |  |
| RS-NN | $\mathbf{99.71}$ | Pignistic entropy | $0.010\pm 0.111$ | $\mathbf{0.0029}$ | $\mathbf{93.89}$ | $\mathbf{93.98}$ | $\mathbf{0.530}\pm\mathbf{0.770}$ | $\mathbf{96.75}$ | $\mathbf{96.58}$ | $\mathbf{0.740}\pm\mathbf{0.917}$ |  |
| LB-BNN | $99.58$ | Predictive Entropy | $\mathbf{0.001}\pm\mathbf{0.085}$ | $0.0032$ | $89.65$ | $90.36$ | $0.287\pm 0.442$ | $95.61$ | $95.65$ | $0.540\pm 0.621$ |  |
| ENN | $99.07$ | Mean Entropy | $0.022\pm 0.127$ | $0.0039$ | $81.79$ | $82.92$ | $0.313\pm 0.464$ | $95.94$ | $95.45$ | $0.703\pm 0.672$ |  |
| CNN | $98.90$ | Softmax Entropy | $0.023\pm 0.135$ | $0.0052$ | $83.77$ | $84.14$ | $0.278\pm 0.426$ | $94.46$ | $93.94$ | $0.616\pm 0.688$ |  |
| ImageNet |  |  |  |  |  | ImageNet-O |  |  |  |  |  |
|  |  |  |  |  | AUROC | AUPRC | Entropy |  |  |  |  |
| RS-NN | $\mathbf{79.92}$ | Pignistic entropy | $3.153\pm 2.549$ | $\mathbf{0.1416}$ | $\mathbf{60.38}$ | $\mathbf{55.16}$ | $3.911\pm 2.318$ |  |  |  |  |
| LB-BNN | $72.48$ | Predictive Entropy | $2.471\pm 2.972$ | $0.5812$ | $41.08$ | $30.99$ | $1.383\pm 0.028$ |  |  |  |  |
| ENN | $71.82$ | Mean Entropy | $\mathbf{1.395}\pm\mathbf{1.510}$ | $0.5961$ | $54.67$ | $43.73$ | $1.617\pm 1.597$ |  |  |  |  |
| CNN | $78.56$ | Softmax Entropy | $6.386\pm 1.388$ | $0.4004$ | $54.28$ | $48.73$ | $\mathbf{6.575\pm 1.512}$ |  |  |  |  |

**Table 2: Test accuracies (%) and inference time (ms) for uncertainty estimation over 5 consecutive runs across methods and datasets. Average and standard deviation are shown for each experiment.**
| Datasets | MNIST | CIFAR-10 | Intel Image | CIFAR-100 | ImageNet (Top-1) | ImageNet (Top-5) | Inference time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CNN | $98.90\pm 0.04$ | $92.08\pm 0.42$ | $90.89\pm 0.10$ | $65.50\pm 0.08$ | $78.56$ | $94.34$ | $\mathbf{1.91}\pm\mathbf{0.03}$ |
| R-BNN | $90.32\pm 0.14$ | $75.21\pm 0.13$ | $82.12\pm 0.08$ | $50.45\pm 0.02$ | - | - | $62.17\pm 0.76$ |
| LB-BNN | $99.58\pm 0.04$ | $89.95\pm 0.81$ | $90.49\pm 0.42$ | $59.89\pm 1.96$ | $72.48$ | $90.85$ | $7.11\pm 0.89$ |
| ENN | $99.07\pm 0.11$ | $91.55\pm 0.60$ | $91.49\pm 0.19$ | $68.02\pm 0.26$ | $71.82$ | $89.48$ | $3.10\pm 0.03$ |
| RS-NN | $\mathbf{99.71}\pm\mathbf{0.03}$ | $\mathbf{93.53}\pm\mathbf{0.09}$ | $\mathbf{94.22}\pm\mathbf{0.03}$ | $\mathbf{71.61}\pm\mathbf{0.07}$ | $\mathbf{79.92}$ | $\mathbf{94.47}$ | $\mathbf{1.91}\pm\mathbf{0.02}$ |

### 4.2 Comparison with the state-of-the-art on accuracy

Tab. 2
shows RS-NN outperform state-of-the-art Bayesian (R-BNN, LB-BNN) and Ensemble (ENN) methods in terms of test accuracy (%) across all datasets. On ImageNet, RS-NN’s accuracy is remarkably close to CNN (standard ResNet50), highlighting its efficacy on large and complex datasets. It is important to note that the comparison with standard CNNs is not based on their role as uncertainty methods, but as benchmarks, underscoring RS-NN’s effectiveness in achieving high accuracy compared to other uncertainty baselines.
Test accuracy for RS-NN is calculated by determining the class with the highest pignistic probability for each prediction
and
comparing it
with the true class.
As RS-NN leverages a random set framework which does not require prior assumptions and is more data driven, it
can better
adapt and capture the inherent complexity of the data, contributing to its superior performance in test accuracy across various datasets (Tab. 2). The pignistic
predictions
derived from the predicted belief functions
appear
more reliable and accurate than standard predicted probabilities of a CNN.
R-BNN results for ImageNet are not reported in Tab. 2 as evaluating it on large-scale datasets
is impractical due to its complexity.
Experiments on inference time (ms) per sample over 5 runs (a single forward propagation) on CIFAR-10 dataset demonstrates the fastest inference time (ms) for RS-NN among all models (Tab. 2).

### 4.3 Out-of-distribution (OoD) detection

We evaluate out-of-distribution (OoD) detection (Tab. 1) against baselines using AUROC and AUPRC scores. OoD detection identifies data points deviating from the in-distribution (iD) training data by measuring true and false positive rates (AUROC) and precision and recall trade-offs (AUPRC), providing insights into the model’s performance at varying confidence levels
(detailed in §C.3).

RS-NN greatly outperforms Bayesian, Ensemble and standard CNN models in OoD detection (Tab. 1), with significantly higher AUROC and AUPRC scores for all iD vs OoD datasets, especially on CIFAR-10 vs SVHN/Intel Image. This demonstrates its effectiveness in identifying data points deviating from training data.
In real-world scenarios, encountering OoD instances is inevitable. Reliable OoD detection is crucial for safety-critical applications to prevent erroneous critical decisions on unfamiliar or anomalous data. Recognising unfamiliar data signals the model about situations beyond its training, allowing it to acknowledge its own limitations and enhance its awareness of ignorance, achieved effectively by RS-NN.
Tab. 1 also reports the Expected Calibration Error (ECE) (detailed in §C.2). A low ECE indicates that the model’s confidence scores align closely with the actual likelihood of events. RS-NN exhibits the lowest ECE indicating well-calibrated probabilistic predictions.

### 4.4 Uncertainty estimation

Entropy. Tab. 1 shows the mean and variance of the entropy distributions for both iD and OoD datasets. Lower entropy values for iD datasets indicate a confident prediction as the model is trained on familiar data, while higher entropy for OoD datasets reflects the model’s uncertainty with
unseen data. RS-NN exhibits both low entropy for iD datasets and high entropy for OoD ones.
The percentage mean iD/OoD shift in entropy is most pronounced in RS-NN, especially evident in ImageNet, where CNN exhibits high entropy for iD and OoD datasets. RS-NN maintains a desirable iD vs OoD entropy ratio.
This ideal behavior, along with superior OoD detection
(Sec. 4.3), is a good indication that RS-NN is
better at estimating uncertainty induced by domain shift (see §D.5.2).

Credal set width.
Fig. 6 displays the estimated credal set width, the difference between the lower and upper probability bounds in (9), for correctly and incorrectly classified classifications of 1000 CIFAR-10 test samples.
Incorrect predictions (orange) correlate with larger credal set widths, indicating higher epistemic uncertainty, and are dispersed in the middle of the plot. Correct predictions (blue) concentrate at the top, exhibiting smaller credal intervals as expected.
Some instances of correct classification still display a sizable credal set width, signifying elevated uncertainty and low confidence.
Tab. 3 reports the credal set widths of RS-NN predictions for iD vs OoD datasets. Larger
intervals
can be observed for OoD datasets. Note that credal set width
is not directly comparable to the entropy or variance of other models, as such metrics have distinct semantics.

Figure: Table 3: Credal set width for RS-NN on iD vs OoD datasets: CIFAR10 vs SVHN/Intel Image, MNIST vs F-MNIST/K-MNIST and ImageNet vs ImageNet-O.
Refer to caption: x6.png

### 4.5 Adaptability
to large-scale architectures

Tab. 4 shows the adaptability of RS-NN to larger model architectures and its ability to leverage
transfer learning. RS-NN demonstrates higher accuracy compared to standard CNNs across various large-scale model architectures, including WideResNet28-10 (WRN-28-10), VGG16, InceptionV3 (IncV3), EfficientNetB2 (ENetB2), and to Vision Transformer (ViT-Base), showcasing its versatility and ease in adopting different architectures and the generality of the random-set concept.
Notably,
using a pre-trained ResNet50 (Pre-trained R50) model with ImageNet weights, RS-NN achieves higher accuracy on CIFAR-10 compared to using only the architecture (no pre-trained weights).

### 4.6 Limitations

The budgeting step can be time consuming but, for a given dataset, it is a one-time procedure
prior to training. For large datasets, t-SNE and GMM can be applied to representative samples rather than the entire dataset, as the purpose is merely to identify the most relevant sets, reducing computational demands.
To further optimise the method, instead of t-SNE, budgeting can utilize various methods such as autoencoders ;
or PCA for dimensionality reduction. However, despite the initial time investment, the efficiency gained from having a budget applicable to any training scenario is substantial.
Another limitation is manually setting the number of focal elements $K$; a dynamic strategy adjusting $K$ based on overlap would enhance flexibility and effectiveness.
The budgeting procedure can also be adapted to work with multiple sources of data or a stream of data (§D.4).
Alternative methods
such as using sparse mass functions ; , could provide a quantitative basis for determining the subsets $A_{K}$ to consider.

## 5 Conclusion

This paper proposes a novel *Random-Set Neural Network (RS-NN)* for uncertainty estimation in classification using belief functions and random sets.
Random-set representation is a foundational method that acts as a versatile wrapper and extends beyond CNNs to any model architecture and classification task (e.g, text).
To our knowledge,
this concept, along with budgeting for optimal selection of sets, is unprecedented.
RS-NN outperforms state-of-the-art uncertainty estimation models and the standard CNN in performance and out-of-distribution OoD tests,
and scales seamlessly to large-scale architectures and datasets.
Given such results, our approach exhibits significant potential impact for safety-critical applications such as medical diagnostics and autonomous driving, where uncertainty estimation and OoD detection is crucial.
In this paper, we discuss target-level representation of RS-NNs. Future work involves parameter-level representation of RS-NN and the extension of this concept from classification to detection tasks.

## Acknowledgement

This work has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 964505 (E-pi).

## Appendix A Further discussion

### A.1 Rationale and applicability

This paper aims to establish a foundational approach for integrating belief functions into standard deep learning models, specifically CNNs. We chose to closely adhere to a standard CNN framework to ensure the model’s applicability to large-scale, real-world datasets—a capability often lacking in other uncertainty models. As a result, we achieve performance comparable to that of a standard CNN while providing improved and reliable uncertainty estimation.

However, the proposed method can be thought of as a generalisation of traditional classification. There is nothing limiting it to CNNs. In fact, Tab. 4 shows that it can be easily adapted to work with a wide range of network architectures, including Vision Transformers (ViTs). Similarly, it can also work with simple Multi-Layer Perceptrons (MLPs).

### A.2 Statistical guarantees

To complement the extensive discussion in the main paper, we provide here
an in-depth discussion on the statistical guarantees underlying RS-NN and the rationale behind our choices for ground-truth representation $\mathbf{bel}$, loss function and training, under how RS-NN learns sets of outcomes.

Unlike, e.g., conformal learning, our RS-NN approach does not, in its current state, provide statistical guarantees such as the coverage of the prediction intervals. However, we can see at least three ways in which this could be done.

Firstly, RS-NN can be employed as the ‘underlying model’ in an inductive conformal learning framework ([https://cml.rhul.ac.uk/copa2017/presentations/CP_Tutorial_2017.pdf](https://cml.rhul.ac.uk/copa2017/presentations/CP_Tutorial_2017.pdf)), which builds an empirical cumulative distribution of the ‘non-conformity’ scores of a set of calibration samples, and at test time outputs the set of labels whose empirical CDF is above a desired significance level $\epsilon$ (e.g., 95%).
Given a test input $x$ and the associated predictive belief function $\hat{Bel}(c|x)$ (the output of RS-NN), we could, for instance, set as non-conformity score

$$ $s(x,c)\doteq 1-\hat{Pl}(c|x)=\hat{Bel}(\mathcal{C}\setminus\{c\}|x)$ (10) $$

(i.e., a label $c$ is
‘non-conformal’
if its predicted *plausibility*, which is defined as $Pl(A)=1-Bel(\Theta\setminus A)$ and has the semantic of an upper probability bound , is low), and compute predictive regions in the usual way:

$$ $\Gamma(x)=\{c\in\mathcal{C}:p^{c}>\epsilon\},$ $$

where

$$ $\begin{array}[]{lll}p^{c}&=&\displaystyle\frac{|(x_{j},c_{j}):s(x_{j},c_{j})>s (x,c)|}{q+1}+u\cdot\frac{|(x_{j},c_{j}):s(x_{j},c_{j})=s(x,c)|}{q+1},\end{array}$ $$

$(x_{j},c_{j})$ is the $j$-th calibration point, $q$ is the number of calibration points, and
$u\sim\mathcal{U}(0,1)$ (the uniform distribution on the interval $(0,1)$).

Experiments on conformal guarantees. We conducted experiments on CIFAR-10 dataset to provide conformal prediction guarantees with non-conformity scores as given in (10). The goal is to assess the reliability of classification predictions by determining a threshold of non-conformity scores that covers 95% of the data.

To achieve this, non-conformity scores are obtained for all classes, and a threshold is calculated such that it contains 95% of the data. This threshold determines the coverage of the classification, which refers to the proportion of predictions that actually contain the true outcome.

This prediction threshold is determined by finding the percentile corresponding to $1-\alpha$, where $\alpha$ is set to $0.05$ to achieve 95% coverage. Fig. 8(a) shows the non-conformity scores for all the calibration data,
$j=1000$
(using the following split: 40000:10000:9000:1000 training, testing, validation and calibration data points respectively for CIFAR-10), with a cut-off threshold that ensures 95% coverage.

The conformal prediction sets for two sample inputs are shown in Fig. 7, along with their predicted probabilities.

Figure: Figure 7: Conformal prediction sets by RS-NN on the CIFAR-10 dataset. The predicted probabilities for each class is shown.
Refer to caption: x7.png

Figure: (a)
Refer to caption: x8.png

Figs. 8(b) and 8(c) show the coverage and average set size for each class on the test dataset.
Coverage is the proportion of prediction sets that actually contain the true outcome, and average set size is the average number of predicted classes per instance. Higher numbers represent more overlap between the classification regions of different classes.

Summarizing the results, (1) the coverage averages across classes over the desired threshold.
This indicates that the conformal prediction method is providing reliable prediction intervals that contain the true class label with the specified confidence level; (2) The spread or variability in set sizes across different samples provides insights into how well the prediction sets adapt to the difficulty of the samples. Ideally, the prediction sets should dynamically adjust based on the difficulty of each example, with larger sets for more challenging inputs and smaller sets for easier ones.
Fig. 8(c) shows that the average set size for classes 1, 2, 3, 4, and 6 is considerably larger than for the rest, indicating that the model found samples from these as challenging inputs.

Alternative approaches to statistical guarantees.
In the future we plan to explore the possibility of generalising the empirical CDF at the basis of conformal learning in a belief function / random set representation, aiming to retain statistical guarantees. Given the complexity of this enterprise, we are working towards this in a separate paper.

Finally, an intriguing alternative approach (and one that is more achievable in the short term) is the study of confidence intervals in a belief functions representation, such as the one we employ in RS-NNs. In fact, recent studies have been looking at extending the notion of confidence interval ([https://en.wikipedia.org/wiki/Confidence_interval](https://en.wikipedia.org/wiki/Confidence_interval)) to belief functions, under the name of *confidence structures* ([https://hal.science/hal-01576356v3](https://hal.science/hal-01576356v3)), which generalise standard confidence distributions and generate “frequency-calibrated" belief functions.

Liu and Martin, in particular, have developed an Inferential Model (IM) approach which produces
belief functions with well-defined frequentist properties ; .
An alternative approach relies on the notion of “predictive" belief function , which, under repeated sampling, is less committed than the true probability distribution of interest with some prescribed probability.

### A.3 RS-NN Learning Mechanism

Recall from Sec. 3.1 that the ground-truth for a Random Set-CNN is the belief-encoded vector $\mathbf{bel}=\{Bel(A),A\in\mathbb{P}(\mathcal{C})\}$, where $Bel(A)$ is the belief function of a focal set $A$ in the power set $\mathbb{P}$ of classes $\mathcal{C}$. In our method, $Bel(A)$ is 1 if the true class is in subset $A$ and 0 otherwise. Consequently, the belief-encoded ground truth $\mathbf{bel}$ will include multiple occurrences of 1.
For instance, in a digit-classification task such as MNIST, if $\{3\}$ is the true class, the belief-encoded ground truth would contain 1s in all sets where $\{3\}$ is present, such as {1, 3}, {0, 3}, {1, 2, 3}, and so forth. This formulation closely resembles multi-label classification.

Since we assign precise labels for belief containing different focal sets, our model begins by penalizing predictions that differ from the observed label in the same manner, regardless of the set’s composition or relationship to the true label. Using MNIST as an example, this means that the loss incurred for predicting label $\{3\}$ is equivalent to predicting label $\{3,7\}$, as both sets contain the true label. This equivalence in losses might seem counterintuitive at first glance. However, despite the identical loss values, the probabilities output by the sigmoid activation function will vary due to differences in the input logit values for each label. For instance, if the correct label is $\{3\}$, the loss for predicting $\{3\}$ or $\{3,7\}$ would be the same. Still, the loss for predicting $\{7\}$ would differ, allowing the model to discern the set structure during training.

During the training process, the model learns to capture and understand these relationships by observing patterns and dependencies in the training data. As the model optimizes its parameters based on the training objective (e.g., minimizing the loss function), it gradually adjusts its internal representations to better reflect these relationships. We use as the basis for our loss function Binary cross-entropy with sigmoid activation. It allows the model to predict the presence or absence of each label separately as a binary classification problem, producing probabilities between 0 and 1 for each class. While the model is unaware that it is learning for sets of outcomes, we leverage this technique to extract mass functions and pignistic predictions from the learnt belief functions. By re-distributing masses to original classes, we obtain the best pignistic predictions that are often more accurate than standard CNN predictions, attributing to the higher accuracy in Tab. 2.

### A.4 Pignistic probability and credal prediction

Figure: Figure 9: Upper and lower bounds (9), in dotted red, to the probability of the predicted most likely class according to the pignistic prediction (in solid red) for 100 samples of CIFAR-10.
Refer to caption: x11.png

For additional illustration,
the lower and upper bounds (9) to the probability of the top predicted class are plotted in Fig. 9, together with the pignistic probability, for 100 samples of CIFAR-10.
The bounds
(9) can be efficiently computed from the finite number of vertices of the credal prediction (, Fig. 4).

## Appendix B Related Work

Researchers have recently taken some tentative steps to model uncertainty in machine learning and artificial intelligence. Scientists have adopted measures of epistemic uncertainty to refuse or delay making a decision , take actions specifically directed at reducing uncertainty (as in active learning ), or exclude highly-uncertain data at decision making time .

Imprecise probability. Significant work has been done in a credal set setting.
Notably, ; have proposed the Naive Credal Classifier (NCC) as an extension of the naive Bayes classifier to credal sets, where imprecise probabilities are included in models in the form of sets of classes. have presented graphical models which generalise NCCs to multilabel data. Expected utility maximisation algorithms, such as Bayes-optimal prediction , and classification with reject option for risk aversion are also based on set-valued predictions. Classification with partial data has been studied by various authors , while a large number of papers have been published on decision trees in the belief function framework . Significant work in the neural network area was conducted in and .
Classification approaches based on rough sets were proposed in . Ensemble classification is another area in which uncertainty theory has been quite impactful, as the problem is one of combining the results of different classifiers. Important contributions have been made by and . Regression, on the other hand, has only recently been considered by belief theorists ; ; . For tasks like image classification in large datasets, most of these approaches have low performance metrics including training time and test accuracy. The epistemic deep learning approach utilizes standard networks to achieve better performance.

Evidential approaches.
Within a proper epistemic setting, a significant amount of work has been done by Denoeux and co-authors, and Liu et al. , on unsupervised learning and clustering in particular in the belief function framework. Quite a lot of work has been done on ensemble classification in the evidential framework (in particular for neural networks ), decision trees , K-nearest neighbour classifiers , and more recently on evidential deep learning classifiers able to quantify uncertainty . Tong et al. proposes a convolutional neural network based on Dempster-Shafer theory called the evidential deep-classifier employs utility functions from decision theory to assign utilities on mass functions derived from input features to produce set-valued observations. Another recent approach by Sensoy et al. proposes an evidential deep learning classifier to estimate uncertainty in the Dirichlet representation. This work is based on subjective logic and learning to form subjective opinions by minimizing the Kullback-Leibler divergence over a uniform Dirichlet distribution. However, epistemic deep learning does not depend on evidence-based parameters for learning.

Conformal prediction.
Conformal prediction provides a framework for estimating uncertainty by applying a threshold on the error the model can make to produce prediction sets, irrespective of the underlying prediction problem. Different variants of conformal predictors are described in papers by Saunders et al. , Nouretdinov et al. , Proedrou et al. and Papadopoulos et al. . Since the computational inefficiency of conformal predictors posed a problem for their use in neural networks, Inductive Conformal Predictors (ICPs) were proposed by Papadopoulos et al. . Venn Predictors , cross-conformal predictors and Venn-Abers predictors were introduced in distribution-free uncertainty quantification using conformal learning.

## Appendix C Algorithms

### C.1 Algorithm for Budgeting

For RS-NN with $N$ classes, generating $2^{N}$ outputs is computationally infeasible due to exponential complexity. Instead, we choose $K$ relevant subsets ($A_{K}$ focal sets) from the $2^{N}$ possibilities.

To obtain these $K$ focal subsets, we extract feature vectors from the penultimate layer of a trained standard CNN with $N$ outputs. We then apply t-SNE for dimensionality reduction to 3 dimensions. Note that our approach is agnostic, as t-SNE could be replaced with any other dimensionality reduction technique, including autoencoders.

Next, we fit a Gaussian Mixture Model (GMM) to the reduced feature vectors of each class. Using the eigenvectors and eigenvalues of the covariance matrix $\Sigma_{c}$ and the mean vector $\mu_{c}$ for each class $c$, we define an ellipsoid covering 95% of the data. The lengths of the ellipsoid’s principal axes are computed as $length_{c,i}=2\sqrt{7.815\lambda_{i}}$, where $\lambda_{i}$ is the $i^{th}$ eigenvalue. The scalar 7.815 corresponds to a 95The class ellipsoids are plotted in a 3D space and the overlap of each subset in the power set of $N$ is computed. As it is computationally infeasible to compute overlap for all $2^{N}$ subsets, we start doing so from cardinality 2
and use early stopping when increasing the cardinality further does not alter the list of most-overlapping sets of classes.
We choose the top-$K$ subsets ($A_{K}$) with the highest overlapping ratio, computed as the intersection over union for each subset.

Figure: Algorithm 1 Budgeting Algorithm

### C.2 Algorithm for ECE

Figure: Algorithm 2 Expected Calibration Error (ECE)

Expected Calibration Error (ECE) is computed by comparing the average confidence and accuracy within each bin. The steps involved are as follows:

- 1.
For each bin, calculate the absolute difference between the average confidence and the accuracy.
- 2.
Weight each difference by the proportion of instances in that bin compared to the total number of instances.
- 3.
Sum up these weighted differences across all bins to get the final ECE value.

The formula for ECE is given by:

$$ $\text{ECE}=\sum_{i=1}^{B}|(\text{Accuracy}_{i}-\text{Confidence}_{i})|\times \text{Weight}_{i}$ $$

where:

- •
$\text{Accuracy}_{i}$ is the accuracy within the $i$-th bin.
- •
$\text{Confidence}_{i}$ is the average confidence within the $i$-th bin.
- •
$\text{Weight}_{i}$ is the proportion of instances in the $i$-th bin compared to the total number of instances.
- •
$B$ is the total number of bins.

The final ECE value represents how much the model’s estimated probabilities differ from the true (observed) probabilities. A low ECE indicates well-calibrated probabilistic predictions, demonstrating that the model’s confidence scores align closely with the actual likelihood of events. RS-NN exhibits the lowest ECE as shown in Tab 1.

### C.3 Algorithm for AUROC, AUPRC

Figure: Figure 10: Receiver Operating Characteristic (ROC) and Precision-Recall Characteristic (PRC) curves for RS-NN, LB-BNN, ENN, and CNN evaluated on the SVHN and Intel Image OoD datasets for CIFAR-10.
Refer to caption: x12.png

Out-of-distribution (OoD) detection involves the identification of data points that deviate from the in-distribution (iD) data on which a model was trained. This process relies on assessing the model’s uncertainty, particularly its epistemic uncertainty, which reflects the model’s lack of knowledge or confidence in making predictions. When exposed to OoD data, which differs significantly from the training data, the model tends to exhibit higher epistemic uncertainty.

AUROC (Area Under the Receiver Operating Characteristic Curve) and AUPRC (Area Under the Precision-Recall Curve) are evaluation metrics commonly used to assess the performance of binary classification models, providing insights into their ability to distinguish between positive and negative instances. In the context of evaluating uncertainty estimation in machine learning models, these metrics quantify how well the model separates iD samples from OoD samples. AUROC is derived from the Receiver Operating Characteristic (ROC) curve, which illustrates the trade-off between True Positive Rate (TPR) and False Positive Rate (FPR) across various classification thresholds. The AUROC value represents the area under this curve and ranges from 0 to 1, with higher values indicating better discriminative performance. AUPRC is based on the Precision-Recall curve, which plots Precision against Recall at different classification thresholds. Precision measures the accuracy of positive predictions, while Recall quantifies the ability to capture all positive instances. AUPRC calculates the area under this curve and provides a complementary perspective, particularly valuable when dealing with imbalanced datasets.

Figure: Algorithm 3 Algorithm for AUROC, AUPRC

Figs. 11 and 11 plot each model’s performance, illustrating the trade-offs between True Positive Rate and False Positive Rate (ROC curve), Precision and Recall (PRC curve) for CIFAR-10 (Fig. 11) and MNIST (Fig. 11). The left two plots depict the AUROC curves, where the blue curve representing RS-NN outperforms others, indicating superior discrimination between true positive and false positive rates. Similarly, on the two right plots displaying Precision-Recall Curve (PRC), RS-NN exhibits the highest curve, emphasizing its precision and recall performance. These results showcase RS-NN’s effectiveness in distinguishing in-distribution and out-of-distribution samples.

## Appendix D Additional Experimental results

### D.1 Robustness to noisy data

We split the MNIST data into training and test sets, train the RS-NN model using the training test, and test the model on noisy and rotated out-of-distribution test data. This is done by adding random noise to the test set of images to obtain noisy data and rotating MNIST images with random degrees of rotation between 0 and 360.

**Table 5: Test accuracies(%) for RS-NN, standard CNN, LB-BNN (Bayesian) and ENN (Ensemble) on noisy samples of MNIST . ‘Scale’ represents the standard deviation of the normal distribution from which random numbers are being generated for random noise.**
| Noise (scale) | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Standard CNN | 93.40% | 79.08% | 79.15% | 58.33% | 40.19% | 28.15% | 28.59% |
| RS-NN | 96.90% | 85.36% | 85.91 % | 68.33 % | 51.46 % | 37.18 % | 38.05 % |
| LB-BNN | 98.47% | 95.26 % | 80.18% | 61.56% | 43.28% | 31.41% | 24.55 |
| ENN | 97.81% | 90.76% | 75.41% | 58.99% | 45.70% | 36.97% | 31.51 |

Tab. 5 shows the test accuracies for standard CNN, RS-NN, LB-BNN and ENN at different scales of random noise. RS-NN shows significantly higher test accuracy than other models as the amount of noise added to the test data increases.

The test accuracies for standard CNN, RS-NN, LB-BNN and ENN on Rotated MNIST images are shown in Tab 6. The samples are randomly rotated every 60 degrees, -180^∘ to -120^∘, -120^∘ to -60^∘, -60^∘ to 0^∘, etc. A fully random rotation between 0^∘ and 360^∘ also shows higher test accuracy for RS-NN at $47.71\%$ when compared to standard CNN with test accuracy $45.86\%$.

**Table 6: Test accuracies(%) for RS-NN, standard CNN, LB-BNN (Bayesian) and ENN (Ensemble) on Rotated MNIST out-of-distribution (OoD) samples. Rotation angle is random between the values given.**
| Rotation (angle) | -180/-120 | -120/-60 | -60/0 | 0/60 | 60/120 | 120/180 | 0/360 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Standard CNN | 36.41% | 21.86% | 74.41% | 80.80% | 23.89% | 37.53% | 45.86% |
| RS-NN | 37.84% | 23.54% | 78.44% | 81.46% | 26.31% | 38.48% | 47.71% |
| LB-BNN | 37.56% | 20.19% | 75.12% | 77.67% | 23.18% | 37.83% | 46.07 |
| ENN | 36.40% | 19.43% | 71.70% | 78.59% | 18.70% | 36.38% | 44.14 |

**Table 7: Standard CNN fails to perform well when tested on noisy noisy and rotated samples of MNIST test data. The predicted results show how a standard CNN predicts the wrong class with a high confidence score, whereas the RS-NN model predicts the right class with varying confidence scores.**
|  | Standard CNN Predictions | Belief RS-NN Predictions |
| --- | --- | --- |
| True Label = 2 |  |  |
|  | Class 0<br>0.628<br>Class 2<br>0.232<br>Class 3<br>0.117<br>Class 8<br>0.015 | Belief values<br>{‘2’}<br>0.985<br>{‘2’, ‘0’, ‘1’}<br>0.981<br>{‘2’, ‘1’}<br>0.980<br>{‘2’, ‘4’}<br>0.979<br>Mass values<br>{‘2’}<br>0.982<br>{‘0’, ‘8’}<br>0.009<br>{‘7’, ‘0’, ‘1’}<br>0.002<br>{‘7’, ‘8’}<br>0.002<br>Pignistic<br>2<br>0.982<br>8<br>0.007<br>0<br>0.100<br>3<br>0.004 |
| True Label = 3 |  |  |
|  | Class 8<br>0.969<br>Class 5<br>0.018<br>Class 9<br>0.006<br>Class 2<br>0.003 | Belief values<br>{‘3’, ‘5’}<br>0.772<br>{‘6’, ‘3’}<br>0.637<br>{‘6’, ‘3’, ‘5’}<br>0.620<br>{‘3’}<br>0.540<br>Mass values<br>{‘3’}<br>0.361<br>{‘7’, ‘8’}<br>0.134<br>{‘0’, ‘8’}<br>0.104<br>{‘8’}<br>0.084<br>Pignistic<br>3<br>0.427<br>8<br>0.205<br>5<br>0.115<br>7<br>0.080 |
| True Label = 1 |  |  |
|  | Class 2<br>1.0<br>Class 1<br>3.39e-08<br>Class 6<br>1.03e-10<br>Class 3<br>4.92e-11 | Belief values<br>{‘2’, ‘1’}<br>0.999<br>{‘1’, ‘9’}<br>0.958<br>{‘1’}<br>0.924<br>{‘1’, ‘5’}<br>0.825<br>Mass values<br>{‘1’}<br>0.556<br>{‘2’}<br>0.423<br>{‘1’, ‘9’}<br>0.020<br>{‘6’}<br>4.31e-05<br>Pignistic<br>1<br>0.566<br>2<br>0.423<br>9<br>0.010<br>6<br>4.31e-05 |
| True Label = 9 |  |  |
|  | Class 5<br>0.988<br>Class 2<br>0.010<br>Class 3<br>0.0004<br>Class 7<br>0.0002 | Belief values<br>{‘7’, ‘5’, ‘9’}<br>0.831<br>{‘7’, ‘9’}<br>0.706<br>{‘5’, ‘9’}<br>0.576<br>{‘3’, ‘9’}<br>0.558<br>Mass values<br>{‘7’, ‘9’}<br>0.171<br>{‘3’}<br>0.134<br>{‘9’}<br>0.132<br>{‘7’, ‘8’}<br>0.102<br>Pignistic<br>9<br>0.699<br>7<br>0.023<br>5<br>0.004<br>3<br>0.001 |

Tab 7 shows predictions for noisy and rotated noisy MNIST samples. In cases where a standard CNN makes wrong predictions with high confidence scores, Random-Set NN manages to predict the correct class with varying confidences verifying that the model is not overcofident in uncertain cases. For example, a noisy sample with true class ‘3’ has a standard CNN prediction of class ‘8’ with $96.9\%$ confidence, while RS-NN predicts the correct class {‘3’} with $42.7\%$ confidence. Similarly, for rotated ‘9’, the standard CNN predicts class ‘5’ with $98.8\%$ confidence whereas RS-NN predicts the correct class {‘9’} with $69.9\%$ confidence. For a rotated ‘1’, the standard CNN predicts class ‘2’ with $100\%$ confidence.

### D.2 Robustness to adversarial attacks

We conducted experiments on adversarial attacks using the well-known Fast Gradient Sign Method (FGSM) . FGSM is a popular approach for generating adversarial examples by perturbing input images based on the sign of the gradient of the loss function with respect to the input. Mathematically, the perturbed image $x^{\prime}$ is computed as

$$ $x^{\prime}=x+\epsilon\cdot\text{sign}(\nabla_{x}J(\theta,x,y)),$ (11) $$

where $x$ is the original input image, $\epsilon$ (epsilon) is a small scalar representing the magnitude of the perturbation, $J$ is the loss function, $\theta$ are the model parameters, and $y$ is the true label of the input image.

Figure: Figure 12: Examples of perturbed images generated using FGSM adversarial attacks on the MNIST dataset for different epsilon values. Epsilon values range from 0 to 0.05.
Refer to caption: x18.png

In our experiments, we applied FGSM adversarial attacks on the MNIST dataset for both standard CNN and Random-Set NN (RS-NN) models. This involved perturbing images from the MNIST dataset with noise generated using FGSM based on the gradient of the loss function (namely, the cross entropy loss for CNN and the $\mathcal{L}_{B-RS}$ loss, see Sec. 3.2, (7), for RS-NN).

**Table 8: Test accuracies of CNN and RS-NN models under FGSM adversarial attacks on the MNIST dataset for different epsilon values. Epsilon values range from 0 to 0.05.**
|  | Model | Epsilon ($\epsilon$) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 0 | 0.005 | 0.01 | 0.02 | 0.03 | 0.04 | 0.05 |  |
| Test acc. (%) | CNN | 98.90 | 98.25 | 94.82 | 72.62 | 49.75 | 38.08 | 32.43 |
| RS-NN | 99.17 | 98.46 | 95.84 | 91.90 | 90.62 | 90.10 | 89.72 |  |

Subsequently, we evaluated the models’ predictions on these perturbed images and computed their test accuracies. Tab. 1 in the anonymous PDF linked to this response presents the test accuracies of both CNN and RS-NN models under FGSM adversarial attacks on the MNIST dataset for different values of $\epsilon$. As can be observed there, RS-NN demonstrates significantly higher robustness to these adversarial attacks compared to CNN, showcasing its superior resilience to adversarial perturbations. For additional information, Fig. 1 in the PDF shows examples of perturbed images generated using FSGM adversarial attacks applied to the MNIST dataset for various epsilon values ranging from 0 to 0.05.

### D.3 Overconfidence in CNNs

Tab 9 shows an noisy example of a rotated MNIST image with a true label ‘$8$’. The standard CNN makes an incorrect prediction (‘$6$’) with $99.95\%$ confidence and a low entropy of $0.0061$, showcasing a noTab limitation in relying solely on confidence scores and entropy of predicted probabilities.
RS-NN provides a correct prediction (class ‘$8$’) with a confidence of $49.7\%$ and pignistic entropy of $2.1626$ and a credal set width of $0.482$. Crucially, the RS-NN’s prediction process considers more than just the predicted class, taking into account both the confidence of the pignistic probability and the entropy. In this example, despite predicting the correct class (‘8’), the RS-NN demonstrates a low confidence (49.7%) and high entropy (2.1626). This holistic approach provides a more nuanced and reliable understanding of the model’s uncertainty,

Figure: Figure 13: Confidence scores of RS-NN on CIFAR-10, Noisy CIFAR-10, and Rotated CIFAR-10
Refer to caption: x24.png

Figure: Figure 14: Entropy distribution of RS-NN on CIFAR-10, Noisy CIFAR-10, and Rotated CIFAR-10
Refer to caption: x25.png

Figure: Figure 15: Confidence scores for Incorrectly Classified samples of RS-NN and CNN
Refer to caption: x26.png

**Table 9: Standard CNN fails to perform well when tested on an noisy (rotated) sample of MNIST test data. The predicted results show how a standard CNN predicts the wrong class with a high confidence score ($99.95\%$), while the RS-NN model predicts the correct class with $49.7\%$ confidence and a high entropy of $2.1626$.**
| True Label = "8" | CNN Predictions | RS-NN Predictions |
| --- | --- | --- |
|  | Class 6<br>0.9995<br>Class 5<br>0.0002<br>Class 8<br>0.0001<br>Class 0<br>1.4e-05<br>Class 4<br>1.1e-06 | Belief values<br>{’6’, ’4’, ’8’}<br>0.7853<br>{’6’, ’8’, ’1’}<br>0.7150<br>{’6’, ’8’}<br>0.6357<br>{’9’, ’8’}<br>0.5529<br>{’8’, ’3’}<br>0.4092<br>Mass values<br>{’6’, ’8’}<br>0.2079<br>{’9’, ’8’, ’1’}<br>0.1999<br>{’8’}<br>0.1959<br>{’8’, ’3’}<br>0.0961<br>{’6’}<br>0.05147<br>Pignistic Probability<br>8<br>0.4978<br>6<br>0.2294<br>9<br>0.1002<br>3<br>0.0502<br>4<br>0.0459 |
| Entropy 0.0061 | Entropy 2.1626<br>Credal set width 0.582 |  |

A distribution of confidence scores for RS-NN in the same noisy experiment are shown in Fig. 13 for both incorrectly classified and correctly classified samples. In Fig. 15, we show how a standard CNN has high confidence for incorrectly classified samples whereas RS-NN exhibits lower confidence and higher entropy (Fig. 14) for incorrectly classified samples. Fig. 14 displays the entropy distribution for correctly and incorrectly classified samples of CIFAR-10, including in-distribution, noisy, and rotated images. Higher entropy is observed for incorrectly classified predictions across all three cases.

### D.4 Budgeting of sets

The budgeting procedure can also be modified to accommodate multiple data sources or a continuous stream of data.
t-SNE can be replaced by an autoencoder (this will need to be trained first) or PCA to add continual inference capabilities in the dimensionality reduction step. Similarly, Gaussian-based dynamic probabilistic clustering (GDPC) can be used instead of GMM clustering to handle a continuous stream of data. GDPC works by first initialising a GMM and then updating its parameters as it sees new data. In alternative, have proposed an interesting approach to learn a GMM from multiple sources of data by maintaining a local and a global mixture model. Both these approaches can be plugged into our budgeting framework
with little modifications
to add continual learning capabilities to it.
The
computation of cluster overlaps
remains the same, so overlapping scores and the resultant focal sets
will need to
be updated as the clusters evolve. However, a cluster tracking strategy can be employed so that the overlap assessment step is only re-done when a sufficient drift has been detected in the clusters. All the proposed changes are efficient enough to not have a significant effect on the overall time of budgeting.

Figure: Figure 16: 2D visualization of the clusters of 10 classes of CIFAR-10 dataset and the ellipses formed by RS-NN based on the hyperparameters of Gaussian Mixture Models.
Refer to caption: x28.png

A 2D visualization of the ellipses formed by calculating the principle axes and their lengths using eigenvectors and eigenvalues obtained from GMMs, for each class $c$ is shown in Fig. 16.

### D.5 Uncertainty estimation for RS-NN

#### D.5.1 Entropy vs Credal Set Width

Figs. 17(a) and 17(b) depict the relationship between the entropy of RS-NN pignistic prediction and the associated confidence level (Fig. 17(a)), and between credal set width and confidence (Fig. 17(b)) for the CIFAR-10 (left) and SVHN/Intel Image (right) datasets. The distribution of iD predictions (left) for both
tests
show a concentration at the top left, indicating high confidence and low entropy or credal set width. Conversely, OoD predictions (middle, right) exhibit a more dispersed pattern.

Figure: (a) Entropy vs Confidence score on iD (left) vs OoD (right) datasets. For CIFAR-10, most predictions are concentrated top left of the plot indicating lower entropy and higher confidence in the predictions. For SVHN and Intel Image datasets, predictions are more distributed.
Refer to caption: x29.png

Entropy, reflecting prediction uncertainty,
is quite correlated with confidence, in both iD and OoD tests.
In contrast, as it considers the entire set of plausible outcomes within a belief function rather than a single prediction,
credal set width better quantifies the degree of epistemic uncertainty inherent to a prediction and,
As a result, credal set width is less dependent on the concentration of predictions and is more reflective of the overall uncertainty encompassed by the model.
Fig. 17(b) shows that,
unlike entropy, credal with
is clearly not correlated with confidence (Fig. 17(b)).

#### D.5.2 Qualitative results of entropy

Figure: Figure 18: Uncertainty (entropy) of MNIST digit ’$3$’ rotated between -90 and 90 degrees. The blue line indicates RS-NN estimates low uncertainty between -30 to 30, and high uncertainty for further rotations.
Refer to caption: x31.png

Fig. 18 depicts entropy-based uncertainty estimates for rotated out-of-distribution (OoD) MNIST digit ‘$3$’ samples. All models accurately predict the true class at $-30^{\circ}$, $0^{\circ}$, and $30^{\circ}$. RS-NN and LB-BNN consistently exhibit low entropy in these scenarios, while ENN shows higher entropy for correct classifications at these angles. As the rotation angle increases, indicating more challenging scenarios, all models predict the wrong class. Notably, LB-BNN fails to exhibit a significant increase in entropy for these incorrect predictions. ENN performs relatively better than RS-NN at $60^{\circ}$ and $90^{\circ}$ rotations but has previously demonstrated high entropy levels even for accurate predictions at relatively minor rotations of $-30^{\circ}$ and $+30^{\circ}$ degrees. Overall, RS-NN provides a more reliable measure of uncertainty.

Figure: Figure 19: Entropy Distributions for RS-NN on MNIST vs Fashion-MNIST/Kuzushiji-MNIST.
Refer to caption: x32.png

Figs. 20 and 20 show the entropy distributions for RS-NN on MNIST (iD) vs Fashion-MNIST (OoD)/Kuzushiji-MNIST (OoD), and CIFAR-10 (iD) vs SVHN (OoD)/Intel-Image (OoD), respectively. There is a clear iD vs OoD shift in entropy for RS-NN, as detailed in Sec. 4.4.

#### D.5.3 Entropy of pignistic predictions

**Table 10: The predicted belief, mass values, pignistic probabilities, and entropy for two CIFAR-10 predictions. The figure on the top (True Label = "horse") is a certain prediction with 99.9% confidence and a low entropy of 0.0017, whereas the figure on the bottom (True Label = "cat") is an uncertain prediction with 33.3% confidence and a higher entropy of 2.6955.**
| Sample | Belief | Mass | Pignistic | Entropy |
| --- | --- | --- | --- | --- |
|  | {‘horse’, ‘bird’}<br>0.9999402<br>{‘horse’, ‘dog’}<br>0.9999225<br>{‘horse’}<br>0.9999175<br>{‘horse’, ‘deer’}<br>0.9998697<br>{‘cat’, ‘truck’}<br>7.0380207e-05 | {‘horse’}<br>0.9999175<br>{‘cat’, ‘truck’}<br>6.859753e-05<br>{‘ship’, ‘bird’}<br>4.094290e-05<br>{‘horse’, ‘bird’}<br>2.250525e-05<br>{‘dog’}<br>1.717869e-05 | horse<br>0.9998833<br>truck<br>3.5826409e-05<br>cat<br>3.3859180e-05<br>bird<br>3.3015738e-05<br>ship<br>2.3060647e-05 | 0.0017040148 |
|  | {‘deer’, ‘cat’}<br>0.4787728<br>{‘deer’, ‘airplane’, ‘bird’}<br>0.4126398<br>{‘horse’, ‘deer’}<br>0.3732957<br>{‘deer’, ‘bird’}<br>0.3658997<br>{‘deer’, ‘dog’}<br>0.3651531 | {‘deer’}<br>0.3104962<br>{‘cat’, ‘truck’}<br>0.1762222<br>{‘dog’, ‘bird’}<br>0.0998060<br>{‘horse’, ‘bird’}<br>0.0954350<br>{‘bird’}<br>0.0524873 | deer<br>0.3332411<br>cat<br>0.2230723<br>horse<br>0.1153417<br>bird<br>0.1086245<br>dog<br>0.1039505 | 2.6955228261 |

Tab 10 shows sample predictions by RS-NN: belief functions, masses, and pignistic predictions for given samples of CIFAR-10. The predicted belief, mass values, pignistic probabilities, and entropy are illustrated for two CIFAR-10 predictions. In the top figure, corresponding to the true label "horse," the model makes a highly confident prediction with 99.9% confidence and a low entropy of 0.0017. Conversely, the bottom figure, associated with the true label "cat," represents an uncertain prediction with 33.3% confidence and a higher entropy of 2.6955. It’s important to note that the second image is slightly unclear and poor in quality.

Figure: Figure 21: Entropy, predicted class and belief value graph for two samples of CIFAR-10 dataset.
Refer to caption: x36.png

Fig. 21 shows the entropy for two samples of CIFAR-10, one is a slightly uncertain image, whereas the other is certain with high belief values and lower entropy. Both results are shown for $K=20$ classes, additional to the 10 singletons.

#### D.5.4 Credal set width

Figure: Figure 22: Credal set widths for individual classes of CIFAR-10 dataset. For Incorrectly Classified samples (top). For Correctly Classified samples (bottom).
Refer to caption: x37.png

Fig. 22 plots the credal set widths for incorrectly classified (left) and correctly classified (right) samples over each class $c$ in CIFAR-10, respectively. The box represents the interquartile range between the 25th and 75th percentiles of the data encompassing most of the data. The vertical line inside the box represents the median and the whiskers extend from the box to the minimum and maximum values within a certain range, and data points beyond this range are considered outliers and are plotted individually as dots or circles. The box plots here are shown for 10,000 samples of CIFAR-10 dataset where most samples within the incorrect classifications have higher credal set widths indicating higher uncertainty in these samples, especially for classes ‘bird’, ‘cat’, ‘deer’, and ‘dog’.

#### D.5.5 Ablation study on hyperparameters

As we discussed in the main paper, the regularization serves the purpose of ensuring that our model generates valid belief functions, which are constrained by maintaining a sum of masses equal to 1 and ensuring non-negativity. These terms are designed to penalize deviations from valid belief functions. However, it is crucial not to assign too much weight to this term, as excessively penalizing deviations may hinder the model’s ability to accurately classify data points. For example, in a Variational Auto Encoder (VAE), if we assign too much weight to the KL divergence term in the loss, the model may prioritize fitting the latent distribution at the expense of reconstructing the input data accurately. This imbalance can lead to poor reconstruction quality and suboptimal performance on downstream tasks.

Hence, it is essential to properly tune the values of $\alpha$ and $\beta$ to ensure that these regularization terms play a meaningful role in the training. The ablation study on hyperparameters is conducted to examine the impact of $\alpha$ and $\beta$ on the model’s accuracy. It demonstrates that small values suffice for these parameters.

In Fig. 23, we show the test accuracies for different values of hyperparameters $\alpha$ and $\beta$ in $\mathcal{L}_{B-RS}$ loss function (7). Hyperparameters $\alpha$ and $\beta$ adjusts the relative significance of the regularization terms $M_{r}$ and $M_{s}$ respectively in $\mathcal{L}_{B-RS}$ loss. The test accuracies are calculated for CIFAR-10 dataset with varying $\alpha$ (blue) and $\beta$ (red) values, $\alpha/\beta$ = [0.001, 0.005, 0.006, 0.009, 0.01, 0.015, 0.020, 0.025, 0.03, 0.04, 0.05]. Test accuracy is the highest when $\alpha$ and $\beta$ equals $0.5$.

In our experiments across various datasets and architectures, we found that a value of 1e-3 for the hyperparameters yields satisfactory results. This includes architectures ranging from ResNet-50 to Vision Transformers (ViT-Base), and datasets ranging from MNIST ($\approx$ 60,000 images of 10 classes) to ImageNet ($\approx$ 1.1M images of 1000 classes). While conducting a parameter search for each dataset could potentially lead to further optimization, it’s worth noting that, in many cases, this step can be omitted without sacrificing performance significantly.
For further optimization one can, of course, perform a tailored parameter search per dataset, but this is not quite necessary.

Figure: Figure 23: Test accuracies of RS-NN on different values of hyperparameters $\alpha$ and $\beta$ in the loss function.
Refer to caption: x39.png

#### D.5.6 Ablation Study on the number of focal sets

Figure: Figure 24: Ablation study on number of non-singleton focal sets $K$ on CIFAR-10 dataset using ViT-Base. The maximum value of K can be 1013 for 10 classes (after excluding the singletons and empty set).
Refer to caption: x40.png

The number of non-singleton focal sets $K$ to be budgeted is a hyperparameter and needs to be studied. A smaller value of $K$ can lead to more similar results to classical classification while a larger value of $K$ can increase the complexity. Therefore, we conducted an ablation study on the $K$ on the CIFAR-10 dataset. It was found that a small value of $K$ (comparable to the number of classes) works best and even performs better than $K=0$ (when there are no non-singleton focal sets) (see Fig. 24). Hence, using set prediction along with the proposed budgeting algorithm not only helps induce uncertainty quantification but also improves the performance of the model. Note that performance comparison was made in terms of accuracy. Smet’s Pignistic Transform was used to compute class-wise probabilities from the predicted belief function. Fig. 24 shows an ablation study on different number of focal sets $K$ for CIFAR-10 on the Vision-Transformer (ViT-Base) model.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Please clarify the validity of belief functions in Eq. 7.
2. Please discuss and compare with some recent uncertainty works in the experiments.
soundness: 3
strengths: 1. The paper is well-written and well-organized.
2. The issue of uncertainty in neural networks is crucial. The proposed method can be easily extended to other baselines, which could benefit the model trustworthiness community.
3. The proposed method is much more efficient than other baselines, making the proposed method more accessible and feasible for adoption.
4. The evaluation is sufficient. The effectiveness of the proposed method is validated on various models and datasets.
summary: This paper focuses on a novel approach called Random-Set Neural Network (RS-NN) for classification tasks. The RS-NN approach aims to predict belief functions over a class list using random sets, which can represent uncertainty induced by limited training data. RS-NN can be presented as a wrapper technique that can enhance existing baseline models by predicting beliefs for sets of classes. Specifically, it efficiently selects relevant sets of classes using GMMs and introduces measures for assessing uncertainty in random-set predictions. In the experiments, RS-NN is compared with Bayesian and Ensemble methods, demonstrating better performance in uncertainty estimation on multiple benchmarks.
weaknesses: 1. One concern lies in the formulation of Eq. 7, where the validity of belief functions is achieved by the involvement of loss terms $M_s$ and $M_r$. However, according to the ablation studies in Figure 26, a large hyperparameter of $\alpha$ and $\beta$ could lead to a decrease in accuracy, which makes it a trade-off between the accuracy and validity of belief functions. How to balance this trade-off is not well-explained in this paper. Although the authors mentioned that any negative masses will be set to zero in post-training, the gap caused by this process is not discussed or explained.
2. Another concern lies in the comparison. Most recent model calibration works that focus on uncertainty have not been discussed or compared in this paper, such as [a,b,c,d]. Furthermore, the evaluation of adversarial robustness could be weak. It is necessary to conduct some evaluation on CIFAR-10 under a stronger PGD attack to demonstrate the robustness.

[a]. Calibrating Deep Neural Networks using Focal Loss. NeurIPS 2020.

[b]. Improving model calibration with accuracy versus uncertainty optimization. NeurIPS 2020.

[c]. Local Calibration: Metrics and Recalibration. UAI 2022.

[d]. Dual focal loss for calibration. ICML 2023.

---

confidence: 4
contribution: 4
initial_rating: 5
presentation: 4
questions: I have compared the reported results for LB-BNN, FSVI, and Deep Ensemble with their sources and was surprised to find large differences in the reported metrics compared to the results reported in the manuscript in Table 2 (OoD detection performance and uncertainty estimation).

While the manuscript claims "Note that the results in the FSVI paper (Rudner et al., 2022) are based on a pre-trained ResNet18", having read the FSVI paper and supplementary, I was not able to confirm this. See also Appendix D.3 of FSVI[1]:

> In this experiment, a standard ResNet-18 network architecture was used. All models are trained for 200 epochs with a mini-batch size of 128 using SGD with a learning rate of 5 × 10−3, momentum (with momentum parameter 0.9), and a cosine learning rate schedule with parameter 0.05

The reported metrics use ResNet-50 everywhere, while LB-BNN, FSVI, and Deep Ensemble use a ResNet-18 on CIFAR-10. Using ResNet-50 (25.6M parameters) should in principle only improve the results as we increase model capacity, compared to ResNet-18 (11.7M parameters).

**CIFAR-10**

| Method                      | Model                                             | Test accuracy (%) | ECE $\downarrow$ | AUROC (SVHN) $\uparrow$ |
|-----------------------------|---------------------------------------------------|:-------------------:|:------------------:|:-------------------------:|
| RS-NN                       | ResNet-50                                         | 93.53             | 0.0484           | 94.91                   |
| LB-BNN (reported)           | ResNet-50                                         | 89.95             | 0.0585           | 88.14                   |
| LB-BNN (source)             | ResNet-18                                         | **95.40**         | 0.3090           | 96.50                   |
| FSVI (reported)             | ResNet-50                                         | 80.29             | 0.0521           | 80.59                   |
| FSVI (source)               | ResNet-18 ($p_{x_{C}} = \text{rand. monochrome}$) | 93.35             | 0.0340           | 94.76                   |
| FSVI (source)               | ResNet-18 ($p_{x_{C}} = \text{CIFAR-100}$)        | 93.57             | 0.0260           | 98.07                   |
| FSVI Ensemble (source)      | ResNet-18 ($p_{x_{C}} = \text{rand. monochrome}$) | 95.19             | **0.0130**       | **99.19**               |
| Deep Ensemble (reported)    | ResNet-50                                         | 92.73             | 0.0482           | 93.84                   |
| Deep Ensemble (FSVI source) | ResNet-18                                         | 95.13             | 0.0190           | 98.04                   |

I appreciate, that the authors took the time and effort to run the compared methods themselves, improving comparability in principle. Nevertheless, it surprises me, that the original (highlighted as *source* in the table above) results for LB-BNN, FSVI, and DE all *outperform* RS-NN in test accuracy, ECE, and AUROC of OoD detection by a significant margin. I have focused on the CIFAR-10 results, as this was a dataset with SVHN as OoD dataset used in all three sources.

Could it be, that the compared methods are, as always, are sensitive to hyperparameters, and the authors probably have unintentionally picked suboptimal hyperparameters for those methods by accident? Or am I missing some important detail?

While I generally really like the manuscript and experimental evaluation, the above leaves me questioning the validity of the comparisons and the claims of "beating state-of-the-art methods" in all experiments. I usually do not particularly care about beating the state-of-the-art, but if a method happens to be misrepresented and suddenly goes from being the best method to being the worst method in the comparison, I get interested as to why that is (as I said, I'm probably just missing an important detail -- I'm not implying that the authors did misrepresent anything on purpose). I will happily raise my score to 6 or 8 if the authors can clarify these discrepancies.

[1] Tractable Function-Space Variational Inference in Bayesian Neural Networks; Tim G. J. Rudner, Zonghao Chen, Yee Whye Teh, Yarin Gal. https://arxiv.org/pdf/2312.17199

**Misc. Questions**:
- L80: "the entropy of a pignistic prediction is more diverse than that of a Bayesian prediction across in-distribution and out-of-distribution data (see Tab. 2)" -> What does "more diverse" mean?
- E.4, Tab. 13: Why was the influence of $\varepsilon$ not evaluated against LB-BNN and ENN as the authors did in Table 12 for different rotations of MNIST and Table 11 for different perturbation noise levels?
- L212, Eq. (4): How is the set of probability distributions $P$ on $\Theta$ selected/constructed?
soundness: 3
strengths: - The manuscript is generally well written and in most parts easy to follow. Examples and visualizations are provided where necessary or helpful for the reader.
- The authors deliver a good example for belief functions in Section 2 which easy to follow, and greatly helps to understand the previous definitions.
- Visualizations in Figure 1, 3, 4 and 5 are well crafted and complement the understanding of the described procedures.
- The experimental evaluation is detailed and covers multiple methods, datasets and aspects to investigate. Most experiments were run multiple times, the standard deviation was reported in addition to the mean result.
- The proposed method impressively beats most of the previous state-of-the-art results in terms of accuracy, uncertainty estimation, and out-of-distribution detection while keeping the inference time of the base model (note, that this does not include the additional time one-time cost for the budgeting step).
- The appendix contains a lot of additional experiments and in-detail explanations and recaps for relevant methods. While being relatively long (23 pages), the authors managed to insert references to the appendix in the main body of the manuscript at many relevant places. I found myself looking into the appendix more than usual compared to other papers with long appendices (which I see as positive).
summary: The authors introduce a novel Random-Set Neural Network (RS-NN) approach for epistemic uncertainty estimation in classification. RS-NN can be applied on top of an existing neural network and predicts distributions (belief functions) over collections of sets of classes and makes use of pignistic prediction to estimate the entropy. The authors theoretically motive their work and empirically show, that RS-NN outperforms state-of-the-art Bayesian and Ensemble uncertainty estimation methods on multiple metrics and datasets.
weaknesses: The experimental results solely focus on image classification tasks. What about a different modality? Tabular data? Text data? The proposed method is not specific to image data and can be applied to arbitrary models and modalities with classification tasks.

Not having worked with belief functions yet, it was at first unclear to me, that Eq. (1) goes over *all possible subsets A* since L185 talks about "mass values to *its* subsets", implying that $m(\cdot)$ is a function which is assigned / is only valid on its own set of subsets. Suggestion: make this consistent with the use of the powerset notation used in the directly following example (L193) and replace $\forall A \subset \Theta$ in the middle of Eq. (1) with $A \in \mathbb{P}(\Theta)$ at the end of the equation.


Minor Errors and Suggestions (these did not impact the score):
- L30: "how confident ti" -> "how confident it"
- L100: "can applied" -> "can be applied"
- L103: "we outline an **ingenious** budgeting method" -> In my opinion, the authors should refrain from calling their own contributions "ingenious".
- L183 - L187: This part is a bit confusing, stating "While [...] mass functions does X, a belief function does Y:", then following with an equation of the mass function (the sentence implies, that the reader will now be shown the definition of a belief function, which only comes in the equation after the next paragraph).
- L269 and multiple times in the appendix: Wrong quotation marks around words were used ("foobar" instead of ``foobar'').
- L305: The definition of $M_r$ should appear directly after introducing $M_r$ in the text. I guess the authors squashed this into Eq. (6) together with $M_s$ to safe space, but it would improve readability to separate both definition, since right now the text talks about $M_s$ ("we also add a mass sum term $M_s$, namely:") and then shows a definition of $M_s$ *and* $M_r$. Since the camera-ready version does not allow an additional page, the authors could also rewrite the text preceding Eq. (6) to make this more clear.
- L432, Tab. 2: The highlighted results (bold-face) for "In-distribution Entropy" for ENN on ImageNet is wrong and should be FSVI (lowest), and for "Out-of-distribution Entropy" for RS-NN on F-MNIST is wrong and should be DE (highest).
- L1362: "to a 95The class" -> "to a 95% confidence interval. The class" (?) -- I guess the "%" was not escaped and commented out the rest of the line in LaTeX.
- L2144, E.7, Tab. 15: The highlighted results (bold-face) for "Entropy (OoD)", "Credal Set Width (OoD)" on SVHN and "Credal Set Width (OoD)" on Intel Image are flipped (in all cases "Budgeted RS-NN" was highlighted as "winner" while "Standard RS-NN" achieved a better performance).
- E.4, L1830: The authors speak of "Tab. 1 in the anonymous PDF linked to this response". This reads like a leftover from a previous submission in which this part of the appendix was a response to a review.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. Please address the point raised in section Weaknesses.
2. The authors claim "Current machine learning models struggle to provide reliable predictions when confronted with unfamiliar data (Guo et al., 2017a)". The paper is from 2017. How did the authors infer no progress has been made since ?
3. The authors claim "RS-NN uses a random-set framework which does not require prior assumptions and is more data driven". Both models are classifying the same data, hence the claim that one is more data driven than the other does not hold and additional details are required to make the text clear.
4. The authors provided code to replicate results. When running eval.ipynb, line:

model = load_model(selected_model, selected_dataset, model_type = "CNN")

threw error:

OSError: No file or directory found at saved_models/CNN_resnet50_cifar10.keras

The weights file was found to be missing. Can the file be provided ?
soundness: 2
strengths: This paper describes a method in uncertainty estimation for deep learning, using the Random-Set Neural Network (RS-NN). The work discusses a theoretical framework using random sets and belief functions to model epistemic uncertainty, develops an innovative "budgeting" technique to make the approach computationally tractable, and proposes new uncertainty measures based on pignistic entropy and credal set width. The paper presents comprehensive empirical validation across multiple datasets and architectures showing improved performance to state-of-the-art Bayesian and ensemble methods in accuracy, out-of-distribution detection, and uncertainty quantification. The presentation is clear, with explanations of complex concepts supported by helpful visualizations and detailed appendices. The work is significant for machine learning deployment in safety-critical domains, offering a wrapper technique applicable to existing architectures while providing uncertainty estimates and robustness to adversarial attacks. The theoretical foundations in random set theory, combined with practical engineering solutions like the budgeting procedure, make this work both mathematically rigorous and implementable in practice.
summary: This paper introduces Random-Set Neural Networks (RS-NN), a novel approach to classification that predicts belief functions over sets of classes rather than probability vectors. The key innovation is using random sets and belief functions to model epistemic uncertainty arising from limited training data. RS-NN extends any baseline neural network by modifying its output layer to predict belief values for relevant sets of classes, identified through an efficient "budgeting" procedure that selects the most informative class combinations. The approach measures uncertainty through both the entropy of the "pignistic" (central) probability prediction and the width of the predicted credal set (set of compatible probability distributions). Through extensive experiments on multiple benchmarks, the authors claim RS-NN outperforms state-of-the-art Bayesian and ensemble methods in accuracy, out-of-distribution detection, uncertainty quantification, and adversarial robustness, while being computationally efficient. The approach is applied to different network architectures and provides statistical guarantees through conformal prediction.
weaknesses: The authors claim in the abstract "In this paper, we propose a novel Random-Set Neural Network (RS-NN) approach to classification
which predicts belief functions" and go on to cite Manchingal & Cuzzolin, 2022 (w.r.t. epistemic uncertainty) where on pg 18 it is "discussed under Sec. 6, the performance and evaluation metrics for random set neural networks", as well as several mentions of belief functions throughout the earlier paper. The works seem closely related hence the claim to originality in the more recent work seems questionable, and a clarification in order.

Meta-review: The paper introduces a novel method to augment neural networks with epistemic uncertainty estimates by providing distributions (belief functions) over the collection of sets of classes. The respective approach is referred to as random-set neural networks (RS-NN). The advantages of RS-NNs are numerous, ranging from its applicability and practical efficiency, to comprehensive evaluation with strong results. Reviewers acknowledged the contributions and thought the paper to be well-structured and well-written. Two main points for requested improvements were to investigate a modality that is not images, and to conduct a deeper investigation with respect to robustness in attack scenarios. The authors have addressed these through additions in the discussion phase to the full satisfaction of two reviewers. The third reviewer’s only raised weakness seems to be a misunderstanding of a lack of novelty because of related work mentioned in the paper. The authors have provided adequate qualification to this point.
Overall, the AC agrees with the reviewers that the paper is well-written, the method important, and its strong results over e.g. ensembles practically useful. The recommends to accept the paper.

Decision: Accept (Poster)