Explaining Bayesian Neural Networks
Authors: Kirill Bykov
Marina MC Höhne
Adelaida Creosteanu
Klaus Robert Muller
Frederick Klauschen
Shinichi Nakajima
Marius Kloft
Venue: Accepted by TMLR ()

Abstract: To advance the transparency of learning machines such as Deep Neural Networks (DNNs), the field of Explainable AI (XAI) was established to provide interpretations of DNNs' predictions. While different explanation techniques exist, a popular approach is given in the form of attribution maps, which illustrate, given a particular data point, the relevant patterns the model has used for making its prediction. Although Bayesian models such as Bayesian Neural Networks (BNNs) have a limited form of transparency built-in through their prior weight distribution, they lack explanations of their predictions for given instances. In this work, we take a step toward combining these two perspectives by examining how local attributions can be extended to BNNs. Within the Bayesian framework, network weights follow a probability distribution; hence, the standard point explanation extends naturally to an explanation distribution. Viewing explanations probabilistically, we aggregate and analyze multiple local attributions drawn from an approximate posterior to explore variability in explanation patterns. The diversity of explanations offers a way to further explore how predictive rationales may vary across posterior samples. Quantitative and qualitative experiments on toy and benchmark data, as well as on a real-world pathology dataset, illustrate that our framework enriches standard explanations with uncertainty information and may support the visualization of explanation stability.

Published in Transactions on Machine Learning Research (09/2025)
Explaining Bayesian Neural Networks
Kirill Bykov'
kirill. bwk.oyykko.blddd.. de
BIFOLD; TU Berlin, Berlin, Germany; ATB Potsdam, Potsdam, Germany TU Munich; MCML, Munich, Germany'
Marina M.-C. Höhne
mhoehne mboehne@@ab-yotsddmm..o de
ATB Potsdam; University of Potsdam, Potsdom, Germany TU Berlin; BIFOLD, Berlin, Germany;
Adelaida Creosteanu
Aignostics, Berlin, Germany
Klaus-Robert Müller
BIPOLD; TU Berlin, Berlin, Germany; Machine Learning Group, TU Berlin, Berlin, Germany
Korea University, Scoul, Korea;
Max Planck Institute for Informatics, Saarbrücken, Germany
Frederick Klauschen
Institute of Pathology, Charité Univvrsstttsmmediii; BIFOLD, Berlin; Institute of Pathology. LMU Munich; German Cancer Consortium (DKTK), Munich, Germany
Shinichi Nakajima
BIFOLD; TU Berlin, Berlin, Germany; RIKEN AIP, Tokyo, Japan
Marius Kloft
Department of Computer Science, RPTU, Kaiserslautern, Germang
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id- tassRtt3u.b
Abstract
To advance the transparency of learning machines such as Deep Neural Networks (DNNs), the field of Explainable AI (XAI) was established provide interpretations of DNNs' predic- tions. While different explanation techniques exist, popular approach is given in the form of attribution maps, which illustrate, given particular data point, the relevant patterns the model has used for making its prediction. Although Bayesian models such as Bayesian Neural Networks (BNNs) have a limited form of transparency built-in through their prior weight distribution, they lack explanations their predictions for given instances. work, we lake a step loward combining these two perspectives by examining how local tributions can be extended BNNs. Within the Bayesian framework, network weights follow probability distribution; hence, the standard point explanation extends naturally to erplanation distribution. Viewing explanations probabiiittiaall,, aggregate and an- alyze multiple local attributions drawn from an approximate posterior to explore variability in explanation patterns. The diversity of explanations offers a way to further explore how predictive rationales may vary across posterior samples. Quantitative and qualitative exper- iments on toy and benchmark data, as well as on a real-world pathology dataset, illustrate that our framework enriches standard explanations with uncertainty information and may support the visualization of explanation stability.
this
"Equal contribution
Work performed while TU Berlin and ATB Potsdam.

---

Published in Transactions on Machine Learning Research (09/202))
Tiesue Slice with Cancer Segmentation
DNN Prediction
BNN Uncertainty estimation (stc)
Patches with high
Lossa cortain
uncertsinty
-0.9
0.05
0.04
004
...
022
0.00 da
Figure 1: Illustrating the practical advantages of BNNs over standard DNNs. From left to right: a whole- slide image from a cancer patient, divided into patches with highlighted cancerous regions; DNN prediction: BNN uncertainty estimation; and patches with the highest uncertainty. Unlike standard DNN, the BNN provides uncertainty estimates for each patch. Blue indicates high certainty, while red indicates low certainty. The rightmost panel shows patches where the model's ' prediction is uncertain and fluctuates between classes.
1 Introduction
Deep Neural Networks (DNNs) have achieved significant success over the years, helping to advance Artificial Intelligence (A1). Driven by the exponential growth in available data and computational resources, DNNs achieve statc-of-tho-art results across various fields of Machine Learning (ML), surpassing human perfor-
mance in various domains tasks (Silver al.) 2016 Merchant et al., 2023 He et al.) [015 Vinyals et al.,
2017 Liu al. 2019 Won al. 2020) and, recently, demonstrating emerging general reasoning abilities with the advances of Large Language Models (LLMs) (Bubeck et al. 2023). DNNs accomplish such high performance by learning mappings from raw data to meaningful representations.
With the increasing complexity of modern Neural Networks, it is a difficult task to explain their decision- making process. Therefore, DNNs have often been considered as "black-box" (Vidovic et al. 2015
Buhrmester et al. 2019). However, especially in security-critical applications (such as autonomous driv- ing or medicine), transparency of the makksinn-making model is mandatory, and therefore, the network's inability to explain its predictions restricts the applicability of ML systems. Indeed, despite showing great performance in test environments, DNNs have not yet reached universal acceptance in the above-mentions areas (Buch et al. 2018). Furthermore, recent studies have confirmed that frequently DNNs are susceptible to learning spurious correlations Geirhos et al. 2020). These correlations, often arising from chance occur- rences or the presence of specific artifacts, are not intended and have undesirable consequences, resulting in poor generalization and potential risks. This phenomenon, termed the Clever Hans effect. (Lapuschkin al., 2019) holds significant importance due its widespread prevalence among various models and datasets (Iz- mailov al., 2022). Notably, substantial number of models trained the ImageNet dataset (Deng al. 2009) display reliance watermarks in Chinese language watermarks (Bykov et al. 2023d significantly impairing their performance (T.i et al. 2023
The field of Explainable AI (XAI) has emerged to address these concerns (see e.g. Samek et al. (2019) XAI aims to develop and study methodologies for explaining the predictions made by advanced learning machines such as DNNs. Recent advances XAT have led to variety of novel methods (Saleem et al. Adadi & Berrada 2018 Das & Rad 2020 Tjoa & Guan 2020). These can be grouped into global and local explanation methods. While global explanation methods interpret the decision-makingg of DNNs
2022
acToss population by visualizing the signals that cause significant activation in neuron (Erhan et al.
2009 Nguyen et al. 2016), analyzing relationships between neurons (Bykov et al. 2023a b) or by linking the
neuron's 8 learned concepts to human--undersaaddabl concepts (Bykov et al. 2023c Bau et al. 2017 Mu &
Andreas 2020: Oikarinen & Weng 2022), local explanations provide interpretations of the prediction for a particular data example by attributing relevances the input features (Bachrens et al. 2010 Bach et al.
2015 Selvaraju al. 2017 Yosinski al. 2015 Kindermans et al. 2017 Montavon et 2018 Lundberg
& Lee 2017
Lundberg al. 2020 Dombrowski al. 2021
2

---

Published in Transactions on Machine Learning Research (09/2025)
Most of the explanation methods, local and global ones, are developed for DNNs trained in a mazimum- a-posteriori (MAP) setting, where the weights of the model are point estimates. Tn contrast, Bayesian Neural Networks (BNNs) learn a distribution of the network weights, which also induces a distribution on the prediction. For a better intuition on the advantage in explainability of a BNN over a DNN, we consider the example image shown in Figure [ where a histopathological image (Janowezyk & Madabhushi 2016
Cruz-Roa et al. 2014) taken from a cancer patient (sce c.g. (Klauschen et al. 2018 Hägele et al. 2020; Binder et al. 2021 Klauschen et al.] 2024b) is shown on the left. Starting from this whole slide image, smaller patches are extracted and fed into a DNN, trained to classify patches into cancer or non-cancer. In the center of Figure 1 we show the prediction score of a regular (non-Bayesian) DNN for the class cancer. While these scores are usually normalized with a softmax function, these scores do not represent actual probabilities (Hendrycks & Gimpel| 2016) the DNN provides no further information on how certain or uncertain the patches' relevances are for the prediction. This additional information is provided by BNNs, o.g. the form of the variances shown on the right in Figure [ We observe that there are distinct regions where the model is significantly more certain about a patch, indicating cancerous areas (highlighted in blue colors) than in other regions (highlighted in red, referring to the most uncertain regions).
Thus, BNNs provide predictive uncertainty, but-like standard DNNs, do not yield local explanations out of the box. Our contribution is a practical, method-agnostic framework that translates predictive epistemic uncertainty into explanation uncertainty: a distribution over attribution maps that can be summarized by mean and percentile- based aggregations and analyzed via clustering. Within the scope of this work, our goal is to visualize and quantify variability in explanatory rationales under an approximate posterior, without making claims about identifying ground-truth decision modes. Importantly, the computational overhead of our aggregation scales lincarly with the number of posterior samples N (i.c., N basc attributions plus constani-time aggregation), and for lightweight approxinations such as MC Dropout, the per-sample inference cost differs marginally from deterministic models. Our model is even applicable to regular DNNs (c.g.. CNNs) trained in a non-Baycsian fashion, which can first be translated into BNNs (e.g., using MC dropout Gaa & Ghahramani 2016) or Laplace approximation (De Laplace 1774, Ritter et al., 2018)), and then using our proposed met for a rigorous explanation. Thus, our approach can enrich the explanation of regular DNNs with additional uncertainty information.
To illustrate this advantage, consider again the medical task illustrated in Fig I The first step towards transparency of the shown model is assigning the input features with relevance scores; this is what explanation methods of regular DNNs do. It helps to identify the areas in the image that were relevant for the prediction. Our BNN explanation would additionally provide information on regions where the method is confident about the relevance scores. This may enable an expert to faster identify the most signíficant cancer areas in an image. On the other hand, the expert might want to look specifically into areas of low confidence to resolve this low confidence by contributing with human expert knowledge to cancer image evaluation. Hence, by determining the level of certainty required by particular case by visualizing multiple levels of certainty once in an explanation, our method could lead to additional insight into the underlying prediction strategy of model.
In this work, we will propose and investigate different techniques for explaining the decision- maaiiig process of (deep) BNNs. Our approach is method-agnostic,, i.e., it can build on any arbitrary explanation method for regular (non-Bayesian) DNNs, which are transformed into a local attribution method for BNNs. The proposed method can be combined with any (approximate) inference procedure of BNNs. In computational experiments, our approach interestingly revealed that BNNs implicitly employ multiple heterogeneous pre- diction strategies. The reason is that BNNs exhibit numerous modes, and approximately one can think of mode as a prototypical prediction strategy. In contrast, in a standard non- -ay-Biasi DNN, the prediction is deterministic and thus cannot extract multi-modal explanations. With our proposed method, we are now able to visualize the different modes, thus revealing the intrinsic multi-modality in the decision- making of BNNs (and by association: DNNs).
In the following, we summarize the main contributions of this worl
"ode and reproduction instructions are available at httpp:///tthhb..ccoc/opp//////pp
3

---

Published in Transactions on Machine Learning Research (09/2025)
Mean explanation under approximate posterior: provide conditions under which the explanation of the predictive mean equals the mean of explanations (Lemma 3.1 Theorem 3.1
UAI (Inion/Intersection) aggregations: percentile- based summaries that expose stable vs. opportunistic features and yield uncertainty-aware heatmaps (including UAT+).
Explanation diversity: cluster the sampled explanations in order to visualize diverse explana- tory patterns that may arisc under the chosen posterior approximation.
Gencrality and cost: the framework method- and inferenco-agnostic and adds only linear overhead O(N Cattr) (with Cattr the cost of the base allribution).
2
Background and Related Work
This section provides a comprehensive overview of BNNs and well-established local explanation methods.
2.1
Bayesian Neural Networks
From statistical perspective, standard DNNs are usually trained using maximum a-posteriori (MAP) 2020):
optimization Wilson
W argmaxw logp(W W Dw), argmaxw log (Drr W) log)(W),
(1)
which reduces to the maximum likelihood estimation when the prior distribution p(W) is flat. The most commonly used loss functions and regularizers fit into this framework, such as categorical cross-entropy for classification or mean squared error for regression. Although this procedure is efficient since the networks only learn a fixed set of weights, it does not provide uncertainty information about the learned weights contrast, Bayesian neural networks (BNNss estimate the posterior and subsequently the prediction. distribution of weights, and thus, provide uncertainty information on the prediction, which can provide confidence information on predictions. Particularly, in critical real-world applications of deep learning Holzinger et al.) 2019 Klauschen et al.) 2018) and autonomous
for instance, modicine (Hägele et al. 2020
driving (Koo et al. 2015 Wiegand et al. 2019) where predictions need to be highly precise and wrong predictions could easily be fatal, the availability of prediction uncertainties can be of fundamental advantage.
Let fw: Rd R. be food-forward noural network with the weight parameter W E W. Given a training dataset Der = {uns yn LN Bayesian learning (approximately) learns the posterior distribution
p(D ((D::W))((W) (WIDu.) P W) (pWW)dW
(2)
where p(W) is the prior distribution of the weight parameter. After training, the output for a given test sample is predicted by the distribution:
plyla, Dwt) = ( plulfw(a)))(W/Pp...
(3)
Since the denominator of the posterior, shown in equation 2 is intractable for neural networks, nu- merous approximation methods have been proposed, e.g., Laplace approximation (Ritter et al. 2018
Variational Inference (Graves] 2011
Osawa et al.] 2019), MC dropout (Gal & Ghahramani 2016
Vari-
ational Dropout (Kingma ct al. 2015 Molchanov et al. 2017), MCMC sampling (Wenzel et al. 2020 and
SWAG (Maddox et al. 2019 Wilson & Izmailov 2020) With these approximation methods, one can now efficiently draw samples from the approximate posterior distribution of the network parameters (equation 3 and compute statistics, c.g., mean and variance, the prediction for given data point æ. Classical MAP training procedures could also be seen as performing approximate Bayesian inference, using the approximate posterior p(.) 5(W W), where 8 is the Dirac delta function.
4

---

Published in Transactions on Machine Learning Research (09/2025)
2.2
Local attribution methods
Local explanation methods altribute relevance to the inpul (Tealures) or intermediate nodes (Bach et al. 2015 Vidovic et al. 2016 Selvaraju et al.) 2017 Montavon et al. 2018) by using relevance attribution operation, which we define as follows:
Definition 2.1 (Relevance Attribution operator). An operator T2,w [ that maps an output function fw Rd + R to a relevance function R: Rd + Rd is called a relevance attribution operator:
(w(x) (Tww(/w((ar).
(4)
The above definition postulates that the relevance of an input leature/nod node depends on the input mainly via the output function, although it can have a direct dependence on x and W.
In this paper, we demonstrate our novel BNN explanation framework mainly using Layer-wit wisc Relevance Propagation (LRP) (Bach al. 2015) as the base explanation method, however, we would like stress that our BNN explanation framework can be applied for any existing explanation method.
Gradient explanation The Gradient explanation method, i.e. T.w = Va. where Va is the weak deriva- tive w.r.t. z, is one of the most basic explanation methods. It visualizes the possible extent of change made by the predictive function in a local neighborhood around the original datapoint x (Erhan et al. 2009:
Baehrens et al. 2010: Simonyan et al. 2013).
LRP Layer-wise Relevance Propagation (Bach et al., 2015) is model-aware explanation technique that can be applied for leed-forward neural networks and can be used for different types of inputs, such as images, videos, text (Anders et al. 2019 Arras et al.) 2017 Montavon et al.) 2018 Binder et al.) 2021). The underlying idea of the LRP algorithm is to use the network weights and the neural activations computed in the forward-pass to propagate the relevant output back through the network until the input layer is reached. This propagation procedure is subject to conservation rule - analogous to Kirchhoff's conservation laws in electrical circuits (Montavon et al. 2019) in each backpropagation step, the relevances from the output layer are distributed towards the input layer, while the sum of relovances should remain the same. Existing variations of LRP are, e.g., LRP-0, IRP-s, IRP-y, and TRP-CMP (Bach et al. 2015: Montavon et. al. 2019
Integrated gradients Integrated Gradients (Sundaraaaaan et al. 2017] is an axiomatic local explanation algorithm that also addresses the "gradient saturation" issue (Samek et al. 2016). It assigns relevance scores to cach feature by approximating the integral of the gradients of the model output with respect to a scaled version of the input. The relevance altribution function, this case, can be defined as
Ta [Sw] (x S." Ofw (o + a(x (() x))
da,
dr
where @ denotes the element-wise product, and a reference point that represents the absence of feature in the input.
3 Explaining Bayesian Neural Networks
Relation to prior work. Work on interpreting Bayesian neural networks (BNNs) has largely focused predictive uncertainty rather than distributions of explanations. Kwon et al. (2020) decomposes moment-basee based predictive uncertainty into aleatoric and epistemic components, and Chai (2018) proposes model-agnostic approach to visualize the contributions of individual features to predictive, epistemic, and
aleatoric uncertainly. These methods analyze uncertainty of the prediction, not the variability of input-level explanations induced by the BNN posterior. Noise-based smoothing methods for deterministic networks take a different route. Bykov et al. (2021) injects multiplicative Gaussian noise into weights to reduce gradient shattering (cf. Samek et al. (2021)) in the spirit of SmoothGrad (Smilkov et al.) (017). While such stochasticity can yield smoother saliency maps, it is not tied to Bayesian posterior and does not
5

---

Published in Transactions on Machine Learning Research (09/202))
Union and Intersection (UAI)
f(z, WT)
.-.>
:
BNN
f(z, W.)
-->
Local Explanation Clustering
g
Input
f(z, Wy)
-
BNN instances sampling
Local Explanation
BNN explanstion
Figure 2: Schematic illustration of proposed methods for explaining Bayesian Neural Networks. Given a particular input cat image we sample models from the posterior distribution and collect local explanations from each instance. These explanations are later aggregated using Union and Intersection (UAT) method: The Union explanation provides global overview of the features learned by the BNN by combining various modes, whercas the intersoction explanation provides the intersection strategy used by the BNN demonstrating the most certain leatures used for the prediction. Further, local explanations be clustered to illustrate the different modalities of the Docision-making strategies.
provide costerior-consiittrn distributional summaries of explanations. A complementary line of work explores variability at the representation level. Grinwald et al. (2023) visualizes diversity latent representations learned by BNNs on the global level using the Feature Visualization method Olah et al. (2018). In contrast, operate at the input-attribution level, producing per-input distributions over local explanations. We also note dopain-specific applications of BNN explainability, e.g., ocean dynamics (Clare et. al. 2022), and
uncertainty-aware XAI for deterministic models (Marx et al. 2023).
Our work complements these directions in three ways: (i) it agghod--ggostic across local attribution techniques and inference schemes; (ii) operates at the input-attributee level, rather than on latent rep- resentations-yiodding a distribution of explanations; and (iii) it introduces percentile-based aggregations and explanation clustering as practical lools for summarizing and visually inspecting variability under an approximate posterior. (i) We treat explanations random variables under the approximate BNN posterior and summarize them with percentiles (Intersection for high agreement; Union for high coverage) and UAI+ for stability-weighted positive relevance. (ii) The framework is method-agnostic across local attribution techniques (e.g., LRP, IG) and inference schemes (ensembles, Laplace, MC-Dropout). (iii) We provide an exploratory pipeline for explaining multi-modality via clustering of per-sample attributions. Concretely, un- like Kwon al. (2020); Chail (2018), we quantify and visualize variability of explanations (not only predictive uncertainty); ); unlike Bykov al. (2021), our variability arises from posterior sampling rather than externally injected noise; and unlike Grinwald et al. (2023), summaries input-level and immediately usable for stability coverage diagnostics.
3.1 Method
In the following, we consider neural network fw(x) and relevance function Rw (x) (defined in equation using an arbitrary explanation method. Note that Rw (x) ) is deterministic mapping for a fixed parameter W. Therefore, the posterior distribution of parameter W induces a distribution over the relevances, resulting we can define the
in a distribution over relevance maps. Given the posterior distribution dist ristrioution of relevance as
p( (R|æ, Dr.) = f Bw(a))(W\D..aaw
(5)
The relevance samples
A~ 2(, D..)
can be obtained by drawing weights from the posterior distribution:
W (Dr...

---

Published in Transactions on Machine Learning Research (09/2025)
A schematic illustration of the process of obtaining the explanations, as well as high-level overview of proposed methods, is given in Figure 2 Here, illustratively, we can observe that different. samples of the network lead to different explanations of the same input image. Applying aggregation strategies, such as the Intersoction, Average, and Union, provides different perspectives on model behavior and may offer additional insights into how predictions are formed.
3.1.1 Average Explanation
For some conditions, the average relevance attribution coincides with the relevance attribution of the average prediction, which we state in the following Lemma:
Lemma 3.1. For any explanation melhod that can be sormalized as in equalion with a linear operalor Ta, = Tx that does not depend on W, it holds that
Ts [Ew [fw]] (x) Ew [w]]()) Ew (R)(
(6)
The claim is held trivially by the lincarity assumption. Equation Gin the above Lemma holds for some existing explanation methods, including LRP-0, which is known to be expressed as equation 4 with
Tz[Sw](a = x ④ V=[Sw](r) One can still rely on equation 4 under slight violation of linearity.
Theorem 3.1. Equation 0 holds almost everywhere in x E Rd for LRP-O with ReLU networks, gradient explonation, and 1G.
The proof is given in the Appendix. Theorem 3.1 states that the explanation of the predictive mean (LHS of equation G can be computed by the sample mean of the relevance maps over the posterior distribution (RHS of equation
This is beneficial since it is computationally exhausting to explain the approximate mean predictive function directly (this requires simultaneously storing all the parameters along with their computational graphs for each of the samples from the posterior distribution), using the results from Theorem 3.1 we can now easily explain it by sampling relevance maps, which drastically cases the process.
3.1.2 Exploring multi-modality of explanations
As result of the non-linear network activations, BNNs known to have multi-modal predictive func- tions (Bishop 2006). Different paramcters sampled from an approximate posterior yield noticcably different local allributions for fixed input (Fig. 2h. Consequently, naive averaging obscure meaningful variability. We therefore cluster-samplee explanations to provide an overview of different explanatory pat- terns within the postcrior. Note that we do not assume that clusters correspond to distinct postcrior modes; rather, they offer an exploratory perspective on potential explanation diversity.
To investigate the "prime" strategies of the Bayesian learning machine and to decompose the behavior of BNNs into groups of inter-similar strategics, we propose to cluster the sampled explanations. Although our method does not restrict user in choosing an algorithm for clustering, Relevance Analysis) clustering method. This method was initially introduced in (Lapuschkin et al. 2019) to solve a similar task an analysis of the class-wise learned decision strategies of DNN in order to obtain a global view on the class related relevant patterns, which also supports the identification behavior, such as Clever Hans artifacts (see Section 5.4 for more information about Clever Hans behavior). Note that SpRAy was originally constructed to investigate the typical traits in the decision-making process over large collection of relevance maps, which were obtained from different data points from the training dataset. In contrast to the original setting of SpRAy, we exhibit and identify typical as well as atypical behavior in the BNN decision-making process for a single input image.
propose to use the SpRAy (Spectral
undesirable
Once the clustering has been performed, "prime" prediction strategies of the BNN for the given input image can be identified by the cluster-wise average explanations. Moreover, the number of saliency maps in each cluster (normalized by the number of sampled relevance maps) could be considered as the "strength" of each strategy. We could visualize the explanations, clusters, and average cluster strategies in a two-dimensional embedding using t-SNE plot MMatenn & Hinton 2008) shown in Figure 3 where the explanations of
7

---

Published in Transactions on Machine Learning Research (09/2023)
Clustee
Clustec
cuater
Figure 3: Visualization of the multi-modality of gradient explanations of BNN (here, LeNet network trained with dropout) is shown exemplarily for image of class "Trousers" from the Fashion MNIST dataset. The explanations were clustered by the SpRAy algorithm into 7 clusters, stated on top, and the first row shows the mean explanation for each cluster respectively, where the shape of the trousers is overlaid over the explanation. The second row depicts the t-SNE visualization of the distribution of explanations, where the points of the particular clusters are highlighted. From the mean cluster explanations, we can observe the variability in the dccision-making process of the Bayesian Neural Networks cach mode illustrates one reciisionmmaking pattern, and the number of elements in each cluster indicates the importance of each cluster to the prediction.
the Fashion MNIST (iiao et al., 2017 input image is mainly divided into seven different clusters, which indicates the patterns of the main modes of the BNN. More practical details about the clustering process can be found in the appendix.
3.1.3 Union and Intersection Explanation
Grasping the multimodality of the network through explanations can be done in several ways. Each relevance attribution map is explanation for individual instance of the Bayesian Neural Network. In our work, observed that differences between instances of BNN reflected in the multi-modal distribution of explanations. Therefore, to aggregate differences in explanations for Bayesian Neural Networks, we propose a mothod called UAI: Union and Intersection. The intuitive idea is illustrated in Figure 2 We treat the relevance of a BNN as a random variable that follows equation
5
Given this distribution of relevance maps p( Dw.), we capture the uncertainty information of the BNN follows:
(A).(x) Pa [p(R]a, D.t)),
(7)
where Pa is an operator computing the entry-wise (e.g. pixel-wise in the case of images) percentiles.
High percentiles (a > 0.5) correspond to what we introduce as Union explanation a resulting relevance map consisting an accumulation of features various relevance maps, iee, providing information across features that were considered of high importance by at least
modes of the BNN by allocating relevance small proportion of samples. In contrast to the Union explanation, small percentiles (a 0.5) illustrate the interscction of features, where at least 95% of explanations agroc. For visualisations, we define Union explanations with 0.95 and Intersection explanations with = 0.05
Furthermore, we introduce UAI+, as the uncertainty information of the BNN relating to positive class attributions only:
F.) - (R(l2,, Du.))
where F. an operator computing the entry-wise (pixel-wise) probabilities of relevance attributed a particular pixel being less than some small predefined value 0. All proposed approaches (Intersection,
8

---

Published in Transactions on Machine Learning Research (09/2025)
Saliency
Intersection
Union
Intorsection
Union
Integrated Gradients
GradientSHAP
Intersection
Union
UAI
Intersection
Union
UAR
UAI Legend: e 100) 100]
a (6006,80)
a E 60)
c t0 201
Figure 4: Illustrative explanations of a Bayesian Neural Network. The BNN was trained with Dropout on the Custom MNTST dataset (Bykov et al. 2021 The input, an MNIST digit zero on a random CIFAR Background, shown on the left and was correctly classified as zero by the BNN, Explanations of the BNN decision are given Intersection (a 5), Average, Union (a 95), and UAI+ explanations using LRP- (first row), Integrated Gradient (IG) (second row). We can observe that the relevance of all explanations correctly emphasizes the digit. However, our proposed Union and Intersection approach, in which the bundled information is contained in the UAI+ explanation, has stronger informational content about the role of the features concerning the model prediction by specifying the model's (un)certainty that a feature contributed to the prediction made.
Average, Union, and UAI+) are mothod-agnostic, and the resulting LRP-c and IG explanations are illustrated in Figure for the case of an MC Dropout network.
As scales various explanation methods differ, instead of thresholding significant positive relevances by fixing e, we perform group normalization: we normalize all positive relevances in all sampled attributions R2 by the maximum relevance value:
R
R
(9)
max r// R, Vie [I Vj (1.d)
where N. is the number of sampled attributions from the posterior and is the number of features of the input. This way, we can set to a small value on the scale of [0,1], such that it will threshold only significant positive relevances. In our visualisations, we set e = 0.05 as we empirically observe this value to be the borderline of visual recognition of positive relevances in the attribution map.
3.1.4 Computational complexity
Let Cattr denote the cost of one base attribution a deterministic network. Our framework draws N poste- rior samples and computes N attributions, then applics constant-time, entry-wise aggregations (percentiles or averages). Thus, the total cost is
O Cattr)
Odd log N)
attributions
percentiles (optional)
where is the number of input features (e.g., pixels). For MC Dropout, each forward pass adds only Bernoulli mask sampling and clement-wise masking, so Cattr close the deterministic cost. The choice of posterior approximation is orthogonal to our method; more expensive samplers (e.g.. HMC) increase Cater proportionally.
9

---

Published in Transactions on Machine Learning Research (09/2025)
4
Evaluation procedure
In the following, we provide the methodology of the qualitative and quantitative evaluation.
4.1 Qualitative Evaluation
For visual inspection of the results, normalize the relevance maps with the MinMax transformation (Bach) et al. 2015), which maps positive relevances onto the interval [0, 1] and negative ones to [-1, 0j. Afterwards, the normalized relevance maps are visualized using the 'seismic' colormap which attributes red tones to pixels with positive relevances and blue toncs to pixels with negative relevances.
4.2 Quantitative evaluation
For quantitative evaluation, we use the localization criterion, where we are interested in measuring the ability of an explanation method to attribute positive relevance to the object interest. Hence, exemplary, the prediction of a model "cat", we assume that the given image, parts of the object cat are responsible for the prediction and subsoquently yield positive relevance attribution by the explanation method. Thus, in order correctly measure the ability of the method to "find" the object of interest, ground-truth segmentations
are required [Arras et al. 2020 Lapuschkin et al. 2016 Kohlbrenner et al. 2020 Zhang et al. 2018 Fawcett,
2006 Bykov et al. 2021
To measure the localization capability of an explanation method, we employ two different metrics, Area Under the ROC Curve and Relevance Mass Accuracy (Arras et al. 2020
For each explanation (e.g., in the form of a heatmap), we calculate the area under the receiver operating characteristic in order to measure how closcly the areas of greatest relevance of the explanation correspond to the classified object. Note that for computing the AUC value, the pixel-wise ground-truth segmentation of the object serves as the true label information, whereas the relevance information from the explanation serves as the predicted label information.
AUC ROC:
Relevance Mass Accuracy (MA): For each explanation, we measure the proportion of the rele- vance mass that lies on the object in comparison to the total relevance mass:
MA Eieo R.
Ejer Rg"
where 1 is the set containing all features and C 1 a subset of features that are part of the segmented object itself.
While the AUC metric can be used both for explanation methods attributing positíve and nogative values, as well as for met hods altributing just positive relevances, MA can only be used for positive relevances from the explanation method. We acknowledge alternative metrics like Fidelity (Samek et al. 2016), ROAR (Hooker ct al. 2019), and F-Fidelity (Zheng et al., (224)), which assess explanation quality via input perturbation. However, these are designed for deterministic models with single explanation per prediction. Our method produces distribution explanations, making direct application of such metrics non-trivial. We therefore use object-localization metrics (AUC-ROC and MA), which are more naturally suited to our probabilistic setting. Extending fidelity-basee based metrics to BNNs remains an important direction for future work.
4.3 Baseline
In our experiments, compare the proposed UAI explanations with the baseline explanation, which uses the expected value of the weights E [W]. In practice, this would correspond to MAP classifier, e.g., Taplace approximation or MC Dropout, where the mean weights of the model are used for prediction. Hence, we refer to the baseline explanation as the one explaining the standard deterministic model using the mean
10

---

Published in Transactions on Machine Learning Research (09/2025)
Input
UA
1000
e 801
[40, 601
(E120,40)
200
Dropout
Laplace
Figure 5: Illustration of UAI+ method explanations based on the Absolute Gradient explanation method for three different Bayesian scenarios: Deep Ensemble, MC Dropout, and Laplace approximation. From the explanations, we can observe the main features that each of the Bayesian Networks used.
weights - this allows us to draw conclusions regarding how standard explanation methods can be enhanced by "Bayesianization..
5 Experiments
In the following, we demonstrate the performance of our proposed method both qualitatively and quantita- lively.
5.1 Experiment Custom MNIST data
We evaluate our proposed methods on the Custom MNIST (CMNIST) dataset [Bykov et al. 2021 where MNIST digits are plotted on randomly chosen CIFAR background. This ensures that the decision-mant basis for the model should rely only the number and not the randomly chosen background. Hence, using the available segmentations of the MNIST digits a ground-truth indicator of the relevant area the evidence should be distributed to, we are able to measure the goodness of the different explanations. To show that our method is applicable to different types of Bayesian Neural Networks and Network Ensembles, we analyze three diffcrent settings, all based on the same standard LeNet architocture (LeCun et al., 1998). We employ three different Bayesian approximation methods Deep Ensemble of 100 dilferent networks, trained with random initialization respectively, Laplace approximation, and MC Dropout (more details about the model architecture and training parameters be found in the appendix). Figure 5 illustrates the differences of U explanations between the three Bayesian scenarios. For each of the three described scenarios, we used test set of 10000 generated images, which was not used during training. For each image, N = 100 relevances were sampled using the LRP-s method from the posterior distribution (in the case of a deep ensemble, each relevance came from a different network instance).
The quantitative results of the localization evaluation for all three scenarios are summarized Table 1 From the results, we can observe that the Union method best-performing in terms of AUC metric, while the Intersection method shows overwhelmingly best results in terms of the Relevance Mass Accuracy metric. From these results, we conclude that the Union method indeed is better in visualizing all the information the Bayesian Network has learned about the object; however, comparatively low MA scores of the Union method imply that explanations attribule positive relevance outside of the object of interest. In comparison, the Intersection method has low AUC scores in almost all scenarios, which implies that the Intersection method does not "cover" the object interest with positive attributions, but high MA scores show high confidence in positive features the Intersection method attributes positive relevance to feature, is most likely to lie inside of object of interest.
11

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Quantitative results for Localisation (AUC) and Relevance Mass Accuracy (MA). Values are mean
+ s.d. rounded to two dccimals.
Ensemble
Laplace Approx.
Dropout
AUC
MA
AUC
MA
AUC
MA
Random
0.50 0.02
0.25 0.01
0.50 + 0.02
0.25 + 0.01
0.50 + 0.02
0.25 +: 0.01
Baseline
0.51 + 0.03
0.85 + 0.10
0.53 + 0.03
0.92 + 0.07
Average
0.50 0.05
0.91
0.05
0.50 + 0.06
0.85 + 0.09
0.51 1: 0.04
0.91 1 0.07
Intersection (a = 5)
0.16 + 0.06
1.00
0.00
0.15 1 0.07
0.78 1 0.41
0.17 + 0.06
0.99 1 0.03
Union (a 95)
0.88 0.05
0.77 0.09
0.86
0.07
0.70 1 0.13
0.86 1 0.05
0.87 1 0.08
UAI+
0.79 + 0.06
0.90 0.08
0.78 1 0.06
0.79 1 0.14
0.71 +: 0.05
0.94 1 0.07
Original Input
Intersection
Average
Union
UAL
ec 100]
e 201
ec e0t
[io, 20]
Figure 6: Exemplary explanations of three images taken from ImageNet (red/blit indicates positive/negative relevance). Each row corresponds to a particular input image. From left to right: original image, intersection explanation, baseline (average LRP) explanation, and Union explanations, as well as the UAI+ explana- tion. We observe that the baseline explanation attributes positive relevance to almost the whole cat except the whiskers. The intersection explanation highlights solely the eyes and the nose of the cat as coherent features whereas the union explanation highlights the same features the bascline explanation but addi- tionally attributes relevance to the whiskers. The explanation of UAI+ provides holistic representation of the different feature importance regarding the model decision: red=high importance, (orange, grey, light bitem=intermediite importance, and dark blue=low importance.
5.2 Experiment ImageNet
We demonstrate the usefulness of the proposed method for explaining the decision-making process for naturally non-Bayesian model that was trained with dropout regularization. We used the broadly applied pre-trained VGG16 network (Simonyan & Zissorman 2014). This network was pro-trained on ImageNet (Rus- sakovsky el al. 2015) and is naturally non- Non-Bayesian. To access the uncertainty of this naturally non-Bayesian model, we employed MC Dropout (Gal & Ghahramani 2016) during the test phase, which can always be applied when a modcl's architecture comprises at least one dropout layer. Furthermore, as a relevance attri- bution function, we used the LRP-CMP rule as explainability method. For the evaluation of the performance of the proposed methods, randomly choose small subset of classes from the ImageNet dataset, consisting
12

---

Published in Transactions on Machine Learning Research (09/2025)
of 3 classes: "tiger cat", "Ilama", and "lemon". For each class, we downloaded 1000 random images. The explanation results for the Intersection, Average, Union, and UAT+ explanations for three randomly chosen images are shown in Figure 6
5.3 Use case: Prioritized and Goverage-Oriented Explanations (medical data)
In this use case, we study binary classification of histopathological images into cancer vs, non-cancer. In medical imaging practice, clinicians need not only accurate predictions but also actionable, uncertainty- aware rationales that support priorilized review of the most reliable evidence and coveruge-oriented sweeps that reduce the risk of overlooking small foci (cf. Klauschen et al. (2018): : Hägele al (2020): Binder et al.)
(001))) Our UAI method summarizes agreement and coverage of explanatory fcatures across posterior samples, yielding (i) prioritized view (stable cores) and (ti) coveragc-oriented view (sensitive, broader evidence).
The dataset comprises 22,302 patches from anonymized non-small-cell lung cancer (NSCLC) cases (n=200) -Uniersitätsmeddzin Berlin, digitized drawn from routine diagnostics at the Institute of Pathology, annotated 28 morphological classes. For using a 3DHistech P1000 whole-slide scanner. Expert pathologists this use case, we reduce to binary carcinoma vs. non-cancer. Data are split into training and test partitions. The training set contains 17,884 labeled patches, with 69.9% non-cancer. Point-level annotations of cancer cells are available for subset of cancer images and are used only for localization evaluation (below).
For this experiment, we trained a VGG-16 (Simonyan & Zisserman 2014) network with an additional dropout regularization layer ap- plied in the feature extractor part of the network (after the first and third MaxPool layer). The network was trained in binary classification fashion for detecting the existence of cancer tissue in histological slide. We trained the network for 100 epochs, using the cross-entropy loss with stochastic gradient descent, where the initial learning rate was set to 0.001. The trained network achieved an accuracy of 86.96% on the provided lest dataset and an F1 score of 0.8205. Afterward, we computed the UAI explanations for differ- ent test images, which allows us to provide an additional estimate of the explanation uncertainty. Results for three different prototypical cancer images are shown in Figure 7 The original histopathological image is shown in the left column with the black dots representing expert-labeled cancerous cells.
Table 2: Localization of expert- annotated malignant cells (point, an- notations). We report AUC (mean + s.d.) for thresholded attribution maps versus cell annotations on cancer- labeled test images with available points. Higher percentiles (Union, large a) increase coverage and yield the best AUC; a=99 performs best here. Intersection (small a favors sta- bility /prit//preiiiio over coverage.
Method
AUC Score
Baseline
0.6534 + 0.1705
Average
0.6635 = 0.1712
In the right. column of Figure 7 the UAT results are plotted over the original image and highlight the relevant parts the image regarding their importance for the classifier. In the Intersection explanation (second column from the left), we show the regions in the image where the classifier is most certain about their relevance to the pro- diction "cancer", and with gray color, features that are absent from the Average explanation are highlighted. Analogously, for the Union explanation, we highlight features in green that are attributed posi- tively in the Union explanation but not in the Average explanation.
UAT
a ==
0.5960 + 0.1733
a' 5
0.6138 + 0.1742
25
0.6536 = 0.1716
a 50
0.6818 + 0.1711
Q 75
0.7026 + 0.1717
a 95
0.7179 + 0.1694
a = 99
0.7201 - 0.1680
Thus, we can visually observe differences between Intersection, Average, and Union explanations while Intersection explanations provide a user with more "conservative" explanations, the Union method allows us to observe all the features that were considered with positive evidence towards the class in question. From the quantitative results shown in Table 2 we can observe that the higher the percentile value, the larger the AUC score for the localization criteria of the cancerous cells. Note that in the quantitative experiment, we used only the images labeled as cancerous, where the annotations of the cancerous areas were available.
3To download subset. of the ImageNet dataset, the following library was used: httpe://github..com ImageNet--Date Daw.seta--ownlleeeeoaaa
13

---

Published in Transactions on Machine Learning Research (09/2025)
In general, the UAI-based heatmaps with different values prove particularly useful with respect to different diagnostic applications. Low value explanations can help to identify tissue regions with the highest likelihood of cancer. High percentile explanations may then be used for Al-based screening applications, where it is important not to overlook even the timiest occurrence of tumor cells in tissue samples. Therefore, combining UAT analyses with low and high a may provide high sensitivity and simultaneously points the pathologists to regions where the machine is most confident about its decision. This additional information will improve diagnostic speed and also reduce the risk of overlooking crucial information in the diagnostic process.
Input with annotated Cancer Colls
Intersection
Average
Uniion
UAI
= #140 62)
E 4.0
=
Figure 7: Comparison of the explanation results for the cancer experiment. VGG16 (Simonyan & Zis- serman 2014 was trained on a set of maemattxylin-ccoin-staaan as on non-cancer histological slides. Three original images labeled cancer, taken from the set, are shown the left, overlaid with black dots, which represent the cancer cells annotated by experts. Behind, the various explanations of the model prediction for the class cancer are shown from left to right inter- section, baseline, union, and UAI+ explanation. For reporting only significant positive relevances, we set a threshold at = 0.05 and visnalize only relevances that surpass this threshold. We highlight by grey and green the additional information gained by our union and intersection approach in comparison to the baseline explanation. In detail, the intersection highlights the most. certain areas, which are indicated as parts of the original image, whereas the areas that are not certain, and thus not visualized in the intersection explanation, are highlighted in grey. In contrast, the union explanation identifies additional information that not shown in the baseline explanation and therefore may point out new areas that may be cancerous, which are highlighted in green. The aggregation of the diverse levels of feature importance is summarized in the UAL, explanation shown on the right. Together, these views support a prioritized review workflow (Intersection) and a covergge-rriented workflow (Union) without retraining the model.
Lung adeno carcinoma (LUAD) as well
5.4 Use case: Confirming the Clever Hans Effect
In this experiment, we revisit the work of Lapuschkin et al. (2019): Samek et al. (2021); KaufTmann el al. (2025) on the Clever Hans effect. This term refers to a problematic solution strategy in which a model achieves correct predictions for the wrong reasons. The name originates from horse named Hans, which appeared to solve arithmetic problems, but was in fact responding subtle cues from its trainer rather than performing actual calculations.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Input: cazton
Intersection
Union
Inpuc: vonicoc
Intorsection
Union
imagetiet VS6 18
RW25
sg
Imputt:
Intersection
Union
Ippu::
Intersection
Union
Intersection
Input: Intersection
Union
UAP
Parcel VOC 2007.
VSG 16
Figure Illustrative visualization of the Clever Hans effect in two datasets: ImageNet and Pascal VOC 2007. The UAI explanations assist in distinguishing between random artifacts in the altribution maps and systematic behavior learned by the model. The enhanced method, UAI+, allows to confirm the Clever Hans effect with high confidence (highlighted in red) in both cases: for ImageNet, where a spurious correlation arises due to a Chinese-language watermark at the center of the image, and for Pascal VOC 2007, where the model relies on photographer watermarks containing name and website (bottom, left part of the images).
In modern machine learning, Clever Hans effect typically involves the model relying spurious correlations or artifacts-sin such watermarks- that are coincidentally associated with specific classes. For instance, an irrelevant visual pattern might consistently appear in training images of one class, leading the model to learn this pattern as a discriminative feature (Lapuschkin et al. 2016 2019).
We conducted our experiments on two datasots: Pascal VOC 2007 and ImageNet. Prior work (Lapuschknn et al.] 2019)) has shown that models trained on Pascal VOC 2007 exhibit Clever Hans behavior for the class horse, often relying photographer watermarks. In the case of ImageNet, Bykov et al. (023d identified spurious correlations between Chinese-language watermarks and certain classes such "carton" and "monitor". For both experiments, we used the VGG16 architecture: model retrained from scratch for Pascal VOC 2007 (see Appendix A.3.3) and standard pretrained model for ImageNet.
Figure presents the UAI explanations for both datasets. In both cases, watermarks appear prominently in the explanations within the top 5th percentile. This means that for least 95% of the model samples, the model consistently identifies the watermark dictions. Specifically, Chinese watermarks are associated with "carton" and "monitor" in ImageNet, while photographer watermarks are linked to the "horse" class in Pascal VOC 2007.
a highly relevant. feature for the respective class pre-
These findings confirm the presence of the Clever Hans effect: the models rely on artifacts rather than semantically meaningful features. Importantly, our UAI method proves effective in identifying such sys- tematic misbehaviors, helping to differentiate between robust explanations and potential artifacts of the explainability technique itself (see also (Dombrowski et al. 2019 Heo et al.) 2019 Samek et al. 2019
6 Concluding Discussion
When lackling real-world learning problems, Bayesian models have been helpful in assessing the intrinsic uncertainties encountered when predicting. The field of XAI could introduce a further safety layer into the inference process when using neural networks because explanations can contribute to, e-g., unmasking flaws in model or data set (Lapuschkin et al. 2019). So far, however, no XAI method for Bayesian Neural Networks conceived.
15

---

Published in Transactions on Machine Learning Research (09/2025)
In this paper, connect BNNs (with built-in uncertainty quantification) and XAI by proposing a practical, method-agnostic framework for explaining BNNs. As demonstrated, our novel technique (called UAT) is applicable to several popular explanation methods.
By appropriately averaging sampled relevance maps, we can obtain an explanation for the expected predictive function. This allows us to shed light on the decision-making process of BNNs: interestingly, UAT allows us to not only inspect the most relevant pixels for a decision but also their (un)certainties. Our method is thus formidable starting point for obtaining novel insight into the behavior of Bayesian learning models.
We found that, with high parameter range of a, UAT explains the behavior of the network more infor- matively, in comparison to a standard mean explanation baseline. By gauging a, users can understand the rationale behind a network: with small parameters of a, we can observe what features are considered to be contributing towards the prediction regardless of the sampled strategy (intersection explanation). With high parameters a, we can understand the features for which at least a small fraction of strategics attribute them with positive relevance (union explanation). Thus, by choosing the parameter a, users can choose bet ween more less risk-averse explanations, depending on the task objective. The proposed UAI explanation ad- ditionally helps to understand and reflect the multiple explanation modes inherent in Bayesian ensemble (cf. Figure 2
The computational complexity of our presented method is linear in the number of posterior samples. Tn practice, we found that as few as 100 samples are sufficient to obtain a stable assessment of explanation uncertainty, which keeps the computational cost low. While more samples can be used for finer-grained analysis, the default choice of 100 samples strikes balance between accuracy and efficiency, which makes our approach fcasible even for large-scale applications.
Our contribution has several limitations: (i) conclusions are conditioned on the chosen posterior approxima- tion (e.g., MC Dropout, Laplace) and do not claim recovery of true posterior multi- modality; (ii) fidelity-style metrics are not yet standardized for distributions of explanations; and (iii) a full human-in-the-lopp evalua- tion is beyond the scope of our work.
Concluding, our CAI framework extends existing attribution methods Bayesian Neural Networks by gen- erating and aggregating distributions of explanations. This enables rescarchers and practitioners to obtain complementary information about the variability of the model reasoning. Our results suggest that incorpo- rating epistemic uncertainty can adjust explanations toward different desiderata: for example, union aggre- gations improve sensitivity (AUC-ROC), while intersections emphasize stability (MA). Our work provides a foundation on which future research can build up in several directions. First, expanding evaluation criteria to better capture the properties of incertainty-wware explanations is necessary next step. Second, practical deployment scenarios deserve further exploration, particularly in safety-critical domains. For instance, in pathology, Bayesian XAI may reveal both spurious and real evidence separate clusters, potentially reduc- ing the risk that users overlook true signals when spurious patterns present. More generally, investigating how explanation diversity can support practitioners in identifying unstable reasoning represents promising direction toward application-speeific deployments.
Acknowledgements
This work was partly funded by the Federal Ministry of Research, Technology and Space (formerly BMBF) through the projects Explaining 4.0 (ref. 011$200551), REFRAME (ref. 011824073B), PIAD (ref. 011S24071A), and BIFOLD - Berlin Institute for the Foundations of Learning and Data (ref. BIFOLD24B). KRM was funded in part by the Institute of Information & Communications Technology Planning & Eval- uation (ITTP) grants funded by the Korea Government (No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University). MK acknowledges support by the DFG through FOR 5359 (KL. 2698/6. 1, KL 2698/7-11, TRR 375 (ID 511263698), SPP 2298 (KL 2698/5-2), and SPP 2331 (KL 2698/11-1), and by the Carl-Zeiss Foundation through the initiatives AI-Care and Process Engineering 4.0.
16

---

Published in Transactions on Machine Learning Research (09/2025)
References
Amina Adadi and Mohammed Berrada. Peeking inside the black-box: survey on explainable artificial intelligence (xai). IEEE access, 6:52138-52108 2018.
Christopher J Anders, Grégoire Montavon, Wojeicch Samek, and Klaus-Robert Müller. Understanding patch-based learning of video data by explaining predictions. In Explainable Al: Interpreling, Explaining and Visualizing Deep Learning, pp. 297-309, Springer, 2019.
Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. what is relevant a text document?": An interpretable machine learning approach. PloS one, 12(8):00181142, 2017.
Leila Arras, Ahmed Osman, and Wojciech Samek. Ground truth evaluation of neural network explanations with clevr-xai. arXiv preprint arXiv:2003.07225, 2020.
Scbastian Bach, Alexander Binder, Grégoire on, Frederick Klauschen, Claus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), 2015.
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robee Müller. How to explain individual classification decisions. Journal of Machine Learning Rescarch, 11 (Jun): 1803 1831, 2010.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 0541-6549, 2017.
Alexander Binder, Michael Bockmayr, Miriam Hägele, Stephan Wienert, Daniel Heim, Katharina Hellweg, Masaru Ishii, Albrecht Stenzinger, Andreas Hocke, Carsten Denkert, et al. Morphological and molecular breast cancer profiling through explainable machine learning. Nature Machine Intelligence, 3(4):355-366) 366, 2021.
Christopher M. Bishop. Pullern recognilion and machine learning. Springer, 2006.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXi::2003.127 2023.
Varun H Buch, Irfan Ahmed, and Mahiben Maruthappu. Artificial intelligence medicine: current trends and future possibilities. British Journal of General Practice, 68(668):143 144, 2018.
Vanessa Buhrmester, David Münch, and Michael Arens. Analysis of explainers of black box deep neural networks for computer vision: A survey. arXie preprint arXw::111.12211, 2019.
Kirill Bykov, Anna Hedström, Shinichi Nakajima, and Marina M-C Höhne. Noisegrad: enhancing explana- tions by introducing stochasticity to model weights. arXie preprinl ar Xixi222006. 10185, 2021.
Kirill Bykov, Mayukh Deb, Dennis Grinwald, Klaus Robert Muller, and Marina MC Höhne. DORA: Explor- ing outlier representations in deep neural networks. Transactions on Mochine Learning Research, 2023a. ISSN 2835-8856. URL httres//eoppareeesss..eerrrrrrrd wtt.:e/eeooorrmmmeeefoeeeeeeee
Kinill Bykov, Laura Kopf, and Marina M-C Höhne. Finding spurious correlations with function-semantic contrast analysis. In World Conference on Explainable Artificial Intelligence, pp. 549 572. Springer, 2023b.
Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina M-C Höhne. Labeling neural representations with inverse recognition. arXiv preprint arXiv:2311 2023c.
Kirill Bykov, Klaus-Robert Müller, and Marina M-C Höhne. Mark my words: Dangers of watermarked images in imagenet. Xie preprint or arXin:2303.05/000 2023d.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Lucy R Chai. Uncertainty estimation in bayesian neural networks and links to interpretability. Master of Philosophy (University of Cambridge), 2018.
Mariana CA Clare, Maike Sonnewald, Redouane Lguensat, Julie Deshayes, and Venkatramani Balaji. Ex- plainable artificial intelligence for bayesian neural networks: Toward trustworthy predictions of ocean dynamics. Journal of Advances in Modeling Earth Systems, 14(1)):20022110033112 2022.
Angel Cruz-Roa, Ajay Basavanhally, Fabio González, Hannah Gilmore, Michael Feldman, Shridar Gane- san; Natalie Shih, John Tomaszewski, and Anant Madabhushi. Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks. In Medical Imaging 2014: Digital Pathology, volume 9041, pp. 904103. International Society for Optics and Photonics, 2014.
Arun Das and Paul Rad. Opportunities and challenges in explainable artificial intelligence (xai): A survey. arXiv preprint ar Xi::20006. 1 11311, 2020.
PS De Laplace. Mémoire sur les suites récurro-récurrentes et sur leurs usages dans théorie des hasards. Mém. Acad. Roy. Sci. Paris, 6:353-371, 1774.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icce, 2009.
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In Advances in Newral Information Processing Systems, pp. 13567-13572 2019.
Ann-Kathrin Dombrowski, Christopher J Anders, Klaus-Robert, Müller, and Pan Kessel. Towards robust explanations for deep neural networks. Pottern Recognition, pp. 108194, 2021.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009.
Tom Fawcett. An introduction to roc analysis. Pattern Recogn. Lett., 2768):611-874, June 2006. ISSN 0167-8655. doi: 10. ptt.://...atree.0/....
URL http::///do..rr/1100// 1016/ patrec..t.... 2005 100010,
Y. Gal and Z. Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of ICML, 2016.
Robert Geirhos, Jörn-Henriz Jacobsen, Claudio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665 673, 2020.
Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. filess/aper/20111
Curran Associates, Inc., 2011. URL https://proceeddnng www.eediiinnnnnninn..eeee File///8Bebbdd1111000000aaaaaaf3
Paper.ppppppp.p pdf
Dennis Grinwald, Kirill Bykov, Shinichi Nakajima, and Marina MC Höhne. Visualizing the diversity of representations learned by bayesian neural networks. Transactions on Machine Learning Research, 2023.
Miriam Hägele, Philipp Seegerer, Sebastian Lapuschkin, Michael Bockmayr, Wojciech Samek, Frederick Müller, and Alexander Binder. Resolving challenges in deep learning- based analyses of histopathological images using explanation methods. Scientific reports, 10:6423, 2020.
Klauschen,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance imagenet classification. In Proceedings of the IEEE international conference compuler vision, pp. (026-1033) 2015.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ar Xio preprint or 1xx::16100001..0 2016.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversarial model manipulation. Advances in Neural Informalion Processing Systems, 32:2925 2936,, 2019.
Andreas Holzinger, Georg Langs, Helmut Denk, Kurt Zatloukal, and Heimo Müller. Causability and explain- ability of artificial intelligence in medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, (4):c133,, 2019.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advonces in neural information processing systems, 32, 2019.
Matthias Humt, Jongscok Lee, and Rudolph Triebel. Bayesian optimization meets laplace approximation for robotic introspection. arXiv preprinl arXin:2000 16141, 2020.
Pavel Ixmailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson. On feature learning the presence of spurious correlations. Advances in Neural Information Processing Systems, 35:38516 38532, 2022.
Andrew Janowezyk and Anant Madabhushi. Deep learning for digital pathology image analysis: A compre- hensive tutorial with selected use cases. Journal of pathology informatics, 7, 2016.
Jacob Kauffmann, Jonas Dippel, Lukas Ruff, Wojeiech Samek, Klbert Müller, and Grégoire Montavon. Explainable ai reveals clever hans effects in unsupervised learning models. Nature Machine Intelligence, 2025.
Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternattributton. arXiv preprint
2017.
D. P. Kingma, T. Salimans, and M. Welling, Variational dropout and the local reparameterization trick. In Advances in NIPS, 2015.
Frederick Klauschen, K-R Müller, Alexander Binder, Michael Bockmayr, M Hägele, P Seegerer, Stephan Wienert, Giancarlo Pruneri, S de Maria, S Badve, et al. Scoring of tumor- iumor-infiltrating lymphocytes: From visual estimation to machine learning. In Seminars in cancer biology, volume 52, pp. 151-157. Elsevier, 2018.
Frederick Klauschen, Jonas Dippel, Philipp Keyl, Philipp Jurmeister, Michael Bockmayr, Andreas Mock, Oliver Buchstab, Maximilian Alber, Lukas Ruff, Grégoire Montavon, and Klaus-Robert Müller. Toward explainable artificial intelligence for precision pathology. Annual Review of Pathology: Mechanisms of Disease, 19:541-570, 2024.
Maximilian Kohlbrenner, Alexander Bauer, Shinichi Nakajima, Alexander Binder, Wojeiech Samek, and Sebastian Lapuschkin. Towards best practice in explaining neural network decisions with Irp, 2020.
Joamin Koo, Jungsuk Kwac, Wendy Ju, Martin Steinert, Larry Leifer, and Clifford Nass. Why did my car just do that? explaining semi-autonomous driving actions to improve driver understanding, trust, and performance. International Journal on Interactive Design and Manufacturing (IJIDeM), 9(4):269-2) 275, 2015.
Yongchan Kwon, Joong-Ho Won, Beom Joon Kim, and Myunghee Cho Paik. Uncertainty quantification using bayesian neural networks in classification: Application to biomedical image segmentation. Computational Statistics & Data Analysis, 142:106816, 2020.
Sebastian Lapuschkin, Alexander Binder, Grégoire Montavon, Robert Müller, and Wojeiech Samek. Analyzing classifiers: Fisher vectors and deep neural networks. In Proceedings of the 1EEE Conference Computer Vision and Pattern Recognation, pp. 2912-2920, 2016.
Scbastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire on, Wojeiech Samck, and Klaus- Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10:1096, 2019.
19

---

Published in Transactions on Machine Learning Research (09/2025)
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to docu- ment recognition. Proceedings of the IEEF, 56(11):2278-2333 1998.
Jongscok Lee, Matthias Humt, Jianxiang Feng, and Rudolph Triebel. Estimating model uncertainty of neural networks in sparse information form. In International Conference on Machine Learning (TCML). Procoodings of Machine Learning Rescarch, 2020.
B Léon. Online algorithms and stochastic approximations. Online Learning and Neural Networks; Cambridge University Press: Cambridge, UK, 1998.
Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one am- plifies others. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20071-20082. 2023.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neural networks via knowledge distillation for natural language understanding, arXiv preprint arXiv:1904.099 09442 2019.
Scott M Lundberg and Su-In Lec. A unified approach to interpreting model prodictions. In Advances in neural information processing systems, pp. 4765-4774, 2017.
Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable ai for trees. Nature Machine Intelligence, 2(1):2522 5839, 2020.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9NNv)):2579 2605, 2008
W. J. Maddox, P. Tamailov, T. Garipov, D. P. Vetrov, and A. G. Wilson. A simple baseline for bayesian uncertainty in deep learning. In Advances in NewrIPS, 2019.
Charles Marx, Youngsuk Park, Hilaf Hasson, Yuyang Wang, Stefano Ermon, and Luke Huan. But are you sure? uncertainty-aware perspective on explainable ai. In International conference artificial intelligence and slatistics, pp. 7375-7391. PMLR, 2023.
Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery, Nature, pp. 6, 2023.
D. Molchanov, A. Ashukha, and D. Vetrov. Variational dropout sparsifies deep neural networks. In Proceed- ings of ICML, 2017.
Grégoire Montavon, Wojciech Samek, and Klaus- Robert Müller, Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojeiech Samck, and Klaus-Robert Müller. Layer-wise relevance propagation: an overview. In Erplainable AI: Interpreling, Explainèng and Visualizing Deep Learning, pp. 193 209. Springer, 2019.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153-17163, 2020.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in neural information processing systems, pp. 3387-3395, 2016.
Tuomas Oikarinen and Tsui- Wei Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. arXiv preprint arXim:2204.10968 2022.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization: How neural networks build up their understanding of images. distill, 2018.
20

---

Published in Transactions on Machine Learning Research (09/2025)
K. Osawa, S. Swaroop, A. Jain, R. Eschenhagen, R. E. Turner, R. Yokota, and M. E. Khan. Practical deep learning with Bayesian principles. In Advances in NewrIPS, 2019.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018-Conferener Track Proceedings, volume 6. International Conference on Representation Learning, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-285) 122112222,,222 2015.
Rabia Saleem, Bo Yuan, Fatih Kurugollu, Ashiq Anjum, and Lu Liu. Explaining deep neural networks: A survey on the global interpretation methods. Neurocomputing, 2022.
Wojcicch Samek, Alexander Binder, Grégoire on, Scbastian Lapuschkin, and Claus-Robert Müller. Evalu- ating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(11):2660 2673, 2016.
Wojeiech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller (eds.). Explainable A1: Interpreting, Explaining and Visualizing Deep Learning, volume 11700 of Lecture Notes in Computer Science. Springer, 2019.
Wojeiech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J Anders, and Klaus-Robert Müller. Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 13):4) 278, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 218-626, 2017.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 429(7587):484-488) 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.
arXiv preprint 4rX:t:1409.1555 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1812.6000 6034, 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: 100.170.0000 038603 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. arXio preprint arXiv:1703.0130 2017.
Erico Tjoa and Cuntai Guan. A survey on explainable artificial intelligence (xai): Toward medical xai. IEEE Transactions on Newral Networks and Learning Systems, 2020.
Marina M-C Vidovic, Nico Görnitz, Klaus-Robert Müller, Gunnar Rätsch, and Marius Kloft. Opening the black box: Revealing interpretable sequence motifs in kernel-based learning algorithms. In Joinl European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 137-153. Springer, 2015.
Marina M-C Vidovic, Nico Görnitz, Klaus-Robert Müller, and Marius Kloft. Feature importance measure for non-lincar learning algorithms. arXiv preprint
2016.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yoo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arxiv preprint Xiv 708. 04 782, 2017.
21

---

Published in Transactions on Machine Learning Research (09/2025)
Chrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17(4):395 416, 2007.
Wenzel, K. Roth, B. S. Veeling, J. Swiatkowski, L. Tran, Mandt, Snoek, T. Salimans, R. Jenatton, and S. Nowozin. How good is the bayes posterior in deep neural networks really? arX1v:2002.02405, 2020.
Gesa Wiegand, Matthias Schmidmaier, Thomas Weber, Yuanting Liu, and Heinrich Hussmann. I drive-you trust: Explaining driving behavior of autonomous cars. In Extended Abstracts of the 2019 CHT Conference on Human Factors in Computing Systems, pp. 1 6, 2019.
A. G. Wilson and P. Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In Advances in NewrlPS, 2020.
Andrew Gordon Wilson. The case for bayesian deep learning. arXiv preprint arXie:2001. 2020.
Dong-Ok Won, Klaus-Robert Müller, and Seong-Whar Whan Lee. An adaptive deep reinforcement learning frame- work enables curling robots with human-like performance in real-world conditions. Science Robotics, 5 (46), 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image datasct for benchmarking machine learning algorithms. arXiv preprint arXiv: NSG::7008.777 077.07777 2017.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint orXiv:1506.068 2015.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan SclarofT. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10):1084 1102, 2018.
Xu Zheng, Farhad Shirani, Zhuomin Chen, Chaohao Lin, Wei Cheng, Wenbo Guo, and Dongsheng Luo. F- fidelity: A robust framework for faithfulness evaluation explainable al. arxiv preprint ar urXiv:2,10.0 00.02970, 2024.
22

---

Published in Transactions on Machine Learning Research (09/2025)
A Appendix
In the following, we provide the supplementary material to our paper Explaining Bayesian Neural Networks.
A.1 Proof of Theorem 3.1
Proof. For LRP-0, it holds that
(w(x Ta (fw)
(10)
with [x] =x v.(/w((x). Furthermore, the linearity
TT(fi(())) + +2T.fll()))))) Talnfi refal(z)
holds for any TI, T2 and any weakly diffcrentiable functions f1, f2 except the non-strongyy-diffeentabbe points of and f2- We will prove, that in this case
Ta [w [fw]l (2) Ew [Talf]]<>)]
(11)
holds except for measure zero points.
Let us fix If fw is strongly differentiable at for all W the support of its posterior distribution g(W), linearity of To holds and therefore Equation 1 holds due to Lemma 1. Let us define
W₂ = (W; fw is not strongly differentiable at 2}, z= = faz / 6(W e WW) > 0}.
where 8()) denotes the Dirac measure. Equation still holds for x E 2, because the contribution from the set of non-differentiable models at x is zero. For any bounded distribution q(x) in the input space, holds that
a(()) f 6(W e Wa)g(W)dWdd dr =
= //(W))//)))
((=))(W (()))(( W.)drdW
due the weakly differentiable assumption on fw. This implics that 2 is a mcasure zero set, which proves the claim for LRP-0.
Similarly, the gradient explanation, well IG, can be written equation 10 with operator Ta linear on all stoongi-diifeeenttablepooi and the same discussion applics to proving the claim.
A.2 Details of the clustering procedure
For the image classification task we employ the SpRAy algorithm follows
Lapuschkin et al. 2019
Relevance maps sampling
collection [ of relevance maps is sampled from the posterior distribution.
2. Preprocessing of the relevance maps.
All relevance maps normalized using the MinMax normalization procodure. If needed, all relc- vance maps should be made uniform in shape and size, for future clustering.
To speed up the clustering process and to produce more robust. results we propose to use down- sampling methods the collection of samples. While the user is free to choose any dimensionality reduction method, we propose to use the Average Pooling method in order to achieve visually dif- ferent strategies in different clusters.
3. Spectral Cluster (SC) analysis pre-processed relevance maps.
Pre-processed relevance maps are clustered by Spectral Clustering (SC) method. The affinity matrix, which is necessary for SC method, based on k-nearest-neigbborhodd relationships. More detailed,
23

---

Published in Transactions on Machine Learning Research (09/2025)
the affinity matrix M (ma) (mi)) measures the similarity my between all N samples R. and R2 of a source dataset and is constructed in the following way:
{1 if R2 is among the k nearest neighbors of R2
mij
else
Since this rule is asymmetric, the symmetric affinity matrix M is created by taking my mar(miy, Mjj) Authors of the original paper highlight that similar clustering results were obtained using the Euclidean distance, with only small differences in the eigenvalue spectra.
The Laplacian L is computed from M as follows:
d; = Σ maj.
= diag [d1, .2., dy]
= - M.
The Matrix D is diagonal matrix, which describe the measure of connectivity of a particular sample i with D being diagonal matrix with entries da describing the degree (of connectivity) of sample i (Lapuschkin et al. 2019
matrix,
Identification of intorcsting clusters by cigengap analysis
By performing an cigenvalue decomposition on the Laplacian L, cigenvalues \1, A2, AN are ob- lained. The number of eigenvalues A = 0 identifies the number of (completely) disjoint clusters within the analyzed set of data.
The final step of SpRAy assigns cluster labels to the data points, which can then be performed using an (arbitrary) clustering method: in our work we use k-means clustering on the N eigenvectors. The number of clusters can be obtained by eigengap analysis (Von Luxburg 2007): can be identified by eigenvalues close zero as opposed exactly zero, followed by an eigengap rapid increase the difference between two eigenvalues in the sequence [At41ds] (Lapuschkin et al. 2019
A.3 Experimental setup
A.3.1 CMNIST experiment
For the CMNIST experiment used simple convolutional network similar to LeNet (LeCun el. al., 1998). For the Ensemble and for the Laplace scenarios standard architecture was used, with only change in number of input channels adjusted to 3-dimensional RGB inputs. For the Dropout scenario, after 2 Average Pooling layers, a 2-d Dropout layer was inserted with dropout probability set to 0.25. 1D Dropout layers were added in the classification part of the net work, with the probability of dropout set to 0.5.
All of the networks were trained with a batch size 32, and with a Stochastic Gradient Descent algo- rithm, (Léon] 1998) with a learning rate of 0.01 and 0.9 momentum. A learning rate scheduler was used with the number of steps set to 7 and multiplicative parameter y 0.1. For the Ensemble scenario, 100 networks were trained for 20 epochs, and for the Laplace and Dropout, the number of epochs was set to 100. For the Laplace approximation, KFAC Laplace approximation was used (Humt et al. 2020 Lee et al. 2020 with Laplace regularization hyperparameters (additive and multiplicative) both set to 0.1.
A.3.2 Carcinoma experiment
For the Cancer experiment, we employed a standard VGG-16 (Simonyan al. 2013) with additional 2D Dropout layers after each MaxPooling layer, with the probability of dropout set to 0.1 and 1D
24

---

Published in Transactions on Machine Learning Research (09/2025)
Dropout layers in the classificator part of the network after each activation function with the prob- ability of 0.5. The network was trained with SGD with 0.001 learning rate and 0.9 momentum. torch. optim saclll___c__eee schedulerr heduceeennnlllaaaeeeeennnna rate scheduler was used with the factor of 0.1 and patience of 10.
A.3.3 Clever Hans experiment
For the Pascal VOC 2007 multi-label classification experiment, employed standard VGG16 network Si- monyan & Zisserman 2014), and adjusted the number of output neurons from 1000 to 20, which is the number of different classes in the Pascal VOC 2007 dataset.
We resized each training image, such that the shorter axis has 224 pixels, keeping the aspect ratio unchanged. Then, we randomly cropped the longer axis and obtained square images with the size 224x224. We trained the network for 60 epochs, by minimizing the Binary Cross Entropy loss preceded with Sigmoid layer We used the Adam optimizer with its paramcters set to 0.0001, 31 0.9, f2 0.999. Our trained VGG16 network achieves 91.6%% in the multi-label classification on the test set, for which center cropping with square size of 224x224 was applied, instead of random cropping.
A.3.4 Fashion MNIST experiment
For the Fashion MNIST experiment trained LeNet network with 2 2D Dropout layers with p 0.5 added after each of the Average Polling layers in the feature extractor part of the Network, and 1D Dropout layer with 0.5 after the Flatten Layer. The network was trained on MNShionMNIST dataset with several augmentations, such as Color Jittering, Random Affine Transformations, and Random Horizontal Flips. The batch size was set to 64, the Network was trained using the Cross-Entropy loss for 50 epochs with Adam optimizer with standard parameters and a learning rate of 0.001.
25


Reviews:
[{"review_id": "OUZRIcSaER", "reviewer": "Reviewer_eXSm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well-written and clearly presents the idea.\n\nThe concept is relatively novel and offers useful insights for the research community.\n\nThe experimental results support the proposed approach.\n\nWeaknesses:\n\nWhile the model performs well, its computational cost is high. Sampling from the posterior distribution leads to significant expense.\n\nThe variations in the explanations observed could be due to local disturbances around a single mode, rather than indicating fundamentally different predictive approaches.\n\nI believe it would be better to employ more standard metrics for evaluating XAI, instead of relying solely on AUC-ROC and Mean Average Precision (MAP)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an explainable AI (XAI) method for Bayesian Neural Networks (BNNs), building on existing XAI techniques for Deep Neural Networks (DNNs). The authors aim to quantify the uncertainty in model predictions. They evaluate their method using datasets from ImageNet and cancer research, demonstrating that their approach works efficiently.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Computational Cost:\n   While the model demonstrates strong performance, its computational cost is notably high. I recommend exploring strategies to reduce the expense associated with sampling from the posterior distribution, such as approximation techniques or more efficient sampling methods, to make the approach more feasible for large-scale applications.\n\nExplanation Diversity:\n   The observed diversity in the explanations might stem from local perturbations around a single mode, which may not truly reflect distinct predictive strategies. It would be beneficial to further analyze whether these variations are genuinely due to different predictive strategies or simply artifacts of the perturbation process. A clearer explanation of how the model handles multiple modes in the data could strengthen this aspect of the paper.\n\nEvaluation Metrics:\n   I suggest incorporating more standard evaluation metrics for XAI, such as Fidelity, Stability, or Interpretability, alongside the AUC-ROC and Mean Average Precision (MAP). Using these additional metrics would provide a more comprehensive assessment of the model's explanatory power and could make the findings more robust.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6afThZnz0d", "reviewer": "Reviewer_A7do", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The framework incorporates uncertainty estimates into explanations, providing a more nuanced understanding of feature relevance and model confidence, which is particularly valuable in safety-critical domains.\n\n2. The method leverages the probabilistic nature of Bayesian Neural Networks, allowing explanations to reflect the inherent epistemic uncertainty in model predictions and feature attributions.\n\n3. Extensive experiments on synthetic, natural, and medical datasets demonstrate versatility and practical relevance, showing the method's robustness in various contexts.\n\nCons:\n1. Sampling from the posterior distribution to generate multiple explanations increases computational cost compared to single-point explanation methods, which may impede real-time applications.\n\n2. There is no time complexity analysis both theoretically and empirically.\n\n3. The authors only include metrics AUCROC and MA are used in the exp, however, for xAI, some more popular metrics such as fidelity. For example, the following papers are discussing it with fidelity:\nA benchmark for interpretability methods in deep neural networks. NeurIPS, 2019\nF-FIDELITY: A ROBUST FRAMEWORK FOR FAITHFULNESS EVALUATION OF EXPLAINABLE AI. ICLR, 2025.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new framework called Uncertainty-Aware Explanations (UAE) tailored for Bayesian Neural Networks (BNNs). It addresses the challenge of quantifying and visualizing the uncertainty inherent in model explanations, a facet largely overlooked by traditional explainability methods primarily designed for deterministic models. Basically, the idea is novel and the paper is well-structured. The theoretical depth is also good. However, the experiments might be improved.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There is no time complexity analysis both theoretically and empirically. Give more discusson on them.\n\nThe authors only include metrics AUCROC and MA are used in the exp, however, for xAI, some more popular metrics such as fidelity. For example, the following papers are discussing it with fidelity:\nA benchmark for interpretability methods in deep neural networks. NeurIPS, 2019\nF-FIDELITY: A ROBUST FRAMEWORK FOR FAITHFULNESS EVALUATION OF EXPLAINABLE AI. ICLR, 2025.\nPlease add more results on fidelity, ROAR, F-Fidelity for comparison if possible, or explain why these are not suitable.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "grdGjiTATo", "reviewer": "Reviewer_Na4v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Paper is well-written and easy to follow overall.\n\n- The histopathology and watermark examples help provide concrete examples over benchmark datasets.\n\n- Understanding uncertainty of explanation is an important facet of explanation in general.\n\n- The basic idea is simple and can be applied to standard DNNs with dropout.\n\n**Weaknesses**\n- The paper (like many attribution-based explanation papers) lacks concrete use-cases of when the explanation will actually be useful to someone. There are a fair bit of statements claiming (without direct evidence) that uncertainty in explanation would be helpful for clinicians or practicioners. But in what way? What decision could it actually help? \n   - The two concrete use cases seem to be: (1) Histopathology prediction and (2) training data debugging (i.e., watermark). Note that these are two very different use cases with very different users. \n   - In the first, the user is the model developer. What other ways could they find \"problematic\" images or wrong attributions in the images? And how would they actually use this? If they are not given ground-truth segmentation boxes, then how would they use your explanation method? Would they have to look at the explanations for every image? This seems time consiuming and probably too difficult in practice. They could also use explanation methods that select important training samples for the prediction. https://arxiv.org/pdf/2212.04612\n   - In the second, the user is the pathologist. In this case, how is showing them more or less information helpful? You need an example where the explanation would confirm the model's prediction AND a case where the explanation would make the pathologist REJECT the model's prediction. If only the first, then it's just reinforcing the bias of the model. You need an example of both cases. AND you need to show that a non-Bayesian approach would fail in one or both of these.\n\n- Relatedly, for those use cases, what are the most natural alternative solutions to the use-case? For example, you would need to show that your method on average would help a histopathologist correctly diagnose patients more often than using a non-Bayesian approach. That should be the goal. In practice, you would need a way to provide at least a proxy measure for this. You might consider an experiment like the one in https://arxiv.org/pdf/1901.09392, where they use a very carefully designed controlled experiment to showcase things. Basically, you need a way to simulate the use-case you care about and show that the best alternative (i.e., non-Bayesian) doesn't solve the task but yours does.\n\n- The above points question the mismatch between the framing of the problem, e.g., \"BNN ...evaluate their faithfulness\", and the evaluation metrics. Figure 1 shows variances of explanation...but why is that important? Yes, the approach can produce a distribution over explanations. But why is that useful for the use cases? Why would having this information actually change what you do?\n   - The evaluation metrics do not match this goal. Merely checking if the explanation overlaps with the object does not actually measure what is needed because the explanation method could game the metric by simply highlighting the foreground object all the time regardless. See e.g., the many papers on adversarial XAI: https://github.com/hbaniecki/adversarial-explainable-ai.  While this might not be true for your method, the metric doesn't seem to measure what is important.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to provide explanations for Bayesian Neural Networks (BNNs).\nThe main idea is that with linear operators functions (or ReLUs almost everywhere), the explanation of the posterior distribution is simply the distribution of explanations.\nThis simplifies the story to simply generate a distribution over explanations using standard attribution techniques and then aggregating these individual explanations.\nThe paper proposes to aggregate based on the percentiles of relevance scores for each pixel (rather than mean or variance of pixel scores) to showcase the variance.\nThe intersection and union parts are based on setting different values of the percentiles.\nFurthermore, the paper claims that this can be applied to standard DNNs that have dropout layers.\nThe paper shows results on how the various explanations method compare by seeing how much of the object the explanation covers and how much of the non-object it covers.\nThe paper illustrates on some toy datasets and several realistic datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Give concrete use-cases (likely framed as a decision problem) with concrete users and explain precisely why the uncertainty matters. You should include cases where the explanation properly increases the confidence in the prediction and where it DECREASES the confidence in the prediction appropriately. It must do both, otherwise it could be fooling the user or be completely useless.\n\n- Explain how the metric is a (proxy) measure for simulating the use case that you mention. Why does a better score on this metric imply that your method would perform better on the use case? Further, make sure to include best alternatives like non-Bayesian and possibly other explanation approaches.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6aWq0XeXcX", "reviewer": "Reviewer_ovyt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Claims and evidence: the maths and the claims about the algorithm performance appear correct, with the authors giving a number of points in support of the claims (theoretical justification, new method, multi-modality of BNNs, generality)\n- Audience: the intersection of uncertainty quantification  and interpretability, and BNNs in particular, is undoubtedly of interest\n\nWeaknesses:\n\nRight now, with both of the strengths above there are caveats which I would hope the author can address. \n\n**Claims and evidence:**\n\nIt seems that the claims are put from the perspective that the combination of BNNs and interpretability is a novel one. However, the authors need to confirm why these claims are not straightforward combination of BNNs and interpretability. Another TMLR work[3] provides a range of empirical findings about the choice of uncertainty quantification (dropout, posterior approximation),  as well as findings on the link between uncertainty and diversity of features. The theoretical and empirical claims, however, are presented as the initial steps towards BNNs and interpretability, despite that in recent years, these steps have been made. In particular, I wonder if the authors could contrast the differences (and demonstrate their importance) compared to [1]-[2]. In particular, I would expect to see both comparisons in Section 6 and Sections 3-4. \n\n**Audience:**\n\nFirst, it seems like a number of other papers have already looked into this question from different perspectives (see, e.g. works [1-3]). The audience would wish to find new insights which cannot be obviously concluded  from other papers. In other words, the authors need to clearly emphasise what this work show in a different light form the existing studies of uncertainty quantification.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes the explanation methods for Bayesian neural networks. It claims that while the research focuses mostly on the non-Bayesian deep neural networks, the Bayesian neural networks can provide a new way of interpreting predictions in neural networks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be important to contrast and compare with some of the works addressing a similar challenge such as:\n[1] Clare, M. C., Sonnewald, M., Lguensat, R., Deshayes, J., & Balaji, V. (2022). Explainable artificial intelligence for Bayesian neural networks: Toward trustworthy predictions of ocean dynamics. Journal of Advances in Modeling Earth Systems, 14(11), e2022MS003162.\n\n[2] Marx et al (2023) But Are You Sure? An Uncertainty-Aware Perspective on Explainable AI, AISTATS 2023\n\n[3] Grinwald et al (2023) Visualizing the Diversity of Representations Learned by Bayesian Neural Networks, TMLR, 2023\n\nThe answers on the questions about Claims and Evidence and Audience are currently No, which is subject to the rebuttal responses.", "broader_impact_concerns": "No further concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "20v1eMPMrU", "reviewer": "Reviewer_noQ2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. Method-agnostic design allows integration with any standard explanation method or BNN inference approach, enhancing flexibility and usability.\n2. The use of relevance map clustering provides an intuitive and effective way to explore predictive diversity in BNNs.\n3. The experimental evaluation demonstrates broad applicability across different inference methods and datasets.\n4. This paper is well-written and well-organized.\n\n**Weaknesses:**\n1. The method relies on approximate inference schemes that may not capture the true multimodality of the posterior, especially in the case of unimodal approximations like Laplace.\n2. The observed diversity in explanations may result from local perturbations around a single mode rather than genuinely distinct predictive strategies.\n3. The approach lacks a principled way to distinguish between epistemic uncertainty and actual mode switching in decision mechanisms.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method-agnostic framework for explaining the decision-making process of Bayesian Neural Networks (BNNs). The approach transforms existing attribution methods for deterministic neural networks into local explanation tools applicable to BNNs by aggregating and analyzing multiple relevance maps sampled from the posterior. Through clustering these explanations, the method reveals the implicit multi-modality in BNN predictions, where different parameter samples may reflect distinct decision strategies. The framework is compatible with various approximate inference techniques, such as Laplace approximation, MC Dropout, and Deep Ensembles.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The central claim that your method reveals the multi-modality of BNN decision-making is compelling but potentially overstated, given that many of the inference techniques used (e.g., Laplace approximation, MC Dropout) are unimodal or cover limited posterior regions. Clarifying the distinction between local explanation variability and genuine multimodal behavior is important to avoid overinterpretation. Please revise the narrative to acknowledge the limitations of the inference methods used and adjust claims about multi-modality accordingly, unless supported by more expressive posterior approximations.\n2. The explanation diversity observed in your results may reflect epistemic uncertainty due to weight variability rather than distinct predictive strategies. Clarifying this distinction is essential for supporting your main claim. Consider proposing a metric or analysis—e.g., relating clusters to prediction differences or uncertainty quantification—that distinguishes between these two phenomena.\n3. While the clusters of relevance maps are visually distinct, it is unclear whether they correspond to meaningful differences in model behavior (e.g., predicted class, spatial focus, or decision boundary shift). A quantitative evaluation—such as comparing prediction confidence or class probabilities across clusters—would strengthen the argument that these clusters reflect distinct decision strategies.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)