Visualizing the Diversity of Representations Learned by Bayesian Neural Networks
Authors: Dennis Grinwald
Kirill Bykov
Shinichi Nakajima
Marina MC Höhne
Venue: Accepted by TMLR ()

Abstract: Explainable Artificial Intelligence (XAI) aims to make learning machines less opaque, and offers researchers and practitioners various tools to reveal the decision-making strategies of neural networks. In this work, we investigate how XAI methods can be used for exploring and visualizing the diversity of feature representations learned by Bayesian Neural Networks (BNNs). Our goal is to provide a global understanding of BNNs by making their decision-making strategies a) visible and tangible through feature visualizations and b) quantitatively measurable with a distance measure learned by contrastive learning. Our work provides new insights into the posterior distribution in terms of human-understandable feature information with regard to the underlying decision-making strategies. The main findings of our work are the following: 1) global XAI methods can be applied to explain the diversity of decision-making strategies of BNN instances, 2) Monte Carlo dropout with commonly used Dropout rates exhibit increased diversity in feature representations compared to the multimodal posterior approximation of MultiSWAG, 3) the diversity of learned feature representations highly correlates with the uncertainty estimate for the output and 4) the inter-mode diversity of the multimodal posterior decreases as the network width increases, while the intra-mode diversity increases. These findings are consistent with the recent Deep Neural Networks theory, providing additional intuitions about what the theory implies in terms of humanly understandable concepts.

Published in Transactions on Machine Learning Research (11/2023)
Visualizing the Diversity of Representations Learned by Bayesian Neural Networks
Dennis Grinwald
Genniss.rinauudd@@tthhiii...
Machine Leorning Group
Technical University of Berlin, Berlin, Germany
BIFOLD - Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
Kirill Bykov
khykeo@aub-potsdaam.cmm
Understandable Machine Intelligence Lab
Leibniz Institute for Agriculture and Bioeconomy (ATB), Potsdam, Germany
Technical University of Berlin, Berlin, Germany
BIFOLD Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
Shinichi Nakajima
nakejima@@tu-belle de
Machine Learning Group
Technical University Berlin, Berlin, Germany
BIFOLD Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
RIKEN Center for AIP, Tokyo, Japan
Marina M.-. Höhne
mboccnne@ad--ootsaamn.
Understandable Machine Intelligence Lab Leibniz Institute for Agriculture and Bioeconomy (ATB), Potsdam, Germany
Technical University of Berlin, Berlin, Germany
BIFOLD Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
Reviewed on OpenReview: http:://ppnnrrsssee////////xxxx
Abstract
Explainable Artificial Intelligence (XAI) aims make learning machines less opaque, and offers rescarchers and practitioners various tools to reveal the decision-making strategics of neural networks. In this work, investigate how XAI methods can be used for exploring and visualizing the diversity of feature representations learned by Bayesian Neural Networks (BNNs). Our goal is to provide a global understanding of BNNs by making their decision- making strategics a) visible and tangible through feature visualizations and b) quantitatively measurable with a distance measure learned by contrastive learning. Our work provides new insights into the posterior distribution in terms of human-understandable feature information with regard to the underlying decision-making strategies. The main findings of our work the following: 1) global XAI methods be applied to explain the diversity of decision- making strategics of BNN instances, 2) Monte Carlo dropout with commonly used Dropout rates exhibit increased diversity in feature representations compared to the multimodal posterior approximation of MultiSWAG, 3) the diversity of learned feature representations highly correlates with the uncertainty estimate for the output and 4) the inter-mode diversity of the multimodal posterior decrcases the network width increascs, while the intra-mode diversity increases. These findings consistent with the recent Deep Neural Networks theory, providing additional intuitions about what the theory implies in terms of humanly understandable concepts.

---

Published in Transactions on Machine Learning Research (11/2023)
1 Introduction
Despite the great success of Doep Neural Networks (DNN), little is known about the decision-making strategies that they have learned. This lack of transparency is a major cause of concern since DNNs are being used in safety-critical applications [Berkenkamp et al. 2017 Huang & Chen 2020 Le et al. 2021 and it has been shown that they tend to encode fallacies, including the memorization of spurious correlations [Lapuschkin et al., 2019 Izmailov et al. 2022 or being biased towards the data set that they were trained on (Tommasi et al. 2017 Turner Lee| 2018 Ribeiro et al., 2016). Therefore, in recent years, the field of Explainable Artificial Intelligence (XAT) has established itself in order to make the decisions of AI models comprehensible to humans. XAI methods allow the user to "open" " the black-box of trained neural networks, that is, understanding what the model has learned and on which features its decisions are based.
Such post-hoc XAI methods can be further categorized into local and global explanation methods. While local XAI methods assign importance scores to input features, e.g. image pixels, that are important for a model's prediction (Sundararajan jan et al., 2017 Simonyan et al., 2014 Lapuschkin et al. 2015 Rs et al. 2017
Smilkov et al. 2017], global XAI methods aim to explain the inner workings of a DNN, e.g. what concepts
are encoded in its parameters, by providing feature visualizations (FVs) (Olah et al. 2017 2018 Bau et al., 2020). XAT methods have been applied extensively on many Deep Neural Network types that were trained in frequentist fashion, namely Convolutional Neural Networks (Lecun et al.
1998 Recurrent Neural Networks
Medsker & Jain 1999.. and others
Hilton et al. 2020 Schnake et al.
2021 Bau et al. 2020
In a recent work by Bykov et al. 2021 explanation methods have been applied to neural networks that were trained in a Bayesian fashion. Inherently, Bayesian Neural Networks (BNN) offer the property of uncertainty estimation that results from the diversity of learned feature representations and prediction strategies of different BNN instances. The authors apply local explanation methods on top of BNN instances and obtain a distribution over explanations, from which they estimate explanation uncertainties. Furthermore, such an approach of introducing stochasticity to the model parameters was shown to be successful in enhancing local explanations of standard models trained in a frequentist fashion (Bykov et al. 2022).
In our work, our primary objective is to explore the fcasibility of utilizing global explanation methods for Bayesian Neural Networks. We aim to answer the following questions:
Can the diversity of BNN instances be explained by global XAI methods?
2. Does the choice of the Bayesian inference method affect the diversity of their feature visualizations?
3. Can the uncertainty estimates provided by a BNN be explained by the diversity of their feature visualizations?
4. How does the network width affect the diversity of explanations of samples from multimodal posterior distribution?
The first question is to make sure that the properties of Bayesian ensembles can be visualized. The other three questions are related to hot topics in Bayesian Deep Learning, i.c., scalable posterior approximation, reliable uncertainty estimation, and the loss surface of Deep neural networks that have been exploited in deep ensembles (Lalshminarayanan et al.) 2017) and their extensions (Wilson & Izmailov] 2020). By answering those questions, we show the capability of our approach in analyzing Bayesian ensembles of neural networks. To the best of our knowledge, we are the first to use global XAI methods to explain the diversity of BNN instances. Our results underpin the latest findings in the field of deep learning theory (Roberts el al. 2022 in a for the first time- illustrative way.
2 Background
In this section, we provide brief overview of Bayesian Neural Networks and global explanation methods, which are used in the subsequent sections.
2

---

Published in Transactions on Machine Learning Research (11/2023)
2.1 Bayesian neural networks
Bayesian machine learning estimates the posterior distribution over unknown variables rather than estimating single value. We focus on neural network classifiers, where the conditional distribution of the model is given
as
plulæ, 0) = SoftMax(fo(x))),
(1)
with fz) being the network output parameterized with 0. Let D == y())), (x(M) y())) x(0) E RD, y(m) E {0, 1) vn I be a training data set, where is the number of training samples, and C is the number of distinet classes. The label y is a one-hot encoded vector, where the veclor-entry corresponding to the correct class is and the rest is set to zero. Given a prior distribution p(0) the
network parameter, the posterior distribution is given by Bayes' theorem (Pearson et al. 1959:
p(0 p(D[0)p(0)
p(D(0)p(()
(2)
P(D) JD(D(0)(0)d0
Then, the predictive distribution ply" | D) for a test point x² given by marginalizing over all possible model parameter sctups:
"'""="")) / ply"l=", ())((()))
(3)
encoding the models uncertainty regarding the predicted label y² for given test point x².
Since the integrals in Eqs. 2 and (3) are computationally intractable for DNNs, several approximation
methods have been proposed (Mackay et al., 1999 Neal
2012
Graves 2011 Blundell et al. 2015 Zhang
et al. 2019). Those methods approximate the posterior
2 by ( p0 and estimate the predictive
13 with T Monte Carlo samples:
pl 1 D)
plu 0.), ~~~g(010)
(4)
T
The resulting averaged output probability vector of the BNN is usually referred to Bayesian Model Average (BMA).
Below we introduce four popular methods for approximating the posterior with unimodal or multimodal distribution families.
2.1.1
Kronecker-Factoree Approximation Curvature (KFAC)
KFAC Martens & Grosse 2015
approximates the true posterior distribution by learning the following
unimodal multivariate normal distribution:
(MAr(0 = (0; MMPP, F-
where (H, 2) denotes the Gaussian distribution with mean μ and covariance Σ. The approximate posterior mean OMA the Marimum A Posteriori (MAP) estimate, which obtained by standard training algorithm such Stochastic Gradient Descent (SGD). The posterior covariance Đ-1 is the inverse of regularized approximation of the Fisher information matrix F. For computational efficiency, the inverse Fisher information matrix f is approximated by the sum of diagonal matrix and a low-rank matrix expressed as Kronecker factorization.
2.1.2 Monte-Carlo Dropout (MCDO)
Deep Neural Networks of arbitrary depth including non-lincarith that are trained with dropout can be used to perform approximate Bayesian inference (Gal & Ghahramani 2016). In MCDO, samples from the approximate posterior are drawn by
0: 07
.ddigg(aas diag([z
M m==1); 21m Bernoull((7),
3

---

Published in Transactions on Machine Learning Research (11/2023)
where of is the weight matrix of layer of dimensions Mi Mi-1 before Dropout is applied, diag(-) is function that maps the vector (=tmm)m=1 |M-1 to a diagonal matrix of dimension Mi-1 x Mi-1 with 21., being the diagonal element in row my and column my for my I.... Mr The elements 21,m are independent binary random variables that follow the Bernoulli distribution, e.g. 21, Bernoulli(w) and n the dropout rate. In practice, users choose fixed value of by that optimizes some metric, c.g. the test accuracy.
2.1.3 Stochastic Weight Averaging Gaussian (SWAG)
SWAG (Maddox et al. 2019) is an efficient method to a Gaussian distribution around a local solution. The algorithm consists of an initial training phase followed by model collection phase. The initial training is performed in a standard way, e-g., by SGD with a decaying learning rate. After the convergence, the collection phase starts, where the SGD update continues with either cyclical or high constant learning rate for collecting SGD iterates. The trajectory of those iterates is then used to estimate the mean and the covariance of the approximate Gaussian posterior distribution:
@SWAG (0 = (0;; ASAAG. ESWAG).
The covariance ÉSWAG is approximated by the sum of a diagonal and a low-rank matrix, similarly to KFAC, for stable estimation from a small number of SGD itcrates during the collection phase.
2.1.4 Deep Ensemble of SWAG (MultiSWAG)
MultiSWAG (Wilson & Izmailov 2020 combines the ideas of SWAG and deep ensemble (Lakshminarryanan ctt al. 2017). It simply performs the SWAG training multiple times from different weight initializations, and each SGD trajectory during the collection phase is used to compute the mean 0₂ and the covariance Σn of a Gaussian distribution. The estimated Gaussians are combined to express the approximate posterior as a (equally-weightee)) mixture of Gaussians (MoG):
K Σ
(MIt) = K
W(0;0e. En).
(5)
Since each SWAG trajectory is expected to converge to a different local solution or mode, MultiSWAG provides a mullimodal approximation of the true posterior. MultiSWAG was shown improve generalization performance and uncertainty estimation quality (Wilson Izmailov 2020).
2.2 Global XAI methods
XAI methods that are decoupled from the architectural choice a neural network and its training procedure are referred to as post-hoc XAI methods, which can be further categorized into local and global explanation methods. Global XAI methods aim to explain the general decision-makiin strategies learned by the representations of DNNs. They reveal the concepts to which a particular neuron responds the most (Olah et al. 2017 2018 Bau
et al. 2020 Nguyen et al.) 2016 by decomposing and quantifying the activations of certain neural network layers in terms of human- understandable concepts (Kim et al. 2018 Koh et al.) 2020). or by identifying and
understanding causal relationships that are encoded between neurons (Reimers et al. 2020
In this work, we focus on the Activation Maximization (AM) framework (Nguyen et al. 2016). The general
idea of AM is artificially generate an input that maximizes the activation of a particular neuron in a certain layer of neural network. The optimization problem can be formulated as follows:
i argmax a(v) R(v), VERV
(6)
where v is the input variable, a(-) is the activation of the neuron of interest, and R(-) a regularizer. This can be casily extended to maximize the activation of a certain channel or layer by maximizing norm of the channel's or layer's activation vectors. We refer to the resulting image, in Eq. as a Feature Visualization (FV) vector. To generate FV that not full of high-frequency noise, the case for
4

---

Published in Transactions on Machine Learning Research (11/2023)
Data set
image size
classes
Training size
Test size
Places365
256 x 256
365
~1.4mil
365k
CIFAR-100
32 x 32 x
100
50k
10k
STT.110
x x
10
5k
SVHN
x X 3
10
~73k
26k
Table 1: Data sets used in the experiments.
Model
# Trainable parameters
Test accuracy
Nes5o
25,557, ,522
55.57%
WRes10
36,546,980 580
82.75%
WRes2
1,481,252
77.45%
WResl
(WideResNet28)
376,356
72.57%
WRes0.7
88,448
62.54%
WRes0.2
5,008
27.96%
Table 2: Network architectures. WRes1 corresponds to the original WideResNet28,, and WRes3 for 3 * 1 is the network with the number of channels 3 times more than the original in cach convolutional layer. The test accuracy on Places365 for ResNet50, and on CIFAR-100 for WRes3, respectively.
adversarial examples (Szegedy et al. 2013 Goodfellow et al. 2014 Athalye et al. 2018). several regularization techniques have been proposed (Olah et al. 2017 Mordvintsev et al. 2018): Iransformation robusiness applies several stochastic image transformations, e.g., jittering, rotating, and scaling, before each optimization step; frequency penalization cither explicitly penalizes the variance between neighboring pixels or applics bilateral filters on top of the input. In order to even further reduce high-frequence patterns that correspond to noise, was proposed to perform the optimization in spatially decorrelated and whitened space, instead of the original image space. This space corresponds to the Fourier transformation of the image based on the spatially decorrelated colors. In this way, high- frequecyyy components are successfully reduced.
Feature Visualization methods have proven effective in explaining the concepts learned within Deep neural Networks (Goh et al.) 2021). These methods have also been employed for the purpose of explaining the circuits within DNNss, which represent computational subgraphs responsible for the transformation of various features (Cammarata et al.) 2020). Additionally, Activation Maximisation approaches have been used identify neurons responsible for spurious correlations (Bykov et al. 2023 and human-implanted backdoors [Cssper el al. 2023).
3
Experimental Setup
Here we describe our experimental setup including the methodologies for the quantitative analysis of feature visualizations.
3.1 Datasets
In experiments, use four data sets, Places365 Zhou et al. (2077), CIFAR-100 (Krizhevsky al. (2009)). STL-10 (Coates et al. (2011b)), and SVHN Yuval 2201 ), of which some statistics are listed in Table I When training a classifier, the Places365 images are clipped to 224 x 224 x 3, and the other data sets are clipped to 32 x 32* x Places365 dataset encompasses numerous classes with diverse uuman-understandable concepts, such as the farm class, which typically includes concepts like farm animals, barns, fields, and others. In computationally intensive experiments with MultiSWAG, we use CIFAR-100 and clipped images of STL-10 and SVHN.
5

---

Published in Transactions on Machine Learning Research (11/2023)
Inference mothod
Test accuracy
KFAC
54.78%
MultiSWAG
56.64%
MCDO-5%
55.68%
MCDO-10%
55.07%
MCDO-25%
51.69%
Table 3: Approximate Bayesian inference methods analyzed in the experiments. For each inference method, the test accuracy ResNet50 on the Places365 test set is evaluated on the predictive distribution that we approximate with T 50 MC samples.
3.2 Network Architectures
We train ResNet50 (He et al. 2016) on Places363, and WideResNet28 (Zagoruyko & Komodakis, 2016) on CIFAR-100, respectively. To study the network width dependence, we also use WideResNet25 with increased and decreased numbers of channels in cach layer. Therefore, we scale the number of channels of the original WideResNe(28 by scaling factor 33 which we refer to as WRes3. For example, WRes2 corresponds to a network that is twice as wide the original network. The number of parameters and the test accuracy are shown in Tabl where the test accuracies are obtained by evaluating the MAP estimates of the corresponding models.
3.3 Inference Methods
The approximate Bayesian inference methods that we analyzed are listed in Table 3 MCDO-y% is the MC dropout model with dropout rate 7%, applied to each layer the CNN encoders except for the last one (same dropout rate for all layers). For MultiSWAG, the approximate posterior is mixture 5 of K 10 Gaussians. In Table 3 the test accuracy of 5e on the Places365 test set with each inference method is evaluated according to Eq. with T 50 MC samples.
3.4 Feature Visualization
In all experiments, we use the AM framework (6) to obtain the FV vectors and analyze their behavior. The FV vector by default of the same size as the input image. However, since the images in CIFAR-100 are small and therefore not very informative in terms of the concepts that we can extract from them qualitatively, we expand the input 128 128 x 3 using bilincar interpolation PyTorch. We solve the AM optimization problem by 512 steps of gradient descent with the step size of 0.05. For regularization, we apply the transformation robustness with random rotation, random scaling, and random jittering. Moreover, the optimization is performed in the decorrelated and whitened space. All transformations correspond to the default setting in Olah et al. (2017). for which we used the PyTorch (Paszke et al. 2019) version of the published source code available at htttps://gite http://ttttuu...e. com/ httae//geeeeettpeeeecco /uuce/
3.5 Quantitative Distance Measure in FV space
One of the main contributions of this work is to analyze the diversity the "concepts", expressed in FV vectors, of BNN instances quantitatively. To this end, we need define a distance measure in the space of FVs. Apparently, the standard norm distances, e.g. L2-distance and cosine-similarity, directly applied to the FV vectors are not appropriate since "concepts" in FV should be invariant to translations and rotations. We, therefore, use non-linear function g: RVV + R that maps FV vectors into low-dimensional latent concept space such that FVs with similar concepts are mapped to close points. Afterward, the distance between two FVss is measured by the cosine-similarity in this space:
d(v,v') g(v) ((v')
(7)
lg(()\ll((>"11

---

Published in Transactions on Machine Learning Research (11/2023)
Compute the predictions of BNN instances for test point (*)) of class Farm.
Predicted class
bom
farm
vegetable_garden garden
Correctly classifying networks search for Red barn to predict the class Farm.
(a)
(b)
BNN
(c)
Wrongly classifying networks Nets wrong search for farm animals predict the class Farm, which they did not find in the image.
x (1)
40.0%
(d)
Compute the predictions of BNN instances æ! (l.ddited)
Manipulating x (1)
Predicted class
1. Find another test image (2) that contains the Farm animal concept and is predicted by all networks in Netswrong Farm.
n
(2)
0%
BNN
2. Extract Farm animal concept from x and insert it into (1 (1,edsted)
2e (Ledited))
(1)
Figure 1: Explaining decision-making strategies of 10 individual BNN instances using FVs. (a) A test sample (11)) of class Farm. (b) 10 BNN instances classify (1), (c) Classification results by 10 instances (only 40% classify (!)) correctly as Farm). FVs of the "correct" instances Noorrect are shown in the upper green box, while those of the "wrong" instances Nwrong are in the lower blue and red boxes. We observe that FVs of NetSwrong contain Farm animal-like concepts, which they did not find the input x(1), (d) Another test. sample (222) that contains a tiny sheep. (e) We cut out the tiny sheep (Form animal concept) patch from (22) and paste it into ())) yielding z (1.odited)
(f) All instances correctly classify z (edtted Farm, which implies that the FVs indeed encode human-understandable concepts that reflect the network's decision-making stratogy.
We learn the function via a contrastive learning scheme (Chen el al. 2020 Li et al.
2020
2021 and more
specifically, we use the SimCLR framework (Chen et al. 2020).
Assume that we are given a training { - of FVs.. We first conduct stochastic data augmentation, which applies M random image transformations to each FV, resulting in M different versions to be used augmented training set H L. 1. 1 Here ( 1M denotes the generated samples by stochastic augmentation applied to the n-th original sample, i.e., Un- Then, the non-linear map go(-) parameterized by & is trained by minimizing the contrastive loss:
Jm=
N
exp del dalun
log
(8)
En En'sen ok
EM
m 2m"mm"==
dola m M. exp(

---

Published in Transactions on Machine Learning Research (11/2023)
a.0.5.v
v.
a.O.S.v
a.0.5.vo
a: 1 1
v.v.
a:0.1.v
a:0.
1.0.1.v
a:0.05.
a:0.05.v
a:0.05.v6 A 0:0.05,v6
a:0.005.v vf a:0. 0.000.
a:0 a:0.005, a: 000..
(a) 128 epochs
(b) 512 epochs
Figure 2: Dependence of the FV the number of epochs, the learning rate a, and the parameter initialization
(og") (1) up a We observe that the number of epochs affects the color scheme slightly, that the learningn rate affects the crispness and contrast drastically, and that the parameter initialization affects the positions and shapes of objects slightly, while the high-level concepts stay the same for different initializations.
or
where de is the distance measure 7 which depends on through the mapping go(+), and is temperature hyperparameter. This contrastive loss aims to minimize the distances between the samples transformed from the same original test sample while maximizing the distances between those transformed from different original test samples. For the architecture of Jp. we use ResNet18 (He et al. 2016) base encoder, that is, all ResNet18 layers up to the average pooling layer, followed by a fully-connected layer, a ReLU non-linearity and another fully-connected layer. We train one contrastive learning model per Bayesian inference method. In particular, we generate 1000 FVs per BNN, except for MultiSWAG. In MultiSWAG, we generate 100 FVs per ensemble member and train our distance on the concatenation of their FVs, thus resulting 1000 total FVs. The number of augmented samples is M (per original sample and epoch), and the augmentation is randomly chosen from "Random Cropping", "Colorjitter", and "Horizontal Flipping". We use stochastic gradient descent (SGD), where the contrastive loss S with the batch size N and the temperature = 0.5 is minimized in each epoch. We run SGD for 150 epochs on the CIFAR-100, STI,110, and SVHN models, and for 250 epochs on the Placcs365 models. For the contrastive learning, we used the implementation of the following public repository available at http:////ttaa./oo///aa/tttttttee Clwsteerrreantnanneeeoe
4 Experimental Results
In this section, we visualize and analyze the diversity of the decision-making strategies of BNN instances in terms of high-level concepts by using feature visualizations. The first three experiments are dedicated to validating our approach. We first demonstrate that the extracted FVs properly reflect the characteristics of cach individual BNN model instance by relating FV the classification prediction, and obscrving the model's uncertainty behavior when manipulating test samples. Then, we analyze the dependence between FVs and the initial parameter setting of the AM algorithm. Finally, analyze the correlation between the diversity of FVs and the uncortainty estimates of BNN model instances. While the first two experiments validate our methodology qualitatively, the third experiment confirms quantitatively that FVs reasonably reflect the property of a Bayesian ensemble.
In the fourth and the fifth experiments, we use our distance measure to analyze the diversity of BNN instances from different posterior approximation methods, and for different scales of the model, which provide us with new insights into the latest. findings of deep learning theory (Roberts et al. 2022 To the best. of
8

---

Published in Transactions on Machine Learning Research (11/2023)
our knowledge, no previous work has used global XAI methods to explain the behavior of Bayesian Neural Network instances.
4.1 Visualizing characteristics of individual BNN instances
The prediction by a BNN is made from an ensemble of model instances drawn from the (approximate) posterior distribution, and such model instances can acquire different decision-making strategies. In this first experiment, we qualitatively show that by using FVs we are able to explain the diverse characteristies of each individual model instance in the Bayesian posterior ensemble. We draw model instances from the MCDO-5% posterior with the ResNet50 architecture trained on Places365. To receive a global explanation for each of the 10 BNN instances for the class Farm, we apply the AM algorithm to the network output, maximizing the logit for the label Farm. From the 10 different FV images, shown in the green, blue and red boxes in Figure 1 we can observe that all FV images contain reasonable high-level concepts, e.g., Red born, Farm animal, Tractor, Crop field, and Pasture fence, implying that all BNN instances learned reasonable decision- making strategies for the class Farm. However, we identify some test samples which are misclassified by some of the BNN instances.
One of the wrongly classified test samples, x²1,, is shown in Figure I Indeed, 4 of the instances classify (x(1) correctly as Farm, while the other 6 instances classify it wrongly either Barn Vegelable garden as shown in the pie chart in Figure [[c). The corresponding FVs of the BNN instance are plotted next to the pic chart, arranged based on their prediction: the 4 FVs in the upper green box correspond to the instances classifying the input correctly (which we refer to as NetScorreet), ), while the 6 FVs in the lower blue and red boxes correspond to those classifying the input wrongly (which we refer to as Netswrong). The networks that correspond to the FVs in the blue box wrongly predict the input as Vegetable garden, while the ones that correspond to the FVs in the red box wrongly predict the input as Barn. Comparing the FVs of NetScorrect and NetSwrong; we notice that most of the FVs of NetSwrang contain fur or farm animal-like objects such as sheep, horses and cows, while F'Vs of NetScorrect do not. This implies that the 6 instances in Netswrong use animals within their decision strategy to classify an image as Furm, and since x()) does not contain any Farm animal concept, they can not classify it correctly. To further demonstrate that FVs can reveal the characteristics of each BNN instance, we manipulate the test sample x(1) by using another test sample a22)) which contains a very small sheep (Farm animal concept) and which is classified correctly by all BNN instances in Netswrong. From 22(2, we cut out the small sheep and paste it into 2(1) manually (see x ((l.edited) in the figure). Now, all instances classify ([1,ddited) correctly as Farm as shown by the green pic chart. This implies that the high-level animal concepts inherent in the FVs of Netswrong are indeed the concepts that the networks are searching for in the input image in order to classily an input Farm.
Two other similar examples are given in Appendix A With this experiment we demonstrated, that the diverse decision- making strategies of different BNN model instances can indeed be explained by global XAT methods, providing evidence for our question 1) from the introduction.
s(dlv =3-
0.15
sld(v(? pr
pldtvp? =)))
0.10
p(ctv(" =e))
0.05
4.2
Dependence on hyperparameters and initialization
0.00
50
100
150
200
250
The AM optimization (6) is highly non-convex, and therefore the obtained FVs can depend on the hyperparameter setting and parameter initialization. Hence, we would need to appropri- ately choose the setting such that the FV's properly extract the concepts that the networks are using for the decision- deciison-making process. Accordingly, we investigate the dependence of FVs on the hyperparameters, e.g., the number of epochs for SGD and the step size a, and the initialization we start from two random initial points, vi and v to solve the optimization problem O From the results, shown in Figure 2 we can ob-
d(v!?, []10-2
Figure 3: Distribution of distances between FVs and their corresponding instance cen- ters (blue) and distribution of the distances between different instance centers (orange). The distances within the instances are much smaller than between different instances, and both distributions are clearly separated. The impact of parameter initialization for the AM optimization is ignorable.
serve that the AM optimization for a 0.005 does not converge even after 512 epochs, and a 0.5 results in over-cont rasted
9

---

Published in Transactions on Machine Learning Research (11/2023)
FVs. We furthermore observe that different initializations can change the location and shape of the concept objects slightly, while the content of the represented concepts is unchanged for different settings and ini- tializations. This result implies that our approach can be used for analyzing the decision-making process without carefully tuning the parameters. Note that the location shift of concept objects by initializations might strongly affect the quantitative analysis of the diversity of BNN instances we would adopt a naive distance measure, e-g.; the L,2 or cosine distance the pixel space. We will make sure in the next experiment that this is not the case when we use the distance measure introduced in Section 3.5
4.3 Quantitative diversity of FVs
Here, we validate our quantitative distance measure in the FV space by showing that suffices two requirements: 1) the diversity caused by parameter initializations for the AM optimization ignorable compared to the diversity of BNN instances, and 2) the measured diversity is highly correlated to the uncertainty of the prediction. In order to evaluate 1), first generate FVs for each of 100 BNN instances and compute the average latent concept vector for each of the instances. We will refer to these 100 mean vectors as instance centers. In Figure 3 we plot the histogram of the Euclidean distance from each FV to its corresponding instance center (blue), as well as to another instance center (orange). We can observe a significant separation between the two histograms, implying that the FV diversity caused by initialization is indeed ignorable.
Next, we investigate the correlation between the diversity of FVs and the predictive entropy,
H(x)) = - Ply" cla" D D) Ply = c,a,
which is measure for uncertainty. We prepare 100 sets, each comprised of 100 model instances with different
FVVar (see Appendix B for how to generate those sets), and plot. the FV variance
FVVar: = I (oo(vr) -
(9)
in the horizontal axis and the empirical mean predictive entropy over the whole data set D
Ep(a) [H)) = Σ ((("))
in the vertical axis in Figure 4 Here, T' is the number of BNN instances in a set, {wr)= are the FV's of them,
and the predictions (for computing the entropy) are made by those T instances. We can observe a high correlation (0.83) between the FVs diversity and the uncertainty estimates, showing that our distance measure in the FV space properly reflects the dis- tance between BNN instances. Overall, we can con- clude that our distance measure satisfies the two requirements above, and we now apply our tools for analyzing BNN instances.
2.9
Linear Regression,
Mean predicti citvve entropy
2.8
Pearson correlation: 0.83
2.7
2.6
2.5
2.4 :4:
2.3
4.4 Comparing representations of different BNN inference methods
2.2
0.0
0.4
0.8
FVVar
Here, we will visually and quantitatively compare the learned representations of models that were trained using different Bayesian inference methods. We train ResNet50 on Places365 with the Bayesian approxi- mation methods listed in Table 3 First, find the
Figure 4: Linear regression and Pearson correlation between feature visualization diversity (FVVar) and mean predictive entropy. High correlation between FVVar and the mean predictive entropy is observed.
10

---

Published in Transactions on Machine Learning Research (11/2023)
KFAC
MultiSWAG
Cluster
Cluster 2
Cluster
(o44 sompprment
MCDO-59
MCDO-2598
Clusser
Cluster
componentt
ISNE comporeett
Figure 5: FVs and cluster structure of the learned concepts for class logit Farm. Four black boxes represent KFAC, MultiSWAG, MCDO-5%, and MCDO-25% respectively. The FVs are clustered using KMeans (where the number of clusters is manually chosen) and plotted in swo-dimensional tSNE space in different colors. We choose certain clusters and plot 3 example FVs of cach cluster. The FVs in a colored rectangle are from the cluster depicted as an ellipse in the same color in the (SNE plot. For each Bayesian inference method, we mostly found the following concepts in the displayed clusters. KFAC cluster 1: Tractor, Crop field, and little details of animals, e-g. Eyes. KFAC cluster 2: Red barn, Crop field, Pasture fence, and little details of animals, c.g. Eyes. MultiSWAG Cluster 1: Tractor, Crop field. MultiSWAG cluster 2: Farm animal, Crop field, and Pasture fence. MultSWAG cluster 3: Red barn and Crop field. MCDO-5% cluster 1: Red barn, Crop field. MCDO-5% cluster 2: Tractor and Crop field. MCDO-5% cluster 3: Farm animal, Pasture fence. MCDO-25% cluster 1: Red barn, Crop field, Pasture fence. MCDO-25% cluster 2:
Farm animal, Pasture fence, and Crop field. MCDO-25 cluster 3 mostly contains a mixture all found Farm concepts in specific grayscale.
cluster structure of BNN instances of cach inference method and compare their Iearned concepts. To this end, we compute the FV for the class Farm of each of the 100 BNN instances, individually. Figure shows 1NNE plots of the BNN instances in the FV space with typical FV images from each cluster for KFAC,
11

---

Published in Transactions on Machine Learning Research (11/2023)
MultiSWAG, MCDO-5%, and MCDO-22 25%. Note that tSNE is applied in the latent concept space, i.e., to the vectors (ge(v))), and therefore reflect our quantitative distance measure. Clustering is performed by applying KMeans (Lloyd 1982) (again to the latent concept vectors {ob(v))}, and the instances that belong to different clusters are depicted in different colors. We can observe that the concepts of the class Farm, which are visually perceptible, generally consist of Red barn, Tractor, Farm animal, Crop field, and Pasture fence. However, different inference methods yield different cluster structures: KFAC yields 2 clusters and MultiSWAG yields 10 clusters indicated by the different colors in Figure 5 respectively. Furthermore, we can observe that MCDO yields 5 ~ 7 clusters, while the distribution looks highly dependent on the dropout rate. KFAC yields the least diverse FVs, in terms of visual comparison, with 2 clusters, and all instances tend to include almost all of the Farm concepts. Nevertheless, we observe a difference between the 2 equally sized clusters which relates to the Traclor concept being more present in cluster and the Red born concept. being more present in cluster 2.
The MultiSWAG instances also include different Form concepts in each of the instances. However, visualizing their clusters individually, we can observe that some clusters include certain concepts more frequently than others. In particular, cluster 1 includes the Tructor and Crop field, cluster 2 the Farm animal and Pasture fence, and cluster 3 the Red barn, Crop field and Pasture fence concepts more frequently than the other clusters. The other 7 clusters contain all Farm concepts and are similar in terms of the content of concepts (see Appendix C For MultiSWAG, the clusters found by KMeans match the MoG structure, i.e., most of the BNN instances generated from the same posterior Gaussian component are clustered together. This connects to the fact that each SWAG ensemble member converges to a different local minimum (Wilson & Izmailov 20200, or posterior mode. MCDO-y shows the most. diverse FVs. We observe in Figure that the BNN instances of cach cluster seem to specialize with respect to certain Farm concepts and can be thus scparated very well by these concepts. For MCDO-5%, cluster 1 primarily includes the Red barn and Crop field, cluster 2 the Tractor and Crop field, and cluster 3 the Farm animal and Pasture fence concepts. Naturally, the diversity of MCDO-y instances increases with increasing Dropout rate, and thus MCDO-25% results in an increased number of clusters. Also, we observe that the quality of FVss decreases with increasing Dropout rate and that more diverse color schemes appear, e.g. two yellow scales, two gray scales, and one blue scale for MCDO-25%. In Appendix [ we additionally show the results for 1CD0--1% and include examples of the remaining clusters of the MultiSWAG model instances. The right plot in Figure G shows the quantitative diversity of FVs, iee, FVVar defined in Eq. 9 which behaves consistently with our qualitative observalions KFAC yields the lowest variance, followed by MultiSWAG, and the MCDO 5%, 10%, 25% yields the largest diversity in this order. Hence, we can answer question 2) from the introduction "Does the choice of the Bayesian inference method offect the diversity of their feature visualization ?" with yes.
The right subplot in Figure G compares the FV diversity and the mean predictive entropy, which exhibits a clear correlation. We observe, for the first time, that the FV diversity correlates with the predicted uncertainty estimates. This answers question 3) from the introduction " "Can the uncertainty estimates provided by BNN be explained by the diversity of their feature visualizations?".
4.5 Visualizing the multimodal structure of the posterior distribution of BNNs.
Here we explain the multimodal structure of the BNN posterior distribution. Specifically, we use BNN instances drawn from the MoG posterior (5) obtained by MultiSWAG, and qualitatively (visually) and quantitatively analyze their behaviors. Furthermore, we investigate the dependence of the multimodality on the network width in terms of humanly understandable concepts using FVs. To this end, we train a Wede and its modified versions listed in Table 2 where WRes3 refers to a WideResNet22 network with the width scaled by 3. The models are trained on CIFAR-100 by MultiSWAG with mixture of K 10 Gaussians posteriors. After training, we draw 100 BNN instances from each Gaussian posterior, resulting in 1000 BNN instances in total, and compute their individual FVs for the class Castle. Figure 7 shows distributions of the 1000 BNN instances in the FV space, where the rows correspond to the networks with different widths, i..,, WResO.2, WResl, and WRes10, respectively (Results with other network widths are shown in Appendix D). For each row, a tSNE plot of FVs is shown in the bottom, and FVs of three BNNs from three hand-picked modes are visualized in the top. Note that the color in the tSNE plot indicates the Gaussian component of MultiSWAG (KMeans is not applied here). From the tNN plots, we observe that.
12

---

Published in Transactions on Machine Learning Research (11/2023)
1.0
0.9
2,4
Mean predictive entropy
2.4
KFAC
MultiSWAG
0.8
MCDO p=5%
2.2
FWVrr
0.7
MCDO
0.6
2.0
p=10%
MCDO
0.5
1.8
p=25%
0.4
1.6
0.3
0.2
1.4
KFAC MCDO
MCDO 10%
MCDO p=25%
0.74
0.76
0.78
0.84
0.86
5=5%
FWVar
Figure 6: Reprosentation diversity of Bayesian inference methods. The left subplot displays the FVVar of BNN instances of each inference method. The diversity of KFAC, MuluSSAA, MCDO 5%, 10,, and 25% is increasing from left to right. The right subplot displays the correlation between FVVar and the mean predictive entropy. They highly correlate with each other, as expected.
the network width strongly affects the multimodal structure: For small network width (WRes0.2). instances from different modes are separable, while, for wide network (WRes10), the instances are overlapped.
For quantifying this observation, compute the inter-mode variance
InterModeVar: K Ek Pk
1 K
(10)
and the intra-mode variance
IntraModoVar t ΣB 100
Ek - mxll
(11)
and plot them in the left panel of Figure Here, 2 is the latent concept vector of the FV of the t-th instance from the k-th mode (k-th Gaussian component), and μu is the latent concept vector of the FV of the k-th Gaussian center. We observe that, with growing network width, the inter-mode variance decreases, while the intra-mode variance increases. The former (decreasing variance) aligns with the implications of recent theory on Neural Tangent Kernels (NTK) (Jacot et al., 2018 Arora et al. 2019). while the latter (increasing intra-mode variance with growing B) has not been explained by theory, to the best of our knowledge. In order to show that the observed tendency is not because of the limited expressivity of the RcsNet18 that we used define our dist ance measure, nor the dataset-specifie properties, we conducted the same experiments with ResNet34 and ResNet50 as the backbone networks and on other datasets including STL-10 (Coates al., 2011ab and SVHN (Netzer et al.) (2011), and observed similar tendency in Appendix E We also confirm in Appendix F that whether starting from pre-trained models or training from scratch does not significantly affect the properties of posterior distribution.
In the following, we will answer question 4) from the introduction regarding the impact of the network width on the diversity of FV of samples from a multimodal posterior distribution. To this end, we first qualitatively investigate the FVs given in Figure 7 We can observe that too narrow network (WRes0.2) gives notably low-quality FVVs, which implies that the network does have sufficient capacity learn good feature representations. The large inter-mode variance reflects the fact that each mode learns different color schemes and different patterns, while the small intra-mode variance results in identical concepts within each mode. For larger network widths, we observe a clear difference in their FVs. While the modes of WResl still learn different color schemes, and at the same time also learn high-level Castle concepts, e.g. castle lowers, the modes WRes10 learn very similar features, and it is harder to distinguish between the FVs of different modes. The difference between WResl and WRes10 implics that, for a moderate network width, each mode plays different roles while for a large network width, modes get mixed and each mode abstracts the concepts well, such that single model alone perform classification well.
13

---

Published in Transactions on Machine Learning Research (11/2023)
WRes0.2 .2
WRes1
WRes10
Cluster
Cluster
Cluster
Cluster
Cluster
Cluster
Cluster
Cluster
Cluster
ISNE component 2
ISNE component 2
tSNE component 2
20
-__00
20
20
40
50
50
550
-25
25
ISNE component 1
tSNE component 1
tSNE component
Figure 7: Multimodal structure of BNNs trained by MultiSWAG in the FV space. The rows correspond to the networks with different widths, i.e., WRes0.2, WResl, and WRes10, respectively. For each network, tSNE plot of 1000 BNN instances shown in the bottom, and FV' of three BNNs from three hand-picked SWAG modes are visualized in the top. We observe that the modes tend to overlap as the network width increases. Moreover, the quality of FVVs drastically improves from WRes0.2 to WResl, and WRes10 successfully abstracts the castle class by replacing discrete shape information (small discrete shapes, c.g., in the bottom of WResl Cluster 1) with smoother ones in each mode (compare the bottom of WRes10 Cluster 2).
We confirm this implication by quantitatively evaluate how strongly ensembling, e.g., with deep ensemble and MultiSWAG, improves the performance. In the right panel of Figure [ we plot the performance gain by MultiSWAG compared to its single-mode counterpart, i.., SWAG, as function of the network width. More specifically, the red curve shows the test accuracy of MultiSWAG (i.e., the test accuracy of the averaged model) subtracted by the average accuracy over the separate SWAG models (i.e., the average of the test accuracies of the models). We observe that the performance gain by ensembling is largest for WResl, and decreases when the network width further increases.
The cyan curve similarly shows the performance gain terms of the test expected calibration error (ECE) (uuo et al.) 2017). Namely, the curve shows ECE of MuluSWAG subtracted by the average of the ECEs over the separate SWAG models. Noting that the lower ECE is the better, uncertainty estimation performance when the network width is small, e.g., WRes0.2 and WRes0.7, while it significantly improves the test ECE when the network width is large, e.g., WRes2 and WRes10. We will investigate this phenomenon further in our future work.
that ensembling degrades the
Our extensive experiments revealed how the width of the undorlying network architecture affects the FVs, answering question 4) in the introduction " HHw does the network width affect the diversity of explanations of samples from mullimodal posterior distribution
14

---

Published in Transactions on Machine Learning Research (11/2023)
0.08
0.8
0.06
0.6
0.04
0.02
InterModeVar
0.4
IntraModeVar
0.00
-0.02
0.2
-0.04
Accuracy difference
ECE difference
-0.06
0.0
WRes0.2 WRes0.7 WRes1
WRes2
WRes10
WRes0.2
WRes0.7
WRes1
WRes2
WRes10
Figure 8: The left subplot shows how the inter- and intra-mode variances change with increasing network width. As the network width increases (from left to right), the inter-mode variance decreases, while the indra-mode variance increases. The right subplot shows the performance gain by ensembling in terms of the test accuracy, well as the test expected calibration crror (ECE), i.c., the test accuracy (ECE) of MultiSWAG subtracted by the average accuracies (ECEs) over the separale SWAG models. Noting that the higher accuracy (lower ECE) is the better, we observe that for larger network widths (WRes2 and WRes10), the performance gain decreases for the test accuracy, while it increases for the uncertainty estimation.
5 Conclusion
Since BNNs provide additional information about the uncertainty of prediction, they are of enormous value, especially in safety-critical applications. Their ability to estimate uncertainties of a prediction is inherent in the learned multimodal posterior distribution. Sampling from this posterior distribution results in BNN instances, exhibiting diverse representations, which in turn lead to different prediction strategies. It has been shown in large number of works that the diversity of these strategies depends on various factors, such the choice of the Bayesian approximation method, the parameter initialization, the model sizc. However, far, this diversity has been analyzed cither in the output, parameter space of the BNN instances, which unfortunately still lacks human understandable intuition. With this work, now deliver this missing but important building block to support human understanding by making the learned strategics visually accessible. To this end, we use feature visualizations as a global explanation method to explain a luman--understnnddble way the different representations and prediction strategies learned by BNN instances. Furthermore, this enables us to examine the diversity of the BNN instances on the feature visualizations both qualitatively and quantitatively, thus adding the visual component the previous analysis.
In order to quantitatively analyze the FVs with their pronounced heterogeneity, we first learn a suitable representation of the FV with the help of contrastive learning, which we can then use to measure the distance. The ability to measure the distances between FVs allows us to investigate and at the same time to visually understand how the use of different. Bayesian inference methods affects the diversity of BNN instances. Indeed, we could demonstrate, that the learned representations vary stronger for multimodal Bayesian inference methods, such MultiSWAG than for unimodal ones, such as KFAC. The greatest variety of learned representations is achieved by Dropout-based models. Here the dropout rate correlated positively with the variety of representations, i.c. the higher the dropout rate is, the more different the representations visible through their FVss are. Furthermore, we showed that the diversity of FVs of BNN samples is positively correlated with the uncertainty estimates that we obtain from this BNN.
We were also able measure- and visually demonstrate the dependence of the multimodal structure of the posterior distribution on the width of the underlying network. Specifically, we have shown that the modes in multimodal posterior distribution of MultiSWAG become more similar with increasing width
15

---

Published in Transactions on Machine Learning Research (11/2023)
of the underlying network. This result is consistent with recent theoretical insights into Neural Tangent Kernels, where it was shown that the local solutions of infinitely wide networks behave similarly. By adding the additional visual component through the lens of global explanations we can easily understand the similar behavior of the modes given their similar FVs. In future work, we will investigate how the observed behavior of posterior modes can help to improve model performance in detecting OOD samples.
Acknowledgements
This work was partly funded by the German Ministry for Education and Research (BMBF)) through the project Explaining 4.0 (ref. 011S200551), the German Research Foundation (ref. DFG KI-FOR 5363), the Investitionsbank Berlin through BerDiBa (grant no. 10174498), and the European Union's Horizon 2020 programme through iToBoS (grant no. 965221), and BIFOLD Berlin Institute for the Foundations of Learning and Data (ref. BIFOLD23BB.
References
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06;/0 70, 2020.
Anish Athalye, Logan Engstrom, Andrew Iyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International conference on machine learning, pp.
PMLR, 2018.
David Bau, Jun-Yan Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Under- standing the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 117:201907375, 09 2020. doi: 10.1073/ 2..107/pnaa...e 1907375177
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30, 2017.
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622 PMLR, 2015.
Kirill Bykov, Marina M-C Höhne, Adelaida Creosteanu, Klaus-Robes Müller, Frederick Klauschen, Shinichi Nakajima, and Marius Kloft. Explaining bayesian neural networks. arXiv preprint arXiv:2108.108: 10346, 2021.
Kirill Bykov, Anna Hedström, Shinichi Nakajima, and Marina M-C Höhne. Noisegrad enhancing explanations by introducing stochasticity to model weights. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6132-6140, 2022.
Kirill Bykov, Mayukh Deb, Dennis Grinwald, Klaus Robert Muller, and Marina MC Höhne. DORA: Exploring outlier representations in deep neural networks. Transoctions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:://ppnrevviii net//orrmmiddnffVeiee
Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 2020. doi: 00.23915/disthi dttps://distill.pub//// /2000/circuite
Stephen Casper, Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield-Menell, Red teaming deep neural networks with feature synthesis tools. arXiv preprint arXiv:2302.1089, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
16

---

Published in Transactions on Machine Learning Research (11/2023)
Adam Coates, Andrew Ng, and Honglak Lec. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and slatistics, pp. 215 223. JMLR Workshop and Conference Proceedings, 2011a. la.
Adam Coates, Andrew Ng, and Honglak Lec. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 213- JMLR Workshop and Conference Proceedings, 2011b.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in doep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Diställ, 6(3):300, 2021.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and harnessing adversarial examples. Xiv preprint 1rXio:1412.633 2014.
Alex Graves. Practical variational inforence for noural networks. Advances in neural information processing systems, 24, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. 06 2017.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In
Proceedings of the IEEE conference compuler vision and pallern recognition, pp. 770 778, 2016.
Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah. Understanding rl vision. Distill, 2020. doi: 10.23915/distill000000 https://distill.ppb///ue http:////iisiiippn////mmm/maaaar unddesstanninng-l-viiiio.
Yu Huang and Yue Chen. Autonomous driving with deep learning: A survey of statc-of-art technologies. arxiv preprint xrXiv:2006.0609,, 2020.
Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35:38516 1885516-3852,, 2022.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Been Kim, Martin Wattenberg, Justin Gilmer, Carric Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 8668-26778 PMLR, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottlenock models. In International conference on machine learning, pp. 5338-5348. PMLR, 2020.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty cstimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Sebastian Lapuschkin, Alexander Binder, Grégoire Montavon, Frederick Klauschen, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10:e0130140, 07 2015. doi: (00.871/journal.pon0000000 00100140.
Müller,
Scbastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Robert Müllrr. Unmasking clever hans predictors and assessing what machines really learn. Nature Communications, 10, 03 2019. doi: 10. 00.1035/41147--019-6565
17

---

Published in Transactions on Machine Learning Research (11/2023)
An Le, Philipp Kratzer, Simon Hagenmayer, Mare Toussaint, and Jim Mainprice. Hierarchical human-motion prediction and logio-geometric programming for minimal interference human-robot tasks. pp. 7-14, 08 2021. doi:
Yann Lecun, Leon Bottou, Y. Bengio, and Patrick Haffner. Gradient-based learning applied document. recognition. Proceedings of the IEEE, 86:2278 2324, 12 1998. doi: 0..1109/5.72677
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint
2020.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 8547-8555, 2021.
S. Lloyd. Least square quantization in pem. IEEE Transactions on Information Theory TIT, 28, 01 1982.
David Mackay, John Bridle, Peter Cheeseman, Sidney Fels, Steve Gull, Andreas Herz, John Hopfield, Doug Kerns, Allen Knutsen, David Kocrner, Mike Lewicki, Tom Loredo, Steve Luttrell, Ken Rosc, Sibusiso Sibisi, John Skilling, Haim Sompolinsky, and Nick Weir. Bayesian methods for adaptive models. 05 1999.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in newral information processing systems, 32, 2019.
James Martens and Roger Grosse. Optimizing neural networks with fronecker-factored approximate curvature. In International conference on machine learning, pp. 1408-2417, PMLR, 2015.
Larry Medsker and Lakhmi C Jain. Recurvent neural networks: design and applications. CRC press, 1999.
Alexander Mordvintsev, Nicola Pezzolti, Ludwig Schubert, and Chris Olah. Differentiable image parameteri- zations. Distill, 2018. doi: 1.229155/distill00011
2ttp:llllpp///222////efeeeeeeeee parameterizations.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised fcature learning. In Advances in Newral Information Processing Systems Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http:://ffll stanfffdd.sttfoodd stanford hww.hhumuessss.....m .pdf
edh/houusenumberrs///ppp10111111 hagpapaeaerrrrpppp00ooooooooaaae
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems, 29, 2016.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.23915/ distill.000....00000
Chris Olah, Arvind Satyanarayan, lan Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexan- der Mordvintsov. The building blocks of interpretability. Distill, 2018. doi: 10.23915 hii/1//disill000100 bbildddgggbbloks..
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-peeformance deep learning library. Advances in neural information processing systems, 32, 2019.
E. Pearson, Maurice Kendall, and Alan Stuart. The advanced theory of statistics, volume i, distribution theory. Biometrika, 46, 03 1959. doi: 10.2307 /2335511.
Christian Reimers, Jakob Runge, and Joachim Denzler. Determining the Relevance of Features for Deep Neural Networks, pp. 330 346. 11 2020. ISBN 078-3-030-58573-0. doi:
20.
18

---

Published in Transactions on Machine Learning Research (11/2023)
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should i trust you?' explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135 1144, 2016.
Daniel A Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory. Cambridge University Press Cambridge, MA, USA, 2022.
Ramprasaath Rs, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basod localization. pp. 618-626, 10 2017. doi: 10.1 20.10 .0017.74.
Thomas Schnake, Oliver Eberle, Jonas Lederer, Shinichi Nakajima, Kristof Schütt, Klaus-Robert Mueller, and Gregoire Montavon. Higher-order explanations of graph neural networks via relevant walks. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP:1-1, 09 2021. doi: N..109//TPAMI.20 3115452.
K Simonyan, A Vedaldi, and A Zisserman. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR). ICLR, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg, Smoothgrad: removing noise by adding noise. arXiv preprint or Xrxiv 221.7006.03333 03825, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pp. 3319 3328. PMLR, 2017.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. 12 2013.
Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. A Deeper Look at Dataset Bias, pp- 3t50//9978-3-1--58888
37-55. 09 2017. ISBN 100-3-819-58844-- doi:
Nicol Turner Lee. Detecting racial bias in algorithms and machine learning. Journal of Information, Communication and Ethics in Sociely, 16, 09 2018. doi:
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4705 2020.
Netzer Yuval. Reading digits in natural images with unsupervised feature learning. In Proceedings of the Advances in Neural Information Processing Systems Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXix:1605.0110 2016.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In International Conference Learning Representations, 2019.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Poltern Analysis and Machine Intelligence, 2017.
19

---

Published in Transactions on Machine Learning Research (11/2023)
A Decision-making strategy manipulation
we demonstrate that we can visualize and extract high-level information about the docision-making strategics of BNN instances from FV and, for the first time, visualize their differences. Here we show lew other examples in Figure
In the introductory experiment in Section 4.1
Test magee
PV at correct precacting networrs Image 1
Teal image
FV of correct predictngg netwerks
image
Teat image? ?
Fy wrong predicting networss on image 1
image?
wrong predicting networks image
Edited
Edited
All natuorks predict adited image correctly
notworks image
comrecty
Figure 9: Two additional image manipulation examples. Left: In this example we can see that the selected images that predicted "Test image I' correctly include the high-level concepts Red barn and Crop fields, which are also present in the image. On the other hand, the chosen networks that predicted the image incorrectly, include the high-level concept Farm animal in them. After manipulating Test image 1 by pasting horse (Farm animal) from "Test image 2" of the test set, all networks do predict the "Edited image" correctly. Right: In this example we can see that the selected images that predieted "Test image 1' correctly include the high-level concept Crop field, which also present in the image. On the other hand, the chosen networks that predicted the image incorrectly, include the high-level concept Red born them. After manipulating Test image 1 by pasting red barn from "Test image 2" of the test set, all networks do predict the 'Edited image" correctly.
B Constructing FV diversity sets
Here, we explain how we formed the sets of BNN instances used in Section 4.3. where the correlation between the FV diversity, FVVar, and the mean predictive entropy is evaluated. We first prepared a pool O = (0,]e = I,... T } of BNN instances, by drawing samples from the posterior distribution. From θ, we generated 100 different sets {S I...., 1000, each of which consists of 100 BNN instances. Each set S; collects samples from θ in the following way: after randomly choosing the first instance 0 θ, we iteratively add the nearest neighbor (in the FV metric space) of the last added instance 0--1 for == 1,..,, 100. Note that, every time we add an instance to set, the corresponding instance removed from the pool, i.e., O O².- Although we did not control the diversity of each set, the resulting sets had different diversity shown in Figure
C Clustering representations of different BNN inference methods
For the experiments in Section 4.4 we cluster the FVs by applying KMeans. We choose the number of clusters by qualitatively analyzing the goodness of clusters, that is, whether the points cluster well in the tSNE plots and whether human-understanddall concepts, c.g. Farm animal, are clustered togother. We show the tSNE plots for the MCDO-10% model in Figure 10 the FVs of MCDO-10% in Figure and the FVs of the MuluSWAG clusters in Figure 12
20

---

Published in Transactions on Machine Learning Research (11/2023)
MCDO-10% clusters
8
NE component 2
4
2
Cluster:
Cluster:
Cluster:
Cluster: 4
Cluster: 5 10
-8
-6
-4 -2 tSNE component 1
2
4
Figure 10: MCDO-10% clusters. The instances cluster into 5 clusters, of which two, Clusters 1 and 4, are well separated, while the other three, Clusters 2, 3, and 5, are connected.
Cluster:
Cluster: 2,
Cluster:
Cluster: 4,
Cluster: 5,
#Elements: 21
#Elements: 22
#Elements: 27
#Elements: 17
#Elements: 13
Figure 11: Five clusters formed by MCDO-10%. Clusters 1-3 are connected, and similar in terms of their color scheme. However, Cluster contains the Farm animal, Cluster the Tractor, and Cluster the Red barn concepts more frequently than the others. The other two clusters, Clusters and 5, contain a mix of all Farm concepts, however in different color scheme.
21

---

Published in Transactions on Machine Learning Research (11/2023)
Guaden
Cuudee
Qludee
Cluder
Cueer
Cuder
Cinten Beneeng
Chesten
Custen
Quster
Figure 12: MultiSWAG clusters. The MultiSWAG clusters mostly cluster with regard the underlying SWAG ensemble members. We can see, that most clusters contain varicty of Farm concepts in them.
D Multimodal structure of the posterior distribution of BNNs WRes0.7 and WRes2.
In Figure 13 we show the tSNE plots and respective FVs of some example clusters of the WRes0.7 and WRes2 models. As can be seen, the FVs are naturally clustered together and well separated for the narrower WRes0.77 model, and overlap more for the WRes2 model.
E
Additional experiment on the width dependence
Here we conducted the same exporiment as in Section 4.5 with different backbone networks for defining the distance measure the FV space, and on different datasets. The left panel of Figure [4]shows the dependence of the inter- (solid lines) and intra-mode (dashed lines) variances on the network width on CIFAR100, STL-10, and SVHN, when ResNet18, ResNet34 and ResNet50 are used as the backbone network. For the CIFAR-100 data set, we base SVHN data set on the class label 8. However, as the consistency across data sets for our results shows, any class label could potentially be used for this analysis. We can observe that, with all three backbone networks and on all three datasets, the inter-mode variance decreases with growing width (x-axis), while the intra-mode variance increases. These results are also reflected in the Feature Visualizations of these different-wii wiith models as shown Figure 15 where we show the FVs for the Bind class of three different modes trained on the STL-10 dataset, and in Figure 16 where we show the FVvs for the 8 class of three different modes trained the SVHN dataset.
FV analysis the class label Castle, for STL-10 the class label Bird, and for the
F Comparing pre-trained models to models that were trained from scratch.
In this section, compare the FVs of models trained in two different scenarios. In the pre-trained scenario, pre-trained models are finetuned on the target dataset, while in the from-scratch scenario, training is performed from scratch. In the pre-trained scenario, collected five ResNet50 models from Ashukha el. al. (2020), which were pre-trained on ImageNet from different initializations, and finetuned them on CIFAR10 using the KFAC method. In the from-seratch scenario, we train five ResNet50 models from scratch on CIFARIO using KFAC. As seen in Figure 17 FVs of class Truck for both scenarios look qualitatively similar. The left subplot of Figure I8 shows the posterior mode structure as 2D projection onto the first two t-SNE components of the latent concept vectors obtained from each mode. Again, the cluster structure for both the pre-trained and from-scratch scenario looks similar. Moreover, the right subplot of Figure I8 shows FVVar of single mode, as well as all modes, for both training scenarios, which quantilatively confirms that the FV distribution is not significantly affected by the training scenario.
22

---

Published in Transactions on Machine Learning Research (11/2023)
WRes0.7
WRes2
Cluster
Cluster
Cluster
Cluster
Cluster
Cluster
40
ISNE component 2
ISNE component 2
20
20
-20
-0
-40
40
-50
50
-50
ISNE component
tSNE component 1
Figure 13: Multimodal structure of WReNet28 with different network widths in the FV space. The black bounding box corresponds to WRes0.7 and WRes2. The colored dashed bounding boxes mark FVss of 3 BNN instances from 3 modes. As the network width increases, the modes overlap more.
23

---

Published in Transactions on Machine Learning Research (11/2023)
Intermooee [+] [/inttrrmm Intre-mode (-) veriance
CIFARI00
inter mode (+) intro mode (-) variance
STL10
mordemoddeeeee (-) variance
0.7
ResNet18
0.6
ResNet34
0.6
ResNet50
0.5
0.5
Inter mode (+) Intra Intrammode
0.5
0.4
ResNet34
0.4
0.3
0.3-
ResNet18
0.2
0.2
ResNet34
0.2
ResNet55
WRes0.2
WReel
WRes10
WRes0.2 2
WRes1
WRes10
Figure 14: Inter- and intra mode variances on CIFAR100 (left), STL-10 (middle), and SVHN (right) using different ResNet backbones.
WRes0.2 2
WRes1
WRes10
1
Mode
Mode
Mode
Figure 15: Feature Visualizations of class Bird that we sample from different modes, c.g. MultiSWAG members, that were trained the STL-10 dataset. We can that the inter-mode variance decreases, and the intra-mode variance increases, when increasing the netw width from left to right.
WRes0.2
WRes1
WRes10
Mode 1
Mode
Mode
Mode
Mode
Mode
Mode
Mode
Mode
8
Figure 16: Feature Visualizations of class that we sample from 3 different modes, e.g. MultiSWAG members, that were trained on the SVHN dataset. We can see that the inter-mode variance decreases, and the intra-mode variance increases, when increasing the network width from left to right.
24

---

Published in Transactions on Machine Learning Research (11/2023)
Pretrained
Scratch
Mode
Mode 2
Mode 3
Mode 1
Mode 2
Mode
Figure 17: Comparison of Feature Visualizations from three different modes, e.g. KFAC models, that were first pre-trained on Net and then fine-tuned CIFAR10 (left), and three different modes that were trained from scratch CIFAR10 (right). The concepts that present in the FVs look similar for both pre-trained and from-scratch scenarios.
pre-trained
40
0.8
from-scratch
tSNE component 2
20
FWar
0
0.4
-20
0.2
-40
-20
40
Avg. single mode
mades
tSNE component
Figure 18: Multimodal structure (left) and FVVar (right) of KFAC models trained in the pre-trained (blue) and the from-scratch scenarios (red). The FVs the models trained in both scenarios have qualitatively similar cluster structures, and quantitatively similar single-mode and all modes FVVar values.
25


Reviews:
[{"review_id": "aMLpBsX0oi", "reviewer": "Reviewer_FCqy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n* The paper tackles a generally important problem with interesting insights: How can the behaviour of BNNs understood such that it's interpretable/exaplainable.\n* A strong empirical result is their demonstration that different Bayesian inference methods affects the diversity of BNN instances.\n\nWeakness\n* The four questions the authors pose in the introduction seem only weakly related. Why would one tackle those 4 aspects jointly? Specifically, what does a very specific architecture related aspect like the width of a network have to do with the other rather generic questions.\n* The paper presents itself as a mostly qualitative type of work. For this however, the breadth of experiments/datasets seems to be too small. The experiments shown in the paper could be easily cherrypicked.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims at improving understanding of the decision-making\nstrategies of BNNs in the realm of Explainable AI. It provides qualitative insights through feature\nvisualizations and quantitaive insights by contrastively learned\ndistance metric.\n\nFeature visualizations are provided through the use of the Activation Maximization\nframework. These features need to be mapped to latent representations in\norder to be quantitatively anlyzed. This mapping is learned through SimCLR.\n\nThe paper gives answers to the following questions:\n1. Can the diversity of BNN instances be explained by global XAI methods?\n2. Does the choice of the Bayesian inference method affects the diversity of their feature visualization?\n3. Can the uncertainty estimates provided by a BNN be explained by the diversity of their feature\nvisualizations?\n4. How does the network width affects the diversity of explanations of samples from a multimodal\nposterior distribution?", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The underlying motivation for the relevant directions the paper investigates needs to be made clearer (or maybe only tackle two out of the 4 questions mentioned in the introduction). For a strong experiment paper, much more datasets need to investigated with the presented methodology, much more concepts (in the way the paper uses it) need to be presented.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V8s0GxgyMs", "reviewer": "Reviewer_gAeF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall Review\n===========\nI found this work very interesting, particularly how neural network width interacts with multi-modal Bayesian solutions and how it relates to the NTK. In that sense, I think this work would be of interest to the research community and it should be accepted.\n\nStrengths\n=======\n* XAI is an increasingly important topic as ML becomes widespread in real-life applications.\n* The comparison between different Bayesian estimator is interesting\n* The paper is well-written and easy to read\n\nWeaknesses\n=========\n* The way the contrastive learning method is trained is not clear to me. If I understand it well, you train a resnet18 with SimCLR on a dataset of FVs. Do you train one resnet per Bayesian network? How do you do it for MultiSWAG? \n* Figure 1's caption is too close to the text below, which makes readers lose time looking for where the text continues.\n* Could you give some details on why the choice of Places and CIFAR? \n\n**Minor**\n* There are many mentions to *Place365*. It should be *Places365* instead.\n* Section 4.1: *which is classify correctly* -> *which is classified correctly*", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors analyze feature visualization (FV) techniques on Bayesian Neural Networks (BNNs). They show that 1) FV can be used on BNNs, 2) With the help of contrastive learning to define a distance measure between FVs, they show that different BNN methods yield different levels of diversity in their FV, 3) the diversity of FVs correlates with uncertainty measures, and 4) wider networks tend to reduce FV diversity between different weight modes while it increases within a mode. The authors complement these points with illustrative qualitative results on the produced FVs, like Figure 1.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Clarify how the FV dataset is built to train SimCLR\n* Fix space after Figure 1\n* Explain the choice of datasets\n* Fix other minor problems (see above)", "broader_impact_concerns": "I do not foresee any relevant ethical concern from this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "skOLgKzywo", "reviewer": "Reviewer_WYUH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is clearly written and well structured (modulo a few needed corrections listed below) \n* The general approach is reasonable, and the experimentation is well designed and leads to interesting insights:\n     * I particularly appreciated the first experiment where the authors showed that the network instances that wrongly classify a test image are focusing on a concept that is not present in the image, and how manipulating the test image by adding a small component from this misleading concept leads to a correct prediction. \n     * I also appreciated the analysis of the impact of width and the Bayesian inference methods on the FVs and linking it to the different networks behavior and to recent DNN theory. \n* The authors provide a plethora of details (including the public repositories they used in their study), making the work highly unpredictable. \n\nWeakness: \n* Minor corrections: \n   * Better separate Figure 1 caption and the main text. \n   * Paragraph 4.5: There is a lot of redundancy in this paragraph.  \n   * Eq (7): The authors gives the definition of cosine similarity instead of cosine distance. Can they confirm that the correct formula has been used in their experiments? \n* The major weakness in my opinion lies in the choice the authors made to measure the distance in the FV space. While the rationale of the choice makes a lot of sense, the distance measure is highly depending on the expressivity of ResNet18, and is then applies to inputs of variable complexity. This can make the analysis misleading, e.g. the higher inter-mode variance and lower intra-mode variance observed for wider models in Figure 7 can be a result of the incapabilities of the ResNet to disentangle them more than the properties of the FVs. Indeed, we can visually see that the FVs of the wider networks are of higher definition. See next section for suggestions. \n* The authors focus on the analysis of width on the FV diversity, and the choice of the Bayesian inference method. There are a plenty of other phenomenons that could be interesting to study for a more complete and impactful contribution. See next section for suggestion.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper looks at the question of explaining the diversity in the decision making strategies of Bayesian neural networks. While prior works looked at local explanations for BNNs (\"Explaining bayesian neural networks\", Bykov et al 2021), the authors focus here on global explanations. They conduct a study along these 4 axes: \n 1. Whether the diversity of BNN samples can be explained with global explanations \n 2. The impact of the Bayesian inference method on the explanation qualitatively and quantitatively \n 3. The correlation between the diversity of explanations and the uncertainty estimates of BNNs\n 4. The impact of architectural parameters on the explanations (the authors focus on the network width)\n\nIn order to conduct this investigation the authors chose the following setting: \n* Architectures and datasets: WideResNet28 (with varying width) for CIFAR100 and ResNet50 for Place365. To vary the width, the authors multiply the number of channels of the WideResNet by 0.2, 0.7, 1, 2 and 10. \n* Bayesian inference method: The authors conduct the study with different methods, either leading to a unimodal and a multimodal posterior: KFAC, Monte Carlo Dropout, Stochastic Weight Averaging Gaussian (SWAG) and its ensembled version (MultiSWAG). \n* Explanation method: The authors chose to focus on  the activation maximization (AM) approach (\"Synthesizing the preferred inputs for neurons in neural networks via deep generator networks\", Nguyen et al. 2016) and regularize it with randomly transforming the input with rotation, scaling and jittering. This approach produce visualization of concepts (of the same size of the input images) that the network instance relies on to make a decision, called Feature Visualization (FV). For interpretability reasons, the FVs obtained from CIFAR100 networks are upsized to 128x128. \n\nThe authors also propose a distance measure for FVs in order to conduct quantitative analyses. The main challenge here is that naive distances could not reflect the invariance of the concepts to translation and rotation. In order to alleviate this, the authors learn to embed FVs into a lower dimensional space with a ResNet18 trained with contrastive loss (SimCLR framework). They compare FVs based on their cosine similarity in this latent space. \n\nThese settings and components are used in different experiments to show the validity of using FVs to explain the decision making of BNNs and reflect their properties reasonably, and to analyse the BNNs diversity and its dependence on Bayesian inference methods and certain architectural choices. For the first set of experiments, they qualitatively analyse FVs and show that they can use them to manipulate test samples in an interesting way, they study the impact of the AM algorithm hyperparameters on the visualizations and quantitatively correlate FVs with BNN uncertainty. In the second set of experiments, they show how the different modelling choices impact the diversity of the FVs and how they cluster in the latent space. This analysis corroborate recent observations and theories for DNNs. For example, they show a high correlation between the predictive entropy obtained with different Bayesian inference methods and the diversity of their FVs. They also provide a novel visualization of the impact of width on the network behavior, in line with recent findings based on Neural Tangent Kernel analysis. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical changes: \nAs mentioned above, the major change that I think is needed to validate the findings of this paper is to remove the gap between the analysed networks and the embedding networks complexities. There two somehow obvious choices for this: \n* Either use the same architecture both for the BNNs and for the distance measure, \n* Or bring back the distance to the pixel space. The invariance issue can be dealt with by a proper use of data augmentations. \n\nStrengthening suggestions: \n* What happens if we vary the depth of the network? \n* Does this analysis generalize well to other families of networks (e.g. transformers)? \n* What is the impact of the training regime on this analysis? More precisely, my understanding is that the networks used here are trained from scratch on the selected datasets. What happens when the training starts from a pretrained network instead? ", "broader_impact_concerns": "I don't think there are any concerns from the ethical implications side in this work. However, I would encourage the authors to consider how their contribution can be used to analyse other aspects beyond the network uncertainty, such us bias. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the breadth of experiments was modest at first, the authors conducted a large number of additional experiments based on the reviewer's input. Reviewers were satisfied with the revision and additional material provided by the authors. They also found the insights valuable and worth publishing.

I recommend acceptance provided the authors conduct the additional experiment discussed with reviewer gAeF.
Decision: Accept (poster)