Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning
Authors: Cuong Pham
Cuong C. Nguyen
Trung Le
Dinh Phung
Gustavo Carneiro
Thanh-Toan Do
Venue: NeurIPS 2023 poster (2023)
Keywords: Bayesian Neural Networks
Deep Mutual Learning

Abstract: Bayesian Neural Networks (BNNs) offer probability distributions for model parameters, enabling uncertainty quantification in predictions. However, they often underperform compared to deterministic neural networks. Utilizing mutual learning can effectively enhance the performance of peer BNNs. In this paper, we propose a novel approach to improve BNNs performance through deep mutual learning. The proposed approaches aim to increase diversity in both network parameter distributions and feature distributions, promoting peer networks to acquire distinct features that capture different characteristics of the input, which enhances the effectiveness of mutual learning. Experimental results demonstrate significant improvements in the classification accuracy, negative log-likelihood, and expected calibration error when compared to traditional mutual learning for BNNs.

Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning
Cuong Pham'
Cuong C. Nguyen²
Trung Le
Dinh Phung
Gustavo Carneiro
Thanh-Toan Do
Department of Data Science and AI, Monash University, Australia
Australian Institute for Machine Learning, University of Adelaide, Australia
*Centre for Vision, Speech and Signal Processing, University of Surrey, United Kingdom
VVinA, Vietnam
I cuong cuong-phaml, trunglm, dinh, phung, toan. domomnashhhh...ooooooooo edu cuong mguyen@adelaide. www.iddeeeeddeee. edu.ed. au, g g.carneiro@surrey. ac.
Abstract
Bayesian Neural Networks (BNNs) offer probability distributions for model pa- rameters, enabling uncertainty quantification in predictions. However, they often underperform compared deterministic neural networks. Utilizing mutual learning can effectively enhance the performance of peer BNNs. In this paper, we propose a novel approach improve BNNs performance through deep mutual learning. The proposed approaches aim to increase diversity in both network parameter distribu- tions and feature distributions, promoting peer networks acquire distinct features that capture different characteristics of the input, which enhances the effectiveness of mutual learning. Experimental results demonstrate significant improvements the classification accuracy, negative higllikl,, and expected calibration error when compared traditional mutual learning for BNNs.
1 Introduction
(BNNs) provide probability distribution over the model parameters, which allows for uncertainty quantification in the model's predictions. This is especially useful in situations where making accurate predictions is crucial, and having a measure of uncertainty can inform decision-making. This measurement is essential for many applications, including continual learning, active learning, and robust operation. To measure uncertainty, we can learn the posterior distribution after observing data through methods like Variational Inference (VI) [6] and Markov The variational approach for BNNs involves selecting prior neural network weights, offering more memory-efficient and flexible approach compared to MCMC.
Bayesian neural networks 22
Chain Monte Carlo (MCMC) [37 21 3
distributions and approximate posterior distributions
However, despite their advantages, Bayesian neural networks using variational inference often find it difficult to match the performance of BNNs using the MCMC approach, as well as deterministic neural networks. To address this performance gap, various methods have been explored 26 28
24 [4]
One promising method is knowledge distillation 13 network (student) to mimic the behaviour of larger, comparable performance with fewer computational resources. This idea has been investigated for compressing BNNs 18 [6] 30). Although it is efficient to train the student in this manner, selecting the right teacher model which can guide the student during training may be difficult to a certain extent. In [18] 36 30), the authors teacher models, which are BNNs trained with Stochastic Gradient Langevin Dynamics (SGLD), resulting in prohibitive storage costs. Additionally, this knowledge
25 which involves training a smaller
complex network (teacher) to achieve
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

distillation requires a multi-stage offline process involving pre-training the teacher, which could potentially limit the mutual learning between peer BNNs.
38 has received increasing interest in recent years. Mutual learning simplifies the training process of knowledge distillation a single stage, which allows two or more networks to learn collaboratively and teach each other simultaneously, even with identical model architectures. In [39] the authors simply encourage multiple networks to learn from each other by reducing Kullback-Leibler (KL) divergence between pairs of predictions. In the same line of research, 40 proposes train a multi-branche architecture by sharing low-level layers and concurrently creating an effective teacher based on soft logits to enhance the learning of the target network. Meanwhile, in [38] 4) the authors match the distribution of feature maps through adversarial training. In [4. the authors also indicate that a direct alignment method, such as the 1.1 or 1.2 distance between features of peer models could make networks become identical. Consequently, this does not improve the performance of peer networks in mutual learning. Even though these approaches show effective results, they mainly concentrate on the soft logits or features of peer networks, with limited research exploring the impact of diversity in the model parameter space for mutual learning. In the context of deep mutual learning, increasing diversity in parameter distributions and feature representations could encourage peer networks to learn distinct perspectives of data from one another. This process would enable models to acquire richer information, leading to more accurate decision- decision-making. making.
Mutual learning or online knowledge distillation 39
40
In this work, we propose a novel approach to enhance the performance of BNNs by leveraging deep mutual learning. Specifically, we aim to increase diversity for deep mutual learning applied to BNNs. First, we propose increasing diversity model parameter space by forcing posterior distributions to diversify. Next, we improve the BNNs' performance in the context of mutual learning by diversifying feature distributions. By promoting diversity in terms of model parameters and feature distributions we encourage peer networks to learn distinct features and characteristics from one another, ultimately boosting their overall performance. Our experimental results demonstrate that increasing diversity in both model parameter distributions and feature distributions is crucial for enhancing the performance of peer BNNs. We conduct experiments on the CIFAR-10. CIFAR-106 and ImageNet datasets and observe substantial improvements in classification accuracy, negative log- log-likelihood, and expected calibration error compared to the traditional mutual learning for BNNs. Our contributions can be summarized follows:
We propose a novel approach for improving the accuracy of BNNs by leveraging the effectiveness of mutual learning. Specifically, we propose promoting diversity in both model parameter space and feature space during the mutual training of peer BNNs. To our best knowledge, this is the first work that leverages mutual learning in the context of BNNs, and also investigates the usefulness of model parameter diversity in mutual learning. Additionally, we are the first to propose maximizing the distance between feature distributions to promote diversity in mutual learning for BNNs.
We extensively evaluate our approach on CIFAR-10, CIFAR- 100, and ImageNet datasets and demonstrate that it outperforms traditional mutual learning for BNNs in terms of accuracy, negative log-likelihood, and expected calibration error.
2
Related work
Bayesian Neural Networks have emerged as an active area of research due their ability to capture uncertainty in predictions. While traditional deep learning models represent model parameters by point estimation and tend to be overconfident in their predictions (9 BNNs focus on estimating the posterior distribution over the weight parameters of neural networks. This approach can result in more calibrated predictions and better uncertainty estimations M. However, BNNs present several challenges, including intractable posterior distributions, unclear prior distributions, and a large number of parameters. To address these issues, researchers have proposed various methods for BNNs such as variational inference [6 5 23), and Markov Chain Monte Carlo (MCMC) [2]. MCMC often results a good performance, but this approach requires prohibitive computational cost and memory storage, making it impractical for real-world applications. In contrast, variational inference or mean-field variational inference (MFVT) presents a more convenient alternative by approximating the complicated posterior distribution with a simpler variational approximation. For instance, this

---

assumes a Gaussian posterior with diagonal covariance matrix M. Variational methods require storing only one model for evaluation, which is more scalable and useful in downstream tasks.
Variational inference for Bayesian neural networks. Given training dataset 8 = { Mn J with zn X' being an input and yn. E y being the corresponding output. The aim is find a model f(; w) : X V/. parameterized by w E W, that can accurately predict the output of each instance x. Bayesian machine learning, that aim is equivalent finding the posterior p( (()(S) of w given the training dataset S with a prior p(w) through the Bayes' theorem:
N
plun [T.. w) p(w)
P(w]S) = II pp TT, w w)dw
(I)
The denominator in Eq. ① is, however, intractable be calculated exactly except for some simple models. To tackle this issue, variational inference is introduced to approximate the true posterior p(w[S) by variational posterior q( u(u; 8) parameterized by 0. The variational posterior g(w; 0) can be obtained by minimizing the following KL divergence:
8" arg min [L[g(w; 8) [p([[(]]
8
arg ming KL 0) [[p[[))]]]))])) - - fq(w; 0)
E log pypmlla w)dw
arg ming KT [T[[(w;; [))[[())] + By(ww)) Em log p(yalln [n,
where p(w) the prior of the parameter w.
Bayes by Backprop (BBB) [ is a simple but efficient variational approach. They suppose the approximate posterior following Gaussian distribution with a diagonal covariance matrix, i.c., q(w;0) 0) = Nu,, diag(a²)) and apply a parametrization trick to represent each weight in the neural network by mean and a standard deviation:
w μ - 6 e~ (00, I).
where: is the Hadamard product and is the identity matrix.
However, the standard BBB method in practice is sensitive to hyperparameters. To address this issue, in (5), the authors introduce variant of BBB called Radial Bayesian Neural Network, where weight parameters are sampled as follows:
lpl for p~NN(,11))
(4)
[e||
where ||-|| denotes the vector norm.
Deep Mutual Learning (DML), also known online knowledge distillation, has gained considerable interest recently due to its adaptability in various applications. Unlike offline knowledge which requires selecting and training teacher network in advance, mutual learning involves distilling and sharing knowledge among peer networks. In [39, the authors argue that multiple models, even with identical architecture, can support each other during the learning process by minimizing the KL divergence between their soft logits. Regarding mutual learning in logits, ONE [40] generates an importance score for each student through the gate module, resulting in ensemble logits used as alignment targets for each network. In [10), optimal ensemble logits are computed and distilled among networks. However, these methods only utilize output logits without considering the potential benefits of incorporating knowledge from intermediate feature maps in peer networks. In A the authors show that directly minimizing the distance between features using L1 or L2 severely degrades the accuracy of peer networks, as it may cause models to learn in identical ways. Therefore for each network, in addition learning useful features for the task on its own, 4 also encourages each network to learn the feature distribution of its peer network through an adversarial training strategy. By doing this, they encourage both networks to learn features that generalize better. In the other words, they encourage the two networks to leam features that work well for both, ultimately resulting better performance. Despite these advancements, existing methods in mutual learning only use output logits or intermediate feature maps in peer networks and do not take advantage of knowledge from parameter space in peer networks, which could further promote diversity and play crucial role in mutual learning.
3

---

adopt knowledge distillation methods enhance the efficiency of BNNs. In [8] the authors train a teacher using SGLD [37] and then distil knowledge into student network, which is deterministic model. In [36,, the authors attempt to reduce the storage issues of MCMC methods by distilling the posterior distribution of the BNN using the Generative adversarial networks (GANs). However, this approach could be difficult for larger models with millions of parameters. With this regard, in [30], the authors treat the knowledge in the teacher as prior to constraining the variational objective of the student. However, the aforementioned works still require getting MCMC samples in advance for distilling knowledge into variational BNNs, leading to expensive computational resources and large storage. This could be unsuitable for training large BNNs. In our work, we propose to apply mutual learning to improve the performance of variational Bayesian neural networks make them applicable in real-work tasks. Specifically, to diversify features of peer BNNs, we propose to enlarge the distance between feature distributions of peer networks which is achieved by careful design of feature combination, an appropriate choice of feature distribution approximation and a distance function. More importantly, in addition to diversity in feature space, also investigate and leverage the optimal transport for encouraging diversity in model parameter space. To our best knowledge, this is the first work that investigates the usefulness of model parameter distribution diversity and also the first enlarge the distance between feature distributions in mutual learning.
Bayesian Deep Mutual Learning. Several works [36 18 30
3 Proposed method
3.1 Deep Mutual Learning for Bayesian Neural Networks
We formulate our proposed method by considering two peer Bayesian neural networks, B1 and B2, with the goal of constructing an efficient framework for BNNs in mutual learning. The core idea of mutual learning [39] allows peer models to provide de feedback on their learning to each other. This encourages peer models to acquire additional information from one another, ultimately leading to improved generalization performance. We adopt the two loss terms for logit-based learning; one is the conventional loss for variational Bayesian neural networks, and the other is mutual distillation loss between peer networks using KL divergence. In setting, the approximate posterior distributions for each peer BNN is assumed to be a Gaussian distribution with a diagonal covariance matrix:
qw: 0,) NN po, (o,g(())))) )), 1=1:2
where He,, 0g, E R4, with d denoting the number of model parameters. The conventional loss for variational Bayesian neural network B,, = 1: is expressed as:
((0,) = KL [g(w; 8,) [[))]]]]] + Bo(w::))) [- E log pa log 1xn, w)
n=
(6)
Let =B1 and B be the logit outputs of the Bayesian neural networks B1 and B2, T be the temperature parameter to control the smoothness of the output distribution. We denote soft logit 3 = softmax((=" and y _ softmax(="**/1) The logit loss terms are defined as follows:
Glggits [(1)) 2 x KL B2 11
B2 logits
[(02) + T x KL [p
B1
1p
B2T
(8)
3.2 Diversity in model parameter space
We aim to enhance diversity by leveraging the benefits of Bayesian neural networks over deterministic neural networks, in which weights in a BNN are represented by a distribution. Specifically, we increase the distance of posterior distributions and subsequently enhance the diversity in mutual learning. To the best of our knowledge, this is the first work that exploits the model parameter space to enhance diversity in mutual learning.
During the training process, for each iteration, we obtain Gaussian distributions with parameters 0, = (Wor Ee,) and 02 = (Ho,, Eo,) for BNN models B1 and B2, respectively. We then measure the distance between two posterior distributions D(g(u; 0, ), q(w; 02)). To this end, we exploit two well-known methods: Kullback-Liek (KL) divergence and optimal transport. Both of them have

---

closed-fome fored-formaae for Gaussian densities. While the KL divergence is well-known, it has limitations, such as being asymmetric and potentially infinite. Optimal transport theory [35] offers an alternative with the Wasserstein distance, which is symmetric and satisfies the triangle inequality. Furthermore, since the approximate posterior of the Bayesian neural network for the MFVT approach is Gaussian with diagonal covariance, this results in a simple form of optimal transport. Thus, we adopt the optimal transport, or Wasserstein distance, to measure the distance between two Gaussian distributions [34]
Diq(u; 01), (w:; 82)) W2(g(ur 01). q(w; 02))² ll - Hoal" B(Eo, Eo,)²
(9)
where (.,.) is the Bures distance
2 represented by:
B (Eo,: Σe.)² = tr + Ee, 2(
-0, E0,-0, 112).
9, Eg,
(10)
With diagonal covariances Me = diag( (o², ), and Lلg = diag(o (0))) we have simple form of distance between two approximate posteriors:
(( 01), g(w; 82)) Ilue, uo,11 + 1100 -0&11?
(11)
To promote diversity in the parameter space, a straightforward approach would be to add a negative D(g(w; 01), <(w; 02)) term to the objective functions in Eqs. ( and (8) for minimization. However, doing so could lead to rapid increases of ((q(w; 01) wr; 02)) which would impair the training process. Therefore, we propose minimizing the following loss function:
Ldiverse param log(1 + ((p(w( 01), (2))).
(12)
The above loss function exhibits desirable properties: Cdiverse param > O, and minimizing Ldiverse param implies maximizing ( 01), q(w; 02)), thus enhancing diversity in the model parameter space. These properties make optimization more stable during training.
3.3 Diversity in intermediate feature space
We aim to leverage intermediate features to increase the distance between the distributions of features among peer networks, rather than directly matching features using 7.2 distance as in offline distillation methods such as FitNets [27] and Attention transfer 17]. This is because directly matching features can lead to peer networks becoming identical, thereby reducing diversity in mutual learning. In our framework, the backbone networks are divided into the same blocks according to their depth. Let F. and 12 be the extracted features at the tt block of the BNN models, B1 and B2, respectively, To increase the diversity of features among peer BNN models, we aim to maximize the distance between their feature distributions. It worth noting that, diversifying peer networks' features is not a straightforward task. Our experimental results (detailed in the Supplementary Material) show that directly maximizing the L1 or L2 distance between peer networks' features does not necessarily enhance the effectiveness of mutual learning. Similarly, we observed the same outcome when directly maximizing the distance between peer networks' feature distributions. The reason could be that directly maximizing the distance between the distributions of original features FB1 and F82 could potentially keep pushing the features apart indefinitely. This leads to BNN models becoming too dissimilar to effectively learn from each other, and then in turn negatively affect the predictive abilities of the BNN models. To mitigate this, we propose to diversify fused feature distributions. By doing so, we indirectly diversify feature distributions through transformed features, which allows for more flexible feature adaptation. Additionally, the use of fused features enables to increase diversity across multiple feature levels. We leverage cross-attention (33 combine features from different blocks of each BNN. Specifically, given extracted features from the 4thh block, and (k + 1)th block of the network B1, the combined feature using cross attention can be expressed as:
BB u(F:1 : F = softmax
(W
WyFBI
(13)
where We. W and Wv represent learnable parameters for query, key, and value, respectively. Similarly, we have combined features FBB2_ u( (F FB2) Specifically, in mini-batch with samples, we denote BI = {F) y and n2 = {F) 1==1 represented fused features derived from BNN models B1 and B2, respectively.
in
Similar to the measurement of distance in the parameter space, we can utilize optimal transport. However the feature distributions may not be clearly represented by Gaussian distributions and
5

---

Algorithm Diversity for Bayesian Neural Networks in Mutual Learning.
1: procedure TRAIN(S, Tbatch size: Repochs)
2:
DSS (3, training datasel
3:
Abbttch size: size of a mini-bati batch
4: nepoohhs: number of training epochs
5:
;0,, 0e, & RANDOM INITIALIZATION
> Initialize net Bl
6:
H0z < PRE-TRAN TRE-TRAINED MODEL.
b Initialize net
7:
Oo, < RANDOM INITIALIZATION
8:
for epoch - nepochs do
9:
for iter in iterations do
10:
sample a mini-batch: == {(rr, y/) (1i4, yi) ~ s]
Bbatch size
11:
> Compute losses shared between two peer networks compute: Dor WASSERSTEIN DISTANCE( (Me, o (,,,,,((H,, ( compute diverse parameter distribution loss: Ldiverse param
12:
Eq. P Eq.
13:
12
14:
b Network B/
15:
compute logit loss for B1: logits
Eq. > Eq.
16:
compute diverse feature loss for B1: CB1
diverse feat.
17:
compute total loss for B1: LB Liogis + aL param + 3. update parameter: Pos f ADAM(C
-.vers Teat
>> Eq.
18
18:
19:
b Network B2
20:
compute logit loss for B2: CB2 logits
Eq.
21:
compute diverse feature loss for B2: CB
diverse feat
> Eq.
22:
compute total loss for B2: _ = logits + divveerse param + 3C
B2
diverse feat
> Eq.
19
23:
update parameter: |102 0o2
BD
24:
return (A01 0b 100g: 002
other forms of optimal transport often require the computation of cost matrices, which are often computationally expensive. Therefore, we adopt KL divergence to diversify feature distributions. To this end, we generate the distribution of fused intermediate features by leveraging the conditional probability density of any two data points within the feature space (32, which is expressed as:
K(F,,F) K(F,
Pilj =
(14)
where K(a, b) a kernel function. our work, we adopt the similarity metric K(a, b) ( "a"" 1) which has been used in (25) for offline knowledge distillation.
Given that Q and are the probability distributions of fused features from BNN models B1 and B2, respectively, the distance between fused feature distributions is represented as:
P(a)) !(x)
DKL B2 (BB),,BBBB-1
KL KL
P(x) log
dz.
(15)
It is worth noting that feature distribution has been investigated for offline knowledge distillation, i.e., in (25). the authors minimize KL divergence to transfer the probability distribution from teacher to student, rather than transferring actual representations. Conversely, in the context of mutual learning, our approach seeks to maximize the KL divergence between fused feature distributions to promote feature diversity. Similar promoting diversity in the parameter space, we propose minimizing the following loss functions for BNN models B1, and B2:
B1 diverse feat. = log + exp(- DD (B))
(16)
Ediverse feat. log + exp(- (DRI (B
B2
(17)

---

Finally, the overall loss of our proposed Bayesian mutual learning for Bayesian neural networks B1 and B2 are defined as:
[ Lingits &LLviverse param + divers feat:
(18)
_ = logits + or diveree param + - feat>
B2
(19)
where and 3 are the hyper-parameters of total loss for optimization.
In proposed BNNs in mutual learning strategy, the peer networks are jointly optimized and collaboratively learned during training. The optimization details are presented in Algorithm
4 Experiments
4.1
Experimental setup
Datasets. We evaluate our approach widely used datasets, including CIFAR-10 (2),. CIFAR-100
19]. and ImageNet [ The CIFAR- 1-0 dataset consists of 60,000 images across 100 classes, with 50,000 and 10,000 images used for training and validation, respectively. Meanwhile, the CIFAR-10 dataset has the same number of images for both training and validation sets but only consists of 10 classes. Additionally, we conduct experiments on ImageNet- large and challenging dataset with 1,000 classes, This dataset contains about 1.2 million images for training and 50,000 images for validation, which we use test set in our experiments.
Implementation details. We evaluate the proposed BNNs-mutual learning with different ResNet models [II]. Specifically, on CIFAR-10 and CIFAR 100 datasets, we evaluate our method with ResNet20, ResNet32 and ResNet56 architectures, while on the large scale ImageNet dataset, evaluate our method with ResNet18 architecture. On CIFAR-10 and 100,R-100, we train all models for 300 epochs, divided into two stages. In the first stage, train peer networks in mutual learning manner until convergence without the loss term that enhances diversity in features by setting the / in Eq. and Eq 19 to O. Specifically, we train BNNs for up 200 epochs. We use the Adam optimizer with an initial learning rate of 0.001, which is divided by 10 after 80, 120, 160, and 180 epochs. In the second stage, which comprises the last 100 epochs, BNNs-mutual learning and use the convergence from the first stage as an initial point for the training procedure. We continue training for the last 100 epochs in which the learning rate is reset to 0.001 and then is divided by 10 after 230, 260, and 290 epochs. The networks are divided into 3 blocks, and we combine features extracted from block and block 3 of each BNN for feature diversity. In all experiments, the temperature T is set to 3 (Eq. 7 and Eq. 8. and a and / (Eq. 18 and Eq. 19) are set and 2, respectively. On the ImageNet dataset, we train BNN models for up to 45 epochs. For the first 30 epochs, we set the / to 0 and use the Adam optimizer with an initial learning rate of 0.001, which is divided by after 15, 20, and 25 epochs. For the last 15 epochs, we reinitialize the learning rate to 0.001 and divide it by 10 after 35 and 40 epochs. We divide the networks into 4 blocks and combine features extracted from block 2, block 3, and block 4 of each BNN for feature diversity. The temperature T, a, and 3 (Eq. T8 and Eq. 19] are set to 1. For all settings, we use pair of Bayesian neural network models with the variational parameters to approximate posteriors (Eq. 6 one is initialized from scratch, and the other is initialized with the mean μ from the deterministic model trained on the same dataset. We empirically find that this setting achieves better results compared to peer models that are initialized under the same conditions. We sample single network during the training phase, while for the inference phase, we conduct an ensemble of 50 sample models across all settings. All the reported results average of 3 trials.
integrate the diversity in feature space into our
4.2 Experimental results
In this section, we compare our proposed method against deterministic models (DNN), Bayesian neural networks trained with mutual learning (DML) and trained without mutual learning (vanilla). In the vanilla approach, each BNN model is trained individually. We evaluate BNNs using three metrics including top-1 classification accuracy (ACC), negative log-likelihood (NLL), and expected calibration error (ECE) (9. We first sample from the posterior distribution of the weights. We then average the predicted probabilities to calculate ACC and evaluate the negative log-likelinn ligeeekld of these probabilities for NLL, Meanwhile, to compute ECE, we categorize data samples into 20 bins based
7

---

Table 1: Top-1 classification accuracy, NLL, and ECE on CIFAR-100 dataset. *Bayesian neural networks are initialized with the mean value from the pre- pre-trained deterministic model.
1
NLL.
ECE.
DNN
Vanilla
DML
Ours
DNN
Vanilla
DML
Ours
DNN
Vanilla
DML
Ours
ResNet20
69.13
63.18
67.27
68.32
1.106
1.447
1.174
1.101
0.065
0.119
0.057
0.041
ResNer20*
67.62
69.61
70.45
1.438
1.073
1.043
0.152
0.047
0.038
Average
65,40
68.44
69.39
1,443
1.124
1.072
0.136
0,052
0,040
ResNet32
71.36
65.20
68.59
70.53
1.074
1.530
1.169
1.029
0.080
0.150
0.087
0.043
ResNet32*
69,76
71,45
72.14
1,542
1,012
0.975
0.173
0,064
0,040
Average
67.48
70.53
71.34
1,536
1.091
1.002
0.162
0.076
0.042
ResNet56
72.88
66.60
70.44
71.90
1.146
1.853
1.117
1.024
0.153
0.204
0.092
0.066
ResNet56
71.94
73.79
74.20
1.764
0.997
0.954
0.189
0.085
0.069
Average
69.27
72.12
73.05
1.809
1.057
0.989
0.197
0.089
0.068
Table 2: Top-1 classification accuracy, NLL, and ECE on CIFAR-10 dataset. *Bayesian neural networks are initialized with the mean value from the pre-trained deterministic model.
1
N
BCE
DNN
Vanilla
DML
Ours
DNN
Vanilla
DML
Ours
DNN
Vanilla
DML
Ours
ResNet20
91.83
89.93
91.09
92.03
0.302
0.445
0.295
0.256
0.067
0.066
0.040
0.029
ResNet20
92.47
92.67
92.89
0.383
0.232
0.228
0.055
0.031
0.025
Average
91.20
91.88
92.46
0.414
0.264
0.242
0.061
0.036
0.027
ResNet56
92.67
91.64
92.47
92.58
0.272
0.488
0.311
0.295
0.063
0.064
0.057
0.053
ResNet56*
93.33
93,50
93,91
0.431
0.243
0.237
0.054
0.041
0.039
Average
92.49
92.99
93.25
0.460
0.277
0.266
0.059
0,049
0.046
Table 3: Top-1 classification accuracy, NLL, and ECE on ImageNet dataset. * *Bayesian neural networks are initialized with the mean value from the pre-trained deterministic model.
1
ECE
DNN
Vanilla
DML
Ours
DNN
Vanilla
DML
Ours
DNN
Vanilla
DML
Ours
R8sNet18
69.76
64.30
66.88
67.36
1.247
1.505
1.357
1,343
0.130
0.134
0.134
0.131
ResNet18*
66.95
67.65
68.16
1.389
1.335
1,311
0.132
0.130
0.129
Average
66.63
67.27
67.76
1.447
1.346
1.327
0.133
0.132
0,130
on their confidence scores and measure the difference between the model's predicted probability and its actual accuracy.
Comparative results on CIFAR-100 and CIFAR-10. We present top-1 accuracy (%) (ACC), NLL, and ECE on both the CIFAR- and CIFAR-IO datasets for various peer BNNs, as shown in Tables I and In terms of the average accuracy of peer BNNs in mutual learning, which represents the mean accuracy from the peer models, our proposed method consistently outperforms the BNNs trained with traditional mutual learning (39], with the highest improvement of 0.95% for ResNet20 on CIFAR-IO0 and 0.58% for ResNet20 on CIFAR- 10, respectively. In terms of BNNs initialized with the mean value from the pre-trained deterministic model, our approach consistently outperforms the others on both datasets. The highest improvement is with ResNet20 architecture, i.e., our method outperforms DML 0.84% in terms of top-1 accuracy on the CIFAR- 100 dataset. Regarding calibration results and negative log- ligliikel,oood, we also achieve better performance compared to other approaches across all pairs. Additionally, our BNN models that are initialized with the mean value from the pre-trained deterministic models outperform the deterministic models (DNN) in all metrics ACC, NLL, and ECE.
Comparative results on ImageNet. To further demonstrate the effectiveness of our proposed method, we conduct an experiment on the large-scale and complex ImageNet dataset, employing peer BNN models of 1esNet18. As shown in Table 3 BNN networks trained with our method outperform those trained with DML [39] showing gain of 0.49% for the average accuracy of two peer BNN networks and a gain of 0.51% % for the BNN initialized with the mean value from the pre-trained deterministic model. This indicates that our method is applicable to large-scale datasets.
8

---

Table Impact diversity for each term in our proposed method terms of top-1 classification accuracy. *Bayesian neural networks are initialized with the mean value from the pre-trained deterministic model.
CIFARI00
Setting
Liogits
Ldiverse param
diverse feat
ResNet20
ResNet20#
Average
(a)
67.27
69.61
68.44
(b)
67.78
70.22
69.00
(c)
V
67.57
70.04
68.81
(d)
68.32
70.45
69.39
Table 5: Comparison of top-1 classification accuracy on CIFAR-100 dataset when only partial data are retained based on the uncertainties of the predictions. *Bayesian neural networks are initialized with the mean value from the pre-trained deterministic model.
Method
Model
20% data retained
40% data retained 60% data retained 80% data retained
ResNet32
98.90
92.75
83.22
74,34
Vanilla
ResNet32
99.10
95.38
87.43
78.29
ResNet32
99.20
94.83
86.33
77.43
DMI.
ResNee332
98.95
95.43
87.73
80.10
ResNet32
99.955
94.80
87.27
78.59
Ours
ResNet32*
99.25
95.88
88.25
80.60
4.3
Ablation studies
Impact of diversity in our proposed method. In this section, we investigate how different diversity approaches in our proposed method contribute to the performance of variational BNNs in a mutual learning setting. All experiments are conducted on the CIFAR-100 dataset with peer BNNs using ResNer20. Table we present the top-1 accuracy BNNs-mutual learning various settings. The results show that when promoting diversity parameter space, there are 0.51% and 0.61% improvements over the mutual learning that only uses Llogits for the BNNs initialized from scratch and initialized with the mean value from the deterministic neural network, respectively. When promoting diversity in feature space, the improvements are 0.3% and 0.43% over the same settings, respectively. This indicates that promoting diversity in parameter space has a more significant impact on model performance. By including both diversity in parameter and feature spaces, the highest improvement is achieved. This demonstrates that both loss terms complementary to each other in achieving better performance in BNNs-mutual learning.
The ability of uncertainty estimation. To evaluate uncertainty estimation, we adopt Bayesian active learning by disagreement (BALD) [7]. which quantifies the mutual information between the posterior distribution of parameters and the predictive distribution. We follow the evaluation methodology presented in Bayesian deep learning (BDL) benchmarks [6] to compare the accuracy in which only partial data are retained depending on uncertainty estimates obtained from the proposed method. Specifically, we compute uncertainty 7 for every testing image. Then we keep top images with the lowest uncertainty and report the top-1 accuracy of those kept images. This provides more comprehensive evaluation metric of model performance and the ability to model uncertainty. Table 5 shows that accuracy increases when fewer data are retained on networks. Furthermore, BNNs trained with our proposed method perform better than those trained without promoting diversity in mutual learning and those trained without mutual learning. This indicates that proposed method produces more reliable uncertainty than the original mutual learning.
5 Conclusion
In this paper, we propose a novel approach to enhance BNNs by leveraging the effectiveness of mutual learning. Our approach encourages diversity in both model parameter space and feature space during the mutual learning of peer BNNs. By promoting such diversity, we encourage peer networks

---

to learn distinct features and perspectives from one another. This enhances the mutual learning of the peer networks. We extensively evaluate our method on CIFAR-IO, CIFAR- 100, and ImageNet datasets, demonstrating that the proposed method outperforms BNNs trained with and without mutual learning while producing reliable uncertainty estimates. We expect that our work will broaden the topic of mutual learning in which the model parameter space is taken into account, in addition to the traditional approaches that only consider the feature space.
Acknowledgments
This work was partly supported by ARC DP23 grant DP230101 and by the Air Force Office of Scientific Research under award number 142386-23-1-4044
References
[1] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In ICML, 2015.
[2] Donald Bures. An extension of Kakutani's 's theorem on infinite product measures the tensor product of semifinite w" -algebrss. Transactions of the American Mathematical Society, 135:199-212, 1969.
[3] Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In ICML, 2014.
[4] Inseop Chung, SeongUk Park, Jangho Kim, and Nojun Kwak. Feature-map-level online adversarial knowledge distillation. In /CML, 2020.
[5] Sebastian Farquhar, Michael Osborne, and Yarin Gal. Radial Bayesian Neural Networks: Beyond discrete support in large-scale Bayesian deep learning. AISTATS, 2020.
[6] Angelos Filos, Sebastian Farquhar, Aidan N Gomez, Tim GJ Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Amoud De Kroon, and Yarin Gal. A systematic comparison of bayesian deep learning robustness in diabetic retinopathy tasks. arXiv, 2019.
[7] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty
in deep learning. In ICML, 2016.
[8] Soumya Ghosh. Jiayu Yao, and Finale Doshi-Veles Structured variational learning of Bayesian neural networks with horseshoe priors. In ICML, 2018
[9] Chuan Guo, Gcoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modemn neural networks. In ICML, 2017.
[10] Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In CVPR, 2020.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016
[12] Byeongho Heo, Jcesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi, comprehen- sive overhaul of feature distillation. In /CCV, 2019.
[13] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the Knowledge in a Neural Network. NIPS Deep Learning and Representation Learning Workshop. 2014.
[14] Mohammad Khan, Didrik Niclsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable Bayesian deep learning by weight-perturbatinn in adam. In /CML,, 2018.
[15] Jangho Kim, Minsung Hyun, Inseop Chung, and Nojun Kwak. Feature fusion for online mutual knowledge distillation. In ICPR, 2021.
[16] Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. NewrlPS, 2015.
[17] Nikos Komodakis and Sergey Zagoruyko. Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
[18] Anoop Korattikara Balan, Vivek Rathod, Kevin P Murphy, and Max Welling, Bayesian dark knowledge. NeurIPS, 2015.
[19] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009,
[20] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced rescarch), 2010.
[21] Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose Bayesian inference algorithm. NeurlPS, 2016.
[22] Radford M. Neal. Rayesian learning for neural networks, volume 118. Springer Science & Business Media,
2012.
[23] Van-Anh Nguyen, Tung-Long Vuong, Hoang Phan, Thanh-Toan Do. Dinh Phung, and Trung Le. Flat seeking bayesian neural network. In NeurlPS, 2023.
[24] Victor M-H Ong, David Nott, and Michael S Smith. Gaussian variational approximation with a factor
covariance structure. Journal of Computational and Graphical Statistics, 27(3)466-478, 2018.
[25] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge
transfer. In ECCV, 2018
10

---

[26] Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks.
In ICLR, 2018.
[27]
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang. Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep
In ICLR, 2015. Simone Rossi, Sebastien Marmin, and Maurizio Filippone. Walsh-Hadamard variational inference for
[28]
Bayesian deep learning. In NewrIPS, 2020.
[29]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bersstein, et al. Imagenet large scale visual recognition challenge.
IJV, 2015.
[30]
Gehui Shen, Xi Chen, and Zhihong Deng. Variational learning of Bayesian neural networks via Bayesian
dark knowledge. In ICCAI, 2021.
[31] Jakub Swiatkowski, Kevin Roth, Bastiaan Veeling, Linh Tran, Joshua Dillon, Jasper Snook, Stephan Mandt, Tim Salimans, Rodolphe Jenatton. and Sebastian Nowozin. The k-tied normal distribution: A compact
parameterization of gaussian mean field posteriors in Bayesian neural networks. In ICML, 2020. [32] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, 2008.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention
you need. NIPS, 2017.
[34] Cédrie Villani. Topics optimal transportation. American Mathematical Soc., 2021.
[35] Cédric Villani et al. Optimal transport: old and new. Springer, 2009.
[36] Kuan-Chich Wang, Paul Vicol, James Lucas, Li Gu, Roger Grosse, and Richard Zemel. Adversarial
distillation of Bayesian neural network posteriors. In ICML, 2018.
[37] Max Welling and Yee W Teh. Bayesian learning stochastic gradient langevin dynamics. In ICMI,,
2011.
[38]
Xiaobing Zhang. Shijian Lu, Haigang Gong, Zhipeng Luo, and Ming Liu. Amin: adversarial- mutual
learning network for online knowledge distillation. In ECCV, 2020.
[39]
Ving Zhang. Tao Xiang, Timothy M Hospedales, Huchuan Lu. Deep mutual learning. In CVPR, 2018. Xiatian Zhu, Shaogang Gong, et al, Knowledge distillation by on-the-fly native ensemble, NeurlPS, 2018
[40]


Reviews:
[{"review_id": "BHjFq8MaDJ", "reviewer": "Reviewer_xPiJ", "summary": "This paper presents a novel method for enhancing the performance of Bayesian Neural Networks (BNNs) by employing deep mutual learning. The proposed approach aims to enhance the diversity of both network parameter distributions and feature distributions, encouraging individual networks to capture unique characteristics of the input data. The effectiveness of the proposed method is demonstrated on datasets, including CIFAR10, CIFAR100, and ImageNet.", "questions": "3 block resnet is used for CIFAR experiments while 4 block resnet is used for ImageNet experiments. Why different form of resents are used for different datasets?", "limitations": "Limitations are shortly addressed in the supplementary.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method improves performance and uncertainty estimation while reducing the expected calibration error (ECE).  The technical approach is novel as the method introduces mutual learning in the context of BNNs and first to propose maximizing the distance between feature distributions and parameter distributions. The paper includes large scale data experiments (ImageNet) and ablation studies to demonstrate the effectiveness of each technical contribution introduced in this paper.", "weaknesses": "The previous studies mentioned in the paper utilize alignments on feature maps [4] or predictions [38], rather than diversifying them. In contrast, the proposed method diversifies both feature distributions and parameter distributions which is an opposite approach to the previous works. Interestingly, both alignment-based and diversification methods improves performance over vanilla BNNs, as indicated in Table 1, 2, 3, and 5. However, the paper does not explicitly explain the reasons behind the performance improvements resulting from these contrasting approaches.\n\nGiven the observed contradicting results in the experiments, where the alignment-based method (DML [38]) also enhances the performance of BNNs, an important question arises: could combining alignment-based methods with parameter diversification further improve BNN performance? Alternatively, is it necessary to diversify both feature and parameter distributions to achieve significant improvements?\n\nIn the experiment section, the proposed method is only compared with [38] and not with [4]. \n\nHyperparameters used for CIFAR experiments and ImageNet experiments are different. However, the paper does not describe details regarding the hyperparameter tuning or determination.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n\n1. The previous studies mentioned in the paper utilize alignments on feature maps [4] or predictions [38], rather than diversifying them. In contrast, the proposed method diversifies both feature distributions and parameter distributions which is an opposite approach to the previous works. Interestingly, both alignment-based and diversification methods improves performance over vanilla BNNs, as indicated in Tables 1, 2, 3, and 5. However, the paper does not explicitly explain the reasons behind the performance improvements resulting from these contrasting approaches.\n\n**Answer:** \nOur method does not contradict to the original DML work [38]. Actually, both ours and [4] are built on top of [38]. Specifically, consider a pair of peer networks, to encourage each network to learn and teach each other, DML [38] tries to match the prediction distributions of the two networks through minimizing the KL distance. Following [38], both ours and [4] have the KL loss term for each network loss.\nComparing to [4], ours and [4] are two different approaches for enhancing the mutual learning. The idea of [4] is that for each network, in addition to learning useful features for the  task on its own, [4] also encourages each network to learn the feature distribution of its peer network through an adversarial training strategy. By doing this, they encourage both networks to learn features that generalize better. In the other words, they encourage the two networks to learn features that work well for both.  Different from [4], our approach encourages each network to learn a different set of features that are good for the task. In the other words, our approach encourages each peer network to capture different characteristics of the input. It is worth noting that our approach encourages the diversity in intermediate features. For the last layer, we have the soft logit alignment similar to DML to encourage each network to learn and teach from each other. \nTo summarize, ours and [4] are two different approaches to enhance mutual learning. [4] aims to learn generalized features, while ours aims to make each network to learn different characteristics of the input. \n\nCombining our idea and [4] may further improve mutual learning, however, this is not the focus of our paper. It is also worth noting that in [4] the authors do not focus on Bayesian Neural Networks. In addition, their implementation is not publicly available. This makes it difficult for us to compare to [4] in the context of BNNs. In the efforts to compare to [4], we reimplement the approach in [4] for the BNNs context. To make a fair comparison, both ours and [4] use ResNet20 as the backbone. For the discriminator of [4], we follow the description in their paper. The comparisons between ours and [4] on CIFAR100 are in the following table. The results show that both ours and [4] improve the performance of BNNs. However, our approach achieves better performance than [4] in terms of ACC, NLL, and ECE.\n\nTable: Comparative results of our proposed method with deterministic models and [4] in terms of Top-1 classification accuracy, NLL, and ECE on CIFAR-100 dataset.*Bayesian neural networks are initialized with the mean value from the pre-trained deterministic model. DNN means the deterministic model.\n\n|             | ACC↑       |        |       |     | NLL↓       |       |       |       | ECE↓       |        |      |      |\n|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n|             | DNN         | DML         | [4]         | Ours        | DNN         | DML         | [4]         | Ours        | DNN         | DML         | [4]         | Ours        |\n| ResNet20    | 69.13       | 67.27       | 68.18       | 68.32       | 1.106       | 1.174       | 1.154       | 1.101       | 0.065       | 0.057       | 0.047       | 0.041       |\n| ResNet20*   | -           | 69.61       | 70.26       | 70.45       | -           | 1.073       | 1.039       | 1.043       | -           | 0.047       | 0.045       | 0.038       |\n\n2. The paper does not describe details regarding the hyperparameter tuning or determination.\n\n**Answer:** Please refer to the general response.\n## Questions\n1. 3 block resnet is used for CIFAR experiments while 4 block resnet is used for ImageNet experiments. Why different form of resents are used for different datasets?\n\n**Answer:**  Regarding the model architectures when evaluating on CIFAR100 and ImageNet, we follow the previous works in knowledge distillation [4, 12, 13, 24, 38, 39]. \nTypically, for large datasets like ImageNet with an input size of $224\\times 224$, people often use ResNet models with 4 blocks such as ResNet18. Meanwhile, for CIFAR-10 and CIFAR-100, which have smaller input size, i.e., $32 \\times 32$, people usually utilize ResNet models with 3 blocks such as ResNet20, ResNet32, and ResNet56. These models have much fewer channels per convolutional layer and a reduced number of blocks (3 instead of 4). \n  It is worth noting that although the ResNet20, ResNet32, and ResNet56 models (3 blocks) are deeper than ResNet18 (4 blocks), they have significantly fewer channels per layer compared to ResNet18. For example, the last conv. layer of ResNet20 has 64 output channels, while the last conv. layer of ResNet18 has 512 output channels.\n"}, {"review_id": "D3RwOqmMN7", "reviewer": "Reviewer_tB5V", "summary": "The paper focuses on improving the accuracy of BNNs by promoting diversity in both parameter space and feature space while training two peer BNNs with mutual learning between them. More specifically, they train two variational BNNs with a mean-field Gaussian variational loss for each along with a KL divergence term between the (temperature-scaled) predictive distributions of the two models, a Wasserstein distance term between the corresponding approximate posterior distributions across the two models (added as a softplus(-distance) term), and a KL divergence term between corresponding feature distributions. On the latter term, instead of directly maximizing the distance between corresponding feature distributions, they instead do so on \"fused feature distributions\". To do so, they use learned cross-attention to fuse the features from multiple feature levels in a model (two at a time). Then, they use the KL divergence between the distributions of the fused feature distributions of the two peer networks. To derive the distributions, they use the conditional probability density defined as $p_{i|j} = \\frac{K(F'_i, F'_j)}{sum_{k=1, k \\noteq i}^n K(F'_k, F'_j)}$, where $K(F'_a, F'_b)$ is a kernel function between two fused feature representations. Given those conditional probs, they compute a KL divergence term. Similar to the parameter space diversity term, they add this term to the loss as softplus(-divergence). The paper claims to be the first to propose maximizing the distance between feature distributions to promote diversity. In terms of experiments, the paper includes results for ResNet models on CIFAR-10/100 and ImageNet, measuring accuracy, NLL, and ECE as metrics, and comparing different approaches.", "questions": "Main:\n- In the experiments, a few details are currently unclear. The following points are on Table 1, but generalize to all three tables. Please clarify these details here and in the paper.\n  - Consider the ResNet20 section of Table 1. Is my understanding correct that the \"ResNet20\" results are for a pair of BNNs trained from scratch, while the \"ResNet20*\" results are for a pair of BNNs trained with the approximate posterior means set to the values from a deterministic model?\n  - Is it correct that all results (all three metrics across all three approaches) are computed after averaging the predicted probs from the pair of models?\n- For the experiments, a deterministic baseline is missing. Given the intro that discusses how BNNs can lag behind deterministic models in acc (though not always), the experiments lack a comparison. It would be helpful to understand how the proposed approach compares to a deterministic baselines, specifically a single deterministic model and a size-2 deep ensemble. Could you add this as a baseline? I would consider this to be a blocker for the paper given the framing and relevant literature.\n\nOther:\n- The KL divergence term is scaled by the square of the temperature -- why?\n- How did you choose the values for temp, alpha, and beta? They differ between CIFAR-10/100 and ImageNet. Did you ablate values?\n\nMinor comments:\n- updating lines 17 & 22 of Alg 1 could be helpful for readability", "limitations": "No limitations are included.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper does a great job of precisely articulating the modeling approach, and discussing the relevant background info. More specifically, the proposed approach of adding terms to promote diversity in parameter and feature space is clear and would be easy to reimplement.", "weaknesses": "My main concern is with the experiment section. More specifically, a few key details are unclear in the text, and importantly a deterministic baseline is missing that I believe should be present given the framing of the paper and relevant literature. Please see the Questions below. Given updates, I believe the paper would be great and I would gladly update my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n1. A few key details are unclear in the text, and importantly a deterministic baseline is missing.\n\n**Answer:** We acknowledge the reviewer's comment. Accordingly, we present here the results of the deterministic baseline in the following table.  The results indicate that our BNN models that initialized with the mean value from the pre-trained deterministic models outperform the deterministic models in all metrics ACC, NLL, ECE. \n\nTable: Comparative results of our proposed method with deterministic models and 2-size deep deterministic ensemble in terms of Top-1 classification accuracy, NLL, and ECE on CIFAR-100 dataset. *Bayesian neural networks are initialized with the mean value from the pre-trained deterministic model. DNN means the deterministic model (ResNet20 or ResNet32). Size-2 DNN means deep ensemble of 2 deterministic models. For a size-2 deterministic deep ensemble, we separately train two deterministic models, which have the same architecture, but with different initializations. After training, we ensemble 2 models by averaging the probability outputs.\n\n|                |          | ACC↑            |          |          |          | NLL↓            |          |          |          | ECE↓            |          |          |\n|----------------|----------|------------------|----------|----------|----------|------------------|----------|----------|----------|------------------|----------|----------|\n|                | DNN      | size-2 DNN       | DML      | Ours     | DNN      | size-2 DNN       | DML      | Ours     | DNN      | size-2 DNN       | DML      | Ours     |\n| ResNet20       | 69.13    | 71.91            | 67.27    | 68.32    | 1.106    | 0.979            | 1.174    | 1.101    | 0.065    | 0.058            | 0.057    | 0.041    |\n| ResNet20*      | -        | -                | 69.61    | 70.45    | -        | -                | 1.073    | 1.043    | -        | -                | 0.047    | 0.038    |\n| ResNet32       | 71.36    | 74.10            | 68.59    | 70.53    | 1.074    | 0.924            | 1.169    | 1.029    | 0.080    | 0.061            | 0.087    | 0.043    |\n| ResNet32*      | -        | -                | 71.45    | 72.14    | -        | -                | 1.012    | 0.975    | -        | -                | 0.064    | 0.040    |\n\n## Questions\n1. Consider the ResNet20 section of Table 1. Is my understanding correct that the \"ResNet20\" results are for a pair of BNNs trained from scratch, while the \"ResNet20*\" results are for a pair of BNNs trained with the approximate posterior means set to the values from a deterministic model? \n\n **Answer:**  As outlined in lines 262-265, we employ a pair of BNN models; one model is initiated from scratch, referred to as \"ResNet20\", while the other is initialized with the mean $\\mu$ is from the pretrained deterministic model, denoted as \"ResNet20*\".  Thus, in Table 1, the results associated with \"ResNet20\" and \"ResNet20*\" are obtained by training a pair of BNNs, one initialized from scratch (ResNet20) and the other initialized (ResNet20*) with the mean $\\mu$ from the pretrained deterministic model. \n\n2. Is it correct that all results (all three metrics across all three approaches) are computed after averaging the predicted probs from the pair of models? \n  \n  **Answer:** The average results in tables 1,2, and 3 in the main paper represent only the average of results from model 1 and model 2 in a pair of models. This will be clarified further in the paper.\n\n3. It would be helpful to understand how the proposed approach compares to deterministic baselines, specifically a single deterministic model and a size-2 deep ensemble.\n\n  **Answer:** For the comparative results between the proposed approach and the deterministic model and a size-2 deterministic deep ensemble, please refer to the table above. The results show that although our method's accuracy and NLL are lower than those of the size-2 deterministic deep ensemble, our method surpasses the deep ensemble in terms of ECE.\n\n4. The KL divergence term is scaled by the square of the temperature -- why? \n\n**Answer:** We adopt the distillation method using soft logits proposed by Hinton et al. [13]. The hyper parameter $T$ controls the smoothness level of the prediction distribution. As $T$ increases, the prediction distribution becomes more smooth.  It's worth noting that the magnitudes of the gradients, which derive from the soft targets, are inversely proportional to $T^2$. Therefore, we scale the KL divergence term by $T^2$. \n\n5. How did you choose the temperature parameter (T) values,$\\alpha$, and $\\beta$? They differ between CIFAR-10/100 and ImageNet. Did you ablate values? \n\n**Answer:**  Please refer to the general response.\n\n6. Updating lines 17 and 22 of Alg 1 could be helpful for readability \n\n**Answer:** We thank the Reviewer for your suggestion. We have updated the line 17 as follows:\n\ncompute the total loss for B1: \n$\\mathcal{L}^{B1} = \\mathcal{L}{\\text{logits}}^{B1} + \\alpha \\mathcal{L}{\\text{diverse param}} + \\beta \\mathcal{L}_{\\text{diverse feat}}^{B1}$\n\nand line 22 as follows:\n\ncompute the total loss for B2:\n$\\mathcal{L}^{B2} = \\mathcal{L}{\\text{logits}}^{B2} + \\alpha \\mathcal{L}{\\text{diverse param}} + \\beta \\mathcal{L}_{\\text{diverse feat}}^{B2}$\n\n"}, {"review_id": "m3NZjHJ5kM", "reviewer": "Reviewer_phoh", "summary": "The paper titled addresses the challenge of improving the performance of Bayesian Neural Networks (BNNs) by leveraging the concept of mutual learning. BNNs provide a means for quantifying uncertainty in predictions through probability distributions of model parameters. However, BNNs often fall short in performance compared to their deterministic counterparts. The authors propose a novel approach that employs deep mutual learning to enhance the capabilities of BNNs.", "questions": "The author should supplement more experiments to prove its effectiveness.", "limitations": "The author should supplement more experiments to prove its effectiveness and strengthen the conclusion.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Innovative Approach: The paper introduces a novel method that combines deep mutual learning with Bayesian Neural Networks. By promoting diversity in both network parameter distributions and feature distributions, the proposed approach enables peer networks to acquire distinct features, capturing different characteristics of the input data. This innovative technique enhances the effectiveness of mutual learning in BNNs.\n2. Detailed algorithm description: The paper provides a thorough and detailed description of the proposed algorithm for improving the performance of Bayesian Neural Networks (BNNs) through deep mutual learning.\n3. Comprehensive Experiments: The authors conduct extensive experiments to evaluate the proposed approach thoroughly. The experimental results are statistically sound and demonstrate significant improvements in classification accuracy, negative log-likelihood, and expected calibration error compared to traditional mutual learning methods for BNNs.", "weaknesses": "1. Limited variety in experimental validation: One weakness of the paper is that the proposed approach and its effectiveness are only verified through experiments conducted on Residual Neural Networks (ResNets). It would have been beneficial to include experiments on a diverse set of network architectures to demonstrate the approach's effectiveness across different model types and complexities. \n2. Lack of detailed explanation for temperature, α, and β: One weakness of the paper is the limited explanation provided for the temperature parameter (T), α, and β, which are crucial components of the proposed approach. These parameters play a significant role in controlling the diversity of network parameter distributions and feature distributions, but their specific effects and optimal values are not thoroughly discussed.\n3. Weakness in the conclusion: The current conclusion merely restates the experimental results and does not highlight the broader implications of the proposed approach or its potential impact on the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n1. Limited variety in experimental validation: One weakness of the paper is that the proposed approach and its effectiveness are only verified through experiments conducted on Residual Neural Networks (ResNets). It would have been beneficial to include experiments on a diverse set of network architectures to demonstrate the approach's effectiveness across different model types and complexities. \n\n**Answer:** In addition to evaluations with the ResNet architectures in the main paper, here we validate our proposed method on AlexNet. The following table shows the results of ACC, NLL, and ECE. In terms of BNNs initialized with the mean value from the pre-trained deterministic model, our approach consistently outperforms the others. Our method outperforms DML 0.48\\% and DNN 0.92\\% in terms of  top-1 accuracy on the CIFAR-100 dataset.\n\nTable: Top-1 classification accuracy, NLL, and ECE on CIAFR100 dataset. DNN means the deterministic model. *Bayesian neural networks are initialized with the mean value from the pre-trained deterministic model.\n\n|          |               | ACC↑     |           |           |               | NLL↓     |           |           |               | ECE↓     |           |           |\n|----------|---------------|-----------|-----------|-----------|---------------|-----------|-----------|-----------|---------------|-----------|-----------|-----------|\n|          | Vanilla       | DNN       | DML       | Ours      | Vanilla      | DNN       | DML       | Ours      | Vanilla      | DNN       | DML       | Ours      |\n| AlexNet  | 50.47         | 52.40     | 51.82     | 52.10     | 2.325         | 2.141     | 2.255     | 2.230     | 0.187         | 0.142     | 0.172     | 0.165     |\n| AlexNet* | 52.23         | -         | 52.84     | 53.32     | 2.105         | -         | 1.921     | 1.904     | 0.169         | -         | 0.099     | 0.089     |\n\n2. Lack of detailed explanation for temperature, $\\alpha$, and $\\beta$: One weakness of the paper is the limited explanation provided for the temperature parameter (T),$\\alpha$, and $\\beta$, which are crucial components of the proposed approach. These parameters play a significant role in controlling the diversity of network parameter distributions and feature distributions, but their specific effects and optimal values are not thoroughly discussed. \n\n**Answer:** Please refer to the general response.\n\n3. Weakness in the conclusion: The current conclusion merely restates the experimental results and does not highlight the broader implications of the proposed approach or its potential impact on the field. \n\n**Answer:**  Our work is the first work that explores the potential of deep mutual learning in the context of BNNs. More importantly, we are also the first to investigate the usefulness of model parameter diversity in mutual learning. We expect that our work will broaden the topic of mutual learning and inspire further researches in which the model parameter space is taken into account, in addition to the traditional approaches that only consider the feature space. In addition, our work can be considered as a baseline for DML-BNNs and we expect that our work inspires further researches that investigate the usefulness of mutual learning in the classical field BNNs. We will update our conclusion to reflect the above.  \n\n\n### Questions\n1. The author should supplement more experiments to prove its effectiveness.\n\n**Answer:**  We present above the experiments with AlexNet architecture. We also compare our work with the work [4] when [4] is applied to BNNs. We also compare our approach with deterministic models (i.e., single deterministic models and size-2 deep ensemble models). The results are presented in the tables in our responses to Reviewer xPiJ and Reviewer tB5V."}, {"review_id": "Bf7vyjA9R6", "reviewer": "Reviewer_fyNb", "summary": "The paper proposes a method to combine deep mutual learning with BNN to diversify the weight distributions of each BNN networks in a pair or ensemble, to improve performance.", "questions": "Line 178-179: The authors said adding the D(...) term will rapidly increase of this term and impact training. Wouldn't putting a smaller scaling factor for this term fix this issue?", "limitations": "None.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. AFAIK this is the first work combining mutual learning with BNN, so the authors can claim this point.\n2. The paper is in general written clearly and easy to follow.\n3. Experiments are adequate with ablation studies on individual features impact on diversity.", "weaknesses": "1. Some design choices are found to be \"empirically\" working well without too much discussion or hypothesis.\n2. Would be interesting to see how the model performs for o.o.d test data, especially uncertainty performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n1. Some design choices are found to be \"empirically\" working well without too much discussion or hypothesis. \n\n**Answer:** Please refer to the general response for the ablation studies on hyper parameters T, $\\alpha$, and $\\beta$. \n\n2. Would be interesting to see how the model performs for o.o.d test data, especially uncertainty performance. \n\n**Answer:**  We thank the Reviewer for the suggestion. To address this, we trained BNN models on the first 5 classes of the CIFAR10 dataset, referred to as CIFAR5. The data from the remaining 5 classes are considered as OOD data. To estimate the uncertainty of predictions on OOD datasets, we compute the entropy of the average prediction from 50 sampled models (for each BNN), called OOD Entropy. \n    We present the top-1 classification accuracy, evaluated on the first 5 classes, as well as the uncertainty estimation on the corresponding OOD data. The comparisons between our proposed method, the DNN model, the vanilla BNN, and the BNN with DML are presented in the following table. The results show that, when trained on CIFAR5, our method outperforms the others in terms of accuracy and uncertainty on OOD data. DNN yields the lowest uncertainty on OOD data, while our proposed method yields the highest uncertainty on OOD data.\n\nTable: Top-1 classification accuracy on CIFAR5 dataset and entropy on corresponding OOD data. *Bayesian neural networks are initialized with the mean value from the pre-trained deterministic model. We report the mean and standard deviation over 3 runs.\n\n| Method  | Model       | ACC (%)                    | OOD Entropy                    |\n|---------|-------------|----------------------------|--------------------------------|\n| DNN     | ResNet20    | 89.46 ± 0.08               | 0.209 ± 0.028                  |\n| Vanilla BNN | ResNet20    | 84.27 ± 0.42               | 0.244 ± 0.030                  |\n|         | ResNet20*   | 90.35 ± 0.45               | 0.261 ± 0.015                  |\n| DML     | ResNet20    | 86.74 ± 0.32               | 0.296 ± 0.005                  |\n|         | ResNet20*   | 91.06 ± 0.32               | 0.325 ± 0.010                  |\n| Ours    | ResNet20    | 86.89 ± 0.12               | 0.308 ± 0.019                  |\n|         | ResNet20*   | **91.36 ± 0.25**           | **0.358 ± 0.022**   \n\n### Questions\n\n1. Line 178-179: The authors said adding the D(...) term will rapidly increase this term and impact training. Wouldn't putting a smaller scaling factor for this term fix this issue? \n\n **Answer:** In our preliminary experiments, we tried with adding a scale factor to control $D(q(w; \\theta_1), q(w; \\theta_2))$, however, we found that it is not an effective solution. The reason is that D(.,.) does not have an upper bound, which implies that it is not easy to control the scaling factor. The value of the D(.,.) term could potentially increase indefinitely during the learning process if we directly maximize it. This could lead to gradient explosion and the model will not converge. The proposed loss $L_{diverse param}$ (eq. 12) has the advantage that the D(.,.) increases gradually until it reaches a saturation point. These properties make optimization more stable during training."}, {"review_id": "NtGNv8bjCr", "reviewer": "Reviewer_5v8y", "summary": "This paper proposed a mutual learning approach to learn a pair of Bayesian Neural Network(BNN). The posterior of BNN is approximated by Variational Inference using a Gaussian distribution with a diagonal covariance matrix. To make the BNN learn different perspective of the data, the author proposed to increase the diversity in parameter space and intermediate feature space by adding the an estimate of distance between parameter distribution and fused feature distribution of two BNN models into the objective function. Empirically, the proposed method outperform existing mutual learning method and vanilla BNN model in terms of accuracy, negative log likelihood loss and expected calibration error. An ablation study is also provided to investigate the usefulness of each component. ", "questions": "1. To my knowledge, it is not very clear if variational distribution(e.g. Gaussian with diagonal covariance matrix) can approximate the true posterior very well, can the author comment a bit on this, e.g. how would different choice of variational family affect the model?\n\n2. In line 264 and line 6 of algorithm 1, it is mentioned that one BNN model is initialized with a trained model and this lead to better results empirically. Can the author discuss more on why this happened? It is a bit wired for me as it seems in the implementation detail, the pre-trained model and the model from scratch are trained with same optimizer and learning rate schedule.\n\n3. Seems like $\\alpha$ $\\beta$ are set to 1,2 for CIFAR and 1,1 for Imagenet, these two parameters controls the strength of proposed penalty to the model, can the author comments a bit more on how sensitive are the model to those parameters, It can help to illustrate how diversity helps model performance.\n\n4. As mentioned in line 268, results are average of 3 trials, I think it would be better to include the standard deviation as well to boost the significance of the results.\n\n5. In figure A.3 in supplementary material, looks like a sharp increase of KL divergence between the fused feature distributions at around 30 epochs, but the penalty for feature is only added for last 100 epochs, can the author explain more on this? ", "limitations": "The authors addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and easy to follow. Increasing the diversity of parameter distribution and intermediate feature distribution of peer BNN models to boost performance is an interesting idea. Experiments and detailed ablation study demonstrate the effectiveness of proposed method.", "weaknesses": "1. It is mentioned in the abstract and introduction that the BNN model with variational inference may underperform deterministic model or BNN obtained by MCMC, the baseline only involves BNN model trained with(DML) or without(vanilla) mutual learning. Would the proposed method close the gap to some extent? Data augmentation, optimizer may all affect performance, so it is still helpful to include deterministic model results follow with same training setup. I would expect the BNN model to outperform deterministic model at least in NLL and ECE, and with the 50 ensemble, it can outperform the accuracy. \n\n2. Continue with last point, for MCMC method (e.g. in line 81 of the paper), I agree that traditional MCMC method(e.g. Metropolis Hasting) may not be feasible for large model, and memory storage can be an issue for MCMC method. But I don't think the stochastic gradient MCMC cited in line 81 would require prohibitive computational cost, it behaves like adding a noise to at each step of standard SGD training. \n\n3. The code is not provided so it may hurt the reproducibility of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n1. BNN model with variational inference may underperform deterministic model or BNN obtained by MCMC, the baseline only involves BNN model trained with (DML) or without(vanilla) mutual learning. Would the proposed method close the gap to some extent? I would expect the BNN model to outperform deterministic model at least in NLL and ECE, and with the 50 ensemble, it can outperform the accuracy.\n\n**Answer:**  Due to limited space we cannot add tables here. We compare our method with deterministic models and also with MCMC-based BNNs. For MCMC-based BNNs, we use the Stochastic Gradient Langevin Dynamics (SGLD). The results are presented in Table 1 in the attached pdf file which can be found at the end of our general response at the beginning.  The experimental results show that our proposed method outperforms the deterministic model and the SGLD [36]. \n\n2. I don't think the stochastic gradient MCMC cited in line 81 would require prohibitive computational cost, it behaves like adding a noise to at each step of standard SGD training. \n\n**Answer:**  We agree with Reviewer that SGLD [3] and Stochastic Gradient Hamilton Monte Carlo (SGHMC) would not require prohibitive computational cost. We will revise the references. We also would like to note that comparing to the stochastic gradient based MCMC such as SGLD and SGHMC, Variational Inference (VI)-based BNNs is more efficient in term of memory storage. In MCMC-based BNNs, we need to store model samples, while in VI-based BNNs, we can explicitly estimate the posterior. \n\n3. The code is not provided so it may hurt the reproducibility of the paper?  \n\n**Answer:**  We will release the code for the reproducibility of the paper.  \n\n### Questions\n\n1. To my knowledge, it is not very clear if variational distribution (e.g. Gaussian with diagonal covariance matrix) can approximate the true posterior very well, can the author comment a bit on this, e.g. how would different choice of variational family affect the model? \n\n**Answer:** The accuracy of the approximation made in VI mostly depends on how expressive the variational family is. The more expressive, the better the approximation. However, in practice, one needs to balance between the accuracy of the approximation and the computation and storage complexity associated. One of the simplest variational families includes the Gaussian distribution with diagonal covariance matrix [1] or Bernoulli distribution [A]. Despite their modest in terms of posterior estimation, they are widely used in the literature due to their efficiency (e.g., closed-form KL divergence) and scalability (i.e., the number of parameters scales linearly) which is suitable for large-scale models such as deep neural networks. Subsequently, one can select a family with higher complexity and expressiveness, such as Gaussian mixture models, to approximate the true posterior accurately. Although such a model could approximate the true posterior precisely, it significantly increases the number of learnable parameters, requiring more computation and storage, and hence, making it less applicable for deep neural networks.\n\n[A] Yarin Gal and Zoubin Ghahramani, Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference, ICLR Workshops, 2016.\n\n2. In line 264 and line 6 of algorithm 1, it is mentioned that one BNN model is initialized with a trained model and this lead to better results empirically. Can the author discuss more on why this happened? It is a bit wired for me as it seems in the implementation detail, the pre-trained model and the model from scratch are trained with same optimizer and learning rate schedule. \n\n**Answer:** One of the benefits of using the pretrained model as initialization for the mean of BNNs is that the pretrained model has been well trained. Using it for BNNs training will lead to better training convergence. In addition, as our aim is to make the model distributions and feature distributions of two peer networks diverse, having a BNN initialized with a pretrained model and a BNN initialized from scratch at the beginning of the training ensures the diversity in model parameter distributions and feature distributions. The above benefits lead to better results. \n\n3. Seems like $\\alpha$, $\\beta$ are set to 1,2 for CIFAR and 1,1 for Imagenet, these two parameters controls the strength of proposed penalty to the model, can the author comments a bit more on how sensitive are the model to those parameters, It can help to illustrate how diversity helps model performance. \n\n**Answer:** Please refer to the general response.\n\n4. As mentioned in line 268, results are average of 3 trials, I think it would be better to include the standard deviation as well to boost the significance of the results. \n\n**Answer:** We present in Table 2 in the attached pdf file the ACC, NLL, and ECE with the standard deviation of pairs of networks on CIFAR100 dataset.\n\n5. In figure A.3 in supplementary material, looks like a sharp increase of KL divergence between the fused feature distributions at around 30 epochs, but the penalty for feature is only added for last 100 epochs, can the author explain more on this? \n\n**Answer:** We would like to clarify that for figure A.3 in the supplementary material, we train BNNs with only $L_{\\text{logits}}$ and $L_{\\text{diverse feat}}$, i.e., we do not add the model diversity. This corresponds to the setting 'c' in Table 4 in the main paper. Our aim is to show that when diversity is encouraged in the feature space, the resulting distance between fused feature distributions is greater than the distance observed during when training with the traditional mutual learning.  We also would like to note that figure A.2 is with training BNNs with only $L_{\\text{logits}}$ and $L_{\\text{diverse param}}$ which corresponds to the setting 'b' in Table 4 in the main paper. "}]

Author rebuttal:
We sincerely thank the reviewers for the constructive feedback.

## General response to the choice of hyper parameters
Regarding hyper parameters $T$, $\alpha$, $\beta$:
For the temperature $T$, we follow the seminal work [13] and [4].  This parameter $T$ controls the smoothness of the prediction distribution. As the value of $T$ increases, the prediction distribution becomes smoother. The hyper parameters $\alpha$ and $\beta$ control the impacts of the diversity of network parameter distributions and network feature distributions on the learning of a pair of peer networks. We present here the ablation studies on the choice of $T$, $\alpha$, $\beta$ on CIFAR100. We denote "*" mean Bayesian neural networks are initialized with the mean value from the pre-trained deterministic model.

For ablation studies for parameter T, we vary the value of $T$ from 1 to 5, and fix the values of $\alpha=1$ and $\beta=2$. The results are shown in the following table. The results show that the best value of $T$ is 3. 
|  T   | 1     | 2     | 3     | 4     | 5     |
|---------|-------|-------|-------|-------|-------|
| ResNet20| 66.6  | 67.66 | 68.32 | 67.95 | 67.93 |
| ResNet20* | 69.62 | 70.14 | 70.45 | 70.12 | 69.97 |

For ablation studies for parameter $\alpha$, we vary $\alpha$ when promoting diversity in parameter space, and set the value of $T=3$ and $\beta=0$. The results are shown in the following table. The results show that by setting $\alpha = 1$, we achieve a better performance compared to other tested values of $\alpha$. 
| $\alpha$ | 0.1   | 1     | 2     | 5     | 10    |
|--------------|-------|-------|-------|-------|-------|
| ResNet20     | 67.25 | 67.78 | 67.18 | 67.45 | 67.29 |
| ResNet20*    | 69.79 | 70.22 | 69.92 | 69.61 | 69.48 |

For ablation studies for parameter $\beta$, we vary $\beta$ when promoting diversity in feature space, and set the value of $T=3$ and $\alpha=0$. The results are shown in the following table. The results show that by setting $\beta = 2$, we achieve a better performance compared to other tested values of $\beta$.
| $\beta$  | 0.1   | 1     | 2     | 5     | 10    |
|--------------|-------|-------|-------|-------|-------|
| ResNet20     | 66.95 | 67.17 | 67.57 | 67.14 | 67.09 |
| ResNet20*    | 69.84 | 69.91 | 70.04 | 69.85 | 69.90 |

In addition, from our experiments, we found that the accuracies are only  slightly different when $\alpha$ and $\beta$ take values in {1, 2}.

Meta-review: The reviewers praise the novelty and originality of the method, the strong empirical results, the comprehensive experiments and the writing of the paper. The main criticisms are the choice of hyperparameters, the connection between this mutual learning with diversification vs existing approaches to mutual learning encouraging alignment, lack of comparison to a deterministic model, limited scope of the experiments in terms of different models, lack of evaluation on OOD data, and lack of comparison to a MCMC baseline. However, most of these points have been addressed in the rebuttal.
Decision: Accept (poster)