Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Authors: Ngoc-Quan Pham
Tuan Truong
Quyen Tran
Tan Minh Nguyen
Dinh Phung
Trung Le
Venue: ICML 2025 poster (2025)
Keywords: Distributional Robustness
Bayesian Inference
Model Finetuning
Promoting Ensemble Diversity
Primary area: probabilistic_methods->variational_inference
TL;DR: We propose an interactive Bayesian inference method to improve ensemble quality.

Abstract: We introduce Interactive Bayesian Distributional Robustness (IBDR), a novel Bayesian inference framework that allows modeling the interactions between particles, thereby enhancing ensemble quality through increased particle diversity. IBDR is grounded in a generalized theoretical framework that connects the distributional population loss with the approximate posterior, motivating a practical dual optimization procedure that enforces distributional robustness while fostering particle diversity. We evaluate IBDR's performance against various baseline methods using the VTAB-1K benchmark and the common reasoning language task. The results consistently show that IBDR outperforms these baselines, underscoring its effectiveness in real-world applications.

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Ngoc-Quan Pham 1
Tuan Truong"
Quyen Tran
Tan Nguyen" 12 Dinh Phung 13 Trung Le
Abstract
maintaining multiple models, especially for large-scale ar- chitectures. To mitigate this issue, variational inference (VI) techniques have been developed to approximate the true pos- terior distribution with tractable approximations known as variational posteriors. These methods optimize a variational lower bound, significantly reducing computational costs while effectively capturing uncertainty. Key contributions in this domain include Kingma & Welling (2013); Kingma et al. (2015), and Blundell et al. (2015), who extended Gaussian variational posterior approximations to neural net- works. Additionally, Gupta & Nagar (2018) introduced greater flexibility in posterior approximations, enhancing the applicability of VI techniques.
We introduce Interactive Bayesian Distributional Robustness (IBDR), a novel Bayesian inference framework that allows modeling the interactions between particles, thereby enhancing ensemble quality through increased particle diversity. IBDR is grounded in a generalized theoretical frame- work that connects the distributional population loss with the approximate posterior, motivating practical dual optimization procedure that en- forces distributional robustness while fostering particle diversity. We evaluate IBDR's ' perfor- mance against various baseline methods using the VTAB-IK benchmark and the common reasoning language task. The results consistently show that IBDR outperforms these baselines, underscoring its effectiveness in real-world applications.
Traditional Bayesian inference faces limitation: the in- dependent sampling of model particles from the posterior often fails to capture interactions and can lead to particle collapse into single mode. To overcome this, we intro- duce a novel framework, Interactive Bayesian Distributional Robustness (IBDR), which establishes a joint distribution over independent posteriors and integrates divergence loss to model this interaction and encourage particle diversity. To further enhance robustness, we leverage Wasserstein- based distributional robustness optimization (Gao & Kley- wegt, 2023; Blanchet et al., 2019; Sinha et al., 2018), as formalized in Theorem 1. This extension generalizes dis- tributional robustness optimization (DRO) to accommodate more general risk functions and product joint distributions, ensuring balance between robustness and diversity. The resulting framework improves ensemble performance by preventing mode collapse while maintaining computational efficiency. To demonstrate the effectiveness and versatility of our framework, we conduct experiments on the image classification task with ViT (Dosovitskiy et al., 2020) and the commonsense reasoning task on LLaMA-2 (Touvron et al., 2023), demonstrating sigmificant improvements over baseline methods.
1. Introduction
Tackling uncertainty remains one of the most challenging problems in deep learning. This uncertainty arises from the real world's inherent randomness and noisy, complex data. To address this challenge, Bayesian inference offers powerful solution by providing a probabilistic framework that enables reasoning under uncertainty. A particularly practical approach within Bayesian inference involves parti- cle sampling techniques, which are well-suited for scenar- ios requiring multiple models. Notable methods include Stochastic Gradient Langevin Dynamics (SGLD) (Welling & Teh, 201 la), Hamiltonian Monte Carlo (HMC) (Neal, 1996), Stochastic Gradient HMC (SGHMC) (Chen al.,, 2014), and Stein Variational Gradient Descent (SVGD) (Liu & Wang, 2016). However, a key limitation of these methods is the computational and storage overhead associated with
Equual contribution 'Qualcomm AI Research, Qualcomm Viet- nam Company Limited "National University of Singapore, Singa- pore MMonahh University, Australia. Correspondence to: Trung Le <trunglm@ monash.edu>.
Our contributions are as follows: (i) We introduce a novel Bayesian framework that explicitly models interactions between model particles during training. Leveraging dis- tributional robustness, we provide theoretical analysis of this interactive framework, generalizing existing results to product distribution spaces. (ii) Building on this analysis.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
we propose practical framework that ensures both ensem- ble diversity and distributional robustness. We validate our approach by fine-tuning ViT (Dosovitskiy et al., 2020) on the VTAB- IK classification benchmark and LLaMA-2 (Touvron et al., 2023) on a commonsense reasoning task, demonstrating significant performance improvements.
more robust training-test set differences (Jiang al., 2020; Petzka et al., 2021; Nguyen et al., 2023a). The link be- tween generalization and minimum width has been explored both theoretically and empirically (Hochreiter & Schmid- huber, 1994; Neyshabur et al., 2017; Dinh et al., 2017; Fort & Ganguli, 2019). Various methods for finding flat minima have been proposed (Pereyra et al., 2017; Izmailov et al., 2018: Foret et al., 2021: Nguyen et al., 2023a). Studies by Keskar et al. (2017), Jastrzebski et al. (2017), and Wei et al. (2020) examine how batch size, learning rate, gradient covariance, and dropout affect flatness. Some methods also use regularization terms in the loss function to promote wide local minima (Pereyra et al., 2017: Zhang et al., 2018: 2019), such as softmax entropy penalties (Pereyra et al., 2017) and distillation losses (Zhang et al., 2018; 2019).
2. Related Works
2.1. Bayesian Neural Networks
Variational Inference. This approach estimates the poste- rior distribution by selecting specific approximation and re- fining variational lower bound. Graves (2011) I) introduced a Gaussian variational posterior for neural network weights, further developed in Kingma & Welling (2013); Kingma et al. (2015); Blundell et al. (2015) using the reparameteri- zation trick for deep latent variable models. Extensions to improve posterior flexibility include Caterini et al. (2021), who used normalizing flows, and Louizos & Welling (2017) and Gupta & Nagar (2018), who employed a matrix-variate Gaussian. Other studies have explored structured variational Gaussian posteriors, such as Kronecker-factorre approxima- tions (Rossi et al., 2020; Eschenhagen et al., 2023), Gaus- sian score matching (Modi et al., 2023), and non-centered or rank-1 parameterizations (Ghosh et al., 2018; Dusenberry et al., 2020). Hybrid approaches combining Variational Inference and Markov Chain Monte Carlo (MCMC) have gained popularity, such as Alexos et al. (2022), which used latent variable averaging to speed up mixing. Applications of Variational Inference to modern architectures, like Vi- sion Transformers (Zhang et al., 2021), highlight its role in uncertainty-aware deep learning.
Among the flat minimizers, Sharpness- Aware Minimiza- tion (SAM) (Foret et al., 2021) has gained attention for its effectiveness and scalability across tasks like domain generalization (Cha et al., 2021; Wang et al., 2023; Zhang et al., 2023), federated learning (Caldarola et al., 2022: Qu et al., 2022), Bayesian networks (Nguyen et al., 2023b; Möllenhoff & Khan, 2023), and meta-learning (Abbas et al., 2022). SAM has also improved generalization in both vi- sion models (Chen et al., 2022) and language models (Bahri et al., 2022). To leverage SAM's generalization ability, Nguyen et al. (2023b) explores the link between flat min- ima and Bayesian inference, proposing methods to use low- sharpness minima in the Bayesian posterior to enhance neu- ral network generalization. Recently, (Truong et al., 2025b) extends the concept of sharpness to the space of functions that govern the movement of the model particles, and incor- porates this theoretical framework to strengthen the general- ization ability of the ensemble in Bayesian inference.
Markov Chain Monte Carlo (MCMC). This approach allows sampling multiple models from the posterior distri- bution, commonly used for neural network inference via Hamiltonian Monte Carlo (HMC) (Neal, 1996). However, HMC requires full gradient estimation, which is computa- tionally expensive. Stochastic Gradient HMC (SGHMC) (Chen al., 2014) uses stochastic gradients for scalability and solution exploration. Alternatively, Stochastic Gradi- ent Langevin Dynamics (SGLD) (Welling & Teh, 201 lb) applies Langevin dynamics in the stochastic gradient set- ting. Stein Variational Gradient Descent (SVGD) (Liu & Wang, 2016) uses particles to approach the posterior distri- bution. MCMC methods, unlike variational methods, can be costly due to the need store multiple models. Nonetheless, SGHMC, SGLD, and SVGD asymptotically sample from the posterior with infinitely small step sizes.
2.3. Bayesian Approach for Model Fine-tuning
Fine-tuning large architectures like Transformers has in- creasingly shifted toward parameter-efficient fine-tuning (PEFT) methods such as prompting (Lester et al., 2021), LoRA (Hu et al., 2022), and adapters (Houlsby et al., 2019). These techniques enable efficient adaptation to new tasks without the computational burden of full model retraining. Prompting facilitates rapid task-switching with minimal up- dates (Lester et al., 2021; Liu et al., 2022), while LoRA (Hu et al., 2022) significantly reduces memory and compute costs by updating only a small subset of parameters. Re- cently, multiple variants of LoRA have been introduced. For example, Liu et al. (2024) proposes to decompose the weight into the direction and magnitude components and learn these components independently. On the other hand, Truong et al. (2025a) proposes to reparameterize the low-rank matrices with small non-linear networks, hence significantly improv- ing the estimation rate of the low-rank matrices. Besides LoRA, Adapter modules (Sung et al., 2022; Hu et al., 2023;
2.2. Flat Minimizers
Flat minimizers improve neural network generalization by helping models find broader local minima, making them
2

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Zhang et al., 2024) introduce specialized layers that refine model behavior without altering its core weights. Together, these methods make fine-tuning large models more accessi- ble and efficient.
metric c, defined as:
J cd7,
inf We (Q', := TEI(Q' .2)
(2)
However, despite their efficiency, PEFT methods can lead to overconfident predictions, especially when fine-tuned on small datasets. This has sparked interest in Bayesian ap- proaches for uncertainty-aware adaptation. Several recent advancements integrate Bayesian principles into PEFT to hance robustness. For example, Bayesian Low-Rank Learn- ing (Doan et al., 2025) applies low-rank perturbations to pre-trained weights, enabling scalable Bayesian neural net- works (BNNs), deep ensembles, and Stein Variational Gradi- ent Descent (SVGD) with minimal computational overhead. BayesTune (Kim & Hospedales, 2023) leverages Bayesian inference for more efficient and principled hyperparame- ter tuning. Laplace-LoRAA (Yang et al., 2023) enhances LoRA's calibration by introducing a Laplace approximation, improving uncertainty estimation in large language mod- els (LLMs). Gaussian SWAG + LoRA (Onal et al., 2024) enables lightweight Bayesian inference with negligible com- putational overhead. Recently, BLoB (Bayesian Low-Rank Adaptation by Backpropagation) (Wang et al., 2024) opti- mizes both the mean and covariance of LoRA parameters, outperforming post-hoc uncertainty estimation methods. By embedding Bayesian principles into PEFT, these methods not only enhance fine-tuning efficiency but also improve model calibration and uncertainty estimation, addressing a key limitation of prior PEFT approaches.
where r (Q', Q) is the set of couplings whose marginals are Q' and Q. Under the assumption that f E T,¹ (Q) upper semi-continuous and e a non-negative lower semi- continuous cost satisfying c(z, 2') 0 iff 2 z', Blanchet & Murthy (2019) shows that the dual form for Eq. (1) is given by:
min { De Eewgg (f (c (3) X
Sinha et al. (2018) further uses a Lagrangian for the Wasserstein-basee uncertainty sets to reach a relaxed version with X 2 0:
max (Egr (5)) (W.
Q'
= -Ezoolmax (s (3') dc (x),
(4)
3.2. Fine-tuning Transformer-based Models
Given a transformer-based foundation model o, the con- ventional approach to fine-tuning involves modifying the model's parameters as 0 = c + A, where A represents addi- tional modules. Popular parameter-efficienn fine-tuning tech- niques include prompt-tuning (Lester et al., 2021), LoRA (Hu et al., 2022), and Adapters (Houlsby al., 2019), which are commonly used to adapt models for downstream tasks such as classification on new datasets. To highlight the effectiveness of our Bayesian Inference framework, we fo- cus on the LoRA technique to fine-tune two instances of Transformer-basee models, including Vision Transformer (ViT) models (Dosovitskiy et al., 2020) for the image clas- sification task and the Large Language Model LLaMA2 (Touvron et al., 2023) for the commonsense reasoning task.
3. Backkround
3.1. Distributional Robustness
This section presents the background on the Wasserstein- based distributional robustness that serves our theory de- velopment in the sequel. Distributional robustness (DR) is an emerging framework for learning and decision-making under uncertainty, which seeks the worst-case expected loss among ball of distributions, containing all distributions that are close to the empirical distribution (Gao et al., 2017).
In brief, LoRA modifies the weight matrix W in the Multi- Head Self-Attention mechanism of the transformer (e.g.. Wº. W and W by applying the transformation W +- W + BA, where and B are low-rank matrices. This low-rank reparameterization ensures that the additional pa- rameters A and B remain lightweight, making the approach well-suited for Bayesian inference methods, benefiting from the efficient incorporation of such lightweight modules.
Consider a generic Polish space S endowed with a distri- bution Q. Let f = S - TR. be a real-valued (risk) function and c 5 x S + R4 be a cost function. Distributional robustness setting aims to find the distribution Q' in the vicinity of Q that maximizes the expected risk (Sinha al., 2018; Blanchet & Murthy, 2019):
3.3. Bayesian Inference with Variational Approach
max Q' ()
Eg' [f
(1)
Consider the model space O over which the parameters 0 follow a prior distribution P with density function p(0). Given a training set S = { (1), y1.,..., (# yx)} whose examples (I,, 34) ~ D, denote [(0; x, y) as the loss induced by using the model 0 to predict x with the ground-truth label
where é > and We denotes the optimal transport (OT) or a Wasserstein distance (Villani, 2008) with respect to the
3

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
y where is a loss function (e.g., the Cross Entropy (CE)
Our framework learns Q such that if 0 01:K QK or 01.K sid Q. the models reside in low-loss, low-sharpness regions while maintaining diversity. To achieve this, we introduce a novel loss function that encourages particle inter- action during training, guiding them toward regions of low sharpness while maintaining high diversity. Then, the i.i.d samples (i.e., the models) sampled from this posterior will also yield high diversity and low sharpness. Notably, parti- cles interact only during training to shape the final posterior Q, but they are sampled independently at inference.
loss). The true posterior is defined as
p(0|S) x II p (ue x2, (0),
where the likelihood p(ya x₁. 0) is defined as
p ((y Ti, 0) x exp 1 (0; 3i 1))
Therefore, the true posterior can be rewritten as
One potential drawback of promoting particle interactions is the risk of uncontrolled instabilities during training. To alleviate this issue and enhance robustness within this in- teractive framework, we employ WS-based Distributional Robustness Optimization (DRO) to develop Theorem 4.1. This theorem characterizes the population loss over the ap- proximate posterior QK providing insights into a practical method for promoting particle diversity and improving gen- cralization. Specifically, Theorem 4.1 can be interpreted as finding particle models that exhibit both diversity and low sharpness, enhancing their ability generalize, as discussed in prior works including Charpness-Awar Min- imization (SAM) (Foret al., 2021). We emphasize that this result is generalization upon prior findings on distri- butional robustness, as elaborated in Corollary 4.2.
p(0 () & exp
Σ 1 (0: p(().
Variational approaches (Graves, 2011; Kingma & Welling, 2013; Kingma et al., 2015; Blundell et al., 2015) can be used to obtain an approximate posterior distribution Q with a simpler density function q()) that approximates the true posterior p(0 |S) as:
Σ 2 (0; ui, ui) + DKL (a.0)
min q
Eg
Finally, we sample K particle models 01:k sid Q and en- semble their prediction outputs to produce the final result. Evidently, the variational approach for Bayesian inference lack mechanism to explicitly enforce interaction between the particle models Ki.k. such as encouraging these parti- ele models to diverge or complement each other, which is crucial for improving ensemble performance.
4.2. Theoretical Development
Let Q be the model space and Q and P be the approx- imate and prior distributions over O. Given a positive number of particle models K > 0. To be able to model the interactions between these K particles, we first define QK = - o Q and pk P PO. P as
Additionally, the Bayesian framework with Stochastic Gra- dient Langevin Dynamics (SGLD) (Welling & Teh, 201 and optimization methods (Nguyen et al., 2023b) encounter the same problem: the particle models 01:k ~ Q lack mechanism to encourage the particle models to diverge or complement each other.
K the joint distributions of K statistically independent parti- cle models sampled from Q and P, respectively. Denote o = f1:K ~ Q* as the concatenation of the K models, we propose the following loss function
K times
std
4. Interactive Bayesian Distributional Robustness
((0: 2, v)
((0,;,,)) + addio (01.k00) v),
K
4.1. Motivations
As discussed earlier, promoting diversity among particles crucial to prevent mode collapse. However, conventional Bayesian frameworks lack explicit mechanisms model interactions between particles 0. during training. To ad- dress this, we first define the approximate posterior distri- bution QK over the product space ØK where samples are concatenated particle models, 0 = 01.k. This formulation allows us to define a loss function over the joint partiele models, thereby facilitating the modeling of the interactions between the individual particles 0
where (I. y) ~ D is sampled from the data distribution (i.e., D is the general distribution data/label pairs), I is the loss function (e.g.. CE loss or Hinge loss), ldso is the divergence loss which encourages the partieles 01.K to be diverse and will be explicitly defined later, and a > 0 is trade-off hyperparameter capturing the extent to which we want to encourage the diversity between the particles.
Given a training set S = {(x,,,)) ~ DN, we define the following population loss functions over a single model
4

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
O and over the approximate posterior QK as:
To further illustrate the connection SAM, we present the following corollary, where we adopt specific form of the cost function establishing clear link SAM (Foret et al., 2021). The proof can be found in Appendix C.
(D) = B(o.a))>D (())z,)) w) CD (2x) ExNqx LD ())
Corollary 4.2. Given a metric d over the model space, consider the following cost function c
Similarly, we define the empirical losses over single model and over the approximate posterior QK as:
d(0,0') if dd 0' S
c(0, 0') = +00
otherwise.
(0) = (( e(0) v0) Ls (0x) - Eanqx Ls [
With the prob. at least 1 8 over the choice of S ~ DN,
LD (=x
S Eenqx
max O'EB,10)
Ls (8))
Note that the population loss CD (o() takes the form:
KDKL (Q. + +log 1 8
LD (Q*) Eangx [CD (c))
+ 1
2N
where the ball (0) 10' d(0, 0') S p).
Σ (0,) + aBD (0 []]
= EBNQR
If we further simplify the analysis by considering the l2 Euclidean distance for d, Corollary 4.2 reveals that the Sharpness-Aware Bayesian Neural Network (SA-BNN) framework from (Nguyen et al., 2023b) becomes spe- cial case of our broader approach. However, compared to SA-BNN, our method, as established in Theorem 4.1 and Corollary 4.2, offers a notable advancement: this framework operates on the joint distribution Q and incorporates a divergence loss ldin, enabling us to model interactions be- tween the particle models 0j.k. This inter-pariccl interac- tion beneficial to achieve strong ensemble accuracy, as we will empirically demonstrate in the subsequent experiments.
By minimizing the general loss LD(()), we simultane- ously encourage the particle models to achieve low general loss while diverging from each other to enhance ensemble performance. A key challenge is that directly minimizing the general loss (D())) is impractical since we do not have access to the data distribution D. To address this chal- lenge, we present the following theorem, whose proof can be found in Appendix C.
Theorem 4.1. With the probubility at least 1 - 8 over the choice of S ~ DN, we have
KDK1. (Q,P) log 8
4.3. Practical Algorithm
CD (x*) S
This section explores the theory above to derive a practical method Interactive Bayesian Distribution Robustness for Model Fine-tuning (IBDR). We first discuss how to model the divergence loss Laiv (01.3) x. v).) Given a pair (z, y) e S, denote f fa; 0,) as the prediction probabilities of the particle model 0; on x. Let f=y(2;; (.) (or f", for short, when the context is clear) be the non-maximal prediction probabilities by eliminating the prediction probability of the ground-truie label y. Inspired by Pang et al. (2019), we encourage the manmmai predictions fhy, (i 1,..., K, where K is the number of particle models) to diverge, while maximizing f (i = 1,... K). Motivated by the theory of Determinantal Point Processes (DPP) (Kulesza et al., 2012), the ensemble diversity can be defined as:
2N
+ min 1xp+ + EBnnu X20
max
-cs (0, 6'
where D is the general data/label distribution, L is the upper-bound of the loss Sunction ((0:x, v), and ck (0,0') Ek c(0,,0)) 0.) represents distance/diver- gence between two models.
((z;8))
Theorem 4.1 offers a practical alternative, it provides an upper bound involving empirical losses over the train- ing set S, which can be computed given the availabil- ity of the training set S. This upper bound is appealing due to its connection with Sharpness-Aware Minimiza- tion (SAM) (Foret et al., 2021). Specifically, in the in- ner maximization, the parameter controls the distance between the adversarial model O' and the center model 0. The outer minimization balances the terms Xp and Ea-qu maxe { ( - (0000 } where an increase in À raises the first term but reduces the second.
Laio (0L xz v) ddet) (
[E[K) [
where fe, = "
5"x and
[ iElK]
R(CC-I [K]
{ 1 {1.... K) and C is the number of classes.
5

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Moreover, according to the matrix theory (Bernstein, 2009),
Algorithm 1 Interactive Bayesian Distributional Robustness (IBDR)
T det (cc(F=)) setk]]
[-oo] sern)) Vol* ([= )
Input: Initial particle means ji.k ascend step size ar
learning rates ax,, az
Output: Optimal particle means P1:K
where Vol (
specifies the volume spanned the
while not converged do
iE[K]
Sample batch B = (1)1))).--
(rb, Wb)} Sample e; ~ N(0, 1) and 0g +- p4 + de;
vectors in f
[E[K] indicating that aim to maximize
the diversity the non-maximal predictions by maximally increasing their spanned volume.
Compute 8 < 0: + 01 Vo, E(0 0;
Compute À x cr VxL(M, Oy 0c;
Compute 14 \ - ag V L(,, 0,
We define the approximate posterior distribution as mix- ture of Gaussians: Q = k Ei N ((1 (i; P with the prior distribution given by P PPN(0. 01 Using this setup, we have the following corollary, which provides valuable insights for developing a practical method. The proof of this corollary can be found in Appendix C.
end while
return H1:K
descent. In practice, we fix 0.1 and do not learn To update P1:K. we first apply the reparameterization trick to obtain 0 and use one-step gradient ascend to obtain &i We next consider 8 = ((.)) = ( + oe,) and apply the chain rule to compute the gradient of the loss w.r.t. μü Similar SAM, we simplify the derivative of 0g w.r.t. 0, as the identity matrix, hence leading to the gradient of the loss w.r.t μ4 is that of the loss w.r.t. 0.. Finally, apply one-step projected gradient descent (e.g., to ensure X 2 0) to update X. The pseudo-code of our approach is summarized in Algorithm 1.
Corollary 4.3. With the probability at least å over the choice of S ~ DN,, we have
EE
1(0(;,,)) v) i==
CD (0*K)
$ min {Ap Es, Es A20
max Fy.K
K
+ aldiv aldio(0 (f.:xiz,)) x,)
Σ K
A
K
c(O,, 83
EK |1al Kd(a log σ) + log |
+ IN
5. Experiments
4N
We focus the fine-tuning problem, where pre-trained model, denoted as & is provided, and the goal is to identify the optimal parameters 0 = + + A, with A representing an additional component. Various parameter-efficient fine- tuning (PEFT) methods, such as LoRA (Hu et al., 2022) and Adapters (Houlsby et al., 2019), have been developed to achieve this objective and have demonstrated remarkable performance compared to the conventional full fne-tuning. Since A is typically much smaller component than the complete model in PEFT methods, Bayesian techniques offer promising applications in model fine-tuning. To as- (5) sess the versatility and effectiveness of our method, we experiment with two tasks on different domains: image classification and commonsense reasoning.
where DD the general data/label distribution, L is the upper-bound of the loss f, and d is the model size.
Inspired by this corollary, apply a minor relaxation to the right-hand side and propose solve the following mini- mization problem:
min min Ap Borx~ K
1
Σ
max ē(0', 0, (,
Pix.o X
6,
||mi| 2 + d(o - -logo)
where E(0%, 0;; x. y) 1(0 x. y) + aldio (f,, 0_ Ac(O, 8)). Here, 3 can be interpreted as a regularization hyper-parameter. We denote the loss function in Eq. (5) as C(), 0% 0;;x, x. v). We employ the reparameterization trick, expressing 0- μ4 ae with 6; ~ (O. Additionally, for each 0g we relax the divergence loss Lase (Ci.xia),)) 2r, to Lai (,, 00 I. where p_; [8)
5.1. Image Classification
We fine-tuned the ViT-B/16 architecture, pre-trained on the ImageNet-212 dataset (Deng al., 2009), using the LoRA framework to learn the parameters A. Within the Bayesian inference framework, our goal is to learn K LoRA particles A, each producing a unique model instance 0, The final output is then generated by averaging the predic- tions from these model instances.
Practical algorithm. To solve the optimization problem in Eq. (5), we alternatively update yı-k and À using gradient
To evaluate the performance of IBDR, we conducted experi-
6

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Table 1. Top-1 Accuracy on VTAB-IK. The accuracies are reported with ViT-B/16 pre-trained on ImageNet-21K
Natural
Specialized
Strachured
SNORD Azim
CIFARIOO
Catesh10)
Flowers
SVHN
Sun397
Camelyon
Resise4S
Recinopathy
Clevr-Coum
Clovr Dior-Dist
DMLab
aSpr-L.cc
JSpr-On
ANOR--Ele
DED
Pots
KIT
Method
AVG
FFT
68.9
87.7
64.3
97.2
86.9
87.4
38.8
79.7
95.7
84.2
73.9
56.3
58.6
41.7
65.5
57.5
46.7
25.7
29.1
62.3
LoRA
67.1
90.7
68.9
98.1
90.1
84.5
94.9
84.4
73.6
82.9
69.2
49.8
78.5
75.7
47.1
44.0
68.4
SAM
72.7
90.3
71.4
99.0
90.2
84.4
52.4
82.0
92.6
84.1
74.0
76.7
68.3
47.9
74.3
71.6
43.4
26.9
39.1
70.5
SA-BNN
91.5
71.0
98.9
89.4
86.2
94.5
86.4
75.2
61.4
63.2
40.0
31.2
68.2
SGLD
68.7
91.0
67.0
98. 6
89.3
83.0
51.6
81.2
93.7
83.2
76.4
80.0
70..1
48.2
76.2
71.1
39.3
31.2
38.4
68.4
Ene
68.6
88.9
67.7
98.9
90.7
85.1
54.5
82.6
94.8
82.7
75.3
46.6
47.1
47.4
68.2
71.1
36.6
30.1
35.6
67.0
TayesTune
68.2
91.7
69.5
99.0
90.7
86.4
51.2
84.9
95.3
84.1
75.1
82.8
68.9
49.7
79.3
74.3
46.6
30.3
42.8
68.5
SVGD
71.3
90.2
71.0
98.7
90.2
84.3
52.7
83.4
93.2
86.7
75.1
75.8
70.7
49.6
79.9
69.1
41.2
30.6
33.1
70,9
IBDR
73.0
92.1
71.7
99.3
91.4
91.3
56.7
85.1
95.0
87.3
76.5
78.1
75.1
53.6
80.4
77.1
49.3
28.9
40.1
73.6
(11)
(31)
(.12)
(0.15)
(0.16)
(36)
(118
(244
(.44
(114)
(11)
(11)
(.24)
(44)
(226)
(229)
(119)
(.13)
(.37)
Table Expected Calibration Errors (ECE) on VTAB- The results are reported with ViT-B/16 pre-trained on ImageNet-22
Natural
Specialized
Structured
KNORB Azim
CIFARI00
Caltesh01
Flowers102
SVHN
San 397
Camelyon
EunSAr
Resiso45
Rotinopunty
Cleve-Count
Cleve-Dist
DMLab
KITH
dSp-LLoc
dSpx-Ori
&NORB-El
DTD
Pots
Method
AVG
FFT
0.29
0.23
0.20
0.13
0.27
0.19
0.45
0.21
0.13
0.18
0.17
0.41
0.44
0.42
0.22
0.14
0.23
0.24
0.40
0.26
LoRA
0.38
0.19
0.18
0.05
0.09
0.10
0.14
0.11
0.09
0.12
0.11
0.12
0.19
0.34
0.18
0.14
0.21
0.18
0.31
0.17
SAM
0.21
0.25
0.20
0.11
0.12
0,15
0.14
0.17
0.16
0.14
0.09
0.12
0.17
0.24
0.16
0.21
0.19
0.13
0.16
0.16
SA-BNN
0.22
0.08
0.19
0.15
0.12
0.12
0.24
0.13
0.06
0,12
0.18
0.14
0.21
0.22
0.24
0.25
0.41
0.46
0.34
0.20
SGLD
0,26
0.20
0.17
0.05
0.18
0.14
0.23
0.18
0.09
0.12
0.32
0.26
0.29
0.21
0,26
0.42
0.39
0.11
0.24
0.22
DeepEns
0.24
0.12
0.22
0.04
0.10
0.13
0.23
0.16
0.07
0.15
0.21
0.31
0.32
0,36
0.13
0.32
0.31
0.16
0.29
0.20
Taye
0,32
0.93
0.20
0.03
0.85
0,12
0.22
0.13
0.07
0.15
0.22
0,12
0.23
0.30
0,24
0.28
0.28
0.31
0.26
0.23
SVGD
0.20
0.13
0.19
0.04
0.16
0.09
0.20
0.15
0.11
0.13
0.12
0.17
0.21
0.30
0.18
0.21
0.25
0.14
0.26
0.18
IRDR
0.16
0.08
0.19
0.02
0.07
0.07
0.13
0.12
0.06
0.11
0.11
0.13
0.24
0.30
0.12
0.11
0.30
0.30
0.16
0.14
(.03)
(.02)
(.02)
(.01)
(.01)
(.0)
(.02)
(.03)
(.02)
(.02)
(.01)
(.01)
(.02)
(.03)
(.01)
(.01
(.05)
(.04)
(.02)
Table Accuracy/ECE on six common-sense reasoning datasets
AdamW, and SAM, as well as five Bayesian inference meth- ods: Sharpness- Aware Bayesian Neural Network (SA-BNN) (Nguyen et al., 2023b), Stochastic Gradient Langevin Dy- namics (SGLD) (Welling & Tch, 201 Bayesian Deep
Metric
Datasets
Type
Method
WG-S
ARC-C
ARC-E
WG-M
OBQA
BoalQ
AVG
MAP
68.62
67.59
86.55
75.61
81.38
86.
77.71
ACC (?)
MCD
ENS
69.57
69.26
66.20
68.43
$4.40
96.077
75.32
76.18
81.38
81.49
$7.09
87.15
77.33
TX.10
Ensembles (DeepEns) (Lakshminaray
et al., 2017),
67.54
68.11
63
73.41
81.72
87.19
77.27
Bayes (Kim & Hospedales, 2023) and Stein Variational
66.78
$0.05
86.95
BLaB
70.89
70.83
86.68
74.55
82.73
86.80
78.75
Gradient Descent (SVGD) (Liu & Wang, 2016).
IBDR
72.51
70.56
$6.95
76.46
84.60
$6,89
79.66
MLE
29.83
29.00
13.12
20.62
12.55
3.18
18.05
In our experiment, we set datasets. We tune using the provided validation set, with the candidate set being p E (0.01, 0.05, 0.13. All mod- cls, except for the deterministic methods (including SAM, LoRA with the AdamW optimizer, and full fine-tuning), were trained using four particles on the same set of LoRA parameters specified by Hu et al. (2022). We conducted three independent runs for each experiment and reported the mean scores and standard deviation. For additional in- formation regarding the experimental setup, please refer to Appendix A.
0.02, 10-4 for all
29.76
29.42
12.07
13.26
3,16
MCD
28.06
27.75
12.31
18.27
15.12
3.40
17.50
BCE (
ENS
28.52
29.16
12:57
20.86
15.34
9.61
19.34
BBB
21.93
25.84
12.42
15.89
11.23
3.76
15.18
8.70
1,30
BLaB
20.62
20.61
9,43
11.23
8.36
2,46
12.12
IBDR
24.17
21.20
9.71
11.19
5.82
1.54
12:27
ments using the 1T benchmark (Zhai et al., 2020),
a standardized framework designed to assess the transfer learning capabilities of models across a diverse range of visual tasks. This benchmark includes 19 distinct datasets encompassing three domains: Natural images. Specialized, and Structured. Each task in VTAB-1K is constrained to 1,000 labeled examples for fine-tuning, presenting chal- lenging scenario for evaluating the model's ability to gener- alize across domains with limited data.
According to Table 1, IBDR outperforms all baselines by large margins. We also note that IBDR surpasses SA-BNN by more than 5%, underscoring the effectiveness of incor- porating inter-particle interaction for ensemble diversity. While the inter-partien iiee interactions can enhance ensemble
We benchmarked IBDR against 8 baselines, utilizing three deterministic fine-tuning approaches: full fine-tuning (FFT),
7

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
126
Resel
115
106
Figure 1. Percentage of OOD samples at different thresholds. Left: model trained on CIFAR- 100 and tested on SVHN. Right: trained and tested on SVHN.
3.00
0.3
3.00
3.0
3.05
Figure 2. Average prediction probabilities of four particles on the SVHN testing set. The blue bar indicates the ground truth label.
diversity, these repulsive forces may compromise model robustness. However, our approach mitigates this issue by incorporating distributional robustness into the interac- tive framework. To evaluate the robustness of IBDR, we report the Expected Calibration Error (ECH) in Table 2. Although there is often a trade-off between accuracy and ECE, IBDR maintains a good balance between these metrics and achieves the best ECE among the baselines. This re- sult highlights the effectiveness of our method balancing distributional robustness and particle diversity.
6. Ablation Studies
6.1. Out-of--istributioo Performance
We conducted additional experiments to evaluate our model's out-of-distribution (OOD) detection capability. As shown in Figure 1, we plotted two line graphs illustrating the percentages of OOD samples at corresponding thresholds. The left graph represents the results for a model trained on the CIFAR-100 100 dataset and tested on the SVHN dataset, while the right graph illustrates the results for model both trained and tested on the SVHN dataset. As expected, when evaluated on a dataset different from the training data, our model exhibits low confidence across all classes and suc- cessfully identifies these samples as OOD. In contrast, the results in the right graph demonstrate that the model trained and tested on the SVHN dataset achieves high confidence in its predictions, further supporting our findings.
5.2. Commonsense Reasoning
Having shown that IBDR excels on the vision task, we ex- tend our experiment the commonsense reasoning task. We fine-tuned the LLaMA2-7B model on six widely-used common-sense reasoning tasks: ARC-Challenge (A (ARC-C) and ARC-Easy (ARC-B) (Clark et al., 2018), Winogrande- Small (WG-S) and Mooogrande-Mediune (WG-M) (Sak- aguchi et al., 2021), OpenBookQA (OBQA) (Mihaylov al., 2018), and BoolQ (Clark et al., 2019). Following the experimental setup outlined in Wang et al. (2024), we benchmarked IBDR against seven bascline methods, includ- ing Maximum Likelihood Estimation (MLE), Maximum A Posteriori (MAP), Monte Carlo Dropout (MCD) (Gal & Ghahramani, 2016), Bayes By Backprop (BBB) (Blundell et al., 2015), Deep Ensembles (ENS) (Lakshminarayanan et al., 2017), the recently proposed LaplaceLoRA (LAP) (Yang et al., 2023) and Bayesian Low-Rank Adaptation by Backpropagation (BLoB) (Wang al., 2024). For all base- line methods, we used the same frozen pre-trained LLM backbone. To maintain consistency, we kept hyperparam- eters identical across all datasets. Except for few IBDR- specific hyperparameters, we strictly adhered to the default settings from Hu et (2022).
6.2. Particle Interactions and Ensemble Diversity
This section delves into the effectiveness of our framework in enhancing ensemble diversity. As shown in Figure 2, which visualizes the average prediction probabilities of four particles, two out the four particles assign the highest probability the ground truth class, while the other two predict the ground truth As discussed in Section 4.3, algorithm is designed to promote diversity in nonmaximal probability predictions. This is evident in Figure 2. Indeed, the third particle pre- dicts class O with high probability. Through interaction among the particles, the other particles tend to assign lower probabilities to class 0. and this effect extends all classes that are not the ground truth. Then, when taking the average of the predictions across all particles, the overall probability for non-ground-truth classes is reduced, leading to the dom- inant probability of 30% for the ground truth compared to the probability of less than 15% for any other class in this case. By fostering particle diversity and maximizing the spanned volume of nonmaximal predictions, we decreased
of the top two probabilities.
The accuracy and Expected Calibration Error (ECE) metrics presented in Table 3, demonstrating that IBDR outper- forms the baseline methods across most tasks, achieving notable performance improvement while maintaining com- petitive calibration. Also, IBDR's ability to balance pre- dictive accuracy with uncertainty calibration highlights its effectiveness in modeling inter-particle interactions.
8

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
the chance of multiple particles making the same mispredic- tion and enhanced the ensemble quality.
profile inference and applications to machine learning.
Journal of Applied Probability, ( 2019
Blundell, C., Cornebise, J., Kavukcuoglu. K., and Wierstra, D. Weight uncertainty in neural network, In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
7. Conclusion
We present Interactive Bayesian Distributional Robustness (IBDR), a novel Bayesian inference framework that models interactions between particles to simultaneously enhance ensemble diversity and distributional robustness. IBDR has been rigorously tested different tasks from various do- mains, where it notably outperformed the baselines, demon- strating its effectiveness across diverse tasks.
Caldarola, D., Caputo, B., and Ciccone, M. Improving generalization in federated learning by seeking flat min- ima. In European Conference on Computer Vision, pp. 654-672. Springer, 2022.
Caterini, A., Cornish, R., Sejdinovie, D., and Doucet, A. Variational inference with contiuously-indexed normal- izing flows. In Uncertainty Artificial Intelligence, pp. 44-53. PMLR, 2021.
Acknowledgments
Trung Le was partly supported by ARC DP23 grant DP230101170 176 and by the Air Force Office of Scientific Research under award number FA9550-23-S-0001.
Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S., Lee, Y., and Park, S. Swad: Domain generalization by seeking flat minima, Advances in Neural Information Processing Systems, 34:22405-22418,, 2021.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are potential societal conse- quences of our work, none of which feel must be specifi- cally highlighted here.
Chen, T., Fox, E.,, and Guestrin, C. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pp. 1683-1691. PMLR, 2014.
Chen, X., Hsieh, C.-J., and Gong, B. When vision transform- ers outperform resnets without pre-training or strong data augmentations. In International Conference on Learning Representations, 2022.
References
Abbas, M. Xiao, Q. Chen, L., Chen, P.-Y., and Chen, T. Sharp-maml: Sharpness-aware model-agnostic meta learning. In International conference on machine learn- ing, pp. 10-32. PMLR, 2022.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pp. 2924 2936, 2019.
Alexos, A., Boyd, A. J., and Mandt, S. Structured stochastic gradient meme. In International Conference on Machine Learning, pp. 414 434. PMLR, 2022.
Alquier. P., Ridgway. J.. and Chopin, N. On the properties of variational approximations of gibbs posteriors. The Journal of Machine Learning Research, 84144, 2016.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint /803v
2018.
Bahri, D., Mobahi, H., and Tay, Y. Sharpness-aware min- imization improves language model generalization. In Proceedings of the 60th Annual Meeting of the Associ- ation for Computational Linguistics (Volume I: Long Papers), pp. 7360-7371, 2022.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 245. Ieee, 2009.
Bernstein, D. S. Matrix mathematics: theory, facts, and formulas. Princeton university press, 2009.
Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. Sharp min- ima can generalize for deep nets. In International Con- ference on Machine Learning, pp. 1019-1028. PMLR, 2017.
Blanchet, J. and Murthy, K. Quantifying distributional model risk via optimal transport. Mathematics of Opera- tions Research, 44(2):565 600, 2019.
Doan, B. G., Shamsi, A., Guo, X.-Y., Mohammadi, A., Hinejad-Rokny. H., Sejdinovic, D.,, Teney, D., Ranas- inghe, D. C., and Abbasnejad, E. Bayesian low-rank
Blanchet, J., Kang, Y., and Murthy, Robust wasserstein

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
learning (bella): A practical approach to bayesian neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 16298-16307, 2025.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-cfficient transfer learning for nlp. In International conference on machine learning, pp. 2790- 2799. PMLR, 2019.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhuu Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Hu, Z.,, Wang, I.., Lan, Y., Xu, W., Lim, E.-P., Bing, L.,, Xu, X., Poria, S., and Lee, R. K.-W. LIm-adapters: An adapter family for guage models. arXiv preprint axXiv2204.0193:3 2023.
Dusenberry, M., Jerfel, G., Wen, Y., Ma, Y., Snock, J., Heller, K., Lakshninarayanan, B., and Tran, D. Efficient and scalable bayesian neural nets with rank-1 factors. In International conference on machine learning, pp. 2782- 2792. PMLR, 2020.
fine-tuning of large lan-
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D. P., and
Wilson, A. G. Averaging weights leads to wider optima and better generalization. In UAL, pp. 876-88 885. AUAI Press, 2018
Eschenhagen, R., Immer, A., Turner, R., Schneider, F,, and Hennig, Kronecker- factored modern neural network architectures. Advances in Neural Information Processing Systems, 36:33624-33653 2023.
Kronecker-fattored approximate curvature for
Jastrzebski, S., Kenton, Z.., Arpit, D., Ballas, N., Fischer, A., Bengio, Y., and Storkey, A. J. Three factors influencing minima in sgd. ArXiv, abs/ 1.0612042,3..0 2017.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. In ICLR. OpenReview.net, 2020.
Fort, S. and Ganguli, S. Emergent properties of the lo- cal geometry of neural loss landscapes. arXiv preprint arXiv:1910.05929, 2019.
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, T. P. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR. OpenReview.net, 2017.
Gal, Y. and Ghahramani, Z. Dropout as a bayesian approx- imation: Representing model uncertainty in deep learn- ing. In international conference on machine learning, pp. 1050-1059. PMIR, 2016.
Kim, M. and Hospedales, T. Bayestune: Bayesian sparse deep model fine-tuning. Advances in Neural Information Processing Systems, 36:65317-65365, 2023.
Gao, R. and Kleywegt. A. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research, (8(2))603-655) 2023.
Kingma, P. and Welling, M. Auto-encoding variational bayes. arXiv preprini arXiv: 1312.6111,..1 2013.
Gao, R., Chen, X., and Kleywegt, A. J. Wasserstein distribu- tional robustness and regularization in statistical learning. arXiv e-prints, pp. arXiv- 1712, 2017.
Kingma, D. P., Salimans, T., and Welling, M. Variational dropout and the local reparameterization trick. Advances in neural information processing systems, 28, 2015.
Ghosh, S., Yao, J., and Doshi Velez, F. Structured varia- tional learning of bayesian neural networks with horse- shoe priors. In International Conference on Machine Learning, pp. 1744-1757 PMLR, 2018.
Kulesza, A., Taskar, B., et al. Determinantal point pro- cesses for machine learning. Foundations and Trends@ Machine Learning, 5(2-3):123-288 2012.
Graves, A. Practical variational inference for neural net- works. Advances in neural information processing sys- tems, 24, 2011.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Gupta, A. K. and Nagar, D. K. Matrix variate distributions. Chapman and Hall/CRC, 2018.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059 2021.
Hochreiter, S. and Schmidhuber, J. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994.
10

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Liu, Q. and Wang. D. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29, 2016.
In International Conference on Machine Learning, pp. 4970-4979, 2019.
Pereyra, G., Tucker, G., Chorowski, J., Kaiser, L., and Hinton, G. Regularizing neural networks by penal- izing confident output distributions. arXiv preprint arXiv: 0rX1i:/701..00 06548,,548, 2017.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight- decomposed low-rank adaptation, 2024, URL https: //arxiv. org/ abs 2abs/2400 2402. 09353.
Petzka, H., Kamp, M.. Adilova, L., Sminchisescu, C., and Boley, M. Relative flatness and generalization. In NeurIPS, pp.
Liu, X., Ji, K. Fu, Y., Tam, W., Du, Z., Yang, Z., and Tang, P-tuning: Prompt tuning can be comparable to fine- tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61 (8, 2022.
2021.
Qu, Z., Li, X., Duan, R., Liu, Y., Tang, B., and Lu, Z. Gen- eralized federated learning via sharpness aware minimiza- tion. In International conference on machine learning, pp. 18250-18288 PMLR, 2022.
Louizos, C. and Welling, M. Multiplicative normalizing flows for variational bayesian neural networks. In Interna- tional Conference on Machine Learning, pp. 2218-2227. PMLR, 2017.
Rossi, S., Marmin, S., and Filippone, M. Walsh-hadamane variational inference for bayesian deep learning. Ad- vances in Neural Information Processing Systems, 33: 9674-9686, 2020.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: 0209,02789, 2018.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Modi, C., Gower, R., Margossian, C., Yao, Y., Blei, D., and Saul, L. Variational inference with gaussian score matching. Advances in Neural Information Processing Systems, 36:29935 -99550, 2023.
Sinha, A., Namkoong, H., and Duchi, J. Certifying some dis- tributional robustness with principled adversarial training, In International Conference on Learning Representations, 2018.
Möllenhoff, T. and Khan, M. E. SAM as optimal re- laxation of bayes. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: fevFqSQcX.
Sung, Y.L.., Cho, J., and Bansal, M. Vl-adapter: Parameter- efficient transfer learning for vision- lsiion-nd-langguae tasks.
//openreview.rr httpr/iwwwwe//oorummmmmm forum? d=
In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 5227-5237, 2022.
Neal, R. M. Bayesian Learning for Neural Networks. Verlag, Berlin, Heidelberg, 1996. ISBN 0387947248.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A.,
Couairon, R., and Authors, M. Llama 2: Open foundation and fine-tuned chat models, 2023.
Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017.
Truong, T., Nguyen, C., Nguyen, H., Le, M., Le, T., and Ho, N. Replora: Reparameterizing low-rank adaptation via the perspective of mixture of experts. In Proceedings of the 42st International Conference on Machine Learn- ing (ICML), Proceedings of Machine Learning Research, Vancouver, Canada, 2025a. PMLR.
Nguyen, V.-A,, Le, T., Bui, A., Do, T.-T.. and Phung, D. Optimal transport model distributional robustness. In Ad- vances in Neural Information Processing Systems, 2023a.
Nguyen, V.A.,, Vuong, T.-L., Phan, H., Do, T.-T., Phung, D., and Le, T. Flat seeking bayesian neural networks. Ad- vances in Neural Information Processing Systems, 2023b.
Truong, T., Tran, Q., Pham-Ngoc, Q., Ho, N., Phung, D., and Le, T. Improving generalization with flat hilbert bayesian inference. In Proceedings of the 42st Interna- tional Conference on Machine Learning (arninggICMLL)) (CML)), Proceed- ings of Machine Learning Research, Vancouver, Canada, 2025b. PMLR.
Onal, E., Flöge, K., Caldwell, E., Sheverdin, A., and Fortuin, V. Gaussian stochastic weight averaging for bayesian low- rank adaptation of large language models. arXiv preprint arXiv:2405.03442 03425, 2024.
Villani, C. Optimal Transport: Old and New. Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2008. ISBN 9783540710493.
Pang, T., Xu, K., Du, C., Chen, N., and Zhu. J. Improving adversarial robustness via promoting ensemble diversity.
URL

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
https://books. google. com. com. www..o//ooooooiideeee.e...... NXXi MAXINAAACA.J.
Zhang, X., Xu, R., Yu, H., Dong, Y., Tian, P., and Cui, P. Flatness-aware minimization for domain generalization. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 1189-5202, 2023.
Wang, Zhang, Z., Lei, Z., and Zhang, Sharpness- aware gradient matching for domain generalization. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3769-3778, 2023.
Zhang, Y., Xiang, T., Hospedales, T. M., and Lu, H. Deep mutual learning. 2018 IEEF/CVE Conference Com- puter Vision and Pattern Recognition, pp. 4320-4328, 2018.
Wang, Y. Shi, H., Han, L., Metaxas, D., and Wang. H. Blob: Bayesian low-rank adaptation by backpropagation for large language models. arXiv preprint arXiv:2406. 1675, 2024.
Wei, C., Kakade, S., and Ma, T. The implicit and explicit regularization effects of dropout. In International con- ference machine learning, pp. 10181-10192. PMLR, 2020.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference International Conference Machine Learning, ICML'11, I, pp. 681-688, Madison, WI, USA, 201 la. Omnipress. ISBN 97819778880 4553306195
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688, 10.
Yang. A. X., Robeyns, M., Wang, X., and Aitchison, L. Bayesian low-rank adaptation for large language models. arXiv preprint 23Xiv:2308.131 2023.
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., Beyer, L., Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., and Houlsby, N. A large-scale study of representation learning with the visual task adaptation benchmark, 2020. URL https: /aaxiviiii..///.i///aaa///aaaavv org/ abs 1910. 04867.
Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., and Ma, K. your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3713-3722. 2019.
Zhang, R., Han, J., Liu, C., Zhou, A., Lu, P., Qiao, Y., Li, H., and Gao, P. Llama-adapter: Efficient fine-tuning of large language models with zero- iro-iintialized attention. In The Twelfth International Conference on Learning Representations, 2024.
Zhang, S., Fan, X., Chen, B., and Zhou, M. Bayesian attention belief networks. In International Conference on Machine Learning, pp.
PMLR, 2021.
12

---

Supplement for "Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models"
The Supplementary Material structured as follows: Appendix A presents the additional experiments to demonstrate the effectiveness and robustness method. Appendix B discusses the experimental information, including hyperparameters choice and the details about datasets. Appendix C presents the proofs of the theoretical results.
A. Additional Experiments
To further evaluate the performance and behavior of IBDR, we conduct additional experiments in this section. First, we examine impact the number of particles, verifying the benefits using multiple diverse particles and supporting our choice for the optimal number. Additionally, we assess IBDR's performance across various hyperparameter values, demonstrating its robustness with respect different settings p and a.
A.1. Effect of Number of particles
conducted experiments on the four Specialized datasets, including the Patch Camelyon, EuroSAT, Resics45, and Diabetic Retinopathy datasets, to assess how the number of particles affects final performance. As shown Table increasing the number of particles enhances the ensemble's quality, improving performance. However, Table shows that increasing the number of particles also results in linear increase in runtime. Considering this tradeoff between time and memory, we determined that using #PARTICLES = 4 offers an optimal balance between accuracy and training cost.
Table Classification accuracy with different #PARTICLES
#PARTICLES
Camelyon
EuroSAT
Resics45
Retinopathy
1p
82.4
93.1
84.2
73.8
2p
84.8
93.9
86.6
74.3
4p
85.1
95.0
87.3
76.5
8p
85.8
95.5
87.4
77.0
Table Runtime per epoch with different #PARTICLES
#PARTICLES
Camelyon
EuroSAT
Resics45
Retinopathy
1p
51
50+1.5
4841.7
51
2p
80
83
9312.1
85
4p
15844.3
161
15644.1
151+2.1
8p
220+6.2
230
21847.3
246+6.8
A.2. Effect of particle interaction via laiv
To further assess the impact of particle interactions, we examine the effect of varying values of a, which indicates the extent to which we enforce diversity among particles through the divergence loss Laiv. There is a significant performance gap between a = and a = 0.02 on all datasets, indicating the importance the divergence loss and particle diversity. As a increases, we anticipate that the behavior of the particles may become increasingly unstable, as the divergence loss begins to
13

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Table 6. Accuracy with different values of DTD, DMLab, and SVHN datasets
DTD
DMLAB
SVHN
0
67.24
51.92
87.03
0.02
71.70
53.61
91.32
0.08
71.73
52.73
91.26
0.5
70.86
53.16
90.63
1.5
70.93
51.27
90.34
68.32
48.64
86.31
Table 7. Classification accuracy with different values of a and
DATASETS
a
0.01
0.03
0.05
2
67.98
67.24
69.24
67.11
65.
0.02
70.11
71.70
70.34
69.26
68.41
0.08
71.02
71.73
70.88
70.01
69.20
DTD
0.5
70.43
70.86
70.61
69.23
69.01
1.5
69.67
70.93
70.93
68.61
67.14
3
68.01
68.32
70.24
67.31
66.09
0
$1.86
51.04
51.92
51.22
49.67
0.02
$2.33
53.24
53.61
51.09
49.30
0.08
53.01
53.26
52.73
50.98
48.12
DMLAB
0.5
52.12
52.87
53.16
48.11
46.87
1.5
50.02
49.64
51.27
47.19
44.62
3
48.96
48.62
48.64
46.23
44.11
dominate the classification loss. However, as shown in Table 6, IBDR maintains robust performance across a reasonable range of
A.3. Hyperparameter Sensitivity
IBDR depends on two key hyperparameters: a and p. In this section, we evaluate IBDR's performance on the DTD and SVHN datasets across various values of a and p to assess the algorithm's robustness with respect these parameters. As discussed in Section A.2, controls the degree to which promote diversity among particles. However, larger values of may cause instability, and we found that 0.02 works well as default across all experiments. Conversely, p serves as the step size for the ascent step, and setting it too high can also destabilize the model. Nonetheless, as shown in Table 7, our method remains robust over a reasonable range of both hyperparameters, demonstrating a desirable level of stability.
B. Experimental Details
The experiments conducted using PyTorch Tesla V100 GPU with 40GB of RAM. We the hyperparameters follows: 0.02, 3 10-4, and {0.01, 0.03, 0.05), with o tuned using the standard validation set. For optimization, we used stochastic gradient descent (SGD) as the base optimizer, combined with cosine annealing learning rate scheduler, 500 warm-up steps, momentum of 0.9, and no weight decay.
B.1. Datasets
B.1.1. IMAGE CLASSIFICATION
The VTAB-1K (Visual Task Adaptation Benchmark) is diverse and challenging image classifcation/prediiiio suite consisting of 19 datasets from various domains. VTAB-1K 1K covers various tasks across different semantics and object categories and designed to evaluate how well pre-teained models can adapt to various visual tasks by fine-tuning on small datasets. Specifically, in the VTAB-1K -Kk setting, only 1,000 training examples are provided for each task, making challenging fine-tune models with limited data.
14

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
Some key features of VTAB-1K including of:
Diverse tasks: VTAB-1K K consists of variety of visual tasks that fall into three broad categories:
Natural: Tasks derived from natural images, such as classification of real-world objects. This category consists of CIFAR100) 00, Ca Caltech101. 01, DTD, Oxford Flowers, Pets, SVHN, and Sun 997 datasets.
-Specialized: Tasks from specific domains requiring more fine-tuned understanding (e.g., satellite imagery, medical images). This category consists of Patch Camelyon, EuroSAT, Resics45, and Diabetic Ret i hopathy datasets.
Structured: Tasks involving synthetic or abstract images that require understanding of structured patterns (e.g.. depth estimation, object counting). This category consists of Smal LNORB, DMLab, aSprites, and KITTI datasets.
Evaluation Process:
Models are pre-trained on large datasets (e.g.. ImageNet- 21k) and then fine-tuned using only 1,000 examples from each task in VTAB- 1K, with an official 80/20 train-validation split. The performance evaluated based on accuracy or other task-specific metrics, testing the model's adaptability and generalization.
B.1.2. COMMONSENSE REASONING
For more details the size of the training and the number of labels for each commonsense reasoning dataset, refer to Appendix B.3 of (Wang et al., 2024).
B.2. Data Augmentations
B.2.1. IMAGE CLASSIFICATION
Our implementation is based on the repository V-PETL. For each dataset, the following data augmentation:
For Smal 1Norb-AAi, SmallNorb-Eil
101tech101, CIFAR100, Clevr-Dist, Dsprites-loc, Dsprites-Ori,
sel transformotoann transforms ([
transforms. (2244, (224, 224)),
transformms.ooensooo (),
Nornsfformss.ormait... (mean- [0. 485, 0.456, 0.406],
st.= 229, 0.224, 0.225]) transfrransor transforms Compose transforms. Resizes (224, 224)),
ToTensor (1 (),
transforms. Norma (mean= 48.485, 0.456, 0.406), [id=10.229) 0.224, 0.2251)
For Clevr-Count, Diabetic Retinopathy, DMLab, DID, EurosAI, KITTI, Flowers102, Pets, Patch Camelyon, Resisc45, Sun397, SVHN:
from timm. data import create_transfor solf.trransfommtraain create_transform
input_size- (224, 224),
is is.training=True: color_jitte:=00.4
rtto_aagmeett=rrad-d----- re_prob=0.0,
re_mode=/pixel',
re_count=l,
diterpolatioonn/iibbii.
transforo.mmmmmmm transforme [0]
Seaeefooms.Resizee (224, 224),
interpolation-3)
15

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
self. cransform_test tranaforms Compose ([
transforms Resize (224, 224)),
FoTeesor transforms JoTensor (),
Normmll (mean- (85, 0.456, 0.406], std= (0.29, 0.224, 0.225])])
B.2.2. COMMONSENSE REASONING
For more details on the data augmentation used for the commonsense reasoning task, refer to (Wang et al., 2024).
C. All Proof
C.1. Proof of Theorem 4.1:
We restate the theorem
Theorem C.1. Ler E (0, 1). With the probability at least 6 over the choice of DN, we have
KDKL (QIIP) + log
LD Lo (x*) <
min (Ap Ferqu max { (s((') - (0,0")))) + I X
(6)
2N
where D is the general data/label distribution, is the upper-bound of the loss function E(O; x. v). and cK 20,00,0 0')
K EK c(0,, 0,) represents distanceedivergencc between two models.
Proof. Under the assumption that the loss (0;I,9) S L is bounded by L > 0, it follows from Theorem 4.1 developed by (Alquier et al., 2016) that for any 3 > 0
g.
Co (Qx) s Ls (2 + DKL (QKIIRK) + log +
(7)
8NN
Consider the term (KL pK), which is equal to:
QK (O....., 0x))
(K |
gk(0...
(O1....., ... 00x log
d0, d0, ..dd0x
Jp.....k 0x
pk (01. ..... 0k)
K
I Q(+x)
II ((0x)) log
d0, d02 ... d0x
Jy..... OK #=1
I P(Ox)
Q(Ow) log Q(fk)
II [ ((ww))dor ddn
P(Ox)
K/ Jew
K
Q(0x) log ((Ok)
d8t
P(O+)
k=1
KDRI. (QIPP)
V8N (Q] (QK pKP +og
Da.( |
By choosing
Eq. (7), the RHS becomes
L. Therefore:
2N
16

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
LD (Q*) s Ls (Q*) + + DKL (QK (K) + log ;
2N
KDKL. (QI + log 1 A
Ls (Q*) + +VV
2N
KDxL (QII + log 8
Ls (Q*) L
max & ( Q²)
2N
According to (Blanchet & Murthy, 2019), the duality problem of the DRO problem implies that
Ls (o*)=
Ap + Ee-Qu max {Ls (0') - Ao (0.0))
max @*=W.(Q*.04)
S min X20
Therefore, we conclude that:
(0,00)) (00 4
KDRI. (Q11P) + log !
LD (Q*) < min X20
Ap Ba
max {ss (0)) Ac
(8)
2N
C.2. Proof of Corollary 4.2: Corollary C.2. Given a metric d over the model space, consider the following cost function
d(0,0')
dd(000)= sp p
c00,0') 0')
to0
otherwise.
With the probability at least 8 over the choice S DN,
we have
KDxi. (QIIP) + log
CD (D(kk) s BBnqu
Cs [s( (
max
+ +
2NV
where define the ball (0) = (0': d(0 0²) p)
Proof. This result follows directly from the fact that
max {Ls (6') - Ac()000)) = max Ls (0')
because (0, 8') becomes +oo when 0' & B, (0) (0' d ( 0*) s p}. Apply this result to Theorem 4.1, the outer minimization in Eq. (8) is obtained when À which concludes the proof.
C.3. Proof of Corollary 4.3: Corollary C.3. With the probability least over the choice of S D.N we have
17

---

Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
CD Co(ok)
- min A20
Ap + E max
{
ΣK K
1(0,; 2r, v)
+ addii (f (01,x;
A
Σ d 8,
K
{)}}}
f2,x
K
K
ΣK | 12 + Kd(o log 0) + 2 log 8
+ LV
4N
where D is the general data/label distribution, I. is the upper-bound of the loss e, and is the model size.
Proof. This result follows directly from Theorem 4.1 and from the fact that
K
K DEL (N (H4, IW (0.1))
DEL (QIIP) = DxL
1
Σ
N
(us, 02) IV (0.10
=
K
K
1=1
1
(Ilssll2 + do dlogo).
2K
18


Reviews:
[{"review_id": "XUfFyMMycG", "reviewer": "Reviewer_djew", "summary": "This paper introduces a distributionally robust method for Bayesian estimation, aimed primarily at fine-tuning foundation models. Central to the contribution is a term to promote particle diversity during optimization. Theoretical results of the proposed method are provided, and extensive fine-tuning experiments for vision transformers and language models are presented.\n\n## Update after rebuttal\n\nThe authors have committed to fixing the errors in the proof of Theorem 4.1 and clarified its claims. This was my primary concern in the submitted manuscript, and thus, I have raised my score from a 1 to a 3 post-rebuttal.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and would like to address the concerns as follows:\n1. **Regarding the Theoretical Claims:**\n- **Regarding the citation of prior work in the proof of Theorem 4.1** : Thank you for the suggestion. Our proof relies on Theorem 2.1 from the cited study. We will clarify this in the revision to better guide readers.\n- **Regarding the $\\beta$ value in our proof:** We appreciate the reviewer for poiting out our typos. In line 858, ${1}/{\\delta}$ should be $\\log\\frac{1}{\\delta}$. In line 878, the correct value $\\beta$ should include a nested square root over the divergence and $\\log\\frac{1}{\\delta}$, as reviewer suggested. Also, “equation (6)” in line 878 should refer to line 858, which is equation (7). We carefully double-check our proof and except these typo, our proof remains true. Specifically, the RHS of this equation (7) actually becomes $\\sqrt{\\frac{\\text{D}_\\text{KL}\\left(Q^{K} \\| P^{K}\\right)+\\log\\frac{1}{\\delta}}{2N}}\\times L$ with the correct value of $\\beta$ above.\n2. **Regarding our claim about promoting diversity**\n- **Conventional Bayesian frameworks lack explicit mechanisms to model interactions between particles θ1:K during training:** Indeed, in Section 3.3, we introduce what we meant by traditional Bayesian framework. Given a training set $S$, we have the closed from for the posterior $p(\\theta|\\mathcal{S})$. To handle this posterior, variational approach aims to learn an approximate posterior $q(\\theta)$ using the ELBO. At the end, we sample $K$ models $\\theta_1,\\dots,\\theta_K$ from $q(\\theta)$ for ensemble the predictions. Certainly, this framework does not allow the particle interaction.\n- **The novelty of our promoting diversity mechanism:** \nOur main contribution is the joint distribution $Q^K$, which models interactions among particles and enables distributional robustness. This links to sharpness-aware minimization and allows flexible definitions of interaction (e.g., prediction divergence), unlike prior work like SVGD, wherein particle diversity is primarily enforced through similarity kernels over weight vectors.\n3. **Comparison with nonlinear Stein VI approach**: \n- Thanks for pointing out us the nonlinear Stein paper. This is an interesting paper that extends Stein Variational Gradient Descend (SVGD). These two share the same form of the objective function  $\\max_{\\rho}{ \\left(F(\\rho)+H(\\rho)\\right)}$ where $H\\left(\\rho\\right)=-\\int\\log\\rho d\\rho$ is the entropy. As pointed out in Eq.(8) in Theorem 1 of the non-linear Stein paper, the second term in this equation relevant to $\\nabla_{\\theta}k\\left(\\theta,.\\right)$ is known as the repulsive term derived from maximizing the entropy $H\\left(\\rho\\right)$. This term encourages the particles to spread out for avoiding the mode collapse.\n- Differently, our **proposed approach** enables the model interaction through  $l_{div}\\left(\\theta_{1:K},x,y\\right)$. Interestingly, we can define this term appropriately to encourage various kinds of diversity (e.g., diversity in the model span or diversity in model predictions). Evidently, in our practical method, motivated by the theory of Determinantal Point Processes, we propose a diversity loss to encourage the diversity in model predictions that is proven to improve the ensemble performance. \n- Finally, we will cite and discuss the non-linear Stein paper.\n\n4. **Typographical comments**: We thank the reviewer for catching these typos. We will correct these typos in the revised manuscript.\n5. **Comparison results with IVON LoRA:** As reported in **Section 5.2** of our paper, we conduct experiments on CommonSense Reasoning, which are also used in IVON-LoRA’s study. For the reviewer’s convenience, we will present our results alongside theirs as follows:\n\n| Accuracy| WG-S | ARC-C | ARC-E | WG-M | OBQA | BoolQ |\n|-------------------|------|-------|-------|------|------|--------|\n| IBDR (ours)|  72.51|   70.56|  86.95 |   76.46 |   84.60 |   86.89     |\n| IVON-LoRA |  72.1|  69.9|  87.5     |  76.6    |  80.9    | 86.1       |\n\n| ECE  | WG-S | ARC-C | ARC-E | WG-M | OBQA | BoolQ |\n|-------------------|------|-------|-------|------|------|--------|\n| IBDR (ours)|   24.17   |   21.20    |  9.71|   11.19|  5.82    |1.54 |\n| IVON-LoRA |  27.5    |  25.8|  10.1 |  23.0|   11.2|   5.6|\n6. **Other Reviewer's Question**\n- **The meaning of \"non-maximal prediction probabilities\"?**: These are the predicted class probabilities excluding the one assigned to the ground-truth class. For example, if the true class is $y_3$ and the original model prediction probability is $[p_1, p_2, p_3, p_4]$ for classes classes $y_1, y_2, y_3, y_4$, then the non-maximal prediction is $[p_1, p_2, p_4]$\n- **Regarding question about entropy-based regularizer**: In our proposed approach, we use the diversity loss $l_{div}\\left(\\theta_{1:K},x,y\\right)$ to encourage the diversity of model predictions instead of the model weight diversity from the entropy maximization.", "claims_and_evidence": "The empirical claims about the performance of IBDR seem well-evidenced. There are several claims about variational inference that I do not agree with (see the \"essential references\" below), but this is more minor. I also have significant comments about the theory (see below).", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense. Standard fine-tuning tasks are chosen for reasonable foundation models, with strong baselines.", "theoretical_claims": "I am not an expert in distributionally robust optimization in particular, but was not able to follow all technical claims, and believe presentation can be improved. For example, the proof of Theorem 4.1 states \"it follows from the PAC-Bayes theorem developed by (Alquier et al., 2016) [...]\" -- but (Alquier et al., 2016) is a 40 page paper with several theorems. It appears that the authors are referring to Theorem 4.1, but then there is a missing logarithm for $1 / \\delta$ (which is introduced later). The proof subsequently states \"by choosing $\\beta = $[...] in Eq. (6)\", but there is no $\\beta$ in Eq. (6). Assuming this is actually referring to the unlabeled equation above, the stated value of $\\beta$ is not correct; unless I'm missing something, even after correcting $1 / \\delta \\rightarrow \\log(1/\\delta)$, the correct value of $\\beta$ should actually be \n$$\\beta = \\frac{\\sqrt{8N}}{L} \\sqrt{D_{\\text{KL}}(Q^K \\parallel P^K) + \\log(1 / \\delta)}$$\nfor the algebra to work out. I did not attempt to verify further claims.", "experimental_designs_or_analyses": "The experimental design and analysis is appropriate as far as I can tell. As mentioned above, the methods and evaluation criteria are appropriate.", "relation_to_broader_scientific_literature": "The proposed method relates to the current and relevant topic of fine-tuning foundation models, through an interesting lens of distributional robustness.", "essential_references_not_discussed": "Several times it is claimed that variational inference methods lack a mechanism to promote diversity. However, approaches like nonlinear Stein VI [1] have very similar explicit forms, optimizing an objective that combines a loss function with a diversity penalty. One may also use mixture variational posteriors, which have recently been shown quite successful in deep learning tasks [2].", "other_strengths_and_weaknesses": "I think clarity can be improved, particularly in the theoretical section.", "other_comments_or_suggestions": "Some typographical comments:\n- (Line 16, Right Column) These citations should use `\\citet{}`.\n- (Line 150, Right Column) There is a missing space in \"cross entropy (CE)\".\n- (Line 249, Right Column) Strictly speaking, $f(x; \\theta_i)$ is generally not a \"predictive probability.\"\n- (Line 321, Left Column) \"Gradient descend\" should be \"gradient descent\".\n- (Appendix C) The notations $D_\\text{KL}(Q \\parallel P)$ and $D_\\text{KL}(Q , P)$ are inconsistently used.\n\nI think it would be interesting to compare results to LoRA with IVON [3], though this was only made publicly available in December and thus falls under \"concurrent work.\"", "questions_for_authors": "1. What is the meaning of \"non-maximal prediction probabilities\"?\n2. The nonlinear SVGD paper [2] found that the repelling regularizer could be quite important, and in particular settles on an entropy-based regularizer rather than a log-determinant-based one. Have the authors considered other similar choices?\n\n## References\n[1] Wang, D., & Liu, Q. (2019). Nonlinear Stein variational gradient descent for learning diversified mixture models. In International Conference on Machine Learning (pp. 6576-6585). PMLR.\n\n[2] Shen, Y., Daheim, N., Cong, B., Nickl, P., Marconi, G. M., Bazan, C., ... & Möllenhoff, T. (2024). Variational learning is effective for large deep networks. arXiv preprint arXiv:2402.17641.\n\n[3] Cong, B., Daheim, N., Shen, Y., Cremers, D., Yokota, R., Khan, M. E., & Möllenhoff, T. (2024). Variational Low-Rank Adaptation Using IVON. arXiv preprint arXiv:2411.04421.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wvxRlnRROa", "reviewer": "Reviewer_NLbp", "summary": "The paper introduces Interactive Bayesian Distributional Robustness, a novel Bayesian inference framework designed to improve ensemble diversity and robustness in fine-tuning foundation models. The core idea of IBDR is to explicitly model interactions between multiple sampled particles in the Bayesian inference process. Unlike traditional Bayesian methods, which treat sampled models as independent, IBDR leverages a joint distribution and introduces a divergence loss to enforce diversity among the sampled models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments and respond to the key concerns as follows:\n\n**\"While IBDR is evaluated on multiple datasets, computational overhead is not thoroughly analyzed. Given that the method involves interactive Bayesian sampling, a study on efficiency would be valuable.\"**\n\nThank you for the insightful suggestion. We would like to point out that an analysis of performance and runtime with varying numbers of particles is already provided in Appendix A.1. For your convenience, we report these corresponding tables below.\n\n| Accuracy         | Camelyon | EuroSAT | Resics45 | Retinopathy |\n|---------------|----------|---------|----------|-------------|\n| 1p            | 82.4     | 93.1    | 84.2     | 73.8        |\n| 2p            | 84.8     | 93.9    | 86.6     | 74.3        |\n| 4p            | 85.1     | 95.0    | 87.3     | 76.5        |\n| 8p            | 85.8     | 95.5    | 87.4     | 77.0        |\n\n\n| Runtime (sec/epoch) | Camelyon     | EuroSAT     | Resics45    | Retinopathy  |\n|---------------------|--------------|-------------|-------------|--------------|\n| 1p                  | 51 ± 1.8     | 50 ± 1.5     | 48 ± 1.7     | 51 ± 0.7      |\n| 2p                  | 80 ± 2.1     | 83 ± 2.4     | 93 ± 2.1     | 85 ± 0.9      |\n| 4p                  | 158 ± 4.3    | 161 ± 4.9    | 156 ± 4.1    | 151 ± 2.1     |\n| 8p                  | 220 ± 6.2    | 230 ± 5.7    | 218 ± 7.3    | 246 ± 6.8     |", "claims_and_evidence": "1. IBDR enhances ensemble diversity compared to existing Bayesian methods  -- Supported by experiments.\n2. IBDR improves robustness through Wasserstein-based distributional optimization  -- Supported by theoretical derivations and empirical results\n3. IBDR generalizes across tasks, from vision (ViT) to language models (LLaMA-2) -- Supported by experiments", "methods_and_evaluation_criteria": "The accuracy and Expected Calibration Error (ECE) metrics are appropriate for evaluating both predictive performance and model uncertainty.", "theoretical_claims": "The paper presents several theoretical claims, particularly Theorem 4.1 and Corollary 4.2, which establish an upper bound for the population loss under the IBDR framework. These claims appear mathematically sound and extend prior results in distributional robustness.", "experimental_designs_or_analyses": "The experimental design is well-structured and follows standard machine learning evaluation practices.", "relation_to_broader_scientific_literature": "- Bayesian Neural Networks: Extends variational inference and Bayesian optimization techniques.\n- Sharpness-Aware Minimization: Connects to robustness-aware training methods.\n- Distributional Robustness Optimization: Uses Wasserstein-based robustness techniques.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "While IBDR is evaluated on multiple datasets, computational overhead is not thoroughly analyzed. Given that the method involves interactive Bayesian sampling, a study on efficiency would be valuable.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "NAAN", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "7d8dDfpwm2", "reviewer": "Reviewer_yWcp", "summary": "The authors introduce a method to encourage diversity in an ensemble of Bayesian neural-net particles.  \nTo achieve this, they combine results from distributional robustness and determinantal point processes\nto derive a PAC-Bayesian-style upper bound on their target objective. An approximation of this bound becomes \n their training objective.\nAs the title already suggests, the paper focuses solely on finetuning existing models and is evaluated extensively on both image classification and LLM reasoning tasks.\n\n___\n_Post-rebuttal update: Switched from weak reject to weak accept given the rebuttal._", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your feedback. We provide detailed responses to the main concerns as follows:\n1. **l195: 'conventional Bayesian frameworks' can't enforce diversity**\n- Indeed, in Section 3.3, we introduce what we meant by traditional Bayesian framework. Given a training set $S$, we have the closed from for the posterior $p(\\theta|\\mathcal{S})$. To handle this posterior, variational approach aims to learn an approximate posterior $q(\\theta)$ using the ELBO. At the end, we sample $K$ models $\\theta_1,\\dots,\\theta_K$ from $q(\\theta)$ for ensemble the predictions. Certainly, this framework does not allow the particle interaction.\n- Our main contribution is the joint distribution $Q^K$, which models interactions among particles and enables distributional robustness. This links to sharpness-aware minimization and allows flexible definitions of interaction (e.g., prediction divergence), unlike prior work like SVGD, wherein particle diversity is primarily enforced through similarity kernels over weight vectors.\n\n2. **The method is evaluated only for the subtask of fine-tuning ...**\n- A key limitation of ensemble-based methods is the need to store and train multiple models, which becomes computationally expensive when dealing with large-scale models. \nBesides, in the era of model finetuning, where fine-tuning pre-trained models is a dominant paradigm, our method becomes particularly advantageous. It mitigates the primary drawbacks of traditional ensemble approaches, by requiring only the storage of multiple lightweight adapters rather than full models.\n- Following your suggestions, we have also conducted additional experiments involving training from scratch on CIFAR100 as follows:\n|Accuracy|Resnet18|Resnet34|\n|-|-|-|\n|Standard Training|76.29|77.31|\n|IBDR (4 particles)|77.45|77.92|\n3. **... why (5) is still supposed to be a valid upper bound as it contains a multitude of changes** \n- Eq (5) is a relaxation of the one in Corolarry 4.3 that keeps its spirit. Particularly, for the one in Corolarry 4.3, given $\\theta_{1:K}\\overset{iid}{\\sim}Q$, we need to find out the perturbation $\\theta_{1:K}^{'}$ all in once that maximizes the loss while staying closely to $\\theta_{1:K}$ via minimizing $\\frac{\\lambda}{K}\\sum_{k=1}^{K}c(\\theta_{k},\\theta_{k}^{'})$. In the relaxation version, we isolate $\\theta_k$ and $\\theta_k^{'}$ in the sense that we find the perturbation $\\theta_{k}^{'}$ around $\\theta_k$ by maximizing the loss and minimizing the distance $c(\\theta_k^{'}, \\theta_k)$, while replacing $\\theta_{-i}^{'}$ by the fixed $\\theta_{-i}$. This is tolerable because $\\theta_{-i}^{'}$ stays closely to $\\theta_{-i}$. Besides being easier to implement, this relaxation still preserves the spirit of the original one. \n4. **The theory depends on the fact that the loss function has to be bounded...but this constraint is dropped completely during application...**\n- We do not break or violate the bounded loss constraint in our application. In particular, we mainly focus on classification tasks with the CE loss, whose formulation can be simplified as $-\\log p_y$ where $y$ is the ground-truth label of $x$ and $p_y$ is the model prediction. We acknowledge that $-\\log p_y$ tends to infinity if and only if $p_y$ tends to 0. This is impossible due to 2 reasons: \n    - At the beginning of training, the model lacks prior knowledge and typically makes near-uniform predictions. This causes the predicted probability $p$ to fluctuate around $1/M$, where $M$ is the number of classes, rather than approaching zero.\n    - During training, the objective is to increase the predicted probability for the correct class (maximize $p_y$). Therefore, the training process inherently pushes $p_y$ closer to 1, not 0.\n- In addition, we notice that in the SAM paper, the McAllester PAC-Bayes bound was used to develop its theory, which is only applicable to the 0-1 loss. In this paper, we leverage with more advanced PAC-Bayes theorem, leading to a less restrictive family of loss functions.\n5. **.., paper's discussion of the field is mostly limited to varying MCMC methods. Other Bayesian approaches,... is completely omitted**\n- We will definitely discuss [D'Angelo & Fortuin, 2021] in the revised version. However, we notice that our approach is fundamentally different from these Stein approaches. \n6. **The LoRA paper reference:**\n- Thanks for your comment. This is indeed an error in our bibtex. We will definitely fix them in the revised version.\n7. **l370: \"Table 1, IBDR outperforms all baselines by large margins...\"**\n- We will change the wording. Indeed, Table 1 includes a column labeled AVG, and our method outperforms all other baselines by more than 2% in terms of average accuracy.\n8. **Why are the baselines between the two experimental setups different**\n- We chose baselines closely related to our method for image classification, and followed BLoB’s setup for commonsense reasoning, including comparisons with popular Bayesian methods in LLMs.", "claims_and_evidence": "The claims are broadly consistent and provided with sufficient evidence. Some restrictions are\n- Claims regarding uniqueness. l195 claims that \"conventional Bayesian frameworks\" can't enforce diversity (see comments on related work below)\n- Despite general derivations and theoretical applicability, the method is evaluated only for the subtask of fine-tuning and lacks a proper empirical evaluation of its general statements.  (see below)", "methods_and_evaluation_criteria": "Both, the methods and evaluation criteria make sense given the restriction the authors impose on themselves in the title, i.e., fine-tuning. \nHowever, the abstract and most of the theory are written and presented as a general new method for particle-based ensemble learning. The experimental setup is not able to evaluate that.", "theoretical_claims": "All theoretical claims are supported by detailed enough proofs whose correctness I checked. \nAn exception is the step from theory to practice, which is not justified, i.e., from Corollary 4.3 to equation (5).\nWhile the authors claim this to be _\"a minor relaxation\"_ (l299) they lack arguments why (5)\nis still supposed to be a valid upper bound as it contains a multitude of changes. \nAdditionally, the theory depends on the fact that the loss function has to be bounded. As far as I can see this constraint is dropped completely during application making leaving the theoretical foundation of the approach on a rather weak foundation.", "experimental_designs_or_analyses": "To evaluate fine-tuning results for classification and common-sense reasoning, the experimental design is adequate and extensive with numerous baselines and ablations.", "relation_to_broader_scientific_literature": "Except for the crucial omission discussed in the next section the paper's relation to the broader literature\nis properly discussed.", "essential_references_not_discussed": "Although the paper belongs to the field of particle-based Bayesian ensembles, its discussion of the field\nis mostly limited to varying MCMC methods and the claim that _\"conventional Bayesian frameworks\" (l198)_ lack \nthe required mechanisms.\nOther Bayesian approaches, e.g., the literature which can be summarized under the keyword \"repulsive ensembles\" (D'Angelo & Fortuin, 2021) is completely omitted.\n\n\n_____\nD'Angelo and Fortuin, Repulsive deep ensembles are bayesian, NeurIPS 2021", "other_strengths_and_weaknesses": "## Strengths \n- clear and understandable writing style\n \n\n## Weaknesses\n- Apart from those already discussed, the major weakness of the paper is its reference section. Many papers point to arxiv preprints instead of the published versions (starting, e.g., with Abbas et al., who published their work in ICML 2022). \nOthers appear multiple times, e.g., Foret et al., Nguyen et al., Welling et al.,... . \nThe most drastic case is the Lora paper reference. It is referenced once as \"Hu, E. and et al.\", then once more with the complete author list \"Hu, E. J., Shen...\" and finally as Bartlett et al., 2023. The claimed author list is disjoint from the Lora authors, and the arxiv reference finally points to a completely unrelated Reinforcement learning paper. Given that there seems to be a researcher called Lora Bartlett that could mean that an llm was used at least for some of the references.", "other_comments_or_suggestions": "- Please follow the ICML style guide when submitting to ICML and use `\\citet` and `\\citep` whenever appropriate\n- l370: _\"According to Table 1, IBDR outperforms all baselines by large margins.\"_ -> That is not what the table shows, there are even situations where IBDR is outperformed by large margins. What is shows is that IBDR outperforms the baselines on average. \n\n### Typos\n- l162 right column is missing a minus sign in the exponential \n- Running title is broken", "questions_for_authors": "- Why are the baselines between the two experimental setups different?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OIKUS6DH0K", "reviewer": "Reviewer_KcS3", "summary": "The authors propose a new Bayesian inference framework called “Interactive Bayesian Distributional Robustness” (IBDR). IBDR is designed to improve the quality and diversity of model ensembles by modelling interactions between individual models in the ensemble in order to prevent them from collapsing into similar solutions. The proposed IBDR framework uses a dual optimization procedure that promotes both distributional robustness and model diversity. The authors demonstrate empirically that using the IBDR framework leads to improved performance on image classification and common-sense reasoning tasks, where IBDR outperforms baseline methods on the majority of tasks.\n\n## update after rebuttal\nAs I stated in my comment below, the authors sufficiently answered all of my questions in their rebuttal and provided additional analysis that I think will strengthen the paper. I have increased my score accordingly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thanks reviewer KcS3 for their supportive review and feedback. We would like to address some question and concern as follow:\n1. **What is the limit on IBDR performance improvement as the number of particles increases?** \nThank you for the helpful suggestion. Following your comment, we conducted additional experiments using a greater number of particles. The results are reported in the table below for your convenience.\nAs shown, the performance of our method generally improves as the number of particles increases. Notably, we observe a slight performance gain when using 12 particles. However, this appears to be the upper limit of improvement—using 16 particles yields comparable or even slightly lower performance than using 12 or 8 particles.\n| Accuracy         | Camelyon | EuroSAT | Resics45 | Retinopathy |\n|-|-|-|-|-|\n| 1p            | 82.4     | 93.1    | 84.2     | 73.8        |\n| 2p            | 84.8     | 93.9    | 86.6     | 74.3        |\n| 4p            | 85.1     | 95.0    | 87.3     | 76.5        |\n| 8p            | 85.8     | 95.5    | 87.4     | 77.0        |\n| 12p            | 86.1     | 95.3    | 87.6     | 77.3        |\n| 16p            | 85.9     | 95.0   | 87.6     | 77.2        |\n2. **The accuracy-runtime tradeoff when choosing between IBDR and other baseline approaches.**\n\nWe thank the reviewer for the insightful suggestion. We have already conducted experiments comparing the runtime of our method with several baseline approaches. As shown in the table below, our method has a slightly higher runtime than some baselines such as SA-BNN and SVGD. Deep Ensemble, being the simplest among the compared methods, achieves the fastest runtime—this aligns with its relatively lower accuracy, as reported in Table 1 in our paper. These results highlight the performance-runtime trade-off among different methods, as the reviewer suggested. We will extend our runtime analysis to include all datasets and baselines, and incorporate the findings into the revised version.\n\n| Model           | Camelyon | EuroSAT | Resisc | Retinopathy |\n|-----------------|----------|---------|--------|-------------|\n| SA-BNN          | 128      | 121     | 125    | 122         |\n| SVGD            | 124      | 118     | 121    | 120         |\n| Deep Ensemble   | 67       | 64      | 66     | 66          |\n| IBDR (4 particles) | 158   | 149     | 156    | 151         |\n\n3. **Regarding some typos in our paper**\n- Typo in **running title** and in **Section title: “B.2. Data augmentations”**:\nThank you for the helpful suggestions. We will correct the typo in the running title by removing the extraneous \"s\". Additionally, we will capitalize “Augmentations” in Section B.2 to ensure consistency with the formatting of other appendix section titles.", "claims_and_evidence": "All claims made in the submission are supported by clear convincing evidence. In particular, the large number of tasks and baseline comparisons done in the experiments section provide clear empirical evidence of claims made regarding the performance improvements obtained by using IBDR.", "methods_and_evaluation_criteria": "Yes, the authors selected a relevant set of tasks in image classification and common-sense reasoning, and compared IBDR directly to all relevant baselines in the literature that I am aware of.", "theoretical_claims": "The authors provide proofs of Theorem 4.1, Corollary 4.2, and Corollary 4.3 in the appendix (section C). I read these proofs thoroughly. As far as I am aware, the proofs are correct and accurately prove the three relevant claims.", "experimental_designs_or_analyses": "I did check the validity of the experimental design. In particular, the authors provided a link to their anonymized codebase which made it easy to check the validity of their experimental setup by looking directly at the code they ran to produce the results provided in the paper. As far as I am aware, the experimental setup is valid.", "relation_to_broader_scientific_literature": "The authors show that IBDR improves performance of ensemble learning. This is relevant to the broader community because ensemble learning is very relevant in the literature, especially for applications such as fine-tuning foundation models. Additionally, mode collapse is a very well-studied and relevant problem in the literature. The authors propose a novel and interesting way to address this problem of mode collapse that I think will be of interest to the broader research community. Additionally, the authors demonstrate that IBDR outperforms relevant baselines in the literature on a variety of relevant benchmark tasks, providing motivation for researchers to use IBDR in practice.", "essential_references_not_discussed": "There is no essential related work missing from the paper as far as I am aware.", "other_strengths_and_weaknesses": "Main Strengths:\n\n1. Novelty: IBDR is a novel Bayesian inference framework that introduces a novel approach to deal with the model collapse problem - one of the most relevant problems in this area in the literature. \n\n2. Theoretical motivation: The authors provide strong theoretical evidence to motivate their IBDR framework (i.e. Theorem 4.1). \n\n3. Empirical evidence: The authors provide compelling empirical evidence to demonstrate that IBDR outperforms other state of the art methods in the literature across a large number of benchmark tasks. \n\nPotential Weaknesses:\n\n1. What is the limit on IBDR performance improvement as the number of particles increases? \nThe authors mention that as the number of particles is increased, the ensemble quality is improved, thus improving performance. However, there is an inherent tradeoff here because increasing the number of particles also increases the runtime linearly. Tables 4 and 5 demonstrate this accuracy-runtime tradeoff with empirical results for 1, 2, 4, and 8 particles. The authors use this result to provide justification of their choice to use 4 particles to balance this tradeoff - which I do agree is a reasonable choice given these results. However, it is clear from Table 4, that up to 8 particles, the performance continues to improve as the number of particles increases. This begs the question: what is the limit on the number of particles after which adding additional particles no longer improves performance? I.e. would accuracy continue to increase as we add up to 20 particles? Up to 100 particles? There must be some limit where adding more particles no longer continues to improve accuracy? I therefore think that it would be very interesting and informative if the authors were to extend the results in Tables 4 and 5 to include larger numbers of particles, ideally up to the number of points where this “limit” on continued accuracy improvement is reached and we see that the accuracy stops improving. This would be useful to demonstrate the best possible performance of IBDR in cases when practitioners are willing to pay higher runtime costs to achieve the best possible accuracy. \n\n2. Runtime of IBDR compared to baselines? \nTable 5 provides runtimes per epoch for IBDR with different numbers of particles. However, I think it would also be good to include the runtime per epoch for all baseline methods compared to in Tables 1-3. This would demonstrate the accuracy-runtime tradeoff when choosing between IBDR and these baseline approaches.", "other_comments_or_suggestions": "1. Typo in running title: “Promoting Ensemble Diversity with Interactive Bayesian Distributional s for Fine-tuning Foundation Models”? What is s? \n\n2. Section title: “B.2. Data augmentations”, augmentations should be capitalized for consistency with other section titles in appendix.", "questions_for_authors": "Please see the two specific questions listed above under “weaknesses”.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel Bayesian inference framework called Interactive Bayesian Distributional Robustness (IBDR) for fine-tuning foundation models, which aims to improve ensemble diversity and robustness by modeling interactions between multiple sampled particles. The reviewers praised the paper for its novelty, theoretical motivation, and compelling empirical evidence. The main criticisms were that the method might struggle with calibration, the limit of IBDR performance improvement as the number of particles increases was not fully analyzed, and some reviewers found the theoretical results and relation to other Bayesian approaches to be unclear.

The authors responded by addressing these concerns and providing additional analyses and explanations. The reviewers indicated that the authors' rebuttal has addressed most of the criticisms, particularly regarding the analysis of the limit of IBDR performance improvement and the runtime comparison with baseline methods. Therefore, the reviewers unanimously agreed to accept the paper. We would still recommend that the authors take the reviewers' feedback into account when preparing the camera-ready version.
Decision: Accept (poster)