Geometry-Aware Collaborative Multi-Solutions Optimizer for Model Fine-Tuning with Parameter Efficiency
Authors: Van-Anh Nguyen
Trung Le
Mehrtash Harandi
Ehsan Abbasnejad
Thanh-Toan Do
Dinh Phung
Venue: NeurIPS 2025 poster ()
Keywords: optimizer
PEFT
parameter efficience
fine-tuning
multiple solutions
Primary area: deep_learning

Abstract: We propose a framework grounded in gradient flow theory and informed by geometric structure that provides multiple diverse solutions for a given task, ensuring collaborative results that enhance performance and adaptability across different tasks. This framework enables flexibility, allowing for efficient task-specific fine-tuning while preserving the knowledge of the pre-trained foundation models. Extensive experiments across transfer learning, few-shot learning, and domain generalization show that our proposed approach consistently outperforms existing Bayesian methods, delivering strong performance with affordable computational overhead and offering a practical solution by updating only a small subset of parameters.

Collaborative Geometry-Aware Multi-Solution Optimizer for Efficient Model Fine- Fine-Tuning
Van-Anh Nguyen
Trung Le Department of Data Science and Al Monash University, Australia trungJm@monash. edu
Department of Data Science and AI Monash University, Australia van-anh. nguyendmonash. edu
Mehrtash Harandi
Department of Electrical and Computer Systems Engincering Monash University, Australia haarrtsh.haraddi woaanddimonas...mmmmm edu
mehrtash. harandi
Ehsan Abbasnejad
Thanh-Toan Do
Department of Data Science and AI Monash University, Australia
Department of Data Science and Al Monash University, Australia toan wwwmonash. edu
ehsan abbasne aabaannajdd@mmnahh...mmm edu
Dinh Phung
Department of Data Science and AI Monash University, Australia dinh. dinh. phung@monash. edu
Abstract
We propose a framework grounded in gradient flow theory and informed by geomet- ric structure that provides multiple diverse solutions for given task, ensuring col- laborative results that enhance performance and adaptability across different tasks. This framework enables flexibility, allowing for efficient task-specific fine-tuning while preserving the knowledge of the pre- pre-trained foundation models. Extensive experiments across transfer learning, few-shot learning, and domain generalization show that our proposed approach consistently outperforms existing Bayesian meth- ods, delivering strong performance with affordable computational overhead and offering a practical solution by updating only a small subset of parameters. The code for our method is at https///gituuccoom/aattAAAA....
Introduction
The rapid growth of foundation models, particularly Transformers 41 based architecture, has fundamentally transformed the field of artificial intelligence. Pre-trained on massive datasets, these models have demonstrated extraordinary capabilities to learn rich, contextualized representations has led to tfte-of-thhear performance across a wide range of applications from natural language
processing [14 43 50
to computer vision [15 39 29
56).
While these models are powerful, adapting them specific downstream tasks remains challenging due to their enormous size and computational demands. Moreover, in many real-world scenarios, generating multiple diverse solutions for task can improve robustness, adaptability, and ensemble performance. However, directly optimizing multiple instances of such large models is computationally
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

infeasible. A practical and efficient way to overcome this problem is to optimize set of compact auxiliary modules that integrate into the foundation model, while keeping most of the original pretrained parameters unchanged and shared across all solutions.
Multiple compact auxiliary modules have been proposed for parameter-efficient fine-tuning (PEFT) frameworks to effectively adapt foundation models to new tasks by tuning only a small fraction of parameters. Initially, full fine-tuning of these models, which often contain millions or billions of parameters, is computationally expensive, memory-intensive, and impractical when multiple tasks are involved. Additionally, limited task-specific data might require careful regularization to prevent overfitting. To address these challenges, pioneering PEFT techniques such as Adapter [20], prompt tuning [22]. LoRA [21]. SCT [55]. and BitFit [52] have demonstrated promising results by maintaining most pre- pre-trained parameters fixed, thus enhancing both computational efficiency and performance.
Building on the efficiency of PEFT, we propose a framework that generates multiple diverse solutions by optimizing lightweight modules while reusing a shared backbone. This design enables collab- orative, robust predictions with minimal overhead, preserving the generalization capability of the pre-trained model. We initiate this process by guiding the posterior toward the target distribution using gradient-based updates. A common approach is Stein Variational Gradient Descent (SVGD) [27], which balances high predictive performance with solution diversity through repulsive interactions in parameter space. However, SVGD ignores the geometric relationships among solutions sese-ssiccccll...... how they align in output space or interact within the loss landscape.
To overcome these limitations, we propose the Geometry-Awane Collaborative Multi-Solution Op- timizer (GAC-MSO), a theoretically grounded and tractable framework for generating diverse, high-collaborative solutions. Unlike SVGD, which promotes diversity solely in parameter space via kernel-based repulsion, GAC-MSO also integrates geometric structure [3] and enforces output- space diversity through divergence term. This leads to efficient solution space exploration while maintaining strong predictive accuracy and calibration, even with limited computational resources.
To summarize, our key contributions in this paper include:
We propose the Geometry-Aware Collaborative Multi-Solution Optimizer (GAC-MSO), a framework grounded in gradient flow theory 3 over the probability space of models. This formulation enables the incorporation of geometric structure 1 2 and promotes the generation of diverse yet collaborative solutions.
We conduct extensive experiments on PEFT across various settings, including model fine- tuning for transfer learning, few-shot learning, and domain generalization. The results demonstrate that our GAC-MSO consistently outperforms baseline methods by a significant margin, highlighting the effectiveness of incorporating geometric structure and promoting diverse yet collaborative solutions.
2 Related works
Parameter Efficiency Parameter Tuning. Parameter-efficien fine-tuning (PEFT) has gained attention for adapting large pre-trained models to downstream tasks by minimizing computational costs. Several methods have been developed achieve this.
- Adapter tuning. These methods adapt Transformer-based models by inserting lightweight neural modules into each layer and fine-tuning only these modules, while keeping the model parameters frozen. These adapters typically adopt a bottleneck architecture comprising two small fully connected (FC) layers 220 and an activation function 9 In the context of vision tasks, certain methods (51) extend the adapter design by incorporating convolutional layers or Normalizing Flows [46] to better capture spatial patterns and complex feature distributions.
- Prompt Tuning.
These methods adapt models to new tasks by introducing additional learnable
visual prompts, which are either inserted into the backbone applied perturbations to existing weights. VPT-Shallow [22] inserts prompts only before the first encoder layer, while VPT-Deep 22 places prompts at each encoder layer for deeper integration.
-LoRA Tuning. These methods introduce additional parameters during training or fine-tune specific subsets of the model, while ensuring that these modifications efficiently integrated into the

---

backbone architecture to minimize inference overhead. A pioneering approach in this category is LoRA (2N) which inserts low-rank decomposition matrices into attention layers and merges them with the original weights at inference time, maintaining both computational efficiency and strong performance. Building on LoRA, several variants have been proposed to further enhance its flexibility and effectiveness. AdaLoRA [54] dynamically adjusts the rank of LoRA modules during training based on their importance, improving parameter efficiency. LoRA-Drop [57] introduces dropout within LoRA modules to regularize training and prevent overfitting, particularly in low-resource scenarios.
Variational Gradient Descent Approach. This strategy enables sampling multiple models from the posterior distribution, a central technique neural network inference often realized through Hamiltonian Monte Carlo (HMC) [3]. Although HMC is effective, it is computationally intensive due to its reliance on full gradient evaluations. To improve scalability, Stochastic Gradient HMC (SGHMC) [TO] uses noisy gradient estimates, facilitating efficcent exploration of the solution space. Similarly, Stochastic Gradient Langevin Dynamics (SGLD) [4] incorporates Langevin dynamics into stochastic gradient framework. In contrast, Stein Variational Gradient Descent (SVG [27] approximates the posterior using a set of interacting particles. Building on this, enhances SVGD with nonlinear transformations that encourage greater partiele diversity, addressing SVGD's tendency to collapse in multimodal settings and improving its ability to learn complex mixture models. Complementarily, [T] proposes repulsive mechanism for deep ensembles that fosters diverse yet plausible members, resulting in more faithful approximation of the Bayesian posterior.
Background
3.1 Gradient Flow in Probability Space
Problem Setting. We start with the problem setting used throughout this paper. Consider the target distribution p (0) exp (-5 (0)} over Rd, where (-) is the energy function, we need to find efficient ways of sampling from this target distribution. It should be noted that this setting can be directly applied Bayesian inference where the energy function is the empirical loss (0) over training set 5 = {(2;, y4) b:)) which is defined as
1 N N
Ls (0)
(((((E))).)))
Σ
where f (xx; 0) is the prediction output of the model with the model parameter 0 and ( is loss function.
It is evident that p (-) is the solution of the following optimization problem:
{ [())= =3 /vdp+ / flog pdp)
min (EP(
(I)
where P(Rd)) is the space of distributions over TRd with the Wasserstein distance 3
The gradient flow of F in the Wasserstein space described by:
V OF
0.ps div Ps
0,
0p.
where OF is the first variation (functional derivative) of and div is the divergence operator. Sp.
3.2 Stein Variational Gradient Descent
Given the current distribution Pe as the time step I, our aim is to find the velocity field 1 = + que using the steepest descent direction:
We = argmin, d + vp))+pt) lo-o.
(2)
dn
where # is the transport operator, id is the identity function, and 17 > 0 is the step size.

---

The next distribution solution P++1 = tp where 8 = id + 1p4. Moreover, by restricting the velocity E k. where H. the Reproducing Kemel Hilbert Space (RKHS) corresponding the E. Stein Variational Gradient Descent (SVGD) 27 reaches the closed-form solution for the optimal velocity Dy as
positive semi-definite kernel (0, 0') θ
[- K (0.0) VoV (0) + VoK (0.0)
0 -
Collaborative Multi-Solution Optimizers
Given the current solution Pt. to find the next solution Pc+11 we use the proximal operator as follows:
min (EP(RA)
(F( (p, pt)},
where d (p, Pr) a proximal operator, which is defined below.
4.1 Proximal Operator
We define the divergence d (p, Pt) as
d (p, Pt) E.0-p. Ee [ (1: 0') ; (x;
(z;
(a;; ())]
(4)
where KL is the Kullback-Leiberr (KL) divergence.
the following lemma, we approximate d(p, pt), exposing the geometry around the current solution Pr. All proof in our theory development can be found in Appendix B in the supplementary material.
Lemma 4.1. The divergence d(p. Pe) in Eq. 4 can be approximated as
[00--0] 0)¹ H (0) (0' - -0]
d (p. p:) ~ op
where (0) = Ex E, V log fy (x: 0) Vo log (x; (x;0)]] 0) and fy (x; 0) is the y-th prediction
Ex
output in the prediction probability vector (x;
4.2 Theory Development
We denote the gradient flow of the optimization problem (OP) in (3) as (Po)aze that satisfies the continuity equation (38) (Chapter 4, Page 110):
a.ps + div (p.v.) = 0, Vs 2 f,
where div is the divergence operator and 0g is the velocity field.
Let ff E M with M P (IR)) be the perturbation function (i.e., describing how changes over time) on the tangent space T, M such that
I div (pive) O,
which further implies f Oipt.
Thus, for small step size 17 > 0, it follows f v P which leads to Pe+- RJ + 1/S. We further derive
"
0F (F(p))
F Fp)) = F -(p(+n))FF(P)
F(p)) = nf.
(6)
Op.
= /
OF (pt) (p.)
(0) d0 !)
OF (p.)
(pi)
-n -1
(0) div (p: (0) 02 (0) do
(7)
Op.
Op:
/
0F (pt)
(0) 2 (0) P: (0) d0,
(8)
Opt
where is due to Eq.
5 and is due to the integral by part.

---

Noting that #p = Prtt by the definition of the velocity field, and relating this to the proximal
operator in we arrive
mine, (F(u##pe) F (px) d (p:pt)))
which can be reformulated into due to (8) and Lemma 4.1
OF (pt)
A H (0) A0 }0}
min
( 24 (0) Pt (0) d0 Ea-ps
Ope
where A0 = 0y ()) 0.
Moreover, Theorem 4.2 characterizes the optimal solution of OP in ( which involves the geometry
of the particles sampled from pr.
Theorem 4.2. The OP in (9) receives the following optimal solution
OF (pt)
0 - 0 Hn
0
(10)
Op:
where 0 (0)=
Ex
By
Ve
log fo x: 0) V log fy
x;
should be noted that although the update formula in Theorem 4.2 has a closed form, is intractable
because V
(FF(p))
i
+ V Po 0 is intractable due the term V log Pr 0 In
Sp.
what follows, we present how to estimate this term to obtain tractable solution.
Tractable Solution.
To develop a tractable solution, we first notice that üf O0) = Ø
V
(P()))) Up:
0 = + yūū (0) is the velocity so that Pr+n = ही #pr minimizes F(p) F(p.)
in vicinity of Pt. To find the optimal increment ūn (0) = - 2F(p))
we seek the steepest
descent direction as in Eq. (2). To this end, we strengthen F(p) by adding the divergence term and then replacing f' log pdp by similar term using the convolution operation inspired by [8]
Inspired by 8), to make smooth the entropy function, we redefine F(p) as
dd / [log (K p) dp,
F = - 3
(11)
where K * (0) K (0, 0') (0') d0' is the convolution operation using the kernel K, which aims to make the entropy function smooth.
Moreover, given the current solution Ps and the velocity ūN id + vpūr, define the divergence term as
M II
Laio (ūe, 7)) = laiv
( 1:Ad
dptial
( (
(12)
m=1
M
f lavv (10m. + rpse (0.. 1
II
P: (0m) d0 d0
where laiv (O1arr) is the loss that encourages the particles 0 more diverge and p ū. Pt- Conceptually, by minimizing Lave (ūe. 17) in 12) we aim to learn velocity é, = id + nūu in such a way that 0 [el] ploel Ue are encouraged to diverge. Eventually, given the current solution Pt. we learn the velocity in = id + quū to minimize
g (ār. 1) F (plirl) + aLaiv (ūr, 1)
(13)
where a 0 is trade-off parameter and pldd] Je#pt.
In particular, we aim to find an optimal velocity ü, that simultaneously minimizes F(p) and pushes the diverge particles. In the following theorem, characterize the steepest descent direction.
5

---

Theorem 4.3. The steepest descent direction has the following form: V (ū,,n) 1==0= (h, ūz).
where (...) is the dot product on H and
Eø' mpe [ (0, 0' VK (0,.)
(-) Eggrpe
BV (0) (0)
Egran [K (0, 0')]
M Ve Lav (01:M) K (m.
+ a.E a.Er 1:Mr rupe
Furthermore, using the first-order Taylor expansion, we obtain
g (ū, n) (ü+, 0) v,V,, (we, m) 1==0 +0(1²).
Therefore, by restricting ün in the ball of radius (h, h) inside Hi, we yield iद = -h, hence of id + nuū id n/. Finally, referring to (IO). we reach
2i 0 ē nH (0)
OF (pe)
(0) 0 +nH (0)
Op:
nH -1 h (0)
Practical Method. In what follows, we present the practical implementation of our method. Similar to SVGD (27). we maintain a set of M particle models, denoted by 01.M- For implementation conve- nience, use the same number of particles in Eq. (2)) To estimate H (0)-), 1, approximate using only its diagonal elements. Furthermore, we compute this estimate using a moving average of (0) accumulated from past to current iterations.
)) + m 0
m 0 = =-diag [Ba.,))BB V log fu (x;;) (..0)vo Vg log fy [x;
where 7 [0:1] is momentum decay and B, is the current mini-batch of (ar, y) at the current iteration.
Moreover, each particle 0² (i.e., (. is updated as follows:
0²+1 0¹
- NV
(0'h. K (0th. 0²) +
mh (0') + M
m=1
Σ D'
K (Øt m. 0t AK (0)
0¹)
Vem lav (0 K (0 0²)
(14)
Em K (0t m. 0tn))
M
where 6 represents particle model at the t-th iteration. The training algorithm is presented in Supplementary.
Divergence Term. Our framework facilitates the integration of diverse computational terms to promote both collaboration and diversity among particle models. To illustrate its effectiveness, introduce a specific term designed to encourage the particle models to generate diverse output predictions, ultimately enhancing the performance the final ensemble.
We now describe the formulation of the divergence loss lai (01.M x. y). For a given data point (xx, v) E S. let fx; 0,) denote the predicted probability distribution produced by the particle model 0,. Define f- (z; 0,) (abbreviated as f"y when context permits) as the non-maximal prediction vector obtained by removing the ground-truth class y from the prediction. Following the approach in
35 we encourage the non-maximal predictions fig. (for i = 1, ... C. where C' is the number of classes) to be mutually dissimilar, while simultaneously promoting the confidence in the ground-true predictions fg. Drawing inspiration from Determinantal Point Processes (DPP) theory (25), we define the ensemble diversity as:
laiv ( (0IM: -llog dett ve(o [F=l)e)))

---

where f-y. fiy. [s-ppl
and
{ SE|C]
E RR(CI)xKK where = (c._11......
Moreover, according to the matrix theory [
det ( [
f [ (eece))) Vol (ol( )
sE(Oy
where Vol (((
specifies the volume spanned the vectors in aim to maximize the diversity of the non-maximal predictions by maximally increasing their spanned volume.
indicating that we se(c]]
iE(C)
Model Fine-tuning with Parameter Efficiency. We focus on the fine-tuning problem, where a pre-trained model, denoted as @, is provided, and the goal is identify the optimal parameters = th + A, with A representing an additional component. Various parameter-ete
fine-tuning (PEFT) methods, such LoRA [11], Adapters (20), prompt-tuning [22 have been developed to achieve this objective and have demonstrated remarkable performance compared to the conventional full fine-tuning. Since A is typically much smaller component than the complete model in PEFT methods, we can conveniently maintain and learn an empirical distribution over several light-weight components A, making our approach feasible.
5 Experiments
In this section, we conduct extensive experiments across various settings to validate the effectiveness of our proposed method: Image classification Benchmark, Domain generalization setting, and Few- shot Learning. Each experiment is repeated with three random seeds, and the mean accuracy is reported.
Detail the experimental setting is presented in Appendix A, which includes the backbone, how to set up multiple particles, the kernel function, and trade-off parameters.
5.1 Image classification
VTAB-1k dataset [53] consists of 19 distinct datasets, which are grouped into three categories: Natural, Specialized, and Structured. Each dataset contains only 1,000 images for training, making the task challenging due the limited amount of data. Additionally, the images show significant variation in data distribution across the datasets, further complicating the learning process.
We conduct experiments using four particles for GAC-MSO and all baselines, except full fine- tuning (FFT), AdamW, and SAM, for which we use a single particle consistent with standard LoRA-based fine-tuning of foundation models. Each particle is randomly initialized the start.
It's important to note that using four particles increases the total number of trainable parameters by a factor of four compared to single-particle methods. However, more parameters do not necessarily lead to improved performance and can sometimes even degrade it. As shown in Table I most baseline methods with multi-solution settings perform worse compared to the single-solution setting methods. This may be due to inefficient model scaling or the trade-off between learning diverse solutions and optimizing individual performance. Despite these challenges, GAC-MSO method outperforms all baselines, achieving the highest average accuracy with a notable 2.3% improvement.
Additionally, we evaluate all methods using the Expected Calibration Error (ECE), which measures how well the predicted probabilities align with actual outcomes. The results, presented in Table 2 show that GAC-MSO achieves a comparable ECE score SAM under single-particle settings and outperforms other SAM-based methods, such as SADA-JEN particle scenario. SAM-baser approaches are known for producing solutions that lie in flatter regions of the loss landscape, correlating with better generalization. Additionally, our method achieves better ECE performance compared to SVGD 28 which records the best ECE score among the other multi-solution baselines.
49 and SA-BNN [32], in the multi-
FGVC dataset. The FGVC benchmark comprises five fine-grained datasets for visual classification tasks: CUB-100-201 [2). NABirds 40). Oxford Flowers R4 Stanford Dogs [12, and Stanford Cars [7]. Each dataset contains between 1,000 and 21,000 images for training, offering diverse

---

Table 1K results evaluated on Top-1 accuracy. All methods applied to finetune the same
set of LoRA parameters on ViT-B/16 pre-trained with ImageNet- dataset.
Natural
Specialized
Strectured
CIFAR
Cabechio
DTD
Plower
Pos
Sun397
EumnSK
Retinopatty
Clove-Di
DM
XITII
d5pr Loc
dSpr
SNORB- -0--
Single selution setting
PFT
68.9
87.7
64.3
97.2
56.9
87.4
38.5
79.7
95.7
84.2
73.9
56.3
58.6
41.7
65.5
57.5
46.7
25.7
29.1
65.6
Admm
M
67.1
90.7
68.9
98.1
90.1
$4.5
54.2
84.1
94.9
84.4
73.6
82.9
69.2
49.8
78.5
75.7
47.1
31.0
44.0
72.0
SAM
16
22.7
90.3
71.4
99.0
90.2
34.4
52.4
82.0
92.6
84.1
74.0
76.7
68.5
47.9
74.3
71.6
43.4
26.9
29.1
70.5
Mulii-slutionn setting
Deeplns 26
88.9
67.2
90.7
54.5
94.8
75.3
47.1
68.2
36.6
30.1
35.5
Boye-Ture
75.1
93.7
70.1
76.2
71.1
SADA-IFM
91.9
91.2
77.0
72.1
79.4
70.7
29.6
SA BNN
65.1
91.5
71.0
39.4
89.3
55.2
85.2
94.5
86.4
75.2
61.4
63.2
40.3
71.3
68.3
34.5
27.2
31.2
68.1
SVGD 25]
71.3
90.2
71.0
98.7
90.2
84.3
52.7
83.4
93.2
86.7
75.1
75.8
70.7
49.6
79.9
69.1
41.2
30.6
33.1
70.9
GAC-MSO (Ours)
73.7
94.9
72.6
99.4
91.6
85.8
58.3
86.2
96.2
86.9
74.0
79.0
63.8
51,0
79.9
84.4
58.3
33.4
46.4
74,5
Table 2: VTAB-1K results evaluated on the Expected Calibration Error (ECE) metric. All methods are applied to fine-tune the same set of LoRA parameters on ViT-B/16 pre-trained with ImageNet-21k dataset.
Natural
Specialized
Structured
CHARIO
Calichto
DTD
Plwerr12
P4ts
SVIN
Sun390
Camelyon
Eun AT
Rosis4
Retinopatly
Clee-Coo
Oleve
DMLab
KITII
48pr Loc
08pp
SNORD gle
Method
AVG
Single solution setting
[2
0.29
0.23
0.20
0.17
0.44
0.23
1140
0.26
0.21
0.31
0.17
SAM
0.30
0.11
0.12
0.16
0.37
0.16
0.19
0.16
0.16
Multi-soutionn sedting
Decpen 25
0.24
0.32
01222
0.04
0.30
0:13
0.23
0.16
0.07
0.15
0.21
0.31
0.32
0.36
0.13
0.32
0.31
01.16
0.20
0.20
BayerTune
A3
0.32
11.08
11.20
11.03
0185
11:12
00.22
0.13
0.117
0.13
0.22
0.12
0.23
0.30
0.24
01.28
0.23
01.31
11.26
0.23
SGLD 18
0.26
0.30
0.17
0.05
0.18
0:14
0.23
0.18
0.09
0.12
0.32
0.26
0.79
0.21
0.36
0.42
0.39
0.11
0.34
0.22
[0]
SVGD
23
0.20
0.13
0.19
0.04
0.16
0.09
0.20
0.15
0.11
0.13
0.12
0.17
0.21
0.18
0.31
0.25
0.36
0.18
GAC-M50 (Ours)
0.14
0.03
0.16
0.60
0.06
0.11
0.15
0.12
0.03
0.18
0.29
0.05
0.25
0.41
0.38
0.16
range of challenges for fine-grained image recognition. Detailed results and experimental setup are placed in Appendix A.
The results demonstrate that our GAC-MSO method achieves notable improvements in both accuracy and ECE score. In particular, GAC-MSO significantly outperforms the SVGD approach on calibration, achieving an ECE score of 0.05 compared to 14 on SVGD. This highlights the effectiveness of our approach not only in enhancing predictive performance but also in improving model confidence and trustworthiness in fine-grained classification tasks.
5.2 Few-shot learning
In this section, we extend our analysis to few-shot learning setting by varying the number of training samples (shots) per class across 2, 4, 8, and 16. We evaluate performance on five fine- grained datasets: FGVC-Aircraft 30). Oxford-Pets (36), Food-101 (5), Stanford Cars (24), and Oxford-Flowerss002 B3
We adopt the same experimental setup as described in Section 2 for standard image classification, using four particles for GAC-MSO, SVGD [28), and Deep Ensemble methods, and a single particle for AdamW. The results for each dataset are shown in Figure 1, where our GAC-MSO achieves highest accuracy across most of the shot settings compared to baselines. The detailed results, including accuracy and ECE scores, are provided in Appendix A. Overall, GAC-MSO also demonstrates a notable improvement calibration performance, achieving lower ECE scores than competing methods.

---

AdamW
Deep-ensemble
GAG-MSO
Food101
OxfordPets
09
Figure 1: Accuracy on the few-shot benchmark FGVC. The x-axis represents the number of shots (training samples per class) in this setting. Evaluation of ECE scores provided in Appendix.
Table Top-1 accuracy on domain generalization experiments. All models are fine-tuned with subset ImageNet-1K K dataset and tested on five datasets.
Accuracy
ECE
Method
Source
Target
Source
Target
ImageNet
Sketchh
-V2
-A
R
ImageNet
Skkttch
V2
-A
-R
Single-solution setting
AdamW 21
70.8
20.0
59.3
6.9
23.3
Multi-solution setting
Deep-ensemble [26
79.4
36.2
68.9
17.6
33.9
0.069
0.028
0.041
0.130
0.034
SVGD 28
77.4
36.6
67.3
17.1
35.1
0.500
0.253
0.435
0,088
0.232
GAC-MSO (Ours)
79.6
37.3
69.1
19.6
35.8
0.058
0.049
0.044
0.123
0.049
5.3 Domain generalization
We analyze the robustness of our method in practical scenarios where domain shift (58 is unavoidable. this setting, the model fine-tuned on a subset of the ImageNet- dataset [13, which includes 16 samples per class. After fine-tuming from ImageNet- 1K: the original ImageNet-lk validation set, ImageNet-Ve V2 [37]. and ImageNet- Sketch [4]. Additionally,
test the model on three widely used validation sets derived
also include two challenging benchmarks: ImageNet-A [19. which consists of naturally adversarial samples, and ImageNet-R [88, which contains artistic and abstract GAC-MSO method consistently achieves
renditions of ImageNet classes. As shown in Table higher accuracy than all baseline methods across all test sets, including both mild (Sketch and V2) and extreme (Adversarial and Rendition) domain shifts. The improvement gap is notable the more challenging ImageNet-A and ImageNet-R datasets. Additionally, GAC-MSO maintains comparable or better ECE score, indicating that the predictions remain well-calibrated even under out-of-distribution conditions.
5.4 Effectiveness of geometry-aware and divergency term
In this section, we analyze the effectiveness of two key components: the geometry term, which encourages diversity in the model space by leveraging geometric structure, and the divergence term, which promotes diversity in predictions within the output space. Results are presented in Table 4 Incorporating the geometry term leads to significant performance improvement compared to models without it, highlighting the benefit of modeling relationships in parameter space, Furthermore, adding the divergence term provides additional gains, demonstrating its effectiveness in enhancing ensemble diversity and improving predictive performance.
5.5 Additional experiments trade-off a of divergence term and number of particles
Detailed experiments and results presented in the Supplementary.

---

Table Results on the VTAB-IK Dataset. We report the average performance across the three task groups: natural, specialized, and structured.
Geometry Divergence
Natural
Specialized
Structured
Average
79.77
84.60
56.25
73.54
82.13
85.73
60.96
76.27
x
X
82.32
85.83
62.03
76.72
6 Conclusion and Limitation
In this work, we have addressed the challenges of adapting large foundation models to downstream tasks, particularly when diverse solutions needed for improved robustness and ensemble perfor- mance. We introduced the Geometry-Awan Collaborative Multi-Solution Optimizer (GAC-MSO), a novel framework that leverages parameter--effcien fine-tuning (PEFT) by optimizing lightweight modules while sharing a common backbone. Grounded in gradient flow theory and geometric struc- ture, GAC-MSO promotes diversity not only parameter space but also output behavior. Our extensive experimental evaluation across transfer learning. few-shot learning, and domain generaliza- tion demonstrates that GAC-MSO significantly outperforms existing baseline methods, providing strong predictive performance with affordable computational cost. These results highlight the poten- tial of GAC-MSO for efficient and effective adaptation of foundation models in resource-constrained settings.
Acknowledgment
Trung Le, Mehrtash Harandi, and Dinh Phung were supported by the ARC Discovery Project grants DP230101 176 and DP250100262. Trung Le and Mehrtash Harandi were also supported by the Air Force Office of Scientific Research under award number FA9550-23-S-000
10

---

References
[1] Shun-ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251- 276, 1998.
[2] Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry, volume 191. Ameri- can Mathematical Soc., 2000. Translated by Daishi Harada.
[3] Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient Flows in Metric Spaces and in the Space of Probability Measures. Lectures in Mathematics ETH Zürich. Birkhäuser, 2. ed edition.
[4] Dennis S. Bernstein. Matrix Mathematics: Theory, Facis, and Formulas with Application to Linear Systems Theory. Princeton University Press, Princeton, NJ, 2nd edition, 2009.
[5] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. mood-i01-mining discriminative components with random forests. In Computer vision-ECCC 2014: 13th European conference, zurich, Switzerland, September 62. 2014, proceedings, part VI 13, pages 446-461. Springer, 2014.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[8] José Antonio Carrillo, Katy Craig, and Francesco S. Patacchini. A blob method for diffusion, 2019.
[9] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances Neural Information Processing Systems, 35:1 15:16666-16678 2022.
[10] Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pages 1683-1691, PMLR, 2014.
[11] Francesco D' Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. In
A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[12] Dataset. Novel datasets for fine-grained image categorization. In First workshop on fine grained visual culegorization, CVPR. Citeseer. Citeseer. Cileseer, volume 5, page 2. Citeseer, 2011.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, Icee, 2009.
[14] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, 2019.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
[16] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021.

---

[17] Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine- grained car detection for visual census estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[18] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of distiriiibtt generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340-8349. 2021.
[19] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262-15271, 2021.
[20] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International conference machine learning, pages 2790-2799. PMLR, 2019.
[21] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[22] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European conference on computer vision, pages 709-727. Springer, 2022.
[23] Minyoung Kim and Timothy M Hospedales. Bayestune: Bayesian sparse deep model fine- tuning. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), volume 36, pages 65317-65365, Curran Associates Inc, December 2023, Thirty- Conference on Neural Information Processing Systems, NeurlPS 2023; Conference date: 10-12-2023 Through 16-12-2023.
[24] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[25] Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. In Advances in Neural Information Processing Systems, volume 24, 2012.
[26] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, 2017.
[27] Qiang Liu and Dilin Wang- Stein variational gradient descent: A general purpose bayesian inference algorithm. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[28] Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose Bayesian inference algorithm. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[30] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint (306553306.555/ 2013.
[31] Radford M. Neal. Bayesian Learning for Neural Networks. Springer-Verlag. Berlin, Heidelberg, 1996.
[32]
Van-Anh Nguyen, Tung-Long Vuong, Hoang Phan, Thanh-Toan Moan--Toanil Do, Dinh Phung, and Trung
Le. Flat seeking bayesian neural networks, 2023.
[33] M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR (06).
volume 2, pages 1447-1454. IEEE, 2006.
12

---

[34] Maria-Elear Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008
[35] Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning, pages 4970-4979, 2019.
[36] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and patlern recognition, pages 3498-3505. IEEE, 2012.
[37] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[38] Filippo Santambrogio. Optimal transport for applied mathermatiiians, calculus of variations, pdes and modeling, 2015.
[39] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[40] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 595-604, 2015.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[42] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-ucsd birds-200-2011 dataset. 2011.
[43] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 07461, 2018
[44] Dilin Wang and Qiang Liu. Nonlinear stein variational gradient descent for learning diversified mixture models. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6576-6585. PMLR, 09-15 Jun 2019.
[45] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in neural information processing systems, 32, 2019.
[46] Yaoming Wang, Bowen Shi, Xiaopeng Zhang, Jin Li, Yuchen Liu, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Qi Tian. Adapting shorteut with normalizing flow: An efficient tuning framework for visual recognition. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 55965-15977 IEEE, 2023.
[47] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-/1), pages 681-688, 2011.
[48] Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML 11, page 681-688, Madison, WI, USA, 2011. Omnipress.
[49] Xiulong Yang, Qing Su, and Shihao Ji. Towards bridging the performance gaps of joint energy- based models. Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pages 15732-15741, 2023.
13

---

[50] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[51] Dongshuo Yin, Leiyi Hu, Bin Li, and Youqun Zhang. Adapter is all you need for tuning visual tasks. arXiv preprint arXiv:2311.15010 2023.
[52] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for wranffrmmerbbsee masked language- models. arXiv preprint arXiv:2/06.10199 2021.
[53] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 191000091, 2019.
[54] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-eeffiient fine-tuning, In The Eleventh International Conference on Learning Representations, 2023.
[55] Henry Hengyuan Zhao, Pichao Wang, Yuyang Zhao, Hao Luo, Fan Wang, and Mike Zheng Shou. Set: A simple baseline for parameter-efffcient fine-tuning via salient channels. International Journal of Computer Vision, 132(3):731-749, 2024.
[56] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a
perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6881-6890, 2021.
[57] Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang. Lora-drop: Efficient lora parameter pruning based on output evaluation. arXiv preprint arXiv:2402.07721, 2024.
[58] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE transactions on pattern analysis and machine intelligence, 45(4):4396-44) 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: our claims match theoretical and experimental results. The overall setting could apply to other PEFT methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [NA]
Justification: There is limitation discussed.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
the proof of theories are presented in the Appendix and Supprementary.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details the experimental setup and full code to reproduce the results.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: We provide the link and the code with detailed instructions for setting up the dataset and hyperparameters reproduce the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli//guu guwde//ddee/ee/ wwt.:Suwwisssispoooooooooooooooi
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yer Yes]
Justification: We provide detailed settings for each experiment along with the code to reproduce.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined, or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experiments are repeated at least times with different random seeds. We report the mean accuracy save space.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification:
We do not include discussion of the resource.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We reviewed and followed the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research poses no risks for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite every dataset and bascline mentioned and used in this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research involves deep neural networks in general.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21


Reviews:
[{"review_id": "XISw7vY6kg", "reviewer": "Reviewer_W6Wr", "summary": "The authors introduce Geometry-Aware Collaborative Multi-Solution Optimizer (GAC-MSO), a parameter-efficient framework that learns **multiple** lightweight tuning modules (e.g., LoRA adapters) on top of a frozen foundation backbone.   They derive an update rule that (i) leverages a **geometry term** to keep the particle models well separated in *parameter* space, and (ii) adds an **output-space divergence term** to encourage complementary predictions\n\nPractically, they keep the backbone frozen and train *M* particles (they use *M = 4*) of LoRA modules with the derived update (diagonal Hessian approximation plus moving average).", "questions": "Please see the weakness.", "limitations": "No. The authors did not discuss any limitations in the section \"Conclusion and Limitation\" even though \"Limitation\" is explicit in the section title. The authors replied \"NA\" to the limitations in the checklist, meaning \"that the paper has no limitation\" according to the answer definitions. However, no paper is perfect without limitations, and this paper definitely contains several limitations, such as does the assumptions in the theorems are realistic in real life and the additional GPU memory consumption and flops during inference.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. The proposed solution is well-motivated based on the existing methods\n2. Provides a clear variational derivation, states lemmas and theorems, and sketches proofs\n3. The evaluation is broad, including multiple settings and datasets. \n\n## Weakness\n1. The goal is to generate diverse solutions for improved robustness and ensemble performance. However, generating diverse solutions requires additional inference cost. With the additional cost, many other approaches are possible. For example, a recent work [1] shows that different PEFT methods generate diverse predictions, and ensembling shows a consistent gain (Figure 4). If GAC-MSO uses 4 particles, it should compare with using 4 different PEFT methods. \n\n2. For robustness, [1] also shows that weight-space ensembles (model merge) can improve robustness (figure 1 (c)).  The authors should compare GAC-MSO with it or discuss the compatibility of GAC-MSO and weight-space ensembles. \n\n3. Additional clarifications are needed to improve the paper. For example\n- A figure demonstrating the training and testing pipeline of GAC-MSO would be helpful \n- \"The final output is then generated by averaging the predictions from all particles\". It would be clearer if the author could explain \"averaging\". It means majority vote, average logits or probability. \n- Some citations are confusing. e.g., tabl1. FFT (full fine-tuning)'s citation is VPT. AdamW's citation is LoRA (the whole table is based on LoRa)\n\n[1]  Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition. CVPR 2025. \n\n[2] Robust fine-tuning of zero-shot models. CVPR 2022", "was_revised": true, "final_justification": "I think the paper's contribution is enough. It provides a clear variational derivation, states lemmas and theorems, and sketches proofs\n\n\nBut given that enhancing PEFT robustness is a central motivation and objective of this paper, I believe it is essential to address the more related works in the revision, discuss the compatitbilities with GAC-MSO and compare with baselines which focus on PEFT robustness.", "rebuttal": "We appreciate the time and effort you put into reviewing our paper. We truly appreciate your thoughtful feedback and valuable comments, and we have addressed the points you raised below.\n\n**Q1: Comparison with ensemble of multiple PETF methods** \n\n We thank the reviewer for pointing out the interesting paper [1]. However, we believe that it does not compromise the novelty and technical contributions of our work. Specifically, our study focuses on particle-based sampling or variational gradient approaches, including well-established methods such as Stochastic Gradient Langevin Dynamics (SGLD) and Stein Variational Gradient Descent (SVGD), which are widely used in Bayesian inference (e.g., Bayesian Neural Networks). Technically, we consider the gradient flow in the probability space that connects the prior and posterior distributions. We then develop theoretical foundations to endow this gradient flow with a geometry, along with a divergence term that promotes collaborative yet diverse particle solutions. In our experiments, we compare our method against relevant baselines in Bayesian inference and particle sampling.\n\nMotivated by your suggestion, we experimented using a deep ensemble of four different PEFT components following [1]. For our approach, inspired by [2], we compute the kernel based on model representations—that is, we feed the current mini-batch into four particle models and use their representations to compute the kernel similarity. It is worth noting that in the original results reported in the paper, the kernel was computed based on particle parameters.\n\n\n_Table 1: Comparison with ensemble of multiple PETF methods with GAC-MSO_\n| Method                             | **Natural** | **Specialized** | **Structured** | Average |\n|------------------------------------|-------------|-----------------|----------------|---------|\n| **LoRA, Adapter, VPT-Deep, SSF**   |             |                 |                |         |\n| Deep-ensemble                      | 82.08       | 85.56           | 62.75          | 76.79   |\n| **4 particles LoRA**               |             |                 |                |         |\n| GAC-MSO                            | **82.33**        | **86.40**            | **63.48**           | **77.40**    |\n\nThe experimental results show that our approach outperforms the ensemble of four distinct particles, demonstrating the effectiveness of our proposed collaborative and divergent solution.\n\nWe also acknowledge the strength of the ensemble approach with four different PEFT particles. This simple strategy provides sufficiently diverse solutions that enhance accuracy. This observation motivates a compelling research question: how can we design a new particle sampling approach in which each particle employs a distinct PEFT type? We leave this promising direction for future work.  \n\n[2] Xingchao Liu, Xin Tong, and Qiang Liu. Profiling pareto front with multi-objective stein variational gradient descent. NeurIPS 2021.\n\n---\n\n&nbsp;\n\n**Q2: Output ensembling**\n\nFor the final output, we average the logits before they are passed into the SoftMax function to calculate the accuracy, which is a standard ensembling approach. This will be clearly explained in our revised version.\n\n---\n\n&nbsp;\n\n**Q3: Training and inference time**\n\nWhen $M > 1$, the training process involves the loss function $l_{div}$, which requires calculating the determinant of the prediction matrix, and the kernel function $K(., .)$, which involves calculating distances in the model or latent space. As a result, increasing M leads to an exponential increase in training time. However, during inference, since the loss and kernel functions are not involved, the shared architecture among particles can be done in parallel. Therefore, increasing M does not significantly impact the inference time. In practice, the number of particles is selected based on the available system resources and the expected performance. We consider the trade-off between performance and training time when increasing the number of solutions M as our limitation.\n\n_Table 2: Training and inference time (s) on 1 epoch_\n| #num solutions | Training | Inference |\n|-----------------|----------|-----------|\n| 1               | 6.0      | 1.25      |\n| 2               | 13.2     | 1.68      |\n| 3               | 20.4     | 2.83      |\n| 4               | 27.6     | 3.42      |\n\n---\n\n&nbsp;\n\n\n**Q4: Training algorithm** \n\nDue to NeurIPS policy, we are unable to use external links for figures. Instead, we present the training algorithm of our GAC-MSO here for a clearer understanding.\n\n--- \n**Input:** Initialize particles $\\theta_1, \\theta_2, ..., \\theta_M$, kernel $K(., .)$, momentum $\\gamma=0.9$, trade-off $\\beta=1$, $\\alpha=0.2$, and learning rate $\\eta$\n   \n**For** each particle $i=1...M$:\n\n**do**\n\n[1]. Compute the empirical estimate of the score function:\n\n$h\\left(\\theta_i\\right)= \\sum_{m=1}^{M}\\left[\\beta\\nabla_{\\theta_m}\\Psi\\left(\\theta_m\\right)K\\left(\\theta_m,\\theta_i\\right)-\\frac{\\sum_{m=1}^{M}\\left[K\\left(\\theta_m,\\theta_i\\right)\\nabla_{\\theta_m} K\\left(\\theta_m,\\theta_i\\right)\\right]}{\\sum_{m=1}^{M}K\\left(\\theta_m,\\theta_i\\right)}\\right]+\\alpha\\sum_{m=1}^{M}\\nabla_{\\theta_{m}}l_{\\text{div}}\\left(\\theta_{1:M}\\right)K\\left(\\theta_{m},\\theta_i\\right).$\n\n$H(\\theta_i) = \\left(\\nabla_{\\theta_m}\\Psi\\left(\\theta_m\\right) + \\nabla_{\\theta_{m}}l_{\\text{div}}\\left(\\theta_{1:M}\\right)\\right)^2$\n    \n[3] Using a moving average with momentum:\n\n$m(\\theta_i) = \\gamma H(\\theta_i) + (1-\\gamma)m(\\theta_i)$\n\n[4] Update the particle using the gradient descent step:\n\n$\\theta_i = \\theta_i - \\frac{\\eta}{M(m(\\theta_i) + \\epsilon)}h(\\theta_i)$\n\n**done**\n\n**Output:** Return the updated particles $\\theta_1, \\theta_2, ..., \\theta_M$.\n\n---\n\n&nbsp;\n\n**Citation confusion** The result of the FFT is taken from paper VPT. And the experiments for single solution using AdamW are taken from the LoRA paper because in this paper, the authors used AdamW to train. We think it will be fair to use the original results."}, {"review_id": "DwywgSMEuo", "reviewer": "Reviewer_jYjW", "summary": "The paper introduces a novel Geometry-Aware Collaborative Multi-Solution Optimizer (GAC-MSO), designed to efficiently fine-tune large pre-trained models for specific downstream tasks. Leveraging gradient flow theory and geometric structure, the authors provide a theoretically grounded optimization solution and a tractable approximation for the method. The proposed method generates diverse yet collaborative solutions by optimizing lightweight fine-tuning modules while capturing the geometric relationships within the model's parameter and output space.  These modules maintain the generalization capabilities of pre-trained backbone models while minimizing computational overhead. Extensive experiments demonstrate that GAC-MSO significantly outperforms existing parameter-efficient fine-tuning (PEFT) approaches across image classification, few-shot learning, and domain generalization tasks.", "questions": "- Can you elaborate on the computational overhead introduced by managing multiple particles?\n- Could you discuss potential challenges or limitations?", "limitations": "potential limitations are:\n- Although the authors present a tractable algorithm, the practical implementation of the proposed Geometry-Aware method might still be complex due to multiple interacting components (e.g., geometry and divergence terms, kernel approximations).\n- The effectiveness of the proposed method likely depends significantly on hyperparameters such as the kernel type, number of particles, the divergence regularization parameter $\\alpha$. This can lead to extensive hyperparameter tuning.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method accounts for the diversity not just in the parameter space like SVGD, but also among solutions in the output space.\n- The proposed approach is theoretically grounded and a tractable algorithm is provided for practice.\n- Performance is strong across diverse benchmarks compared to other baselines, surpassing other established PEFT methods.\n- The method enhances model robustness through diversified collaborative solutions, maintaining strong performance in domain generalization.\n\nWeaknesses:\n\n- Overall, the paper is a bit hard to follow since the main focus the theories. I think the author can spend a bit more effort on motivation, for instances, explain what are the geometric relationships and why are they important. \n- No discussion about limitations", "was_revised": true, "final_justification": "A technically solid paper and the performance of the method have demonstrated great potential. The author addressed the concerns of the computational budget for the method and the ablation studies for the hyperparameters. I've I carefully checked the other reviews and based on my limited expertise in this field, I decided to keep my score.", "rebuttal": "We sincerely thank the reviewer for your time to review our work. Below, we provide a response to the questions you raised.\n\n**Q1: The importance of geometry**\n\nGeometry plays a crucial role in our approach. It is derived from the proximal operator in Eq. (3), which incorporates the Fisher information as defined in Eq. (4). From this perspective, the resulting term acts as a geometry-aware regularization. Furthermore, as shown in Eq. (4), the geometry influences the update along each dimension of the model parameters.\n\n&nbsp;\n\n**Q2: Training and inference time** When $M > 1$, the training process involves the loss function $l_{div}$, which requires calculating the determinant of the prediction matrix, and the kernel function $K(., .)$, which involves calculating distances in the model or latent space. As a result, increasing M leads to an exponential increase in training time. However, during inference, since the loss and kernel functions are not involved, the shared architecture among particles can be done in parallel. Therefore, increasing M does not significantly impact the inference time. In practice, the number of particles is selected based on the available system resources and the expected performance. We consider the trade-off between performance and training time when increasing the number of solutions M as our limitation.\n\n_Table 1: Training and inference time (s) on 1 epoch_\n| #num solutions | Training | Inference |\n|-----------------|----------|-----------|\n| 1               | 6.0      | 1.25      |\n| 2               | 13.2     | 1.68      |\n| 3               | 20.4     | 2.83      |\n| 4               | 27.6     | 3.42      |\n\n---\n\n&nbsp;\n\n\n**Q3: Training algorithm**  Here we present the training algorithm of our GAC-MSO. In our algorithm, we need to compute the loss function $\\psi(.)$, $l_{div}(.)$, the kernel function $K(.,.)$, and their respective gradients. In practice, the loss function $\\psi(.)$ (which we use as the cross-entropy loss) and $l_{div}(.)$ are calculated using the output logits, and the gradients of these functions are derived simultaneously using the standard backpropagation algorithm. For the kernel $K$, we employ the RBF kernel, which is applied to the model space. The RBF function uses the $l_2$ distance, a computationally simple and efficient measure with a closed-form for its gradient. However, we agree that the overall process adds to the computational cost of the training.\n\n\n**Input:** Initialize particles $\\theta_1, \\theta_2, ..., \\theta_M$, kernel $K(., .)$, momentum $\\gamma=0.9$, trade-off $\\beta=1$, $\\alpha=0.2$, and learning rate $\\eta$\n   \n**For** each particle $i=1...M$:\n\n**do**\n\n[1]. Compute the empirical estimate of the score function:\n\n$h\\left(\\theta_i\\right)= \\sum_{m=1}^{M}\\left[\\beta\\nabla_{\\theta_m}\\Psi\\left(\\theta_m\\right)K\\left(\\theta_m,\\theta_i\\right)-\\frac{\\sum_{m=1}^{M}\\left[K\\left(\\theta_m,\\theta_i\\right)\\nabla_{\\theta_m} K\\left(\\theta_m,\\theta_i\\right)\\right]}{\\sum_{m=1}^{M}K\\left(\\theta_m,\\theta_i\\right)}\\right]+\\alpha\\sum_{m=1}^{M}\\nabla_{\\theta_{m}}l_{\\text{div}}\\left(\\theta_{1:M}\\right)K\\left(\\theta_{m},\\theta_i\\right).$\n\n$H(\\theta_i) = \\left(\\nabla_{\\theta_m}\\Psi\\left(\\theta_m\\right) + \\nabla_{\\theta_{m}}l_{\\text{div}}\\left(\\theta_{1:M}\\right)\\right)^2$\n    \n[3] Using a moving average with momentum:\n\n$m(\\theta_i) = \\gamma H(\\theta_i) + (1-\\gamma)m(\\theta_i)$\n\n[4] Update the particle using the gradient descent step:\n\n$\\theta_i = \\theta_i - \\frac{\\eta}{M(m(\\theta_i) + \\epsilon)}h(\\theta_i)$\n\n**done**\n\n**Output:** Return the updated particles $\\theta_1, \\theta_2, ..., \\theta_M$.\n\n---\n\n&nbsp;\n\n\n**Q4: The choice of hyperparameters**  In this paper, we formulate the problem in a general form, so that multiple hyperparameters are used, such as the number of particles $M$, the trade-off parameters $\\alpha$ and $\\beta$ for the loss functions, and the kernel function $K(., .)$. However, in the main experiments, we set $M = 4$, $\\alpha = 1$, $\\beta = 0.2$, and use an RBF kernel for $K(., .)$ to focus on analyzing the effectiveness of our algorithm in learning diverse solutions with geometry collaboration. The sensitivity of $M$ and $\\beta$ is explored in the supplementary material, which demonstrates the effectiveness of each hyperparameter. For the kernel function $K(.,.)$, we use the RBF kernel, as it is a common function to measure the similarity between two points in space. This simple kernel is effective for high-dimensional spaces, such as model or latent spaces."}, {"review_id": "2ta34fsoSQ", "reviewer": "Reviewer_dwXi", "summary": "The authors propose a learning framework grounded in gradient flow theory which leverages geometric structure that is supposed to enhance performance and adaptability across different tasks. The authors conduct experiments across transfer learning, few-shot learning, and domain generalization against several existing Bayesian methods.", "questions": "The authors take ideas from 8 and 25, but they eventually do not compare against these methods. Are these methods not applicable in this scenario?", "limitations": "the authors did not discuss limitations, but I can also not think of any. so might be ok?", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The authors support their method by extensive theoretical grounding. The authors proof several theorems and derive their method on existing work [8, 25]\n- Comparison against many baselines and a wide range of datasets and problem settings.\n- Ablations investigating the impact of different aspects of the method.\n\nOverall, the paper is well-written and the authors provide strong support for their claims.\n\n**Weaknesses**\nI can't think of any", "was_revised": true, "final_justification": "I had only a simple question and the authors addressed it to my satisfaction. Given my low expertise, I carefully checked the other reviews. In my understanding, there are no reason that require me to alter my rating.", "rebuttal": "Dear Reviewer,\n\nWe sincerely thank you for taking the time to review our work. Below, we provide a response to the questions you raised.\n\n&nbsp;\n\n**Q1: Idea from [8] and [25]**\n\nThanks for this comment. The paper [8] introduces a general family of a Blob approach for diffusion processes. In our paper, we leverage this Blob approach to our framework. Moreover, drawing inspiration from Determinantal Point Processes (DPP) theory [25], we develop our divergence term.   \n\n\n&nbsp;\n\n**Q2: Training and inference time** When $M > 1$, the training process involves the loss function $l_{div}$, which requires calculating the determinant of the prediction matrix, and the kernel function $K(., .)$, which involves calculating distances in the model or latent space. As a result, increasing M leads to an exponential increase in training time. However, during inference, since the loss and kernel functions are not involved, the shared architecture among particles can be done in parallel. Therefore, increasing M does not significantly impact the inference time. In practice, the number of particles is selected based on the available system resources and the expected performance. We consider the trade-off between performance and training time when increasing the number of solutions M as our limitation.\n\n_Table 1: Training and inference time (s) on 1 epoch_\n| #num solutions | Training | Inference |\n|-----------------|----------|-----------|\n| 1               | 6.0      | 1.25      |\n| 2               | 13.2     | 1.68      |\n| 3               | 20.4     | 2.83      |\n| 4               | 27.6     | 3.42      |\n\n---\n\n&nbsp;"}, {"review_id": "mq2bU9JyKN", "reviewer": "Reviewer_x89M", "summary": "This paper proposes the Geometry-Aware Collaborative Multi-Solution Optimizer (GAC-MSO), a gradient flow-based framework that leverages geometric structure to generate diverse yet collaborative solutions for machine learning tasks. Unlike traditional methods like SVGD, which rely solely on parameter-space diversity, GAC-MSO incorporate geometric insights and enforces output-space diversity through a divergence term, enhancing exploration while maintaining predictive accuracy and calibration. Experriments in transfer learning, few-shot learning, and domain generalization show that the proposed approach outperforms existing methods with affordable computational overhead.", "questions": "1. How does the number of particles affect ensemble diversity, individual model performance, and computational cost?", "limitations": "The authors have not discussed the limitations of the proposed framework in the Conclusion and Limitation section.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. This work offers conceptual innovation by combining SVGD with geometry-aware optimization, backed by a theoretical foundation in gradient flow theory and geometric structure. The proposed divergence term promote diversity in both parameter and output spaces.\n2. Experiments are performed across various settings (Image classification, Domain generalization, and fewshot learning) to validate the effectiveness of the proposed method. Ablation studies on the geometry term and the divergence term further confirm the design's effectivenss.\n\n## Weaknesses\n1. The proposed method using four particles increases the total number of trainable parameters by a factor of four compared to standard single-particle methods, which may greatly increase computational overhead and reduce practical usefulness. However, there appears to be no analysis or results on computational costs.", "was_revised": true, "final_justification": "As the computational cost analysis has been provided, I have raised my score to 4. But I remain uncertain about my evaluation of this submission.", "rebuttal": "Dear reviewer,\n\nWe thank the reviewer for your valuable the time to review our work. Below, we address the point you have raised.\n\n---\n\n**Training and inference time** When $M > 1$, the training process involves the loss function $l_{div}$, which requires calculating the determinant of the prediction matrix, and the kernel function $K(., .)$, which involves calculating distances in the model or latent space. As a result, increasing M leads to an exponential increase in training time. However, during inference, since the loss and kernel functions are not involved, the shared architecture among particles can be done in parallel. Therefore, increasing M does not significantly impact the inference time. In practice, the number of particles is selected based on the available system resources and the expected performance. We consider the trade-off between performance and training time when increasing the number of solutions M as our limitation.\n\n_Table 1: Training and inference time (s) on 1 epoch_\n| #num solutions | Training | Inference |\n|-----------------|----------|-----------|\n| 1               | 6.0      | 1.25      |\n| 2               | 13.2     | 1.68      |\n| 3               | 20.4     | 2.83      |\n| 4               | 27.6     | 3.42      |\n\n---\n\nWe agree that increasing the number of particles leads to a linear increase in training time. However, this is a common characteristic shared by particle-based sampling methods (e.g., SVGD and SGLD) as well as ensemble approaches.\n\nWe kindly ask the reviewer to consider the theoretical and technical contributions of our work. Our study focuses on particle-based sampling and variational gradient approaches, including well-established methods such as Stochastic Gradient Langevin Dynamics (SGLD) and Stein Variational Gradient Descent (SVGD), which are widely used in Bayesian inference (e.g., Bayesian Neural Networks). Specifically, we consider the gradient flow in the probability space that connects the prior and posterior distributions. We develop theoretical foundations to endow this gradient flow with a well-defined geometry and introduce a divergence term that encourages collaborative yet diverse particle solutions. In our experiments, we compare our method against relevant baselines in Bayesian inference and particle sampling, demonstrating its significant improvement over state-of-the-art methods.\n\n&nbsp;"}]

Author rebuttal:

Meta-review: The paper proposes an optimization approach for fine-tuning a small component of a pre-trained model.  Experiments show performance advantages over competing parameter-efficient fine-tuning strategies.  After the rebuttal and discussion phase, all reviewers lean toward accept, noting solid technical development and sufficient experimental validation.  The Area Chair agrees with the reviewer consensus.
Decision: Accept (poster)