Title: Geometry-Aware Collaborative Multi-Solutions Optimizer for Model Fine-Tuning with Parameter Efficiency
Venue: NeurIPS 2025
Accepted: yes
Keywords: optimizer

PEFT

parameter efficience

fine-tuning

multiple solutions
Abstract: We propose a framework grounded in gradient flow theory and informed by geometric structure that provides multiple diverse solutions for a given task, ensuring collaborative results that enhance performance and adaptability across different tasks. This framework enables flexibility, allowing for efficient task-specific fine-tuning while preserving the knowledge of the pre-trained foundation models. Extensive experiments across transfer learning, few-shot learning, and domain generalization show that our proposed approach consistently outperforms existing Bayesian methods, delivering strong performance with affordable computational overhead and offering a practical solution by updating only a small subset of parameters.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Geometry-Aware Collaborative Multi-Solution Optimizer (GAC-MSO), a gradient flow-based framework that leverages geometric structure to generate diverse yet collaborative solutions for machine learning tasks. Unlike traditional methods like SVGD, which rely solely on parameter-space diversity, GAC-MSO incorporate geometric insights and enforces output-space diversity through a divergence term, enhancing exploration while maintaining predictive accuracy and calibration. Experriments in transfer learning, few-shot learning, and domain generalization show that the proposed approach outperforms existing methods with affordable computational overhead.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. This work offers conceptual innovation by combining SVGD with geometry-aware optimization, backed by a theoretical foundation in gradient flow theory and geometric structure. The proposed divergence term promote diversity in both parameter and output spaces.\n2. Experiments are performed across various settings (Image classification, Domain generalization, and fewshot learning) to validate the effectiveness of the proposed method. Ablation studies on the geometry term and the divergence term further confirm the design's effectivenss.\n\n## Weaknesses\n1. The proposed method using four particles increases the total number of trainable parameters by a factor of four compared to standard single-particle methods, which may greatly increase computational overhead and reduce practical usefulness. However, there appears to be no analysis or results on computational costs.\n\nQUESTIONS:\n1. How does the number of particles affect ensemble diversity, individual model performance, and computational cost?\n\nLIMITATIONS:\nThe authors have not discussed the limitations of the proposed framework in the Conclusion and Limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors propose a learning framework grounded in gradient flow theory which leverages geometric structure that is supposed to enhance performance and adaptability across different tasks. The authors conduct experiments across transfer learning, few-shot learning, and domain generalization against several existing Bayesian methods.

STRENGTHS AND WEAKNESSES:
**Strengths**
- The authors support their method by extensive theoretical grounding. The authors proof several theorems and derive their method on existing work [8, 25]
- Comparison against many baselines and a wide range of datasets and problem settings.
- Ablations investigating the impact of different aspects of the method.

Overall, the paper is well-written and the authors provide strong support for their claims.

**Weaknesses**
I can't think of any

QUESTIONS:
The authors take ideas from 8 and 25, but they eventually do not compare against these methods. Are these methods not applicable in this scenario?

LIMITATIONS:
the authors did not discuss limitations, but I can also not think of any. so might be ok?
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel Geometry-Aware Collaborative Multi-Solution Optimizer (GAC-MSO), designed to efficiently fine-tune large pre-trained models for specific downstream tasks. Leveraging gradient flow theory and geometric structure, the authors provide a theoretically grounded optimization solution and a tractable approximation for the method. The proposed method generates diverse yet collaborative solutions by optimizing lightweight fine-tuning modules while capturing the geometric relationships within the model's parameter and output space.  These modules maintain the generalization capabilities of pre-trained backbone models while minimizing computational overhead. Extensive experiments demonstrate that GAC-MSO significantly outperforms existing parameter-efficient fine-tuning (PEFT) approaches across image classification, few-shot learning, and domain generalization tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The proposed method accounts for the diversity not just in the parameter space like SVGD, but also among solutions in the output space.\n- The proposed approach is theoretically grounded and a tractable algorithm is provided for practice.\n- Performance is strong across diverse benchmarks compared to other baselines, surpassing other established PEFT methods.\n- The method enhances model robustness through diversified collaborative solutions, maintaining strong performance in domain generalization.\n\nWeaknesses:\n\n- Overall, the paper is a bit hard to follow since the main focus the theories. I think the author can spend a bit more effort on motivation, for instances, explain what are the geometric relationships and why are they important. \n- No discussion about limitations\n\nQUESTIONS:\n- Can you elaborate on the computational overhead introduced by managing multiple particles?\n- Could you discuss potential challenges or limitations?\n\nLIMITATIONS:\npotential limitations are:\n- Although the authors present a tractable algorithm, the practical implementation of the proposed Geometry-Aware method might still be complex due to multiple interacting components (e.g., geometry and divergence terms, kernel approximations).\n- The effectiveness of the proposed method likely depends significantly on hyperparameters such as the kernel type, number of particles, the divergence regularization parameter $\\alpha$. This can lead to extensive hyperparameter tuning.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Geometry-Aware Collaborative Multi-Solution Optimizer (GAC-MSO), a parameter-efficient framework that learns **multiple** lightweight tuning modules (e.g., LoRA adapters) on top of a frozen foundation backbone.   They derive an update rule that (i) leverages a **geometry term** to keep the particle models well separated in *parameter* space, and (ii) adds an **output-space divergence term** to encourage complementary predictions\n\nPractically, they keep the backbone frozen and train *M* particles (they use *M = 4*) of LoRA modules with the derived update (diagonal Hessian approximation plus moving average).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The proposed solution is well-motivated based on the existing methods\n2. Provides a clear variational derivation, states lemmas and theorems, and sketches proofs\n3. The evaluation is broad, including multiple settings and datasets. \n\n## Weakness\n1. The goal is to generate diverse solutions for improved robustness and ensemble performance. However, generating diverse solutions requires additional inference cost. With the additional cost, many other approaches are possible. For example, a recent work [1] shows that different PEFT methods generate diverse predictions, and ensembling shows a consistent gain (Figure 4). If GAC-MSO uses 4 particles, it should compare with using 4 different PEFT methods. \n\n2. For robustness, [1] also shows that weight-space ensembles (model merge) can improve robustness (figure 1 (c)).  The authors should compare GAC-MSO with it or discuss the compatibility of GAC-MSO and weight-space ensembles. \n\n3. Additional clarifications are needed to improve the paper. For example\n- A figure demonstrating the training and testing pipeline of GAC-MSO would be helpful \n- \"The final output is then generated by averaging the predictions from all particles\". It would be clearer if the author could explain \"averaging\". It means majority vote, average logits or probability. \n- Some citations are confusing. e.g., tabl1. FFT (full fine-tuning)'s citation is VPT. AdamW's citation is LoRA (the whole table is based on LoRa)\n\n[1]  Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition. CVPR 2025. \n\n[2] Robust fine-tuning of zero-shot models. CVPR 2022\n\nQUESTIONS:\nPlease see the weakness.\n\nLIMITATIONS:\nNo. The authors did not discuss any limitations in the section \"Conclusion and Limitation\" even though \"Limitation\" is explicit in the section title. The authors replied \"NA\" to the limitations in the checklist, meaning \"that the paper has no limitation\" according to the answer definitions. However, no paper is perfect without limitations, and this paper definitely contains several limitations, such as does the assumptions in the theorems are realistic in real life and the additional GPU memory consumption and flops during inference.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}