Title: RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness
Venue: NeurIPS 2025
Accepted: yes
Keywords: Model merging

Multi-modal large language model

Multi-task learning

Parameter-efficient tuning

Robust Fine-tuning

Federated Learning
Abstract: Fine-tuning pre-trained models with custom data leads to numerous expert models on specific tasks. Merging models into one universal model to empower multi-task ability refraining from data leakage has gained popularity. With the expansion in data and model size, parameter-efficient tuning becomes the common practice for obtaining task-specific models efficiently. However, few methods are dedicated to efficient merging, and existing methods designed for full fine-tuning merging fail under efficient merging. To address the issue, we analyze from low-rank decomposition and reveal that direction robustness during merging is crucial for merging efficient modules. We furthermore uncover that compensating for the gap between stark singular values contributes to direction robustness. Therefore, we propose RobustMerge, a training-free parameter-efficient merging method with complementary parameter adaptation to maintain direction robustness. Specifically, we (1) prune parameters and scale coefficients from inter-parameter relations for singular values to maintain direction stability away from task interference, and (2) perform cross-task normalization to enhance unseen task generalization. We establish a benchmark consisting of diverse multimodal tasks, on which we conduct experiments to certify the outstanding performance and generalizability of our method. Additional studies and extensive analyses further showcase the effectiveness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the directional robustness in the merging process from the perspective of low-rank decomposition, showing that compensating for the significant disparity between singular values helps improve directional robustness. It proposes a parameter-efficient merging method that does not require training, maintaining directional stability through complementary parameter adaptation. The paper prunes parameters based on their inter-relationships and scales the singular value coefficients to maintain directional stability and avoid interference between tasks. Experimental results demonstrate the superior performance and generalization ability of this method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper proposes a novel Lora parameter merging method, which prunes parameters based on their inter-relationships and scales the singular value coefficients.\n2. The paper conducts extensive experiments and analysis on different models and benchmarks, validating the effectiveness and generalization of the proposed method. It also performs a thorough set of ablation experiments.\n\nWeaknesses:\n\n1.Some of the baseline methods used for comparison are somewhat outdated. Comparing with more recent merging methods would help better highlight the effectiveness of the proposed method.\n2. Can the merging method proposed in the paper be extended to other PEFT methods, such as PISSA, etc.?\n3. No OOD results.\n\nQUESTIONS:\n1. What is the specific transformation applied to matrix B using the method of complementing parameter scaling in Section 3.3?\n2. How does the method proposed in this paper compare with current model fusion methods such as Fisher[1], RegMean[2], BayesOpt[3], and LORE[4]?\n[1] Merging Models with Fisher-Weighted Averaging\n[2] Dataless Knowledge Fusion by Merging Weights of Language Models\n[3] Checkpoint Merging via Bayesian Optimization in LLM Pretraining\n[4] LORE-MERGING: Exploring Low-Rank Estimation For Large Language Model Merging\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

comments: SUMMARY:
This paper introduces the method of CoPA-Merging, a training-free method for the merging of PEFT modules in multimodal large language models. The innovation of the method lies in addressing direction robustness. This happens due to large differences of singular values between LoRA modules. This is addressed through pruning and complementary parameter scaling to maintain proper stability of direction. It also does cross-task normalization to improve generalization to unseen tasks.

STRENGTHS AND WEAKNESSES:
---- Strengths ----

The concept of directional robustness is very new from the viewpoint of low rank adaptation merging.

The great thing is that is training-free and is efficient and it does not require additional training, storage etc.

It shows strong empirical results and outperforms baselines like Task Arithmetic, DARE and Ties-Merging.

There has been thorough evaluation on seen and unseen tasks.

Analysis and Mathematical rigor is well defined in the form of variation in singular values.

---- Weakness ----

The method has only been carried on LoRA and not other PEFT tasks.

The method has only been tried on multimodal tasks. It has not been applied to image generation tasks.

QUESTIONS:
How would the distribution of singular values change with different layers ?

How would the performance change if the number of merging models/loras change ?

How would the performance change if the merging was done differently for different layers ?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the challenge of merging multiple task-specific parameter-efficient models into a single high-performance and generalizable multimodal large language model. The authors introduce CoPA-Merging, a training-free merging framework that maintains direction robustness in low-rank space through adaptively scaling singular values. Comprehensive experiments demonstrate CoPA-Merging’s effectiveness in maintaining multi-task performance across seen and unseen domains.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. Efficient merging is an important topic for current large-scale pre-trained models.\n\n2. The analysis from the perspective of direction robustness of singular values is interesting and reasonable for multi-task merging.\n\n3. Comprehensive experiments well demonstrate the effectiveness of CoPA-Merging.\n\n**Weaknesses:**\n\nI think the motivation is sufficient and the experiments well support the claims, so I only have several minor suggestions that may further strengthen the paper in my mind.\n\n1.  More PEFT methods. The exploration mainly focuses on LoRA-based mechanisms. How does it work on other PEFT methods, such as P-Tuning [1] or LLaMA-Adapter [2]?\n\n2. More MLLMs. Only LLaVA is employed in the paper. How does CoPA-Merging perform on other MLLMs, such as Qwen2-VL [3]?\n\n3. More analysis or visualization on task relationships. The paper mentions that the small singular values of one task may be altered by other tasks, reducing the performance. Is there experimental evidence indicating this phenomenon, and are there direct visualizations or analyses showing how CoPA-Merging mitigates it?\n\n***References:***\n\n[1] Xiao Liu et al., “P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks”, ACL, 2022.\n\n[2] LLaMA-Adapter-V2 Multi-modal, https://github.com/OpenGVLab/LLaMA-Adapter/tree/main/llama_adapter_v2_multimodal7b, 2024.\n\n[3] Peng Wang et al., “Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution”, arXiv, 2024.\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nLIMITATIONS:\nLimitations are discussed in Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
In this paper authors look at the problem of merging parameter-efficient fine-tuned (PEFT) models. Typically, model merging approaches fail under PEFT settings. Authors claim the reason of poor performance as direction instability along directions of small singular values. Authors propose to resolve this by 1) pruning smallest magnitude weights and  2) renormalizing the PEFT A matrix. 
Authors empirically show that this procedure leads to improvement in the model merging performance.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper looks at an important problem merging PEFT models and proposes a novel methodology (CoPA merging) for solving the problem. 
- Thorough empirical evaluation on VLMs with detailed ablation studies.
- Results in Table 4 and 5 are promising and CoPA merging leads to improvement in model merging

Weaknesses:
- Even though authors identify directional robustness as a critical property for achieving model merging for PEFT models, there is no formal definition and they lack evaluation of when one can identify if PEFT models are robust or not. It needs to be formalized
- I can’t seem to understand how pruning and scaling lead to directional robustness so justification is not entirely clear to me. I think this is related to not having a proper definition established in the paper.

QUESTIONS:
- It’ll be helpful to resolve the weaknesses in the paper

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-10