RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness
Authors: Fanhu Zeng
Haiyang Guo
Fei Zhu
Li Shen
Hao Tang
Venue: NeurIPS 2025 spotlight ()
Keywords: Model merging
Multi-modal large language model
Multi-task learning
Parameter-efficient tuning
Robust Fine-tuning
Federated Learning
Primary area: applications
TL;DR: An effective parameter-efficient model merging method for multimodal large language models from the perspective of direction robustness in low-rank space

Abstract: Fine-tuning pre-trained models with custom data leads to numerous expert models on specific tasks. Merging models into one universal model to empower multi-task ability refraining from data leakage has gained popularity. With the expansion in data and model size, parameter-efficient tuning becomes the common practice for obtaining task-specific models efficiently. However, few methods are dedicated to efficient merging, and existing methods designed for full fine-tuning merging fail under efficient merging. To address the issue, we analyze from low-rank decomposition and reveal that direction robustness during merging is crucial for merging efficient modules. We furthermore uncover that compensating for the gap between stark singular values contributes to direction robustness. Therefore, we propose RobustMerge, a training-free parameter-efficient merging method with complementary parameter adaptation to maintain direction robustness. Specifically, we (1) prune parameters and scale coefficients from inter-parameter relations for singular values to maintain direction stability away from task interference, and (2) perform cross-task normalization to enhance unseen task generalization. We establish a benchmark consisting of diverse multimodal tasks, on which we conduct experiments to certify the outstanding performance and generalizability of our method. Additional studies and extensive analyses further showcase the effectiveness.

RobustMerge: Parameter-Efficent Model Merging for MLLMs with Direction Robustness
Fanhu Zeng' Haiyang Guo' Fei Zhu Li Shen
Hao Tang"
MAIS, Institute of Automation, Chinese Academy of Sciences
Centre for Artificial Intelligence and Robotics, HKISI-CAS
Shenzhen Campus of Sun Yat-sen University "Shenzhen Loop Area Institute
State Key Laboratory Multimedia Information Processing,
School of Computer Science. Peking University
challenngzzenggh@@maail com, guohaiyang20236a..cg cn,
2hfei2218@@mail.com com, shenl 16mmail. waw..yyauuu......
byauuedu..o edu. haotang@pku.co edu.
Abstract
Fine-tuning pre-trained models with custom data leads numerous expert models on specific tasks. Merging models into one universal model to empower multi-task ability refraining from data leakage has gained popularity. With the expansion in data and model size, eeffiietr-efficientt tuning becomes the common practice for obtaining task-specific models efficiently. However, few methods are dedicated efficient merging, and existing methods designed for full fine-tuning merging fail under efficient merging. To address the issue, we analyze from low-rank decompo- sition and reveal that direction robustness during merging is crucial for merging efficient modules. We furthermore uncover that compensating for the gap between stark singular values contributes to direction robustness. Therefore, we propose RobustMerge, training-free free parameter-efficiet merging method with comple- mentary parameter adaptation to maintain direction robustness. Specifically, prune parameters and scale coefficients from inter-parameter relations for singular values to maintain direction stability away from task interference, and (2) perform cross-task normalization to enhance unseen task generalization. We establish a benchmark consisting of diverse multimodal tasks, on which conduct experi- ments to certify the outstanding performance and generalizability our method. Additional studies and extensive analyses further showcase the effectiveness. Code is available at https:: : // thub. com/ com/Aurore wwwrrrrzengfh/eee Roous Meege..
(1)
Introduction
Rapid development of foundation models has facilitated the construction of expert models from custom data. Modern models like large language models (LLMs) are pre-trained on various datasets to obtain general knowledge and employing pre-trained models typically involves fine-tuning on task-specific data to gain ability in specific areas. When dealing with tasks of different domains, multi- task learning [47] common paradigm mitigate performance variations. However, particular knowledge may be required progressively over time. As the model becomes larger [4, 59], once the model is specialized on specific datasets, it is time-consuming and resource- inoourceintenssvvee retrain models to gain knowledge of another area, even encountering catastrophic forgetting [67]. Furthermore, issues regarding data privacy may obstruct its practical application. To address these issues, model merging [51] has been proposed to integrate multiple separate models of specific knowledge off- fff-thhe- shelf into one model with multi-task ability without the demand of training accessing data. Its effectiveness and convenience show great potential in various downstream tasks [10, 49].
"Correponning authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite its popularity, crucial problems for model merging remain unsolved, restricting its real-world deployment. First, with larger model sizes like multimodal large language models (MLLMs) and massive data, parameter-effiieen tuning (PEFT) [16] has become the most popular and effective tuning approach for large models. However, existing model merging methods focus on full fine-tuning (FFT) techniques [62, 11], which struggle with distribution shift and undergo performance drops when directly applied to parameter-efficien model merging, as is illustrated in Fig. 1. Moreover, another issue lies in that current perh-perrormance methods rely extra information of seen tasks (e.g., validation data [64], extra storage [18]) to boost the performance. Therefore, they can only handle seen tasks and fail to generalize to unseen tasks, questioning their robustness and extensibility in real-world scenarios, as is concluded in Tab. 1. The most related work is LoraHub [17]. However, its requirement for coefficient optimization through test-time adaptation severely hinders its application.
Inspired by the stated shortcomings, we aim to de- velop merging algorithm for parameter-efficient modules with generalizability. First, we analyze
Partomunce oF Jraaee Saar50
MaltiTank
the reason behind the performance drop. We ob-
Tee--merging o PCB-merging
serve (1) stark singular values and (2) a distinct wider distribution in efficient parameters that differ
Task Artthmetic
from full fine-tuning teeeettnng..... Moreover, starting from the perspective of low-rank decomposition, that direction robustness, i.e., maintaining directions of singular values, is crucial for efficient merging. From the above observations, we propose Robust- Merge, novel parameter-efficienn method for high- performance merging of multimodal large models and introduce effective complementary" parameter adaptation to maintain directions for performance enhancement. Concretely, we prune ineffective pa- rameters and construct scaling coefficients from inter- parameter relations directly on LoRA components to mitigate interference between tasks aroused from stark singular values difference. Additionally, we perform cross-task normalization to balance tasks of different data scales and enhance unseen task gener- alization. It is notable that our method is free from any additional data or storage and does not require explicit decomposition, which equips the method with more flexibility and efficiency.
reveal
5B Perlomence of shen bookth))
62
66
Figure 1: Performance balance between seen task enhancement and unseen task generalization.
Table I: Prerequisites and application scope of different methods.
Methods
Validation Extra Storage Unseen Pimmeteer
Frze
Tooks
Merging
Task Arishmetic
mergingging Pch-Merging
X
X
LoraHtu
X
AdiMerging
x
X
EMR Merging
x
RobustMerge
We conduct experiments on a benchmark consisting of eight seen tasks and four unseen tasks with diverse fields to evaluate the ability on multimodal generative tasks. We also report results on common evaluation benchmarks, and it shows that our method promotes both seen (3.4% %% unseen tasks (4.5% and comprehensive common ability with a substantial margin, demonstrating the effectiveness and generalizability of our method. We additionally perform experiments on vision tasks along with extensive analyses to validate the utility of our method. Our contributions are summarized as follows:
We focus on parameter-efficieet model merging, highlighting the necessity of -eghomerforeecc parameter-efficient merging algorithms free from additional data storage.
We analyze from the perspective of direction robustness of singular values in low-rank decompo- sition and propose an effective training-free merging algorithm with complementary parameter adaptation to maintain direction for merging performance enhancement.
We conduct extensive experiments and achieve superior results compared to existing approaches. which strongly validates the effectiveness and generalizability of the method.
2 Related Work
Multimodal large language models. With the surge in data volume and model size, large language models (LLMs) [46, 56, 1] have shown their powerful performance. They are constructed with decoder-only blocks and respond to inputs an auto-regressive way, which shows their potential
'In contrast to individual, we use the term to distinguish between subspace multiplication (along dimension) and original multiplication (along do dimension) of matrix.

---

Magnitue
Task A
Task
Ta
Ta
Tu
Inteook
Inatabiitty
Model
Taa
Tt
Ta
It
2i
Maggtbuds
Tan
Tin
13
Tai
Tin
Tin
Correspording sing. par.aaraa values
Direction Instability
Direction Robustness
Direction Instabiliit
Direction Robustness
Figure 2: Illustration of merging A and B in low-rank space for evaluation of each task. The magnitude of vector represents numerical singular value. Left: Stark singular values exist within task, leading to instability when merging between tasks. Right: As directions of large singular value are naturally robust, direction instability is more tikely to happen for small values when merging specific singular vectors. Scaling tail values contributes to direction robustness and promotes the performance.
in both classification [57] and generative tasks [7]. Furthermore, multimodal large language mod- els (MLLMs) enhance large models with vision perception ability. They obtain visual features with a vision encoder and align image-text features with cross-modality module [32, 26] and so on. Current research on large models dedicated to directly fine-tuning one independent model with task-specific data to get better results. Rather than improving the performance of a certain domain, focus integrating models into one model to boost efficiency and handle multiple tasks simultaneously.
Parameter-efficient tuning, When fine-tuning pre-trained model with task-specific data, training the whole model would not only disrupt the representations obtained from billions of data but also become issurce-intensive. To address the issue, parameter-efficienn tuning [13] is introduced to refrain from tine-tuning the whole model. It typically trains lightweight modules to make the model adapt to downstream tasks and achieves competitive results compared full fine-tuning models. Various efficient tuning techniques have been explored, like prompt learning [20, 23, 66]. adapter learning including LoRA [16, 60, 34, 41, 33], 3 [31] and
In this paper, focus LoRA, as it is the most commonly utilized PEFT method and has demonstrated its usefulness various fields [68, 9] especially for large models [32].
Model merging. Model merging [63, 52, 27] refers to merging multiple models of different capabili- ties to handle multi-task learning with one universal model [21, 40). Task Arithmetic [19] presents a paradigm that obtains task vectors from subtracting a pre-trained model from fine-tuned models and treats model merging as arithmetic operations of task vectors. It has gained widespread attention in various fields [53]. Ties-merging [62] trims and elects signs to reduce interference. DARE [65] randomly drops parameters and rescales the remaining ones approximate the original embedding. PCB- OCB-merging [11] introduces parameter adjustment with competition balancing to address potential confficts. However, most of them focus merging models with FFT on classification tasks [5], and the distribution shift prevents their ability to acquire satisfying performance [54]. Some re- cent works [29, 30] also focus merging checkpoints during pre-training to enhance downstream performance. By contrast, we concentrate on parameter-efffcient merging with multimodal tasks.
3 Methodology
We first describe basic notations for efficient merging, then show our observation and motivation for reducing task interference when merging efficient modules, and finally introduce our method to improve the performance of parameter-efficient merging for multimodal large language models.
3.1 Preliminary and Notations
Parameter-efficieet tuning keeps the pre-trained model frozen and fine-tunes a lightweight module adapt downstream tasks. In this paper, we focus on LoRA [16], low-rank adaptation technique that decomposes additional parameters into two low-rank matrices. Formally, for weight matrix
Wo E Rdd xd; the updated matrix is depicted as:
W Wo + AW W. + B-A,
(1)
where B I xrd,xr, A A xd and rank < min(d,, do).
3

---

02-
Origial Prgnn
-ah ScceCCOA
2003
Praenter Biliod
0.15
SCAW www.rroo.....oo
13
10
16
-..0 0.50
Figure 3: (a) Magnitude of singular values for original and pruned matrix. Stark singular values are observed in original matrix and pruning effectively scale tail ones. (b) Effectiveness of RobustMerge by adaptively reducing interference with larger scale smaller singular values. (c) Distribution of FFT PEFT modules. Parameters of FFT. and different components in efficient tuning have different distributions.
Model merging targets combining multiple models of the same structure (0,,... 00v) that fine-tuned from pre-trained model prre into one new model 6m and maintaining multi-task ability in training-fre frree manner. Existing full fine-tuning (FFT) methods follow the paradigm proposed by Task Arithmetic (TA) [19]. Typically, they construct the task vector by performing subtraction operation Ty O₂ Ore E IRd conduct merging algorithm on the task vector subspace, and obtain
the final merged model by adding the pre-trained model, iee, 6. = Opre + + A ((TT), where (.) stands for the merging algorithm.
Parameter-efffiien model merging differs from the traditional model merging, as the backbone of the foundation model is frozen and the updated matrices to be merged are randomly initialized. Consequently, we use W to represent merging modules for simplification, and exploit the model merging method on these parameter-efficient modules, i.e., W. = Wo + A > ((AW,.)
3.2 Motivation and Observation
While existing methods perform well on FFT merging, challenges remain unsolved when it comes to PEFT merging with suboptimal performance. To have a better understanding of the difference, we (1) first analyze the parameter distribution and low-rank decomposition of a single task, (2) then reveal key factors for parameter-efficicet merging multiple tasks, and (3) finally propose an effective merging algorithm for parameter-efficient modules built on these observations.
Direction robustness is crucial for merging models of multiple tasks. To illustrate the uniqueness of parameter-effiieent tuning in merging compared to full fine-tuning, i.e., the low-rank matrices, we decompose them using singular value decomposition (SVD) to obtain singular values with corresponding directions and introduce the notation of Direction Robusiness that plays vital role in merging. Concretely, for a single matrix, the direction for each singular value can be viewed as task-specific knowledge in low-rank space and the magnitude of the singular value is the extent to which the knowledge is utilized in the current task. Theoretical analysis is provided in Appendix A.
We visualize the distribution of singular values for efficient modules in Fig. 3a and observe stark difference between head and tail singular values (intra-task). Therefore, for models of diverse tasks (inter-task) to be merged, directions of large singular values are inherently prone to direction change. When merging models and evaluating on a certain task, i.e., task ssec-fpei knowledge, the corresponding small singular values are more likely to alter the direction, challenging the stability. The same direction instability appears on other singular vectors when the evaluated task changes. Therefore, direction robustness, which refers to maintaining the direction of each singular vector during low-rank matrix merging, is crucial for reducing task interference, as each of them represents task-specific knowledge and contributes merging performance. We give illustration of merging models fine-tuned on specific tasks A, B and evaluating on each task separately in Fig. 2.
Mitigating gap between singular values is effective for nigh-performance merged model. As different tasks have their principal singular directions, certain directions may possess large singular values in one task and small ones in another. Based on the observation above and in Fig. 3, it can therefore be inferred that the direction tail singular values for certain tasks is more likely to cause instability when merging, and mitigating the gap is crucial for resolving the interference between different tasks. One direct way is to adaptively scale tail values, which has less impact on vectors with larger singular values, while greatly contributing to small ones. This can be confirmed by Fig. 3a

---

tasks
Unseen
(b)
Parameter Efficient Merging
Inference
Pruning Complementary
Individual Training
I
Known
AW;
AW,
2 Task Normolization
W
AW
Porometer
Figure 4: Diagram of RobustMerge. Tasks are divided into seen and unseen ones. Checkpoints of seen tasks are trained employing the standard individual training and merged following the pipeline of inter- parameter adaptation. During inference, the merged model is required to both enhance seen tasks and be generalizable to unseen tasks with an unknown distribution.
and Fig. 3b, which clearly show that our method introduced in Sec. 3.3 changes the distribution of singular values and adaptively adjusts the singular values by scaling smaller singular values by larger multiple, thereby alleviating direction instability and achieving better performance. Detailed illustration of singular values is shown in Appendix B.
Parameters of efficient modules have distinct distributions. We also depict the distribution of elements in Fig. 3c figure out the difference between two types of merging. It can be found that most parameters of full fine-tuning have a much smaller and concentrated distribution (distribution in dark blue), where the problem of sign conflict becomes particularly prominent [62]. Conversely, parameters in efficient components have a relatively wider range of distribution (light blue and gray), and direction instability rather than sign conflict is the main issue for interference between tasks, which we give detailed comparison in Sec. 4.3.
Parameter-ee effficcet modules have intrinsic relations. The two LoRA matrices have asymmetric functions in PEFT [69, 55]. As pointed out by AsymmetryLoRA [69], random untrained A performs well fine-tuned one and B improves the bound. HydraLoRA [55] reveals that shared A can reserve knowledge. To determine the distinct function of the two matrices in merging, we depict the distribution of A. B respectively in Fig. 3c. It turns that B follows Gaussian distribution and A has an approximately uniform distribution. corresponds with existing research that also has a more unique and crucial role in PEFT merging. Due to their expression, we aim scale the singular value directly on two LoRA modules to avoid explicit and time-consuming decomposition.
3.3 RobustMerge: Parameter-Effiiiien Model Merging for MILMS
Motivated by the observations, we introduce novel model merging method for parameter-efficient components, which targets maintaining direction robustness and compensating for the gap between singular values by adaptively scaling tail singular values. As is illustrated in Fig. 4, our approach is divided into pruning and complementary parameter scaling and cross-task normalization.
Pruning and complementary parameter scaling. Due to significantly wider distributions, changing larger parameters are more likely to alter directions in low-rank space. Therefore, rather than electing parameters of the same sign [62, 18] with delicate design, we simplify the definition of ineffective parameters to be those with small values in magnitude. In this way, the direction of matrices is not greatly changed by reserving larger parameters, and knowledge of the specific task is therefore retained when mitigating interference between models of different tasks. Consequently, with M(.) as the binary operation matrix, the updated matrices can be formulated as:
À = O == =BB(k) 3 B,
(2)
5

---

where @ stands for element wise multiplication, and l is the pruning rate of parameters. Formally, it sets percentage of parameters with small values sorted by magnitude to zero. Fig. 3a shows its utility in altering distribution and scaling tail singular values.
After pruning ineffective parameters, the remaining ones should be refined to scale tail singular values and complement the performance gap caused by task interference. Considering that explicitly operating on decomposed matrices is time-consuming, we directly adjust original low-rank matrices to achieve the same goal inspired by the asymmetry and correlation between two LoRA modules stated in Sec. 3.2 and that is almost orthogonal in high- nigh-dimenssonal space. Specifically, we propose to adaptively adjust singular values through complementary parameter scaling for transforming B to compensate for performance deficiencies resulting from the gap between singular values. As A follows uniform distribution, we construct scaling coefficients from the statistical characteristics of A for singular values. We define the scaling matrix diagonal matrix, and the elements on the diagonal are:
E abs (A..))
S
i
2==1 abs(MaRs) @ A
This can be viewed as singular value adaptation in low-rank space, in which small values of each model are effectively increased in larger proportions (Fig. 3b), thereby contributing to minimizing task conflicts aroused by direction instability while refraining from explicit computation of decomposition.
Cross-task normalization. Complementary parameter scaling coefficient S is determined in a task-independent manner. On the one hand, the imbalance in data size between different seen tasks leads to overfitting for data-abundant tasks and underfitting for data- scarce tasks. Additionally, it also poses a neg- ative effect the generalization to unseen tasks. Consequently, we conduct normaliza- tion on scaling matrices across all tasks to reduce the impact of coefficient imbalance, mathematically formulated as:
Algorithm Procedure of parameter-efficient merg- ing with complementary parameter adaptation.
Input: Fine-tuned models (A,,Ba, J pruning rate k, rank and X Output: Merged Parameter-Efficieen Model W
Step 1: Pruning and Complementary Parameter Scaling.
MA (k) = (eetarysee_top______
A, k))
MB(A) =
(B, k))
A = (A) @ A
B = (B(kk) @ B
forall .................. do
N
St > abs( A ( 2 Maj() A(aa))
S = S n = /
S
N.
(4)
end
n=1
Step Creas-Task Nornalization.
The normalization provides more balance across diverse types of tasks and therefore achieves more stable performance. It also enhances the ability unseen tasks, which
forall n = 1, ... N do
S. S DN S
end
Obtain pareneter- efficiien nodules. forall n = 1,..., N do
shown in Fig. 5c. The final efficient param- eter can be rewritten as follows:
S., Diag(S
end
AW- = A (B, S A,, (5) and the merged model weights can be ob- tained by adding merged parameter-efficient modules of all tasks. It should be emphasized that during the whole merging process, no validation data extra information storage of seen tasks is required, certifying the superiority of the method.
Herge parameter modules.
AW + EN (B.. S.) S.)-E., Ä., W. +AW
return W
Experiments
Implementation details. We conduct experiments on multimodal generative tasks, unseen task generalization, and vision tasks using multimodal models [32, 45]. We comprehensively extend our approach on the scale of the model, number of tasks, rank, and otherwise stated, all models are trained with a rank of 16. More details can be found Appendix C.
certify the utility. Unless
Datasets and baselines. For multimodal task merging, we establish a MultiModal Merging Benchmark (MM-MergeBennch) which comprises eight multimodal generative tasks including Sci- enceQA [36], ImageNet [5], VQAv2 [7], REC-COCO [22, 39], OCRVQA [42], Flickr30k [44],

---

Table Performance of MM-Merge-Bor Bench on eight seen and four unseen tasks.
SEEN TASKS
UNSEEN TASKS
Method
SciQA Image VQA REC OCR VizWiz Flickr IconQA Average
AVQA Image-R S2w TabMWP Average
Individual
83.74 96.02 67.58 43,40 65.50
64,80
57.29
75.54
69.23
Zero-Shot
61.73 40.87 62.88 36.10 41.16
41.03
49.07
14.09
43.37
51.62
28.27
5.98
15.01
25.22
Multi-Task
76.90 74.08 67.05 35.98
65.37
66.67
56.09
66.87
63.62
76.33
41.39
8.34
18.20
36.06
Task Arithmetic
71.94 57.49 67.06 38.90
62.87
44.80
49.20
39.21
53.93
74.78
37.37
7.52
13.57
33.31
DARE
71.59 57.25 66.26 39.38 62.56
44.93
49.13
39.59
53.84
73.75
37.67
7.56
13.62
33.15
Ties-merging
71.49
55.88 66.73 39.67 65.12
44.35
47.06
34.46
$3.09
73.43
38.44
7.47
13:23
33.14
PCB-merging
71.10
57.82
67.59
38.22
64.35
44.58
48.90
37.01
53.70
74.57
36.28
7.84
15.44
33.53
Merge
73.43 65.54 67.20
44.80 62.97 46.61
52.80
45.90
57.33
79.30
45.79
9.23
17.62
37.99
VizWiz-caption [12], IconQA [38]. It includes diverse multimodal tasks across various areas like question answering, grounding. classification, captioning and can comprehensively evaluate the perfor- mance of different merging methods in generative tasks. To demonstrate the generalizability unseen tasks, we evaluate the merged models on four diverse datasets, ImageNet-R [15], VOKVQA [48], Word [58]. TabMWP [37]. Detailed interpretation can be found in Appendix E. Besides, we also evaluate on general benchmarks like POPE [28]. MME [6] and MMBench [35]. Experiments on vision tasks are provided in Sec. 4.2 and more results shown in Appendix F.
For comparison methods, we re-implement Task Arithmetic [19], Ties-merging [62], DARE [65] and PCB- MCB-megging [1] ] on varameter-efficies modules of MLLMs to have fair comparison. Detailed information about these baselines can be found in Appendix D.
4.1 Experiments on MILM with Generative Tasks
We systematically evaluate parameter-efficient merging methods on multimodal generative tasks, in which LLaVA [32] used as the foundation model, with CLIP-L-336 [45] as the image encoder.
RobustMerge is efffective in parfmetee-efffcient tuning, We evaluate the performance of various model merging methods. Concretely, we obtain independent models from fine-tuning separate datasets and merge models without re-accessing data. It is indicated from the left part of Tab. 2 that existing approaches suffer from a severe performance drop when merging parameter-efficien models, even worse than zero-shot in some cases. Also, they do not necessarily perform better than simple Task Arithmetic, showcasing the challenge in PEFT merging. By contrast, our method achieves superior results, consistently and substantially outperforming all previous methods by solid margin (3.4% improvements on average). Notably, our approach even achieves comparable performance with multi-task learning. These results strongly validate the effectiveness of the method.
RobustMerge enhances performance on unseen tasks. Generalizability is crucial for evaluating merging methods as domain shifts are unavoidable and frequently oc- cur in real-world scenarios. On the right of Tab. 2, we report merging performance directly evaluated on four unseen tasks. It is harder as the merged models have no clue for the distribution of unseen tasks. This is further confirmed by the poor per- formance of existing merging methods (TA, DARE, Ties), which is even worse than zero-shot on some occasions. Conversely, method significantly enhances performance with substantial 4.5% average improvements and even outperforms multi-task learning. Notably, our method successfully promotes domain trans- fer (ImageNet-R) and specific knowledge task (TabMWP), further demonstrating the generalizability.
Table 3: Performance of different merging models general multimodal benchmarks.
Method
POPE
MME
MMBench
Zero-Shot Traditional MTL
86.4
1476.9
66.1
86.9
1433.5
62.9
Task Arithmetic
87.0
1465.2
67.3
DARE
86.4
1475.7
67.4
Ties- Ties-merging PCB-merging
86,7
1489.4 ,4
66.6
86.6
1490.7
66.3
RobustMerge
87.2
1494.9
68.1
RobustMerge outperforms on general multimodal benchmarks. We additionally report results on general multimodal benchmarks POPE [28], MME [6] and MMBench [35] in Tab. 3 to evaluate
7

---

Table Results of merging eight vision tasks with CLIP-ViT-E as pre-trained foundation model.
Method
Cars
MNIST
EuroSAT
GTSRB
DTD
RESISC45
SUN397
SVHN
Average
7ero-Shot
59.7
48.5
62.3
32.6
60.7
45.5
31.4
48.0
Individual
74.3
99.3
65.2
92.9
88.7
58.4
99.1
96.4
84,2
Task Arithmetic
63.2
37.6
62.8
50.9
37.6
51.1
DARE
60.4
52.4
63.1
37.5
62.8
44.0
50.3
37.7
51.0
Ties-merging
60.7
56.4
62.4
33.9
61.3
43.1
51.1
42.9
51.5
Merge
61.4
65.0
43.1
63.3
52.2
52.4
55.9 (+4.4)
Table 5: Results of merging eight vision tasks when pre-trained model scales CLIP-
Method
Cars
MNIST
EuroSAT
GTSRB
DTD
RESISC45
SUN397
SVHN
Average
Zero-Shot
77.7
76.3
66.8
50.5
71.0
55.3
59.9
58.4
64.4
Individual
99.7
99.4
80.0
97.2
95.8
70.3
98.6
97.9
92.4
Task Arithmetic
78.6
79.7
68.5
53.6
73.5
55.8
65.7
60.9
67.0
DARE
79.5
81.4
68.8
56.5
75.0
56.6
65.8
62.8
68.3
Ties-merging
79.4
83.4
69.5
59.4
76.0
55.7
64.0
64.4
68.9
RobustMerge
79.7
82.8
70.6
62.4
78.4
58.2
64.7
70.3
70,9 (+2.0)
base capabilities merged models like hallucination and so on. It shows that multi-task learning achieves inferior results. indicating the challenge. By contrast, our method enhances zero-shot perfor- mance, substantially outperforms cxisting methods, and retains common knowledge on challenging benchmarks with outstanding outcomes, certifying the effectiveness.
4.2 Experiments on VLM with Vision Tasks
For vision language model (VLM) with vision tasks, we follow the experimental setup outlined by Task Arithmetic [19] and fine-tune eight models with LoRA on corresponding vision datasets. The datasets consist of Cars [24], MNIST [25], EuroSAT [14], GTSRB [50], DTD [3], RESISC45 [2],
SUN397 [61] and SVHN [43]. See Appendix E for details.
RebustMerge is effective in vision tasks. We evaluate our methods on CLIP-ViT-B-32 [45] and showcase the results in Tab. 4. is indicated that when fine-tuning models with parameter--fffiient techniques, previous methods do not observe significant improvements against zero-shot performance, questioning their utility in vision task-efficient tuning. By contrast, our method obtains a considerable 7.9% promotion against the ability of zero-shot and outperforms previous merging methods by substantial margin (4.4%). It strongly validates the effectiveness of our method when merging vision models in parameter-efficient way.
RobustMerge scales well to large VLM models. We also apply method to larger models to certify the scalability of the method. Concretely, we fine-tune CLiP-ViT-L-14 on eight vision tasks separately and evaluate models with merged parameter-efficient components. The results in Tab. exhibit that the performances of all methods improve with larger foundation models. Furthermore, RobustMerge achieves the best results with 2.0% average improvement, demonstrating superiority.
Table 6: Influence of each component. Prune&scale and norm refer to pruning and complementary scaling.
Scale
Norm
SciQA
Image
VQA
REC
OCR
VizWiz
Flickr
IconQA
Average
71.94
57.49
67.06
38.90
62.87
44.80
49.20
39.21
53.93
73.03
64.18
67.50
43.12
52.24
44.54
56.14 (+2.21)
73.43
65.54
44.80
57.33 (+3.40)
4.3 Ablation Study and Further Analysis
Effectiveness of each component. We progressively apply key components of our method, i.e., pruning and complementary parameter scaling and cross-task normalization, to substantiate their effectiveness. Results in Tab. 6 illustrate that pruning and complementary parameter scaling fun- damentally contribute to direction robustness and mitigating interference in model merging, and integrating them all further achieves more advanced results.

---

22
20
12
Soan Taxt
SYopeT Tart
Piuziing
Figure 5: (a) Average performance of merging models with different pruning rates. Non-zero parameters decline according to the ineffective parameter criteria as the pruning rate increases. (b) Performance of different pruning techniques averaged
and unseen tasks. The aggregated sign achieves poor performance. (c) Comparison of average performance and standard deviation with existing methods. Cross-task normalization enhances performance with stable deviation.
Impact of rank. To explore the performance as the rank of LoRA varies, train parameter- efficient components on different ranks. Results in Fig. 6 illustrate that the model obtains pro- motion from the improvements of rank, which increases the storage of knowledge in the update subspace. Moreover, our approach continuously outperforms existing methods by substantial margin (3.4% in 16 and 3.3% in 128), validating the scalability of the method.
TA DARE
Averpg Acturnay
Rank
128
Figure 6: Average performance with different ranks.
Influence of pruning rate. As revealed in Sec. 3.2, parameters of small values have less influence on the direction robustness of low-rank decomposition. Therefore, prune parameters according to the magnitude to facilitate the merging procedure. To further validate this point of view, we gradually increase the pruning rate and show the variation of average performance in Fig. 5a. It is elucidated that: (1) As the pruning rate increases, the performance gradually boosts due to reduced interference between tasks; (2) Finally, the performance undergoes a sharp decrease as pruning larger parameters significantly influences directions and task knowledge. Consequently, the results are in accordance with the aforementioned analysis, underlining the utility of the ineffective parameter pruning strategy in our method.
Parameter pruning in magnitude provides su- perior results. We additionally compare with pruning techniques employed in DARE [65] and Ties-merging [62] to showcase the effectiveness of the proposed parameter pruning technique. Concretely, we employ random pruning and ag- gregated sign as pruning criteria, respectively, and evaluate their performance. The results in Fig. 5b reveal that sign conflict is not crucial in efficient merging, with performance worse than random. By contrast, our magnitude- based parameter pruning technique achieves better results in multimodal tasks and outperforms existing approaches by substantial margin (2.3% and 4.0%, respectively). It indicates that the value in magnitude, other than sign interference, plays vital role in parameter-efficient model merging. We attribute the promotion to significantly wider distribution parameter-efficien models than full fine-tuning models, and pruning according to sign inevitably changes direction in low-rank space. Conversely, our method avoids task conflicts with less impact on principal direction.
Table 7: Influence of complementary parameter scaling. Coefficients solely dependent on specific module (A, B or both) perform inferior to adaptive coefficients with inter- pher-parameter relations.
Method
Seen Tasks
Unseen Tasks
Baseline (w/o adaptation)
54.1
32,5
Ours (individual, A)
54.5 (+0.4)
32.1 (-0.4)
Ours (individual, B)
55.4 (+1.3)
34.1 (+1.6)
Ours (individual, A + B3)
51.7 (-2.4)
35.0 (42.5)
Ours (inter-parameter)
57.3 (+3.2)
38.0 (+5.5)
Complementary parameter scaling effectively compensates for performance drop. It elucidated in See. 3.3 that we construct coefficients with influence interwoven between parameters. To figure out its effectiveness, we replace it with different scaling strategies. Concretely, we decouple the interaction between the two modules, employing coefficients from A, B, individually. We additionally conduct scaling for A and B simultaneously and report quantitative results in Tab. 7. The study certifies the benefit of scaling coefficients from complementary parameter adaptation, and adaptively adjusting coefficients of B effectively promotes performance, which is in accordance with the analysis above. also demonstrated that the performance does not necessarily become better by utilizing more complex scaling coefficients (2.4% decrease in seen tasks).

---

Taak Anthmmtic
0.5
Pronebbccle (Dars
14
PPrussdcaa (un))
0.3
0.8
0.1
0.2
Similarity of largest singular vector
Average similarity of remaining singular vectors
Ratio of singular value
Average reti of remaining singular values
Figure 7: Similarity of singular vector and ratio of singular value on different merging techniques.
Cross-task normalization provides more stable performance. As is illustrated in Sec. 3.3, cross- task normalization provides not only consistent and stable performance on seen tasks but also advanced promotion on unseen tasks. We analyze the correlation between performance and variance in Fig. 5c. Concretely, compared with existing methods, approach achieves 3.4% better perfor- mance (57.3% v.s. 53.9%) with significantly smaller variance (1.2%). Notably, employing cross-task normalization strengthens the advantage. Specifically, it improves 1.2% average performance while reducing 0.2% on standard deviation, showcasing the superiority of the proposed method.
Further analysis of direction robustness. We formulate the evaluation metrics for measuring direction robustness to better understand parameter-efficient model merging. We use the average similarity of each corresponding singular vector between task-specific models and merged models as the criterion, which reflects the direction deviation in merging. Larger similarity means the direction possesses robustness and is not prone to changing direction during merging, thereby maintaining the performance of a specific task. Moreover, we also incorporate the ratio of singular value between merged and original models comprehensively reflect the degree of specific knowledge learning. As depicted in Fig. 3, the largest singular value displays more direction robustness, so we divide the value into the largest and the average of the remaining parts in Fig. 7 for better illustration of different model merging approaches.
It can be concluded that: (1) During merging, the largest vector tends to be stable, while remaining vectors are extremely dissimilar (direction instability), which leads to a performance drop evalua- tion. By contrast, promoting merging performance; (2) The results of the ratio of value also reflect that for a specific model, existing methods would decrease the largest singular value and fail to sufficiently strengthen smaller singular values. By contrast, our method better enhances smaller values and maintains task-specific knowledge during merging, which is consistent with view that scaling smaller values contributes to direction robustness. Furthermore, these analyses also give a clear explanation about the function of each component in the proposed method: (1) Prune to resolve the interference between tasks while exhibiting the least influence on the direction, and the sparsification also boosts the robustness of small values; (2) Scaling after prune aims compensate for the singular value drop raised by pruning, thereby enhancing the direction robustness.
method substantially improves the similarity of remaining vectors, strongly
5
Conclusion
This paper focuses on parameter-efin eeffiiitr-efffcennt model merging for large foundation models. We analyze from low-rank decomposition and reveal that direction robustness is crucial for merging efficient modules. We furthermore uncover that scaling tail singular values can effectively mitigate task interference and maintain direction robustness. Therefore, we introduce RobustMerge, an effective merging technique to maintain directions in low-rank space. We conduct extensive experiments and comprehensive analyses showcase the superiority and scalability of the approach. This is the first attempt at parameter-efficicet model merging from the perspective of direction robustness, and we hope it can inspire more advanced parameter-efffiiient merging methods.
Limitations and future work. We do not validate the method on more structures and tasks. However, since our method is a model-agnostic and task aagk-agnstiic post- processing algorithm, this will not be bottleneck given numerous models on platforms like Huggingface. Also, we propose the concept of direction robustness in parameter-efficient merging, but we do not design specific algorithm on decomposed matrices for the purposes of efficiency. We believe they would be promising directions that are left for future development in various downstream areas of parfmeter-efffiient learning.
10

---

Acknowledgments and Disclosure of Funding
This work partially supported by the Fundamental Research Funds for the Central Universi- ties, Peking University, and the InnoHK program. Li Shen is supported by the NSFC Grant (No. 62576364), Shenzhen Basic Research Project (Natural Science Foundation) Basic Research Key Project (No. ECX120024122212443004
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
[2] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 165510):1855-183,, 2017.
[3] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[4] Mostafa Dehghani. Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480 7512. PMLR, 2023.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[6] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
[7] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904 6913, 2017.
[8] Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu- Yao Zhang, and Cheng- Lin Liu. Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), page 13572-13586, 2025.
[9] Haiyang Guo, Fanhu Zeng, Fei Zhu, Wenzhuo Liu, Da-Han Wang, Jian Xu, Yao Zhang, and Cheng-Lin Liu. Federated continual instruction tuning. arXiv preprint arXiv:2503. 022.03...88991 12897, 2025.
[10] Haiyang Guo, Fei Zhu, Fanhu Zeng, Bing Liu, and Yao Zhang. Desire: Dynamic knowledge consolidation for rehearsal-free continual learning. arXiv preprint urXiv:2411.19154. 2024.
[11] DU Guodong, Junlin Lee, Jing Li, Runhua Jiang. Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, al. Parameter competition balancing for model merging. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[12] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 2018
[13] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine- fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608 2024.
11

---

[14] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):21172226, 2019.
[15] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributinn generalization. ICCV, 2021.
[16] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[17] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307. 13269, 2023.
[18] Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-menging: Tuning-free free high- begh-performance model merging. In The eiirth- Annual Conference Neural Information Processing Systems, 2024.
[19] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hanmanch Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
[20] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision, pages 709-727. Springer, 2022.
[21] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro. and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference Learning Representations, 2023.
[22] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[23] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-moda prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[24] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[25] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nnistl/ 1998.
[26] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730 19742. PMLR, 2023.
[27] Lu Li, Tianyu Zhang, Zhiqi Bu, Suyuchen Wang, Huan He, Jie Fu, Yonghui Wu, Jiang Bian, Yong Chen, and Yoshua Bengio. Map: Low-compute model merging with amortized pareto fronts via quadratic approximation. arXiv preprint arXiv:2406. 07529, 2024.
[28] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292-305, 2023.
[29] Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang, Zecheng Wang, Fangying Rong Qingbin Liu, Yanchao Hao, Bo Li, Xi Chen, et al. Pruning via merging: Compressing lmms via manifold alignment based layer merging. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17817 2024.
12

---

[30] Deyuan Liu, Zecheng Wang, Bingning Wang, Weipeng Chen, Chunshan Li, Zhiying Tu, Dianhui Chu, Bo Li, and Dianbo Sui. Checkpoint merging via bayesian optimization in Ilm pretraining. arXiv preprini arXiv:2403. 19390, 2024.
[31] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-cfficient fine-tuning better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
[32] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[33] Jun Liu, Zhenglun Kong, Peiyan Dong, Xuan Shen, Pu Zhao, Hao Tang, Geng Yuan, Wei Niu, Wenbin Zhang, Xue Lin, al. Rora: Efficient fine-tuning of IIm with reliability optimization for rank adaptation. In ICASSP, 2025.
[34] Shih- Yang Liu, Chien-Yi Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
[35] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216-233. Springer, 2025.
[36] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
[37] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song- Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In The Eleventh International Conference on Learning Representations, 2023.
[38] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chus Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021.
[39] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11-20, 2016.
[40] Michael S Matena and Colin A Raffel. Merging models with fisher-weighte averaging. Advances in Neural Information Processing Systems, 15:17703-1 2022.
[41] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37:121038-121072. 2024.
[42] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947-952. IEEE, 2019.
[43] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 201 1, page 4. Granada, 2011.
[44] Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. IJC,, 123(1):74 2017.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
13

---

[46] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 1lya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9 2019.
[47] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey. et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[48] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146-162. Springer, 2022.
[49] Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. In European Conference on Computer Vision, pages 422 238.4 Springer, 2024.
[50] Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian lgel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 201 international joint conference on neural networks, pages 1453-146) IEEE, 2011.
[51] George Stoica, Daniel Bolya, Jakob Brandt Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. In The Twelfth International Conference on Learning Representations, 2024.
[52] Yi-Lin Sung. Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. An empirical study of multimodal model merging. In Findings of the Association for Computational Linguisties: EMNLP 2023, pages 1563-1575, 2023.
[53] Anke Tang. Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. In Forty-first International Conference on Machine Learning, 2024.
[54] Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization. In The Twelfth International Conference Learning Representations, 2024.
[55] Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng zhong Xu. HydraloRA: An asymmetric loRA architecture for efficient fine-tuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[56] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2rXiv:2302.13991 2023.
[57] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In The Twelfih International Conference on Learning Representations, 2024.
[58] Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning. The 34th Annual ACM Symposium on User Interface Software and Technology, pages 498-510, 2021.
[59] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing laniunnl model's perception of the world at any resolution. arXiv preprint 2rXiv:2409.12191, 2024.
[60] Taiqiang Wu, Jiahao Wang. Zhe Zhao, and Ngai Wong. Mixture-of-suuspaac in low-rank adaptation. arXiv preprint arXiv:2406.1 /909, 2024.
[61] Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 119:3-22, 2016.
14

---

[62] Prateck Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
[63] Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in Ilms, milms, and beyond: Methods, theories, applications and opportu- 2024.
nities. arXiv preprint arXiv:2408.0768
[64] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference Learning Representations, 2024.
[65] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024.
[66] Fanhu Zeng. Zhen Cheng, Fei Zhu, Hongxin Wei, and Xu- Yao Zhang. Local-prompt: Extensible local prompts for few-shot out-of-distribution detection. In The Thirteenth International Conference on Learning Representations, 2025.
[67] Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu-Yao Zhang, and Cheng-Lin Liu. Modalprompt: Dual- modality guided prompt for continual learning of large multimodal models. arXiv preprint arXiv:2410.05849, 2024.
[68] Fei Zhu and Zhaoxiang Zhang. Trustlora: Low-rank adaptation for failure detection under out-of-distribution data, arXiv preprint 4rXiv:2504.14545, 2025.
[69] Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry low-rank adapters foundation models. Forty-first International Conference on Machine Learning, 2024.
15

---

Appendix
A
Theoretical Analysis of Singular Value Decomposition in Merging
As we focus on the merging of low-rank matrices, we first introduce basic notations of singular value decomposition and then describe its application in merging.
A.1
Background of Singular Value Decomposition
Denote the parameter-efficient module W x A, W E Ruxn and W as low-rank matrix, ie.,
Rank(W) = < n. The singular values of the matrix W are 2 02 2 ... 2 >> 0.
Based singular value decomposition, the matrix W be decomposed as:
W UΣV*
(6)
where U [uq ug. Un E Raxr and V [1 U2 ... Ur] R are orthogonal matrices with left and right normalized singular vectors, respectively, and Σ E Rxx is a diagonal matrix containing singular values of the original matrix, which can formulated as:
Φ1
02
Σ
Tr
The low-rank matrix therefore be rewritten
W =1 U20222 + U,
(8)
=i + 020222 + .... U,
A.2 Theoretical Analysis in Merging
For better illustration, we consider merging in simplified situation and take two parameter-efficient modules an example.
Let W1, W2 be two decomposed modules that are fine-tuned on task A and B, respectively:
W, =amviiin + 021 T 0,14
(9)
W2 10 + 022 .... 0,20
where oij, Uty and Uty represent the tt singular value/left singular vector/right singular vector the jth low-rank matrix, respectively.
Given the practical significance of singular vectors in LoRA, consider two permutations to r, Le., (T), (2), (r), and (1), (2),..... (r), merging the two modules therefore be rewritten
W =><x + W2)
=M + "2 + "ym)
to D2 + t + + o 222
(10)
-Mloon" D 22 (2) on + 2022)22"))))) +...
((m)))))) " + 20
where each pairwise subscript (0) (i) 1, 1,... r stands for similar singular components, i.e.. similar knowledge of two different matrices in low-rank space.
Empirically, U contains more general knowledge in low-rank space with larger similarity across tasks. Consequently, the merging process can mathematically be expressed as merging each of the task-specific vectors in low-rank space:
Ao (0)1 101 + Ao AaEa + Avefee i= 1,...,. .r
(11)
16

---

Given that U, V are normalized, it can be inferred from Eqn. 11 that merging in low-rank space can be seen as vector addition for each group of task specific singular vector, with singular vector & indicating the direction and singular value o; indicating the magnitude. Based on vector synthesis, direction change for task A and task B would be complementary to each other for each singular value.
It can be seen from the above derivation that due to the difference in original direction and magnitude, the singular values with larger magnitude are more likely determine the direction and magnitude of the merged singular vector. As result, the change in singular vector angle will vary from the perspective of singular value vectors belonging to different tasks due to the stark difference in singular values, e.g., for task A, the direction angle change for vector 1 is small while the angle change for vector 2 is relatively large, and the situation is just the opposite for task B. Therefore, the key for merging would be maintaining direction robustness for vectors with small singular values. Without the loss generality, the derivation can be extended to merging any number of models.
B Distribution of Singular Value in Different Layers
We depict the distribution of attn. v in Layer 1, 18, and 32 to show the distribution of singular value changes with different layers. It clearly shows in Fig. 8 that: (I) As the position of the layer becomes higher, the maximum singular value becomes larger, and the tail singular values become smaller, making the distribution more stark. This shares a similar observation with HiDe-LLaVA [8] that lower layers carry more general knowledge and higher layers contain more task-specific knowledge, so in the first layers, the gap between top and tail values would not be as large as in the last layers. Therefore, the distribution becomes more stark with increased layer, highlighting the necessity to properly handle direction instability during merging: (2) Moreover, compared with the original model, our method successfully and consistently scales both top and tail singular values across different layers, thereby contributing to robust and efficient merging with improved performance, which strongly demonstrates the effectiveness and rationality of the method.
Layer
Layer 18
Layer 32
0.4
Original
04
Original
0.56
Orgirnl
Magnitule
Ours
Ours
Oun
0.3
0.3
0.22
Index
Index
Figure 8: Singular value distribution of original and our model in attn. v of layer 18 and 32.
C More Implementation Details
We build multimodal codebase for multimodal tasks and vision tasks upon LLaVA and CLIP 4, respectively. For training, we follow the standard training procedure described in LLaVA, i.e., training each task individually and obtaining parameter-efficier modules. LoRA added to linear layers in foundational blocks, and all models are trained for I epoch for merging.
For inference, the tasks evaluated by accuracy. The vision task is conducted by constructing textual prompts for each category and calculating the similarity. Each evaluated sample is classified into the category with the largest similarity. Hyper-parameter À is set to 2 by default. All merging experiments are carried out on a single NVIDIA A6000 with the temperature set to 0.
D Details of Comparison Methods
The crucial operation of merging method involves designing a merging algorithm. i.e., 2(-) defined in the paper. We primarily compare with DARE, mies-merging, and Task Arithmetic in the main
hhrps://githhbb "hanps//gituub.co//attiia http://hhh...ccmmmaaaaiaiillllll "bapps//gittubbco//oppaaaaCCCCCt
17

---

results. For comparison to these traditional approaches, we employ traditional approaches to LoRA components. Specifically, we fine-tune the foundation model using LoRA, merge LoRA components using traditional approaches, and finally evaluate performance with merged LoRA attached to base models. Their ways of merging are briefly introduced in the following paragraph.
Task Arithmetic views all parameters as vectors. It obtains task vectors by subtracting pre-trained models from fine-tuned models and performs standard arithmetic operations like addition and subtraction on them. It sums the parameters from checkpoints of different tasks and constructs a strong baseline for multi-task learning.
mies-ng reduces the conflicts between parameters of different tasks by a trim, elect sign, and merging paradigm. Concretely, it first keeps parameters with the highest magnitudes, and then determines the aggregated sign based on the summation of remaining parameters. Finally, it merges the parameters with the same sign as the aggregated sign mitigate disagreements.
DARE empirically observes the sparsity in parameters. It randomly drops parameters with a fixed ratio p. and rescales the remaining parameters with 1/(1 - p) to match the expectation of parameters lost from dropping ones.
E
Details of Different Tasks
E.1
Composition of Instruction Tuning Datasets
The instruction tuning datasets follow the format of instruction tuning and are composed of image- text pairs and additional instruction templates. Instruction templates provide a clear and ex- pressed task environment and purpose in natural language and are crucial for instruction tuning. The templates are shown in Appendix 8. In mul- timodal generative tasks, we carefully design the instruction template for each dataset. The templates are concatenated with task- task-specific in- puts of image and text to the model to generate responses in an auto-regressive way. The lan- guage model is set be trainable with the visual encoder frozen.
Table 8: Instruction templates for each dataset.
Dataset
Instruction
Answer with the option's letter from the given choices directly.
ScienceQA
What is the object in the image?
ImageNet
Answer the question using single word or phrase.
Answer the question using single word or phrase.
VQAv2
Please provide the bounding box coordinate of the region this sentence describes: edescription
Grounding
Answer the question using a single word or phrase.
IconQA
What happening in the image?
VizWiz
Generate a brief caption for the image.
E.2 Datasets of Vision Tasks
What is happening in the image?
Flickr30k
Generate a brief caption for the image.
All the vision tasks are traditional datasets con- taining common objects across wide domains like cars, texture, traffic signs, and age classification. Categories for them vary from 10 to 397. We fine-tune VLMs with LoRA each task and merge them employing different types of merging methods. Only the visual en- coder is trainable, and the text encoder remains frozen for label embedding extraction.
Answer the question using single
OCR-VQA
word or phrase.
for im-
Answer with the option's 's letter from
AOKVQA
given choices directly.
ImageNet-R
What is the object in the image?
You are given phone UI screen.
Screen2Word
Describe the screen in sentence.
Answer the question using single word or phrase.
TabMWP
F More Experimental Results
RobustMerge generalizes to the number of tasks. We gradually increase the number of tasks to substantiate the robustness of our method. As is illustrated in Fig. 9, in seen tasks, the performance undergoes slight drop as merging more models interferes with specific tasks. In unseen tasks, the performance first improves and then declines modestly. It could be attributed to the fact that in the first stage, seen tasks transfer knowledge and enhance unseen tasks of similar distribution; in the second stage, interference dominates merging rather than knowledge transformation. Under both circumstances, our method consistently outperforms existing approaches by a substantial margin as the task number varies, indicating the superiority and stability of our method.
18

---

Table 9: Results of merging models fine-tuned with DORA.
Method
SciQA
Image
VQA
REC
OCR
VizWiz
Flickr
IconQA
Average
Task Arithmetic
69.91
67.45
66.18
41.43
58.57
46.60
52.68
40.57
55.42
Ties-merging
69.01
64.06
66.60
40.68
61.94
46.51
51.97
35.82
S4.57
RobustMerge
70.95
68.25
66.48
41.67
58.39
46.72
52.78
43.40
56.08
Seen Tasks
Unseen Tasks
72
Task Arithmetic
Arthmert
Average Accuracy (%6
Ties
64
Average Accuracy
Ours
28
48
40
Number of tasks
Number of tasks
Figure 9: Average performance of seen and unseen tasks when number of tasks increases. Our method consistently outperforms TA and Ties with significant improvement.
RobustMerge extends well to different PEFT methods. We primarily test our method on LoRA since it is the most commonly used and comparable PEFT technique. To demonstrate the extensibility of the proposed method, we additionally conduct experiments on DoRA [34], which is a LoRA-base efficient technique with advanced algorithm to improve the performance of PEFT learning. Results shown in Tab. reveal that our method achieves consistent and substantial improvements against existing merging methods in different PEFT methods. strongly certifies that the problem of direction robustness is widespread in merging different kinds PEFT modules, where attention primarily paid improving the performance of single task. By contrast, our method handles the issue some extent, thereby promoting the multi-task performance when the PEFT technique varies.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers,
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See limitations in Conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximation: only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms
and how they scale with dataset size.
If applicable, authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
See in Appendix.
Guidelines:
20

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See in Sec. 4 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We released the code, dataset and model.
Guidelines:
21

---

The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.ce
mublic/guides SoweieesssSSSSemiiiiiiiiii.. cassssinnolly for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wttpcccpppcccccccccccccccccccllt /gcii/gusss....ee. /CodeSubmies wwwtubimsssoooo aesioPoolcyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See in Sec. 4 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: See in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
22

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
See in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethies
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : /nwrriserr...//
Question:
Answer: [Yes]
Justification:
The authors have reviewed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: No societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
23

---

The answer NA means that the paper poses such risks. If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We correctly cite the original paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
24

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or research with human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing or research with human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: No LLMs used in core method development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy (httpa: http///eenn..p/eeee htt///uips.c//ooffeeeeeeeeeeeeee wttpa/ceeee//////2/e// for what should or should not be described.
25


Reviews:
[{"review_id": "hotCZkjQUp", "reviewer": "Reviewer_f2pm", "summary": "In this paper authors look at the problem of merging parameter-efficient fine-tuned (PEFT) models. Typically, model merging approaches fail under PEFT settings. Authors claim the reason of poor performance as direction instability along directions of small singular values. Authors propose to resolve this by 1) pruning smallest magnitude weights and  2) renormalizing the PEFT A matrix. \nAuthors empirically show that this procedure leads to improvement in the model merging performance.", "questions": "- It’ll be helpful to resolve the weaknesses in the paper", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper looks at an important problem merging PEFT models and proposes a novel methodology (CoPA merging) for solving the problem. \n- Thorough empirical evaluation on VLMs with detailed ablation studies.\n- Results in Table 4 and 5 are promising and CoPA merging leads to improvement in model merging\n\nWeaknesses:\n- Even though authors identify directional robustness as a critical property for achieving model merging for PEFT models, there is no formal definition and they lack evaluation of when one can identify if PEFT models are robust or not. It needs to be formalized\n- I can’t seem to understand how pruning and scaling lead to directional robustness so justification is not entirely clear to me. I think this is related to not having a proper definition established in the paper.", "was_revised": true, "final_justification": "I thank the authors for clarification in the rebuttal and the proposed metrics for measuring directional robustness. Given the novelty in addressing PEFT model merging, and the promising results across multiple benchmarks, I lean toward acceptance.", "rebuttal": "Dear Reviewer f2pm:\n\nWe sincerely thank you for your time and effort in reviewing our paper with several insightful suggestions, and we will respond to each question respectively.\n### **Q1 Evaluation of robustness of PEFT models**\nThank you for your constructive suggestion. We totally agree that formulating the robustness of specific model contributes to better understanding of the effectiveness of merging method. To better illustrate this part, we use the **average similarity of each corresponding singular vector** between task-specific models and merged models as the criterion, which reflects the direction deviation in merging. Larger similarity means the direction possesses more robustness and is not prone to change direction during merging, thereby maintaining performance of specific task. Moreover, we also incorporate the **ratio of singular value** between merged and original models to comprehensively reflect the degree of specific knowledge learning.\n\nEmpirically taking merging ScienceQA and ImageNet in Fig.3b as an example, we present similarity of singular vector and ratio of singular value between task-specific model and merged model and compare two merging methods. As is analyzed in the paper (Fig.3a and Fig.3b), the largest singular value displays more direction robustness, so we divide the value into largest and average of remaining parts for better illustration. Results are shown below.\n\n|Method|Similarity of largest singular vector $\\uparrow$ | Average similarity of remaining singular vectors $\\uparrow$|Ratio of largest singular value $\\uparrow$| Average ratio of remaining singular values $\\uparrow$\n|-|-|-|-|-\n|Task Arithmetic|0.63|0.16|0.6|1.3\n|Ours|**0.67**|**0.27**|**1.3**|**1.8**\n\nIt can be concluded that **(1)** during merging, **the largest vector tends to be stable, while remaining vectors are extremely dissimilar** (direction instability), which leads to performance drop in evaluation. By contrast, our method substantially improves the similarity of remaining vectors compared to Task Arithmetic (**x1.7**), strongly promoting merging performance (line144-146); **(2)** Moreover, ratio of value also reflects that compared to specific model, existing method would decrease the largest singular value and fail to sufficiently strengthen smaller singular values. By contrast, our method **better enhances smaller values and maintains knowledge during merging**, which is in consistent with the paper that scaling smaller values contributes to direction robustness (line153-157).\n\n### **Q2: Intuitive contribution of each component**\nFrom the definition established above, we conduct ablation study about the similarity to certificate the effectiveness of the proposed pruning and scaling, and the results are shown below.\n\n|Method|Similarity of largest singular vector $\\uparrow$ | Average similarity of remaining singular vectors $\\uparrow$|Ratio of largest singular value $\\uparrow$| Average ratio of remaining singular values $\\uparrow$\n|-|-|-|-|-\n|Task Arithmetic|0.63|0.16|0.6|1.3\n|Prune|0.62|0.20|0.5|1.2\n|Prune&Scale (Ours)|**0.67**|**0.27**|**1.3**|**1.8**\n\nThese analyses give a more clear explanation about the function of each component: **(1)** Prune is to **resolove the interference between tasks while exhibiting the least influence on the direction** (line189-192), and the sparsification also boosts the robustness of small values (two opposite conflicting parameters would be mitigated by setting one value to zero); **(2)** Equipping scale after prune aims to **compensate for singular value drop raised by pruning** (line 201-203), thereby enhancing the direction robustness, which can be reflected by both similarity and ratio increase.\n\nQuantitative experiments are as follows (Tab.6 in paper) to intuitively display performance gain from the two techniques.\n\n|Component|SciQA|ImgNet|VQAv2|Grounding|OCR|Viz|Flickr30k|IconQA|Avg\n|-|-|-|-|-|-|-|-|-|-\n|Task Arithmetic|71.94|57.49|67.06|38.90|62.87|44.80|49.20|39.21|53.93\n|Prune|73.03|64.18|**67.50**|43.12|58.19|46.36|52.24|44.54|56.14(+2.21)\n|Prune&Scale (Ours)|**73.43**|**65.54**|67.20|**44.80**|**62.97**|**46.61**|**52.80**|**45.90**|**57.33(+3.40)**\n\n\nDue to the constraint of the policy, we are only able to present table instead of better-illustrated figures. We will add the additional analysis properly and detailedly into updated manuscripts to strengthen our method. \n\nWe thank again for your insightful and professional comments. We hope the additional definition and analysis can help you have a better understanding of how our method contributes to direction robustness. If you have any remaining question that needs to be further clarified, do not hesitate to respond to us and we will try our best to address all the concerns!"}, {"review_id": "MJL56FRYn2", "reviewer": "Reviewer_Rk6Q", "summary": "This paper focuses on the challenge of merging multiple task-specific parameter-efficient models into a single high-performance and generalizable multimodal large language model. The authors introduce CoPA-Merging, a training-free merging framework that maintains direction robustness in low-rank space through adaptively scaling singular values. Comprehensive experiments demonstrate CoPA-Merging’s effectiveness in maintaining multi-task performance across seen and unseen domains.", "questions": "Please refer to Weaknesses.", "limitations": "Limitations are discussed in Appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. Efficient merging is an important topic for current large-scale pre-trained models.\n\n2. The analysis from the perspective of direction robustness of singular values is interesting and reasonable for multi-task merging.\n\n3. Comprehensive experiments well demonstrate the effectiveness of CoPA-Merging.\n\n**Weaknesses:**\n\nI think the motivation is sufficient and the experiments well support the claims, so I only have several minor suggestions that may further strengthen the paper in my mind.\n\n1.  More PEFT methods. The exploration mainly focuses on LoRA-based mechanisms. How does it work on other PEFT methods, such as P-Tuning [1] or LLaMA-Adapter [2]?\n\n2. More MLLMs. Only LLaVA is employed in the paper. How does CoPA-Merging perform on other MLLMs, such as Qwen2-VL [3]?\n\n3. More analysis or visualization on task relationships. The paper mentions that the small singular values of one task may be altered by other tasks, reducing the performance. Is there experimental evidence indicating this phenomenon, and are there direct visualizations or analyses showing how CoPA-Merging mitigates it?\n\n***References:***\n\n[1] Xiao Liu et al., “P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks”, ACL, 2022.\n\n[2] LLaMA-Adapter-V2 Multi-modal, https://github.com/OpenGVLab/LLaMA-Adapter/tree/main/llama_adapter_v2_multimodal7b, 2024.\n\n[3] Peng Wang et al., “Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution”, arXiv, 2024.", "was_revised": true, "final_justification": "After the author rebuttal, all of my concerns have been addressed. I think the paper provides a novel insight for keeping the robustness while merging model. The technique is reasonable and the experiments are comprehensive. I therefore suggest accept.", "rebuttal": "Dear Reviewer Rk6Q:\n\nThank you for your professional comments about our paper and recognition about the analysis from direction robustness! We will answer each question respectively.\n\n### **Q1. More PEFT methods**\nOur motivation is built upon matrix analysis of singular values and design specific techniques to resolve interference of direction robustness when merging LoRA components, which is based on the consideration that LoRA has been the most popular and effective parameter-efficient tuning technique across various fields. Therefore, it can not be directly applied to prompt-based methods. In this part, we do not know any existing work about merging prompts to the best of our knowledge. Nevertheless, we respectully agree with your suggestion that validating the extensibility across different PEFT methods is beneficial to further demonstrate the effectiveness of the method. Therefore, we give some additional results on other PEFT techniques like PISAA[1], DoRA[2], which are LoRA-based efficient modules with advanced techniques to improve the effectiveness of PEFT learning, to demonstrate the extensibility of the proposed method. Detailed results and analyses are shown below.\n\n\n|Strategy|SciQA|ImgNet|VQAv2|Grounding|OCR|IconQA|Avg\n|-|-|-|-|-|-|-|-\n|LoRA+Task Arithmetic|71.92|59.78|66.31|40.51|63.54|42.10|57.36\n|+Ours|74.35|69.22|66.79|46.70|64.17|49.74|**61.82(+4.46)**\n|DoRA+Task Arithmetic|72.21|61.27|67.10|38.29|64.02|44.31|57.86\n|+Ours|74.89|69.40|67.14|44.24|64.85|52.26|**62.29(+4.26)**\n|PISSA+Task Arithmetic|72.36|60.89|67.43|39.11|64.51|45.99|58.38\n|+Ours|75.01|69.56|67.60|44.79|65.09|52.42|**62.57(+4.03)**\n\nIt can be concluded from the results that **(1)** more advanced PEFT technique is beneficial to better performance of model merging, and exhibits overall better performance; **(2)** our method achieves consistent and substantial improvements against existing merging methods across all PEFT methods. It strongly certificates that **the problem of direction robustness widespread exists in merging all kinds of PEFT modules**, where attention is primarily paid to improving the performance of single task. By contrast, our method handles the issue to some extent, thereby promoting the performance when the PEFT technique varies.\n\nWe firmly agree with your opinion and will treat exploration to more types of parameter efficient tuning merging as potential future direction. We will cite and add discussion about this in the updated manuscript.\n\n[1] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. NeurIPS 2024.\n\n[2] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024.\n\n### **Q2. More experiments on other MLLMs**\nThank you for your constructive suggestions. We firmly agree with you that validating on other MLLMs is crucial for the extensibility of the proposed method. Following your suggestion, we conduct additional experiments on **Qwen-VL** and the results are shown below.\n\n|Method|SciQA|ImgNet|VQAv2|Grounding|OCR|Viz|Flickr30k|IconQA|Avg\n|-|-|-|-|-|-|-|-|-|-\n|Task Arithmetic|66.02|54.20|67.74|29.76|64.77|47.10|49.01|38.57|52.14\n|Ties|66.27|54.06|67.02|30.31|**66.90**|46.81|46.65|33.15|51.39\n|Ours|**69.45**|**63.91**|**69.02**|**35.61**|65.04|**49.28**|**51.89**|**43.06**|**55.90(+3.76)**\n\nIt is indicated that compared with existing merging strategies, our method also achieves significant improvement on Qwen-VL (**+3.76%**), strongly demonstrating its effectiveness across different architectures.\n\n### **Q3. More analysis on task relationships**\nWe totally agree with you that demonstrating the phenomenon of task interference would further strengthen the effectiveness of the method. To this end, would explain it from three aspects.\n\n**1)** We calculate the **average similarity of each corresponding singular vector** between task-specific models and merged models to reflect the direction deviation in merging. Larger similarity means the direction possesses more robustness and is not prone to change direction during merging, thereby maintaining performance of specific task. Moreover, we also incorporate **ratio of singular value** between merged and original models to comprehensively reflect the degree of specific knowledge learning.\n\nEmpirically taking merging ScienceQA and ImageNet in Fig.3b as an example, we present similarity of singular vector and ratio of singular value between task-specific and merged model. As is analyzed in the paper (Fig.3a and Fig.3b), the largest singular value displays more direction robustness, so we divide the value into largest and average of remaining parts for better illustration. Results are shown below.\n\n|Method|Similarity of largest singular vector $\\uparrow$ | Average similarity of remaining singular vectors $\\uparrow$|Ratio of largest singular value $\\uparrow$| Average ratio of remaining singular values $\\uparrow$\n|-|-|-|-|-\n|Task Arith|0.63|0.16|0.6|1.3\n|Ours|**0.67**|**0.27**|**1.3**|**1.8**\n\nIt can be concluded that **(1)** during merging, **the largest vector tends to be stable, while remaining vectors are extremely dissimilar** (direction instability), which leads to performance drop in evaluation. By contrast, our method substantially improves the similarity of remaining vectors (**x1.7**), strongly promoting merging performance (line144-146); **(2)** Moreover, ratio of value also reflects that compared to specific model, existing method would decrease the largest singular value and fail to sufficiently strengthen smaller singular values. By contrast, our method better **enhances smaller values and maintains specific knowledge during merging**, which is in consistent with the paper that scaling smaller values contributes to direction robustness (line153-157).\n\n**2)** From the definition established above, we additionally conduct ablation study about the similarity to certificate the effectiveness of the proposed pruning and scaling, and the results are shown below.\n\n|Method|Similarity of largest singular vector $\\uparrow$ | Average similarity of remaining singular vectors $\\uparrow$|Ratio of largest singular value $\\uparrow$| Average ratio of remaining singular values $\\uparrow$\n|-|-|-|-|-\n|Task Arithmetic|0.63|0.16|0.6|1.3\n|Prune|0.62|0.20|0.5|1.2\n|Prune&Scale (Ours)|**0.67**|**0.27**|**1.3**|**1.8**\n\nThese analyses give a more clear explanation about the function of each component and how our method mitigates the problem of direction instability: (1) Prune is to **resolove the interference between tasks while exhibiting the least influence on the direction** (line189-192), and the sparsification also boosts the robustness of small values (two opposite conflicting parameters would be mitigated by setting one value to zero); (2) Equipping scale after prune aims to **compensate for singular value drop raised by pruning** (line 201-203), thereby enhancing the direction robustness, which can be reflected by both similarity and ratio increase.\n\nQuantitative experiments are as follows (Tab.6 in paper) to intuitively display performance gain from the two techniques.\n\n|Component|SciQA|ImgNet|VQAv2|Grounding|OCR|Viz|Flickr30k|IconQA|Avg\n|-|-|-|-|-|-|-|-|-|-\n|Task Arithmetic|71.94|57.49|67.06|38.90|62.87|44.80|49.20|39.21|53.93\n|Prune|73.03|64.18|**67.50**|43.12|58.19|46.36|52.24|44.54|56.14(+2.21)\n|Prune&Scale (Ours)|**73.43**|**65.54**|67.20|**44.80**|**62.97**|**46.61**|**52.80**|**45.90**|**57.33(+3.40)**\n\n**3)** We furthermore give an analysis from the perspective of distribution of singular values in different layers in **Q3 of Reviewer 5p9X** to demonstrate the effectiveness of our method. The summarized conclusion is that **higher layers exhibit more stark distributions**, which means larger gap between task-specific knowledge with more direction variation, highlighting the need for handling the issue of direction instability, and our method designed for direction robustness well mitigates the problem.\n\nUnfortunately, we are not able to present a picture to display it due to the constraint of the policy. We show qualitative results above and we will add extensive analyses detailedly into the updated manuscript to strengthen our method.\n\nWe express our respectful gratitude to the reviewer for your time and effort, and recognition of our paper. We hope the various additional experiments could strengthen our paper and we will revise the paper according to your valuable feedback. If you have any remaining questions or think some experiments are insufficient, feel free to respond to us at your convenience and we will do our best to address all the concerns!"}, {"review_id": "9NjMQxm4yl", "reviewer": "Reviewer_5p9X", "summary": "This paper introduces the method of CoPA-Merging, a training-free method for the merging of PEFT modules in multimodal large language models. The innovation of the method lies in addressing direction robustness. This happens due to large differences of singular values between LoRA modules. This is addressed through pruning and complementary parameter scaling to maintain proper stability of direction. It also does cross-task normalization to improve generalization to unseen tasks.", "questions": "How would the distribution of singular values change with different layers ?\n\nHow would the performance change if the number of merging models/loras change ?\n\nHow would the performance change if the merging was done differently for different layers ?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "---- Strengths ----\n\nThe concept of directional robustness is very new from the viewpoint of low rank adaptation merging.\n\nThe great thing is that is training-free and is efficient and it does not require additional training, storage etc.\n\nIt shows strong empirical results and outperforms baselines like Task Arithmetic, DARE and Ties-Merging.\n\nThere has been thorough evaluation on seen and unseen tasks.\n\nAnalysis and Mathematical rigor is well defined in the form of variation in singular values.\n\n---- Weakness ----\n\nThe method has only been carried on LoRA and not other PEFT tasks.\n\nThe method has only been tried on multimodal tasks. It has not been applied to image generation tasks.", "was_revised": true, "final_justification": "I think the authors have addressed my comments and I will increase the score.", "rebuttal": "Dear Reviewer 5p9X:\n\nThank you for your professional suggestions and we greatly appreciate your positive comments about our analysis and empirical results. We will respond to each question respectively. \n\n### **Q1. More PEFT methods**\nThank you for your valuable advice. Evaluating the method on different PEFT modules is of great significance to demonstrate the extensibility of the proposed method. We primarily test our method on LoRA since it is the most commonly used and comparable PEFT techniques. Following your suggestions, we additionally conduct experiments DoRA[1] and PISSA[2], which are two LoRA-based efficient techniques with advanced algorithms to improve the performance of PEFT learning, to demonstrate the extensibility of the proposed method. Detailed results and analyses are shown below.\n\n|Strategy|SciQA|ImgNet|VQAv2|Grounding|OCR|IconQA|Avg\n|-|-|-|-|-|-|-|-\n|LoRA+Task Arithmetic|71.92|59.78|66.31|40.51|63.54|42.10|57.36\n|+Ours|74.35|69.22|66.79|46.70|64.17|49.74|**61.82(+4.46)**\n|DoRA+Task Arithmetic|72.21|61.27|67.10|38.29|64.02|44.31|57.86\n|+Ours|74.89|69.40|67.14|44.24|64.85|52.26|**62.29(+4.26)**\n|PISSA+Task Arithmetic|72.36|60.89|67.43|39.11|64.51|45.99|58.38\n|+Ours|75.01|69.56|67.60|44.79|65.09|52.42|**62.57(+4.03)**\n\nIt can be concluded from the results that **(1)** more advanced PEFT technique is beneficial to better performance of model merging, and exhibits overall better performance; **(2)** our method achieves consistent and substantial improvements against existing merging methods across all PEFT methods. It strongly certificates that **the problem of direction robustness widespread exists in merging all kinds of PEFT modules**, where attention is primarily paid to improving the performance of single task. By contrast, our method handles the issue to some extent, thereby promoting the performance when the PEFT technique varies.\n\n[1] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024.\n\n[2] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. NeurIPS 2024.\n\n### **Q2. More task types**\nWe truly appreciate the reviewer for the professional suggestion. Due to the large gap between image generation and understanding task, we do not find existing merging methods that conduct additional experiments on image generation task. To this end, we conduct experiments on vision tasks with VLM model to partially demonstrate the extensibility of our approach. With respect to image generation task, the most related work would be ZipLoRA[3], which focuses on style transfer with merging strategies. It trains a content LoRA and a style LoRA and then merges them to get one LoRA with both style and content knowledge. However, it **requires post-training merged LoRAs with specific design**. \n\nAs the reviewer suggests, we conduct experiments on style transfer to validate the extensibility of the proposed method in image generation task. Following the setting from ZipLoRA[3], we train a content LoRA and a style LoRA based on SDXL. We calculate the feature similarity of subject, style and text between specific and merged models to reflect the alignment and fidelity of the merged model. Results are shown below.\n\n|Method|Style-alignment $\\uparrow$|Subject-alignment $\\uparrow$|Text-alignment $\\uparrow$\n|-|-|-|-\n|Direct Merge|0.472|0.271|0.213\n|Task Arithmetic|0.474|0.269|0.233\n|Ties|0.481|0.274|0.231\n|Ours|0.484|0.288|0.240\n \nDue to the time limitation and merging without post-training, the model might not be that converged with overall smaller similarity in value, but the advantage of our method can still be confirmed that our method outperforms existing merging techniques in evaluation of all style, subject and text alignment, thereby showing the extensibility of the proposed merging method.\n\n[3] ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs. ECCV 2024.\n\n### **Q3. Singular value distribution of different layers**\n\nWe take merging ScienceQA and ImgetNet in Fig.3 as an example to show the distribution of singular value changes with different layers. Due to the constraint of the policy, we are only able to present the results in table, so we present average singular value first layer, middle layer and last layer to represent the overall change in distribution of singular values. We select the maximum, medium and minimum of singular value to show the distribution. Results are shown in table below.\n\n|Layers|First|Middle|Last|\n|-|-|-|-\nOriginal\n|Maximum singular value|0.1177|0.1645|0.2772\n|Minimum singular value|0.0183|0.0173|0.0142\n|Medium singular value|0.0328|0.0272|0.0179\nOurs\n|Maximum singular value|0.2102|0.2841|0.4842\n|Minimum singular value|0.0284|0.0262|0.0205\n|Medium singular value|0.0426|0.0311|0.0244\n\nFrom the table above, it can be concluded that:\n\n**(1)** as the **position of the layer becomes higher, the maximum singular value becomes larger, and the tail singular values become smaller** (reflected by both smaller minimum and medium singular values), making the distribution more stark. This shares similar observation with HiDe-LLaVA[4] that lower layers carry more general knowledge and higher layers contain more task-specific knowledge, so in the first layers, gap between top and tail values would not as large as the last layers (This could also be validated by the results of merging different layers in **Q5** that naive merging of higher layers leads to more performance drop). Therefore, the distribution becomes more stark with increased layer, highlighting the necessity to properly handle direction instability during merging.\n\n**(2)** Moreover, comparing our model with original model, we can conclude that **our method successfully and consistently scales both top and tail singular values across different layers**, thereby contributing to robust efficient merging with improved performance, which strongly demonstrates the effectiveness and rationality of the proposed method.\n\nIt would be clearer to present the results in the form of figure, and we would properly display it in the updated manuscript.\n\n### **Q4. Different number of merging models**\nWe provide results of merging different number of models in Fig.A of Appendix. It can be concluded that **(1)** in seen tasks, the performance undergoes slight drop, which is in consistent with the intuition that merging more models interferes specific task more; **(2)** in unseen tasks, the performance first improves and then declines modestly. The phenomenon could be attributed that in the first stage, seen tasks transfer knowledge and enhance unseen tasks of similar distribution; in the second stage, interference dominates merging rather than knowledge transformation with increasing merging models; **(3)** It is notable that under both circumstances, our method consistently and substantially outperforms existing approaches as task number varies, indicating the superiority and stability of our method.\n\n### **Q5. Merging different layers**\nTo get the influence of merging different layers, we conduct experiments on merging low, middle and high layers of the model to observe the performance variation. Specifically, we perform the proposed method on selected layers and naive average merging on the rest layers. The results are shown in table below. \n\n|Merging layers of our approach|SciQA|ImgNet|VQAv2|Grounding|OCR|Viz|Flickr30k|IconQA|Avg\n|-|-|-|-|-|-|-|-|-|-\n|First 8 layers|70.85|58.69|67.66|39.40|**65.25**|43.48|46.41|31.48|52.90(-4.43)\n|Middle 8 layers|71.44|58.32|66.65|39.22|64.60|44.44|48.50|37.29|53.80(-3.53)\n|Last 8 layers|71.64|56.50|66.86|42.07|63.76|46.16|51.40|35.71|54.26(-3.07)\n|Full (Ours)|**73.43**|**65.54**|**67.20**|**44.80**|62.97|**46.61**|**52.80**|**45.90**|**57.33**\n\nIt can be inferred from the results that **(1)** As observed from **Q3** that higher layers exhibit more stark distribution, **naive merging higher layers is more prone to poorer performance** (reflected by the first two rows). This shares similar conclusion with results from **Q3** and HiDe-LLaVA[4] that lower layers carry more general knowledge and higher layers contain more task-specific knowledge. As the specific knowledge dominates the distribution, the rest singular values are more likely to alter their directions during merging. Therefore, merging higher layers without specific design inevitably leads to direction instability. Therefore, when operating our approach on the higher layers (Last 8 layers in table), performance would be the best compared to naive merging of highest layers. This is of great significance that it gives a clue for which layers are more prone to direction instability during merging; **(2)** Our approach conducts from the perspective of direction robustness. and all layers merging with our method achieves the best results, furthermore certificating the effectiveness of the proposed merging method.\n\n[4] HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model. ACL 2025.\n\nWe would like to express our sincere gratitude to the reviewer for all the thoughtful suggestions, and we hope these additional experiments could address the concerns and better strengthen the proposed method. We will update the manuscript with additional citations, experiments and analyses. If you have any remaining questions, do not hesitate to respond to us, and we are pleased to address any of your concerns!"}, {"review_id": "TtZXbeaM4L", "reviewer": "Reviewer_e3Kd", "summary": "The paper analyzes the directional robustness in the merging process from the perspective of low-rank decomposition, showing that compensating for the significant disparity between singular values helps improve directional robustness. It proposes a parameter-efficient merging method that does not require training, maintaining directional stability through complementary parameter adaptation. The paper prunes parameters based on their inter-relationships and scales the singular value coefficients to maintain directional stability and avoid interference between tasks. Experimental results demonstrate the superior performance and generalization ability of this method.", "questions": "1. What is the specific transformation applied to matrix B using the method of complementing parameter scaling in Section 3.3?\n2. How does the method proposed in this paper compare with current model fusion methods such as Fisher[1], RegMean[2], BayesOpt[3], and LORE[4]?\n[1] Merging Models with Fisher-Weighted Averaging\n[2] Dataless Knowledge Fusion by Merging Weights of Language Models\n[3] Checkpoint Merging via Bayesian Optimization in LLM Pretraining\n[4] LORE-MERGING: Exploring Low-Rank Estimation For Large Language Model Merging", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a novel Lora parameter merging method, which prunes parameters based on their inter-relationships and scales the singular value coefficients.\n2. The paper conducts extensive experiments and analysis on different models and benchmarks, validating the effectiveness and generalization of the proposed method. It also performs a thorough set of ablation experiments.\n\nWeaknesses:\n\n1.Some of the baseline methods used for comparison are somewhat outdated. Comparing with more recent merging methods would help better highlight the effectiveness of the proposed method.\n2. Can the merging method proposed in the paper be extended to other PEFT methods, such as PISSA, etc.?\n3. No OOD results.", "was_revised": true, "final_justification": "The author addressed my concerns about the applicability of other PEFT methods and validated the effectiveness of their approach. I will maintain my score.", "rebuttal": "Dear Reviewer e3Kd:\n\nThank you for your effort in reviewing our paper and we are glad you are impressed by our analysis and thorough evaluation. We will respond to every question respectively.\n\n### **Q1. Comparison with more fusion methods**\nThank you for mentioning the valuable papers. Fisher[1] and RegMean[2] are popularly employed comparing methods for evaluation of model merging. They approximate the merging parameter by fisher matrix and finding closed-form solution for a least-squares regression problem, respectively. Both Fisher and RegMean need validation set to compute the coefficient of merged models, as shown in Tab.1, and our method refrains from validation set. BayesOpt[3] explores a new paradigm that merges checkpoints during pretraining to enhance downstream performance, while our method mainly targets at merging fine-tuned PEFT modules of different tasks. Since all models are fine-tuned with 1 epoch, [3] can not be directly applied to the setting. LORE-merging[4] iteratively solves a single-variable minimization problem with coordinate descent method to obtain better task vectors. By contrast, our method does not need the time-consuming iteration, with one straightforward calculation, which is much more efficient. As [4] does not provide its code, we can not reproduce satisfying results. Comprehensively considering reproducibility and comparability, we compare with Fisher[1] and RegMean[2] to better illustrate the effectiveness of the proposed method. Results are shown below.\n\n|Method|SciQA|ImgNet|VQAv2|Grounding|OCR|Viz|Flickr30k|IconQA|Avg\n|-|-|-|-|-|-|-|-|-|-\n|Fisher|69.17|56.21|64.05|38.61|58.74|42.35|46.79|38.06|51.74\n|RegMean|68.60|47.24|65.37|37.66|59.36|41.65|39.10|34.98|49.24\n|Ours|**73.43**|**65.54**|**67.20**|**44.80**|**62.97**|**46.61**|**52.80**|**45.90**|**57.33(+5.59)**\n\nIt can be seen in the table that without validation set, our method still gets substantial promotion compared with Fisher and RegMean, strongly certificating the effectiveness of the proposed method. This could be attributed to in-depth analysis from the perspective of low-rank decomposition and specific merging algorithm design for PEFT components.\n\nWe totally agree with the reviewer that comparison with recent merging methods would help highlight the effectiveness of the proposed method. Following the suggestions from the reviewer, we find that checkpoint merging[5, 6] would be a promising direction for merging techniques, and we would treat it as potential future direction. We promise to properly cite and add discussion with all these six valuable works in the updated manuscript.\n\n[1] Merging Models with Fisher-Weighted Averaging. NeurIPS 2022.\n\n[2] Dataless Knowledge Fusion by Merging Weights of Language Models. ICLR 2023.\n\n[3] Checkpoint Merging via Bayesian Optimization in LLM Pretraining. ICML 2025.\n\n[4] LORE-MERGING: Exploring Low-Rank Estimation For Large Language Model Merging.\n\n[5] Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging. EMNLP 2024.\n\n[6] Towards enhanced LLM pretraining: Dynamic checkpoint merging via generation quality. Information Fusion.\n\n### **Q2. Extension to other PEFT methods**\nThank you for your constructive suggestion, comparison with more PEFT methods is significant to validate the extensibility of our method. In addition to PISSA[7], we also implement a commonly used PEFT method DoRA[8] that supports multimodal tasks. Due to time and character constraints, we conduct the experiment on a subset of 6 datasets and compare with two merging methods. Results are shown below.\n\n|Strategy|SciQA|ImgNet|VQAv2|Grounding|OCR|IconQA|Avg\n|-|-|-|-|-|-|-|-\n|LoRA+Task Arithmetic|71.92|59.78|66.31|40.51|63.54|42.10|57.36\n|+Ours|74.35|69.22|66.79|46.70|64.17|49.74|**61.82(+4.46)**\n|DoRA+Task Arithmetic|72.21|61.27|67.10|38.29|64.02|44.31|57.86\n|+Ours|74.89|69.40|67.14|44.24|64.85|52.26|**62.29(+4.26)**\n|PISSA+Task Arithmetic|72.36|60.89|67.43|39.11|64.51|45.99|58.38\n|+Ours|75.01|69.56|67.60|44.79|65.09|52.42|**62.57(+4.03)**\n\nIt can be concluded from the results that **(1)** more advanced PEFT technique is beneficial to better performance of model merging, and exhibits overall better performance; **(2)** our method achieves consistent and substantial improvements against existing merging methods across all PEFT methods. It strongly certificates that **the problem of direction robustness widespread exists in merging all kinds of PEFT modules**, where attention is primarily paid to improving the performance of single task. By contrast, our method handles the issue to some extent, thereby promoting the performance when the PEFT technique varies.\n\n[7] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. NeurIPS 2024.\n\n[8] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024.\n\n### **Q3. OOD results**\nWe firmly agree with the reviewer that validating OOD ability should be highlighted in model merging to certificate the generalization ability and extensibility of merging algorithm. In Tab.1, we use seen and unseen tasks to validate the generalization ability of merging model, where we merge eight fine-tuned models, and test them on four unseen tasks (line263-274). As can be seen from the table, **(1)** this can be viewed as OOD generalization that **existing methods perform poorly or even worse than zero-shot**, demonstrating its difficulty and that merging towards specific datasets would harm the generalization ability; **(2)** our method outperforms existing methods by a large margin, which could be attributed to the design of the merging method. Therefore, the generalizability of our proposed method is firmly demonstrated.\n\nAlso, following the OOD setting of BayesOpt[3], we set VQAv2 as ID dataset and test merged models on three diverse OOD datasets: ScienceQA, IconQA and Grounding to further demonstrate the OOD ability. Results are shown below.\n\n|OOD Dataset|ScienceQA|IconQA|Grounding\n|-|-|-|-\nID: VQAv2\n|Task Arithmetic|61.17|24.70|35.23\n|Ties|62.44|23.68|36.10\n|Ours|**64.09**|**27.91**|**39.46**\n\nSimilar conclusion can be obtained from the additional experiment that our method achieves better OOD performance with less ID dataset, strongly demonstrating the effectiveness and generalization ability of the proposed method.\n\n### **Q4: Form of specific transformation**\nThe transformation matrix $W$ is applied between $A$ and $B$ in the form of matrix multiplication. As a result, the matrix can be seen operating on $A$ and $B$ simultaneously and no matrix operates independently on B. With respect to the specific form of the transformation matrix, it is a diagonal matrix based on statistical characteristics of $A$, which is defined in Eqn.(3) with detailed analysis in line203-214. Briefly speaking, considering that explicitly operating on decomposed matrices is time-consuming, we directly adjust original low-rank matrices to achieve the same goal inspired by the asymmetry and correlation between two LoRA modules. By designing this, singular values can be enlarged as illustrated in Fig.3a and Fig.3b.\n\nWe once again thank the reviewer for time and effort in reviewing our paper with several insightful suggestions, and we hope the additional experiments and analyses could tackle your issues. We will properly cite the mentioned papers and add discussion in the updated manuscript. If you have any further questions, respond to us and we will try our best to address all your concerns!"}]

Author rebuttal:

Meta-review: The paper received (A, A, A, BA) with confidence (3, 4, 4, 3).  All reviewers agreed that the work tackles an important and timely problem, is novel, and shows strong empirical performance with extensive ablations. The novelty of the methodology and the empirical verification are convincing, and the consensus  after rebuttal was positive. The decision is to accept.
Decision: Accept (spotlight)