OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging
Authors: Yongxian Wei
Runxi Cheng
Weike Jin
Enneng Yang
Li Shen
Lu Hou
SiNan Du
Chun Yuan
Xiaochun Cao
Dacheng Tao
Venue: ICLR 2026 Poster (2026)
Keywords: Model Merging
Task Vector
Data-Free Optimization
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We introduce the first MLLM merging benchmark, along with a novel approach and theoretical insights.

Abstract: Foundation models update slowly due to resource-intensive training, whereas domain-specific models evolve rapidly between releases. Model merging seeks to combine multiple expert models into a single, more capable model, reducing storage and serving costs while supporting decentralized development. Despite its potential, previous studies have primarily focused on merging visual classification models or Large Language Models (LLMs) for code and math tasks. Recently, Multimodal LLMs (MLLMs) that extend LLMs through large-scale multimodal training have gained traction. However, no benchmark exists for model merging research that clearly divides the tasks of MLLM training and evaluation. In this paper, $(i)$ we introduce a model merging benchmark for MLLMs, which includes multiple tasks such as VQA, Geometry, Chart, OCR, and Grounding, studying both LoRA and full fine-tuning models. Moreover, we explore how model merging can combine different modalities (e.g., vision-language, audio-language, and video-language models), moving toward the Omni-language model. $(ii)$ We implement 10 model merging algorithms on the benchmark. Furthermore, we propose a novel method that removes noise from task vectors and robustly optimizes the merged vector based on a loss defined over task vector interactions, achieving an average performance gain of 2.48\%. $(iii)$ We find that model merging offers a promising way for building improved MLLMs without requiring training data. Our results also demonstrate that the complementarity among multiple modalities outperforms individual modalities.

Published as a conference paper at ICLR 2026
OptMerge: UNIFYING MULTIMODAL LLM CAPABILI- TIES AND MODALITIES VIA MODEL MERGING
Yongxian Wei Runxi Cheng' Weike Jin"
Enneng Yang²
Li Shen
Lu Hou
Sinan Du' Chun Yuan Xiaochun Cao²
Dacheng Tao
Tsinghua University; HHuawei Noah's 's Ark Lab;
Sun Yat-sen University: "Nanyang Technological University
bst weiyx23@mails tainghua. edu. mathshenl wtthshenli@gee 18gmail. com; yuanc www..tss..nieeii..... edu.cn
ABSTRACT
Foundation models update slowly due to resource-intensive training, whereas domain-specific models evolve rapidly between releases. Model merging seeks combine multiple expert models into single, more capable model, reducing storage and serving costs while supporting decentralized development. Despite its potential, previous studies have primarily focused on merging visual classification models or Large Language Models (LL.Ms) for code and math tasks. Recently, Multimodal LLMS (MLLMs) that extend LLMs through large-scale multimodal training have gained traction. However, no benchmark exists for model merging research that clearly divides the tasks of MLLM training and evaluation. In this paper, (i) we introduce model merging benchmark for MLLMs, which includes multiple tasks such as VQA, Geometry, Chart, OCR, and Grounding, studying both LoRA and full fine-tuning models. Moreover, we explore how model merging combine different modalities (e.g.. vision-language, audio-language, and video- language models), moving toward the Omni-language model. (ii) We implement 10 model merging algorithms on the benchmark. Furthermore, we propose novel method that removes noise from task vectors and robustly optimizes the merged vector based on loss defined over task vector interactions, achieving average performance gain of 2.48%. (iii) We find that model merging offers a promising way for building improved MLLMs without requiring training data. Our results also demonstrate that the complementarity among multiple modalities outperforms individual modalities. All code and checkpoints are publicly available here
INTRODUCTION
Foundation models experience slow develop- ment cycles due to resource-intensive training re- quirements, while Nomain-speeifie models con- tinuously improve during interim periods (Fang al. 2025). Various developers release their fine-tuned models on open-source communities such as Hugging Face. Model merging (Yadav ct al.) 2024) aims to combine multiple expert models into a unified model with multiple ca- pabilities. This approach reduces storage and serving costs through model reuse, while sup- porting decentralized development by enabling independent contributors to build models that can later be merged. Despite its potential, pre- vious studies (Akiba et al. 2025 Ilharco et al.) 2023 Yang et al. (024b) have primarily focused on merging visual classification models across
Oven LiavA-o1
Pelaate canting
InteenVL
Jutte
Data: Trve
Druet
eenglngg
Google
welasio
Cent NLLM
aonglig
Figure Unifying the capabilities modali- ties of MILMs from open-source communities via model merging, which a data-free, cost-effective post-hoc method.
*Corresponding author

---

Published as a conference paper at ICLR 2026
multiple datasets to extract representations, merging Large Language Models (LLMs) specifically for code and math tasks.
Recently, Multimodal Large Language Models (MLLMs) that extend LLMs with broader capabiliities through large-scale multimodal training have gained traction. Model merging offers a cost-effective way combine fine-tuned MILMs with task-specific skills into unified model (see Fig. [ However, no benchmark exists for model merging research that clearly divides the tasks of MILM training and evaluation. Specifically, AdaMMS (Du al. (025b) proposes an unsupervised hyper- parameter selection method, but can only merge two MLLMS at time. For example, merging LLaVA-One ViniinnnnnnnOOOOn Onwiisinn-Qeen (Li al. 2025a) into Qwen2- VL (Wang et al., 2024b) on the Qwen2 architecture. UQ-Merge (Qu et al., 2025) treats each fine-tuning dataset in LLaVA-v1.5 (Liu et al. 2024a) as separate task without categorization of MILLM capabilities, fine-tuning the base model for each dataset, and using LLaVA- 1.5 as the mixture training baseline.
In this paper, we introduce a model merging benchmark for MLLMs, which includes diverse spe- cialized models such as VQA, Geometry, Chart, OCR, and Grounding. For each corresponding task, we collect comprehensive public datasets with least 100k samples to ensure effective supervised fine-tuning, and select corresponding benchmarks to evaluate distinct capabilities. We derive an upper bound on the error between the merged model and expert models, proving that merging performance is influenced by the learning rate and iterations, which control the extent of parameter drift. Smaller parameter changes sacrifice task performance but lead to more effortless merging. We choose two types of vision-language models: Intern V1.2.5 and Qwen2-VL, providing both LoRA and full fine- tuning checkpoints. Moreover, most existing MLLMs specialize in dual modalities, and incorporating new modality encoders requires re-training on new modality-text data. Generating high-quality multimodal instruction data is resource-consuming (Jiang et al. 2024). Therefore, we explore how model merging can efficiently combine different modalities (e.g.. Dision-language lanio-lan and rideo-language models), moving toward the Omni- Omni-language model. This offers data-free way to reuse and integrate modality-speciffe encoders into a unified L.I..
Based on our benchmark, we conduct an in-depth comparison and analysis of of-e-oof-the-at merging methods in capability and modality merging settings. Furthermore, we propose a novel merging method that improves the task vector (i.e., the parameter change between fine-tuned and base models) optimization. OptMerge optimizes the merged model based on a loss defined over task vector interactions and applies low-rank approximations to reduce redundant noise, achieving the best results. Combining multiple MLLMs without requiring data, the merged model can even outperform expert MI.MMs in their respective capabilities and mixture data training. We also find that merging methods effectively integrate inputs from multiple modalities, outperforning models trained on individual modalities, thus emphasizing the complementary nature of modal information.
In summary, our main contributions are threefold:
We introduce the first model merging benchmark that provides fine-grained categorization of MLLM capabilities, and evaluates how merging integrates multiple modal- ities. We train expert models for each task and publicly release their weights and code. This benchmark is designed to help the model merging community better evaluate the generalizability of their methods.
Benchmark:
Methodology: We further propose a simple yet effective method, OptMerge, which removes noise from task vectors and enhances the robustness of merged vector optimization. Ablation studies show an average performance improvement of 2.48%.
Experiments: We conduct comprehensive experiments and analyses on our benchmark. Our empirical results suggest that model merging can outperform mixture training, offering a viable path to omni-model alignment and a scalable approach to developing MLLMs with reduced computational cost and training time.
2 RELATED WORK
Model merging. Model merging has emerged as a cost-effective approach to developing improved models by combining multiple expert models to leverage their complementary capabilities (Yaday ct al. 2024, Ahmadian et al. 2024). These expert models typically share a common base model, with specialization achieved through fine-tuning distinct datasets. This approach offers flexible
2

---

Published as a conference paper at ICLR 2026
and modular method for post-training MLLMs and facilitates the integration of new capabiliities into uop-prforming models. Current research on model merging falls into two primary categories: static merging and dynamic merging. Static merging compresses multiple models into single standard- sized model without adding additional computation or memory overhead. Dynamic merging (aka MoE-like methods) (Tang al.] 2024b Huang al. [024]] Lu et al., 2024b) requires the dynamic loading of task-specific modules based on test inputs, involving training routers or prior knowledge.
The storage parameters for dynamic merging are larger.
Static merging can be further divided into data-free methods and test-time adaptation methods. Data-free methods merge fine-tuned models without requiring additional data. We categorize these methods into four groups: (i) Linear interpolation methods that perform arithmetic operations on
task vectors (Wortsman et al. 2022 Iharco et al. 2023 Goddard et al. 2024; Chen et al.) 2025 ; (ii)
Sparsification-basel methods that reduce redundancy in task vectors (Yadav et al. 2023 Yu et al.
He et al.] 2025); (iii) XVD-based methods that identify and exploit the low-rank features of
task vectors (Gargiulo et al. 2025 Marczak al.) 2025 Choi et al. 2024| [Sooca et al. 2025); and
(iv) Optimization-based methods that optimize task vectors via gradient descent (Wei et al. 2025c
Cheng al. 2025). Test-time adaptation (Yang et al. 2024d Daheim al. 2024) assumes access
to unlabeled test datasets, which can be considered a form of transductive learning.
Although test-time adaptation and dynamic merging achieve remarkable results, their practical appli- cability is limited due to challenges including data privacy concerns, additional storage requirements, and insufficient parallelism in merged models. Therefore, we focus data-free static merging.
Model merging in MLLMs. Recently, several works have attempted model merging for MLLMs. but with different objectives. VL-merging (Sung et al. 2023) merges modality-speciffc models to create modality-agnostic models, evaluating their effectiveness through fine- fine-tuning on downstream tasks (e.g., image classification). VisionFuse (Chen et al. 2024d) employs task arithmetic to merge LLMs with concatenated visual encoder outputs, primarily focusing on enhancing MILMs' visual capabilities. Firstly, UnIVAL (Shukor et al., (023) proposes study multimodal model merging via weight interpolation of models trained on different multimodal tasks, showing their benefits in particular for generalization. DAMC (Chen al. (024a) composes MLLMs across image, audio, video, and point cloud modalities while reducing modal interference through parameter decoupling.
Several approaches similar to ours aim to merge multiple MLLMs improve multi-task performance. AdaMMS (Du et al.) 2025b) proposes an unsupervised hyperparameter selection method for model merging. However, requires generating responses for each candidate hyperparameter, making tine-consuming and assuming test set availability during merging. Furthermore, it can only into Qwen2-VL on
merge two models time. For example, merging LLaVA-OneVision--ee the Qwen2 architecture, or merging LLaVA-v1.5 into CogVILM-Chat on the LLaMA architecture. UQ-Merge (Qu et al. (2055) considers uncertainty quantification on text and vision inputs to examine MLLM prediction confidence, requiring unlabeled test sets to calculate prediction and determine merging sequence. This approach measures uncertainty across all candidate models and repeatedly evaluates merged models to find optimal combinations. UQ-Merge treats each fine-tuning dataset in LLaVA-v1.5 (Liu et al. 2024a) as a separate task without categorization of MLLM capabilities, fine-tuning the base model for each dataset and using LLaVA-v1.5 as the mixture training baseline. In contrast, our benchmark collects more comprehensive data with clearer MLLM task divisions for fine-tuning, and propose a data-free method that requires hyperparameter search.
RETHINKING MODEL MERGING
we begin by introducing common model merging algorithms. In Sec. 3.2 we revisit empirical findings from prior work, and provide a theoretical explanation of the relationship between model fine-tuning and merging performance. Building on this, we analyze the statistical properties of our benchmark, demonstrating both its validity and the challenges it presents.
In Sec. 3.1
3.1 MERGING BASELINES
Model merging aims to integrate multiple fine-tuned models, all derived from a base model 0,, into a unified model that consolidates knowledge from diverse sources. Given n fine-tuned models denoted
3

---

Published as a conference paper at ICLR 2026
as 01. 2n the objective is to produce single merged model p... that effectively inherits the capabilities of all individual models. We categorize merging methods into four groups.
Linear interpolation methods: Weight Averaging (Wortsman al. 2022) averages the weights of
models fine-tuned on different tasks. Task Arithmelic (llharco et al. 2023) computes task vectors
T, 0, 0g for individual tasks and sums them to form mulli-task vector Tm >== T3. This vector is scaled by a coefficient and added to the base model 0g to obtain the merged model.
Sparsification- methods: Ties-Merging (Yadav al. 2023) combines steps like trimming, parameter sign determination, and disjoint merging to produce the Tm- The final model is defined as On = 0o XTm. where is tuned using the validation set. DARE (Yu al. 2024) randomly drops redundant task vectors and rescales the remaining ones to mitigate parameter interference.
SVD-based methods: TSV Merging (Gargiulo et 2025) quantifies task-specific feature overlap in weight space by measuring the singular task interference of TÜ It then reduces task interference through decorrelation. The method seeks orthogonal matrices V1 and U1 reconstruct the parame- ters of the merged model. Iso-C (Marezak et al.) 2025) proposes an isotropic merging framework that flattens the singular value spectrum of task matrices, and enhances alignment between singular components of task-specific and merged matrices.
Optimization-baser methods: WUDI Merging Cheng et al. 2025) proves that task vectors 7 form an approximate linear subspace of the fine-tuning data æ. This property allows the implicit utilization of training data information through task vectors alone. They define layer-wise interference between the merged vector and task vector as Tm. T for task at layer I. To optimize the merged vector Tm.,, they minimize this interference (Tm., T.,, with respect to data T, Leveraging the linear subspace relationship, they substitute the transpose of Tea for TT,1:
min Li Tm,i
[[ (T.)
(I)
This formulates model merging a data-free optimization problem over parameters. Using the Adam optimizer, we obtain the merged vector Tm. which minimizes interference with task vectors on multiple tasks, iee. the hidden activation satisfies (fo.1 + Tm, R (0o.1 + T,,)a,.
3.2 PARAMETER CHANGES DURING FINE-TUNING MATTER
Model merging exhibits sensitivity to task vectors T; (i.e., parameter changes between fine-tuned models and the base model). Several studies (Yu et al. 2024 Li et al.) 2025b) demonstrate that less intensive fine-tuning can yield superior merging performance, even when these models achieve lower accuracy on their respective tasks. In App. B.I. we conduct experiments on the impact of fine-tuning steps on merging performance, and observe that performance tends to rise initially and then decline. This counterintuitive finding suggests that higher-performing expert models do not necessarily produce better merging outcomes. Fine-tuned models tend to converge around the base model in parameter space (Merlin et al. [2023] Chung et al.] 2024). When constructing our benchmark, we minimize parameter changes by adjusting the learning rate while maintaining performance improvements on specific tasks. We analyze the upper bound the loss incurred by model merging:
Theorem 3.1. Consider task trained for T iterations of gradient descent with a fixed step size r) E (0, 1/L., where L. is the Lipsehitz constant. Let := qp E (0, 1) denote the PL convergence factor. Then the merged update := - - OJT satisfies
L. TC) + (6 +(, 01TTT))
where O(:TT) is the residual error from incomplete convergence on task i, O68 is the cross-task interference term, and T²2 is the curvature term from L-smoothness. This indicates that both the learning rute and iterations influence model merging results. Pleuse refer to App. Aor detailed assumptions and proofs.
Remark. Theorem 3.1 provides the first theoretical explanation of how model fine-tuning affects merging performance. The target task's 's gains dominate in the early training stage, but as convergence approaches, cross-task interference and curvature errors (growing with 17T and TT2 can undermine
4

---

Published as a conference paper at ICLR 2026
12
DCR
14-
35
Geametty y
Geumeity
Chart
30
Chart
Grounding
Grounding
1.5
1.-
0.0
-5
0.0
7.
455
-5.0
5.0
-5.0
-45
-3.0
The magnitude the parameter (Log10)
The magnitude the parameter (Log10)
(n) Task vector magnitude distribution (IntemVL2.5)
(b) Task voctor magnitude distribation (Qwen2-VL)
30008
03809
V26
Cwn
33000
048006
3.000
A3002
(c) Frobenius norm of different layers (12.ernVL2.5))
(d) Frobenius norm of different layers (Qwen2-VL)
Figure Visualization task vectors from the benchmark, revealing the small extent of parameter changes during fine-tuning. Intern VL.2.5 (full fine-tuning) and Qwen2-VI. (low-rank adaptation) exhibit distinct distribution patterns across different tasks.
merging performance. Thus, in the convergence phase, is essential to control directional Ieakage (small 6) and limit T/T ensure high-quality merging. It supports previous empirical observations: fine-tuned models in current benchmarks typically remain within the same basin near the base model. For example, merging Qwen2.5-Math (Yang al. 2024a) and Qwen2.5-Coder (Hui et al. 2024) yields poor performance. This is likely due to excessive post-training, which causes significant parameter drift. This insight suggests that poor merging results may not reflect algorithmic flaws, but rather issues with the fine-tuned models. When selecting models from Hugging Face, it's helpful to choose fine-tuned models that are better suited for merging minimize multi-task degradation.
We examine the weight magnitude distribution of task vectors across our benchmark (see Fig. 2 (a-b)). Our analysis reveals that Intern VI.2.5, which undergoes full fine-tuning, exhibits right-skewed distribution. In contrast, Qwen2-Ve VL. fine-tuned using LoRA, displays multi-modal distribution. This is due to the low-rank nature and scaling factors LoRA, which constrain the task vectors to be linear combinations in reduced subspace, causing them to cluster along a few dominant magnitudes. Both models demonstrate distinct magnitude distribution patterns across different tasks. We also compute the normalized Frobenius norm of parameters (i.e., divided by the number of parameters). As shown in Fig. 2 (c-d), the Frobenius norm varies significantly across tasks and layers, which presents challenges that will address in approach. The small task vector magnitudes suggest that fine-tuned models and base models exist in adjacent regions of the loss landscape with linear connectivity (Wu al. 2023), facilitating effective model merging.
4 METHODOLOGY
Eq. [I] defines a loss between the merged vector and the task vectors. However, data-free optimization often suffers from instability and convergence issues. To address this, we propose OptMerge, a novel method that improves task vector optimization. Specifically, fine-tuning and LoRA fine-tuning scenarios, as they yield model parameters with distinct properties (e.g., s., low-rank sparsity and varying optimization difficulty). These differences naturally necessitate
approach accommodates both full
tailored merging strategies, as detailed in Sec. 4.1 and Sec. 4.2
4.1 MERGING FULL FINE-TUNING MODELS
Task vectors contain significant redundancy and noise, leading to mutual interference during merging, Redundancy stems from different tasks re-learning shared foundational skills, while noise reflects non- essential parameter updates. Directly adding task vectors amplifies these issues, hindering effective
5

---

Published as a conference paper at ICLR 2026
Ausragge norm layerss
Optimizing Tm
000027
0
0,0002
Fm
000037
WUDI Merging
Fm
Fm
4
0.0000
0o
Fi
orthogonal
0:
0,0000
Optimization iteraiion
Figure 3: When optimizing Eq. ( Tm tends to take shortcuts by increasing its magnitude to achieve orthogonality.
Figure 4: We plot the progression of the Frobe- nius norm of the merged vector during optimiza- tion (average by layers).
merge vector optimization. To address this issue, we propose reducing inter-task interference through low-rank approximation. First, we calculate the average task vector Σ;= Ti, and use it to center task vectors (Choi et al. 2024). Next, we perform SVD to isolate core task-specific knowledge from noise present in the top and lower singular vectors
SVD(TL, 7)) UEVT, where U e RRXr DER V Ruxr.
(2)
We then apply low-rank approximations to eliminate redundant noise, where U1:k Ei:k, VI represent the top-k singular components. Moreover, we find that substituting Σ1: ViT for the task vector Tu, as the input subspace æ; allows us to discard secondary row space information, focusing only on the column feature space. Thus, we can optimize Tm, via gradient descent on the loss:
min Tm.1
Σ
[[ T
UnA ViL (2 ViTETTIE..
By truncating singular values, we preserve critical features VT which is similar selecting principal components in Principal Components Analysis (PCA) (Abdi & Williams (010). This yields more accurate estimates of a,, than using (Tr.))
4.2 MERGING LORA FINE-TUNED MODELS
The inherent low-rank nature of LoRA fine-tuning presents unique optimization challenges for the merge vector. When optimizing m, gradients become effective only in directions corresponding to non-zero singular values of Ti,,, while approaching zero in other directions (null space). This constraint limits parameter update freedom, preventing Tm. from properly exploring the parameter space. We observe that Tm. tends to take shorteuts by increasing its magnitude to minimize loss. This occurs because the merge vector must simultaneously accommodate multiple task vectors in different directions. Without constraints, Eq. (T) achieves orthogonality by increasing the length of the merge vector (see Fig. 3. When added to the base model, such large-norm task vectors cause deviation from the original distribution, resulting in collapsed language ability.
To address these challenges, we introduce set of practical techniques: (1) We replace Adam with SGD, which better escapes flat local optima and offers greater stability under sparse gradients. Notably, SGD provides implicit regularization (Smith et al.) 2022 Wang et al. 2022). constraining task vector optimization and navigating flat regions induced by null spaces. (2) We apply direct low-rank approximation to Ti using truncated SVD(T.,) ss Ei ViTe without centering. The Frobenius norm equals the sum of squared singular values 11 Σ}=10 After truncation, we
drop the tail energy Σj> of and thus reduce the norm. (3) We also introduce initializing the merged vector with the mean of task vectors to mitigate the issue of excessive merge vector magnitude. As shown in Fig. [ our approach maintains relatively consistent norm throughout the optimization process while minimizing loss successfully.
We optimize only the model's linear layers, representing each task vector X matrix. The remaining layers merged by simple parameter averaging.

---

Published as a conference paper at ICLR 2026
Table I: Summary of collected training datasets, with their corresponding sizes and languages.
Task Category
Size
Datasets (Language)
VQA
588K
GQA (en) (uudson & Manning, 2019]
VQAv2 (en) (Goyal al. 2017) OKVQA (en) (Marino al.
2019
In-Atr-ns (2h) (I [L al. 2024n Wii..iiiiglledddddcoo ten&tz) [Wang et 2024c
Cog LogVLM-Mutiround (en&zin) Wang et 20240
Geometry
190K
GeoQA+ (zh)
(Coo & Xiao
2022
G-LLaVA (cn)
(Gno et al
2023
Chart
218K
ChariQA (en)
(aasry al.
2023
DVQA (en) (Kalle et
2018
OCR
238K
OCRVQA (en)
Mishra et al.
2019
TextCaps (en) Sidorov el 2020 SynthDoG (en) Kim al.
20221. LLaVAR (en) (Zhang et
2023]. VQA [en) (Biten al. 2019) TextVQA (en) (Singh
ct al. 2019 DocVQA (cn) (Mathew et al
2021). DecpForm (cn) (Sveilichnaya 2020
KLC
Stanislwekk et al. 2021 20. TabFact (en)
al. 2019]
Grounding
135K
RefCOCO (cn) (Y al. 2016
Mao 2016).
VG (cn) Krishna ct 2017
MILMs MERGING BENCHMARK
We begin by detailing the benchmark, including its checkpoints, datasets, and evaluation protocols, as well as the implementation of merging algorithms (Sec. (Sec. Next, Sec. 5.2 presents extensive experiments that empirically validate the benchmark, and summarizes the key findings.
5.1 BENCHMARK DETAILS
Checkpoint construction. To cover two practical scenarios, namely fine-tuning base models and fine-tuning instruction-tuneds models, we select two models that differ in intended use: MnterVLL..5 IB-Instruct (Chen et al. (202c)), lightweight model aligned for instruction following. and Qwen2- VL-7B-Base (Wang et al.j 22024)) a general foundation model. Qwen2- VL-7B-Bast is among the few publicty available pretrained models, so use it for the base-model scenario. These choices span different training strategies and scales, enabling a broad assessment of merging methods.
For modality merging, select Vicuna-7B-v1.5 (Zheng al., 2023 as the shared LLM. The vision-language model uses CLIP- -ip--vxT---336px (Radford et al. 2021) as the image encoder, paired with MLP projection as the connector. The audio-language model adopts BEATs-Iter3+ (Chen et al. (202)) as the audio encoder, with Q-Former as the connector. The video-language model employs LanguageBind (Zhu et al. 2023) as the video encoder. See App. Cpp.Iite flor details.
Training data. We collect a broader range of domain-specific data, divided into VQA, Geometry, Chart, OCR, and Grounding tasks. The datasets used are summarized in Table [ For effective supervised fine-tuning, we gather at least 100k public dataset samples for each task, ensuring maximum diversity wherever possible. We process all data into the instruction tuning format.
Evaluation benchmark. Current benchmarks (Liu et al. 2024b Li et al., 2024 Chen et al. 2024b Fu et al., 2024) predominantly evaluate a model's 's overall performance but provide limited insights into specific capabilities. Therefore, we curate carefully selected suite of specialized datasets to evaluate five capabilities: VQA, geometric reasoning, chart understanding, OCR based VQA, and referring expression grounding. See App. for details. All evaluation results are obtained using the VLMEvalKit (Duan et al. 2024) and LMMs-Eval (Zhang et al., 2024 libraries under the same settings ensure fair comparison. All experiments are conducted using 8x NVIDIA V100 GPUs.
For languae models, we assess Audio-VQA, which requires multimodal understanding and spatiotemporal reasoning about visual objects, sounds, and their relationships in videos.
Merging details. Following Task Arithmetic (Ihharco et al., 2023), we employ a single coefficient to scale the merged vector before adding it to the base model. For all model merging methods, determine the optimal merging coefficient À by searching within the range [0.1, 0.3, 0.5, 0.7, 1.0, 1.5]. In our implementation, the rank size Ao in Eq. (3) is simply defined as the rank of each task vector divided by the number of tasks (i.e., 5). We use the Adam optimizer with a learning rate of le-5 for Intern VL while applying the SGD optimizer with a learning rate of lc-4 for QwenVL. The number of optimization iterations is set to 300. We apply our method exclusively to the linear layer in the model.
5.2 EXPERIMENTAL RESULTS
Capability merging. As shown in Tables 2 and merging individually specialized models outper- forms expert MILMS their target tasks. For example, the merged Qwen2- VI. achieves 51.05 and
7

---

Published as a conference paper at ICLR 2026
Table 2: Capability merging results on VL2.5 (full fine-tuning) across multiple tasks. For the merging methods, we highlight the best score in bold and the second-best score in underlining.
Methods
VQA
Geometty
Chart
OCR
Grounding
Ang.
VizWiz GQA (Bes)) Vista (mina) MATH Viion (minn) [ChinQA (1es0)
TexVVQA (val) OCRVQA (test) RefoOCo ReCCOCO ReCCOCOg
Instruct
25.15
54.62
46.30
18:42
09.48
72.51
41.08
71.09
00.41
67.40
55.06
Ialiesded VQA
17:01
48.76
36,04
Ialieihal Gounetty
32.80
25.00
51.76
19.51
Indieiiall Char
2016
Indieiiall
68.01
Cronciing
Weisht
29.96
$4,89
19.60
18.42
71.64
74.54
41.
45.29
52.39
Tuck Arithmetic
30,67
56.34
21,05
72.88
76.26
68.15
56.18
TIES Mergiia
30,63
56,48
23.68
72.28
76.29
44.01
68.45
56.70
DARE
30,61
56,48
21,05
73.06
76.30
43.03
74.94
68.07
TIES DARE
30,05
22.63
72.32
76.19
43.33
68.46
73.55
56.76
TSV Merging
31,15
56.67
52.45
18.95
70.56
75.66
65.19
58.51
54.37
Iaa-C
25.21
55.36
48.96
21,05
70.56
69.34
46.51
72.72
66.56
68.50
54.78
WUDI Merging
31,02
36.96
$3.00
17.01
09.19
75.95
45.12
76.06
70.14
74.48
$7.00
OytMerge (Ours)
30,97
57.33
34.48
21.05
68.72
76,01
45.35
75.97
09.73
73.94
$7.44
Misture Training
29.0 79
61.33
$2.85
23.68
70.32
72.56
60.25
72.06
65.95
67.46
157.66
Table 3: Capability merging results on Qwen2-VI. VL (L.RA fine-tuning) across multiple tasks. For the merging methods, we highlight the best score in bold and the second-best score in underlining.
Methods
VQA
Geonetey
Chart
OCR
Geounding
Axg.
VizWiz GQA (Dest)
Motth (min)) MATH Visen (zimi] [ ChamqqA (1e51)
TertVQA (val) OCKVQA (test) RetDac0 RetUOD0+ ReCC
OwenZ Bise
5.52
47.85
23:69
37.55
21.852
12.71
Indicihal Geometry
$5.57
78 05
45.95
9.31
Inlicihal Char
$2.89
63.67
Ialicihal OCR
37.53
13.16
64.68
Inlisiiaal Groudding
19.74
18.08
75.05
Weight Avergg
41.47
57.33
50.31
34.21
59.56
$1
57.85
50.72
65.37
77.68
660.55
Tink Asillmatic
40.52
62.31
40.36
26.31
70.67
1.009
$0.50
75.06
61.33
75.85
60.29
TIES Merging
41,38
39.08
46.37
34.21
67.34
$1.5
55.53
$0.63
63.36
77.65
61.24
TA wi DARE
40.64
62.38
40.67
26.31
79.76
$1.04
59.34
75.85
61.41
75,80
60.32
TIES w/ DARE
41.63
39,96
45.72
35.53
70.68
$1.53
59.63
80.73
65.65
22.27
61.88
TSV Merging
41.43
57.31
51.05
34.21
59.44
$1.25
$7.81
80.31
05.34
77.76
00.6
Iso
12.31
13.44
39.56
27.65
2,50
30,05
6.12
53.68
38.96
41.50
26.09
WUDI Merging
37.19
56.45
42:56
27.63
67.84
79,92
68.56
76.25
60:72
71.99
58.65
OptMerge (Ours)
41.01
51.16
48.06
40.79
74108
$1.54
60005
80.92
68.90
78.24
63.30
QwenZ VL Instruct
64.09
02.18
46.02
19.73
70.04
78.28
65.42
$2.39
77.57
75.63
662.5
40.79 on Geometry (vs. 42.50 and 28.95 for individual models) and 79.76 on Chart (vs. 61.08). We observe similar gains for OCR and Grounding, with complementary benefits between these tasks. For Intern VL2.5-IInsruce we conduct mixture training by combining all task-specife training data for SFT. For Liven--V--Base we directly use Owen2-VL-Instruce as the upper bound for mixture training, given its extensive prior SFT with diverse datasets. Notably, our best model merging methods closely match or even surpass mixture training and instruct versions. These results demonstrate that model merging potentially surpasses multi-task learning, while providing a scalable solution for creating high-performing MLLMs with reduced computational cost.
exhibit distinct behaviors. Linear interpolation of task vectors, while ignoring parameter conflicts, is robust but only moderately effec- tive. Sparsification-ben based methods such TIES struggle control sparsity and often underperform relative to task arithmetic. DARE reliably provides plug-and-play gains through simple rescaling. SVD-based methods are sensitive to the spectral structure of task vectors. For example, Iso-C fails on Qwen2-VL. because the LoRA-tuned task vectors are already low-rank, and averaging singular values further reduces their Frobenius norm, creating instability in LLMs. Even increasing X, as recommended in their paper, only marginally improves results. TSV merging excels in modality merging because its orthogonalization mitigates modal conflicts, but delivers ordinary performance in multi-task settings. In contrast, our approach achieves superior average results across various scenarios, benefiting from stable task vector optimization.
Categorization of merging methods. Different merging methods
Improved task vector optimization. Our method en- hances task vector optimization stability, achieving optimal results. In Table we evaluate each component's contribu- tion to overall performance. Starting with WUDI Merging, we incrementally add one component at a time, reporting performance for both LoRA model merging (Qwen2-VL) and modality merging (Vicuna-7BB 7B). Replacing Adam with SGD alone does not necessarily improve performance; however, when combined with initializing the merged vector using the mean of task vectors, we observe a significant 4.43% improvement.
Table 4: The ablation study.
Qwen2-VL
Vicuna-7B
WUDI Merging 58.65
64.65
SGD Initialization
48.88 (-9.77%) 66.91 (42.26%)
63.08 (+4.43%) 67.07 (+2.42%)
Low-rank
63.30 (+4.65%) 67.00 (+2.35%)
8

---

Published as a conference paper at ICLR 2026
Table Modality merging results on zero-shot imaee-audio-video question answering tasks by merging vision- vision-language, audio-language, and video-langune models. The "Individual Modalities" columns show baseline performance for each single-modalis model.
Individual Modalities
Merging Methods
Online Composing
Datasets
Vision Audio
Video
Weight Average
Task Arithmetic
Merging
Ties
TSV Merging
Iso-C
WUDI Merging
OptMerge (Ours)
NaiveMC
DAMC
MUSIC-AVQA
50.77 27.93
49.02
47.75
52.14
50.35
53.78
52.77
52.43
$3.17
53.50
$2.80
AVQA
75.55
47.57
79.20
69.39
78.62
75.84
80.90
77.51
76.86
80.82
80.26
80.78
Avg.
63.16 37.75
64.11
58.57
65.38
63.10
67.34
65.14
64.65
67.00
66.88
66.79
Table 6: Merging results on actual fine-tuned checkpoints collected from Hugging Face.
Methods
VQA
Gouneryy
Charr
OCR
Graunding
Avg.
VizWiz CQA (Dest) Vista (nin6) SENTH Vision (mist)) Channga (test) TeuVVQ (vail
OCRVQA (tstt) RerCOCO ReCCOCO RetCOCO
Quea2-VL VL. RB-GRPO--L
79:12
Qwea2 Pokemoo
76.75
71.55
EAN
70.70
40.59
43.26
Task Aitlmnttic
41.57
60.95
42.99
23.58
75.25
81.95
71.78
87.72
81.60
55.63
65.32
TIES Merging
46.17
60.54
42.52
27.95
75.48
$2.40
71.09
99,86
83.52
56.44
65.42
TA w/ DARE
43.33
61:15
44.17
26.95
26.45
83.03
72.00
85.00
82. 70
16.07
66. 90
TIES w/ DARE
44.37
60.78
18.37
27.53
76.04
82.61
70.93
89.40
82, 93
$6.77
66.58
TSV Merging
43273
61.00
4352
21.01
75,
83.15
71.05
00.53
25
66.41
66.50
Ba-C
01.34
76.96
83:27
WUDI Merging
41.39
74.36
80-78
83.50
83:48
64.50
OptMarge (Can))
87.97
Qwen2 VL. Instrntt
44.09
70.04
76.38
77.87
75.05
60.25
Low-rank approximation further enhances performance, demonstrating its effectiveness in preserving critical knowledge from task vectors while maintaining the stability of the Frobenius norm. For full fine-tuned models, Tables 2 and o show average improvements of 0.44% and 1.9% for OptMerge over WUDI Merging, respectively. This highlights the necessity of Eq. (3) over Eq. T for denoising task vectors and achieving robust merged-vector optimization.
Modality merging. As shown in Table merging methods effectively integrate information from three modalities, outperforming models trained on individual vision, audio, or video inputs. This highlights the complementary nature of modal information and its potential for merging. Online composing dynamically merges activations in the LLM from different modalities during inference. requiring separate parameter storage for each modality (i.e., 3x static merging). NaiveMC (Chen et al. 2024a) performs simple activation averaging, while DAMC (Chen et al. 2024a) decouples parameters during training to reduce modal interference. Notably, the best merging method even outperforms these online composition methods. Advancing Omni models through model merging offers promising direction for future research.
Computational requirements. As illustrated in Ta- ble 7 we compare the solving time and GPU mem-
Table 7: Model merging Data mixing.
ory usage of our approach against mixture train- ing. Our approach optimizes the merged vector over 300 iterations while incurring minimal com- putational overhead and requiring significantly less
Methods
Solving Time GPU Memory
VT.2.5V (Ours)
0.22h
2.62GBB
VLe.5.LL..5-11 (Mixed)
25.38h
240GB
Qwen2-VL-7D (Ours)
3.78h
21.97GB
Qwen2-VL-7B (Mixed)
24.56h
256GB
GPU memory than data-based training. This effi- ciency is achieved through layer-by-layer optimization without requiring training data, Our results confirm that the proposed method is computationally efficient and highly scalable on devices with modern GPUs, facilitating the rapid development of new models based on existing ones.
Actual checkpoints from Hugging Face. To evaluate the practicality of model merging in com- munities, we collect fine-tuned models released by different developers on Hugging Face. Our collection includes a model specialized in math reasoning via multimodal reinforcement learning personalized model for the Pokemon domain") a model focused on converting PDF documents into tex and model with OCR and VQA capabilities in Vietnamese As shown in Table 6 OptMerge
hhtps:/huuginggface hhtgss//ingiiggaeee.cc//mas/st// http:///ww......aooooo//oeeeeenn
'https//hugginnfacccco wwtp://wwww...oooooooo/o//////..

---

Published as a conference paper at ICLR 2026
Table Ablation study on rank size is ratio for OptMerge. The rank size & 10%, 20%, 30%, 40%, and 50% of the rank of each task vector.
ratio
VQA
Geumetry
Chart
OCR
Grounding
Arg.
VizWiz
GQA (lest) MathVista (mini) MATH Vision (mini) ChanQA (test) TexVVQA (val)
OCRVQA (lest) RefCOCO RefCOCO+ ReCCOCOg
10R
30,90
57.06
$1.49
76.10
76
56.91
30%
31.55
57.15
21,05
68.72
76.27
43.67
73.63
66.84
70.92
409
11.49
56.92
55 75.77
25.00
67.36
76.06
45.96
65.5 65:55
58.40
59.64
$4.22
50%
31.37
$6.68
56.75
23.68
68.08
75.81
45.02
61.45
51.50
56.19
$2.98
Table 9: Capability merging results on Qwen2.5- Qwen2.5-VI.-32H. -1-32BB-nnttt across multiple tasks. For the merging methods, we highlight the best score in bold.
Methods
VQA
Geometry
Chart
OCR
Grounding
Avg.
VicWis GQA (ual) Vadia (mint) MATH-Vishm (mins) ChanQA (tet)) TraVVA (val) OCRVQA (loe)
RefOOCO ReICO0O Re1C000g)
Swea2.5 VL-32B
41.39
59.01
$2.41
$4.06
77.96
42.67
50.34
43.42
$4.56
IadiiQuall Groundin
78.85
42,10
66.19
82.95
Indivital Cher
43.01
61
14 38
41.42
36.96
31.71
67.90
69.72
81.52
14.11
71.71
Indivitull VQA
43.34
62.78
78
42.11
36.68
31.04
67.06
89.34
83.90
$4.72
71.86
Individual OCR
42.65
61.04
75.25
34.21
$7.00
31.42
67.32
59.63
83.76
$4.62
70.09
OptMerge (Duns)
43.52
62.50
50.01
43.42
$8.93
$1.91
66.37
$9.94
83.97
$4.68
72.52
achieves performance that surpasses that of the individual models, effectively integrating knowledge from diverse models to construct more robust system.
Rank size k. To further investigate the impact of rank size, we conduct additional ablation studies by setting
10%. 20%, 30%, 40%, and 50% of the rank of each task vector. The results are summarized in Table 8 As shown, the performance remains relatively stable for k ratios between 10% and 30%, indicating that OptMerge is robust to moderate changes rank size.
Model scales. We extend our evaluation to the larger - Instructt model and augment training with additional high-quality fine-tuning data. As shown in Table 9 OptMerge effectively combines multiple fine-tuned models while mitigating cross-task interference, achieving the best overall performance and surpassing the base Qwen2.5-VL, TTLL-2B-Instren These results indicate that OptMerge remains effective and beneficial at larger model scales.
Table 10: Evaluation of the merged model on general multimodal QA benchmarks.
Doc MMMU
ScienceQA
A12D
Infographie
Individual Geometry
33.67
64.29
73.25
62.27
29.79
Individual Grounding
34.22
76.54
63.24
33.82
Individual Chart
30.33
57.13
40.01
29.86
26.02
Individual VQA
26.00
62.93
50.83
44.59
39.07
Individual OCR
38.00
77.67
63.66
54.39
41.97
OptMerge (Ours)
39.33
84.18
91.89
79.44
56.84
General tasks. We further evaluate the merged model (based Intern VL2.5- on set of general multimodal QA benchmarks that require combinations of multiple abilities. The results are shown Table IO On these integrated benchmarks that require multiple abilities, single- ability models cannot solve the tasks effectively. In contrast, our OptMerge, which merges all specialized models, demonstrates emergent integrated capabilities and consistently outperforms the best individual model for each task, with an average improvement of 10.85% across benchmarks.
6 CONCLUSION
Model merging aims to combine multiple expert models into single model without requiring data. In this paper, we introduce the model merging benchmark with detailed categorization of MLLM capabilities, and explore how model merging can effectively combine different modalities of MLLMs. We further propose a novel merging method that effectively removes noise from task vectors and improves the robustness of merged vector optimization. Our results demonstrate that model merging potentially surpasses mixture training, serving as a way for omni- omni-model alignment, while offering a scalable solution for developing MLLMMs with reduced computational cost and time.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Key R&D Program of China (2022YTB-401400//70) SSTIC (TT2(KK/2D202300 1.02111225222222222222222222 2.KIZZ202339231
149114491,,,,22222224200 1114400088)))
This work supported by National Key R&D Projects (NO. 2024YFC3307100) NSFC Grant (No.
62576364. No. U2541229), Shenzhen Basic Research Project (Natural Science Foundation) Basic Research Key Project (NO. JCY1202221122222444441 Dr Tao's 's research partially supported by NTU RSR and Start Up Grants.
REFERENCES
Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2010.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.088: 0037477744 2023.
Arash Ahmadian, Seraphina Goldfarb-Tamen Beyza Ermis, Marzieh Fadace, Sara Hooker, el. Mix data or merge models? optimizing for diverse multi-task learning, arXiv preprint arXiv:2410.10800 2024.
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 2025.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawa- har, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, 2019.
Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In COLING, 2022.
Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, et al. Model composition for multimodal large language models. In ACL. 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In NeurIPS, 2024b.
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-trate pre-training with acoustic tokenizers. In ICML, 2023.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. TabFact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909, 01.99.02244. 2019.
Yan-Lun Chen, Yi-Ru Wei. Chia-Yi Hsu, Chia-Mu Yu, Chun Ying Huang, Ying-Dac Lin, Yu-Sung Wu, and Wei-Bin Lee. Layer-aware task arithmetic: Disentangling task-specific and instruction- following knowledge. arXiv preprint 200 2025.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024c.
Zhuokun Chen, Jinwu Hu, Zeshuai Deng, Yufeng Wang, Bohan Zhuang, and Mingkui Tan. En- hancing perception capabilities of multimodal Ilms with training-free fusion. arXiv preprint arXiv:2412.01289, 2024d.
Runxi Cheng, Feng Xiong, Yongxian Wei, Wanyun Zhu, and Chun Yuan. Whoever started the interference should end it: Guiding data-free model merging via task vectors. In ICML, 2025.
Jiho Choi, Donggyun Kim, Chanhyuk Lee, and Seunghoon Hong. Revisiting weight averaging for model merging. arXiv preprint 2r1iv: /2/53, 2024.

---

Published as a conference paper at ICLR 2026
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-fnetuued language models. JMLR, 2024.
Nico Daheim, Thomas Möllenhoff, Edoardo Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by uncertainty-based gradient matching. In ICLR, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In NeurlPS, 2023.
Sinan Du, Jiahao Guo, Bo Li, Shuhao Cui, Zhengzhuo Xu, Yifu Luo, Yongxian Wei, Kun Gai, Xinggang Wang, Kai Wu, et al. Vqrae: Representation quantization autoencoders for multimodal understanding, generation and reconstruction. arXiv preprint ar arXiv:2511.23388 2025a.
Yiyang Du, Xiaochen Wang, Chi Chen, Jiabo Ye, Yiru Wang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Zhifang Sui, et al. AdaMMS: Model merging for heterogeneous multimodal large language models with unsupervised coefficient optimization. In CVPR, 2025b.
Haodong Duan, Junming Yang. Yuxuan Qiao. Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong. Yuhang Zang. Pan Zhang, Jiaqi Wang, et al. VLMEvalKit: An open-source toolkit for evaluating large multi-modality models. MM, 2024.
Junfeng Fang, Houcheng Jiang, Kun Wang. Yunshan Ma, Xiang Wang, Xiangnan He, and Tat-seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. In ICLR, 2025.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2024.
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-LLaVA: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312. 2023.
Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà. Task singular vectors: Reducing task interference in model merging. CVPR, 2025.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: A toolkit for merging large language models. In EMNLP, 2024.
Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass. Listen, think, and understand, ICLR, 2024.
Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD: General analysis and improved rates. In ICMI,, 2019.
Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
Yuchen Guan, Runxi Cheng, Kang Liu, and Chun Yuan. Enhancing logits distillation with plug&play kendall's 7 ranking loss. In ICML, 2025.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. VizWiz grand challenge: Answering visual questions from blind people. In CVPR, 2018
Yifei He, Yuzheng Hu, Yong Lin, Tong Zhang, and Han Zhao. Localize-and-stitch: Efficient model merging via sparse task arithmetic. TMLR, 2025.
Zixuan Hu, Yongxian Wei, Guozheng Ma, Jiahe Wang, Tianci Luo, Sinan Du, Runxi Cheng, Qianpu Sun, Haotian Luo, Xiaoyu Ma, et al. Synthetic data beyond generative models: A comprehensive survey of how, why, and where. 2026.
12

---

Published as a conference paper at ICLR 2026
Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. EMR-Merging: Tuning-fie free peg--performacce model merging. In NeurIPS, 2024.
Wenke Huang, Jian Liang, Xianda Guo, Yiyang Fang, Guancheng Wan, Xuankun Rong, Chi Wen, Zekun Shi, Qingyun Li, Didi Zhu, et al. Keeping yourself is important in downstream tuning multimodal large language model. arXiv preprini arXiv:2503. 2025.
Drew Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2.5-coder technical report. arXiv preprint arXiv:2409. 12186............ 2024.
Gabriel Ilhareo, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023.
Shixin Jiang. Jiafeng Liang, Ming Liu, and Bing Qin. From specific- to omni-mllm: A survey about the milms alligned with multi-modality. arXiv preprint arXiv:2412.1169. 2024.
Kushal Kafle, Brian Price. Scott Cohen, and Christopher Kanan. DVQA: Understanding data visualizations via question answering. In CVPR, 2018.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014.
Ahmed Khaled and Peter Richtárik. Better theory for SGD in the nonconvex world. TMLR, 2023.
Geewook Kim, Teakgyu Hong. Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonscok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In ECCV, 2022.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-onevision: Easy visual task transfer. TMLR, 2025a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench. Benchmarking multimodal large language models. In CVPR, 2024.
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning answer questions in dynamic audio-visual scenarios. In CVPR, 2022.
Lu Li, Tianyu Zhang, Zhiqi Bu, Suyuchen Wang, Huan He, Jie Fu, Yonghui Wu, Jiang Bian, Yong Chen, and Yoshua Bengio. MAP: Low-cempute model merging with amortized pareto fronts via quadratic approximation. ICLR, 2025b.
Zhong-Zhi Li, Duzhen Zhang, Ming- Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, al. From system system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502. /7419, 2025c.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Yuan. Video-LLaVA: Learning united visual representation by alignment before projection. In EMNLP, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024a.
13

---

Published as a conference paper at ICLR 2026
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is your multi-modal model an all-around player? In ECCV, 2024b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wes Chang, Michel Galley, and Jianfeng Gao. Math Vista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024a.
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-Merging: Dynamic integration of modular expertise in model merging. In NeurIPS, 2024b.
Ruipu Luo, Ziwang Zhao, Min Yang. Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207, 2023.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChttPPT: Towards detailed video understanding via large vision and language models. In ACI, 2024.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
Daniel Marezak, Simone Magistri, Sebastian Cygert, Barthomiej Twardowski, Andrew D Bagdanov, and Joost de Weijer. No task left behind: Isotropic model merging with common and task- specifiic subspaces. In ICML, 2025.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In CVPR, 2019.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, 2021.
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. WavCaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. TASLP, 2024.
Gabriele Merlin, Vedant Nanda, Ruchit Rawal, and Mariya Toneva. What happens during finetuning of vision transformers: An invariance based investigation. In CoI.As, 2023.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCRVQA: Visual question answering by reading text in images. In ICDAR, 2019.
Jingchen Ni, Keyu Lyu, Yu Guo, and Chun Yuan. Semantic alignment and hard sample retraining for visible-infrared person re-identification. In ICME, 2025.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In NeurlPS, 2023.
Artemis Panagopoulou, Le Xue, Ning Yu, Jumnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction- aware representations to llms and emergent cross-modal reasoning. arXiv preprint arXiv:2311.18799, 2023.
Haolong Qian, Xianliang Yang, Ling Zhang, Lei Song, Jiang Bian, and Chun Yuan. Nadro: Leverag- ing dual-reward strategies for llms training on noisy data. In NeurlPS, 2025.
Huaizhi Qu, Xinyu Zhao, Jie Peng, Kwonjoon Lee, Behzad Dariush, and Tianlong Chen. UQ-Merge: Uncertainty guided multimodal large language model merging. In ACL., 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., 2021.
14

---

Published as a conference paper at ICLR 2026
Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. UnIVAL: Unified model for image, video, audio and language tasks. TMLR, 2023.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. TextCaps: a dataset for image captioning with reading comprehension. In ECCV, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
Samuel L Smith, Benoit Dherin, David Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In ICLR, 2021.
Tomasz Stanistawek, Filip Graliński, Anna Wróblewska, Dawid Lipiński, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemyslaw Biecek. Kleister: key information extraction datasets involving long documents with complex layouts. In ICDAR, 2021.
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots. In ICLR, 2025.
Zihan Su, Hongyang Wei, Kangrui Cen, Yong Wang, Guanhua Chen, Chun Yuan, and Xiangxiang Chu. Generation enhances understanding in unified multimodal models via multirrepresentation generation. arXiv preprint arXiv:2601.2 22466 2026.
Qianpu Sun, Changyong Shu, Sifan Zhou, Runxi Cheng, Yongxian Wei, Zichen Yu, Dawei Yang, Sirui Han, and Yuan Chun. Gsrender: Deduplicated occupancy prediction via weakly supervised 3d gaussian splatting. arXiv preprini arXii:2412.14455 2024.
Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. An empirical study of multimodal model merging. In EMNLP, 2023.
Stacey Svetlichnaya. DeepForm: Understand structured documents at scale, 2020.
Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Fusionbench: A comprehensive benchmark of deep model fusion. arXiv preprint arXiv:24006.32288 03280, 2024a.
Anke Tang. Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. In ICML, 2024b.
Bohan Wang, Qi Meng, Huishuai Zhang, Ruoyu Sun, Wei Chen, Zhi-Ming Ma, and Tie-Ya Liu. Does momentum change the implicit regularization on separable data? In NeurIPS, 2022.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In NeurIPS, 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing dision-languse model's perception of the world at any resolution. arXiv preprint arXiv:2409.12199 12191, 2024b.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, al. CogVLM: Visual expert for pretrained language models. In NeurlPS, 2024c.
Yuanrui Wang, Cong Han, Yafei Li, Zhipeng Jin, Xiawei Li, SiNan Du, Wen Tao, Shuanglong Li, Yi Yang, Chun Yuan, et al. Uniglyph: Unified segmentation-condiiionn diffusion for precise visual text synthesis. In ICCV, 2025a.
Zile Wang, Hao Yu, Jiabo Zhan, and Chun Yuan. Alphavae: Unified end-to-end rgba image reconstrue- tion and generation with alpha-aware representation learning. arXiv preprint Xxxx::5577.00000., 2025b.
Hongyang Wei, Shuaizheng Liu, Chun Yuan, and Lei Zhang. Perceive, understand and restore: woal-world image super-resolution with autoregressive multimodal generative models. In ICCV, 2025a.
15

---

Published as a conference paper at ICLR 2026
Hongyang Wei, Hongbo Liu, Zidong Wang, Yi Peng, Baixin Xu, Size Wu, Xuying Zhang, Xianglong He, Zexiang Liu, Peiyu Wang, et al. Skywork unipic 3.0: Unified multi-image composition via sequence modeling. arXiv preprint arXiv:2601.1564, 2026.
Yongxian Wei, Zixuan Hu, Zhenyi Wang. Shen, Chun Yuan, and Dacheng Tao. FREE: Faster and better data-free meta-learning. In CVPR, 2024.
Yongxian Wei, Zixuan Hu, Li Shen, Zhenyi Wang, Chun Yuan, and Dacheng Tao. Open-vocabulary customization from CLIP via data-free knowledge distillation. In ICLR, 2025b.
Yongxian Wei, Anke Tang, Li Shen, Chun Yuan, and Xiaochun Cao. Modeling multi-task model merging as adaptive projective gradient descent. In ICML, 2025c.
Yongxian Wei, Yilin Zhao, Li Shen, Xinrui Chen, Runxi Cheng, Sinan Du, Hao Yu, Xiaohan Wang, Gang Liu, Jiahong Yan, et al. Learning to pose problems: Reasoning-driven and solver-adaptive data synthesis for large reasoning models. arXiv preprint wwxXiv2251.009900 2025d.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the the-ttheert natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML, 2022.
Chengyue Wu, Teng Wang, Yixiao Ge, Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Luo. r-tunnng: Transferring multimodal foundation models with optimal multi-task interpolation. In ICML, 2023.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES TIES-merging: Resolving interference when merging models. NeurIPS, 2023.
Prateck Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale? arXiv preprint arXiv:2410.036/7, 2024.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 12122, 2024a.
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in Ilms, milms, and beyond: Methods, theories, applications and opportunitics. arXiv preprint ar0.v:2408.000 076666, 2024b.
Enneng Yang, Li Shen, Zhenyi Wang. Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. In ICML, 2024c.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In ICLR, 2024d.
Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. AVQA: A dataset for audio-visual question answering on videos. In MM, 2022.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. RI RI-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025.
Hao Yu, Jiabo Zhan, Zile Wang, Jinglin Wang, Huaisong Zhang, Hongyu Li, Xinrui Chen, al. Omnialpha: A sequence-to te-eueee----seeeneeeeeee framework for unified multi-task rgba generation. arXiv preprint arXiv:2511.20211 2025.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICMI. 2024.
16

---

Published as a conference paper at ICLR 2026
Licheng Yu, Patrick Poirson, Shan Yang. Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, 2016.
Kaichen Zhang, Bo Li. Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Li, al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:240..12772 2024.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. LLaVAR: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17100 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurlPS, 2023.
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. LanguageBind: Extending video-language pretraining to n- modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023.
Junhao Zhuang, Shi Guo, Xin Cai, Xiaohui Li, Yihao Liu, Chun Yuan, and Tianfan Xue. Flashvsr: To- wards real-time diffusion- basddd streaming video super-resolution. arXiv preprint arXhv:550011247, 2025.
17

---

Published as a conference paper at ICLR 2026
A
THEORETICAL PROOFS
NOTATION AND SETTING
Tasks and losses. For tasks i, let the loss be Lg R R evaluated parameters O E R
Task vectors. For task i, after T steps (deterministic) gradient descent (GD) with fixed step size > 0 starting from a common initialization Q the task vector is
T
Merged update. Let the merged vector be
m
Tm aUTY
j=1
with nonnegative weights ox 0. We study the loss of task at the merged point θ Tm-
Norm and inner product. ||-1| is the Euclidean norm and (.. the Fuclidean inner product. For nonzero vectors u, n, cos(u, v) (u, v) /[[/w]] [[e])).
Assumption A.1 (L-smoothnesss) Each L; has L-Lipschitz continuous gradients; that is, for all O, O':
< |||@ [
[VV,(e)
O'll
Equivalently, for any A:
L < (O) (VL( + 2 |
Assumption A.2 (Polyak-E.ojasiewizl (PI.) condition). Each L; satisfies, for μ 0:
1 전
| - :
where C ;= inf (.).
Assumption A.3 (Directional similarity). For each d:
cos VL,(O), 2 kr K (0, 1],
cquivalently:
(V), 7) S VV) |T||]-
This ensures that the update is indeed a descent direction for task i, with alignment quantified by s.
Assumption A.4 (Approximate orthogonality). For all i + j:
cos(T,, T,) I e E 1).
Prior works (Thaaroo et al. 2023 Ortiz-Jimenez et al. 2023) show that task vectors are nearly orthogonal, which helps explain the success of model merging. This likely reflects general property of high-dimensional spaces: independent directions tend to be almost orthogonal. A small means that tasks nearly orthogonal in update space, reducing negative transfer effects.
18

---

Published as a conference paper at ICLR 2026
Assumption A.5 (Bounded gradients). There exists G considered:
such that for all i and all O
|V S.
This assumption widely adopted in the optimization literature Gower et al. 2019; Khaled Richtárik 2023), where similar boundedness conditions imposed to control the variance of stochastic gradients and to derive finite-step convergence rates.
Lemma A.6 (Cross-task cosine leakage). Under Assumptions and Tj * to ensure the cosine well bell-defined, for i + we have
A.3 A.4
with (e)
cos [V.(())=))| 5 6: v1
V1 V2- E?.
Use Assumption A.3 to get (u, uz) 2 K and Assumption 1.4gg to get (u; 12) then decompose and uj along v; and its orthogonal complement and apply Cauchy-Schee Schwarz.
Proof sketch. Normalize u = V VL |l. 1 = / -, IT
Lemma A.7 (PL convergence under GD). Under Assumptions A. A.2 and 1j (0,1/L) the GD iterates for task satisfy
(,(T) - C S (1 (1- (()
-
Proof. Let one step of GD be 0411 = 0 - -,,) By L-smoothness, for any æ,
L
(.(0) - (z) (VL(())) 2) lly z²|²
2
Plug x O,, y O.+1:
(,(0)))) < (,,) 1/ (1 Lnj
[V,,
Since S 1/L, we have
In >
2 2. thus
(,(0+1) L.)))) - (V.(
Apply the PL inequality:
|V,( 2 (0,) Li),
to get
0. - + s ((1 LT).
Unrolling the recursion yields the claim.
In Ei'd vL,()()) and [c, C
Lemma A.8 (Task vector norm). !ʃT, -1 all t, then
||Ty|| S nTG.
Proof. By the triangle inequality,
S T-1
|Ty S n
VC,
19

---

Published as a conference paper at ICLR 2026
Lemma A.9 (Inner-product lower bound). Under Assumptions A./ A.2 and vy E (0,1/L)
L ||
(VL.(0), (1-12-111)) (C:()) f.)
Proof. Apply the L-smooth upper bound with = T; and rearrange; then use Lemma A.7 to bound L, + Ti) (()
Theorem A.10 (Finite-step bound), Consider task trained for T iterations of gradient descent with fixed step size y [)) Let y m convergence factor. Then the merged update Tm == === OjTz satisfies
qu (0,1) denote the PL
c( & C. + +0+")) + 0(6 nT) + O
where ()7²) the residual error from incomplete convergence on task i, (ō the cross-task interference term, and TT) is the curvature term from L-smoothness.
Proof. Define the Tinndependent constant
C = - -,)).
By L-smoothness,
C. +m)) (V Tm) + ||
Decomposing the inner product yields
(V,,, T..) = a; (VL,, T.) + Σaz (VL,, T5)
jtt
For the self term, Lemma A.9 implies constant part absorbed into C, and residual term of order O(?T), plus a curvature correction T via Lemma A.8 For the cross terms, Lemma A.6 and Assumption A.5 give
(VC, Ty) < 8 G2
so the sum over j + is (T)) Finally, ||Tm.l S n/TG E, aj implies the smoothness term is TT)22)) Combining all contributions yields the stated bound.
Theorem A.11 (Bound in the near-convergence regime). Suppose the residual PL error after steps is below a given tolerance Ç >
(1 mn) (C:(O) 22) SC.
Equivalently,
(
T>
In(1 mpp)
Then the merged loss satisfies
C.( + Tm) +((+0
with the same C2 as in Theorem A.10
Proof. Starting from Theorem A.10 replace the residual term O(?T) by O()) using the near- convergence assumption. The tross-task and curvature terms remain unchanged.
20

---

Published as a conference paper at ICLR 2026
Remark:
When the learning rate is fixed and the model has not yet converged, the improvement from finetuning on the target task (captured by - -") typically outweighs the influence other task vectors, especially when these vectors close orthogonal (small hence small d). In this stage, merging different task updates remains stable and can be beneficial.
However, training approaches convergence, the potential negative impact from other task vectors becomes more significant. Even if each single-task loss continues to decrease, the merged model's loss worsen due accumulated cross-task interference, which grows linearly in T ())) and curvature effects from L-smoothness, which grow quadratically as O (172T²). This means that over-training on individual tasks can harm the quality the merged model.
In the convergence regime, a fixed learning rate can lead to increased norms of task vectors. While the single-task losses may remain similar, the larger norms amplify interference and curvature errors. further degrading merged performance. Once the PL. error (1 - nn4)² (C; L falls below tolerance Ç, the main residual terms are the interference and curvature contributions. Reducing directional leakage (small 8) and controlling the product 1/T are therefore essential for high-quality merging.
B MODEL MERGING BENCHMARKS
B. FINE-TUNING INFLUENCE ON MODEL MERGING
To demonstrate the sensitivity of model merging task vectors T; (i.e., parameter changes between fine-tuned models and the base), we conduct experiments using the standard CLIP- LiT merging benchmark, following the fine-tuning setup of FusionBench (Tang et al. [2024a). We train with Adam (learning rate le-5) for 4,000 steps with a batch size of 32. Models are saved every 500 iterations and evaluated for accuracy on the test dataset, as illustrated in Fig. Across eight tasks, convergence typically occurs around 3,000 steps.
Additionally, we evaluate four classical merging methods various fine-tuning stages, reporting their average accuracy in Fig. The results indicate that increasing the number of fine-tuning steps does not consistently enhance merging performance. Instead, performance typically improves initially before declining. This finding motivates our derivation of Theorem 3.1 which demonstrates that both the learning rate and the number of iterations affect model merging performance. MLLM training is typically organized by full passes over the data (epochs), rather than discrete iteration steps. Accordingly, we set the number epochs to and reduce the learning rate to limit parameter changes. This keeps the fine-tuned models close to the base model in parameter space while still yielding improvements on specific tasks.
100.0
Task Task- Specific Accuracy (%)
90.0
70.0
60.0
EuroSAT
MNIST
40.0
Starford Cars
SUN397
30.0
500
1000
1500
2000
2500
3000
3500
4000
Fine-tuning Steps
Figure 5: Accuracy of CLIP pre-trained VIT-B/32 fine-tuned separately eight downstream datasets. As training steps increase, performance on each dataset gradually converges.
21

---

Published as a conference paper at ICLR 2026
Task Arithmetic
Weight Average
Average Accuacy 850
67.0
70.6
70.5
60.6
68.4
72.4
66.2
65.0
5.0
1090
1900
2030
2500
3000
3500
4209
590
1000
1590
2000
2500
3000
3500
4009
Fine- Tine-tuning Steps
Fine-tuning Steps
70.8
DARE
83.6
TSV Merging
(ouras 180
Acuusit
72.6
Average Perrage Accurary
83.5
83.4
70.4
03.3
76-2
1980 Fine- Fine-tuning Steps
2000
25000
4009
1500
2000
2500
4009
Fine-tuning Steps
Figure 6: Average accuracy of different model merging methods across eight datasets. Increasing fine-tuning steps does not consistently improve merging performance; instead, performance tends to rise initially and then decline.
B.2 CHALLENGES OF MLLMS MERGING BENCHMARK
Disconnection between training and evaluation of MLLMs. Training and evaluation of MLLMs developed independently rather than being split from the same dataset. (i) Recent benchmarks (Liu et al. 2024b Li et al.) 2024] Fu et al., 2024) often assess models' comprehensive abilities through pre-defined choice questions, with each benchmark emphasizing different nuanced aspects. (ii) Domain-specific training data is also proprietary and confidential. Consequently, models demonstrate varied capabilities based on their training foundations. For example, LLaVA (Liu al. 2003) excels in conversational visual reasoning, while InstructBLIP (Dai et al. (003)) performss better on traditional short-answer VQA tasks. These differences present challenges for developing a unified benchmark suitable for multi-task model merging.
Trade-off between instruction-following and task-specific capabilities. Public vision datasets provide strong task-specific supervision but rarely use instruction cottruuctioooolooiinnn formats. Conversely, instruction data generated by models such GPT-4 (Achiam et al. 2023) often lacks task-specific grounding. This mismatch creates a trade-off between instruction adherence and task expertise.
Further SFT may lead to overfitting. Many publicly released models are already instruction tuned on diverse sources, including open-source, licensed, and private datasets. Additional supervised fine-tuning (SFT) therefore yields diminishing returns and can further overfit models to widely used training distributions (Huang et al. 2025).
C IMPLEMENTATION DETAILS
Checkpoint construction. For Intern -t-...---..-----strcccc we perform full fine-tuning with a learning rate 4e-5 and a warmup ratio of 3e-2. For Qwen2-VL-7B- rank of 8, learning rate of 1e-5, and warmup ratio of le-1. Both models are trained for one epoch using a cosine learning rate scheduler. Different training strategies and scales help evaluate the generalizability of merging methods.
VL-7B- VL-7B-Base, we apply LoRA fine-tuning with
22

---

Published as a conference paper at ICLR 2026
We follow prior work (Chen et al. 2024a) by pairing Vicuma-7B-v1.5 with modality-speciiff encoders and connectors. Separate models are trained using bi-modal data across three modalities: vision, audio, and video. Additional details are presented in Table IT The training approach consists of two phases: an alignment stage where only connector parameters are trainable, and a fine-tuning stage where we tune all connector and language model parameters. During fine-tuning, we apply LoRA with rank of 128 across all linear modules within the LLM. For our merging strategy, we preserve each modality's 's unique encoder and connector components while merging only the language model parameters, enabling the model to process inputs from all three modalities simultaneously.
Table 1I: Overview of modality components and training data.
Modality
Modality Encoder
Connector
Aligument Data
Fine-tuning Data
Referenced Work
Visson
CLIL-VIT-- 336px (Rafford al.
MLP
LC'S Liu
-aixed 665K (Liu al.
LLaVA-15 Liu ct al. 2024a
2023
2024a
Audio
BEATs- Chen 2023
Q-Former
WaveCaps
OpenAQA filtered 350K Gong
X- InstructBLIP (Panago at
400K (Mei
20224
2024
2024
Vidco
LanguageBind Zhu
MLP
LCS 558K
Video-ChatGPT 100K (Maaz
Video-LLaVA
et al. 2023
2023
Valley
at 2024 LLaVA-mixod
et 2024
702k
subsct 140K aau et 2024n
2023
Training data. Following (Chen et al., 2024c), we collect a broader range of domain-specific data, divided into VQA, Geometry, Chart, OCR, and Grounding tasks. For each dataset, we use only the training split, containing question, answer, and image. Samples exceeding 8192 tokens in combined question-answer length or with corrupted images are removed. The remaining data are converted into the ShareGPT instruction-tunin separately, so cross-task balancing is unnecessary. We collect at least 100k public samples per task to ensure diversity, following common practice for fine-tuning models in the IB-7B parameter range. Specifically, for grounding tasks, we map coordinates to the [0, 1000) range and add the special token notation <| <lboossttrt |>< box_end| [Wang et al. 2024b). During Qwen2-VL-Base fine-tuning, we observe that Chinese datasets consistently degraded performance, possibly due to lower data quality or because evaluation benchmarks are primarily in English. Consequently, we only English datasets for instruction tuning of Qwen2-VI-Bass Intern VL.2.5-Instrruc, already possessing multilingual catruction-followiin capabilities, is fine-tuned using all available data.
format. Tasks (e.g., VQA, OCR) are trained
Evaluation benchmark. We carefully select specialized datasets to evaluate distinct abilities across tasks. (i) For VQA, we utilize VizWiz (Gurari al. 2018) and GQA (Hudson & Manning, 2019) to assess general visual question answering proficiency. (ii) For Geometry. we incorporate multiple challenging subsets: "geometry reasoning". " "algebraic reasoning" and "geometry problem solving" from Math Vista (Lu et al.) 2024a), complemented by "metric geometry angle". "metric geometry area", "metric geometry- length" and "solid geometry" from MATH- Vision (Wang et al., al.] (024a). (iii) For Chart, we employ ChartQA (Masry et al. (022)), which tests reasoning and interpretation ability with charts and graphs. (iv) For OCR, our evaluation suite includes TextVQA (Singh et al.. 2019) and OCRVQA (Mishra al., 2019). (v) For Grounding, implement referring expression comprehension using RefCOCO (Kazemzadeh et al. 2014), RefCOCO+ (Kazemzadch et al. 2014). and RefCOCOg (Mao al., 2016), which require models identify specific objects in images based on natural language descriptions. All evaluation results are obtained using the VL.EvalKit (Duan) al. 2024) and LMMs-Eval (Zhang al. 2024) libraries under the same settings to ensure fair comparison. When evaluating Math Vista and MATH- Vision benchmarks, we utilize the GPT-4o-mini API extract answers from the output. The following prompt used, where (question) denotes the question text and (prediction) denotes the original output from the evaluated model.
Please read the following examples. Then extract the answer from the model response and type it the end of the prompt.
Hint: Please answer the question requiring an integer answer and provide the final value, e.g., 1, 2, 3, at the end. Queation: Which number is missing?
23

---

Published as a conference paper at ICLR 2026
Model reaponse: The number missing in the sequence is 14.
Extracted answer:
Hint: Please answer the question requiring a floating-point number with one decimal place and provide the final value, g.g., 1.2, 1.3, 1.4,
the end.
Question: What is the fraction of females facing the camera? Model response: The fraction of females facing the camera is 0.6, which means that six out of ten females in the group are facing the
Extracted answer:
Hint: Please answer the question requiring a floating-pan number with two decimal places and provide the final value, e.g., 1.23, 1.34, 1.45, at the end.
Question: How much money does Luca need to buy a sour apple candy and a butter-scotch candy? (Unit: $)
Model response: Luca needs $1.45 to buy a sour apple candy and butterscotch candy.
Extracted answer: 1.45
Hint: Please answer the question requiring a Python list as an answer and provide the final list, e.g., [1, 2, 31, [1.2, 1.3, 1.4), at the end.
Question: Between which two years does the line graph saw its maximum peak?
Model response: The line graph saw its maximum peak between 2007 and 2008.
Extracted answer: [2007, 2008]
Hint: Please answer the question and provide the correct option letter, .g., A, B, C, D, at the end.
Question: What fraction of the shape is blue?
Choices: (A) 3/11 (B) 8/11 (C) 6/11 (D) 3/5 Model response: The correct answer is (B) 8/11.
Extracted answer:
[question] Model response: (prediction)
Extracted answer:
For Omni-language models, select audio-visual question answering task. This task requires multimodal understanding and spatio-temporal reasoning over audio-visual scenes. AVQA Yang al., (002)) targets real-world objects and activities. MUSIC-AVQA (Li et al. 2022) specifically focuses on musical performances.
D DISCUSSIONS
D.1 UNDERSTANDING THE TASK VECTOR
WUDI Merging (Cheng al. 2025) substitutes the transpose of the task vector for the input We reconsider the update process the task vector, which can be formulated as follows:
00
T& D1=l
(4)
30.
ET
E1=1
11 Σ N -
0 "i
8C
0(0'-1
00
" n-1
(5)
ΣT
E 000 x
ac
(x)T1) 1),
(6)
coeflicient
24

---

Published as a conference paper at ICLR 2026
where Ti, denotes the task vector of task i in linear layer I, and 7 represents the parameters of task i in linear layer I at time 1. Each parameter in the linear layer can be interpreted as weighted sum of input vectors across training iterations, with gradients serving as coefficients.
D.2 LIMITATION AND FUTURE WORK
Due to resource constraints, our experiments were limited to models of 7B parameters. The public datasets we collected may contain lower-quality data. Future work will explore multilingual or reasoning- focusse MLLM merging, incorporating visual chain-of-though datasets YYng al.) 2025 Li et al. 2025c to support expert reasoning models. For evaluation, we plan to develop new benchmarks specifically designed assess the reasoning capabilities of MLLMs.
D.3 BROADER IMPACTS
Various developers release fine-tuned models open-source platforms such Hugging Face (Wolf et al. 2019 Wei et al. 2025b). Model merging reduces storage and serving costs through model reuse and helps preserve data privacy. It also supports decentralized development by enabling independent contributors to train models that can later be merged. We hope this benchmark will help the model merging community better evaluate the generalizability of their methods and accelerate progress in MLLM development.
E
LLM USAGE
This study utilizes large language models to correct grammatical errors.
F
REPRODUCIBILITY STATEMENT
We have open-sourced the code and checkpoints, and provided a detailed description of the imple- mentation details.
25


Reviews:
[{"review_id": "NHoKhE97uV", "reviewer": "Reviewer_Kqrb", "summary": "This paper focuses on the \"no data\" model fusion problem in multi-modal large language models (MLLMs), with main contributions including:\n- Establishing the first benchmark for model fusion aimed at MLLMs, covering five capability tasks: VQA, Geometry, Chart, OCR, and Grounding, as well as visual-audio-video tri-modal fusion scenarios, and publicly releasing the corresponding data and checkpoints;\n- Proposing the OptMerge method, which enhances fusion stability and performance based on the existing WUDI-Merging by low-rank denoising of task vectors, introducing SGD, and mean initialization strategies;\n- OptMerge outperforms 10 existing baselines in multi-task and multi-modal scenarios, exceeding mixed training methods in certain tasks while having significantly lower computational and storage costs. The paper also provides an O(ηT) theoretical analysis linking fine-tuning steps/learning rates to fusion upper bound errors.", "questions": "1. How sensitive is k (the SVD truncation rank)? It is suggested to provide ablation studies with different k values.\n2. λ only globally searches 6 points; has there been an attempt at hierarchical λ?\n3. The mathematical evaluation answer analysis relies on GPT-4o-mini; what is the complete prompt?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Sufficient motivation for the problem: The value of model fusion in terms of storage, inference costs, and community collaboration is prominent.\n2. Detailed benchmark construction: Adequate data volume, clear task decomposition, covering both LoRA and full parameter fine-tuning scenarios.\n3. The paper is generally easy to read.", "weaknesses": "1. General algorithm innovation: The core ideas (SVD low-rank truncation, SGD implicit regularization, mean initialization) are relatively common and represent engineering improvements.\n2. The \"no data\" expression is somewhat exaggerated: λ relies on validation set grid search, and the choice of k also references test set statistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RWbKTZ9nvG", "reviewer": "Reviewer_3WMa", "summary": "This paper focuses on **low-cost integration of specialized MLLMs** (e.g., VQA, OCR, audio-language models) into a unified, high-performance model via model merging, addressing pain points like fragmented domain models and high retraining costs for new modalities. Its core contributions are as follows:  \n\n### 1. First MLLM Merging Benchmark  \nIt establishes the first standardized benchmark for MLLM merging, covering:  \n- 5 key MLLM capabilities (VQA, Geometry, Chart, OCR, Grounding) and cross-modal scenarios (vision/audio/video-language);  \n- Checkpoints of 2 mainstream models (InternVL2.5, Qwen2-VL) with 2 fine-tuning formats (full-tuning, LoRA);  \n- Public training/evaluation data and code for reproducibility.  \n\n\n### 2. Novel Merging Method (OptMerge)  \nOptMerge solves noise and instability in existing merging:  \n- For full-tuned models: Uses SVD to denoise task vectors, retaining core knowledge;  \n- For LoRA models: Adopts SGD optimization and truncated SVD to avoid unstable vector scaling;  \n- Outperforms baseline methods (e.g., WUDI, TIES) by 2.48% on average.  \n\n\n### 3. Validating Merging Feasibility  \nExtensive experiments show:  \n- Merged models outperform single specialists and approach mixed-data fine-tuning results;  \n- Cross-modal merging (e.g., vision+audio) boosts multimodal tasks (e.g., Audio-VQA accuracy);  \n- 8.6% less memory and 84% less time than mixed-data training.", "questions": "Same as the Section of **Weaknesses**.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### 1. Well-Aligned and Impactful Research Motivation  \nThe paper addresses two long-standing, practical pain points in MLLM development that prior work has largely overlooked: (1) the fragmentation of domain-specialized MLLMs (e.g., VQA, OCR, geometry reasoning) in open-source communities, which incurs high storage/deployment costs and fails to leverage cross-task synergy; (2) the lack of standardized benchmarks for MLLM merging—existing frameworks focus on single-modal LLMs or vision classifiers, leaving MLLM-specific merging (e.g., cross-modality integration, LoRA compatibility) unevaluable. By directly targeting these gaps, the research motivation is highly relevant to both academic progress and industrial deployment, laying a clear foundation for its significance.  \n\n\n### 2. Strong Originality with Two Key Innovations  \nThe paper demonstrates notable originality in both benchmark design and merging methodology:  \n- **First MLLM-Specific Merging Benchmark**: Unlike generic MLLM evaluation benchmarks (e.g., MME, LLaVA-bench) that only measure single-model performance, this work constructs the first benchmark tailored for merging—covering 5 core MLLM capabilities (VQA, Geometry, etc.) and cross-modal scenarios (vision/audio/video-language), with publicly available checkpoints (full-tuning/LoRA) and datasets. This fills a critical void in standardizing MLLM merging research.  \n- **OptMerge: Fusion-Method Adaptation for MLLMs**: Instead of repurposing single-modal merging techniques (e.g., TIES, WUDI), OptMerge addresses MLLM-specific challenges: it uses SVD denoising for full-tuned models (to eliminate redundant task-vector noise) and SGD+truncated SVD for LoRA models (to avoid unstable vector scaling). This targeted design—rather than one-size-fits-all—removes key limitations of prior methods in MLLM scenarios.  \n\n\n### 3. Rigorous and Reproducible Experimental Design  \nThe experimental work exhibits high quality and clarity, supporting its conclusions convincingly:  \n- **Comprehensive Comparisons**: It evaluates 10 mainstream merging methods (e.g., Weight Average, Task Arithmetic) across 3 base models (InternVL2.5, Qwen2-VL, Vicuna-7B) and 2 fine-tuning formats, controlling variables (e.g., learning rate, hardware: 8×V100) to ensure fair comparisons.  \n- **Full Reproducibility**: All checkpoints, training/evaluation datasets, and code are made public. Key details (e.g., SVD top-k selection, optimizer parameters) are explicitly reported, eliminating ambiguity for follow-up work.  \n- **Theoretical + Empirical Support**: Beyond experimental results, the paper derives a theorem (Theorem 3.1) linking fine-tuning intensity (learning rate/epochs) to merging performance, providing a theoretical basis for model selection—avoiding purely empirical-driven conclusions. Results are also presented clearly via tables/figures (e.g., cost comparisons vs. mixed-data training), enhancing readability.", "weaknesses": "## 1. The MLLM merging benchmark lacks coverage for practical scenarios  \nThe benchmark’s design is restricted in two key ways that limit its utility for real-world merging:  \n- **Model scale gap**: Experiments only use small-to-medium parameter models (1B InternVL2.5, 7B Qwen2-VL/Vicuna), while practical deployments rely on large-scale MLLMs (70B+; e.g., Qwen2-VL-72B). Large models have unique traits (higher parameter redundancy, sparser gradients) that may break OptMerge’s current logic (e.g., SVD denoising could become computationally prohibitive or fail to capture fine-grained task knowledge).  \n- **Training data bias**: The benchmark focuses heavily on vision-related tasks (VQA, geometry, OCR) but omits critical MLLM use cases—text-centric capabilities (multi-turn dialogue, summarization) and vertical domain data (medical imaging, industrial defect detection). This makes it hard to compare merging methods across the diverse scenarios MLLMs actually serve.  \n\n\n## 2. OptMerge’s generalization is unvalidated  \nThe benchmark’s limitations directly raise doubts about OptMerge’s practicality:  \n- The benchmark covers only 5 types of vision task, which fails to meet the comprehensive evaluation standards required for practical MLLM applications. \n- It lacks validation on non-vision tasks (text-centric/vertical domains), so it is unclear if OptMerge can retain task-specific knowledge outside the vision scope.  \n\n\nThese weaknesses are actionable, not fatal. The work’s core value—establishing a standardized MLLM merging benchmark and exploring data-free multimodal integration—remains highly impactful. Future improvements could expand the benchmark to 70B+ models, heterogeneous architectures, and more vision and non-vision tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wer0Vze217", "reviewer": "Reviewer_iMYB", "summary": "This paper introduces OptMerge, a novel, data-free method for unifying multiple specialized Multimodal Large Language Models (MLLMs) into a single, more capable model. The authors present a new MLLM merging benchmark that is the first to evaluate unifying both task-specific capabilities (like VQA, OCR, and Geometry) and different modalities (vision, audio, and video). The proposed OptMerge algorithm achieves an average performance gain of 2.48% by robustly optimizing the model parameters, using techniques like low-rank approximation to remove noise and stabilize the merging process.", "questions": "Could you explain the relationships between model merging and federated learning? It seems they are very similar.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Building benchmark for model merging in Multimodal LLMs (MLLMs). The paper introduces the first model merging benchmark specifically designed for MLLMs. It is a good contribution as the author claims that they are the first to evaluate the unification of both diverse task-specific capabilities (e.g., VQA, Geometry, Chart, OCR, and Grounding) and different modalities (vision, audio, and video).\n\n2. The proposed method OptMerge, is simple and effective. OptMerge method is data-free and exceptionally efficient compared to traditional \"mixture training\". It may be a cost-effective solution for developing powerful MLLMs.\n\n3. Experiments demonstrate the robustness and effectiveness of the method. The OptMerge algorithm demonstrates good empirical performance, outperforming 10 other merging methods and even matching or surpassing the results of full mixture training. Its strength lies in its tailored approach: it robustly handles full fine-tuned models by using low-rank approximation (SVD) to remove noise and effectively merges LoRA models by using techniques like SGD and mean initialization to prevent the common problem of norm explosion and performance collapse.", "weaknesses": "1. Lack of discussion on model scale. This paper's experiments are based on InternVL2.51B-Instruct and Qwen2-VL-7B-Base. However, they are different model series. To observe the generalization of the model scale, it is more reasonable to conduct experiments on the same model series. For example, InternV2.5 has 1B, 2B, and 8B model scales.\n\n2. Lack of evaluating the merged models on general multimodal QA tasks. This paper merges the checkpoints of 5 abilities, including VQA, Geometry, Chart, OCR, and Grounding. Does the merged show emergent integrated capabilities of these 5 abilities? For example, if a question requires both capabilities of OCR and Grounding, it can not be solved by the model with only the capability of OCR or Grounding, but the merged model may have integrated capabilities to solve it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "trYFGth4ZP", "reviewer": "Reviewer_4XK6", "summary": "This paper explores data-free model merging as a scalable alternative to retraining multimodal large language models (MLLMs). Model merging offers a cost-effective path to consolidate multiple expert models into a single MLLM capable of broader multimodal understanding.\n\nThe paper makes three main contributions.\n(1) Benchmark: It introduces the first comprehensive benchmark for MLLM model merging, covering diverse capabilities such as visual question answering (VQA), geometry, chart reasoning, OCR, and grounding.\n(2) Methodology: The authors propose OptMerge, an optimization-based algorithm that denoises and regularizes task vectors (parameter differences between fine-tuned and base models) through singular value decomposition and robust loss minimization. This design stabilizes data-free optimization, mitigates parameter interference, and adapts to both full and low-rank (LoRA) fine-tuning regimes.\n(3) Experiments: OptMerge achieves the best or second-best results, improving average performance by 2.48% over prior methods and even surpassing mixture-training baselines. It effectively merges heterogeneous modality-specific models (vision, audio, video), confirming strong cross-modal complementarity. The approach reduces training time by over 90% and memory by more than 10× compared with data mixing, showing significant efficiency gains.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This work introduces a new benchmark for Multimodal LLM (MLLM) model merging, covering five capabilities (VQA, Geometry, Chart, OCR, Grounding), with carefully curated training and evaluation datasets. authors proposes OptMerge, a new merging method that improves robustness via task vector denoising and SVD-based low-rank approximation.  The proposed method unifies capability and modality merging within one framework, including vision, audio, and video encoders, which has not been systematically explored in prior literature.\n\nAs for technical contributions, this work\n\n- Implements and evaluates 10 merging baselines, categorizing them into linear interpolation, sparsification, SVD, and optimization-based methods.\n- Provides theoretical analysis (Theorem 3.1) on how parameter drift during fine-tuning affects mergeability, offering insights into fine-tuning strategy design.\n- Includes ablation studies and detailed comparisons on both synthetic and real-world Hugging Face checkpoints, demonstrating consistent improvements.\n\nGenerally, this work demonstrates that model merging can outperform mixture training without any data, offering a scalable, resource-efficient solution to building Omni-MLLMs.", "weaknesses": "1. Unclear Motivation for a New Benchmark\nThe paper motivates the introduction of a new MLLM merging benchmark by arguing that no existing dataset separates model training and evaluation tasks. However, the necessity of building a new benchmark is not fully justified. Prior multimodal evaluation suites such as MMBench, MME, or VLMEvalKit already cover a broad range of MLLM capabilities (e.g., VQA, OCR, grounding). The authors could better clarify why existing benchmarks are insufficient—e.g., whether they lack fine-grained control of task specialization, incompatibility with merging-based evaluation, or insufficient modality coverage. Without this clarification, the benchmark may appear redundant rather than essential.\n\n2. Incomplete Description of Benchmark Construction and Data Distribution\nAlthough Table 1 summarizes datasets (e.g., GQA, GeoQA+, ChartQA, TextVQA, RefCOCO), the paper does not detail how these datasets are split, standardized, or balanced across tasks. The rationale for setting the minimum of 100k samples per task is not explained, nor is there any discussion on inter-task overlap or label consistency. Furthermore, the benchmark’s evaluation distribution (training–validation splits, number of samples per modality, or balance among English and Chinese data) remains vague. A more transparent benchmark construction pipeline—such as annotation filtering, normalization steps, or sampling policies—would greatly enhance reproducibility and fairness.\n\n3. Limited Novelty in Algorithmic Design\nWhile OptMerge introduces a low-rank denoising mechanism for task vectors, it conceptually extends prior optimization-based approaches such as WUDI Merging (Cheng et al., 2025) and TSV Merging (Gargiulo et al., 2025). The innovation mainly lies in the combination of SVD truncation and loss reformulation rather than a fundamentally new optimization paradigm. The paper could clarify theoretical differences or provide stronger justification for why its design uniquely improves over existing baselines. In addition, Can authors explain the meanin of O in Eq.2? It is hard for the reviewers such as me who are not familiar with related topics.\n\n4. Restricted Theoretical Depth\nTheoretical analysis (Theorem 3.1) only establishes an upper bound proportional to the fine-tuning learning rate and iteration count. However, this bound is loose and lacks quantitative verification or deeper connection to empirical results. A stronger theoretical framework—e.g., convergence analysis of the merged vector optimization—would improve credibility.\n\n5. Practical Deployment Considerations\nThe discussion overlooks issues like merging models with different tokenizer vocabularies, scaling to larger parameter sizes (e.g., 30B+), or stability under mixed fine-tuning frameworks. Addressing these limitations would make the contribution more applicable to real-world model integration scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)