Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking
Authors: Dengming Zhang
Xiaowen Ma
Zhen-Liang Ni
Zhenkai Wu
Han Shu
Xin Jiang
Xinghao Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Model Merging
Model Editing
Large Language Model
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Model merging, which combines multiple domain-specialized experts into a single model, offers a practical path to endow Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) with broad capabilities without the cost of joint training or serving many models. However, training-free methods rely on hand-tuned coefficients, whereas training-based methods primarily align parameters rather than downstream task behavior and typically treat all layers uniformly, ignoring inter-layer heterogeneity. We introduce Expert Merging, a training-light method that learns a small set of layer-wise coefficients using only unlabeled calibration data. The coefficients are optimized to explicitly align the merged model’s hidden states and logits with those of the corresponding experts, with a coefficient regularizer for stability and task-weighted losses for controllable trade-offs. To capture inter-layer variation, Expert Merging++ augments this design with importance-guided chunking: a normalized layer-importance metric, derived from learned coefficients, task-vector magnitudes, and parameter counts, allocates more chunk-wise coefficients to high-importance layers while keeping low-importance layers lightweight. The result is a label-free, parameter-efficient, and scalable approach to multi-expert model merging across LLMs and MLLMs. Across MLLM backbones (InternVL and Qwen2-VL) and the LLM backbone (Mistral), our method surpasses strong training-free and training-based merging baselines, with Expert Merging++ delivering further gains and, in some cases, even exceeding supervised Mixture Training. Our code is available at https://github.com/Littleor/ExpertMerging.

Published as a conference paper at ICLR 2026
EXPERT MERGING: MODEL MERGING WITH UNSU- PERVISED EXPERT ALIGNMENT AND IMPORTANCE-
GUIDED LAYER CHUNKING
Dengming Zhang Xiaowen Ma Zhenliang Ni
Zhenkai Wu1.2 Han Shu²
Xin Jiang' Xinghao Chen't
1 Huawei Technologies ZZejjang University
ABSTRACT
Model merging, which combines multiple domain-specialized experts into sin- gle model, offers a practical path to endow Large Language Models (LLMs) and Multimodal Large Language Models (MILMs) with broad capabilities without the cost of joint training or serving many models. However, training-free meth- ods rely on hand-tuned coefficients, whereas training-based methods primarily align parameters rather than downstream task behavior and typically treat all lay- ers uniformly, ignoring inter-layer heterogeneity. We introduce Expert Merging, a training-light method that learns small set of layer-wise coefficients using only unlabeled calibration data. The coefficients are optimized to explicitly align the merged model's hidden states and logits with those of the corresponding experts, with a coefficient regularizer for stability and task-weighted losses for control- lable trade-offs.. To capture inter-layer variation, Expert Merging++ augments this design with amportance-guidees chunking: a normalized layer-importance metric, derived from learned coefficients, task-vector magnitudes, and parameter counts, allocates more thunk-wise coefficients to high-importance layers while keeping low-importance layers lightweight. The result label-free, parameter- efficient, and scalable approach to multi-expert model merging across LLMs and MILMS. Across MLLM backbones (Intern and Qwen2-VL, VL) and the LLM backbone (Mistral), our method surpasses strong training-free fareing--reeaee and training- based merging baselines, with Expert Merging++ delivering further gains and, some cases, even exceeding supervised Mixture Training. Our code is available at https:://ithuue wtm.iluteeoc//ttteeeeeoe..ooo. /ttpe///t/////p/etttteeerrggigi
INTRODUCTION
Large language models (LLMs) show strong general abilities but often lag on domain-specific tasks that require specialized knowledge (Hadi et al. 2023 Ling et al., 2023). A standard remedy is Su- pervised Fine- Fine-Tuning (SFT) that turns a base model into domain domain-specffic expert models for code, math, or reasoning (Zhao et al. 2023 Dodge et al., 2020 Chung et al. 2024). However, maintain- ing and serving a separate model per domain imposes substantial storage, memory, and engineering overhead, hindering deployment at scale (Yadav et al., (024). Model merging offers an attractive alternative: combine multiple domain experts into a single multi-competent model while retaining
their respective strengths (Yu et al. 2024: Wortsman et al., 2022 I]harco et al. 2022). Existing model merging methods broadly fall into two groups: training free methods with predefined coeffi-
cients and training- based methods that learn coefficients or parameters (Wei et al. 2025).
Among training-free approaches, Task Arithmetic (Ilaarco et al.) 2022) as a classic method forms merged model by adding expert task vectors (i.e., subtracting the base model parameters from the SFT model parameters) to the base model with fixed, predefined coefficients. Building on this idea, subsequent training-free approaches (Yadav et al. (002)) Yu et al. 2024 Gargiulo et al. 2025) refine the merging process to mitigate parameter redundancy and interference, reporting measurable
"Equal contribution.
Corresponding author; xinghao.chen huawei.com

---

Published as a conference paper at ICLR 2026
improvements. However, these methods typically rely on hand-specified hyperparameters (e.g.. per- task coefficients), which are selected via manual tuning or grid search. More importantly, these methods often focus solely parameter-space alignment while neglecting task alignment.
Training- bbaedd methods address these limitations by learning merging parameters or coefficients through gradient-based optimization. WUDI (Cheng et al.) 2023) reduces cross-task interference by minimizing a layer-wise wise interference objective, improving merged model performance. Building on this idea, WUDI v2 (Wei et al.) 2025) performs singular value decomposition (SVD) before training to isolate task-salient signals from noise, further enhancing performance. Nevertheless, these approaches remain primarily parameter-space alignment procedures and overlook the merged model's 's downstream performance. AdaMerging (Yang et al.) 2024) optimizes task-wise or layer- wise coefficients by minimizing prediction entropy to obtain a favorable combination of coefficients. While this accounts for task performance, entropy minimization provides no explicit task alignment signal and can induce overconfidence under distribution shift; moreover, assigning a single trainable coefficient per layer ignores inter-layer heterogeneity, ultimately constraining performance.
To address both task alignment and inter-layer beterogeneity, we introduce Expert Merging and its extension, Expert Merging++. For task alignment, Expert Merging learns trainable layer-wise coef- ficients that align the merged model's hidden states and logits with those of the corresponding expert models small set of unlabeled inputs (5-10 samples), thereby explicitly matching downstream behavior. To mitigate distribution shifts during merging, we incorporate a coefficient regulariza- tion term that stabilizes optimization. For inter-layer heterogeneity, our analysis reveals substantial variation across layers in optimal coefficient magnitudes and parameter number. We therefore pro- pose a layer-importance metric quantify each layer's contribution and chunk-wise coefficient method: high-importance layers are partitioned into structured parameter chunks with distinct co- efficients, while ow-importancce layers remain lightweight. We evaluate these methods on LLMs and multimodal large language models (MLLMs): Expert Merging consistently outperforms strong training-free free and prior training-based baselines both settings, Expert Merging++ yields additional gains, and on MILMs the approach can even surpass supervised Mixture Training.
Our contributions are: (1) Expert Merging that learns layer-wise coefficients to align the merged model's hidden states and logits to expert models on unlabeled data, with coefficient regulariza- tion for stability; (2) Expert Merging++ with chunk-wise coefficients guided by a layer-importance metric to allocate more capacity by chunking figh-imprttance layers while keeping low-importance layers lightweight; and (3) comprehensive results and ablations on both an LLM (Mistral-7B) and MILMS (Intern VI., Qwen2-VI.) showing Expert Merging surpasses training-free and prior training- based baselines, with Expert Merging++ yielding further gains.
2 RELATED WORK
Model merging aims to combine multiple task-specifie models into a unified model with diverse ca- pabilities (Wortsman et al. 2022). This approach has gained significant attention the large model domain, particularly for large language models (Wei et al. 2025 Zhou et al.) 2024 Goddard et al., 2024), due to not requiring access to original training datasets and its cost-effective nature (Stoica al. 2023). Existing model merging approaches can be categorized into two main classes based on whether they require training: training-free methods and training-based methods (Wei et al. 2025).
2.1 TRAINING-FREE MODEL MERGING METHODS
Training- free methods (Wortsman et al. 2022 Gargiulo et al. erate directly on model parameters without requiring additional optimization procedures. These approaches typically manipulate pre-trained model weights through mathematical operations to achieve effective model combination. Task Arithmetic (Ilharco et al.) 2022) represents founda- tional approach in this category, computing task vectors for each task task-specific model by subtracting the base model parameters from fine-tuned parameters. The method then merges multiple task- specific models through predefined scaling coefficients. However, this approach often suffers from parameter interference when combining multiple tasks. TIES Merging (Yadav al. 2023) addresses the parameter conflict problem inherent in Task Arithmetic by proposing a sophisticated conflict resolution mechanism. The method analyzes parameter changes based on both magnitude and di-
2025 Marezak et al. 2025) op-
2

---

Published as a conference paper at ICLR 2026
Predefined Coefficient
Single per-lnyer learnalle confficient
Aduptive per-layer coefficient count
Th 0g- Psam
Bengee
Beac
Σ
6
82
An
of
B
A7
6
2
(a) Task Vector
(b) Task Artihmetic
(c) Expert Merging
(d) Expert Merging++
Figure I: Conceptual overview: (a) Task Vector, (b) Task Arithmetic (TA), (c) Expert Merging, and (d) Expert Merging++ with importance-guiie guided chunk- wise coefficients.
rection, trimming conflicting parameters before merging. Specifically, TIES identifies parameters with conflicting signs across different task vectors and resolves these conflicts through magnitude- based voting schemes. DARE (Yu et al., 2024) introduces different perspective by focusing on parameter sparsification during the merging process. The method applies drop and rescale opera- tions to task vectors, randomly dropping fraction of parameters and rescaling the remaining ones to maintain the expected magnitude. This approach reduces parameter interference by randomly dropping parameters, demonstrating that sparsification can actually improve merging performance. Despite their simplicity, training-free approaches typically rely on fixed, hand-tuned coefficients and operate purely in parameter space without explicit task-level alignment, making them sensitive to hyperparameters and prone to degraded performance when many experts are combined.
2.2 TRAINING-BASED MODEL MERGING METHODS
Training-based methods optimize model combination through gradient-based updates, enabling more adaptive merging strategies. These methods be further categorized by their data require- ments. Sample-independent methods eliminate the dependency on training samples while still em- ploying optimization. WUDI (Cheng et al.) 2025) reduces cross-task interference by minimizing layer-wise interference objective, aligning the merged model's activations with those of the experts and improving robustness. Building on this idea, WUDI v2 (Wei et al., 2025) performs singu- lar value decomposition (SVD) of task vectors before training to isolate task-salient signals from noise, further improving performance. Sample-dependent methods leverage data to optimize merg- ing parameters. AdaMerging (Yang et al.) 2024) optimizes task-wise or layer-wise coefficients by minimizing prediction entropy to obtain a favorable combination of coefficients. While effective, en- tropy minimization provides explicit task-alignn mask-alinmment signal and can induce overconfidence under distribution shifts; moreover, assigning a single trainable coefficient per layer overlooks inter-layer heterogeneity, which can limit performance.
Despite progress, most training-bast based methods emphasize parameter-space alignment or implicit distribution matching, provide limited control over task trade-offs, and ignore layer heterogeneity. We instead align hidden states/logits on unlabeled data with regularized layer-wise and importance- guided chunk- coefficients, improving alignment, trade-off control, and performance.
3
METHOD
We propose Expert Merging, a training-light method that merges multiple domain experts via layer- wise coefficients learned from unlabeled data to align hidden states and logits; Expert Merging++ further allocates capacity by layer importance with chunk-wiss coefficients (Figure II.
3.1 PRELIMINARIES
Let frase denote the parameters of a base model f(; Oosse) with L layers, and let (0x K be the parameters of K domain experts obtained by SFT on tasks (Tx 1 1 Define the task vector of expert k as Tı 02 Orase and its layer-l block as of. Training-fre free Task Arithmetic forms merged model by 0TA = Ornc + >k seek trainable, layer-wiss/chunkwwiii coefficients that learned from unlabeled data.
1 As Tk with predefined coefficients {A& (panel (b) of Figure l). In contrast,
3

---

Published as a conference paper at ICLR 2026
We assume access to an unlabeled, lask-partitionee calibration set D U Dj where Dj contains inputs reflecting the distribution of task Ta (text-only for LLMs; image+text for MLLMs). From D we learn layer-wise (and later chunk-wisel coefficients, with the objective of aligning the merged model's performance with expert lt on inputs x E DD.-
Limitations of existing trainable merging. Task vector based methods such as AdaMerging and WUDI Merging also introduce trainable coefficients on top of Task Arithmetic, but their objec- tives do not explicitly align the merged model with each expert on its own domain. For example, K
AdaMerging writes the merged parameters as (A) Orose + 2k=1 TRT Tk. and optimizes the coefficients on multi task unlabeled data via entropy minimization
K Σ
Cada (A)
Σ Jomn ME
H(ply) x; (A)),)
where H-) is Shannon entropy and B5 C. De is a batch of unlabeled inputs from task k. Minimizing LAda mainly pushes the merged model to make high confidence predictions, because any very peaked distribution has low entropy, even if the model becomes very confident about an answer that expert would not choose. In particular, very different logit distributions can share the same entropy, so the model can decrease LAda by becoming confidently wrong. Moreover, the entropy is averaged over all tasks, so the coefficients can be driven by tasks where confidence is easiest to increase, without any term that requires ply æ;; Ad (A)) to align with the expert distribution on Dp. As result, some experts may be underrepresented even when they are strong on their own domains.
WUDI Merging instead analyzes the interference between the merged model and each expert at selected linear layers. Let 0% and @eergeed denote the parameters of layer for expert and the merged model, and let 2 be the corresponding layer input. The interference on task layer & can be written as
Jk,e = Er,, II (meegged 0.
(2)
and WUDI Merging seeks merged model that minimizes this interference aggregated over tasks and layers. This controls a local linear discrepancy, but it is defined only on selected linear layers and does not directly match the full hidden trajectories or the predictive distributions, with small interference can still produce noticeably different predictions on complex, multi-modal tasks. These limitations motivate us to directly align the merged model with each expert at both the representation and prediction levels on task dask-partitioned calibration data, which leads to the hidden state and logit alignment losses in equation and equation
models
3.2 EXPERT MERGING
The key idea of Expert Merging is to learn per-layer coefficients by explicitly aligning the merged model's internal representations and predictions to those of the experts on unlabeled inputs from their respective domains. The end-to-end training pipeline is depicted in Figure 2 the base and expert models are frozen, the coefficients are the only trainable parameters, and alignment is applied on both hidden states and logits.
Parameterization. Expert Merging uses one trainable coefficient per layer (panel (c) of Figure [] For each layer f, we set 0f mmerged of + Σ of Tf- where of E IR is a learnable coefficient for expert & at layer f. This yields only K x L trainable coefficients, initialized with a fixed value āμ.
Alignment losses. For an input x, we denote the merged model's hidden state at layer e by hy; ( and its pre-softmax logits by z(x; meergdd) Likewise, h (k) (x)) and 2 (1r) denote the corresponding values from expert k. Specifically, for inputs x D; of task k, we minimize combination of hidden- state and logit alignment (green and yellow dashed connections in Figure 2
Lnd Σ EyWD, be(:;; Omergedd) - h1/ (x)
C
logit
2 EEND, KL (softmxx(())))))))))))))
softmax(z(x;;
mmerged) 0merge /TT/TT))
(4)
where S is a subset of layers used for hidden alignment (e.g., every Transformer layer's 's output), and T is the temperature. Hidden-state matching enforces intermediate alignment, and logit alignment
4

---

Published as a conference paper at ICLR 2026
x=
Math Question
Code Question
Frozen
Trainable
Align Loss
SFT Model
Code Model
Model
Merged Model
Logits Loss
Layer
Layer
DO- 00
00
Hidden State
Layer
O
Logits
Correct Answer
Correct Answer
Respoase
Figure 2: Architecture of Expert Merging++ The base and expert models are frozen (snowflakes) and only the chunk-wise coefficients {a } are trainable (flames). For unlabeled inputs from each domain, the merged model aligns both hidden states be (green) and logits z (yellow) to the corre- sponding expert, producing single model that preserves all task performance.
promotes task-faithful outputs, which are crucial for downstream performance. The logit loss is multi task alignment objective that minimizes the Kullback-Leibler divergence between the expert distribution and the merged model distribution Dx, so when experts are accurate this directly reduces the supervised loss without requiring labels. In contrast to entropy based objectives that only push predictions to be confident, equation encourages the merged model align with the expert on which predictions should be confident. The hidden state loss complements this by matching the full representation trajectories (he(z))ess les, which implicitly controls layer level interference terms such |10 "T. eel2 while also constraining non linear layers that are not covered by the original WUDI analysis.
Controllable task trade-offss Unlike prior trainable merging that implicitly balances tasks with- out offering any external control, we introduce explicit nonnegative task weights (3x to con- trol domain priority. Conceptually, each task contributes a weighted alignment signal: Latign Ek ( + (lgit) with 3x 2 O. Setting 33 modulates how strongly expert k is preserved, providing direct, interpretable control over cross-task trade-offs. This explicit weighting makes the trade-offs transparent and tunable, addressing the concern that entropy based or interference based objectives may implicitly favor some tasks over others without exposing a clear control knob.
Coefficient regularization. To stabilize optimization and mitigate distribution shifts, we regularize the coefficients around the initial value āg:
K
L
R(a) KL
a - ox)-
(5)
The overall objective is min(ag) Lalign 7 R((a), where 7 controls regularization strength. Fur- thermore, since the initial values have significant impact on the optimization process, we use coef- ficients validated by Task Arithmetic as the initial values. This regularizer can be viewed as prior that keeps the solution close to the training free Task Arithmetic point, and only allows coefficients to move away when the hidden and logit alignment losses provide consistent evidence. In practice this prevents overfitting the small calibration sets and avoids degenerate solutions that might arise optimized the coefficients using only entropy based interference based objectives.
3.3 EXPERT MERGING+
Layer-wist wise coefficients assume uniform importance across layers and parameter groups, which is violated in practice: (i) different layer types (e.g., attention vs. MLP) have very different parameter

---

Published as a conference paper at ICLR 2026
counts; (ii) even within type, deeper layers can be more influential; and (iii) tasks impact layers unevenly. Expert Merging++ addresses this by estimating layer importance and allocating finer- grained, chunk-wise coefficients where they matter most (panel (d) of Figure TT.
Layer-importance metric. After training Expert Merging, we compute a normalized importance score per layer based three factors: learned coefficient magnitude, task-vector weight, and param- eter count. Specifically, let sy denote the task-vector vector weight of expert & at layer l, computed from the SFT task vector (e.g., mean absolute magnitude $ mean( and normalized across layers.
Then 1 Norm (E | sf. nz), where n4 param cowntmmcou (2) is the number of parameters in layer é, and Norm(-) denotes l normalization to [0, 1] across layers. Intuitively, I is data driven proxy for how much the experts and the learned coefficients rely on layer & across tasks, so it gives principled signal for where additional merging capacity is likely to yield the largest gains.
Importance-guided chunk-wise coefficients. As we have calculated the importance of different layers Lt, the next step is to chunk the layers according to their importance, with more chunks representing more trainable coefficients. Given a total coefficient budget for each task, we allocate to layer l a chunk count
my B
2 0,
Ej-1
and partition its parameter tensors into mg parameter chunks. Here & is an allocation steepness hyperparameter: = 0 yields near-uniform allocation; K = allocates proportional to importance; > concentrates chunks on high-importance layers; 0 x < flattens the distribution. In implementation, we flatten the tensor and split it into my contiguous chunks. Each chunk receives a distinct trainable coefficient of for expert k. For some low-important layers, we set m and use a fixed, untrainable scalar, preserving compact parameterization.
Objective. We reuse the alignment objective with chunk-wise coefficients in equation and equation Let (-) denote chunk-selection operator that extracts chunk from the layer 7, and
define kkss Se,a(TE): For each chunk k.
we will have a trainable chunk-wist coefficient of
Then
m
of merged
0f
c TK."
For brevity set a (a ................ a k. and Th (Th (T Th kmmm)) shorthand also used in Fig- ure The loss is identical Section 3.2 replacing at by xh.-- Regularization includes a chunk- 1. aware term BK Exe., jaf aæ which prevents the chunk-wise coefficients from deviating ex-
cessively from their initial distribution.
importannc-guidded chunking allocates learnable capacity to layers that matter most, improving per- formance especially when there are significant differences among experts or layers (as observed in our experiments). Moreover, since the value of B remains close number of chunk-wise coefficients is nearly the same as that of the layer-wise coefficients, thereby preserving strong sparsity in the merged model while improving the performance. Notably, Expert Merging++ is an enhanced version of Expert Merging. In practice, we first run Expert Merging to learn layer-wise coefficients, then use these learned coefficients to compute the layer-importance scores I and determine how many chunks my each layer receives. In the second stage, we keep the base and expert models frozen and optimize only the chunk-wise coefficients with the same hidden-staie and logit alignment objectives as before.
practice (0.9-1.2), even the
EXPERIMENT
4.1 SETUP
Backbones. We evaluate our method on both LLMs and MLLMs. For LLMs, we use Mistral-7B- v0.1 (Tiang al. 2023) the base model and experts specialized for three capabilities: instruc-

---

Published as a conference paper at ICLR 2026
tion following (Chat) mathematical reasoning (Math P and code generation (Code For MLLMs, (Chen et al, following prior merging methods (Wei et al. (025), we adopt VL2.5-1
2024 and Owen2-VL-7B-Bast (Wang et al. (2024b) as backbones, and use experts for five capa- bilities: visual question answering (VQA), geometry reasoning (Geometry), chart understanding (Chart), optical character recognition (OCR), and visual grounding (Grounding).
Benchmarks. For LLMs, we assess instruction following on AlpacaEval (Li et al. 2023 mathe-
matical reasoning on GSM8K (Cobbe et al., (021) and MATH (Hendrycks al., 2021). and code generation on HumanEval (Chen et al. (021) and MBPP (Austin et al. [022]) ) Unless otherwise noted, we report zero-shot accuracy; for code generation we use pass to measure functional correctness. For MLLMs, we evaluate VQA on VizWiz (Gurari et al. (2018) and GQA (Hudson
& Manning 2019), Geometry on Matha (Lu al., (024) and MATH- 2024a), Chart on ChartQA (Masry et al.) 2022), OCR on TextVQa (Singh et al.) 2219)) and OCRVQA (Mishra et al. 2019), and Grounding on RefCOCO (Kazemzadeh et al. 2014
(Wang et al,
COCO+Kazemzaadel Kazemzadeh et al. |014//e/COOOOg (Mao al. 2016). We run the MLLM evaluations using the VLMEvalKit (Duan et al.) 2024) and EvMs-Eval (Zhang et al., 2024) toolchains under matched settings to ensure fair comparison. When evaluating AlpacaEval, MathVista, and MATH- Vision, we employ the GPT-40-mini API to evaluate or extract answers from the output.
Baselines. We compare against strong training-free free and training- based model meeeengg baselines:
Weight Average (Wortsman et al. 2022). Task Arithmetic (TA) (Iharcco et al. 2022). TIES Merg-
ing (Yadav et al.
2023), DARE Yuu al.; 2024), TSV Merging (Gargiulo et al. 2025). ), Iso-C (Mar-
czak et al. 2025). WUDI Merging (Cheng et al. 2025), WUDI v2 (Wei et al. 2025). and AdaMerg-
ing/AdaMergging++ (Yang et al. 2024). Detailed baseline descriptions are provided in Appendix B
Implementation details. For each task, we sample 5-10 unlabeled examples from the training or test sets as the dataset. We first perform Expert Merging, and then apply Expert Merging++ using the learned coefficients. All experiments are repeated five times and results are averaged, using 8x 32G GPUs. For optimizing the Expert Merging/+ objective, we use the AdamW optimizer with a learning rate of 5 x 10-1 and weight decay of 1 x 10-5, and run 5 epochs with early stopping. The learning rate follows a cosine annealing schedule (Loshchilov & Hutter (016) over optimization steps. Additional hyperparameters and details are provided in Appendix B.2
4.2
MAIN RESULTS
Across both the MLLM and LLM backbones, our approach delivers the strongest overall perfor- mance while using only unlabeled data for calibration. On InternVL (Table I). Expert Merging++ at- tains the best average of 58.45, outperforming the best prior merging method (+1.49 over WUDI v2) and even exceeding supervised Mixture Training (+0.79). The gains are particularly pronounced on grounding benchmarks, where Experi Merging/++ achieves 80.05/80.53 on RefCOCO, 73. 73.85/74..7 on RefCOCO+, and 79.04/79.31 on RefCOCOg, substantially surpassing all baselines. At the same time, VQA performance remains competitive (e.g.. 31.16 on VizWiz, the best in table) and geometry scores are strong (26.32 on MATH-Vision with Expert Merging), indicating that the method effec- tively balances recognition, reasoning, and localization. While certain OCR-centric metrics (c.g., OCRVQA) favor Mixture Training, the overall average and grounding improvements demonstrate superior cross- domain integration. Additional analysis and results are provided in Appendix [
Results on Qwen2- VL (Table 2 show the same pattern. Expert Merging++ achieves the best av- erage of 63.63, outperforming the strongest prior merging baseline (+1.00 over WUDI v2) and Mixture Training (+1.40). It sets or matches state-of-the-art results on several domains, including MATH-Vision (44.74) and TextVQA (81.65), while delivering excellent grounding (c.g.. 79.00 on RefCoCOg), Although some methods peak on individual tasks (e.g.. TA+DARE on ChartQA), they do so at the expense of other domains, whereas Expert Merging/++ maintains consistently high per- formance across VQA, geometry, OCR, and grounding, yielding the best overall trade-off. Across Intern and DWen2-VL, Expert Merging++ thus consistently attains the highest average over ten evaluation tasks while remaining competitive on each individual benchmark.
httas:///uggingfaac....//////// caae-aaaaaaaaappaaaaaaa 1paca- lora
7

---

Published as a conference paper at ICLR 2026
Table 1:
Main results on Intern across multiple tasks.
VQA
Geometry
Chart
OCR
Grounding
Method
VieWiz GQA
VittiVista MATH-Vision
ChariQA
TextVQ OCRYQA ReCCOCO ReCcoCO+ ReCCOCOg
InternV1.2 IntermV1.2.-Isstraut
29.15
54.62
46.80
18.42
69.48
72.51
41.08
71.89
65.41
67.40
53.65
Expert
60.91
36.04
Geometry Expert
13,45
52.80
55.20
25.00
51.76
56.91
35.35
24.73
19.61
23.81
33.86
Chart Expert
20,16
40.39
23,84
10.53
69.52
54.36
34,83
36.23
Expert
10.53
73.00
54.79
73.68
68.01
Grounding Expert
19.09
25.88
28.91
14.47
41.32
58.39
74.87
76.67
71.35
70.09
48.10
Trudning- Free Madel Merging Methoul
Weight Average
29.96
51.89
19.60
18:42
71.61
74.51
11.96
$2.62
15,29
52.39
49.12
Task Arithmetic
56.34
56.17
TIRS Merging
30.63
56.48
44.50
23.68
72.28
76.29
44.01
76.01
58.44 15.45
73.69
56.59
TA DARE
30.61
56.48
48.45
21,05
73.08
76.30
43.03
74.94
68.00
73.02
56.50
TIES w/ DARE
30.65
56.11
13.85
27.63
72.72
76.19
43.33
75.10
68.48
73.55
56.76
TSV Merging
31,15
$6.67
52.45
28,95
70.56
75.66
45.38
65.19
58.51
59.17
54.36
28.21
55.36
72.72
66.56
Training- Rosed Model Merging Methods
Merging
56.07
WUDI v2
31,00
57.23
52.61
21.05
65.32
76.20
45.83
76.30
69.91
74.16
56,96
AdaMerging
57.20
AduMerging++
30.94
57.18
51.77
19.74
66.20
76.02
45.93
76.45
70.47
74.41
56,91
Mixture Training
29.79
61.33
52.83
23.68
70.32
72,96
60.25
72.06
65.93
67.46
57.66
Expert Merging
31.16
56.77
47.71
26.32
65.28
75.20
45.31
80.05
23.85
79.04
58.11
Expert Merging+
31,14
56.53
52.35
22.37
65.92
76.06
46.00
80.53
74.37
79.31
58.45
Table 2: Main results on Qwen2-VL. across multiple tasks,
VQA
Goumetry
Grounding
Method
VizWiz GQA
Math
MATH- Vision
ChariQA
TexVVQA OCRVOA RaCCOCO ReCCOCO+ RdCOCCO Avg.
Owen2-VL -LeBaee
5.52
5.39
47.85
23.68
0.36
20.22
1.07
45.32
37.55
31.26
21.82
VQA Expert
41.38
62.60
33.71
28.94
66.56
80.21
55.33
39:31
32.71
38.01
47.88
Geumetry Expert
35.57
44.63
42.50
28.95
14.56
73.95
45.96
5.57
2.31
3.90
29.79
Chart Expert
38.58
24:24
49.28
32,89
61.08
79.75
63.67
46.28
36.67
34.06
46.65
OCR Expert
28.38
37.53
31.81
13.16
$7.40
70.50
64.68
0.59
0.46
0.26
30.48
Grounding Expert
38.60
32.92
36.17
19.74
18.08
75.05
48.27
72.14
65.33
66.48
47.28
Training-Free Free Model Merging Methods
34.21
60.55
Task Arithmetic
40.52
62.31
40.36
26.31
79.62
81.09
59.50
75.96
61.33
75.85
60.29
TIES Merging
41.38
59.08
46.87
34.21
67.24
81.42
58.53
80.63
65.36
77.65
61.24
TA DARE
40.64
62.38
40.67
26.31
79.76
81.04
59.34
75.83
61.41
75,80
60.32
TIES DARE
41.63
$9.96
45.72
35.53
70.68
81.53
59.63
80.73
65.65
77.77
61,88
TSV Merging
41.43
57.31
$1.05
34.21
59.44
81.25
57,81
80.71
65.34
77.76
60.63
Iso-C
12.31
13.44
39.96
27.63
2.80
30.05
6.12
53.68
38.96
41,90
26.69
Training Bared Model Merging Method'
WUDI Merging
37,19
56.45
42,96
27.63
67.
79.
65.56
76.25
60.72
71.99
58.65
WUDI v2
41.13
61.17
46.98
38.
74.54
79.54
60.03
80.43
65.85
78.42
62.63
AdaMerging
41.83
60.28
49.08
35.53
71.68
81.52
60.22
53.31
47.27
57.62
55,83
AdaMergingt+
41.67
60.09
47.82
35.53
69.76
81.59
60.42
80.57
65.98
78.74
62.22
Misture Training
44.09
62.18
46.02
19.73
70.04
78.38
65.42
82,89
77.87
75.63
62.23
Expert Merging
41.54
60:51
48.80
42.11
73.00
81.41
60.19
80.64
66.15
79.02
63.34
Expert Merging++
41.49
60.42
49.33
44.74
72.60
81.65
60.19
80.71
66.21
79.00
63.63
For Mistral-7B (Table 3. Expert Merging+ attains the highest average (48.71), improving over strong baselines such as WUDI v2 (+3.23) and AdaMerging++ (+3.61). It achieves the best Code score among merged models (53.66 on HvmanEval), with strong performance AlpacaE- val (75.51) and GSM8K (59.44). Notably, Expert Merging attains the highest Chat score overall (77.84 AlpacaEval). Our merged models approach match specialist experts within their do- mains while consolidating all capabilities into a single model, validating the effectiveness of our coefficient learning and alignment objectives. Together, these results support our key claim: align- ing both hidden states and logits with per-layer coefficients yields a robust, training-efficien path to unify domain experts. The chunk-wise coefficients of Expert Merging++ (guided by the importance analysis in Section 4.3) further improve cross cross-domain balance with negligible overhead, leading to state-of-the-as ahr averages across backbones without supervised finetuning.
4.3 LAYER-WISH IMPORTANCE ANALYSIS
Merging performance is not uniform across layers or depths. To investigate this heterogeneity, examined the coefficient magnitudes and parameter sizes of each layer across different component types (attention projections Q/K/V/O and MLP Cate/Up//Dww)) and depths (early, middle, late). We
8

---

Published as a conference paper at ICLR 2026
Citte
Up
Down
Coeffiiient Mugnitude
Parameter Size
0.2
0.15
0.14
0.13
IntenVL
Qwen2-VL
Mistral
InternVL.
Qwan2-VV 2VLL
Mistral
Purameter
Size
Coefficient Magnitude
Figure 3: Model-wise parameter size and learned coefficient across backbones after normalizing. Full stage-wise trends for panel (b) are provided in Appendix (Figure 5
Up
0.00
Barly
Midele
Early
Middle
Late
Eurly
Middle
Late
InternVL
Qwen2-VL
Mistral
Figure 4: Layer importance by stage and submodule across backbones.
then combined these statistics with the task vector weights to derive per-layer importance score, quantifying each layer's contribution to the final merged model.
Parameter size across backbones. Despite architectural differences. the distribution of parameters by submodule type highly consistent (panel (a) of Figure 3 In Qwen and InternV,, the MLP dominates capacity: Gate/Up/Down together account for roughly 87% of the parameters, while attention uses the remaining 13%. Mistral has slightly larger attention parameters (about 19%) but the same qualitative pattern. This structure suggests that most of the representational capacity resides in the MLP, with Q/O providing relatively smaller but non-negligible attention-side capacity.
Learned coefficient magnitude. After malizing the learned coefficients within each model. we aggregate them across submodule to obtain panel (b) of Figure The magni- tudes are fairly balanced overall, but the lead- ing submodule varies by backbone: Down is largest for InternVL, Q for Qwen2-VL, and K for Mistral. The O projection is consistently the
Table Main results on Mistral across multiple tasks.
Chat
MATH
Code
Method
Alpuca Eval
GSM8K Math
HomanBsal MBPP
Avg.
Mistral-TB-vtt.
5.10
30.49
Chat Expert
31.37
41.09
7.26
35.37
32:20
37.46
Expert
53.18
Code Expert
69.00
30.78
4.84
54.88
38.70
39.72
Training
Merging
smallest across all three. Notably, Down ex- hibits the greatest variance; it is highest In- VL while being lowest on Mistral. This in- dicates bacchiie- allocation rather than a universal ranking. This pattern contrasts with panel (a), where MLP (Gate/Uph (Batt// clearly dominates by parameter size.
Task Arithmetic
74.18
$2.69
13.16
45.12
26.00
42.23
TIFS Merging
71.66
53.68
13,86
38.41
40 10.00
43.50
DARE
13.32
TIES DARE
76.33
52.08
12.00
34.76
43,40
43.71
ISY Merging
60.61
47.38
11.18
37,20
42.20
39.71
Iso-C
69.76
12.82
40.50
Training Based Model Merging Metheds
WUDI
64.95
53.37
14.04
32,99
43,40
WUDD
14.88
AdaMerging
75.94
5481
12:14
43.29
39.00
45.06
AdaMerging++
75.94
$481
12.34
43.29
39.30
45.10
Importance:
where capacity actually mat- ters. As described in Section [.3] we com- pute layer importance by stage and submodule across backbones, with results shown in Figure Up consistently rank as the two largest contributors, followed by Down; attention K and con- tribute the least; Q and O fall in between. Second, importance concentrates in the late-stage third of the network for all types, with monotonic or near-monotonic increases from early to late blocks (particularly pronounced for Mistral). Third, backbone-specifie nuances remain, InternVL: Down- projection is the single largest contributor, whereas Mistral distributes MLP importance more evenly between Gate and Up. but the overarching ranking is consistent.
Expert Merging
77.84
53.68
13.88
06.34
40.10
07:43
Expert Mergings+
75.51
59.44
13.94
53.66
41.00
48.71
First, MLP components dominate: Gate and
Takeaways for Expert Merging++. The analysis reveals substantial variation across submodules and depths in different architectures, indicating that the prior practice of assigning single coeffi-

---

Published as a conference paper at ICLR 2026
cient per layer overlooks critical inter-layer heterogeneity. This observation motivates Expert Merg- ing++, which replaces the layer-wise coefficient with chunk-wise coefficients, enabling finer mner-grained control in high- iighiimportance layers. In particular, high-importance layers exhibit larger merged task vectors and influence more parameters, explaining why chunking only these layers recovers nearly all the benefits of dense chunking, whereas chunking low-importance layers yields little gain.
4.4 ABLATION STUDIES
We study how each component of Expert Merging and Expert Merging++ contributes to perfor- mance across the two MLLM backbones and the Mistral LLM. For brevity, we report a consolidated summary in Table 4 and defer the full ner-benchmark ablation tables to Appendix C.3
Removing either alignment term degrades performance across all backbones, con- firming that matching both internal representations and predictive distributions is beneficial. On InternVL and Mistral, logit alignment accounts for the larger gains (-3.39 and -3.81 average without it), while hidden-state alignment provides additional improve- ments. On Qwen2-VI. the two terms contribute com-
Aligmment objectives.
Table 4: Ablations on key design choices.
Varinnt
InternVI.
Qwmn-VI.
Mistra
Expen Marging weo hidlen aligumest
$8.11
61.34
47.43
57.73
02:43
47,30
wis alignment
$4.72
61.71
43.62
wii coeffacien regulatioation
47.57
62.17
40.81
wil takk trale-ofis
57.71
62,69
46.30
Expen Merginge+
58.45
6363
48.71
wro chunking
58.11
63.34
$7.43
chank all layers (2x)
56.76
02.61
46,82
parably. Qualitatively, hidden alignment stabilizes cross-domain behavior (notably OCR and grounding on InternVi.), while logit knowledge aligning brings task-faithful outputs.
Coefficient regularization. The coefficient regularizer is critical for stability. Without it, averages drop markedly (-10.54 on InternVL, -6.62 on Mistral, 1.17 on Qwen2-VL), accompanied by pronounced degradation on grounding metrics (e.g., ReiCOCO/+/g on InternVL). This supports our design in Section 3 regularization curbs coefficient drift during unsupervised alignment and prevents catastrophic forgetting in high-capacity. late layers.
Controllable task trade-offs. Removing explicit
task weights consistently reduces averages (-0.4 to -.7)). Beyond the modest aggregate gains, the weights enable practical control over cross- domain balance, aligning with our goal of exposing interpretable knobs for deployment.
Chunk-wise coefficients. Expert Merging++ extends layer-wis- wise coefficients (Expert Merging) by assigning chunk-wise coefficients, yielding further improvements: +0.34 on InternVL, +0.29 on DW and +1.28 on Mistral. These gains concentrate in regions where our importance analysis (Section 4.3) indicates capacity matters most, namely late-stage MLP blocks. For example, chunk- ing substantially boosts code generation Mistral (HumanEval +7.3 points) while preserving or slightly improving other domains. contrast, indiscriminately chunking all layers (2x coefficients) consistently degrades performance across backbones. This underscores that targeted capacity allo- cation, rather than uniform over peee-arraettrizzaion, is crucial for addressing inter-layer heterogeneity, and further validates both our importance metric and the chunk-wise design of Expert Merging++.
Summary. The ablations validate our core choices: (1) alignment losses both help (logits stronger on Intern VL.MMssalllstaal,, hidden states aid robustness); (2) coefficient regularization critical for stabil- ity; (3) task weights enable controllable trade-offs; and (4) chunk- coefficients in Expert Merg- ing++ deliver further gains. Together, these components yield the best balance between accuracy and stability and deliver consistent cross-domain improvements. Combined with the main tables, these component wise degradations support our claim that the gains of Expert Merging and Ex- pert Merging++ come from the hidden state and logit alignment objectives plus importance guided chunking rather than incidental engineering choices.
CONCLUSION AND FUTURE WORK
We introduce Expert Merging, label-free, training-light approach that learns layer-wise coefficients to align hidden states and logits, and Expert Merging+ which adds importance-guided. chunk-wiss capacity. Across InternVL, Qwen2-Ve VL, and Mistral, they consistently surpass training-free free and prior training- bassed baselines; Expert Merging++ provides further gains and in some cases exceeds supervised Mixture Training, with ablations confirming each design choice. In future work, we plan to further explore model merging solutions for heterogeneous models.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Algorithmic details, coefficient parameterization, and the alignment objectives are specified in Sec- tion (with Figures [ and 2 The complete experimental setup inncuding backbones, expert checkpoints, benchmarks, evaluation toolchains, and baselines- is described Section [ Hyper- parameters, data sampling protocols, coefficient intiaaizaationreggllaizzzii compute settings are documented in Appendix [..2] and baseline descriptions and evaluation settings summarized in Appendix B We report consolidated ablations in Table and provide the full per-benchmark ablations in Appendix C.33 For evaluation consistency, we rely on public toolchains (VLMEvalKit and LMMs-Eval) under matched settings cited in Section Our code is available to further facilitate reproduction of
chunking budgets, and
at htttps pettps///i thub. wwwm///tteeo//eeeeeecttteeeeeeet
method.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint urXiv:2108.0777 07732, 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao. Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
Runxi Cheng, Feng Xiong, Yongxian Wei, Wanyun Zhu, and Chun Yuan. Whoever started the interference should end it: Guiding data-free model merging via task vectors. arXiv preprint arXiv:2503.08099 2025.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finttuee lan- guage models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 14 166, 2021.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06303 2020.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluat- ing large modti-yydalliyy models. In Proceedings of the 32nd ACM international conference multimedia, pp. 11198-11201, 2024.
Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola. Task singular vectors: Reducing task interference in model merging. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18695- 18705, 2025.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark MoQuade, and Jacob Solawetz. Arcee's s MergeKit: A toolkit for merg- ing large language models. In Franck Dernoncourt, Daniel Preotiuc-Pietro, and Anastasia Shimorina (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing: Industry Track, pp. 477-483 Miami, Florida, US. November 2024. Associ- ation for Computational Linguistics. doi: 10. 10.18653/w1/2024.cmmm //aclantho Taaclaantolloo..oor////oo ogy. 2024 emn 1p- ndust
URL https:

---

Published as a conference paper at ICLR 2026
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3608-3617, 2018.
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint Sit.::1033.38744, 2021.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04088, 2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https:
//arxiv org.iiiirrrgaao aaaa/////110 06825
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacaeval: An automatic evaluator of instluction-folloving models, May 2023.
Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et al. Domain specialization as the key to make large language models disruptive: A comprehensive survey. ACM Computing Surveys, 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0 03983, 2016.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1 11-20, 2016.
Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartlomiej Twardowski, Andrew D Bagdanov, and Joost van de Weijer. No task left behind: Isotropic model merging with common and task- specific subspaces. arXiv preprint 2502.2500204955 2025.
Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947-952. IEFF, 2019.
12

---

Published as a conference paper at ICLR 2026
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. arXiv, 2023.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li. Measuring multimodal mathematical reasoning with math vision dataset. Advances in Neural Information Processing Systems, 37:95095-95169, 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-lange lannuala model's ' perception of the world at any resolution. arXiv preprint arXiv:2409. /2/91, 2024b.
Yongxian Wei, Runxi Cheng, Weike Jin, Enneng Yang. Li Shen, Lu Hou, Sinan Du, Chun Yuan, Xiaochun Cao, and Dacheng Tao. Unifying multimodal large language model capabilities and modalities via model merging. arXiv preprint arXiv:2505. /99022225..99.,, 19892, 2025.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference machine learning, pp. 11965-23998... PMLR, 2022.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 36:7093-7115, 2023.
Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale? arXiv preprint
arXiv:2410.03617, 2024.
Enneng Yang. Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo. Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. The Twelfth International Conference on Learning Representations, 2024.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Ab- sorbing abilities from homologous models as free lunch. In Forty-first International Conference on Machine Learning. 2024.
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint 1277v 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint 203. 2023.
Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. Metagpt: Merging large language models using model exclusive task arithmetic. arXiv preprint arXiv:2406. 1385, 2024.
13

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS
We used large language models (LLMs) to aid and polish the writing of this manuscript. Concretely, used LLMs suggest improvements to grammar, phrasing, and clarity for select passages.
B EXPERIMENT SETTING
B. BASELINES
We compare against the following baselines:
Weight Averaging (Wortsman 2022
simply averages the weights (or deltas) of
models fine-tuned on different tasks.
define task vectors Tk = 0g Bases and the merged model follows the notation in Section 3.1 as 0TA = Obose + Σm=1 A Tu (with
Task Arithmetic (TA) (Iharcco et al.
2022
predefined {Ax)). DARE (Yu et al. 2024 randomly drops redundant task vectors and rescales the remaining ones to mitigate interference.
TIES Merging Yadav et al. 2023): sparsification-bis pipeline with Trim, Elect Sign, and Disjoint Merge to produce merged vector Tmi the final model is Oesse X Am with tuned; we also evaluate TIES+DARE.
TSV Merging (Gargiulo et al., 2025): an SVD-based method that measures singular task interference and reduces it via decorrelation formulated as an orthogonal Procrustes prob-
lem.
Iso-C (Marezak al. 2025
isotropic merging that flattens the singular value spectrum of task matrices and improves alignment between singular components.
WUDI Merging (Cheng et al. 2025); optimization based merging that treats task vectors spanning an approximate linear subspace and minimizes layer-wise interference between the merged vector e and each task vector Ti, using only task vectors (no labels).
WUDI v2 (Wei et al. 2025): adds an SVD-based component to decompose task matrices, yielding stronger multi domai- performance.
AdaMrggng YYang et al. 2024): taining--bsed approach that learns small, typically per- layer coefficients on unlabeled calibration data to combine experts.
variant that learns per-layer coefficients for each task vector within the TIES- Merging pipeline, improving robustness to conflicts.
Adggengggg++ (Yang al.) 2024):
Mixture Training: supervised multi-task finetuning on the union of data, serving as an upper bound reference when available.
Analysis. Across backbones (Tables I 2 and 3 training-free methods exhibit clear trade-offs. TA/DARE variants often peak a few domains (c.g., ChartQA) but underperform grounding or geometry tasks, while sparsification- and SVD-based methods (TIES/TSV/Iso reduce con- flicts yet still lag cross-domain balance. Optimization-orrineee WUDI/WUDI v2 deliver stronger macro averages among training-fre baselines but remain below Expert Merging/++: especially on grounding (MLLMs) and code (LLMs).. Notably, Mixture Training is not strict upper bound in our setting: Expert Merging+ surpasses it on macro average for both InternVL and Qwen2-VL, echoing the main that careful, unlabeled calibration can rival exceed supervised multi-task fine-tuning.
B.2 IMPLEMENTATION DETAILS
Backbones and experts. We consider both LLMS and MLLMs. For LLMs, the backbone is Mistral- 7B-v0.1 I with three domain experts: instruction following (Chat), mathematical reasoning (Math), and code generation (Code). For MLLMs, following prior multimodal merging work (Wei et al, 2025), -ntemVL2.5-1B-mmm experts: VQA, Geometry, Chart, OCR, and Grounding. Expert checkpoints follow the sources referenced in Section 4
and Owen2-VL-7BB-o -Bsee backbones, with five capability
14

---

Published as a conference paper at ICLR 2026
We report the key hyperparameters used; other settings follow defaults and are omitted due to space.
Hyperparameters.
Train samples per task (N). In2em: 5; Mistral-7B-vO.I: 5; Qwen2-VL-77-Ban
10.
Initialization (ā prior). InterVV2..5; 0.1; Mistral-7B-vO.1: 0.3; Qwen2-VL-7B-Ban
0.3.
Hidden alignment layers (S). For all models, supervise only the middle Transformer layer's s hidden state, i.e., S|=1 with layer index [L/2].
Coefficient regularization weight (y). Intern 5; Mistral-7B-vO.1: 0.8; Qwen2- VL-7B-Base: 0.8.
Total coefficient budget (B). Set to 0.9-1.2 reduce trainable parameters and en- courage sparsity.
Allocation steepness (s). Set to 1.2 for each model to accentuate differences in layer importance.
C EXPERIMENT RESULTS
C.I LAYER- WASE IMPORTANCE ANALYSIS
This section provides the full stage-wise wise (carly/middle/late) breakdown of learned coefficients for each submodule and backbone, complementing the model-wise aggregation in Figure Bp.
Learned coefficients across depth. Coefficients learned by Expert Merging are distinctly non- uniform depth Figure 5 shown. Across all three backhones, late-stage blocks receive larger shares than early ones for both attention and MI.P For example, on Mistral the Q and Gate shares grow from about ~0.04 (carly) to ~~006 (late). Intern shows a pronounced late-stage skew on the MLP poojerrojcction, while Qwen exhibits a milder early-to-late increase. Averaged over types, model-wise coefficient shares remain relatively even (roughly 0.13-0.16 per type), suggesting that raw coefficient fractions alone weakly indicate capacity hot spots unless viewed alongside parame- terization across depth.
The late-layer emphasis aligns with the role of deeper blocks in consolidating task-specific signals (e.g., cross-modal grounding MILMS and semantic control LLMs).. This observation motivated the chunk-wise refinement in Expert Merging+ which reallocates more pacity primarily within later regions. Consistently, Appendix C.3 shows targeted chunking improves grounding/OCR Intern and boosts code/math on Mistral with minimal impact on VQA.
Interpretation.
Up
Down
Coefficient Magnitude
0.05
Early
Middle
Late
Early
Middlee
Iate
Errly
Middle
Late
InternVL
Qwen2-VL
Mistral
Figure 5: Learned coefficient share across depth (earlt/middle/late) for each submodule. Late-stage blocks consistently receive larger shares for both attention and MLP; Intern shows a stronger late-stage skew on the MLP poon-prooection, while Qwen's shift is milder.
C.2 TASK- VECTOR COSINE SIMILARITY ANALYSIS
To better understand interactions among experts, we compute cosine similarity between task vectors O Figure reports heatmaps for InternVI. Qwen2-VI., and Mistral.
Tk 02
15

---

Published as a conference paper at ICLR 2026
Table 5: Ablation studies Intern V1.2.. We measure the effect of each component in Expert Merging and the chunking strategy in Expert Merging+t.
VQ4
Goometry
Chart
Grounding
Variant
ViaWiz GQA
Math Vista MATHI-Vbion
ChariQA
TextVQA
OCRVQA
ReCCOCO
ReCCOCO+ ReCCOCOg
Avg.
Expert Merging
31.16
26.32
65.28
$0.05
73.85
wio hidden alignment
31.07
27.63
64.52
75.55
45.15
78.06
76.98
57.75
wio logit aligoment
31,53
57.08
48.82
22.37
67.16
75.82
44.76
69.81
63.05
66.81
54.72
task trad--ufis
31,16
56.72
51.17
26.32
64.60
75.52
45.05
78.12
71.65
76.82
57.71
regularization
29.63
52.11
65.08
Expert Merginge+
31.14
56.53
52.36
22.37
65.92
76.06
46.00
80.53
74.37
79.31
5K.45
no chunking
31.16
56.77
47.71
26.32
65.28
75.70
45.31
$0.05
73.85
70.04
58.11
all layers
On VL, most off diagonal entries are close to zero, with the largest similarity between OCR and Grounding (Rs 0.14). This suggests that the visual experts largely complementary and that conflicts arise mainly from few pairs (OCR and Grounding) with modest overlap. On Qwen2-VL, similarities are more uniformly positive (+= 0.05 0.13), especially among OCR, VQA, and Chart, indicating shared structure among text heavy visual tasks, while Geometry and Grounding remain less aligned with them. For Mistral, the Chat and Code experts show relatively high similarity ( 0.42), whereas Math is almost orthogonal to both. This matches our observation that code and instruction tuning are more synergistic than math, yet the merged model retains strong performance across all three domains due to the behavior space alignment objective and coefficient regularization.
L.0
VQA
0.03
VQA
0.10
0.04
0.8
Cosine similarity
Goometry
0.00
0.1
0
0.00
Geometry
0,06
1,00
0.06
0.05
0.04
0.6
Chart
0.01
0.01
1.00
0.32
0.01
Chart
0,10
0.05
1.00
0.12
0.07
Math
0.01
1.00
0.02
0.4
OCR
0.03
0.00
0.02
1,00
0.14
OCR
0.33
0.05
0.12
1.00
0.06
Code
0.02
1.00
0.2
Grounding
0.11
0.14
1.00
Grounding
0.07
VQA
Goumteyy
Chart
OCR
Gromding
VQA
Geometyy
Chart
OCR
Grounding
Chat
Mah
Code
0.0
(a) InternVi
(b) Qwen2-Ny
VL
(c) Mistral
Figure 6: Task-vector cosine similarity heatmaps for (a) Intern (b) Qwen2-VL, and (c) Mistral.
Darker blocks indicate higher similarity.
C.3 ABLATION STUDIES
This section provides the complete ablation results for each backbone, expanding the summarized findings reported in Table See Tables mnd7for detailed results.
Summary. Three consistent trends emerge across backbones: (i) logit alignment essen- tial-remoingg it causes the largest average drops and notably hurts geometry/grounding on MLLMs and code LLMs; (ii) coefficient regularization is critical for withol it, per- formance collapses on Intern and Mistral, indicating over-allocation and interference; and (iii) chunking helps when guided by importance the targeted ftunk-wise variant in Expert Merging+ yields macro-average gains, whereas naively chunking all layers degrades multiple domains.
InternV1.2.5. Table shows that, relative to Expert Merging, removing hidden alignment slightly raises geometry (e.g., Sattiiitt/MATH-vssiinn but reduces grounding by 2-3 points, yielding lower average-indicating hidden features help localization. Removing logit alignment severely hurts geometry and grounding (while sometimes inflating ChartQA), confirming logits are indis- pensable for cross-domain consistency. Eliminating coefficient regularization leads to a dramatic collapse, especially grounding benchmarks. Guided chunking in Expert Merging++ improves OCR and grounding and achieves the best average, while chunking all layers uniformly harms ge- ometry and grounding.
16

---

Published as a conference paper at ICLR 2026
Table 6: Ablation studies Qwen2-VL. We report the impact of each loss component and chunking strategy.
VQ4
Goometry
Grounding
Variant
ViaWiz
GQA
Math
MiTTo-Vibion
ChariQAA
TextVQA
OCRVQA
Re(COCO
ReCCOCO+ RefCOCOg
Avg.
Expert Merging
41 14.44
42.11
13,00
10
66.15
79.02
wio hidden alignment
41.6c
47.66
34.21
22.84
81.59
60.38
80.65
78.87
62,43
wio logit alignment
41.80
60.18
47.82
38.16
72.04
81.44
60 16
90.64
65
78.76
62.71
task trade-ufis
41.56
60.53
18.38
36.84
72,80
81.49
59.96
80.51
66.00
78.82
62.69
regularization
41.67
36.84
72.20
Expert Merginge+
41.49
60.42
49.33
44.74
72.60
81.65
60.19
80.71
66.21
79.00
63.63
no chunking
41.54
60.51
48. 80
42.11
13,00
81.41
60. 19
80.64
66 15
79.02
63.34
all layers
Table 7: Ablation studies Mistral. We evaluate the effect of each loss component and chunking choice on text benchmarks.
Chat
MATH
Code
Variunt
AlpucaEval
GSM8K
MATH
HvmanEEal
MBPP
Avg.
Expert Merging
58.68
13.88
46.34
40.40
47.43
w/o hidden alignment
77.99
57.09
13.92
45.12
41.40
47.10
w/o logil uligmment
69.68
56.10
13.32
38.41
40.50
43.62
task trade-offs
78.95
54.81
13.50
45.12
41.60
46
w/o coefficient regularization
65,32
52.01
12.08
39.02
35.60
40.81
Expert Merging++
75.51
59.44
13.94
53.66
41.00
48.71
chunking
77.84
58.68
13.88
46.34
40.40
47.43
chunk all layers (2x)
77.84
56.79
13.74
44.51
41.20
46.82
Qwen2-VL. Table 6 shows that sensitivity is milder: removing hidden logit aligmment changes the average only modestly, though MATH-Visee Vision degrades without logits. Dropping regularization shifts scores unevenly (e.g., slight gains GQA/Math Vista but clear declines in grounding), under- scoring the stability-variane trade-off. importance-guidee guided chunking improves MATH-Vision and the macro average; indiscriminate chunk-all-layers again underperforms.
Mistral-7B. Table 7 shows that removing logit alignment yields the largest drop among ablations, impacting both instruction following and code. The coefficient regularizer is likewise crucial: with- out it, average performance falls sharply. Expert Merging++ delivers the best average, with gains dominated by large improvement on HumanEval and consistent lifts on GSM8K/MBPP, while AlpacaEval remains strong, Uniform chunking degrades results relative to targeted chunking.
C.4 REGULARIZATION SENSITIVITY
We evaluate the effect of the coefficient regularization strength on Expert Merging, keeping all other settings fixed. Metrics and datasets match the main ablations.
Summary. Across backbones, moderate coefficient regularization yields the best macro averages: VL255 prefers a stronger setting (y ss 5), while Qwen2-Vi and Mistral-7B favor smaller values (y z 0.8). Gains primarily from grounding (MLLMs) and code (LLMs),, with mild trade-offs on Chart/geometry at very large This behavior matches the role of the coefficient regularizer in curbing interference and preventing over-allocation. See Tables and 10
Backbone-speeifie results. For Intern Table shows the average rises from 43.49 at 7=0 to 56.61 at y=5.0 (best), then plateaus (56.42 at 7=8.;; 55.93 at >=10.0). The gain is driven largely by grounding: RefCOCCORerCCCOO/RiCCCC to 79.08/72.91/77..07 at 7=5.0 (79.63/73.42/78.. at 8=8.0). ChartQA and HAATH-Vssion favor smaller 7 (66.76 at 0.6; 26.32 at 0.8), while VQA/OCR change little. Overall, y s 5 is the best trade-off.
improve from 02.46/29.06/40.0
For Qwen2- VL, Table shows sensitivity is mild: the average peaks at 63.34 for y=0.8 and remains within about 0.5 over E 10). MATH- MisioVision benefits from moderate regularization (42. at 0.8 vs. 39.47 at 0), whereas VQA/OCR are nearly unchanged and grounding remains stable around 80.6/66.2/79 2/7900 A smaller default (7 Rs 0.8) is appropriate.
17

---

Published as a conference paper at ICLR 2026
Table 8: Regularization sensitivity InternV1.22. for Expert Merging. Rows vary the coefficient
regularization strength.
VQA
Geometry
Chart
OCR
Grounding
VizWiz GQA
VathVista
MATH-Vision
ChartQA
TextVQA
OCRVQA
RefCOCO
RefCOCO+ RefCOCOg
Avg.
7
29.15
51.06
51.93
19.74
65.64
72.52
42.42
32.46
29.06
40.97
43.49
1=0.2
52.97
51.04
36.85
44.79
0=0.1
30.16
54.41
50.88
22.37
66.44
73.94
44.08
38.02
32.85
38.54
45,16
7 0.666 0.6
30.49
55.12
49.47
22.37
66.76
74.43
44.69
40.26
34.15
37.03
45.47
-=0.8
47.06
0=9
30.83
55.72
48.19
25.00
65.76
74.92
45.38
46.68
40.46
41.85
47.47
1=100
30.96
55.92
47.77
21.05
65.24
75.04
45.38
49.32
43.21
44,83
47.87
1=1.5
31.15
56.12
46.45
21.05
75.42
62.61
56.18
57.41
51.65
2.0
31.23
56.27
47.73
21.05
64.16
75.76
44.95
71.77
65.62
68.14
54.66
3-00
31.26
56.29
49.12
18.42
62.64
75.89
45.31
71.97
71.67
75.74
$6.43
5500
31.05
56.49
47.71
19.74
60.12
75.87
45.44
79.08
72.91
77.70
56.61
8.0
30.94
56.70
47.29
18.42
58.44
75.74
45.64
79.63
73.42
78.00
56.42
7= 10.0
31.07
56.85
43.96
17.11
58.24
75.53
45.57
79.50
73,42
78.08
55.93
Table 9: Regularization sensitivity on Qwen2- VL for Expert Merging. Rows vary the coefficient regularization strength.
VQA
Geometry
Chart
OCR
Grounding
VizWiz GQA
Vatt
Misson
ChartQA
TextYQA
OCRVQA
ReCCOCO RefCOCO+ ReCCOCOg
Avg.
7=
41.48
60:51
48.80
39.47
73.24
81.39
59.99
80.59
66.17
78.93
63.06
1=
41.45
60.42
47.96
39.47
73.36
81.40
$9.99
80.59
79.01
62.97
0.4
41.39
60.47
49.22
39.47
73.20
81.39
60.06
80.55
66.
78.91
63.08
0.66
41.43
60.52
49.22
39.47
73.32
81.34
60.06
80.57
66.24
78.94
63.11
-=0.8
41.54
60.51
48.80
42.11
73,00
81.41
60.19
80.64
66.15
79.02
63.34
009.9
41.39
60.47
48,35 38
73.12
81.32
80.56
66.
78.91
62,99
1-0.0
41.42
60.49
47.54
39.47
73.28
81.40
59.93
80.59
66.14
78.94
62.92
1=15
41.44
60.49
49,22
39.47
73.28
81.36
60.06
80.53
66.15
78.92
63.09
2=2.0
60.44
48.38
78.89
62.87
3.00
41.52
60.43
48.38
36.16
73.28
81.35
60.09
80.57
66.16
78.98
62.89
550
41.50
60.43
47.96
38.16
73.16
81.39
60.06
80.55
66.16
78.96
62.83
78.92
63.01
10.0
41.38
60.46
48.92
38.16
73.36
81.44
59.93
80.56
66.13
78.96
62.93
For Mistral-7B, Table 10 shows the average improves from 40.85 at 7=0 to 47.43 at 7=0.8 (best), then drifts slightly (47.27 0.9; 46.34 1.0). Task-wise, HumanEval peaks 48.78 (0.9), MBPP at 43.00 (3.0), and AlpacaEval increases up to 79.64 (3.0). A default of zz 0.8 gives the strongest overall average.
C.5 SAMPLE EFFICIENCY
We report the impact the number of unlabeled calibration samples used by Expert Merging. Met- rics follow the same protocol as above.
Summary. Across all backbones, performance saturates with very few unlabeled samples: best averages occur at N E {5. 10). and larger N brings marginal or negative returns. This indicates that layer-wise wise coefficient learning has low sample complexity, consistent with the small parameter count and the stabilizing effect of the regularizer and coefficient budget. See Tables [2 and T3
Backbone-speeificc results. For InternVV.2.5. Table shows the average 56.46 at N=3, improves to 18 at N=5 (best), and is 57.35 at ==10. Grounding peaks at N (Ref- (OCORRiCOCOOReCCCOOS 80.05/73.85/1 85/77334//79 while geometry varies (Math 51.61 at 10; MATH--Viioon 26.32 at 5). Larger N yields little benefit.
For (wen2-VL, Table 12 shows results plateau: the average is 63.01 at N=3, 63.04 at N=5, and 63.34 at N=10 (best). Vision benefits most from -=10 (42.11), while other domains change marginally (e.g., TextVQA 81.21-81.666 Beyond N=10, improvements are small or nega- tive.
18

---

Published as a conference paper at ICLR 2026
Table 10: Regularization sensitivity Mistral for Expert Merging. Rows vary the coefficient regu-
larization strength.
Chat
MATH
Code
7
AlpacaEval
GSM8K
MATH
HumanEval
MBPP
Avg.
y=
65,32
52.01
12.08
39.02
35,80
40.85
0=2.2
75.53
55.42
13.58
43.90
39.40
45.57
0.44
75.61
57.32
14,02
45.73
41.00
46.74
0=666
77.52
$8.68
13.84
46.95
39.40
47.28
0=0.8
77.84
58.68
13.88
46.34
40.40
47.43
>=0.9
76.66
56.71
13.22
48.78
41.00
47.27
y=1.0
13.60
39.00
46.34
1=1.5
79.01
$6.25
12.24
45.73
42.60
47.17
2=0.0
79.28
56.41
12.00
44.51
42,60
46.96
3-0
5.0.
79.09
55.34
13.20
43.90
42.40
46.79
8.8.
79.00
55.65
13.66
43.29
40.60
46.44
1=10.0
Table 11: Sample efficiency on Intern for Expert Merging. Rows vary the number of unlabeled calibration samples.
VQA
Geometry
Chart
OCR
Grounding
VizWiz GQA
MathVista MATH- Vision
ChartQA
TextVQA
OCRVQA
ReCCOCO RefCOCO+ Re(COCOg
Avg.
N=3
31.10
56.75
46.91
25.00
60.04
75.37
45.44
77.41
70.75
75.86
56.46
N-5
31.16
56.77
47.71
26.32
65.28
75.70
45.31
80..5
73.85
79.04
58.11
N=10
31.01
56.61
51.61
21.05
59.96
75.68
45.93
79.76
73.61
78.33
$7.35
N=15
30.93
56.57
50.18
22.37
60.84
75.38
44.95
78.84
72.68
77.08
56.98
N=20
31.05
56.63
50.06
21.05
62.04
75.74
44.73
79.20
72.45
76.61
56.95
N=30
30.89
56.63
49.92
25.00
59.80
75.39
44.79
78.48
72.10
76.23
$6.92
N=50
30.71
56.57
50.34
21.05
56.28
75.22
45.28
78.38
71.75
76.39
56.19
-=100
56.71
48.25
60.32
75.28
78.84
72.06
76.53
56.45
For Mistral-7B, Table s3 the average is 47.17 N and 47.43 at N=5 (best). Tasks prefer different regimes (HumanEval 48.17 at 3; MBPP 40.60 at 10), but overall handful of samples per domain suffices.
Regularization and data efficiency work in tandem: modest reduces cross-domain interference so that a small calibration set captures task signatures reliably, and sypportance-guddee chunking (Section 3.2) further channels capacity to later layers where gains concentrate (grounding/code task).
C.6 INITIALIZATION SENSITIVITY
We examine the sensitivity of Expert Merging to the initialization prior ā. Tables 14 report the results.
15 and 16
Summary. Performance is generally robust the choice of across backbones, with optimal val- ues falling in the range [0.1, 0.3]. A moderate uniform initialization helps avoid poor local minima, particularly for tasks requiring complex reasoning or generation (e.g., Code, Grounding).
Backbone-speeific results. For Intern VL2.5 (Table the average score peaks 2=0.1 (58.11) and gradually decreases as increases, though Grounding remains strong at lower priors. For Qwen2-VI. VI. (Table 15). the best performance achieved at 6=0.3 (63.34), with notable stability in OCR and VQA tasks across the range. For Mistral-7B (Table [6). the average score peaks at 0=0.3 (47.43). Very low priors (a=0.1) lead to significant drop (41.77), especially Code tasks, suggesting that sufficient initial allocation is crucial for text generation capabilities.
The stability across a wide range confirms that our method not overly sensitive to the starting point. However, moderate prior (+s 0.1-0.3) strikes the best balance, providing enough initial capacity for task adaptation without overwriting the pre-trained knowledge necessary for general capabilities.
19

---

Published as a conference paper at ICLR 2026
Table 12: Sample efficiency Qwen2-VL. for Expert Merging. Rows vary the number of unlabeled
calibration samples.
VQA
Geometry
Chart
OCR
Grounding
N
Viz GQA
MathVista
MATH- MATH-Vision
ChartQA
TextVQA
OCRVQA
RefCOCO RefCOCO+ RefCOCOg
Avg.
N=3
41.42
60.53
48.80
39.47
74,08
81.21
59.93
80.41
65.63
78.58
63.01
N=5
47.40
80.72
79.02
63.04
N=10
41.54
60.51
48.80
42.11
73.00
81.41
60.19
80.64
66.15
79.02
63.34
N=15
41.75
60.34
49.75
34.21
71.84
81.66
60.29
80.60
66.
78.93
62.35
N=20
80.63
N=30
41.60
60.31
49.93
32.89
73.00
81,60
60.12
80.35
65.99
78.87
62.49
N 5=
41.60
60.38
48.92
35.53
72.80
81.58
60.35
80.67
66.28
79.03
62.71
80.69
62.90
Table 13: Sample efficiency on Mistral for Expert Merging. Rows vary the number of unlabeled calibration samples.
Chat
MATH
Code
N
AlpacaEval
GSMSK
MATH
HumanEval
MBPP
Avg.
N=3
75.41
58.07
13,80
48.17
40.40
47.17
N=5
47.43
N=10
75.69
56.71
13.66
45.12
40.60
46.36
N=15
76.22
57.24
14,30
45.12
38.00
46.18
N=20
77.88
57.47
13.42
39.00
45.85
N=30
76.89
57.62
13,80
45.73
39.00
46.61
N=50
76.65
57.85
13.82
47.56
39.40
47.06
N=100
77.23
57.54
13.88
46.34
39.00
46.80
Table 14: Initialization sensitivity Intern for Expert Merging.
VQA
Geometry
Chart
OCR
Grounding
VizWiz GQA
Math MATH-Vision
ChariQA
TextVQA OCRVQA
ReCCOCO Re(COCO+ ReCCOCOg
Avg.
6=0.1
31.16
56.77
47.71
26.32
65.28
75.70
45.31
80.05
73.85
79.04
58.11
6=0.2
30.83
56.73
53,54
21.05
59.48
75.58
45:70
78.86
72.89
77.80
57.24
6=0.3
31.05
56.49
47.29
19.74
60.12
75.87
45.44
79.08
72.91
77.70
56.56
6=0.4
31,26
57.22
52.45
25.00
61.28
75.82
44.99
73.89
67.05
71.77
56.07
a=0.5
31.10
56.61
50.44
23.68
69.64
75.38
43.95
63.95
56.78
59.33
53.08
Table 15: Initialization sensitivity on Qwen2 VL for Expert Merging.
VQA
Geometry
Chart
OCR
Grounding
6
VizWiz GQA
MathVista MATH-Visson
ChartQA
TextVQA
OCRVQA
RefCOCO ReCCOCO+ ReCCOCOp
Avg.
a=0.1
40,68
60.92
41.97
34.21
72.88
79.83
$4.92
80.35
66.14
78.17
61.01
G=0.2
41.50
62.68
47.70
38.16
72.12
80.83
58.40
80.04
65.74
78.33
62.55
41.54
48.80
42.11
73.00
81.41
60.19
80.64
66.15
63.34
6=0.4
41.00
61.75
43.48
39.47
75.76
81.37
60.06
80.54
66.
78.84
62.86
0=0.5
40.70
68.18
43.48
30.26
78.16
81.13
59.64
80.43
66.60
79.12
62.17
Table 16: Initialization sensitivity on Mistral for Expert Merging.
Chat
MATH
Code
AlpacaEval
GSM8K
MATH
HumanEval
MBPP
Avg.
6=0.1
69.02
48,90
11.52
38.41
41.00
41.77
0=2.2
79,07
57.09
13.50
41.46
43.00
46.82
d 0-3.3
77.84
58.68
13.88
46.34
40.40
47.43
0-4.4
76.95
56.86
14.10
46.95
39.00
46.77
0-5.
78.05
57.62
13.82
47.56
39.20
47.25
20

---

Published as conference paper at ICLR 2026
C.7 HIDDEN ALIGNMENT LAYER ANALYSIS
We investigate the effect of the choice of the hidden alignment layer S. Tables 17 the performance across different layers.
18 and 19 show
Summary. Supervising the middle Transformer layers consistently yields the best results across all models. This confirms that middle layers capture the most transferable semantic structures suitable for alignment, whereas early or late layers are either too generic or too task-specific.
Table 17: Hidden alignment layer sensitivity on Intern VL2.5 for Expert Merging.
VQA
Geometry
Chart
OCR
Grounding
S
VizWiz GQA
Vasa
MATH-Vision
ChartQA
TextQQA
OCRVQA
ReCCOCO
ReCCOCO+ ReiCOCOp
Avg.
S=0
30.96
56.57
47.11
26.32
60.88
75.72
45.61
79.37
73.01
77.80
$7.33
$=4
31.10
56.61
48.38
26.32
63.92
75.75
45.57
79.28
72.92
77.92
57.77
S=8
31.19
51.61
73.05
77.84
58.02
S=12
31.16
56.77
47.71
26.32
65.28
75.70
45.31
80.05
73.85
79.04
58.11
S=16
30.91
56.41
50,22
26.32
60.28
75.69
45.61
79.32
73.17
78.04
57.59
5-20
30.86
47.41
72.84
77.86
57.18
S=23
31.16
56.45
50.24
23.68
61.52
75.74
45.31
78.60
72.15
76.61
57.14
Table 18: Hidden alignment layer sensitivity on Qwen2- VL for Expert Merging.
VQA
Geometry
Chart
OCR
Grounding
S
VizWiz
GQA
MathVista
MATH- Vision
ChartQA
TextVQA
OCRVQA ReSCOCO RefCOCO+ RefCOCOg
Avg.
41.57
60.37
47.24
36.84
72.68
81.55
60.48
80.66
66.05
78.87
62.63
S=4
48.80
66.19
78.93
62.99
S=8
41.41
60.49
48.80 80
39.47
73.12
81.38
60.06
80.57
66.19
78.94
63.04
S=12
41.57
60.49
48.38
39.47
73.20
81.40
60.03
80.53
66.
78.98
63.02
S=14
41.54
42.11
73.00
81.41
60.19
66.
79.02
63.34
8-16
41.48
60.48
48.38
39.47
73.36
81.31
60.12
80.52
66.08
78.91
63.01
S=20
41.45
60,36
49.08
36.84
72.88
81.37
59.93
80.49
66.00
78.86
62.73
S=24
41.56
60.38
47.82
36.84
73.60
81.26
59.83
80.37
65.78
78.59
62.60
S=27
41.64
48.66
59.24
65.55
78.18
62.49
Table 19: Hidden alignment layer sensitivity on Mistral for Expert Merging.
Chat
MATH
Code
AlpacaEval
GSMSK
MATH
HumanEval
MBPP
Avg.
S=0
77.99
57.24
13.18
47.56
39.20
47.03
S-4
78.92
57.16
13.30
46.95
39.00
47.07
S=8
77.18
57.09
13.44
48.78
40.40
47.38
8-12
77.52
57.24
14.06
46.34
41.00
47.23
S=16
77.84
58.68
13.88
46.34
40.40
47.43
S-20
77.43
56.33
13.70
46.95
41.40
47.16
S-21
77.72
56.71
13.76
46.34
41.00
47.11
S-28
76,82
56.94
13.60
46.95
41.80
47.22
S-31
76.34
55.80
13.48
48.17
39.40
46.64
Backbone-specific results. For IL2.5 (Table 17 performance peaks at layer 12 (58. 11) and remains high in the middle block (layers 8-16). For Qwen2-VL VL (Table 18). the results are stable across middle layers, with the best average at layer 14 (63.34). For Mistral-7B (Table the performance peaks at layer 16 (47.43) and robust across layers 12-24. Aligning at the very first (layer 0) or last (layer 31) layers results in slightly lower averages.
C.8 ALL.OCATTON STEEPNESS ANALYSIS
We analyze the impact the allocation steepness parameter π, which controls how sharply the budget is allocated across layers. Tables 20 and 22 present the results.
21

---

Published as a conference paper at ICLR 2026
Summary. A moderate to high steepness (k 2 1.0) generally benefits performance, allowing the model to allocate capacity more aggressively to critical layers. This is particularly effective for tasks that rely specific depth-wise processing, such as code generation.
Table 20: Allocation steepness sensitivity on Intern for Expert Merging++.
VQA
Geometry
Chart
OCR
Grounding
VizWiz GQA
Vath MATH-Viste Vision
ChartQA
TextVQA
OCRVQA
RefCOCO RefCOCO+ RefCOCOg
Avg.
km000
31.16
56.77
47.71
26.32
65.28
75.70
45.31
80.05
73.85
79.04
58.
k=
31.39
56.61
51.61
22.37
65.08
75.97
45.93
80.08
73.91
78.62
58.15
k=0.5
31.39
56.44
50.08
23.68
64.60
75.96
45.74
79.28
73.52
79.74
58.04
31.39
56.29
52.45
25.00
64.76
75.68
44.79
80.72
74.74
78.17
58.39
1.2
31.14
56.53
52.35
22.37
65.92
76.06
46.00
80.53
74.37
79.31
58.45
K=2.0
31.10
56.28
49.40
23.68
67.00
75.91
79.08
72.87
77.47
57.91
Table 21: Allocation steepness sensitivity on Qwen2-VL. Expert Merging++.
VQA
Geometry
Grounding
VizWiz GQA
Vathiisst MATH- Vision ChariQA
TextVQA OCRVQA ReCCOCO ReICOCO+ RefCOCOg
Avg.
41.54
60.51
48.80
80.64
66.15
79.02
63.34
K=0 .25
41.51
60.45
48.38
38.16
73.12
81.36
60.03
80.56
66.
78.94
62.87
0.5
41.72
60.54
49.22
39.47
73.92
81.37
60.32
80.59
65.84
78.89
63.19
50.33
78.85
63.44
1.2
41.49
60.42
49.33
44.74
72.60
81.65
60.19
80.71
66.21
79.00
63.63
2.0
41.76
60.50
48.24
43.42
73.92
81.55
60.22
80.58
65.95
78.81
63.50
Table 22: Allocation steepness sensitivity on Mistral for Expert Merging++.
Chat
MATH
Code
AlpacaEval
GSMSK
MATH
HumanEval
MBPP
Avg.
k=
77.84
58.68
13.88
46.34
40.40
47.43
0.25
78.80
57.70
13.18
46.95
40.80
47.49
k 0.5
78.87
56.56
13.18
47.56
41.20
47.47
k=1.0
78.89
57.24
13.58
50.00
39.00
47.74
1.2
79.07
57.09
13.50
50.61
43.00
48.65
2-2
57.54
13.76
52.44
42.20
48.52
Backbone-specifie results. For VL2.5 (Table 20 a steepness of x=1.2 provides the best balance (58.45), improving Grounding and OCR. For Qwen2-VL. (Table 21). performance is rela- tively stable peaks at 1=1.2 (63.63). Notably, MATH-Vision sees a significant improvement (44.74 vs 42.11 at k=0), suggesting that sharper allocation benefits geometric reasoning. For Mistral-7B (Table 22). increasing λ from to 1.2 consistently improves the average score, peak- ing 48.65 with 1.2. Notably, higher steepness significantly boosts HumanEval (50.61 vs 46.34 at K=0), indicating that sharper allocation benefits complex reasoning.
The benefits of 2 1.0 suggest that "winner-take--ll tendency in coefficient allocation is ad- vantageous. By sharpening the distinction between important and unimportant layers, the method prevents capacity dilution and ensures that critical layers receive the dominant influence they need to drive performance.
C.9 TASK WFIGHT SENSITIVITY
We analyze the impact of varying the task weights for different tasks. Tables 23 and present the results.
Summary. Adjusting task weights allows for a trade-off between different capabilities. Increasing the weight of a specific task generally improves performance on that task, sometimes at the cost of others. However, method maintains robust average performance across range of weights.
22

---

Published as a conference paper at ICLR 2026
Table 23: Task weight sensitivity Intern for Expert Merging.
VQA
Geumetry
Chart
OCR
Grounding
Weight
VizWiz GQA
MathVista
MATII- Vision
ChariQA
TexVVQ
OCRVQA
ReCOOCO
ReCOCC++ ReCCOCOg
Avg.
0=1
31,16
56.72
51.17
26.32
64.00
75.52
45.05
78.12
71.63
76.82
57.71
Avox
79.00
0vox=1.5
31,01
56.57
47.67
18.42
64.12
75.96
45.28
79.90
73,68
78.57
57.11
Pvex=20
30.97
56.64
47.67
22.37
63.72
75.99
45.51
80.02
73.75
78.96
57.56
Acsmur
Apessar
1.5
30.91
56.54
$2.03
26.32
60.84
75.88
45.41
79.83
73.94
78.66
58.03
30.97
56.53
51.61
23.68
60.64
75.75
45.57
79:12
72.86
77.72
57.44
0.5
30.76
56.57
48.27
22.37
58.76
75.93
45.77
80.15
74.07
78.90
57.15
A0.5=115
31.22
56.65
46.99
18.42
68.32
75.96
45.18
79.80
73.86
78.98
57.53
Acaan=
79.92
78.37
A.cx=0.5 0.5
30.89
56.77
47.39
22.37
64,52
75.95
44.79
79.74
74,02
78.76
57.52
4ocx=1.5
30.97
56.51
48.09
23.68
63.76
76.12
45.57
80.12
73.77
78.72
57.23
Aocx=2.0
76.02
80.07
Atrnniin =0.5
30.91
56,80
49.28
25.00
61.28
75.45
45.48
76.59
72.42
76.94
57.01
1.5
31.16
56.77
47.71
26.32
65.28
75.70
45.31
80.05
73.85
79.04
58.11
31.05
56.34
48.13
23.68
63.68
76.14
45.44
80.02
73.65
78.74
57.68
Table 24: Task weight sensitivity on Qwen2-VL. VL for Expert Merging.
VQA
Geometry
Chart
OCR
Grounding
Weight
VizWiz
MathVissa
MATH-Vision
ChartQA
TextVQa OCRVQA
ReCCOCO ReCCOCO+ RefCOCOg
Avg.
A-1
41,56
60.53
48.38
36.84
72.80
81.49
59.96
80.51
66.00
78.82
62.69
3vgx=0.5
41.51
60.30
47.96
35.53
72.76
81.45
60.25
80.57
66.16
78.87
62.54
Ovox
72.96
80.56
78.98
0vox=2.0
41.66
60.48
48.38
38,16
72.92
81.56
60.48
80.49
65.98
78.89
62.90
Accesstry
005
41.41
60.53
47.96
35.53
73.36
81.48
60.32
80.67
66.20
78.94
62.64
Aeesnary
-2.0
41.82
60.27
48.24
36.16
71.32
81.44
39.83
80.30
65.55
78.40
62.53
Acaan 0.5
41.64
60.48
46.71
39.47
72,76
81 .444
60.16
80.57
10
78.95
62.83
41.58
60.46
49.22
38,16
73.12
81.44
60.09
80.53
66.02
78.91
62.95
41.71
60.42
47.12
39.47
73.16
81.36
60.06
80.54
63.97
78.85
62.87
A.c
41.69
60.37
39.47
72.28
81.27
80.50
78.82
62.93
Aoca=1.5
41.50
60.43
18.92
38.16
73.60
81.46
60.19
80.72
66.22
78.88
63.01
1ocx=2.0
41.51
60.38
47.66
36.84
72.96
81.51
60.48
$0.64
66.25
78.80
62.70
72.92
Arennding
=1.
1.5
41.52
60.36
48.38
38.16
72.56
81.51
60.16
80.62
66.26
78.93
62.85
-2.0
41.61
60.38
45.85
38.16
72.68
81.38
60.22
80.79
66.45
78.97
62.65
Table 25: Task weight sensitivity Mistral for Expert Merging.
Chat
MATH
Code
Weight
AlpacaEval
GSM8K
MATH
HumanEval
MBPP
Avg.
8=1
78.95
54.81
13.50
45.12
41.60
46.80
Ah 0.5
78.15
57.54
45.73
41.60
47.32
Bch -1.
78.95
54.81
13.50
46.95
41.00
47.04
Ach =
78.93
53.68
13.04
47.56
42.80
47,20
Acua -2.0
79.07
52.39
12.24
46.34
42.60
46.53
AMATH= 0.5
79.75
52.01
12.08
43.29
40.60
45.55
MMTHH -1 1.2
80.17
57.24
13.18
45.73
39.00
47.06
MMATH 1.5
79.44
58.53
14.40
45.12
39.20
47.34
MMTTH 2.0
78.00
58.15
14.64
45.12
40.00
47.18
Code = 0.5
81.41
56,10
13.48
44.51
41.20
47.34
ACade 1.2
77.84
58.68
13.88
46.34
40.40
47.43
Bodde 1.5
76.86
57.62
13.40
46.95
41.20
47.21
Code
75.23
53.15
12.02
49.39
42.20
46.40
Backbone-specific results. For InternVL2.5. (Table 23). increasing the weight of Grounding im- proves its performance significantly (e.g., RefCOCO reaches 80.79 with Grounding-2.0), while the
23

---

Published as a conference paper at ICLR 2026
average performance remains stable. For Qwen2- VL (Table 24 the model shows high stability. In- creasing the weight of MATH improves MATH- Vision and the overall average (63.34 with MATH- 1.5). For Mistral-7B (Table 25). increasing Code weight boosts Code tasks (HumanEval reaches 49.39 with Code-2.0) and achieves the best average (47.43 with Code-1.2).
This sensitivity analysis demonstrates the flexibility of our approach. Users tune task weights to prioritize specific domains (e.g., boosting Code or Grounding) to meet deployment needs. Crucially, the method's underlying robustness ensures that such prioritization does lead catastrophic forgetting of other skills, maintaining a high competitive baseline.
24


Reviews:
[{"review_id": "jjqiQAwF9A", "reviewer": "Reviewer_RyyA", "summary": "The paper addresses the problem of model merging, assuming a pool of K pre-trained models is available, and these have *expertise* on certain tasks. From the perspective of both training-free methods and optimization-based ones, problems related to the inter-layer heterogeneity and variation of the neural nets are not yet solved. For this reason, a new framework consisting of two methodologies (Expert Merging & Expert Merging ++) is proposed, whose main feature is to describe a new procedure of several steps, where the key ideas are i) learn per-layer coefficients within alignment of representations and ii) estimation of the layer importance. Experimental results with LLMs (i.e., Mistral-7B-v0.1 and InternVL2.5-1B-Instruct) show that the performance is on-pair and sometimes better than a wide range of SOTA methods for model merging.", "questions": "I certainly do not agree with the statement \"*We will release the source code upon acceptance*\". Understandably, under industrial competition and similar issues, this reviewer can understand certain practices or not releasing full systems to *operate* models at scale; however, I do think not providing any source code, and promises of doing it conditioned on acceptance, is, in general, a bad practice. I prefer not to raise additional concerns or points regarding this issue with the submission, but I do recommend that authors review the code of ethics of ICLR, particularly the sections on *standards of scientific excellence* (related to reproducibility) and on *trustworthy and transparent research.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **SOTA & Related Work:** I do think the paper and the proposed method are correctly based on evidence, driving the concerns around the limitations of current SOTA methods. I particularly enjoyed reading the division of such methods into the two families, one for the training-based and the non-training-based methods. That was clear enough to me and for experienced readers interested in these topics. In general, the introduction and presentation of the model merging problem are clear and understandable for the ICLR audience.\n- **Critical goals and key points:** I also perceived a very powerful identification of the main goals to solve for the model merging challenge. For instance, alignment of logits, layer importance, or just comprehensive results on some multi-task validation test. The mission is certainly not easy, but sometimes you don't see these sorts of goals and issues as clearly stated as they are in this paper on other model merging contributions.\n- **Alignment of losses:** One of the parts that I was more positively surprised about and which I would've liked to know more about is the subsection on alignment of losses and logits. Although I didn't clearly understand the motivations for making the choices on Eq. 1 and Eq. 2, I see the power of the proposed approach, and this is something I've seen repeatedly missing in some minor works in the last years on model merging.\n- **Empirical results:** The paper has a significant empirical contribution, and experiments have the will to rigorously demonstrate that the performance of the ExpertMerging/ExpertMerging++ is worth being used and superior to other SOTAs. I am, however, not entirely convinced of this superiority, looking at Tables 1 and 2, or at least I get a bit lost on the heterogeneity of results and some lack of clear analysis and identification of issues/strengths related to the performance", "weaknesses": "- **Arithmetic merging/linear regression between models:** My main concern comes from the fact that the proposed model basically merges parameters according to a linear regressor with real-valued coefficients, as shown in Eq. 5 and in section 3.2, second paragraph. In that sense, I don't see much novelty or at least i am not convinced of the great progress that these propositions make, apart from being applied to big LLMs or MLLMs (i.e. Mistral one) from an academic point of view.\n- **Specification and highlighting issues in SOTA models:** Unfortunately, I don't see great research contribution in the way the work and procedures are introduced. I do miss understanding (at an Equation level) what the problems and limitations of, for instance, AdaMerging or similar baselines. In the manner that section 3 describes things and introduces procedures, one can barely understand the motivations and issues that drove each one of the modelling/learning decisions taken. From this lens, it is just more of a description of engineering features and steps taken during the development of some tool with a reasonable purpose.\n- **ExpertMerging/ExpertMerging++:** I had a hard time understanding if they can be combined, one is the enhanced version of the other, or just focus on different challenges. Sorry about this, if I may have missed something, but it's not clear to me, and I would need additional clarification.\n- **Performance and results:** I don't have doubts about the honest empirical results provided apart from the fact that the code and software are not accessible to the reviewer ( I mentioned this in the next section). The main problem that I see is that I don't perceive superior performance from the experiments provided for a method and procedure, which I don't know how to reproduce or to understand from a research decision-making viewpoint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SH3XR9taOZ", "reviewer": "Reviewer_4vJ1", "summary": "This paper addresses the issues of task alignment and inter-layer heterogeneity in model merging, and presents the training-light Expert Merging and Expert Merging++ methods. Expert Merging learns layer-wise coefficients to align the merged model’s hidden states and logits with those of expert models using unlabeled data, while Expert Merging++ introduces importance-guided, chunk-wise coefficients to further capture inter-layer heterogeneity. Extensive experiments on both LLMs (Mistral with Chat, Math, and Code experts) and MLLMs (InternVL2.5 and Qwen2-VL across five visual reasoning tasks) demonstrate that the proposed methods consistently outperform training-free and prior training-based baselines, including WUDI v2, AdaMerging++, and even supervised Mixture Training in some cases.", "questions": "1. Why does aligning hidden states + logits offer better task alignment than entropy-based or interference-based objectives? Beyond the intuitive justification, is there any strong analysis or evidence?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Well-motivated issues and clear solutions:** The paper identifies two critical limitations in model merging research, i.e., task alignment and inter-layer heterogeneity, and proposes direct and effective solutions for them.\n\n- **Comprehensive evaluation and insightful analysis:** The experiment is well-designed and comprehensive. Its evaluation spans multiple architectures, diverse tasks, and includes comparisons against many strong baselines. The consistency of improvements strengthens the initial claims. Moreover, this paper presents some meaningful discussions, such as the layer-wise importance analysis.", "weaknesses": "- **The proposed methods are somewhat incremental:** While Expert Merging and Expert Merging++ are both effective, hidden-state and logit alignment losses are well established in knowledge distillation and model compression. If the novelty and effectiveness of the proposed methods are attributed mainly to the combination of techniques such as layer-wise coefficient learning and importance-guided chunking, the contribution seems somewhat incremental.\n\n- The proposed methods involve some empirical tricks/techniques, such as the selection of hidden alignment layer and task weights, would this affect their adaptation and transferability?\n\n- In addition, the proposed methods involve a considerable number of coefficients/hyperparameters that require tuning, would this affect their robustness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "StpYUlYvzs", "reviewer": "Reviewer_6nLa", "summary": "This paper proposes a new method, Expert Merging (and its extension Expert Merging++), for merging multiple domain-specialized expert models into a single large language model (LLM) or multimodal large language model (MLLM). The approach is “training-light,” requiring only a small set of unlabeled calibration data to learn layer-wise (and chunk-wise) coefficients that align the merged model’s hidden states and logits with those of the experts. The method introduces a layer-importance metric to guide parameter allocation, aiming to improve performance and efficiency. Experiments on several LLM and MLLM backbones (Mistral, InternVL, Qwen2-VL) show that the proposed method outperforms strong training-free and training-based baselines, and in some cases even surpasses supervised mixture training.", "questions": "* Please provide more empirical or theoretical evidence that your layer-importance metric is robust and generalizes across architectures. For example, does it correlate with actual task performance sensitivity per layer?\n* Can you provide more theoretical justification for the choice of alignment objectives (hidden states and logits)? Under what conditions does this guarantee downstream task preservation?\n* Please analyze and report what happens when expert models have overlapping or conflicting capabilities. Does the merged model degrade on some tasks? Are there cases of negative transfer?\n* In your experiments, the merged model (Expert Merging/Expert Merging++) sometimes outperforms the individual domain expert models on certain tasks. Could you provide a detailed explanation or analysis for why the merged model is able to surpass the best domain expert (or mixture training), given that it is only aligning to the experts' hidden states and logits using a small amount of unlabeled data? Is this effect consistent across different tasks and domains, or could it be due to overfitting, ensembling effects, or other factors? Please clarify the underlying mechanism and provide supporting evidence or ablation studies.\n* Moreover, is Expert Merging (for example, on InternVL2.5 grounding tasks) effectively a performance ceiling, or—given the same data budget (i.e., access to all domain data)—could a domain-targeted method or supervised fine-tuning achieve strictly better results in that domain? Please clarify the underlying mechanism and provide supporting evidence or ablation studies.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper addresses a real and growing problem—how to efficiently combine multiple expert models without the cost of joint training or serving many models.\n* The method requires only a handful of unlabeled samples per task, making it highly practical for scenarios where labeled data is scarce or unavailable.\n* By aligning both hidden states and logits, the method goes beyond parameter-space alignment and attempts to preserve downstream task behavior.", "weaknesses": "* The method assumes that expert models are complementary, but does not analyze or address what happens when experts have conflicting or overlapping capabilities. There is no discussion of negative transfer or catastrophic forgetting.\n* The method relies on aligning hidden states and logits, but there is little theoretical analysis or justification for why this should guarantee optimal downstream performance, especially when expert models may have conflicting behaviors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dp3mPMD11l", "reviewer": "Reviewer_4ve8", "summary": "This manuscript addresses the limitations of existing model merging methods—where training-free methods rely on manual coefficients and training-based methods overlook task behavior alignment and layer heterogeneity—and proposes two lightweight training-based model merging schemes: Expert Merging and Expert Merging++.\nThe former learns layer-wise coefficients using only 5–10 unlabeled calibration samples. It explicitly aligns the hidden states and logits of the merged model with those of the expert models, and introduces coefficient regularization to ensure optimization stability, as well as a task weight loss to achieve controllable cross-domain trade-offs.\nBuilding on the former, the latter combines learned coefficients + task vector magnitude + parameter count to construct a normalized layer importance metric. It allocates more block-wise coefficients to high-importance layers and keeps low-importance layers lightweight, further adapting to layer heterogeneity.\nThe manuscript validates these two methods across multi-task scenarios on LLM and MLLMs. Both methods outperform training-free and existing training-based baselines, with Expert Merging++ delivering better performance—it even surpasses supervised Mixture Training in some scenarios.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1 The authors' experiments cover 3 different models (1 LLM and 2 MLLMs) as well as multiple datasets, making the experimental design relatively comprehensive.\n\n2 The proposed method is relatively lightweight and relies on only 5–10 samples.", "weaknesses": "1 Regarding the selection of unlabeled calibration samples, the authors do not provide clear details. Are the samples selected randomly? If random selection is adopted, the randomness of sample selection may reduce the reproducibility of the results.\n\n2 For the selection of hyperparameters α and κ, although the authors provide the final values in the appendix, there is a lack of ablation experiments for hyperparameters and the range of parameter selection. This makes it impossible to further verify the parameter sensitivity.\n\n3 As shown in Figure 2, the authors also merged the base model. What would the result be if only the expert models were merged?\n\n4 What are the specific detailed settings for the optimization process of the objective function? For example, what are the values of parameters such as the number of iterations?\n\n5 It is suggested that the authors classify the different comparative methods in Tables 1, 2, and 3 by type, such as training-based and training-free types", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)