A Systematic Study of In-the-Wild Model Merging for Large Language Models
Authors: Oğuz Kağan Hitit
Leander Girrbach
Zeynep Akata
Venue: Accepted by TMLR ()

Abstract: Model merging combines multiple fine-tuned checkpoints into a single model without additional training, offering an attractive approach to reusing models and efficiently improving performance. However, it remains unclear whether the advantages reported for settings where all merged experts have distinct roles and are tuned on clearly separated tasks also hold in settings where the merged experts do not have clearly distinct roles, but are trained on overlapping or even conflicting objectives. To evaluate this setting, we present a large-scale, systematic evaluation of “in-the-wild” model merging of heterogeneous experts, that may have been trained on overlapping or conflicting objectives. Concretely, we evaluate six state-of-the-art merging methods, including recent subspace methods, across four open-weight LLMs, twelve fine-tuned checkpoints per base model, and sixteen standard LLM benchmarks. Evaluating through standardized benchmarks, we measure both the probability that a model merged from a heterogeneous set of experts outperforms the base model and we measure relative gains over the best individual checkpoint. Our results show that the oldest and simplest method, Task Arithmetic, is the only approach that reliably yields performance gains on LLMs in this “in-the-wild” setting. Other interference-aware and subspace merging methods typically do not result in notable improvements over the base model. Our findings indicate that current merging techniques mostly do not enable extracting useful weight updates from heterogeneous and potentially conflicting versions. This motivates the design of LLM-specific merging algorithms and merging-aware fine-tuning methods.

Published in Transactions on Machine Learning Research (03/2026)
A Systematic Study of In-the-Wild Model Merging for Large Language Models
Oguu Kağan Hitit* Kog University
ohstit20@ku.co tdu.
Leander Girrbach
Technical University of Munich
Munich Center for Machine Learning Helmholtz Munich
Zeynep Akata
zeynep, hawpepp.oaattccccccmoooeemmmeeee
Technical University of Munich
Munich Center for Machine Learning
Helmholtz Munich
Reviewed on OpenReview: https: // openreview. net/ forum? id- 6zSIyrgS7J
Abstract
Model merging combines multiple fine-tuned checkpoints into a single model without addi- tional training, offering an attractive approach to reusing models and efficiently improving performance. However, it remains unclear whether the advantages reported for settings where merged experts have distinct roles and are tuned on clearly separated tasks also hold in settings where the merged experts do have clearly distinct roles, but are trained overlapping
conflicting objectives. To evaluate this setting, present large- scale, systematic evaluation of "in-the-wild" model merging of heterogeneous experts, that may have been trained on overlapping conflicting objectives. Concretely, we evaluate six state-of-the-art merging methods, including recent subspace methods, across four open- weight LLMs, twelve fine-tuned checkpoints per base model, and sixteen standard LLM benchmarks. Evaluating through standardized benchmarks, we measure both the probabil- ity that model merged from a heterogeneous set of experts outperforms the base model and mcasure relative gains the bost individual checkpoint. Our results show that the oldest and simplest method, Task Arithmetic, is the only approach that reliably yields performance gains LLMs in this "in-the-wil"" setting. Other interference-aware and sub- space merging methods typically do not result in notable improvements over the base model. Our findings indicate that current merging techniques mostly do not enable extracting useful weight updates from heterogeneous and potentially conflicting versions. This motivates the design of LLM-spocificc merging algorithms and merging-aware fine-tuning methods. Code available at https://gittut: com http://....oo/aggaaaiiii////rg//
1 Introduction
Recently, model merging has gained considerable attention due empirically strong efficacy combining different models with the same architecture. Among the most intriguing observations is the phenomenon of constructive interference, where merged model outperforms its individual base models (Stojanovski et. al., Yadav et al.) 2023 Roth al., 2024). In this paper, we focus specific instantiation of this phe- nomenon: whether we can improve the general capabilities of large language models (LLMs) by merging a heterogeneous set of "in-the-wild" fine-tuned versions. Answering this question is interesting because, if
2022
Work done while at Technical University Munich.

---

Published in Transactions on Machine Learning Research (03/2026)
possible, it allows us to derive an improved version of the base model with minimal additional cost. This model can, in turn, be used to produce stronger specialized versions, like the setup in (Roth et al. 2024). Previous work (He et al. 2025 Cohere et al. 2025) has shown that final merged models can retain significant in-domain performance of the individual merged chockpoints but do not surpass them. Orthogonal to this setting, our research evaluates whether merging multiple helerogeneous, potentially overlapping or conflict- ing, checkpoints can produce models that outperform any individual checkpoint, as measured by average performance across wide range of tasks. In summary, we evaluate whether merging heterogeneous models can lead to an overall improved model, beyond equipping the base model with task-specific performance from one or more fine-tuned versions (He et al. 2025
Understanding this question is important for both scientific and practical reasons. On the practical side, organizations often accumulate dozens of fine-tuned checkpoints tailored to specific domains, tasks, or use cases. These checkpoints do not necessarily harmonize or contribute towards common improvement di- rection. If they can jointly improve the underlying model beyond any individual checkpoint, this provides practical application for their reuse. Additionally, understanding which methods and settings enable this kind of improvement provides insight into how knowledge is distributed in the parameter space of LLMs, offering clues about the geometry of fine-tuning and the limitations of weight-space interpolation.
In this study, we address this gap by conducting a large-scale, systematic evaluation of state-of-the-rt model merging techniques across multiple LLM families, a heterogeneous, "in-the-wild" set of fine-tuned checkpoints, and a wide suite of benchmarks. Our work seeks to answer the following research questions: (1) Can we produce an improved version of the base LLM by simply merging multiple fine-tuned versions? (2) Which weight interpolation-baseed model merging techniques enable such improvement? (3) Do recently proposed merging methods that operate on the subspaces of weight matrices also improve performance of "inlthe-wild" merging in LLMs?
In summary, our main contributions are: (1) We systematically evaluate six model merging methods on four LLMs across 16 benchmarks; (2) We find that most merging methods do not produce models that outperform all involved individual checkpoints. This motivates further research on how to leverage capabilities of existing heterogeneous model versions and how to combine them; (3) Among all six evaluated merging methods, only Task Arithmetic, the oldest and simplest of the methods, consistently yiclds models that outperform all involved individual checkpoints. However, performance gains are limited. The partial success of Task Arithmetic shows that even in heterogeneous pools of model versions, there is complementary knowledge that can be used to improve the base model, but it is non-trivial to extract, and more sophisticated merging methods are not better suited to do so.
These claims are supported by extensive experiments: We evaluate four LLMs, spanning different model families (Qwen3 and Llama3) and different model sizes (3B, 4B, and 8B), on 16 standard LLM benchmarks, which allows for generalizable insights. Observed trends are consistent across the evaluated models and benchmarks, so they can be assumed to hold for other models as well. Finally, our insights are relevant to the model merging and broader machine learning community, as systematic evaluation of subspace merging methods on LLMs has been lacking so far, and our results are likely to inspire future research on model merging, specifically targeting LLMs and heterogeneous, "in-the-wild" merging.
2 Related Work
Model merging for LLMs has been surveyed extensively. Li et al. 2025b) review model fusion across archi- tectures and disjoint training runs. Yang et al. (2026) group LLM merging approaches into "Pre-Merging Methods" (weight alignment)), "During-Merging sis". We use "merging methods" " to denote the second category. Ruan et al. (2025) classify merging approaches with emphasis on pruming, while Yadav et al. (2025) systematically study merging across model scales up to 64B parameters. Our work complements these analyses by incorporating additional recent subspace methods and evaluating widely used open-weight models (Qwen3 and Llama. 3) rather than proprietary PaLM models.
Methods" (weight combination), and "Theories and Analy-
2

---

Published in Transactions on Machine Learning Research (03/2026)
Base LLM
Sampled Checkpoint Subsets
TA-Based Methods
LM-Eval-Horness
Apply Model
Llomo.32 (3B)
n=2
Task Arthmmetic
TruthfuuiA
Pair each model with 12 finetuned
TEES Meraing
Liamo 3. (88)
De4
Merging
Model Stock
Evaluate
MMLU
mergekit
Subspace Methods
Merged
Qwwnns (48)
checkpoints
m=10
Models
TSV-Merge
@wen3 (BB)
n=12
1sc-C Subsspoce Boosting
Hallsswwog
Figure 1: Our evaluation protocol pairs each base large language model (LLM) with 12 publicly available heterogencous, i... "in-the-wild", chockpoints and repeatedly samples subsets to merge. The sampled check- points are merged using three task arithmetic (TA) and three subspace merging methods. Resulting merged models are evaluated on 16 standard LLM benchmarks from Im-eval-harness to analyze trends in which merging methods consistently work well on LLMs.
2.1 Background on Motivations and Theoretical Foundations of Model Merging
Stochastic weight averaging (SWA) shows that combining weights from multiple chockpoints of the same
model improves performance (Izmailov et al., 2018 Guo et al.] 2023). By averaging points along a training
trajectory, SWA benefits from mode connectivity (Draxler al. 2018 Garipov et al. 2018 Kuditipudi al.
2019 Benton et al. 2021), i.e. the observation that distinct optima are linked by low-loss paths. Thus, model variants sharing optimization trajectory can be interpolated with negligible performance loss (Frankle et al. 2020). Robustness to small weight perturbations further supports such combinations (Arora el. al.
2018 However, merging models trained from different bases requires neuron alignment (Tatro et al.) 2020 Entezari et al. 2022), and several methods address this (Ainsworth et al. 2023 Peña et al. 2023 Rinaldi et al. 2025). Here, however, we restrict our focus to fine-tuned LLM checkpoints derived from a common base and therefore do not consider neuron alignment.
2.2 Detailed Overview of Model Merging Techniques and Paradigms
Weight Interpolation Based Methods. Wortsman et al. (2022) introduce Model Soup, which averages
greedily aggregates aligned models. For finc-tuned variants of shared basc, Ilharco et al. (0023 propose
Task Arithmelic (TA), main method in our study (detailed in Section 3.1p. Several approaches refine TA to reduce interference across merged models. DARE Yu et al. 2024) drops fraction of delta parameters and rescales the rest, and DAREr (Deng et al. 2025) adapts this for extreme pruning rates. DELLA (Deop) et al. 2024) prunes by ude, preserves consistent parameter signs, and fuses selected updates. Model Breaderumbs (Davari Belilovsky] 2024) applies layer-wise masking to remove large outliers and small noise, while EMR--Mrrying (Huang al. 2024b) masks and rescales task vectors individually. TIES- Merging (Yadav al.) 2023) trims small updates, enforces sign consensus, and merges only aligned parameters. SLERP (Shoemake 1985) performs geodesic interpolation to preserve geometric structure.
Training-Based Methods. Others optimize parameters such as interpolation coefficients, for instance LoraHub (Huang et al.) 2024a] merges LoRA adapters (Hu et al. 2022) via weighted averaging with gradient-
free coefficient tuning on validation data. Routing- methods combine components in MoE. architectures
[Kan et al. 2025 Li et al.] 2024a] Muqeeth al.] 2024 Tang et al.] 2024a Lu al.] 2024). Additional
techniques use data statistics
validation sets to select averaging coelficients (Yang et al. 2024b;
Zhou
et al.
2024
Zhang et al. 2024 Li et al.) (0255)), pruning masks (Wang et al.
2024 Tang al.
2023
Kong
ct al.
2024
or parameter rescaling (Matena & Raffel 2022 Jin et al. 2023
Daheim et al. 2024
Akiba
et al.
2025)
optimize merging strategies via evolutionary search. Post-training or model linearization can
further improve mergeability (Yang et. al. 2024a Ortiz-Jimenez et. al. 2023 Tang et. al.
2024b Liu et al.
2024).
Subspace Merging Methods. Recent approaches treat merging as a problem within low-rank task sub- spaces rather than full parameter space. (Skorobogat. al. (025)) address the rank collapse of task vectors
3

---

Published in Transactions on Machine Learning Research (03/2026)
with subspace-boosted merging, using SVD to preserve expressive directions. In parameter-cffiient fine-tuning (PEFT), Stoica et al. (2025) introduce KnOTS, which aligns LoR updates into shared subspace to improve compatibility. Marczak et al. (2025) analyze singular value spectra to decompose updates into common and task-specific subspaces, mitigating interference. (aam et al. (2024) frame merging as solving linear systems in lask parameter subspaces. Finally, Gargiulo et al. (2025) use per-layer SVD to isolate task-relevant directions, showing that singular vectors can guide merging to reduce destructive interference.
Constructive Interference. Constructive interference is the main focus of this study. It occurs when merged model outperforms its constituent experts by leveraging their complementary strengths. WWotsmaan 2022 show that averaging fine-tuned weights improves generalization compared to single checkpoints. Ilharco et al. 2023) demonstrate that linear combinations of task vectors enable transfer and domain gen- eralization. Yadav et al. (2023) highlight that resolving weight conflicts produces merged models that consistently outperform their parents. Similar findings exist in reinforcement learning (Ramé et al. 2023b; 2024b) and continual learning (Stojanovski et al., 2022). However, most evaluations focus on modcrato-scale Transformers like BERT (Devlin et. al.; 2019) T5 (Raffel et. al. 2020), leaving the generalization to modern large-scale LLMs open question.
ct al.
2.3 Practical Applications of Model Merging
Model merging naturally enables multi-task models derived from task-specific variants (Wang et al. 2024 Matena & Raffel 2022 Dahcim et al., 2024). For example, [Awasthy (awastyy et al. (2025) build a strong teacher for distillation by merging models trained on different objectives. Merging also mitigates catastrophic forgetting
during fine-tuning and continual learning, helping models retain base-model knowledge (Alexandrov et al.
2024
Porrello et al.] 2025; Zhhu et al.] 2024 Marczak et al.) 2024 Xiao et al.) 2024 Chitale et al. 2023
Qazi et al. 2024 Stojanovski et al. 2022). Weight averaging further enhances out-of-distributii
[Izmallov
et al. 2018 Ramé et al.) 2022: 2023a 2024b
Jolicoeur-Martineau Martineau et. al. 2024 Jain et al. 2023 el al.
2025c: and out-of-domain generalization (Arpit et al.] 2022 Li et al. 2024b), strengthening robustness to
adversarial attacks and jailbrcaks (Cong et al., 2023
Croce al. [2023] Gallego 2024). Finally, morging
supports instruction luning and aligument of RLHH-tunne LLMs (Fu el al. 2024
Ramé et al. 2024a
3 Do Methods Based on Task Arithmetic Enable Constructive Interference?
Our goal is to systematically evaluate if existing model merging techniques can achieve constructive inter- ference in LLMs by merging heterogeneous fine-tuned versions. We focus on methods similar to the seminal Task Arithmetic method (Ilharco et al., 2023), which merge models by interpolating their weights. Our evaluation includes three merging techniques, four base LLMs, 12 fine-tuned versions of each LLM, and 16 benchmark tasks. This allows us to provide a comprehensive overview of the strengths and limitations of merging methods when applied to in-the-wild merging of LLMs.
3.1 Merging Methods in this Study: Task Arithmetic, TIES-Merging, and Model Stock
We evaluate three popular algorithms that represent distinct paradigms for model merging: Task Arithmetic [Ilharco et al. 2023), TIES-Merging (Yadav et al. 2023), and Model Stock (Jang et al. 2024). These meth respectively capture linear vector arithmetic, interference-awar adjustment, and geometric interpolation. We do not include other recent. approaches such as Consensus Merging (Wang et al. 2024 Model Soups [Worssman et al. 2022), these methods are likely to perform similarly to simple averaging under large-scale conditions or rely on domain-specific heuristics that make systematic comparison difficult. In the following,
we briefly introduce all evaluated merging methods, and we visualize them in Fig. 2
Task Arithmetic. Task Arithmetic MTlaacoo et al.) 2023) frames model merging as vector addition and subtraction in weight space, treating fine-tuning updates as fask vectors. Given a base model Wo and its fine-tuned variant W,, the corresponding task vector is defined
AW, W Wo-
(1)
4

---

Published in Transactions on Machine Learning Research (03/2026)
Task Arithmetic
Model Stock
Ideal Center
WA
W
WB
Weighted
TTA
We
TB
Averaging
TA
TMS
Wc
TA
Wo
W
Task Vectors
Projected Center
WA
Task Vectors Parameter-wise
Trimmed Task Vectors
Masked Task Vectors
TIES-Merging
(()
A++1l y
WA
Trim
72185
WB
Merge
KG-22
M)+22
TA
TB
TO
T
Wc
Elect Sign
Wo
W
Task Vectors
Sign Vector
Overview of task-arithmetic; based model merging met hods: Task Arithmetic,, TTES-Merging, and Model Stock. Given base model Wo and finc-tuned checkpoints W,, Task Arithmetic computes task vectors AW; W. Wo and merges them via weighted addition. TIES- Meringg extends this by (1) trimming small- magnitude parameter updates, (2) enforcing sign-consistent updates across checkpoints, and (3) merging only aligned parameters to reduce interference. Model Stock instead interpolates between Wo and the geometric center of the fine-tuned checkpoints based on estimated inter-model angles.
Figure 2:
These task vectors encode learned task-specific knowledge and can be algebraically combined to transfer, compose, or remove capabilities across models. A merged model W Weerged can thus be expressed as
n AWTA Σ AWW,
W Weergod W. XAWTA.
(2)
where a; denotes the coefficient assigned to each expert model, and À is global, scalar scaling factor. Setting = 1 for a target task and a, = -1 for an undesired task allows additive or subtractive transfer, respectively, enabling "forgetting by negation" and "learning by addition". In our experiments, we set a, 1, and À for all checkpoints.
TIES-Merging. TIES (Yadav et al. 2023 also uses task vectors, but attempts to mitigate conflicts between merges in weight space. Given set of fine-tuned weights (Wi and common initialization Wo, each task vector AW; is defined in Task Arithmetic (Eq. ( The method proceeds in three stages. (1) Trim: within each layer, only the top-&% of parameters in AW; based on absolute magnitude are retained, and the rest are reset lo zero, producing a sparsified update AW!riimeed This slep removes weak or noisy signals. (2) Select signs: for each parameter, a sign consensus across all checkpoints A AWtrimmed computed. Parameters in A AWtriimeed whose sign disagrees with the sign consensus are maskod out, yiclding AW.masked This sign selection ensures that only updates with consistent directional agreement contribute to the merge, while conflicting parameters reset to the base value. (3) Disjoint merge: Similar to Task Arithmetie, the
5

---

Published in Transactions on Machine Learning Research (03/2026)
final merged model is computed as
A TWTISS
A W W.maseed
W Werrged Wo + XAWTIES.
(3)
Intuitively, TIES preserves the relevant lask updales while filtering out contradictory ones.
Model Stock. Model Stock (Jang et al., 2024) moves the merged weights toward the geometric center of set of fino-tuned chockpoints: given pre-trained weights Wo and fine-tuned checkpoints (W:) Model Stock selects the point that is geometrically closest to the unknown center, i.e. the true geometric midpoint of the shell that the checkpoints would define in weight space by a layerwise interpolation between Wo and the average of the fine-tuned variants (Wavg). Mathematically, the method computes the merged model as
N Σ Wi, ==1
N cos 0 1 (N 1) cos
W Wave
1 1=
W Wmergod = Wavg + (1 (1-t) t) Wo-
(4)
N
where N denotes the number of fine-tuned variants, t denotes the interpolation factor, 0 denotes the mean inter-model angle (measured layerwise) among the fine-tuned variants. When the checkpoints are tightly aligned (small 0), & is larger and the merge relies more on Wavgi when they are more diverse (large 0), decreases and the merge leans toward Wo- We acknowledge that Model Stock, in its original formulation, is intended to merge multiple checkpoints from the same training trajectory. However, formally, there no constraint against applying Model Stock to checkpoints fine-tuned on different datasets. Thus, we include it in our comparison for more complete comparison, and an explicit "stress-test" for Model Stock.
3.2 Experimental Setup
Models and Checkpoints. We evaluate four open-weight LLMs spanning two familics and paramcter scales: LLAMA 3.2 3B, LLAMA 3.1 8B (Dubey al. 2024h, QWEN3 4B, and QWEN3 8B (Yang et al. 2025). This diversity supports generalizable conclusions. For each base model, we merge 12 publicly available fine- tuned checkpoints that cover various objectives and domains (Appendix A). Merging methods use mergekit (Goddard et al., 2024) with hyperparameters fixed to values identified in Appendix C We set A = 1.0 for Task Arithmetic and Model Stock, and = 0.1 for TIES-Merging. We use top-10% magnitude threshold for TIES- Merging
Sampling Checkpoints Merge. To study how performance scales with the number of merged models, follow progressive merging strategy. For each base model and method, we evaluate the base model, all 12 individual fine-tuned checkpoints, and merged models containing (2, 4, 6, 8, 10) and (12) checkpoints. Because the number of possible combinations grows combinatorially, we uniformly sample 15 subsets for each merge size and report the mean performance. The same subsets are used across methods, ensuring differences arise from the merging algorithms rather than checkpoint selection.
3.3 Evaluation on Standard LLM Benchmarks
Benchmarks. We evaluate every base model and merged configuration with the im-enaluation-tarress library (Biderman et al., 2024). using its standardized implementations for the following Open LLM Leaderboard tasks: arc_easy, arc arc_challln are_challenge, hellaswag, winogrande, boolq, piqa, openbookqa, mmlu, medmcqa, leaderboard_geee _appbaarrdd_pa.,,.oa leaderboard bbh, and leaderboard, mmlumprlloooompppp pro. These benchmarks collectively cover multiple evaluation axes including com- monsense and scientific question answering (e.g., commonsense _aa medmoqa), multi-step reasoning (e.g., are pre_challenge, bbh), and instruction-followigg (e.g., hellaswag, winogrande). We use the default decod- ing setup and per-task n-fewshot configuration of im-eval-harness for all benchmarks. We do not apply any chat, templates, and we report. accuracy for each task. The exact. n-fewshot values are provided in Appendix
commonsense_cgr _a, headga, prost, truthfulqa_nse
Results. Fig. 3 we show the average performance across benchmarks for all merged models and merging methods (task-wise accuracies are in Appendix B). For merging methods, notice clear trends that hold

---

Published in Transactions on Machine Learning Research (03/2026)
1aama332-3B-Instruct
Qwen3-4B
-lama-3.-88-Instrucc
Qwen3-8B
Model Stock
0.60
0.60
0.60
0.60
lask Arthhmetic
Average Accuracy
TIES
Baseline
0.55
0.55
0.55
0.55
0.50
0.50
0.50
0.50
0.45
0 012
6 8
10
12
0.45
0
1.22
6
8
10
12
0.45
0 0001
8 10
12
0.45
0
1
4
10 12
Number of merged models (n)
Number merged models (n)
Number of merged models (n)
Number of merged models (n)
Figure 3: Average accuracy and standard deviation of the models across all benchmarks. From left to right, models are LLAMA 3.2 3B, QWEN3 4B, LLAMA 3.1 8B, QWEN3 8B, respectively. Shaded areas indicate the standard deviation over different samples of merged checkpoints.
regardless of the model. Task Arithmeelic steadily improves as more experts are combined, becoming reliably superior to the base model once moderate number of experts merged. This clearly demonstrates the existence of constructive interference in LLMs: merging several independent fine-tuned checkpoints can produce a model that surpasses both the base LLM and any individual expert. At the same time, the improvement achieved through merging is modest, generally less than 1% averaged over all tasks. At most, we can achieve 13.07% improvement for prost task LLAMA 3B when all twelve checkpoints are merged with Iso-C. C. Model Stock does not deviate significantly from the performance of the basc model, and also weights stay very close to the base model. This shows its limited ability in finding interpolations of different fine-tuned versions that generalize better. Finally, TIES-Mergine despite building top of Task Arithmetic and using& more sophisticated approach, remains tightly clustered around the base model's performance. While it demonstrates improvements similar to Model Stock, it consistently falls short of the gains achieved by Task Arithmetic, maintaining a lower average accuracy across all merged model counts.
These observations are quantified in Table I which reports both the probability of surpassing the base model and the corresponding relative improvement for each n. Across all four models, Task Arithmetic exhibits a clear, monotonic trend: both the success probability and the relative gain steadily increase more models are merged. For example, for LLAMA 3B, TA improves over the basc model in only 20% of combinations n=2, but already reaches 80% at n-4 and 100% for all 2 6, with the average relative improvement rising from -0.27 at n=2 to +0.89 at n=12. This pattern consistently appears in the other models as well: TA reaches 100% success for all n 2 4 in LLAMA 8B and all n 2 6 in both Qwen models, with relative gains reaching as high as +1.62 (LLAMA 8B, n=10) and +0.88 (QWEN 4B, 1=1 Model Stock follows a similar but weaker pattern: improvements are small but consistently positive at higher n values, aligned with its conservative update rule. For instance, LLAMA 8B shows gains growing from +0.06 at n=2 to +0.36 n=12, and +0.36 is the highest relative improvement that Model Stock achieves across all models. TIES- Merging achieves a high probability of improving over the base model (averaging 75 87% across n 2), but the magnitude of these gains remains small and plateaus quickly, with the average relative improvement hovering around +0.17. TIES also exhibits instability at higher merge counts for certain models. For QWEN- 8B, performance degrades from a peak of +0.16 at n = 4 to -0.08 at n = 12. As discussed in Appendix C this behavior likely a consequence of the method's sensitivity to task vector magnitude in heterogeneous settings.
It is also important to note that individual fine-tuned checkpoints rarely outperform their own base model: at n=1, fewer than 50% of the checkpoints exceed the accuracy of their corresponding base. In other words, randomly selected expert is more likely to underperform than improve upon the base model. This confirms that the gains observed at higher n do not stem from simply picking stronger experts, but rather from the constructive interference produced by merging multiple weaker ones.
Beyond improvements over the base model, we also examine whether merging can surpass the strongest individual fine-tuned checkpoint. As shown in Table 2 Task Arithmetic reliably exceeds the best expert. for
7

---

Published in Transactions on Machine Learning Research (03/2026)
Model
Method
Base
n=1 (12)
n=2 (15)
n=4 (15)
n==6 (15)
n=8 (15)
n=10 (15)
n= (1)
TA
50.3
17 -0.85
20 /0.27
80 / +0.39
100 +0.64
100
/ +0.66
100
/ +0.68
100
+0.89
LLAMA-3B Model Stock
50.3
17
-0.85
60 +0.01
87 +0.02
87 +0.05
100
/ +0.05
93 +0.07
100 / +0.09
TIES
50.3
17
-0.85
47 +0.02
60 +0.04
87 +0.24
+0.18
+0.28
100 / +0.27
TA
56.9
25
-1.28
60 / +0.38
93 / +1.05
100 +1.36
100
+1.60
100
/ +1.62
100 / +1.56
LLAMA-8B Model Stock
56.9
25
-1.28
93 +-0.06
100 +0.19
100
++0.30
100
+0035
100
+-032
100 / +0.36
TIES
56.9
25
-1.28
100 -0.28
93 +0.21
100 / +-025
100
+0.28
+0.17
100 / +0.08
TA
55.3
25
-1.34
-0.09
80 / +0.17
100
+0.44
100
+0.64
100
/ +0.76
100
+0.88
QWEN-4B
Model Stock
55.3
25
-1.34
13 -0.01
20 -0.02
47 -0.01
27
-0.01
27 -0.01
100 / +0.01
TIES
55.3
25
-1.34
66 +-0.66
93 40.14
100
40.17
100 / +..22
100 / +-025
100 / +.024
TA
58.8
33
-0.97
67 -0.09
100 +0.39
100
+0.50
100
+0.56
100
+0.67
100
+0.74
QWEN-8B
Model Stock
58.8
33/
0.97
47 +0.01
100
+0.04
98 +0.03
100
/ +0.03
100
+0.05
100
/ +0.05
TIES
58.8
33
-0.97
100
00.11
93 40.16
60
0.0.6
53
1.004
40
+0002
-0.08
TA
55.3
25
-1.11
49 -0.02
88 +0.50
100
4.0.4
100
/ +0.87
100 / 4.093
100 / +.002
Average
Model Stock
55.3
25
-1.11
53
+0.02
77
+0.06
82
+0.09
82 +0.10
80
+0.11
100 / +0.13
TIES
55.3
25
-1.11
78/
/ 10.11
85
40.14
87
+01.8
85
4-0.7
80
0.17
+-0.13
Table 1:
Constructive interference results for Task Arithmetic-basee merging methods applied to models. Each entry contains two quantities: the percentage of merge combinations that exceed the base model's accuracy, and the mean relative accuracy improvement for those combinations. Column headers use the
notation n = m (k), where n is the number of models merged and k is the number of evaluated merge combinations for that value of n. Base indicates base model accuracy.
three of the four model families once n 2 4. For example, in QWEN-4B, TA delivers +1.02 improvement at n-4, which increases steadily to +1.72 at n=12. QWEN-8B shows almost identical pattern, with gains rising from +1.14 at n=4 to +1.49 at n=12. LLAMA-3B also surpasses its best expert once n 2 4, improving from +0.32 at n to +0.82 at 1212. The only exception is LLAMA-SB, whose strongest. fine-tuned check- point is unusually strong: merging never exceeds it, although the deficit shrinks meaningfully from -1.76 at n=2 to only -0.58 at n=12. These results demonstrate that heterogeneous, "in-tho-wild" model merging frequently produces models that outperform not only the base model but also the best available fine-tuned checkpoint in general capabilities.
To better undersland the mechanism behind these performance differences, we measure the magnilude of the task vector, [[0merged - Obaseell |2, as function of n in Fig, 4 Across all model families, Task Arithmetic, Task Arithmetic with Subspace Boosting, TIES, TIES with Subspace Boosting, and Model Stock remain very close to the base model, with lask-vector norms generally below 50 for all n. In contrast, Iso-C and TSV-M produce substantially larger deviations, with distances often in the 100-30 300300 range for L.LAMA-BB and QWEN-4BB, and exceeding 300 for LLAMA-8B and QWEN-8B, These displacements in parameter space correlates strongly with the performance degradation observed in Fig. and Fig. G supporting the hypothesis that merging algorithms that aggressively change the weights and move outside the base model's loss basin are responsible for the observed catastrophic forgetting.
4 Do Subspace Merging Methods Enable Constructive Interference?
In Section we found that only Task Arithmetic consistently achieves constructive interference in LLMs when merging heterogeneous experts, whereas Model Stock and TIES Merging, which alternative meth- ods operating in weight space, do not yield significant gains. However, recently, subspaco-based model merging methods have achieved significant improvements when applied to vision-language models. Unlike weight interpolation methods that directly operate in full parameter space, subspace- approaches merge models by aligning or projecting their task updates into subspaces. This approach mitigates rank collapse, isolates compatible update directions, and improves robustness during model composition. Therefore, we also evaluate subspace-based model merging methods, which have been primarily evaluated vision-language
8

---

Published in Transactions on Machine Learning Research (03/2026)
Liama-3.2-3B-Inse
Qwen3-48
Hama-3.1-88-instruce
Qwen3-88
1000
250
500
Average Norm
100
200
75
150
300
100
200
25
200
0
0
10
Number merged madels (n)
Number of merged models (n)
Number of merged madels (n)
Number of merged models (n)
Iso-C
Stack
Task Artthmatic
Task Artthmatic
SB
TIES DES-Merging
TIES 5B
TSV-M
Figure 4: Average L2-norm of the task vectors with respect to the base model as function of the number of merged checkpoints. Each curve reports the mean Euclidean distance [[0merged Obasel2 across samples of merged models, with shaded regions indicating the standard deviation. Higher values indicate larger deviations from the base model in parameter space.
Model
Best FT
n=1 (12)
n=2 (15)
n=4 (15)
n=6 (15)
n=8 (15)
n=10
(15)
n=12 (1)
LLAMA-3B
50.4
17 / -0.92
20
-0.34
80
+0.32
100
+0.58
100
+0.60
100
+0.62
100
+0.82
LLAMA-8B
59.0
17 -3.41
13 -1.76
0 0/-1.00
0/ / -0.78
0 -0.54
0 -0.52
0 -0.58
QWEN-4B
54.5
75
/ -0.50
93
+0.76
100
+1.02
100
/ +1.28
100
+1.48
100
+1.60
100
+1.72
QWEN-SB 58.1
67
/ -0.22
80
+0.66
100
+1.14
100
+1.25
100
+1.32
100
+1.42
100
+1.49
Average 55.5
44
-1.22
52
+0.08
70
+0.60
75
/ +0.84
75 +0.97
75
+1.03
75
+1.13
Table 2: Constructive interference results for Task Arithmotic comparing merged models to the best finc- luned checkpoint across all bases. For each base model, the best fine-tuned checkpoint selected based on average performance across all evaluated tasks and used as fixed reference for all merge comparisons. Each cell reports (i) the percentage of merge combinations that surpass this best fine-tuned model and (ii) the mean relative accuracy difference. Column headers use the notation m (k), where is the number of merged models and k is the number of evaluated merge combinations.
models or small language models, such as T5, for LLM model merging, using the same setup introduced in Scction 8 Below, we give a brief overview the evaluated mothods.
4.1 Merging Methods in this Study: TSV-M, Iso-C, Subspace Boosting
We assess three representative subspacc-oriented merging methods, namely, TSV-Merge (Gargiulo et al. 2025), Iso-C (Marezak et al. 2025), and Subspace Boosting (Skorobogat et al. 2025
TSV-Merge. TSV-Merge (Gargiulo et al., 2025) compresses each task's update into dominant low-rank di- rections, orthogonalizes them across tasks, and recombines the resulting bases into an interference-minimized update. Similar to Task Arithmetic, for each fine-tuned variant E (l...., T}, task vectors (AW:(!) are created for each layer e. Then, TSV-Merge computes SVD of every layer-wiso task vector,
AW = =00000
(5)
where the singular vectors D and v are called Task Singular Vectors (TSVs) and the diagonal entries of 2X()) quantify their importance. TSV-Merge then retains only the top fraction of singular components for each (i, to control capacity and suppress noise, keeping the highest-energe directions. Then, the truncated TSVs are aggregated (suppressing & for brevity) by concatenation,
U U1 U2
Ur
E blook--iaag(E..... ET), ET),
Vi
V/2
Vr..
(6)
Bocause different tasks may emphasize overlapping directions, TSV-Merge removes this redundancy via an orthogonal Procrustes projection. Computing SVDs of the concatenated matrices U and V, the closest
9

---

Published in Transactions on Machine Learning Research (03/2026)
TSV-Merge
Decomposed Task Vectors
Orthogonalized Singular Vectors
Wa
PrDrQu
TSV TSV-Merge Update
U_ PoQu
Orthogonalize
7M
V PyDrQF
W
V. Pr4y
Took Vectors
Up
35.0
Iso-C
Iso-C Singular Value
Wx
V7
Weighted
Compute Meon Singular Value
Averoging
VT
TY4
7180
To
We
E000
22-11
E1 o1,
Subspace Boosting
Subspace Boosted Singular Values
W,
y
Subspace Boosting Update
Wa
Averaging
Weighted
Compute 5B Singular Values,
TYA
U
Wo
j min j. ny 20}
W
= (1 54 (イ>
Task Vectors
Figure 5: Overview of subspace- based model merging methods: TSV-Merge, Iso- C, and Subspace Boosting. These methods operate in low-rank task-update subspaces rather than full weight space. TSV-Merge ex- tracts dominant singular directions for each task update, orthogonalizes them via Procrustes aligmment, and recombines the aligned subspaces into a unified low-rank update. Iso-C flattens the singular value spectrum of the Arithmeitie update, producing an isotropically scaled representation of its principal directions. Subspace Boosting mitigates rank collapse by clevating weaker singular directions above cumulative-energy threshold, broadening the effective subspace captured by the merged update. In the illustration, we show the TA+SB variant, but any mask-vectorbbe vector-based merging method (e.g. TIES) could be substituted by modifying only how the merged task update is computed before applying the Subspace Boosting operation.
orthogonal factors in Frobenius norm are obtained in closed form as:
= DU.
Pv
U1 PoQT.
V PvQT.
(7)
With the aligned bases U1 and V1 in hand, -Sr-Merge reconstructs the merged variant by creating single low-rank update by reintroducing the (block-diagonal) singular values, and applying weighted addition:
WWTSV-M = U1 EVI.
Wmerred
W. + A
(8)
Conceptually, TSV-Merge is subspacc-alignment mechanism: it comprossos each task into its principal sin- gular directions, aligus those directions across tasks to enforce mutual independence, and fuses them through single low-rank reconstruction. The truncation regulates signal-noise trade-offs, Procrustes removes inter- task overlap, and the final scaling tunes how far the merged model moves from the base.
Iso-C. Iso-C MMrrzzak et al. 2025) introduces an isotropic model merging method designed to improve subspace alignment across task updates by flattening their singular value spectrum. Starting from the cunulative task vector AWTA obtained via Task Arithmetic (Eq. (2), Iso-C performs the following operation layerwise (we suppress the layer index & for brevity). It computes an SVD
TWTA = U.VT
Σ = diaiag(o)......0
where contains the singular values and r denotes the effective rank. Rather than retaining the original (typically skewed) singular value distribution, which may overemphasize few dominant. task directions.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Iso-C replaces all singular values with their mean to enforce isotropy: o - os and Enso oI,. The isotropically rescaled update and merged variant is then reconstructed as:
A USioo VT,
W mergeed W. + -AW Is
(10)
This operation equalizes the contribution of each principal direction, yiclding more balanced representation of task information. Conceptually, Iso-C can be viewed as spectrun-flattened extension of Task Arithmetic: it preserves the same subspace spanned by AWTA while imposing uniform scaling of its singular values.
Subspace Boosting, Subspace Boosting (Skorobogat et al., 2025) counteracts rank collapse, i.e. the ten- dency of merged task vectors to compress variance into a few dominant singular directions as multiple fine-tuned variants are combined. The method is applied layerwise; for clarity, we suppress the layer index throughout. Subspace Boosting performs an SVD of the merged update (AW = -UEV), ) where the diagonal entries of Σ diag(o1....,.o) .,, represent the energy of the corresponding subspace directions. The cumu-
Eis," and boosting threshold / determines the spectral lative normalized energy is computed as nj Lizi oz
cutoff index jr = min(j n, 2 5}. Singular values beyond this threshold are elevated to the cutoff value oyr, producing a flattened spectrum. The boosted update is then constructed as
f0 Jg. j Jj j> jr,
A WW boosted = -VVV,
of
W merged = Wo + AAW boosted.
(11)
Conceptually, Subspace Boosting broadens the effective subspace spanned by the merged variant by redis- tributing energy from dominant to weaker singular directions. The method is agnostic to the underlying merging strategy and can be seamlessly applied to any task-vector-basee approach, such as Task Arithmetic TeESsMerging.
4.2 Experimental Setup and Results
Experimental Setup. Apart from the merging algorithms, our setup mirrors Section 3 We integrated all available implementations into the mergekit library to provide a single, unified pipeline. We reuse the same base models as in Section 3. the same 12 checkpoints per base model, the identical subset-sampling over merge sizes, and the same evaluation configuration to isolate the effect of the merging algorithm itself. Following ablation studies (Appendix(C CJ, set X = 0.1 for TIES+SB and X 1.0 for all other methods, while fixing the Subspace Boosting threshold to 3 = 0.2.
Results. Fig. G shows the average performance across benchmarks for all merged models and subspace merging methods. Trends for the diffcrent methods are consistent across LLMs: Both TSV-Merge and Iso-C exhibit steady declines in average accuracy as the number of merged models increases, indicating that their dimensional truncation and orthogonalization operations progressively discard informative components when aggregating multiple checkpoints. In contrast, methods utilizing Subspace Boosting avoid this degradation. TIES + SB demonstrates highly stable profile, consistently remaining slightly above the base model, though it does not achieve significant. scaling behavior. TA + SB exhibits high variance at small merge sizes but improves steadily with scale, eventually matching or even surpassing the base model's accuracy at large n, at a similar level to the original Task Arithmetic. These results indicate that subspace projection and flattening generally do not produce constructive interference LLMs, whereas Task Arithmetic paired with Subspace Boosting remains the only sctup that benefits from scaling the number of exports. However, given that this trend mirrors pure Task Arithmetic (see Fig. 3 this is mostly due to TA, while Subspace Boosting is not harmful here.
In Table B we again quantify these trends by reporting the probability of surpassing the base model and the average relative improvement across merge sizes. TA + SB consistently transitions from unstable early performance to strong, near-100% success as n increases. At small merge sizes, success rates remain low- 23% n=2 with an average relative change of -4.22- but they rise stcadily to 98% at n=10 and reach 100% at n=12, with corresponding improvements of +0.89 and +1.07. Also, TIES + SB improves from 72% success (n=2) to 100% (n=12), but with capped relative gains (+0.06 to +0.36). In contrast, TSV-Merge
11

---

Published in Transactions on Machine Learning Research (03/2026)
Llama-3.23B-Instruuc
Qwen3-4B
Llama-3.188-Innstee
Qwen3-88
TA s0
0.60
0.60
0.60
0.60
Average Accuracy
TSV-M
Iso-C
0.55
Baseline
0.55
0.55
0.50
0.50
0.50
0.50
0.45
0
6
8
10
12
0.45
0
8
10
12
0.45
0 10100
8
10
12
0.45
0
10 12
Number of merged models (n)
Number merged models (n)
Number of merged models (n)
Number of merged models (n)
Figure 6: Average accuracy and standard deviation of the models across all benchmarks. From left to right, models are LLAMA 3.2 3B, QWEN3 4B, LLAMA 3.1 8B, QWEN3 8B, respectively. Shaded areas indicate the standard deviation over different samples of merged checkpoints.
Model
Method
Base
(12)
n=2 (15)
==4 (15)
==6 (15)
==8 (15)
==10 (15)
x=12 (1)
TA SB
50.3
17 -0.85
13 -1.74
67 -0.22
60 / -0.90
87 +0.50
100
-0.77
100
+1.22
TIES + SB
50.3
17 -0.85
47 +0.01
47 -0.07
87 +0.28
93 +0.21
100
+0.29
100
+0.26
LLAMA-3B
TSV-M
50.3
17 0.85
47 -0.34
27 0.544
47 0.00
27
-.200
-0.33
Iso-C
50.3
17
-0.85
33 -0.30
47
-0.39
20
-0.98
13 -0.99
0 -1.81
0
-2.48
TA SB
56.9
25
/ -1.28
27 3.10
67
8.55
87 +0.72
100 +1.79
100
-1.81
100
+1.83
TIES | SB
56.9
25
-1.28
100
10.27
100
10.33
100
10.33
100
10.31
100
0.25
100
0.20
LLAMA-SB
TSV-M
56.9
25
-1.28
33 -1.90
20
-2.39
7 -2.92
13 -2.70
0 -3.36
-3.79
Iso-C
56.9
25
/ -1.28
53 -0.65
40
-1.19
27
-2.77
20
-3.60
0 -5.94
0
-8.84
SB
55.3
25
-1.34
33
33
-0.77
47 -0.22
+0.28
+0.49
100
+..61
TIES + SB
55.3
25
-1.34
73 +0.06
87
-0.09
100
+0.14
100
+0.19
100
+-021
100
+0.23
QWEX-IB
TSV-M
55.3
25
-1.34
18 -0.98
7 / -1.95
0 -1.50
7 -1.91
0
0 -1.69
0
-2.04
Iso-C
55.3
25
-1.34
27
-0.62
7 -3.05
-3.03
7 -4.58
-4.96
-6.95
TA SR
58.8
33 / -0.97
20 -7.95
67 -3.10
60
/ /-1007
67 -0.84
100
-0.51
100
+0.60
TIES + SB
58.8
33 / -0.97
66
/ -0.00
100
4.0.8
100
4-0.99
100
+.0.6
100
-0.66
100
40.73
QWEX-8B
TSV-M
58.8
33 -0.97
13 -2.03
13 -2.33
-3.21
-3.26
-3.32
-3.28
Iso C
58.8
33/ -0.97
20 -..30
27 -0.35
0 -1.10
0 -1.64
0
-2.23
0
-3.16
TA SB
55.3
25 / -1.11
23 -4.52
58 -2.55
63 -0.37
86 10.43
98
0.889
100
1.07
TIES + SB
55.3
25
/ -1.11
+0.06
-0.22
+0.31
+0.32
+0.35
+0.36
Average
TSV-M
55.3
25
/ -1.11
22 -1.1
22 -1.75
8 -2.04
17 -2.00
7
-2.14
0
-2.36
Iso-C
55.3
25
-1.11
33
-0.47
30
-1.23
12
-1.97
10
-2.70
0
-3.73
0 -5.36
Table 3: Constructive interference results for Subspace- based merging methods across models. Each entry contains two quantities: the percentage of merge combinations that exceed the base model's accuracy, and the mean relative accuracy improvement for those combinations. Column headers the notation m (k)), where is the number of models merged and & is the number of evaluated merge combinations for that value of n. Base indicates base model accuracy.
and Iso-C deteriorate monotonically in both probability and relative improvement as the number of experts grows. TSV- Merge decreases from 22% success and -1.41 at n=2 to 0% and -2.36 at n=12, while Iso-C moves from 33% and -0.47 at n=2 0% and -5.36 at n=12. On average, subspace projection- basod methods suppress rather than exploit beneficial diversity, whereas Task Arithmetic with Subspace Boosting remains the only configuration whose performance scales constructively with increasing model diversity.
12

---

Published in Transactions on Machine Learning Research (03/2026)
5 Discussion and Limitations
Why Merging Methods Underperform in "h-thhe-Wid" Scenarios. Subspace-based merging meth- ods rely on strong assumptions about the geometry of fine-tuned checkpoints, which typically hold when models specialize
distinct, well-defined tasks. In such scttings, coherent update dircctions enable opera- tions like SVD truncation, orthogonalization, or isotropization to align or reshape task subspaces construc- tively. In our setup, however, we merge randomly sampled checkpoints, which also is practically relevant and directly evaluates the promise of merging methods of reusing the vast repository of publicly available model variants (Ramé et al.) 2023a). Their update directions need not form stable subspaces and may conflict substantially with each other. Consequently, subspace transformations can distort the combined update and push the merged model outside the lincarly mode-connccted region around the basc LLM, incrcasing the risk of performance degradation.
In contrast, Task Arithmetic makes subspace assumptions and effectively averages task vectors. When chockpoints are diverse, this averaging remains close to the basc model, yiclding modest but consistently positive gains. This explains why Task Arithmetic is more successful under random sampling, whereas subspace-based methods, though effective in their intended regimes, often underperform in our setting.
Homogencous vs. Heterogencous Merging. To further isolate the source of interference, we ana- lyzed homogeneous merges in Appendix B where experts were drawn from distinct, non-conflicting domains (Mathematics and Medicine). Unlike the random "in-tho-wild" sampling, we found that merging experts from these clearly defined domains, even when combining both domains simultaneously, resulted in negligi- ble performance loss compared to domain-specific merges. For instance, a joint model merging both Math and Medical experts performed nearly identically to specialized merges on their respective tasks. This con- trast confirms that the performance degradation observed in our main evaluation stems specifically from the unstruct ured heterogeneity of randomly sampled checkpoints. When experts possess conflicting update directions or high variance without clear task separation, advanced merging mechanisms struggle to extract useful signals, whereas they succeed when task roles distinct.
Limitations and Future Directions. While our evaluation is extensive, it is not exhaustive. First, we intentionally focused on LLMs and did not evaluate sucoder-decoder or multimodal models, where subspace geometry and finc-tuning dynamics may differ. Second, our experimental design omits pro-merging alignment or clustering steps to isolate intrinsic effects of merging methods. Future work should investigate whether pre-merging strategies like spectral filtering of task vectors or clustering improve performance.
6 Conclusion
We present a large-scale study of "in-the-wild" model merging for LLMs. Across four model families, twelve fine-tuned checkpoints per base model, and sixteen benchmarks, we find that only Task Arithmetic reliably produces constructive interference, i.e., improving upon both the base model and all individual checkpoints. In contrast, interference-awart aware and subspace- approaches (TIES- Model Stock, TSV-Merge, Iso-C, Subspace Boosting) fail to provide gains and degrade performance when not properly tuned.
These findings suggest that it is difficult, but not impossible, to improve the base model by simply merging a heterogencous set of finc-tuned versions and outperforming every checkpoint involved. Additionally, we find that Task Arithmetic yields better results on this task, while more sophisticated methods, such as TIES subspace-based based methods, do not successfully extract knowledge from heterogeneous checkpoints to improve base model performance.
A priority for future work is designing merging algorithms tailored to LLMs and validating them directly in this setting rather than relying solely on image-llassfficttion benchmarks, Our implementation, which combines mergekit with im-enal-harness, provides a standardized framework for such evaluations. Finally, merging-awar fine-tuning, which explicitly encourages complementary specializations, may further amplify the benefits of model merging, as our results with arbitrary checkpoints already suggest its potential.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Broader Impact Statement
This work investigates the reliability and limitations of model merging techniques for large language models. By clarifying when constructive interference occurs, our findings can help practitioners combine fine-tuned models more efficiently, potentially reducing computational cost and energy consumption associated with retraining. The study may also support open research by enabling reuse of publicly available fine-tuned checkpoints.
At the same time, model merging raises ethical and practical concerns. Automatically combining models without understanding their data provenance or domain biases can amplify undesirable behaviors, privacy risks, or misinformation learned from individual experts. Our results highlight that merging not universally reliable and should applied cautiously, with careful monitoring of model behavior and documentation of mergod chockpoints. Overall, we believe that greater transparency and empirical rigor in evaluating merging methods contributes positively to responsible large-model development.
Acknowledgements
This work was partially funded by the ERC (853489- DEXIM) and the Alfried Krupp Bohlen und Hal- bach Foundation, for which we thank them for their generous support. The authors gratefully acknowledge the scientific support and resources of the Al service infrastructure LRZ Al Systems provided by the Leibniz Supercomputing Centre (LRZ) of the Bavarian Academy of Sciences and Humanitics (BAdW), funded by Bayerisches Staatsministerium für Wissenschaft und Kunst. (SIMWK).
References
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetrics. In ICLR, 2023.
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. In Nature Machine Intelligence, 2025.
Anton Alexandrov, Veselin Raychev, Mark Mueller, Ce Zhang, Martin Vechev, and Kristina Toutanova. Mitigating catastrophic forgetting in language transfer via model merging. In ACL (Findings), 2024.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In JCML, 2018.
Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model
selection and boosting performance in domain generalization. In NeurIPS, 2022.
Parul Awasthy, Aashka Trivedi, Yulong Li, Meet Doshi, Riyaz Bhat, Vishwajeet Kumar, Yushu Yang, Bhavani Tyer, Abraham Daniels, Rudra Murthy, et al. Granite embedding r2 models. In arXie, 2025.
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling. In ICML, 2021.
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alhamn Fikri Aji, Pawan Sasanka Amnanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models. In aiv, 2024.
Rajas Chitale, Ankit Vaidya, Aditya Kane, and Archana Santosh Ghotkar. Task arithmetic with loRA for continual learning. In R0--FoMo-Robustnes of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
Team Cohore, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeod Alnumay, Sophia Althammer, Arkady Arkhangorodskyy Viraat Aryabumi, Dennis Aumiller, al. Command a: An enterprise-ready large language model. In arXis, 2025.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Tianshuo Cong, Delong Ran, Zesen Liu, Xinlei He, Jinyuan Liu, Yichen Gong, Qi Li, Anyu Wang, and Xiaoyun Wang. Have you merged my model? the robustness of large language model ip protection methods against model merging. In ACM Workshop on Large Al Systems and Models with Privacy and Safety Analysis, 2023.
Francesco Croce, Sylvestre-Alve Alvise Rebuffi, Evan Shelhamer, and Sven Gowal. Seasoning model soups for robustness to adversarial and natural distribution shifts. In CVPR, 2023.
Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Tryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by uncertainty-based gradient matching. In ICLR, 2024.
MohammadReza Davari and Eugene Belilovsky, Model breadcrumbs: Scaling multi-tass model merging with sparse masks. In ECCV, 2024.
Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Poria. Della-merging: Reducing interference in model merging through magnitude-basee based sampling. In arXiv, 2024.
Wenlong Deng, Yize Zhao, Vala Vakilian, Minghui Chen, Xiaoxiao Li, and Christos Thrampoulidis. DARE the extreme: Revisiting delta-parameter pruning for fine-tuned models. In ICLR, 2025.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In ICML, 2018
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. In arxio, 2024.
Rahim Entezari, Hanic Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In ICLR, 2022.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML, 2020.
Pushing the limits of Victor Gallego. Merging improves self-critique against jailbreak attacks. In ICML Workshop on Foundation Models in the Wild, 2024.
Tingchen Fu, Deng Cai, Lemao Liu, Shuming Shi, and Rui Yan. instruction tuning via alignment tax reduction. In ACL (Findings), 2024.
Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolå. Task singular vectors: Reducing task interference in model merging. In CVPR, 2025.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In NeurIPS, 2018.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's MergeKit: A toolkit for merging large language models. In EMNLP, 2024.
Hao Guo, Jiyong Jin, and Bin Liu. Stochastic weight averaging revisited. In Applied Sciences, 2023.
Yifei He, Siqi Zeng, Yuzheng Hu, Rui Yang, Tong Zhang, and Han Zhao. Mergebench: A benchmark for merging domain-specialized lmss. In NeurIPS, 2025.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic loRA composition. In COLM, 2024a.
Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr- emr-merging: Tuning-free high-performance model merging. In NewrIPS, 2024b.
Gabriel Ilharco, Marco Tulio Ribciro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In UAI, 2018.
Samyak Jain, Sravanti Addepalli, Pawan Kumar Sahu, Priyam Dey, and R. Venkatesh Babu. Dart: Diversify- aggregate-repeat training improves generalization of neural networks. In CVPR, 2023.
Dong-Hwan Hwan Jang, Sangdoo Yun, and Dongyoon Han. Model stock: All we need is just few fine-tuned models. In ECCV, 2024.
Xisen Jin, Xiang Ren, Danicl Prootiuc-Pictro, and Pengxiang Cheng, Dataless knowledge fusion by merging weights of language models. In ICLR, 2023.
Alexia Jolicoeur-Martineau, Emy Gervais, Kilian FATRAS, Yan Zhang, and Simon Lacoste-Julien. Popula- tion parameter averaging (PAPA). In TMLR, 2024.
Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob Hansen, James R. Glass, David Daniel Cox, Rameswar Panda, Rogerio Feris, and Alan Ritter. Self-moc: Towards compositional large language models with self-specialized experts. In ICLR, 2025.
Fanshuang Kong, Richong Zhang, and Ziqiao Wang. Activated parameter locating via causal intervention for model merging. In Xiv, 2024.
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. In NeurIPS, 2019.
Lu Li, Tianyu Zhang, Zhiqi Bu, Suyuchen Wang, Huan He, Jie Fu, Yonghui Wu, Jiang Bian, Yong Chen, and Yoshua Bengio. MAP: Low-compute model merging with amortized parcto fronts via quadratic approximation. In ICLR, 2025a.
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystiffy efficient SMoe with hints from its routing policy. In ICLR, 2024a.
Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey, In IEEE Transactions on Neural Networks and Learning Systems, 2025b.
Wenyi Li, Huan-ang Gao, Mingju Gao, Beiwen Tian, Rong Zhi, and Hao Zhao. Training- free model merging for multi-target domain adaptation. In ECCV, 2024b.
Yunshui Li, Yiyuan Ma, Shen Yan, Chaoyi Zhang, Jing Liu, Jianqiao Lu, Ziwen Xu, Mengzhao Chen, Minrui Wang, Shiyi Zhan, et al. Model merging in pre-training of large language models. In NewrIPS, 2025c.
Tian Yu Liu, Aditya Golatkar, and Stefano Soatto. Tangent transformers for composition.priaccy and
removal. In ICLR, 2024.
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic intogration of modular exportise in model merging. In NeurIPS, 2024.
Daniel Marczak, Bartlomicj Twardowski, Tomasz Trzciński, and Sebastian Cygert. Magmax: Leveraging model merging for seamless continual learning. In ECCV, 2024.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartlomiej Twardowski, Andrew D. Bagdanov, and Joost van de Weijer. No task left behind: Isotropic model merging with common and task-specific sub- spaces. In JCML, 2025.
Michael Matena and Colin Raffel. Merging models with fishtt- averaging. In NeurIPS, 2022.
Mohammed Mugeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing. TMLR, 2024.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing pre-trained models. In NeurIPS, 2023.
Fidel A Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differentiation. In CVPR, 2023.
Angelo Porrello, Lorenzo Bonicelli, Pietro Buzzega, Monica Millunzi, Simone Calderara, and Rita Cucchiara. A socond-order perspoctive on model compositionality and incremental learning. In ICLR, 2025.
Mohammad Areeb Qazi, Ibrahim Almakky, Anees Ur Rehman Hashmi, Santosh Sanjeev, and Mohammad Yaqub. Dynammo: Dynamic model merging for efficient class incremental learning for medical images. In MIUA, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. In JMLR, 2020.
Alexandre Ramé, Matthieu Kirchmeyer, Thiband Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. In NeurIPS, 2022.
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouilee: Recycling diverse models for out-of-distribution generalization. In JCML, 2023a.
Alexandre Ramé, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In NewrIPS, 2023b.
Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre- Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. Warp: On the benefits of weight averaged rewarded policies. In Xiv, 2024a.
Alexandre Ramé, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Gcoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the benefits of weight averaged reward models. In ICML, 2024b.
Filippo Rinaldi, Giacomo Capitani, Lorenzo Bonicelli, Donato Crisostomi, Federico Bolelli, Elisa Ficarra, Emanuele Rodolå, Simone Calderara, and Angelo Porrello. Update your transformer to the latest release: Re-basin of task vectors. In ICML, 2025.
Karsten Roth, Vishaal Udandarao, Sebastian Dziadzio, Ameya Prabhu, Mchdi Cherti, Oriol Vinyals, Olivier Hénall, Samuel Albanie, Matthias Bethge, and Zeynep Akala. A practitioner's 's guide to continual multi- modal pretraining. In NewrIPS Workshop on Scalable Continual Learning for Lifelong Foundation Models, 2024.
Wei Ruan, Tianze Yang, Yifan Zhou, Tianming Liu, and Jin Lu. From task-specific models to unified systems: A review of model merging approaches. In Xáv, 2025.
Ken Shoemake, Animating rotation with quaternion curves. In SIGGRAPH, 1985.
Ronald Skorobogat, Karsten Roth, and Mariana- Iuliana Georgescu. Subspace- boosted model merging. arXiv, 2025.
17

---

Published in Transactions on Machine Learning Research (03/2026)
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots. In TCLR, 2025.
Zafir Stojanovski, Karsten Roth, and Zeynep Akata. Momentum-based weight interpolation of strong zero- shot models for continual learning. In NewrIPS Workshop on Distribulion Shifls: Connecting Methods and Applications, 2022.
Derek Tam, Mohit Bansal, and Colin Raffel. Merging by matching models in task parameter subspaces. In TMLR, 2024.
Anke Tang, Li Shen, Yong Luo, Liang Ding, Han Hu, Bo Du, and Dacheng Tao. Concrete subspace learning based interference elimination for multi-task model fusion. In arXiv, 2023.
Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. In ICML, 2024a.
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du Yixin Chen, and Dacheng Tao. Parameter- efficient multi-task model fusion with partial linearization. In ICLR, 2024b.
Norman Tatro, Pun-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing mode connectivity via neuron alignment. In NewrIPS, 2020.
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In ICML, 2024.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongsook Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Xingrun Xing. Lm-cocktail: Resilient tuning of language models via model merging. In ACL (Findings). 2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In NeurIPS, 2023.
Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsend- suren Munkhdalai. What matters for model merging at scale? In TMLR, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. In arXiv, 2025.
Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. In ICML, 2024a.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerg- ing: Adaptive model merging for multi-task learning. In ICLR, 2024b.
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in Ilms, mllms, and beyond: Methods, theories, applications, and opportunities. In ACM Com- puting Surveys, 2026.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024.
Frederic 7. Zhang, Paul Albert, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad. Knowledge composition using task vectors with learned anisotropic scaling, In NewrIPS, 2024.
Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. Melagpt: Merging large language models using model exclusive task arithmetic. In EMNLP, 2024.
Didi Zhu, Zhongyisum Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Chao Wu, and Kun Kuang. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. In ICML, 2024.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Supplementary Material
A Fine-tuned Checkpoints
For cach base model, we used 12 publicly available fine-tuned checkpoints from the Hugging Face Hub, The complete list is provided below for reproducibility.
metholaann//lmmms222-----nttuu
1. Morggkkent//llammm2222mmmmoooooo instrrucioon
MerggeBenchhLLlama-322222 Instruct Bact-BB-innsuuttmmmmiiiit mmultiiiguallllll
3. MergeBBench/Llannne Saaareecdo//lamm--222----sstt math
4. BarggBBand//Llmmm32--BBmmmttttt
MerggelenchhlLaammm322333------- safety
6. bogpaeef/lamm33322222itttttmhmmm
8.
9. Oopiiimmm/Aammmm2222/itttttttttn
10. Fue.ll//hucchhaallaaaa2223113113
11. hatta
12.
2.
Deep.Nountt0//lanne331.111111111
4.
curiositytccc//MAASSS52222
7. StniaankA//boooby--iiiiiiinillee 3--8 1-8B
8. UW-.Madisnn-ce-LaalllaaaaPPPPPP
9. hatr://Lauees.AAAA/P/Pppponunnln
10.
11. TheFInAl//inol-88
12.
Owen/Qwen3-A4
19

---

Published in Transactions on Machine Learning Research (03/2026)
1. makiaa//hena------ooooooonnn
2. Menlo/Jan-nano
3.
maboonee/QQvaa33BBBbblteeeee
Valiantlabss/Qve333-------P5p5
Inteliieat-Intennee/l--eSaaaaad
10.
Devd122//gwenn-ffinnnnoooddddd
11. (ghoo360//ightt1RR-441
12.
Sethivvllmodd/Drcconi--ppoooo Batch-B-Prevveee
Qwen/ Qwen3-8B
Trandpo/TrendyoLLLLSS8---111
2.
3.
Vallautlabbs/Qvwn33334---Pspp
Cocde-IA\/Gooxd-PPoeee---1
mnlaoonee/Qvee333S4aallieenn
goobil22/Grayyiiineqqvveis-
TheFinAl//in-oll-3
AXCXEPPT/Qyna--E22-8B-baaaa
10. bougg-rroupuuutt/yyyuaaaaaaall]
11. Normalesardd/Apoolo-118
12. (kchmoolss/OViikn----B----aae Instruction
B
Taskwise Accuracy of Models
In Figs. to 10 we provide detailed wisk performance breakdowns for all evaluated base models. Across all model families and sizes, we observe consistent behavioral patterns that align with the aggregated results reported in the main text. Specifically, Task Arithmetic and subspace-booster variant demonstrato robust scaling, maintaining improving accuracy on diverse benchmarks such arc arr_chaalengge and winogrande increases. In contrast, Iso-C and TSV-M suffer from performance degradation knowledge-intent intensive and reasoning tasks like medmcqa and mmlu, particularly the number of merged checkpoints grows. Model Stock, TIES-Merging, and its subspace- boosted variant rarely deviates significantly from the base model's performance profile. These task-level visualizations confirm that the superior average performance of Task Arithmetic is driven by consistent gains across a wide range of evaluation dimensions rather than outliers in specific tasks.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Llama-3.1-8B-Inttruc
challenge
boolq
0.6
0.9
Accuraty
0.5-
Accura
Agcuracy
0.8
Accuracy
0.6
0.5
0.5
0.4
0.3
0.4
12
0.3
12
headqa
hellaswag
leaderboard bbh
leaderboard opqa
0.55
Accurary
0.40
Accuracy
Accurscy
0.50
0.32
0.5
0.45
0.30
0.30
0.40
0.35
10
12
10
8
12
Number Models
Number Models
Num ber of Models
Number Models
leaderboard pro
openbookqa
0.5
0.7
0.35
Accuracy
Accuracy
Accuracy
0.6
Accuracy
0.5
0.4
0.20
0.4
0.3
0.20
2
6
10
12
10
0
4
3
12
2
10
12
Number Models piga
Number of Models
Num ber of Models
Mumber of Models
prost
truthfulga, mc1
winogrande
0.50
0.40
Acturacy
0.75
Accuracy
Accuracy
0.35
0.70
0.40
0.70
0.65
0.30
0.65
0.30
0.25
0.55
2
6
10
12
1
10
0
4
8
10
12
6
10
12
Number Models
Number of Models
Number of Models
Number of Models
Model Stock
Task Arlthmetic
TIES
Task Arithmetic
SB
TIES SB
TSV-M
Iso-C
Figure 7: Taskwise accuracy and standard deviation of LLAMA 3.1 SB.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Llama-3.2-3B-Instruks
challenge
bookq
0.475
0.82
0.75
0.80
0.450
0.80
0.70
Accuracy
Accuracy
Accuracy
Accuracy
0.425-
0.65
0.70
0.400
0.76
0.74
0.55
0.60
10
10
12
hellaswag
leaderboard bbh
leaderboard gpqa
0.575
0.46
0.550
Aecuracy
Acturacy
0.525
Accuracy
0.29
0.42
0.30
0.28
0.40
0.27
8
4
10
6
10
12
12
Number of Models
Number Models
Number Models
Number Models
pro
openbookqa
0.65
0.32
0.32
Accurscy
Accuracy
0.60
0.30
0.28
0.475
0.55
0.26
0.450
0.50
8
10
12
2
4
10
6
10
12
8
12
Number of Models
Number of Models
Number Models
Number of Models
piqa
prost
truthfulqa, mmc
winogrande
0.42
0.70
Accuracy
Accurary
0.40
Accuracy
0.32
0.68
0.38
0.30
0.36
0.64
0.28
0.62
2
8
10
12
4
10
12
10
12
2
8
10
12
Number of Models
Number of Models
Number Models
Number of Models
Model Stock
Task Arlthmetic
TIES
Task Arithmetic
SB
TIES
SB
TSV-M
Iso-C
Figure 8: Taskwise accuracy and standard deviation of LLAMA 3.2 3B.
22

---

Published in Transactions on Machine Learning Research (03/2026)
Qwen3-4B
challenge
boolq
0.55
0.90
0.8-
0.85
Accurtty
Accuracy
Arguracy
Accuracy
0.80
0.45
0.7
0.40
0.70
0.5
10
12
0.4
Models
hellaswag
leaderboard bbh
leaderboard gpqa
0.375
0.55
Accuraty
0.350
Accuracy
0.50
Accuracy
0.50
0.32
0.325
0.45
0.45
0.300
0.40
0.40
0.275
0.28
8
4
10
6
10
12
12
Number of Models
Number Models
Number Models
Number Models
pro
openbookqa
0.7
0.32
Accurecy
0.40
Accurccy
Accuracy
0.30
0.28
0.30
0.22
8
10
12
111
4
10
6
10
12
8
12
Number of Models
Number of Models prost
Number Models
Number of Models
piqa
truthfullq__m mc1
winagrande
0.700
0.775
0.45
0.675
Accuracy
0.750
Accuracy
0.40
Accurary
0.34
0.655
0.725
0.32
0.625
0.700
0.35
0.30
0.600
0.675
0.30
0.28
0.575
2
8
10
12
4
10
12
6
10
12
8
10
12
Number of Models
Number of Models
Number Models
Number of Models
Model Stock
Task Arlthmetic
TIES
Task Arithmetic
SB
TIES
SB
TSV-M
Iso-C
Figure 9; Taskwise accuracy and standard deviation QWEN3 4B.
23

---

Published in Transactions on Machine Learning Research (03/2026)
Qwen3-8B
challenge
boolq
0.94
Accurtty
0.50
Accurscy
0.8
Accuracy
0.7
0.6
0.45
0.40
0.7
0.35
0.6
10
10
12
Number Models hellaswag
leaderboard bbh
leaderboard gpqa
0.60
0.400
0.55
Accuary
0.375
Accuracy
Accuracy
0.350
0.5
0.325
0.45
0.4
0.40
2
8
4
10
6
10
12
12
Number of Models
Number Models
Number Models
Number Models openbookqa
pro
0.8
0.5
0.60
Accuracy
0.55
Accuracy
0.30
0.50
0.6
0.28
0.3
0.26
0.24
0.2
0.35
0.4
2
8
10
12
4
10
6
10
12
2
8
12
Number of Models
Number of Models
Number Models
Number of Models
piqa
prost
truthfulqs_mm mc1
winogrande
0.800
0.375
0.775
0.350
Accuracy
Accurary
0.45
Accuracy
0.65
0.725
0.40
0.300
0.700
0.35
0.60
0.275
0.30
2
8
10
12
4
10
12
10
12
2
8
10
12
Number of Models
Number of Models
Number Models
Number of Models
Model Stock
Task Arlthmetic
TIES
Task Arithmetic
SB
TIES
SB
TSV-M
Iso-C
Figure 10: Taskwise accuracy and standard deviation QWEN3 SB.
24

---

Published in Transactions on Machine Learning Research (03/2026)
C Hyperparameter Ablations and Sensitivity Analysis
In this scction, analyze the scnsitivity of the merging methods to their key hyperparameters: the scaling coellicient X, the pruning density & (for TIES-Merging), and the spectral threshold / (for Subspace Boosting). We specifically investigate the structural reasons why Task Arithmetic and TIES-Merging exhibit divergent behaviors regarding the scaling coefficient À when applied to heterogencous, "in-the-wild" chockpoints. Un- less otherwise stated, all ablations are performed by merging all 12 tnn-ttned variants and sweeping the corresponding method-specific hyperparameters. We report average accuracy over the evaluation suite.
C.1 Task Arithmetic: Stability via Normalization and Cancellation
In Fig. analyze the sensitivity of Task Arithmetic to the scaling coefficient X. We initially observed flat performance curve across the standard range À E [0.1, 1.9]. To test whether this stability holds indefinitely or is merely a matter of scale, we exlended the sweep to X = 10. As shown in the rightmost portion of the plot, performance drops significantly at this extreme, confirming that the method is not invariant to scaling but rather highly robust within the typical hyperparameter range. We attribute this robustness to the normalization configuration employed by mergekit. By default, we consistently enabled normalization in experiments, which controls how individual task vectors are aggregated. Formally, let & be the task vector of model i and a; its scalar weight. With normalization enabled, the merged update A is computed as weighted average:
a Σ.a
(12)
Rorm
As shown in Appendix D the task vectors of the randomly selected checkpoints used in this study are largely orthogonal. Consequently, when averaged via normalization, the incoherent directions cancel out, resulting in a merged task vector with a very small magnitude. Because the base update vector is close to zero, scaling it by E 10.1, results negligible movement in parameter space, keeping the model within the low-loss basin of the base checkpoint. Only when À pushed to extreme values (e.g., 10) does the update magnitude become large enough to show significant performance change.
To validate that this stability is indeed an artifact of veoragig-induuedd cancellation, we perform an ablation where normalization is disabled. In this setting, the merge becomes a weighted sum (A = Σ, a;o;). Without the normalizing divisor, the heterogeneous updates accumulate magnitude rather than averaging out, yielding a task vector that is more likely to disrupt model performance. The results in Fig. I strongly support this analysis: in the absence of normalization, the model exhibits scnsitivity to X, with accuracy degrading rapidly as the scaling factor increases, even within the standard range.
C.2 TIES VEES-Merging: Sparsity Prevents Cancellation
For TIES-Merging, we perform a grid search over the pruning density k (top-k%) and the scaling factor X. Figs. 13 to 16 illustrate the relationship between the Lz-norm of the merged task vector and average accuracy.
We observe two distinct trends that differentiate TIES from Task Arithmetic:
1. Unlike the flat curve of Task Arithmetic, TIES exhibits monotonic degradation in accuracy À increases from 0.1 to 1.9, regardless of the densily setting.
2. Increasing X in TIES causes a rapid increase in the Lz-norm of the task vector, which correlates with performance degradation.
One might expect normalization to stabilize TIES just as did for Task Arithmetic. However, our results show that while Task Arithmetic maintains stable performance across X [0.1, 1.9), TIES exhibits an increase
in both the norm and the performance degradation as X grows (see Figs. 13 to 16 We hypothesize that
25

---

Published in Transactions on Machine Learning Research (03/2026)
0.50
0.5
030
Aweroge Accuracy
0.45
Ulams-3.1 48--nstrcct
Qwen1-48
Qwen3 88
0.25
13
17
Figure 11: Task Arithmetic: Effect of mixing coefficient X. We sweep the interpolation weight À used to combine task updates in Task Arithmetic.
0.50
Ubrra 3:2 38 Instruct
Usma 3.1 8B Instruct Owen3-48
0.5
Averase Accuracy (No Nerralization
0.50
0.45
0.40
0.35
0.30
0.25
4
9
Figure 12: Task Arithmetic (Without Normalization): Effect of mixing coefficient X. We sweep the interpolation weight À used to combine task updates Task Arithmetic. Task Arithmetic more sensitive to À scaling factor without normalization.
26

---

Published in Transactions on Machine Learning Research (03/2026)
this sensitivity arises because the design of TIES structurally limits the cancellation effects that otherwise stabilize heterogeneous merges. Specifically:
Trimming: By retaining only the top-k% magnitude parameters, TIES filters out the low- magnitude parameters that would typically facilitate averaging towards zero in standard mean operation.
Sign Consensus: By masking paramcters that disagrec on sign, TIES enforces directionality among the remaining weights.
In the context of random, heterogeneous checkpoints, we hypothesize that these mechanisms isolate high- magnitude parameters that, due to the lack task alignment, do not encode a coherent shared skill but rather high-variance weights. Because these weights are forced into alignment by the consensus step, they accumulate rather than cancel out during the merge. This hypothesis is supported by the observed expansion of the task vector's L2-norm in Figs. 13 16 which pushes the model far from the pre-trained weights even at moderate À values, resulting in the significant accuracy drops seen in sweep.
Based on this analysis, we select X = 1.0 for Task Arithmetic and X = 0.1 for TIES in our main experiments.
2-0.9 2-10
0.45
2-15
Averoge Accuracy
0.40
Tap-2'
0.30
20
90
50
242
Norm Task Vector
Figure 13:
Average accuracy versus L2-norm of the merged task vector for TIES on LLAMA 3.2 3B, under varying À and to density settings.
C.3 TIES top-k% Density
Fig. illustrates the impact of the pruning density k in TIES-Merging. The results reveal distinct "U- shaped" trajectory: accuracy is maximized when the density is either very low very high, while degrading significantly in the intermediate range. Although performance recovers as k approaches 100%, we did not select this setting because at full density, the pruning mechanism is effectively disabled, making the method behaviorally nearly identical to standard Task Arithmetic. Therefore, to faithfully evaluate the sparsification properties that distinguish TIES- MES-MMegingg from simple averaging, we selected the top-10% density for our main evaluation.
27

---

Published in Transactions on Machine Learning Research (03/2026)
Tesk: Arthmmetic Auinaage Acourscy
1:3.3
0.93
A.0
Average Accuracy
0.43
Top Dera
1-1.3
1-18
TLak Rom
1-1.7
0.3
130
Figure 14: Average accuracy versus Iz-norm the merged task vector for TIES on LLAMA 3.1 8B, under varying À and top-k% density settings.
Arthmeli
100
Average Accuracy
1-1.7
260
360
Figure 15: Average accuracy versus Lg-norm the merged task vector for TIES on QWEN3 4B, under varying and top-k% densily settings.
28

---

Published in Transactions on Machine Learning Research (03/2026)
1-03
Nunragg Acouncy
2-05
0.5
2-1.
0.53
Average Accurary
0.45
0.49
0.35
Top 45 Demity
50
102
252
202
Norm Task Vector
Figure 16: Average accuracy versus Lo-norm of the merged task vector for TIES QWEN3 8B, under varying \ and top-k% density scttings.
0.50
Owen3 48
Qwen1--0
0.525
Nverage Accuracy
0.475
0.225
0.40
20
30
40
65
70
90
100
Top-X% Dersity
Figure 17: TIES-Merging: effect of top-A% density. We sweep the top-k% density, defined as retaining the top-k% largest-magnitudd weights in TTES-Merging. Higher density (larger k) keeps more parameters active (less sparsily), whereas lower density (smaller k) enforces stronger sparsity.
29

---

Published in Transactions on Machine Learning Research (03/2026)
C.4 Subspace Boosting Threshold
Fig. 18 depicts the performance of Subspace Boosting function of the spectral threshold B. We observe rapid performance gain as / increases from 0 to 0.05, after which the accuracy stabilizes and remains robust across a wide range of values (3 E [0.1, 0.5)). This indicates that Subspace Boosting is not highly sensitive to the exact threshold, provided large enough. We therefore chose B = 0.2 for our experiments to ensure robust spectral flattening.
Average Accuracy
0.54
0.52
0.50
Usma: 3.2 Instruct
0.6
Uama: B--B-untttutt
Owen
5.0
0.1
0.2
03
6.4
Boosting Threshold (8)
Figure 18: Subspace Boosting: effect of the boosting threshold B. We sweep the raw-proportion threshold 3 E [0, in Subspace Boosting. For each SVD, singular values whose normalized cumulative sum is < 3 are left unchanged; subsequent singular values are boosted by clamping them to the cutoff singular value. Accuracy vs. highlights how strengthening lower-energy directions mitigates interference and impacts overall performance.
D Pairwise Cosine Similarities of Task Vectors from Tuned Variants
Figs. to 22 present the pairwise cosine similarities between task vectors obtained from the fino-tuned checkpoints used in this study. Across all model families, the cosine similarity values are concentrated near zero, indicating that the corresponding task vectors are largely orthogonal in parameter space. Importantly, each model family exhibits both positive and negative cosine similarity values, suggesting that while some fine-tuned variants induce mildly aligned task directions, others produce updates in opposing directions.
To complement these aggregate measurements, we additionally report layerwise sign heatmaps in Figs. 23 to 26 For each pair of fine-tuned models, cosine similarity is computed independently at each layer, and we record the number of layers exhibiting positive versus negative cosine similarity. These matrices therefore summarize the distribution of alignment signs across layers, rather than a single scalar similarity value. The layerwise analysis reveals that even when the global cosine similarity between two task vectors is close to zero, individual layers may induce aligned updates, opposing updates, or remain effectively neutral. This sign heterogeneity provides direct evidence for partial cancellation effects in Task Arithmetic, arising from mixtures of positively and negatively aligned layers rather than uniformly orthogonal updates.
Finally, across all base model families, we observe a non-negligible number of layers whose cosine similarity is equal to (or numerically indistinguishable from) zero. As result, for a given model pair, the sum of positive and negative layer counts does not necessarily equal the total number of layers.
30

---

Published in Transactions on Machine Learning Research (03/2026)
1.000
0.007
0.004
0.029
0.101
0.005
0.013
0.000
0.000
0.054
0.000
0.000
1.00
(2)
0.007
1.000
0.006
0.004
0.000
0.002
0.006
-0.000
0.001
0.003
(3)
0.004
0.001
1.000
0.026
0.010
0.002
0.007
0.000
0.000
0.025
0.000
0.000
0.75
(4)
0.029
0.006
0.026
1.000
0.051
0.004
0.013
0.000
0.000
0.025
0.000
-0.000
0.50
(5)
0.25
(6)
0.005
0.000
0.002
0.004
0.005
1.000
0.002
0.000
0.000
0.005
-0.003
-0.000
0.00
(7)
1.000
-0.25
0.000
0.006
0.000
0.000
0.000
0.000
0.000
1.000
-0.000
0.000
0.016
0.000
(9)
0.000
-0.000
0.000
0.000
0.000
0.000
0.001
.0.000
1.000
-0,000
0.000
-0.000
-0.50
(10)
0.054
0.001
0.025
0.054
0.005
-0.000
1.000
0.000
0.000
-0.75
(11)*
0.000
0.003
0.000
0.000
0.000
-0.003
-0.000
0.016
0.000
0.000
1.000
0.000
-1.00
(12)
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
Figure 19:
Pairwise cosine similarities ween task vectors fine-tuned LLAMA 3.2 3B variants. Indices 1 12 correspond to the checkpoint ordering listed in Appendix A
31

---

Published in Transactions on Machine Learning Research (03/2026)
1.000
0.000
0.000
0.000
0.001
0,000
0.001
0.000
0.001
0.001
0.002
0.001
(2)
0.000
1.000
0:254
0.010
0.007
0.000
0.004
0.001
0.063
0.005
0.019
(3)
0.000
0.254
1.000
-0.002
0.000
0.001
0.000
0.001
0.000
0.439
0.001
-0.000
0.75
(4)
0.000
0.010
-0.002
1.000
-0.000
-0.000
0.005
0.001
-0.000
0.001
0.032
0.522
0.50
(5)
0.25
(6)
0.000
0.000
0.001
-0.000
-0.000
1.000
0.001
0.000
0.000
0.000
0.001
0.000
0.00
(7)
1.000
-0.25
0.000
0.001
0.001
0.001
0.001
0.000
0.001
1.000
0.001
0.000
0.005
0.001
(9)
0.001
0.006
0.000
.0.000
0.020
0.000
0.007
0.001
1.000
0.001
0.012
0.014
-0.50
(10)
0.001
0.063
0.001
0.001
0.002
0.001
1.000
0.002
0.006
-0.75
(11)*
0.002
0.005
0.001
0.032
0.013
0.001
0.031
0.005
0.012
0.002
1.000
0.014
-1.00
(12)
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
Figure 20: Pairwise cosine similarities ween task vectors fine-tuned LLAMA 3.1 8B variants. Indices 1 12 correspond to the checkpoint ordering listed in Appendix A
32

---

Published in Transactions on Machine Learning Research (03/2026)
1.000
0.001
0.000
-0.000
-0,000
0,000
0.001
0.003
-0.000
0.001
0.0D1
0.000
1.00
(2)
0.001
1.000
0.000
0.000
0.000
0.000
0.000
0.002
-0.000
0.000
0.000
(3)
0.000
0.000
1.000
-0.001
-0.000
0.002
0.001
0.002
0.000
0.001
0.002
0.000
0.75
(4)
-0.000
0.000
-0.001
1.000
-0.000
-0.000
-0.000
-0.003
-0.000
-0.000
-0.000
-0.000
0.50
(5)
0.25
(6)
0.000
0.000
0.002
.0.000
0.000
1.000
0.005
0.001
0.000
0.000
0.001
-0.000
0.00
(7)
1.000
-0.25
0.003
0.002
0.002
-0.003
-0.000
0.001
0.004
1.000
0.001
0.005
0.012
0.001
(9)
.0.000
.0.000
0.000
.0.000
0.000
0.000
.0.000
0.001
1.000
-0,000
0.001
0.020
-0.50
(10)
0.001
0.000
-0.000
0.000
0.000
0.005
-0.000
1.000
0.002
0.000
-0.75
(11)*
0.001
0.000
0.002
-0.000
0.000
0.001
0.003
0.012
0.001
0.002
1.000
0.000
-1.00
(12)
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
Figure 21:
Pairwise cosine similarities between lask vectors fine-tuned QWEN3 4B variants. Indices 1-12 correspond the checkpoint ordering listed in Appendix A
33

---

Published in Transactions on Machine Learning Research (03/2026)
1.000
0.000
0.002
0.003
0.005
0,000
0.000
0.005
0.003
0.005
0.002
0.000
(2)
0.000
1.000
00.000
0.000
-0.000
0.033
0.001
0.001
0.000
0.000
-0.000
(3)
0.002
-0.000
1.000
0.001
0.003
0.000
0.000
0.001
0.001
0.001
0.000
0.000
0.75
(4)
0.003
0.000
0.001
1.000
0.008
0.001
0.006
0.010
0.004
0.009
0.002
0.001
0.50
(5)
0.25
(6)
0.000
0.033
0.000
0.001
-0.002
1.000
0.009
0.008
0.000
0.002
-0.000
-0.001
0.00
(7)
1.000
-0.25
0.005
0.001
0.001
0.010
0.010
0.008
0.008
1.000
0.045
0.052
0.010
0.005
(9)
0.003
0.000
0.001
0.004
0.002
0.000
0.001
0.045
1.000
0.021
0.025
0.006
-0.50
(10)
0.005
0.000
0.009
0.010
0.002
0.052
1.000
0.011
0.005
-0.75
(11)*
0.002
-0.000
0.000
0.002
0.002
-0.000
0.001
0.010
0.025
0.011
1.000
0.008
-1.00
(12)
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
Figure 22: Pairwise cosine similarities between lask vectors fine-tuned QWEN3 8B variants. Indices 1-12 correspond the checkpoint ordering listed in Appendix A
34

---

Published in Transactions on Machine Learning Research (03/2026)
Positive Cosine Simliarity Layer Counts
Negative Cosine Simllarity Layer Counts
12
:23
2:4
127
:06
220
203
21
242
230
232
1:3
2:4
103
32
148
277
13
2
23
11
43
23
13
200
23
243
1.5
223
216
232
31
22
23
130
233
16
106
112
1.5
27
151-
214
24B
204
200
204
101
161
21%
14
10
171-
16
14
22
15
110
104
150
195
82
291-
1200
313
215
153
167
132
1201-
123
123
277
130
1:1
100
105
162
130
55
12:1
53
14
64
75
50
194
117
33
24
16
14
164
11
(4)
010
011
1120
111 24 00
16 14
091
06
1100
1101
Figure 23: Layerwise sign heatmap of pairwise cosine similarities between task vectors of fine-tuned LLAMA 3.2 3B variants, showing the number of layers with positive and negative alignment. Indices 12 correspond to the checkpoint ordering in Appendix A
Positive Cosine Similarity Layer Counts
Negative Cosine Similarity Layer Counts
12
11c
115
305
140
303
105
120
123
112
100
11
307
162
112
392
120
21
42
27
24
59
51
31
124
143
34
112
165
106
246
152
162
37
26
30
124
23c
146
164
232
204
65
150
40-
34
391
35
1.0
112
230
220
54
363
333
1.60
303
200
18
53
324
10
1:66
128
205
100
1:56
20
180
1209
123
247
:09
105
1201
010
228
.02
152
230
220
40
233
203
201
110
204
1111
26
23
23
26
235
200
252
136
234
19
22
18
21
00
14
14
16
171
(8)
(6)
(1.)
01)
(12)
121
211
60
14
141
661
(7
(16)
Figure 24: Layerwise sign heatmap of pairwise cosine similarities between task vectors of fine-tuned LLAMA 3.1 8B variants, showing the number of layers with positive and negative alignment. Indices 1-12 correspond to the checkpoint ordering in Appendix A
35

---

Published in Transactions on Machine Learning Research (03/2026)
Positive Cosine Simliarity Layer Counts
Negative Cosine Simllarity Layer Counts
12
323
20:
252
121
232
104
276
143
136
42
76
147
234
235
175
228
172
19
216
245
39
86
183
93
124
197
151
210
144
274
160
123
115
545
4x
130
16%
144
114
115
73
143
181
37
47
17
307
167
141
505
104
174
314
42
171-
57
163
92
182
70
164
2X7
372
230
903
204
62
100
391-
1200
174
233
175
194
173
1201
108
146
1230
232
204
172
102
48
14:
306
302
363
184
222
1211
126
124
87
261
24
70
37
224
230
176
222
55
319
237
24
16 184
164
11
691
14)
010
01
1120
111
01
00
16
141
09
06
1000
1101
Figure 25: Layerwise sign heatmap of pairwise cosine similarities between task vectors of fine-tuned QWEN3 4B variants, showing the number of layers with positive and negative alignment. Indices 1-12 correspond to the checkpoint ordering in Appendix
Positive Cosine Similarity Layer Counts
Negative Cosine Similarity Layer Counts
12
1:7
223
206
234
203
21
120
35
27
58
37
42
32
31
20
37
21
14
6s
135
136
143
56
68
123
86
138
18
76
23
56
61
264
36h
246
185
150
15
150
40-
143
103
121
174
102
391
25
147
2/17
72
14C
2012
200
324
:53
127
171-
15
41
103
125
241
200
333
145
24
1209
234
223
265
302
137
1208
010
125
37
78
1/2
318
102
152
173
222
220
145
1111
87
23
88
57
74
15
120
74
32
207
127
143
349
157
543
1:3
42
11
21
00
14
161
171
08)
(4)
(1.)
(1)
(12)
111
24
(4)
141
061
(7)
04
(16)
Figure 26: Layerwise sign heatmap of pairwise cosine similarities between task vectors of fine-tuned QWEN3 8B variants, showing the number of layers with positive and negative alignment. Indices 1-12 correspond to the checkpoint ordering in Appendix
36

---

Published in Transactions on Machine Learning Research (03/2026)
E
Experiments on Homogeneous Merges
In addition to the random subset sampling strategy presented in the main text, we investigate the behavior of merging methods when applied to domain-specific groups of experts. Specifically, we aim to determine whether merging models from distinct domains (Medical and Math) introduces destructive interference compared to merging experts from a single domain.
E.1
Experimental Setup
For both QWEN3-4B and QWEN3-8B, we selected three fine-tuned checkpoints specialized in medical tasks and three specialized in mathematics from the Hugging Face Hub. The specific models used are listed in Table 4
Table 4: List of domain-specific chockpoints used for homogencous merging experiments.
Base Model
Domain
Checkpoint Name
prithicMLmmodss/Deccnii---eevre Math-4B-Prewse
Math
QWEN3 4B
Medical
Math
mifoundatioon-dev/aal math formulas
QWEN3 8B
Medical
BlueZeros/EHRRRRi-55
We construct domain-specific evaluation suites, Medical and Math, each composed of relevant subsets drawn from mmlu, medmcqa, headqa, and bbh. The specific task subsets included in each suite are listed in Table 5
Table 5: Evaluation tasks used for homogencous morging experiments, grouped by domain.
Medical Tasks
Math Tasks
medncqa
mmlu_abstirn aebstract_algebare
headqa_en
mmlu
amlu_anatomy
mmlu www.eetarryeatteeetttttooo m_aahhemtiicce
Btlu_cclinical_kooolee.
milu_formal_ _logic
mmiu_neddica__enntics
leaderboard_ Baered_bbbbbooo_eeeeeee
moollann_expessiiee
mmlu.proffesiiona__meeeeee
leaderboard tat_forral_faaaacii
amlu_virology
leaderboard, bbb__gemeerrc_eheee _hapee
haa__coollee_nediicie
leaderboard_ cettrrbarrdbbbb____e wbb__bbectt___...oooeoeeeeeeeeee ceteccttttoiinnggeeee
leaderboard www.bbbbbeee__e_eee oaabb_of_iee__
To evaluate the impact of mixing domains, we performed three merge configurations for each base model and method: (i) Merge Medical Only (n = 3), which merges only the three medical experts and is evaluated on medical tasks; (ii) Merge Math Only (n = 3), which merges only the three math experts and is evaluated on math tasks; and (iii) Merge All (n 6), which merges all six experts (three medical and three math) and is evaluated both medical and math tasks.
37

---

Published in Transactions on Machine Learning Research (03/2026)
E.2 Results
The results for QWEN3 8B and QWEN3 4B are presented Fig. 27 and Fig. 28 For QWEN3 8B, we observe stability across all merging methods. Specifically, models merged from all experts perform nearly identically those merged from domain-specifi subsets. This indicates that, at this scale, combining disjoint domains like Math and Medicine introduces negligible interference, allowing the merged model to retain the specialized capabilities of both groups simultaneously.
A similar trend holds for QWEN3 4B. Standard approaches like Task Arithmetic and TIES effectively preserve task performance, yielding multi-domain models that match their single-domain counterparts. However, we observe more noticeable deviations in subspace-based variants. For instance, TA + SB and Iso-C exhibit larger performance gaps between single-domain and multi-domain merges, likely reflecting the increased sensitivity of subspace selection smaller parameter scales. Despite these exceptions, the overall pattern reveals that diverse experts generally be combined without significant negative transfer.
1.0
Only
Modical Aug Indistiual
Merge Tasks
0.8-
Math
Marge lasca
0.6
Average
0.4-
0.2
0.9
Task Arithmetic
TIES
Model Stock
TA SB
TIES SB
TSV
Iso.C
Figure 27: Performance comparison of homogeneous vs. heterogeneous merging on QWEN3-8B. We compare merging only sppeifc experts (Medical Only, Math Only) against merging all experts together.
1.0
Merge Only Merge Medical Tasks Menge Math Theies
Medical Medical Aug Indistaual Math Maln
0.5
0.6
0.4
0.2
Task Arithmetic
TIES
Model Stock
TA SB
TIES S8
TSV-M
Iso-C
Figure 28: Performance comparison of homogeneous vs. heterogeneons merging on QWEN3-4B, We compare
merging only domain-specific experts (Medical Only, Math Only) against merging all experts together.
38

---

Published in Transactions on Machine Learning Research (03/2026)
F Evaluation Details and Configuration for Im-eval-harness
All evaluations follow the default Im-eeal-harness inference and task configuration. Decoding is performed us- ing the harness default generation setup. Models are evaluated using float16 precision with batch size=auto. For each task, the number of in-context examples (n-fewshot) determined by the task definition shipped with the harness. We do not apply model-specific chat templates, and no task-specific prompt engineering or template modifications are applied. Table summarizes the n-fowshot values used for each benchmark.
Task
n-fewshot
arc_easy
arc_challenge
boolq
commonsense_qe
0
hellaswag
winogrande
0
piqa
openbookqa
headqa
prost
truthfulqa_ne mc1
medmcqa
0
leaderboard_gpga
leaderboard_bbl
3
mmlu
0
leaderboard mmml__rr
Table 6: Number in-context examples (n-fewshot) used for each evaluation task, following the default
configuration of Im-eval-harness.
39


Reviews:
[{"review_id": "dYO6KcMwc3", "reviewer": "Reviewer_AyLX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is an empirical study evaluating recent model merging methods on various LLMs.\n\nStrengths:\n1. The paper provides a much needed systematic evaluation of recent model merging methods on recent open-weights LLMs.\n\nWeaknesses:\n1. In my opinion, the evaluation protocol presented in the paper entangles two independent factors: (a) merging of LLMs, and (b) merging \"in the wild\". The authors claim to investigate (a) merging of LLMs, but (b) merging \"in the wild\" may obscure it. \n   Most merging methods implicitly assume that each checkpoint is useful and can positively contribute to the final merged model. It is always the case in the standard controlled evaluation protocols of multi-task image classification or natural language inference. However, when selecting some checkpoints from an open model hub, it is not clear if the model can positively contribute. Therefore, the superiority of one merging method over the other, can come from the fact that it secretly reduces the influence of \"harming checkpoints\".\n2. Main results are questionable.\n\t1. In Fig. 3 and Fig. 6, \"Baseline\" (which I assume is an evaluation of a base model) is significantly lower than merging 0 models. They should be exactly equal which indicates some inconsistency in the evaluation protocol.\n\t2. Figure 4: How is the L2-norm of Task Arithmetic decreasing with the number of merged models? For a constant $\\lambda=1$, it could happen only if the task vectors had significant opposite components that would manifest itself in negative cosine similarity between task vectors. For cos_sim >= 0, L2-norm grows between $\\sqrt n$ (cos_sim = 0) and $n$ (cos_sim = 1), where $n$ is the number of merged task vectors.  \n3. Hyperparameters selection:\n\t1. I am very sceptical of Figure 11. It contradicts previous results (eg TIES merging paper, Fig 8 left), my own experiments for merging LLMs and common sense: summing 12 task vectors and scaling the sum by 1.9 achieves the best results for llama models. This does not make much sense.\n\t2. What values of $\\lambda$ are used for other merging methods?\n4. Phrases like \"multi-talented model\" seem LLM-generated\n5. Model Stock is a method designed to merge models fine-tuned on the same dataset with the same hyperparameters. It is an odd baseline for multi-task merging \"in the wild\".", "claims_and_evidence_yn": "No", "claims_explanation": "In my opinion, the current evaluation setup entangles two orthogonal problems: merging of LLMs and merging \"in the wild\" (see Weakness 1). Therefore, the claims about some merging methods working or not working for LLMs are not necessarily true. Moreover, some of the presented results are not convincing (see Weakness 2 and 3).", "audience_yn": "Yes", "audience_explanation": "I believe that the results could be interesting for model merging community **once the evaluation issues are resolved**. In the current state, I find the results presented in the paper confusing.", "requested_changes": "Critical changes:\n1. Make evaluation protocol more clear (see Weakness 1). I can see 2 possible ways of achieving it. First (better) would be to train the models on tasks related to the evaluation benchmarks. Second would be to evaluate the individual models and filtered the ones that do not improve over the base model. Once we know which merging methods work for LLMs,  the current setup could serve as an extension -- it would tell which methods are robust to inclusion of harmful checkpoints. \n2. Address evaluation issues specified in Weakness 2\n3. Clarify hyperparameters selection (Weakness 3). \n\nImprovements:\n1. Among the most intriguing observations is the phenomenon of constructive interference, where a merged model outperforms its individual base models. -- this statement would benefit from citation\n2. Section 2.2 -- TIES-Merging is missing citation\n3. Fig 2.2 -- first step in TIES-Merging diagram is captioned as \"Task Vectors Layerwise\" while it should be \"Task Vectors Parameter-wise\"\n4. Table 2 -- it is unclear what \"best fine-tuned checkpoint\" is. Is it a single checkpoint that performed best on average across all the evaluated tasks or is it the best model for each evaluation task?", "broader_impact_concerns": "In my opinion, Broader Impact Statement provided by the paper is sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PJfZUdTJ8b", "reviewer": "Reviewer_62PW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission conducts a broad empirical study of model merging for modern LLMs (e.g., Llama 3 and Qwen families) using a set of publicly available fine-tuned checkpoints and a wide benchmark suite. The paper’s central finding is that simple Task Arithmetic (TA) yields the most consistent performance improvements as the number of merged checkpoints increases, whereas more elaborate methods such as TIES and subspace-based approaches often degrade performance in the evaluated setting.\n\n**Key strengths:** broad empirical coverage across model families/benchmarks; practical motivation aligned with \"in-the-wild\" checkpoint reuse; an attempt to provide a mechanistic interpretation via deviation-from-base diagnostics.\n\n**Key weaknesses:** (i) baseline scope mismatch for Model Stock under highly heterogeneous merging, (ii) conclusions risk overgeneralization from the specific \"random heterogeneous subset\" regime, and (iii) potentially asymmetric hyperparameter tuning across methods, especially for TIES (top-k and scaling interaction).", "claims_and_evidence_yn": "No", "claims_explanation": "The evidence provided does not sufficiently support the broad claim that modern merging methods \"fail on LLMs\" due to three critical flaws in the experimental design:\n1. **Heterogeneity vs. Method Failure**: The evidence convincingly supports a *narrower* claim: *under heterogeneous, randomly mixed checkpoint merging, TA appears more robust than TIES/subspace methods in the authors’ evaluation pipeline.* However, the submission at times reads as making a broader claim (e.g., that these methods \"fail for LLMs\" generally), while the experiments do not sufficiently disentangle LLM-specific failure from strong heterogeneity-driven failure.\n2. **Misapplication of Baselines (Model Stock)**: One baseline (Model Stock) is evaluated primarily in a regime that does not match its intended scope (merging models trained under the same task/setup with different random seeds). In such an out-of-scope regime, underperformance is difficult to interpret as evidence against the method, which weakens the clarity and fairness of the comparative conclusions.\n3. **Unfair Hyperparameter Tuning**: The analysis attributes poor performance (especially for TIES) to large deviation from the base model, yet the tuning protocol does not clearly ensure that TIES method is given comparable opportunity to control distance via scaling (λ), unlike TA. Without distance-constrained or distance-matched controls (and/or a fair tuning budget across methods), the causal explanation remains under-validated.", "audience_yn": "Yes", "audience_explanation": "The topic of model merging is highly relevant to the TMLR audience, given the increasing cost of training LLMs and the proliferation of fine-tuned checkpoints. A rigorous \"reality check\" on whether complex merging methods justify their complexity over simple averaging is valuable. If the experimental flaws are addressed, the finding that simple methods are robust would be of significant practical interest to engineers and researchers aiming to reuse public models efficiently.", "requested_changes": "To secure a recommendation for acceptance, the following critical adjustments are required:\n1. **Rectify the Model Stock Comparison (Critical)**: The evaluation of Model Stock on heterogeneous tasks must be fundamentally revised. Either remove Model Stock from the main comparative analysis or explicitly frame it as an out-of-distribution stress test, acknowledging that the setup violates the method's intended scope (same-task merging). The current framing implies a failure of the method itself, which is misleading.\n2. **Control for Heterogeneity (Critical)**: To support the claim that these methods fail on LLMs (rather than just on conflicting tasks), the authors must add experiments with homogeneous merging.\n    - Cluster checkpoints by domain (e.g., merge only Math models, or only Coding models) and evaluate TIES/Subspace methods in this setting.\n    - If TIES/Subspace methods work well in homogeneous settings but fail in random settings, the paper's conclusion must change to reflect that these methods lack robustness to task conflict, not that they are incompatible with LLMs.\n3. **Implement Fair Tuning Protocols (Critical)**: The comparison between TA and TIES/Subspace methods must be standardized regarding hyperparameter optimization.\n    - Perform a grid search for TIES over both density (k) and scaling factor (λ).\n    - Specifically, conduct a \"Distance-Constrained\" comparison: Tune the scaling factor of TIES/Subspace methods so that their final merged model has a similar L2 distance from the base model as the TA model. Compare performance at this iso-distance point to determine if the failure is due to the direction of the update or simply the magnitude.\n4. **Refine Claims**: Soften the broad claims about method failure to be specific to the experimental context (random heterogeneous mixing) until the above controls are implemented.", "broader_impact_concerns": "The paper includes a Broader Impact Statement discussing the efficiency benefits and potential risks of uncontrolled merging. There are no additional specific ethical concerns requiring a new statement, provided the technical claims are corrected.", "additional_comments": "**Reject**. \nThe paper is timely and potentially valuable, especially as a \"reality check\" for practitioners. At present, I believe the study’s empirical results are interesting but the broader interpretation is not yet fully supported due to (i) ambiguous baseline scope for Model Stock, (ii) insufficient control/measurement of checkpoint heterogeneity, and (iii) lack of distance-constrained and uniformly tuned comparisons. Addressing the critical items above would substantially strengthen the paper and make the conclusions more general and reliable.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8tdxF1EN1V", "reviewer": "Reviewer_24CT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a systematic evaluation of model merging for LLMs, pairing each of four base models with 12 publicly available fine-tuned checkpoints, sampling multiple subset sizes, and evaluating merged models on standard benchmarks via LM-Eval-Harness. It compares six merging methods, three weight space and three subspace, with implementations applied through the mergekit library. The main empirical finding is that Task Arithmetic is the only method that consistently achieves constructive interference under the paper’s random-subset protocol. In contrast, interference-aware and subspace-based methods often degrade as more checkpoints are merged. The paper also reports a correlation between larger parameter space displacement from the base model and worse average benchmark accuracy, using the L2 norm of the merged update as a diagnostic.\n\nKey strengths are the scale and consistency of the evaluation grid across model families and subset sizes. Key weaknesses include limited merging regimes and some reporting details needed for exact replication that are not fully specified in the main text.", "claims_and_evidence_yn": "Yes", "claims_explanation": "* The core claim that Task Arithmetic improves reliably with more merged checkpoints, while TIES Merging often degrades, is directly supported by the average accuracy trends and the constructive interference tables across four base models.\n\n * The claim that merges can surpass not only the base but also the strongest individual fine-tuned checkpoint for three of four model families is directly supported by the reported comparisons against “Best FT.”\n\n * The negative results for subspace methods under the same protocol are also directly supported by the accuracy trends and the success probability and mean change table for subspace-based methods.", "audience_yn": "Yes", "audience_explanation": "The topic aligns with active interest in post-training reuse of open weight checkpoints, since the paper studies when merging can replace or reduce additional training across multiple LLM families and sizes. This paper provides a large, standardized comparison that clarifies which prior merging ideas transfer poorly to LLM settings under a realistic “public checkpoint soup” style protocol, which is directly relevant to researchers who use mergekit-style tooling.", "requested_changes": "* The authors are suggested to add at least one additional merging regime beyond random subsets, such as similarity-based clustering of checkpoints or filtered subsets, and report whether the rankings of methods change under that regime.\n\n* The authors are suggested to specify the evaluation harness configuration in sufficient detail to reproduce results, including few-shot settings, prompts or templates if applicable, and any decoding details for tasks.\n\n* There is a typo in the caption of Table 1: “Arithmetic.”", "broader_impact_concerns": "This paper’s broader impact statement notes that merging can amplify undesirable behaviors, privacy risks, or misinformation when checkpoints have unknown provenance or biases, and it recommends caution and monitoring.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)