Navigating the Accuracy-Size Trade-Off with Flexible Model Merging
Authors: Akash Dhasade
Divyansh Jhunjhunwala
Milos Vujasinovic
Gauri Joshi
Anne-Marie Kermarrec
Venue: ICLR 2026 Poster (2026)
Keywords: Model Merging
Model Fusion
Task Arithmetic
Multi-task Learning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Model merging has emerged as an efficient method to combine multiple single-task fine-tuned models. The merged model can enjoy multi-task capabilities without expensive training. While promising, merging into a single model often suffers from an accuracy gap with respect to individual fine-tuned models. On the other hand, deploying all individual fine-tuned models incurs high storage costs. We propose FlexMerge, a novel data-free model merging framework that: (a) flexibly generates merged models of varying sizes, spanning the full spectrum from a single merged model to retaining all individual fine-tuned models; and (b) supports multiple merging algorithms in a unified framework. Using FlexMerge, we systematically characterize the accuracy–size trade-off of different  algorithms. Our study reveals two key findings: first, even modestly larger merged models can yield steep accuracy gains (up to 13.5% when just doubling the size); second, algorithm rankings are not consistent as size increases, with some methods overtaking others beyond the one-model regime. These results uncover a new design dimension for model merging: developing and comparing algorithms across the full spectrum of sizes rather than only at the single-model limit. Extensive experiments on vision and NLP benchmarks, with up to 30 tasks, confirm the generality and practicality of FlexMerge.

Published as a conference paper at ICLR 2026
NAVIGATING THE ACCURACY-SIZE TFADE-OFF WITH FLEXIBLE MODEL MERGING
Akash Dhasade's Divyansh Jhunjhunnala' Milos Vujasinovic', Gauri Joshi?, Anne-Marie Kermarrec*
EPPF. Switzerland CCarnegie Mellon University, USA akash. www.ade@epsl...commo .ch milos.
ABSTRACT
Model merging has emerged as efficient method to combine multiple single-task fine-tuned models. The merged model can enjoy multi-task capabilities without expensive training. While promising, merging into single model often suffers from an accuracy gap with respect to individual fine-tuned models. On the other hand, deploying all individual fine-tuned models incurs high storage costs. We propose
a novel data-free model merging framework that: (a) flexi- bly generates merged models of varying sizes, spanning the full spectrum from a single merged model to retaining all individual fine-tuned models; and (b) supports multiple merging algorithms in unified framework. Using FLEXMERGE, we systematically characterize the accuracy-size trade-off of different algorithms. Our study reveals two key findings: first, even modestly larger merged models can yield steep accuracy gains (up to 13.5% when just doubling the size); second, algorithm rankings are not consistent as size increases, with some methods overtaking others beyond the one-model regime. These results uncover a new design dimension for model merging: developing and comparing algorithms across the full spectrum sizes rather than only at the single-mode model limit. Extensive experiments vision and NLP benchmarks, with up to 30 tasks, confirm the generality and practicality of FLEXMERGE
INTRODUCTION
In recent years, the pre-training followed by fine-tuning paradigm has become the leading approach in both natural language processing (NLP) and computer vision, showcasing remarkable success on wide range of tasks (Devlin et al. 2018, Dodge et al. 2020 Dosovitskiy et al. 2021 Bommasani et al. 202T). Pre-trained models (PTMMs), which learn generalized features from aarge-scale datasets, serve as powerful starting points, enabling fine-tuning to achieve superior performance on downstream tasks with less labeled data. This has to an exponential growth the number of fine- fnnettund models driven further by the availability of open-source repositories (maintainers & contributors, 2016]] Wolf cet al. 2019). However, deploying individual fine fine-tuned models for specific tasks incurs high storage and deployment costs. The alternative is Multi-task learning (MTL), which aims jointly train single model across multiple tasks (Vandenhende et al., 2021]] Sanh et al. 2022) But MTL comes with its drawbacks, such significant computational overhead and the need to simultaneously access the data from all tasks, which might be infeasible due to privacy constraints (Jin et al. (202)).
To mitigate these limitations, model merging has emerged
promising solution, allowing the combination of multiple fine-tuned models into a single model without access to training data. To this end, several model merging methods have been proposed (Gargiulo al. 2025 Huang al. 2024
Yang et al. 2024 Yadav et al. 2023 Ilharco et al.) 2023 Matena & Raffe] 2022). However, single model is often unable to perfectly resolve parameter conflicts between tasks, leaving an accuracy gap with respect to the individual fine-tuned models (Zhang et al. 2025 Huang et al. (024)) This gap
becomes more significant as higher number of models are merged (Yadav al. | Ilharco et al.)
2023). To mitigate this issue, some methods leverage additional data to facilitate merging (Lu et al.
Work done during rescarch visit to Camegie Mellon University.
Our code is available at wtthhttt://// /tttpp://gitie com/ www.o///acseppppoo ffwmm/eeeeeeeeeeeeeee

---

Published as a conference paper at ICLR 2026
TA
Avg
TSV-M
Original models size
Flex size 1.75x
TIES
PCB
Avg. Avg. Accurcy
0.92
0.92
Bay Aocuncy
task
2
3
4
[u.e]
task b
2
3
d
ba.b.c, d.e)
la.d,e) lb.ll
ba. d.e)
0,88
0.88
[b.c]
0.84
0.84
task
3
4
0.80
0.80
task
4
task
--
2 3
Size 1.0x
Size 3.0x
(a) FEEX illustration
(b) FLEX 8 tasks (ViT-L/144
Figure 1: (a) Fine-tuned models are sequences of blocks. FLEXMERGE iteratively merges block pairs until reaching the desired size (e.g., size 1.75x). (b) Algorithm rankings change as size is increased.
2024
Yang et 2024a; Tang al. 2024a Yang al. 2024b). Yet, the data-dependency might be
difficult to meet in practice due to privacy constraints or proprietary restrictions, leading to growing focus on data-free model merging techniques (Gargiulo al. 2025 Huang al. [024 Du et al..
2024; Yu et al., 2024: Yadav et al.) 2023). Nevertheless, in the absence of data, the accuracy gap remains significant, highlighting the need for novel solutions.
We argue that an effective solution to this challenge is to go beyond the conventional one model approach, and merge into model(s) of bigger sizes. Merging multiple fine-tuned models naturally presents trade-off between maintaining accuracy and achieving model compactness, dictated by the size of the merged model. This trade-off spans a spectrum: at one extreme, retaining all individual fine-tuned models for each task achieves maximal accuracy but at the cost of larger overall size; at the other, fully merging all tasks into a single model minimizes storage size but sacrifices accuracy. Despite this clear trade-off, systematic investigation of the accuracy size relationship in model merging has been lacking. In this light, we pose two key research questions: (RQI) How can we derive merged models across the full range of model sizes in data-free manner? and (RQ2) What is the nature of the accuracy-size trade-off exhibited by different data-free merging algorithms?
In response (RQ1), we propose FLEXMERGE, flexible frame- work that enables data-free fusion into model(s) of any desired size. At its core, FLEXMERGE treats each fine-tuned model as composed of sequential blocks, as illustrated in Figure [a), whose granular- ity can be controlled (e.g., a transformer block, few layers, or even single layer). It then takes bottom-uy up approach starting with all fine-tuned models with their respective blocks and greedily merging pair of blocks with the highest cosine similarity in each merging iteration. This merging can leverage any existing data-free merging method such as Task Arithmetic (TA) (Ilharco al. 2023).
FlexMerge + TA
Avg. Accuracy
90
80
full
+13.5
acc.
70
4
6
FlexMerge Cons.
TIES MERGING (Yadav et al.] 2023). EMR- MER-MNGGING (uunn et al. 2024), TSV-M (Gargiulo et al. 2025), etc., applied the bloek-level. With each merging iteration, the size of the deployed model is reduced, and the process can be halted once the desired size
Arg. Accuracy
90
85
full
is met. For instance, in Figure [a), the merging is halted when the
80
+8.5
acc.
merged model is 1.75x the size of single fine-tuned model. The
75
entire merging process in FLEXMERGE needs no additional data or tuning, making FLEXMERGE fully data-free.
3
10
15
20 25 30
Deployed model size
In response to (RQ2), we demonstrate with FLEXMERGE that a range of data-free merging algorithms exhibit highly favorable accuracy-siz trade-offs. Remarkably, the accuracy-size trade-off is characterized by steep gains accuracy for even modestly bigger merged models beyond one model, followed by steady improve- ments, reaching near fine-tuning accuracy well before the maximum size. To illustrate this in practice, Figure 2 charts the merged model accuracy versus deployed size for 8 tasks (top) and 30 tasks (bot- tom) using the ViT-B/32 model, with TA (Ilharco et al. 2023 and
[in model units]
Figure 2: FLEXMERGE
ables large accuracy gains when just doubling the de- ployed model size and attains full accuracy well before the maximum size.
2

---

Published as a conference paper at ICLR 2026
CONSENSUS (Wang al. 2024) the respective merging methods. O and D annotate the accuracy at both ends of the spectrum i.e., lowest fused size and retaining all fine-tuned models respectively. FLEXMERGE TA gains 13.5% in average accuracy when going from 1x to 2x while FLEXMERGE + CONSENSUS gains 8.5% when doubling the size from approximately 3x to 6x. We note that CONSENSUS requires storing masks and the pre-trained parameters alongside the unified parame- ters (Wang et al. 2024), resulting in the lowest possible size of 2 3x for 30 tasks. We observe that the steep rise is followed by relatively slower accuracy growth in the middle. Yet, near fine-tuning accuracy is attained well before the maximum size. For 8 tasks, this is obtained around size 6x and for 30 tasks, around size x Secondly, we observe that algorithm rankings are not consistent even at modestly bigger sizes. Figure[]]b shows that vanilla averaging exceeds TIES- TA attains the performance of PCB-MERGING at size 3x despite starting from large gap at 1 x. Our findings open new design dimension: encouraging algorithm development and comparison for sizes 1x instead of restricting only to 1x.
while
Contributions. To the best of our knowledge, present the first study of model merging that:
Generates merged models across full spectrum of sizes, including non-integer sizes; Supports a wide range of data-free merging algorithms, within a unified franework;
Provides a systematic characterization of the accuracy-size trade-off in data-free model merging, revealing general trends, highly favorable regions and inconsistency of algorithm rankings;
Demonstrates that larger merged sizes incur negligible inference-time overhead, enabled by our efficient implementation.
We confirm our findings through extensive experiments spanning language and vision modalities, multiple model families, multi-modal datasets, using both full-parameter fine-tuning (FFT) and parameter efficient fine-tuning (PEFT), scaling up to 30 tasks.
2 RELATED WORK
Initial studies on model merging focused on vanilla averaging as a way of combining models obtained from same or different training runs of a task into one higher performing model (Izmailov et al.
2018 Gupta al., 2020 Wortsman al. 2022 Cha 202II. Vanilla averaging also used in federated learning to merge different client models (McMahan et al. 2017,) Konečný et al. (016). Ilharco et al. (203) introduced task vectors, representing the difference between fine-tuned and pre-trained models, enabling model combination through vector arithmetic.
Data-based merging methods leverage validation data to facilitate merging. Techniques like FISHER MERGING (Matena & Raffel 2022) and MEGMEAN (Jin et al. 2023) compute the Fisher Information and Gram matrices, respectively, for weighted averaging of model parameters. SURGERY (Yang el al. 2024 trains task-specific adapters to debias the representations produced by the merged model. ADAMERGING (Yang et al. 2024b) introduces per-task, per-layer merging co-efficients, and proposes to learn these co-efficients by solving an entropy minimization objective. WEMoR (Tang al. 2024a) merges all modules except for task-specific MIPs, which are retained as weight-ensembled mixture-of-experts (MoE) with learned routers. MERNINGGGING (Lu al. (2024) leverages MoE on difference vectors i.e., the difference between the fine-tuned models and the merged model. While the availability of validation data enhances accuracy, such data might be difficult obtain in practice.
Data-free merging directly merges model parameters without any data. TIES-MERGING (Yaday al.) 2023) resolves parameter interference by trimming redundant parameters and resolving sign (Du et al.) 2024) considers both intra- and inter-paranen parameter competition balaneing. DARE YYu al. 2024) reduces parameter interference by randomly dropping parameters and proportionally rescaling remaining ones. EMR EMRMMERGING (Huang et al. (2024) introduces the paradigm of maintaining weight task specific masks in addition to the merged model to enhance performance. CONSENSUS (Wang et al. 2024) also relies on task specific masks, but creates them differently compared to
conflicts.
Both approaches significantly improve accuracy over previous methods, albeit at the cost of test-time reconstruction overhead (Gargiulo et al.] 2025). TS V (Gargiulo et al.) (025) merges SVD-decomposed task singular vectors, reducing interference by retaining only prominent singular directions and orthogonalizing them across tasks.
3

---

Published as a conference paper at ICLR 2026
Recent work by Zhang al. (2025) explores merging into sizes 1x. Their method, CHANNEL MERGING, relies on layer-wise K-Means clustering followed by merging within each cluster using only TA. However, this approach is restrictive it cannot generate fractional-sized models. Despite the emergence of advanced methods and attempts at merging into bigger sizes, to the best of our knowledge, prior work has systematically investigated the accuracy-size trade-off in model merging under single unified framework. For completeness, we provide additional related work and a taxonomy of existing algorithms based on their caita--ree/data-biii based nature in Appendix
FLEXMERGE
3.1 PRELIMINARIES
We consider a set of M tasks: (Ti...., where the fine-tuned model parameters for task T. are denoted by O,. These fine-tuned parameters are typically obtained by adapting pre-trained model, such as ViT (Dosovitskiy et al., 2021 or T5 (Raffel et al. 2020) using either full parameter fine-tuning (FT) or garameter-efficiee assumed that all the fine-tuned models have the same size and the model architecture as the pre-trained model, as also considered in prior work (Ilharco al. 2023 Yadav et al., (023). To analyze the changes introduced by fine-tuning, we use the concept of task vectors 7; introduced by Ilharco et al. (2023), where T; = 0, Ppe-- with Pee being the pre-trained weights. These task vectors capture the specific modifications needed for each task and provide a compact representation for merging.
fine tuning (PEFT) methods (Liu et al. (022). Thus, it is
Standard model merging approaches involve combining the task- (TI,..., TM } into a unified task vector Tui = TM 1) and then adding the unified task vector to the pre-trained weights to get the final merged model. um = pa Tuni- Here denotes the merging algorithm used to obtain the unified task vector's 's weights. For example, the unified task vector Tuni can be computed via simple averaging T Tumi M E T, or via TA (Ilharco al. 2023) that uses coefficient À to weigh the contribution of the unified task vector Tuni = A. 37 2i=1 Ti in the final merged model. It is shown that just by Tuning X, one can outperform weight averaging (Ilharco et al. 2023).
Motivation. Merging into one model uni may cause accuracy deterioration due to parameter interference between different fine-tuned models (Zhang et al., 2025; Yadav et al. 2023). This behavior becomes prominent as more and more fine-tuned models are merged, as discussed in Section I On the other hand, retaining all fine-tuned models preserves full fine-tuning accuracy but results in a net size x that of one fine-tuned model, which is impractical due to the high memory requirements. In this work, we investigate the problem of generating models of any desired size in the range [1, M). including models with fractional size such as 2.25x x model units.
3.2 PROPOSED APPROACH
To enable a more granular fusion, we consider the model to composed of B sequential blocks, for instance transformer blocks in a ViT model or even layers within each transformer block such as attention or MLP layers could be considered as unique blocks. Assuming B total blocks, we consider the task vectors for each block as { corresponding to the original task vector ти for task k. Our proposed framework, FLEXMERGE, takes a greedy approach to efficiently merge task vectors from multiple tasks at the granularity of blocks, aiming to reduce the deployed model size while maintaining utility. The pseudo-code for FLEXMERGE is presented in Algorithm [
(Lines 1-6). The merging proceeds bottom-up. Initially, no merging has occurred, and we retain 7he for all tasks & E [M and all blocks b E [B] (see Figure
Initialization
[[ For each block b, we initialize a set of tuples: go = { ((k), +)) l E [M] Each tuple in gd consists of: (i) a task set {k} (tracking which tasks are represented) and (ii) the corresponding block task vector Th. For example, in Figure (a) for the first block, we would have G = { (fa), T2) ..... (fe), T!) }. When the merging terminates, the resulting g¹ for Figure 1 a) would be G1 {{aa.... e), ( }. where 41 is the merged task vector for the first block for all tasks. The initial size S is calculated as the cumulative size of all block parameters across M tasks.
We add scaling factor of 1 //M to the standard definition Tuni X EM, T; given in [lharco et al. 2023
better suit its usage in MERGE where M can vary across blocks.
4

---

Published as a conference paper at ICLR 2026
Algorithm 1: The FLEXMERGE framework
Input: Task vectors {T for all E [M], b E [B]; merging algorithm F; target size Sunget Output: Merged task vectors with reduced size
S x-0 0
Initialize deployed size
2 for b
B do
G +-0
for k = to M do
g + +g" O ((
S S + size(TT)
while S > Surzer or not all blocks merged do
Find block b" and pair (gs-, .3-) E 9 with the highest similarity:
(b' 9.. 9))
arg max SIMILAARITY(J..0) bEIB), g.ggeeh
10
TBe T + gg, 95- 95-(0) (0)
Get task subsets
11
Thi t T UTJ
Merge task subsets Merge task vectors b. Update the block Update current size
Tuni + F( k E })
12
13
g - gh (Tii T mmi
9
S S size(T
Iteration (lines 7-14). In each iteration, the algorithm identifies a block b* and pair of tuples (9-) gh which have the highest similarity (as defined below). Then they merged as follows. Let T and TT² be the subset of tasks associated with g;- and Øj- respectively, i.e., the first elements of gi- and gj- respectively. First, T. and T are merged via a union operation: Thi TH UTF Next, the merged task vector corresponding block b' and set T is created as follows: Th = F((Tk | k E 1). Here F can be any data-free merging algorithm. The tuple set gh is then updated by removing the tuples gr,, .9* and adding the new merged tuple (T T Each merge reduces the model size by the size of the task vector corresponding block D", and the process continues until the current size S meets the desired size Suaget or no further merges are possible.
Similarity function. We measure the similarity between two groups Ji, 9j in any block b using the lowest cosine similarity between any pair of original task vectors corresponding to the tasks in the sets T, and T/:
SIMILARITY (gi,
min cosine_sim(Ti, ET KETT
Our choice of the min similarity derives from our ablations comparing different strategies-max, min, and average-ar well as computing similarity between merged group task vectors directly. Among these, min yields the best performance. Thus at each iteration, we merge the pair of groups with the highest of these minimum similarities (line 9, Algorithm
While the cosine similarity between full task vectors can be relatively low (Ihharco et al.) 2023). the block-level similarities tend be higher
and effective for merging. CHANNEL MERGING (Zhang et al. 2025) also employs cosine similarity.
Enhancing efficiency, The pairwise similarities can be precomputed once for all pairs and accessed in constant time during the merging process. Furthermore, we implement Go using the Disjoint Set Union (DSU) (Cormen et al.) 2009) data structure efficiently track and unify task sets for each block. Our design enables FLEXMERGE perform very efficient merging even under many tasks (see T).
Complexity Analysis. Algorithm | identifies most similar block pairs in each iteration. We pre- sented the algorithm in this form for conceptual clarity. However, in practice, we can generate the global merging order for all blocks first and then apply merges. To analyze the time complex- ity of FLEXMERGE, we consider three distinct stages of this process: similarity pre-computation, generating merging order, and actual parameter merging.
5

---

Published as a conference paper at ICLR 2026
Table 1: Summary of existing data-free merging methods. Column F(IT) result of merging. Figure (Section B) provides an illustrative diagram.
Timi)) denotes the
Algorithm
F(lT? | Tm)))
Final Model
What is stored?
TA
[lharco et al. 2023
TIES
Yadav et al.
2025
Avg.
Uharco et a.. 2023
6 00 + Tun
6
PCB
(Du et al
2024
TSV-M
Gargiulo ct al. 2025
6% Ope + + me (reconstructed per-task k))
6% Time {rrg l c )am))
CONSENSUS
Wang al.
Tii 1mmg & E Tani)
6"_ 0dm + Th The m (reconstructed per-task k)
O Thm {m 18 Tuui
EMR-MERGING
Huang
2024
Thm (mik 7h k T
Similarity pre-computation: We compute the pairwise cosine similarities between M tasks in each block, across all B blocks. Let the maximum size of any block task vector be dmax then the similarity computes takes (dmmx) per pair. With (2) pairs per block, this step is O(BM (( ).
Generating merging order: Our greedy merging using Equation (T) an instance of a specific form of clustering. called single-linkage clustering. We thus use the SLINK algorithm (Sibson 1973) which takes input the similarity matrix and generates a sorted list of merge orders for each block in (M²)) For B blocks, this takes O ((BM²). We then need to combine these per-block sorted lists, each of size M 1, into a single global sorted list. Using a min-heap, this takes O(BM log B) (Knuth 1997). In total, this step takes O(BM² + BM log B) and results in global merge ordering across all blocks. Once the global merge ordering is obtained, we replay merges until the target size is met, simultaneously tracking task clusters via a DSU data structure, one per block. This gives the final groups { to which parameter merging is then applied. The per-block DSU overhead is (M))))) where (-) is the Inverse Ackermann Function (Cormen et al. 2009). For all practical purposes, or( .) < 5, resulting in x O(BM) for blocks.
We now merge parameters according to the final groupings in go. For linear algorithms like TA, merging a group g of size [g| with task vectors of size d takes ( d)) Summing over all groups within a block takes Egeg O(Ig d) O(E gegs I9l)d) = ((M)). Repeating this for B blocks and upper bounding d with dmmx results in ((BMdnax))
Applying parameter merging:
The total complexity is dominated by the similarity pre-computation (as dmax is typically larger than B), resulting in a final complexity of O ). Note however that dmax is much smaller than the total model dimension, at only corresponds the maximum size of any block of the model.
3.3 EXISTING MERGING METHODS IN COMBINATION WITH FLEXMERGE
FLEXMERGE provides the flexibility to choose any data-free merging algorithm from diverse set of existing approaches. Unlike traditional methods that operate at the level of full task vectors, FLEXMERGE applies merging algorithms at the block level, fusing block task vectors. We detail the exact bloek-level merging procedure for different algorithms next. In standard approaches like TA, TSV-M. and BCB-MERGING, task vectors are merged into a single unified task vector. When applied the block-level, the merging outcome for any block b can be denoted where F is the specific merging algorithm and T is the subset of tasks for which the merging occurs. The final block parameters are then computed as 0² 6tt + T'm-- Approaches such as CONSENSUS generate spec-fie masks in addition to the unified vector: Th (ml k E Tam < F(IT 1s 1ii Then during inference, the task specific weights for task k are reconstructed as - + Th o m CONSENSUS thus stores O pre Th and the binary masks (mh | k E } and defers per-task reconstruction to the inference time. This leads storage cost exceeding 2x that of standard methods, which only store O EMR-MEGGING further generates task-specific scalars {ok & E Tami in addition the masks, however the storage cost of these scalars is negligible. Table I summarizes the merging outcomes for different algorithms, applied at block-level within FEE Figure ( (Section B) provides illustrative diagram.
Thm F((Tk l E Thh)

---

Published as a conference paper at ICLR 2026
VIT-B/32
VIT-L/14
FlexMerge + TA
FlexMerge TIES
FlexMerge Cons.
FlexMerge EMR
Avg. Accuracy
0.95
0.95
0.95
0.94
0.85
0.85
0.92
0.90
0.90
0.75
0.75
0.88
0.65
0.65
0.85
2
4
6
2
4
6
8
2
4
8
4
6
Avg. Accuracy
0.95
0.94
0.90
0.90
0.90
0.92
0.80
0.80
0.85
0.90
0.70
0.70
0.80
0.88
0.60
0.75
0.86
5 10
20
30
5
20
30
5
10
20
30
1 5
10
20
30
Deployed model size [in model units]
Figure Merging 8 (top) and (bottom) tasks. The accuracy-size trade-off shows rapid initial gains, followed by gradual improvement, reaching near fine-tuning accuracy well before the maximum size.
4 EXPERIMENTS
We split evaluation follows: (i) Merging vision, PEFT and FFT benchmarks in Section 4.1 (ii) FLEXMERGE vs CHANNEL MERGING in Section 4.2 and (iii) ablation and efficiency analysis in and OOD results
Section 4.3
Lastly,
Appendices C.4 and C.6
Merging algorithms. We investigate the accuracy-size trade-off for several data-free merging algorithms including Vanilla Averaging, TA (Ilharco al.) 2023), TIES-MERGING (Yadav et al. 2023), PCB-MERGING (Du et al. 2024). TSV-M (Gargiulo et al. 2025),
(Cheng 2025). CONSENSUS (Wang al. 2024) and EMR-MERGING (Huang al. 2024) extensive vision and NLP benchmarks. As noted earlier, the focus of our work is data free model merging, Hence, existing data-based algorithms such SURGERY (Yang et al. 2024a), ADAMERGING (Yang
et al. 2024b), TWIN-MERGING (Lu et al. 2024), etc. are not directly comparable in our setting.
1.5. For TIES-MERGING 0.1 and employ the recommended value of A 1. For CONSENSUS, we set the hyperparameter responsible for controlling the amount of information extracted by masks to 0.6 for all tasks and use TIES-MERGING
Hyperparameters. For TA, we set
sparsity ratio of
the algorithm to generate unified task vectors. For FLEEXERGEE we set the block granularity at the level of individual components within the transformer layer, i.e., the attention, MLP, and layer normalization modules are treated as separate blocks during the merging process.
4.1 MERGING RESULTS
Merging and 30 vision models. For the image classification tasks, follow the setup from existing work (Huang et al., 2024 Yadav et al. 2023). Specifically, we use two versions of the CLIP model (Radford et al. 2021), incorporating B/3- and ViT-L/14 as visual encoders (Dosovitskiy et al., 2021). We evaluate on the standard 8 task benchmark (Iharcoo et al. 2023) well as an extended 30 task benchmark (detailed Appendix B.2)). F average accuracy vs. deployed model size (in multiples of single fine-tuned model). For FLEXMERGE + TA, the accuracy increases fairly rapidly as the model size grows beyond 1x. The gains are significant (top row), where the accuracy reaches 80% at size 2x from only 67.5% at size 1x for the ViT-B/32 model in the 8 task setup. Similar gains are also observed for 30 tasks (bottom row).
Masking- based approaches, CONSENSUS and EMR-MERGINE accuracy than TA and
begin with substantially higher but their smallest size exceeds 1x due to the need to store pre-trained weights and binary masks (Section 3.3 On 8 tasks, CONSENSUS shown match

---

Published as a conference paper at ICLR 2026
(a) (IA)² PEFT
(b) Full-Parameter FT (FFT)
FlexMerge + TA
FlexMerge + EMR
FlexMerge + TA
FlexMerge EMR
0.72
0.87
Avg. Accuracy
0.85
0.70
0.70
0.65
0.75
0.86
0.68
0.60
0.65
0.85
1
7
9
11
5 7 9
11
3
4
5
6
7 2
4
5
Deployed model size [in model units]
Figure 4: FLEXMERGE TA gains 7.2% for (IA)² going from 1x to 3x and more than 9% for FFT when just doubling the size from 1x to 2x. EMR begins with higher accuracy, yet, substantially benefits from increased size.
fine-tuned accuracy at small sizes, but only when its extraction parameter is separately tuned per task (Wang al. 2024). FLEXMERGE CONSENSUS also shows strong gains, improving from 76% at x 3x to 84.5% at = 6x for ViT-B/32 in 30 tasks. EMR-MERGING maintains high accuracy even at the smallest size. Yet, it exhibits an accuracy gap w.r.t the fine-tuned models, which can be effectively reduced by increasing the deployed model size. Larger ViT-L/I4 models achieve higher accuracy across all methods, but the accuracy-size trade-off remains similar: rapid initial gains followed by gradual improvements. Most algorithms approach the fine-tuning accuracy (denoted by [) well before maximum size, around 6x for 8 tasks and 23.5x for 30 tasks. Thus, in cases requiring storage of all fine-tuned models, FLEXMERGE can reduce size by about 25% with little accuracy loss.
Merging 11 PEFT models. We adopt the experimental setup from prior work (Huang al. 2024
Yadav et al. 2023). Specifically, we employ the (IA)s (Liu et al. 2022) PEFT method on the
TO-3B (Sanh al. 2022) base model using 11 diverse datasets sourced from (Yadav et al. 2023)
(detailed in Section B.3 Figure a demonstrates the benefits of deploying larger model sizes.
where in this case the model size is measured with respect to the (IA)3 modules. FLEXMERGE
+ TA achieves notable gains, increasing accuracy from 59% at size 1x to 66.2% at 3x, a 7.2% improvement. Similarly, FLEXMERGE EMR MERGING surpasses 70% accuracy at 5x, starting from 67. 67.6% at the lowest size of 2.34x. We observe similar trends for other algorithms, included in Appendix C.2
Merging 7 FFT models. For this experiment, we closely follow the setup from prior work (Du et al. 2024 Yadav et al., 2023). We use T5-Base and T5-Large as base models, applying full- datasets sourced from (Yaday et al., 2023) (detailed in Appendix [B.4p.
parameter fine-tuning Figure 4b) illustrates the trade-off between model size and accuracy for the L5-Large model. Here, one unit of model size corresponds the full size of single model. FLEX + TA gains more than 9% to reach an accuracy of 75% when just doubling the size from X to 2x. Similarly, FLEXMERGE EMR- MERMIIGGGING surpasses 86% size 4x, starting from 85.2% its lowest size of 2.2x. Consistent with our observations on vision tasks, FLEXMERGE + TA reaches very close to the fine-tuning accuracy around size 5x, much in advance of full size 7x Thus, scaling the model size benefits both ends of the spectrum. Results for other combinations are included in Appendix C33
Cross-algorithm analysis. Thus far, we evaluated the accuracy-sizz trade-off per algorithm. We now compare algorithms at same size, yielding two interesting findings: (i) the performance gap between different algorithms significantly narrows at slightly larger sizes; and (ii) the algorithms rankings also alter in many cases, with simpler algorithms rivaling or surpassing advanced ones. This behavior aligns with the phenomenon we observed earlier: as scale increases, parameter interference between tasks reduces substantially, leading to greater natural parameter disentanglement and higher accuracy, Consequently, the explicit interference reduuciio mechanisms built into advanced approaches such as TRIM in TIES competition balancing in PCB offer marginal added benefit, because much of the interference is already mitigated simply by the increase in scale. Thus simple algorithms begin rivaling their more sophisticated counterparts larger sizes. In Figure 5 on vision tasks,
8

---

Published as a conference paper at ICLR 2026
FlexMerge 'TA
Avg
TSV-M
Consensus
Channel Merge TA
TIES
PCB
EMR
0.92
0.94
0.94
Avg. Accuracy
0.92 0.69
0.69
0.87
0.92
0.82
0.90 0.66
0.66
0.90
0.77
0.88
0.88
0.63
0.63
0.72
Size 2.25x
Size 3.25x
Size 2.5x
Size 4.5x
Deployed model size [in model units]
8 Vision tasks (ViT-L/14)
11 PEFT tasks (TO-3B)
Figure (Left) FLEXMERGE + TA outperforms CHANNEL MERGING + TA across all sizes. (Center, Right) Algorithm rankings shift even at modestly larger sizes, with simpler methods rivaling advanced ones. We show sizes just over CONSENSUS and EMR-MERGING'S lowest size for wholistic comparison.
vanilla averaging exceeds TIES- MERGING at size 3.25x while TA overlaps with PCB. While EMR-MERGING and CONSENSUS stay atop vision, they are surpassed by PCB on PEFT at size 4.5x. Crucially, algorithms remain within 4% on both benchmarks increased sizes despite originating with much larger gap size 1x (see Figure Our findings provide encouraging evidence to develop and compare algorithms at sizes > 1x rather than only at 1x.
4.2 FLEXMERGE CHANNEL MERGING
CHANNEL MERGING Zhang et al. 2025 uses K-Means clustering per layer, following fixed same value K for every layer. Each choice K (2,3..... -11 1} results merged model of the corresponding size. Figure[5]chart, the average accuracy with TA and ViT-B/32 for a set of integer model sizes, excluding the extremes 1x and 8x where both approaches have the identical accuracy. Recall that CHANNEL MERGING does not support fractional sizes. FLEXMERGE achieves higher accuracy than CHANNEL MERGING in all cases, thanks to its greedy pairwise merging approach which allows flexible number of groups per layer instead of restrictive clustering. Results with TIES-MERGING and visualization of clusters is included in Appendix C.5
4.3 ANALYSIS
Ablations the merging procedure. We ablate the similarity functions (min, max, average, comparing unified vectors) for Equation (T) and merging orders (left-to-right, right-to-left, greedy) in FLEXXERGG using the ViT-B/32 model on tasks. We also investigate random block selection
cosine similarity. 6 shows that the min strategy performs the best, though other strategies are also competitive. For merging order, right left performs the worst as expected since the final layers in neural networks tend to be more specialized and merging them first hurts accuracy. While left to right seems ideal, it can be too strict and therefore greedy emerges the best. We further analyze the merging order of greedy in Appendix CC.00 Random selection is competitive, but generally underperforms when compared across algorithm. Based on these findings, we FLEXMERGE to use greedy with cosine similarity (min strategy) by default. For more ablations, see Appendix C.8
Merging and inference efficiency. Table 2 shows that FLEX achieves highly efficient data-free merging, generating all deployed sizes in about 20 sec for up to 30 tasks. For inference with FLEXMERGE, each request follows a unique forward path through the merged model using task-specific blocks (Figure aa).. For model of size 1x, all tasks share a single path, but the classification heads are always applied separately. We load the tensors of merged model (size > 1x) into the GPU memory once and create M task-specific model views that reference these shared tensors to process task batches in parallel. Standard merging, by contrast, processes all tasks in single batch before splitting for task-specific heads. We simulate the worst case arrival, where

---

Published as a conference paper at ICLR 2026
Similarity Function
Merging Order
Selection strategy
max
avg
left-right greedy right-left
random
uni min
cosine
FlexMerge TA
FlexMerge + TA
FlexMerge
TA
FlexMerge + Cons.
Avg. Accuracy
0.9
0.9
0.9
0.90
0.8
0.85
0.8
0.8
0.88
0.80
0.7
0.7
0.7
0.86
1
6
2
4
8
1
6
8
Deployed model size [in model units]
Figure 6: Ablation results for FLEXMERGE reveal that the min similarity strategy and greedy merging perform the best, while cosine similarity generally outperforms random selection.
inference batches corresponding to tasks arrive at once. We consider 50 consecutive batches size 256 (totaling 12800 samples). Each batch contains 32 samples per task across 8 tasks. Table shows that FLEXMERGE maintains inference speed comparable to standard merging for both ViT-B/32 and ViT-L/14, demonstrating that larger models can enhance accuracy without slowing inference.
Table 2: FLEXMERGE's merging time.
Table 3: Comparing inference time of FLEXMERGE against standard model merging. The overheads are negligible.
Method
VIT-R/32 (6) (sT-L/14
Tiasks
Model
Algorithm
Size
Inference Cost (/12500 items)
TA (size 1x) FLEXMERGE (all sizes)
0.8
2.6
VIT-B-32
Standard Merging
12.30 0.21 ms
3.4
FLEXMERGE
1x
12.21 0.41
VIT-L-14
Standard Merging FLEXMERGE
1x
118.70 1.78
>> 1x
120.53 +: 0.32 ms
TA (size 1x)
1.9
6.1
FLEXMEROE (all
20
DISCUSSION AND CONCLUSION
Benefits. Different merging algorithms have different advantages: EMR and CONSENSUS achieve high accuracy but require task-specific reconstruction during inference, incurring heads. FLEXMERGE can also mitigate this overhead as larger deployed models need fewer blocks be reconstructed (see Appendix C.9). In contrast, TIES and TA avoid reconstruction but have lower accuracy. FLEXMERGE provides flexibility, letting practitioners choose algorithms and balance accuracy, reconstruction overhead, and model size for various deployment scenarios.
Limitations. Most works, including FLEXMERGE, are limited to merging models with the same architecture as merging heterogeneous models remains challenging (Singh Jaggi 2020 Imfeld et al. 2024). Secondly, the theoretical insights for effective model merging are limited Ortiz-Jimenes al. 2023). For FLEXMERGE, how obtain the optimal merged model for any given size remains unclear. Although extensive ablations help guide (Section 4.3), further investigation needed to understand the bounds the accuracy-size trade-off.
We introduced FLEXMERGE, flexible, data-free model merging framework that extends beyond tra- ditional single-mode fusion and offers precise control over fused model size. Extensive experiments show that the accuracy-size trade-off exhibits favorable properties for several algorithms, benefit- ing from rapid accuracy gains with modest size increments. Future work may explore specialized algorithms for block-level merging.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Akash was supported by the EPFL DobiMityl fellowship during his research visit Carnegie Mellon University. This work was also partially supported by the NSF grants CCF 2045694, CCF 2428569, ENS-2112471, CPS-2111751, ONR grant N00014-23-4 1------2--999 A12C Seed grant, and by the Swiss National Science Foundation, under the project "FRIDAY: Frugal, Privacy-Aware and Practical Decentralized Learning", SNSF proposal No. 10.001.796 The authors are thankful to Martijn de Vos and Arian Raje for their helpful feedback during the preparation of this work.
REFERENCES
M Israk Ahmed, Shahriyar Mahmud Mamun, and Asif Uz Zaman Asif. Denn-based vegetable image classification using transfer learning: A comparative study. In 2021 5th International Conference on Computer, Communication and Signal Processing (ICCCSP), pp. 235-243 IEFF, 2021.
Sarder Iftekhar Ahmed, Muhammad Ibrahim, Md Nadim, Md Mizanur Rahman, Maria Mehjabin Shejunti, Taskeed Jabid, and Md Sawkat Ali. Mangoleaftd: comprehensive image dataset to classify diseased and healthy mango leaves. Data in Brief, 47:108941, 2023.
Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. Promptsource: An integrated development environment and repository for natural language prompts. arXiv preprint arXiv:2202.01279, 2022.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65 72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL 09097
Punect Bansal. Intel image classification. Available on https://www. kaggle. com/punect66000/mte image- classe-lassiffcation, Online, 2019.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07255 2021.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 Mining Discriminative Components with Random Forests. In IEEE European Conference on Computer Vision (ECCV), 2014.
CCHANG. Garbage classification. https: http::://wwwwwe.. httpo///www.aaglle..coo///////1
2018
Junbum Cha. Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho. Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: domain gencralization by secking flat minima. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 22405- 22418, 2021. URL Uttpttss://///peeeiiing wwwp:/oceeee..n......ee...eeeeee www.iierippp. http://wwwooo.oo..oooooo.oo..opp pccp///aapp////2///sss/./ 41
BataaaAAbbrrract html
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10): 1865-1883,,, 2017.
Runxi Cheng, Feng Xiong, Yongxian Wei, Wanyun Zhu, and Chun Yuan. Whoever started the interference should end it: Guiding data-free model merging via task vectors. In Forty-second International Conference on Machine Learning, 2025. URL
orum? d=
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
11

---

Published as a conference paper at ICLR 2026
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv, 2018. URL /ttppp////xvvv//////
abs/1812. 01118v1
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011. https::/pprceeetee
press / v1 5 coates1la, html
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending MNIST to handwritten letters. In International Joint Conference on Neural Networks (ICNN), 2017.
Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. Introduction to Algorithms, Third Edition. The MIT Press, 3rd edition, 2009. ISBN 0262033844.
Carrie-Catheineee De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp. 107-124, 2019.
DeepNets. https//ww.....aao.oooooooooo///a
Jia Deng, Wei Dong, Richard Socher, Li-Jia Ii, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 10.1109. 5206848.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18Xiv:/810.0440 2018.
Jesse Dodge, Gabriel Ilharco. Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprini arXiv:2002.06300 06300, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In Proceedings of the International Conference on Learning Representations (ICLR). 2021.
Guodong Du Junlin Lee, Jing Li, Runhua Jiang, Yifci Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, and Min Zhang. Parameter competition balaneing for model merging. In The eighth Annual Conference on Neural Information Processing Systems, 2024.
Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34: 17501-27516, 2021.
Antonio Andrea Gargiulo. Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola. Task singular vectors: Reducing task interference in model merging. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18695- 18705, 2025.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9, 2007.
Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Hyug Lee, et al. Challenges in representation learning: A report three machine learning contests. In International Conference on Neural Information Processing (ICONIP), 2013. URL
org/ abs/1307 04144 4v1
Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 30. 2020. OpenReview.net, 2020. URL https::/openreet net het:/Forumiid=rygwaaaa.
12

---

Published as a conference paper at ICLR 2026
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2233 2019.
Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. EMR-merging: Tuning- free high-performance model merging. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL /tt.o///e////////eeees.ss..eeeer www.://nteee//ooto-------- djzx3DYu
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL
wf8 jrj
Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Transformer fusion with optimal transport. In The Twelfth International Confer- ence on Learning Representations, 2024. URL httpt::://ppereeeeee/e/ htwparreweeeeeeeeeooommmmmmmm ferum?irrmm LjecMvQpen
net/
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Amir Globerson and Ricardo Silva ....). Proceedings of the Thirty-Fouud Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, Califormia, USA, August 6-10, 2018, pp. 876 885. AUAI Press, 2018. URL pdf
wttpppeeww.n//pppppeeeeeeeeeeeee ngs http:////aaaaaooooo/oooooooooooa
http: /tt///aair/g/a///1111///
Mona Jalal, Kaihong Wang, Sankara Jefferson, Yi Zheng, Elaine O Nsoesie, and Margrit Betke. Scraping social media photos posted in kenya and elsewhere to detect and analyze food types. In Proceedings of the 5th International Workshop Multimedia Assisted Dietary Management, pp. 50-59, 2019.
Ruochen Jin, Bojian Hou, Jiancong Xiao, Weijie J Su, and Li Shen. Fine- tine-tuning attention modules only: Enhancing weight disentanglement in task arithmetic. In The Thirteenth International Conference Learning Representations, 2025.
Xisen Jin, Xiang Ren, Daniel Preotiuc- Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023.
net/ forum www.oooimnnnooouuuraaa....
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qase: dataset for question answering via sentence composition. Proceedings of the AAAI Conference Artificial Intelligence, 34(05):8082 8090. Apr. 2020. doi: 10. 7/ojs. . aaai org/ index.ndexxee/ php AAAI httpp///aaaaatcee/eeeeeeea 6319
334105.6319, URL https
Donald Ervin Knuth. The art of computer programming, volume 3. Pearson Education, 1997.
Jakub Konečný, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. CoRR, abs/1610.02527, 2016. URL
http://a /xt///////vvv/////////v/ orgg abs 610. 02527
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009. https: http:p//www...cccaaacacoor//rrap
Makerere Al Lab. Bean disease dataset, January 2020. URL /Makereere/ieenn
13

---

Published as a conference paper at ICLR 2026
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. con/exdb/nmist// 1998.
Yeoreum Lec, Jinwook Jung, and Sungyong Baik. Mitigating parameter interference in model merging via sharpness-aware fine-tuning. In The Thirteenth International Conference on Learning Representations, 2025. httpst www.revveeeeeeeeeoooeee net. for:m Ftw.rrrmmmmaaaaaapppp......
Hector Levesque, Emnsst Davis, and Leora Morgenstem. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning.
2012.
Chin-Ye Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. |RL http::tpps//.tm........t //cl antho ogg. wrgyoo0/g/04-1133
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1..
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang. Mohit Bansal, and Colin A Raffel. Few-shot parameter-effiient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems (NeurIPS), 35: 15:1999-1965, 2022.
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. In The Thirty-eighth Anmual Confer- ence on Neural Information Processing Systems, 2024. URL
orum?id=81YIee 63TTn
Vischn maintainers and contributors. Torchvision: Pytorch's 's computer vision library. https 2016.
//github. com http://pytcccch.h...pe//
Michael S Matena and Colin Raffel. Merging models with fisher- averaging. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htttps::/open.revieet
d= wwt.tee///fort ftt////rmmmmiiiis111 p_ace
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Arcas. Communication-effie efficient learning of deep networks from decentralized data. In Aarti Singh and Xiaojin (Jerry) Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FI., USA, volume 54 of Proceedings of Machine Learning Research, pp. 1273-1282. PMLR, 2017. URL http: //proceedings.mi:: mir. press /v44
httm///maaaaan//a///a///a/.aaa.a https///////mmmmaammaaa/aaa 7a html
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14555 2019.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, gruphics & image processing, 2008.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Thirth-seventh Conference on Neural Information Processing Systems, 2023. URL
n Eorum? 2i:::1i::c222 jZDGW
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pp. 311 311-318, USA, 2002. Association for Computational Linguistics. doi: 0111115511773033177311333 URL http:///ppp////a...../////0////0
3.1073155
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Mohammad Taher Pilehvar and Jose Hamacho-Colladons Wie: the word-in-context dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv: 1808 09/2/, 2018.
14

---

Published as a conference paper at ICLR 2026
Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter The- lin Schmidt, et al. Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In Proceedings of the 8th ACM on Multimedia Systems Conference, pp. 164-169, 2017.
Languku Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
Models are Unsupervised Multitask Learners, 2019.
Lnguagge models/
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(140):1-67, 2020.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011. 1.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106. 2021.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL
Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 752-757, 2018
R. Sibson. Slink: An optimally efficient algorithm for the single-link cluster method. The Computer Journal, 16(1):30-34, 01 1973. ISSN 0010-4620. doi: 10. 1003/comjn//16.111o URL https
1/doi.orge 1093/ com j.1.1..1.1110
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. heurips ccppapercppppp...pp f
Curran Associates, Inc., 2020. URL http:httts/spooceiiiiiii
Dosbold-Papez.c.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP), 2013. https?
7/aclantho htwp:hoo_ogyorrr//33-11100 orq/D13-1170,
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In The 2011 international joint conference on neural networks, pp. 1453-1460. IEEE, 2011.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International conference on machine learning, pp. 9120-9132. PMLR, 2020.
15

---

Published as a conference paper at ICLR 2026
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots. In The Thirteenth International Conference on Learning Representations, 2025.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs, 2019, URL http::t/arxii..or g/ 3/abs/1811 00491
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. QuaRTz: An open-domain dataset of qualitative relationship questions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joini Conference on Natural Language Processing IEMNLP-JJCNLP,, pp. 5941-5946, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.18653/v I/D19- 1608, URL /ttpttts///caaaatoollooo////
1608/
Anke Tang. Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging mulli-task, models via weight-ensembling mixture of experts. In Forty- International Conference on Machine Learning, 2024a. URL htttps://openree
ew. net/ forum? www.mmmdddnnnnnnna d=nLLKn07 4RB
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization. In The Twelfth International Conference on Learning Representations, 2024b.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3614 4-333,, 2021.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4566-4575, 2015. doi: 10.1100/CVPR.220555229953
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivari- ant CNNs for digital pathology. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2018. http:ttp:://aa...oo/o/aaaaaa////
0
3962v1
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In Forty-first Interna- tional Conference on Machine Learning, 2024. URL htttps://pperreiieeee
met/fooum
Pid=DWTuuiGjxT
Ke Wang, Nikolaos Dimitriadis, Alessandro Favero, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Lines: Post-training layer scaling prevents forgetting and enhances model merging. In The Thirteenth International Conference Learning Representations, 2025.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and lasion-language tasks, 2022. URL https://aaxx
v.org/abs/2200 2208 10422
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, et Huggingface's transformers: Statc-of-the-a natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the International Conference on Machine Learning (ICML), pp. 23965-23998,, 2022.
Haixia Xiao, Feng Zhang, Zhongping Shen, Kun Wu, and Jinglin Zhang. Classification of weather phenomenon from images by using deep convolutional neural network. Earth and Space Science, 205)=2200 604, 2021.
16

---

Published as a conference paper at ICLR 2026
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: novel image dataset for benchmarking machine learning algorithms. arXiv, 2017. URL WRLLbtt::/axxxv..oo http://.....g//////.///////// /ttpa////aaau/..o//////////////0
V2
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information P Processing Systems, volume 36, pp. 7093-7115. Curran Associates, Inc., 2023. URL tR www.://eww....ee..o.............
aper- Coer-Conferrecee.rr pdf
Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. -3332-56356. PMLR, 21-27 Jul 2024a. URL https::/prrceediing
www.ss.ressssee.e yaggg//2.///gn...... html
Enneng Yang, Zhenyi Wang. Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference Learning Representations, 2024b. htttts:t///peeeteeee
net/ formm
Hid=n2P6NGD30X
Yi Yang. Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open-domain question answering. In Lluís Märquez, Chris Callison-Buis Burrch, and Jian Su (eds.). Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2013-2011111 Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.11863/11/D155 1237. URL https: http:///aaaaaahhhhoo ottp://oooooooro//11 1237/
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference Machine Learning, 2024.
Netzer Yuval. Reading digits in natural images with unsupervised feature learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905007830000 2019.
Mingyang Zhang, Jing Liu, Ganggui Ding, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Channel merging: Preserving specialization for merged experts. Proceedings of the AAAI Conference Artificial Intelligence, 39(11) https: www.aaaaaa...m.........aa....... org/ http:/nnexx...m php/ AAAAAAAAAATS....... http///AA/aaaaaaaeaaaaeaa
Apr. 2025. doi: 16.1609/aaii v39.2.......911.......... .34405. URL
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2298-1308, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 1/119 URLphttss //aclantholoog.orrg/////o/ 1131/
Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, and Fei Wu. Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering. In The Thirteenth International Conference Learning Representations, 2025.
17

---

Published as a conference paper at ICLR 2026
ORGANIZATION OF THE APPENDIX
Extended related work
B
Additional details
B.
How to apply existing merging algorithms at the block-level?
B
Vision benchmark
B.3
Parameter efficient fine-tuning (PEFT) benchmark
B.
Full- parameeer fine-tuning benchmark
B.5
Multi-modal benchmark
Additional results
CA
Vision benchmark
Parameter efficient fine-tuning (PEFT) benchmark
C
Full-parameter fine-tuning benchmark
Multi-modal benchmark
FLEXMERGE vs CHANNEL MERGING
C.6
OOD performance of FLEX
Scaling laws for flexible model merging
C.8
Ablations of the merging procedure
Reconstruction latency masking based approaches
C.
Merging order analysis
Main results presented with normalized accuracy
Dataset-wise results
D Compute Resources
LLM Usage Statement
A EXTENDED RELATED WORK
Table Sabbeaccharattrizzze different merging algorithms on their data-based/data--ree nature and merged model size. Below we discuss additional related work.
Multi-Tash Learning (MTL). The traditional approach to obtaining a model with multi-task capabili- ties MTL which trains single model using training data from multiple tasks together (Vandenhende al. 2021 Sanh al. 2022 However, MTL suffers from not only (i) the expensive computational cost for training, but also (ii) the limited data availability due to data privacy (Jin et al. 2023 Yang al. 2024b). In comparison, model merging bypasses these challenges by combining the fine-tuned model weights directly, without training data, thus offering a more cost-effective approach to building multi-task model. Existing research in MTL also focuses on grouping tasks i.e.. identifying subsets of task that derive positive benefit from training together (Standley et al.] [2220] Fifty et al. 2021). Since training one model for all tasks can lead suboptimal performance due task conflicts and competition for model capacity, these methods train separate mulli-task models for specific task groups. This can be seen as conceptually similar to our approach of having different task subsets merged block to improve performance (see Figure [[a). While our approach performs grouping during merging, these approaches perform grouping during training.
Model Merging. Besides the works discussed in Section 2 recent works also focus on merging models fine-tuned specifically with Low-Rank Adaptation (LoRA) (Stoica et al. 2025 Zhao et al. 2025 Tang et al. 2024b). Other efforts focus on developing approaches for tuning that result in lower interference during downstream merging (Lee et al. 2025) Jin et al. 2025 Ortiz-Jimenes et al. 2023). LiNeS (Wang al.) 2025) proposes post- traininainng editing technique to reduces negative interference between parameters by scaling parameter updates based on their layer depth.
FI.EX seamlessly incorporate these recent advances.
18

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of merging algorithms by data dependency and merged model size.
Algorithm
Data Free
Size
Storage beyond the unified model
Weight Average
1x (Fixed)
TA (Ilharco et al.
2023
1x (Fixed)
TTES-MEKGING Yaday ct al.
2023
1x (Fixed)
PCB- BCB-MERGING (Du
2020
(Fixxed)
CONSENSUS (Wang cl al. 2024
2x (Fixed)
Stores masks and pce
EMR-MERGING (Hung a.
2 (Fixed)
Stores masks and Ppe
TSV-M (Gargiulo ct al.
(Fixed)
WUDI-MERRGNG (Cheng al. 2025
1x (Fixed)
REGMEAN Jin
2023
(Fixed)
FISHER MERGING (Matena & Raffel
2022
(Fixed)
ADAMERGING (Yany
20241
1x (Fixed)
SURGERY (Yang
202e4
X
1x (Fixed)
Stores task-speciffic adapters
WEMoE Tang al 20244
X
> 1x (Fixed)
Strres MLP modules for all tasks Stores compressed diff. vectors and ppe
TWIN-MEKGING [uu al.
2024
X
2x (Fixed)
Pre-trained block
Masks
Merged block
2
2
(a,d,e)
{8,8)
a.d.e)
a.0
(a.b.c.
(a,b,c,
(b.c)
(a, c, d,e)
(a
(b.c)
d,e)
(b.c)
d,e)
(b,c)
d
(d)
(d)
6
pze
pee.
pre
prn
FlexMerge
(a) FlexMerge (Consensus, EMR)
(TA, TIES, PCB)
Figure 7: FLEXMERGE in combination with different merging algorithms. Standard methods such TA, TIES-MERGING, etc. generate the merged parameters per block. Recent methods such as CONSENSUS and EMRMMERGING generate binary masks in addition to the unified parameters. They also store the pre-trained parameters for task-specific reconstruction.
B ADDITIONAL DETAILS
B. HOW APPLY EXISTING MERGING ALGORITHMS THE BLOCK-LEVEL
While most merging algorithms can be directly applied at the block-level, we list them below and explain any specific adaptations that help improve performance. Figure provides an illustrative diagram for FLEXMERGE in combination with different merging algorithms.
Averaging. Applied directly to obtain T
F(lT k T
kk
Je Thh
TA (Ilharco al. 2023 Applied directly obtain Tuni-
F(IT) & Thh X
T
uni
KET
For all experiments with TA, we use X = 1.5. Note that our definition of À for TA excludes the 1/TTil factor (see Section 3.11. in contrast to prior work. This modification is essential for ensuring reasonable merging performance with TA, as the number of tasks being fused per block varies throughout the bottom-up fusion process, ranging between 2 and M.
19

---

Published as a conference paper at ICLR 2026
TIES-MERGING (Ydaa al. 2023). We apply trimming to the full task vectors Tk: [M] to retain the top 10% of the parameters before beginning the greedy merging process. This is because selecting the top parameters globally performs better than selecting them within each block. During the bottom-up merging process, only the elect sign and disjoint merge steps of TEES-MERIING are performed. Lastly, we employ the recommended value of À 1.
PCB-MERGING (Du al. 2024). Applied directly at the block-level by executing the steps of intra-balancing, inter-balancing and drop and rescale at the block-level. In PCB- MERGING, the drop operation for any task vector depends on other task vectors that being merged with. This contrasts with TIES-MERGING is solely dependent on the magnitude values in the task vector. Therefore, global trimming is not possible in PCB-MERGING and we execute it block-wise. We set the sparsity ratio to 10% for vision tasks and 20% for PEFT and FFT experiments. Additionally, we employ the recommended value of À across all experiments.
where the drop (or trim) operation
CONSENSUS (Wang et al., 2024). Applied directly wherein the merging results in not only the unified task vector but also the task specific masks.
> {m' A E Tim) F((Tk E E
The task-specific weights for task & E T corresponding to this merged block are then reconstructed during the reconstruction process as:
0 = + Tuni m
We use TIES- MERGING as the algorithm generate Th within CONSENSUS. Note that the above version of CONSENSUS corresponds to the compression application in (Wang et al., 2024). The alternative version of CONSENSUS that corresponds to merging using masks can be also directly leveraged within FLEXMERGE.
EMR-MERGING (Huang et al. 2024). Applied directly wherein the merging results in the unified task vector, the task specific masks and task-specific rescalers.
Timi {m > k E Thh + F(1T) k E T
The task-specific weights for task & E Th corresponding to this merged block are then reconstructed during the reconstruction process as:
0% O' The Th m
TSV-M (Gargiulo et al.
2025). Applied directly per- as the method originally also operates layerwise. 2D parameters are merged using their task singular vectors while ID parameters just use averaging for merging as done in their original work.
WUDI-MERGING (Cheng et al. 2025). Applied directly per-block the method originally also operates layerwise. We use the same learning rate 10 as used by the authors within the method. Lastly, we set the number of iterations to 300 for 8 tasks and 1000 for 30 tasks.
B.2 VISION BENCHMARK
The 8 task vision benchmark (Iharcoo et al. 2023) comprises the following datasets: 1. SUN397 Xiao
et al. 2010 2. Cars (Krause et al. 2013). RESISC45 (Cheng et al., 2017), 4. EuroSAT (Helber
et al. 2019), 5. SVHN (Yuval [201]]
6. GTSRB (Stallkamp et al.) 2011)). 7. MNIST (LeCun 1998]
and DTD (Cimpoi et al. 2014). We extend the task vision benchmark with 12 additiona
datasets sourced from (Wang et al. 2024), including: 9. CIFAR 100 Krizhevsky et al. 2009).
10. STL10 (Coates et al. 2011). 11. Flowers 102 (Nilsback & Zisserman 2008). 12. OxfordIlTTP
Parkhi et al. 2012). 13. PCAM (Veeling et al.) 2018), 14. FER2013 GGoddellow et al. 2013).
15. EMNIST (Cohen et al. 2017). 16. CIFARTO (Krizhessky et al. 2009). 17. Food101 [Bossard
et al. 2014), 18. FashionMNIST (Xiao et al.) 2017). 19. ReRnneeeddSS (Socher et al.) 2013 Radford
ct al. 2019) and 20. KMNIST (Clanuwat et al.) 2018). The remaining 10 datasets for our 30
task benchmark are sourced from (Huang et al. 2024), which include: 21. Weather (Xiao et al.
2021b. 22. Vegetables (Ahmed et al. 2021 23. MangoLeafBD (Ahmed et al. 2023). ) 24. Landscape
Recognition (DeepNets), 25. Beans (Lab) 2020), 26. Intel Images (Bansal 2019), 27. Garbage
Classification (CCHANG) 2018) 28, Kvasir (Pogorelov et al. 2017), 29. KenyanFood (Jalal et al.
2019) and 30. Dogs (Khosla et al. 2011
20

---

Published as a conference paper at ICLR 2026
B.3 PARAMETER EFFICIENT FINE-TUNING (PEFT)
We fine-tune (IA)² modules on 11 diverse datasets, including RTE
Giampiccolo et al. 2007).
CB (De Marneffe et al. 2019), Winogrande (Sakaguchi et al. 2021). WIC (Pilehvar & Camacho-
22188) WSC (Levesque et al. 2012), COPA (Roemmele et al. 1011). H-SWAG (Zellers
al. 2019), Story Cloze (Sharma et 2018), and ANLI (Nie al. al. 2019) (RI R3). In addition,
we leverage prompt templates from the Public Pool of Prompts (P3) (Bach et al.] 2022) which
convert each dataset example into text-to-text format, where each label is mapped
unique string representation. We report the median performance across all templates for each dataset. Evaluating
multiple prompt templates increases the evaluation runtime significantly. To ensure the runtime remains manageable, we cap the maximum number of test samples at 1000 per dataset.
B.4 FULL PARAMETER FINE-TUNING
The 7 datasets that we consider for fine-tuning include: PAWS (Zhang et al.
2019h, QASC (Khot
et al. 2020), QuaRTz (Tafjord et al., 2019), Story Cloze (Sharma et al. 2018].
WikiQA Yang et al.
2015). Winogrande (Sakaguchi ct al. 2022 and WSC (Levesque et al. 20122) During training and
evaluation, we apply a specific prompt template from P3 (Bach et al. 2022) to each dataset. Each model is trained for up to 75 000 optimization steps, with early stopping if validation accuracy does not improve over five consecutive evaluation rounds. Performance is evaluated every 5 steps for WSC and every 100 steps for other datasets, using the full validation set. Training is conducted using the Adam optimizer with a constant learning rate of 0.0001 and an effective batch size of 1024. The maximum sequence length is set to 128, and 6 precision is used for both training and evaluation.
B.5 MULTI-MODAL BENCHMARK
We evaluate the performance of merging fine-tuned checkpoints of BEiT-3-base model (Wang et al.
2022) under FLEXMERGE. We consider four datasets: COCO Captioning (Lin al. 2014) (Image Captioning), ImageNet-Ik -k (Deng et al. 2009) (Image Classification), NLVR2 (Suhr et al. 2019)
(Visual Reasoning) and COCO Retrieval (Lin et al. 2014) (Image-Text Retrieval). The individual checkpoints, each fine-tuned on one of these datasets, are available in UniLM repository We merge layers that are common to all checkpoints i.e., excluding the task-specific classification heads. We report accuracy for ImageNet-Ik, NLVR2 and COCO Retrieval, meanwhile for COCO Captioning, we report BLEU-4 (Papineni et al. 2002). CIDEr (Vedantam et al., 2015), ROUGE-L (Lin 2004)
and METEOR (Banerjee & Lavie 2005).
C ADDITIONAL RESULTS
In this section, we provide the remaining results for each benchmark.
C.1 VISION BENCHMARK
Figures g and 9 show the results on 8 and 30 tasks respectively, for Averaging, PCB-MERGING, TSV-M and W WUDI-MERGING extending our results in Figure FLEXMERGE Avg. gains over 15% in accuracy by just doubling the size from 1x to 2x in the 8-task benchmark under the ViT-B/32 model. PCB-MERGING starts with higher accuracy at size 1x than vanilla averaging, thanks to its competition balancing procedure. Yet, it significantly benefits from increasing the deployed model size. Under the same setup as above, FLEXMERGE + BCB-MERGING gains 7.5%, by just doubling the size from x to 2x. TSV-M and WUDI-MERGING significantly lead both prior algorithms at size 1 xx still showing steep improvements of over 4% from size 1x to 2x. We observe similar steep initial gains for both the algorithms even with 30 tasks. Notably, in all cases, observe that near fine-tuning accuracy is reached well in advance of the maximum size, around 6x for tasks and around 23.5x for 30 tasks.
21

---

Published as a conference paper at ICLR 2026
ViT-B/32
ViT-L/14
FlexMerge Avg.
FlexMerge + PCB
FlexMerge + TSV-M
Flex + WUDI
0.95
0.95
0.95
0.95
Avg. Accuracy
0.85
0.90
0.90
0.75
0.75
0.85
0.85
0.65
0.65
0.80
0.80
6
8
2
8
Deployed model size [in model units]
Figure 8: Accuracy-size trade-off for tasks.
ViT-B/32
VIT-L/14
FlexMerge Avg.
FlexMerge + PCB
Flex + TSV-
Flex WUDI
0.95
0.95
0.95
Avg. Accuracy
0.9
0.90
0.85
0.8
0.85
0.85
0.75
0.7
0.80
0.80
0.65
0.75
0.6
0.75
0.55
0.70
5
10
20
30
5
10
20
30
1 5
10
20
30
5
10
20
30
Deployed model size [in model units]
Figure 9: Accuracy-sio trade-off for 30 tasks.
FlexMerge TIES
Flex.Merge + Cons.
FlexMerge Avg.
FlexMerge + PCB
Avg. Accuracy
0.72
0.72
0.72
0.72
0.70
0.70
0.68
0.69
0.68
0.68
0.64
0.66
0.66
0.66
0.60
0.63
0.64
0.64
0.56
0.60
3
9
11
5
7
3
5
11
3
7
9
Deployed model size [in model units]
Figure 10: Remaining combinations on the (IA)3 PEFT benchmark with TO-3B.
C.2 (IA)3 PEFT BENCHMARK
We extend the results of Figure with four additional algorithms in Figure TO All algorithms demonstrate steep initial rise accuracy, except TIES TIES-MERGING for which the gains appear sharply after a steady initial rise. Notably, at size 7x, all algorithms are within 1% of the fine-tuning accuracy (denoted by D) Therefore, practitioners aiming to deploy all 11 fine-tuned models for high accuracy can be benefit from a reduction of 4 model units without losing too much accuracy.
http:://ggitthbb...mmmmoooorrfre 1m
22

---

Published as a conference paper at ICLR 2026
FlexMerge +
TIES
FlexMerge Cons.
FlexMerge Avg.
FlexMerge + PCB
0.9
Avg. Accuracy
0.87
0.85
0.85
0.85
0.8
0.80
0.80
0.83
0.7
0.75
0.75
0.81
0.6
0.70
0.70
2
4
6
2
3
6
3
4
5
6
2
Deployed model size [in model units]
Figure 11: Remaining combinations the FFT benchmark with T5-Large.
FlexMerge + TIES
FlexMerge + Cons.
FlexMerge Avg.
FlexMerge + PCB
0.85
0.85
0.85
Avg. Accuracy
0.83
0.80
0.80
0.80
0.81
0.75
0.75
0.75
0.79
0.70
0.70
0.70
0.65
0.77
2
4
5
6
7
1
2
3
4
2
3
4
5
6
7
2
6
7
Deployed model size [in model units]
Figure 12: Remaining combinations on the FFT benchmark with T5-Base.
TA
Avg
Consensus
TIES
PCB
EMR
Avg. Accuracy
0.85
0.85
Ave A.c.Aaeuaa
0.80
0.80
0.75
Size 2.25x
Size 3.5x
Figure 13: Results on T5-Large. Algorithms rankings not consistent across sizes. In particular, MER-MEGGING and TIES- MERGING surpass the accuracy of masking masking-based approach CONSENSUS while averaging almost attains the accuracy of TA at size 3.5x.
C.3 HCLL-PARAMETER FINE-TUNING (FFT) BENCHMARK
We extend the results of Figure 4 with four additional merging algorithms two models. Figure I] and Figure 12 chart the results for T5-Large and T5-Base respectively. Just doubling the deployed model size from 1x 2x significantly improves the accuracy across all combinations. We note gains of more than 7%, 12% and 4.3% for SEES-MERGING Averaging and PCB- PCB-MERGING respectively under the T5-Large model. Similarly, we note gains of nearly 9% for both Averaging and PCB- MERGING under the T5-Base model. a sharp rise around size 2x, reaching 80.6% accuracy at just 2.7x from an accuracy 68.7% at size 1x, gain of nearly 12%. Except for FLEXMERGE CONSENSUS under T5-Large which demonstrates a linear trade-off. all other combinations exhibit a highly favorable trade-off, validating the benefits of our approach across diverse scenarios. Lastly, also observe that algorithm rankings remain inconsistent on the FFT benchmark as well when sizes are increased, as illustrated in Figure [3
MERGING under the T5-Base model demonstrates
23

---

Published as a conference paper at ICLR 2026
C.4
MULTI-MODAL BENCHMARK
FlexMerge TA
COCO-Cap. (BLEU4)
(CIDEr)
(ROUGE-L)
(METEOR)
1.5
0.4
0.6
0.3
1.0
Performance metric
0.2
0.5
0.4
0.2
0.0
0.0
0.1
ImageNet (Acc)
NLVR2 (Ace)
COCO-Ret. (Acc)
85
80
75
80
50
75
60
25
3
2
Deployed model size [in model units]
Figure 14: Accuracy-size trade-off for FLEXMERGE + TA on the multi-mode benchmark with BEiT-3-base. The performance metrics for each dataset are shown in parentheses.
FlexMerge + EMR
COCO-Cap. (BLEU4)
(CIDEr)
(ROUGE-L)
(METEOR)
0.40
1.35
0.60
0.31
0.38
1.30
0.59
0.30
Performance metric
0.36
0.58
0.34
1.20
0.57
0.29
ImageNet (Acc)
NLVR2 (Ace)
COCO-Ret, (Acc)
2
3
85.0
85.0
84
82.5
82.5
82
80.0
77.5
80
1
2
4
2
3
2
4
Deployed model size [in model units]
Figure 15: Accuracy-size trade-off for FLEXMERGE + EMM-MERGING on the multi-modal bench- mark with BEiT-3-base. The performance metrics for each dataset are shown in parentheses.
In this section, we present the accuracy-size trade-off for the multi-modal benchmark detailed in and FLEX EMR- MERGING respectively. The performance metrics differ across datasets, therefore show each dataset separately. The first row each figure corresponds to the COCO-Captioning dataset with 4 metrics while the second row corresponds to ImageNet, NLVR2 and COCO-Retriews datasets, each evaluated with accuracy. For all metrics, higher values indicate better performance.
Section B.5 Figures [4] and 15] the results for FLEEXMEGGE.
Across all datasets, observe significant gap between the performance at lowest merged size and the tne-tuned model performance. FLEXMERGE TA shows significant benefit of increased size on the COCO-Captioning dataset where the performance steeply grows across all four metrics between
24

---

Published as a conference paper at ICLR 2026
size 1x and 2x. For ImageNet, NLVR2 and between size 2x and 3x. Interestingly. NLVR2 shows a drop in performance before rising sharply, indicating that merging with related tasks can provide complementary benefits. EMRRMEERON has a lowest size of over 2x due to the cost of storing the pre-trained model and the masks. However, this provides significantly higher starting performance than TA. The performance improvements for FLEXMERGE + EMR-MERGING are similar to FLEXMERGE + TA, where COCO-Captioning demonstrates sharp rise between 2x and 3x while the remaining datasets demonstrate sharp rise between 3x and 4x. Thus, larger sizes can confer beneficial improvements across both algorithms, obviating the need to deploy all fine-tuned models for high performance.
the performance improves sharply
C.5 FLEXMERGE VS CHANNEL MERGING
In this section, we compare FLEXMERGE with CHANNEL MERGING under the TIES-MERGING algorithm, following the same setup as our experiments with TA in Figure 5 As shown in Figure [6] FLEXMERGE consistently achieves signif- icantly higher accuracy than CHANNEL MERGING across all sizes. This is because CHANNEL MERGING enforces fixed number of clusters per block, whereas FLEXMERGE allows vastly different number of clusters per layer by design. This flexibility, enabled by iterative pairwise merging, leads to sub- stantial accuracy improvements. To better understand this effect in FLEXMERGE, we visualize the number of clusters per block in Figure 17] corresponding to merged model sizes of 2. 16x. For clarity, we focus on Attention and MI.P blocks, which con- stitute the bulk the model size, omitting smaller blocks such as layer norms. In Figure 17 we observe that the number of clusters per block varies significantly- from I to whereas CHANNEL MERGING, at similar model size, would enforce exactly 2 clusters per block, limiting its effectiveness.
FlexMerge TIES
Channel Merge + TIES
Avg. Norm. Accuracy
1.00
0.95
0.90
0.85
0.80
3
4
5
Deployed model size [in model units]
Figure 16: FLEXMERGE achieves higher accuracy than CHANNEL MERGING across all sizes.
Size 2.16x
Number of Clusters
O-artn
0-mip
1-attn
1-mip
2-attm
2-mip
3-attn
3-mlp
4-aitn
4-mip
5-atim
5-m
6-attm
6-m
-atin
7-mlp
8-attn
8-mlp
9-attn
9-mlp
10 10-atm
10-mlp
Il -atin
11-mlp
Blocks in Sequential Order
Figure 17: Visualizing the number of clusters per block in FLEXMERGE model of size 2.16x. The number of clusters vary significantly across blocks, ranging from 1 to 4. In contrast, for a model of size 2x, CHANNEL MERGING will have exactly 2 clusters in each block. This flexibility enables greater accuracy for FLEXMERGE over CHANNEL MERGING.
C.6 OUT-OR-DISTRIBUTION (OOD) PERFORMANCE OF FLEXMERGE
To assess the OOD performance of FLEXMERGE, we conducted additional experiments on 6 OOD tasks using models merged from 8 in-domain tasks. For each OOD task, the prediction is obtained by ensembling the outputs of the 8 in-domain branches of the merged model (see Figure Ia).
Setup: In-domain tasks are SVHN, Cars, RESISC45, EuroSAT, SUN397, GTSRB, MNIST and DTD. Oft-of-domman tasks are Weather, PCAM, Flowers 102, Landscapes, Beans and Food101. We consider TIES as the merging algorithm as it known for its good performance OOD.
25

---

Published as a conference paper at ICLR 2026
Results. Table presents the average accuracy of each in-domain expert the 6 OOD tasks. Table 6 presents the average accuracy of FLEXMERGE'S 's merged model for varying sizes on the OOD tasks. We observe that across all model sizes > 1x, FLEXMERGE consistently outperforms the best single in-domain expert on these OOD tasks, suggesting that our method not only preserves but can also improve OOD generalization.
Table 5: Average OOD Accuracy of Individual In-domain Experts. Bold-faced values indicate the best in-domain expert.
Model 4 Datasel +
SVHN
Cars
RESISC45
EuroSAT
SUN397
GTSRB
MNIST
DTD
ViT-B-32
0.5538
0.6230
0.6268
0,5768
0.6366
0.5083
0.5060
0.5910 0.7193
V1T-L-14
0.7259
0.7152
0.6908
0.6797
0.7096
0.7328
0.7298
Table Average OOD Accuracy of FlexMerge Varying Sizes. Bold-faced values indicate higher accuracy than the best in in-domain expert in Table
Model 4 Merged Size +
1x
1.5x
2x
2.5x
3x
4x
6x
8x
VIT-B-32
0.6289
0.6410
0.6475
0.6539
0.6567
0.6586
0.6548
0.6595 0.7488
ViT-I.-14
0.7390
0.7394
0.7398
0.7403
0.7416
0.7453
0.7464
C.7
SCALING LAWS FOR FLEXIBLE MODEL MERGING
8 Tasks
30 Tasks
8 Tasks
30 Tasks
Tasks
30 Tasks
FlexMerge + TA
Avg. Norm. Accuracy
FlexMerge + TIES
Avg. Norm. Accuracy
Avg. Avg.Non Norm. Accurccy
1.00
FlexMerge + Consensus
0.90
0.9
0.80
0.8
0.98
0.70
0.7
0.96
0.60
2M
3M
4M 10
3M
3
4.5M
SM
6M
IO
TO
TO
TO
10
10"
TO
10
Deployed model size [in model units]
Deployed model size [in model units]
Deployed model size [in model units]
Figure 18: Deploying model sizes in proportion to the number of tasks (M) scales better than fixed size models (1x or 3x) as shown.
As the number of tasks increases, maintaining high accuracy in the merged model becomes challeng- ing. While newer methods enhance accuracy scaling, they encounter limitations when the model size fixed. Thanks to flexible model merging, this scaling can be significantly improved the deployed model size chosen in proportion the number of tasks. To illustrate this, we chart in Figure [8] the drop in average normalized accuracy from 8 tasks to 30 tasks, when deploying model of fixed size 1x sizes chosen proportionally to M. Noticeably, the drop becomes smaller as the proportions increase, becoming < 5% at size 4M/10 from > 16.5% at size of 1x for TIES- EEES-MERIINGG..
We observe similar results across other algorithms. For TA at size 1x, the average normalized accuracy drops by 9.2% when scaling from 8 to 30 tasks. With FLEXMERGE, this degradation can be significantly mitigated by adjusting the deployed model size proportion to the number of tasks (M). For instance, setting the deployed model size to 3M/110 reduces the drop to 3. 3,7% and increasing to 4M/10 brings the drop down further to 2.5%. Even for advanced algorithms such as CONSENSUS, fixed-size models can suffer substantial degradation. At fixed size of 3x, the accuracy drop from 8 to 30 tasks for CONSENSUS is 14.5% (beyond the y- y-axis limits of the figure). However, increasing the deployed model size to 4.5M/10 reduces the drop to 2.5%, and to less than 1% at 6M/10. While fixed-size models struggle to scale effectively with the number tasks, our results highlight the need to rethink accuracy malling-advocalle for model sizes that grow proportionally with task count.
26

---

Published as a conference paper at ICLR 2026
Merging Order
Selection strategy
left-right right-left
greedy
random
cosine
FlexMerge Cons.
FlexMerge + EMR
FlexMerge Cons.
Avg. Accuracy
0.90
0.91
0.9
0.89
0.89
0.80
0.8
0.83
0.75
0.87
5
10
1
30
10
20
Deployed model size [in model units]
Figure 19: Greedy emerges the best over left-to-right and right-to-left for both FLEXMERGE CONSENSUS and FLEXMERGE + EMR-MERGING whereas cosine performs better than random.
Selection strategy
random
cosine
FlexMerge + TIES
FlexMerge EMR
Avg. Accuracy
0.90
0.85
0.80
0.89
0.75
0.88
2
4
4
8
Deployed madel size [in model units]
Figure 20: For TIES-MERGING, its trimming component makes similarity comparisons noisy. Hence, random selection performs slightly better than cosine. In contrast, cosine similarity performs better
on CMMR-MERGIN and most other cases (see also Figures and 19).
C.8 ABLATIONS OF THE MERGING PROCEDURE
In this section, we include the remaining ablation results. In particular, we explore the impact of merging order CONSENSUS and -oght and greedy. In the left-to-right order, we execute pairwise iterative merging using cosine similarity in block 1, followed by block block 3 and block are fully merged before carrying out any merging in block i++1. The right-to-left merging does exactly the same, but in reverse order, starting from the last block. contrast both, greedy merging can select any pairs in any block depending upon their cosine similarity, without any restriction on the order. Figure 19 shows the results for FLEXMERGE + CONSENSUS and FLEXMERGE + EMR-ME MERR-GGG Similar to our results in Figure 6 right-to-left performs the worst. As final layers tend to be more specialized, merging them first significantly hurts accuracy. Intuitively, left-to-right merging seems ideal. However, strictly following this order proves too rigid, and more flexible greedy merging approach performs best. We further analyze the ordering greedy in Section C.10
We consider three merging orders: left-to-right,
(see Figure T]. Hence, all task-blocks in
We also explore the impact of random selection over cosine similarity based selection on more merging algorithms, namely TIES- MERGING and CMM-MEERING extending our results in Figure In Figure 19 cosine performs better than random on FLEXMERGE CONSENSUS. However in Figure 20 random selection shows slightly better performance for FLEXMERGE + TIES -EEGGIGG This is likely due to the trim component of TIES- MERGING which makes similarity comparisons noisy. As detailed in Section B.I we retain only the top 10% parameters in the full task vectors before beginning the bottom-up merging process when using
While sparsification enables
27

---

Published as a conference paper at ICLR 2026
better merging due to reduced interference Yadav et al. (2023), it also renders similarity comparisons noisy, leading to a slight drop in performance compared to random selection. With FLEXMERGE + EMR-MERGING random shows high variance, and once again cosine performs better average than random. In summary, cosine is generally superior across most cases in Figures
19 and 20)
C.9 RECONSTRUCTION LATENCY OF MASKING BASED APPROACHES
C.9.1 HOW DO WE MEASURE THE RECONSTRUCTION LATENCY?
Approaches such as CONSENSUS and EMR-MERGING merge into a unified task vector Tum and task-specific masks (m1...., mar). To reconstruct the task-specifi model, they also store 6 pr and reconstruct as follows:
0, 0 pre + Tuni
Under FLEXMERGE, the size of the deployed model varies. Consider for certain task i, two sets Bf and B\B," comprising the set of all blocks that were fused and retained as original respectively. Notice the subscript which indicates that these sets could be different for different tasks depending on how the greedy fusion occurred. For each fused block b E B let T denote the subset of tasks which got fused. Then FLEXMERGE will store the corresponding Thn along with the task specific masks {mi]keT> and 0 Under FLEXMERGE, we then reconstruct follows:
for E BF 6 -0000".6 pre + T m
(3)
And, no reconstruction is required for unfused blocks:
for b E BAB, =0
(4)
The reconstruction latency therefore depends upon the size of B," Ideally, we would measure the
reconstruction latency as the time required to execute Equation (3 averaged across all tasks E [M]. However, this would mean that the reconstructed parameters O, occupy additional storage alongside pre. To restrict this extra storage, we assume that the reconstruction happens in- in-place on pre as follows:
for E Bf 0t, Once the inference for task / completed, the pre-trained parameters are restored back to be ready for the next reconstruction:
pre = pre + Tmi o m
(5)
for b E Bf 0 = Ofe - - - m Thus, we report the reconstruction latency to be the total time required to execute both Equation (5) and Equation (6), averaged across all tasks.
C.9.2 How DOES INCREASING THE DEPLOYED MODEL SIZE LOWER THE RECONSTRUCTION LATENCY?
As noted earlier, the reconstruction latency depends on the size of Bf As the deployed model size progressively increases, more and more blocks move from B," to B B,B,"" and the time to reconstruct consequently reduces. The lowest deployed size i.e., where all blocks are fused, Bf" = B, incurs the highest reconstruction overhead. Conversely, the maximum possible deployed size (of Mx), with B. = b. incurs zero reconstruction latency. Thus by employing larger deployed models generated by FLEXMERGE, practitioners can effectively reduce the reconstruction overhead in time-critical applications.
Results. Figure 21 shows the average reconstruction latency for CONSENSUS, measured on both CPU and GPU. The latency for scalar multiplications in the application of masks. As illustrated in Figure 21 merging into larger models using FLEXMERGE can significantly reduce reconstruction latency. For V on CPU, latency drops from 240 ms to 0 ms, and for ViT-B/32, from 60 ms to () ms, depending on the deployed model size. On GPU, the latency similarly dicreasee-frowd 6 iss to ms for ViT-L/I4 and from 4 ms to ms for ViT-B/32. To put this in perspective, the forward pass takes 12 ms for VIT-L/14 and 7 ms for ViT-B/32. On GPU, reconstruction latency can exceed 50% of the forward pass time, while on CPU, the overhead is several times higher. These reconstruction costs can be significant in time-critical scenarios. By enabling ane-grained control over the deployed model size, FLEXMERGE allows CONSENSUS and EMR-MERGUE to achieve not only higher accuracy but also lower reconstruction latency.
closely matches that of CONSENSUS, as the additional incur negligible overhead: the dominant cost arises from
28

---

Published as a conference paper at ICLR 2026
VIT-B/32
VIT-L/14
(a) GPU
(b) CPU
Mean latency [ms]
300
6
240
180
4
120
2
60
0
2
4
Deployed model size [in model units]
Figure 21: FLEXMERGE enables lowering of the reconstruction latency overheads for approaches such as CONSENSUS Wang et al. (2024) and control over the deployed model size. The default overhead at the lowest size can be significant, exceeding 50% for the forward pass time, which 12 ms for ViT-L/14 and ms for ViT-B/32 on GPU.
Huang al. (2024) by offering flexible
C.10
MERGING ORDER ANALYSIS
Avg. Selection
100
Time Step
50
0
0-aitn
0-mip
I-attn
1-mip
2-attn
2-mlp
3-athn
3-mpp
4-atm
4-mlp
5-attn
5-mlp
6-attn
6-m
7-atth
7-mip
8-atth
8-mlp
9-attm
9-mlp
10-attn
10-mlp
Il 1lattn
l] !1-mlp
Blocks in Sequential Order
Figure 22: Visualizing the order of greedy selection blocks for FLEXMERGE + TA on 8 task vision benchmark with ViT-B/32. We focus only Attention and MLP blocks they share the biggest portion of the size of the model. We observe that the initial and final blocks get selected ahead of the intermediate blocks on average, indicated by the smaller values of their average selection time step.
Figure 22 visualizes the order of greedy block selection for the run corresponding FLEXMERGE + TA on 8 task vision benchmark with ViT-B/32 (shown in Figure 33. We focus only Attention and MLP layers as they share the biggest portion of the size of the model. There are 12 transformer layers in the ViT-B/32 model, resulting in total of 24 blocks that we consider in the sequential order. The number of possible merges total to (M 1) x 24 = 24 = 168, with at most (M - 1) merges possible for each of the 24 blocks. The time steps 167. We report the average selection time step for every block in Figure 22 We observe that the initial and final blocks are, average, selected earlier than the intermediate blocks, indicated by their smaller average selection time step. While the overall trend aligns with the common understanding that earlier layers in neural networks learn general features shared across tasks and later layers capture more task-specific features with lower correlation, we find that the final few blocks may also exhibit higher cosine similarity and be selected early for merging. This flexibility, enabled by FLEXMERGE'S greedy selection process, facilitates the efficient trade-off between model size and accuracy.
assigned in order of selection starting from 0 to
C.11 MAIN RESULTS PRESENTED WITH NORMALIZED ACCURACY
We presented results using average accuracy Figures and For completeness, Figures 23 and 24 show the corresponding results using average normalized accuracy, computed by dividing the
29

---

Published as a conference paper at ICLR 2026
VIT-B/32
VIT-L/14
FlexMerge + TA
FlexMerge TIES
Flex + Cons.
FlexMerge EMR
1.0
1.0
1.00
1.00
Avg. Nomalized Accuracy
0.9
0.98
0.99
0.8
0.8
0.96
0.98
0.97
0.7
0.7
0.94
2
4
6
8
2
4
6
2
6
8
2
4
1.00
1.00
1.00
1.00
0.90
0.90
0.95
0.98
0.80
0.80
0.90
0.70
0.70
0.85
0.96
0.60
0.60
5 10
20
30
1
5
10
20
5
20
30
1.5
10
20
30
Deployed model size [in model units]
Figure 23: Vision benchmark results (Figure 3 shown with average normalized accuracy.
(a) (IA)² PEFT
(b) Full-Parameter FT (FFT)
Flex.Merge + TA
FlexMerge EMR
FlexMerge TA
FlexMerge EMR
Avg. Norm. Accuracy
1.0
1.00
1.0
1.00
0.98
0.9
0.9
0.99
0.96
0.8
0.8
0.94
0.98
3
9
11
2
4
5
2 3
Deployed model size [in model units]
Figure 24: PEFT and FFT benchmark results (Figure shown with average normalized accuracy.
accuracy of the merged model by the fine-tuning accuracy on each task, and then averaged across tasks.
C.12 DATASET-WISE RESULTS
Figures 25 to 27
chart the dataset-wise results for FLEXMERGE + TA, corresponding to the plots in
Figures
and
D COMPUTE RESOURCES
All experiments were conducted on an internal compute cluster comprising 2 AMD EPYC 7543 32-Core 2.8GHz CPU Processor, equipped with x NVIDIA A100 SXM4 80GB GPU. All of our experiments individually use only I out of the 8 GPU units. While the merging itself is efficient, the evaluation of test accuracy consumes bulk of the time and compute. The wall-clock time can range anywhere between 3 h for 8 tasks to up 12 h for 30 tasks under the ViT-L/14 model for evaluating up to 20 merged sizes in the size range [1, 30]. Similarly, the wall clock times range up to 2 for hull-parameter fine-tuning, up to 8 h for PEFT and up 5 h for modli-lodal test evaluations. Across all experiments presented in this article, we estimate the total virtual CPU and GPU time to be approximately 1200 each.
30

---

Published as a conference paper at ICLR 2026
VIT-B/32
VIT-L/14
SUN397
RESISC45
DTD
MNIST
1.00
0.8
0.8
0.9
0.7
0.8
0.95
Accuracy
0.7
GTSRB
SVHN
EUROSAT
CARS
1.0
1.0
0.8
0.9
0.9
0.8
0.8
0.6
0.8
0.6
2
Deployed model size [in model units]
Figure 25: Dataset-wiss results for FLEXMERGE TA on the vision benchmark (Figure
RTE
WIC
CB
STORY CLOZE
0.825
0.7
0.9
0.96
0.800
0.8
0.6
0.95
0.775
0.7
ANLI-RI
WINOGRANDE
COPA
WSC
0.88
0.7
Accuracy
0.7
0.86
0.6
0.84
0.5
0.6
0.5
0.82
ANLI-R3
ANLI-R2
H-SWAG
3
5
0.50
0.50
0.44
0.45
0.45
0.42
3
5
7
9
11
1
3
5
11
5
7
9
11
Deployed model size [in model units]
Figure 26: Dataset-wise results for FLEXMERGE TA on the PEFT benchmark (Figure 4 a)).
E LLM USAGE STATEMENT
We acknowledge the use of LLMs this work, limited to coding assistance, identifying potentially relevant related work, and improving the clarity and grammar of the manuscript. All LLM-generated content was reviewed and verified by the authors.
31

---

Published as a conference paper at ICLR 2026
Base
T5-Large
QUARTZ
STORY CLOZE
WIKI_Q _AK
QASC
0.95
0.8
0.90
0.9
0.7
0.85
0.8
Accuracy
0.6
WINOGRANDE
PAWS
WSC
0.7
0.9
0.6
0.6
0.8
0.4
2
3
4
6
7
4
5
2
3
6
Deployed model size [in model units]
Figure 27: Dataset-wise results for FLEXMERGE TA on the FFT benchmark (Figure 4 (()).
32


Reviews:
[{"review_id": "UPT5nJElpz", "reviewer": "Reviewer_hggn", "summary": "The paper introduces a block-level merging enhancement technique: it separates each expert into block groups and greedily merges the most similar pair within the same block using a given merging algorithm. During inference, it requires knowledge of the task ID and the specific block in which the task’s parameters reside, then routes the input to that block. The authors conduct extensive experiments on models across image domains (e.g., CLIP ViT‑B/32, ViT‑L/14) and PEFT/NLP setups.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- S1: This paper proposes a conceptually novel block-wise merging technique, serving as the interpolation between the single merged model and “retain‑everything”. It can balance performance and memory cost.\n- S2: FlexMerge can plug into existing algorithms (TA, TIES, EMR, Consensus). The authors propose recomputing pairwise similarities and using a DSU structure to speed up the process.\n- S3: The paper tests multiple base models (ViT‑B/32, ViT‑L/14), 8‑task and 30‑task suites, and PEFT scenarios (T0‑3B tasks), and conducts extensive ablation studies.", "weaknesses": "- **W1 (Major):** FlexMerging requires specific task iD. It needs a task ID to route task-specific inputs to the corresponding blocks, which imposes at least two constraints:  \n  - It cannot be applied in scenarios where we do not know which task the input originates from. Requiring a known task ID is unfair when comparing against methods that make no such assumption—e.g., task arithmetic or TIES.  \n  - It cannot handle out-of-domain evaluation, which is very common in practice. The method assumes that an input from task A must be assigned to the merged block containing the expert parameters for task A. If an unknown task B is presented, the system cannot assign it appropriately.\n\n- **W2 (Major):** The computational complexity for $M$ tasks × $B$ blocks (involving pairwise similarity matrix computation and greedy merging steps) is not fully analyzed. For instance, with very large $M$ (e.g., 100+ tasks) and large $B$ (e.g., DeepSeek with 61 layers or Qwen with 94 layers), the merging cost in terms of memory and time remains unknown.\n\n- **W3 (Minor):** The target size appears more empirical and may count different stored artifacts (e.g., masks for Consensus, per-block scalars for EMR). Choosing a size given a memory or accuracy target is heuristic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YYzXsHT4pM", "reviewer": "Reviewer_EyDU", "summary": "The authors propose FlexMerge, a unified framework for data-free, block-level greedy model merging, which enables the combination of multi-task fine-tuned models at arbitrary scales from 1× to M×. The paper systematically demonstrates a favorable trade-off between accuracy and model size across vision, NLP, and multimodal scenarios, showing that scaling from 1× to 2× typically yields substantial accuracy improvements, while the performance approaches the single-task fine-tuning upper bound at scales far below M×, a phenomenon consistently observed across multiple merging algorithms.", "questions": "1, How about the experiments on merging LLMs ?\n\n2, Definition & Construction of Fractional Models：\n\nThe paper introduces fractional model units (e.g., 2.25→), but the mechanism remains unclear.\n\nHow are fractional models constructed at the parameter level?\n\nDoes “fractional” refer to partial layers, partial parameter blocks, or continuous interpolation over task vectors?\n\nHow do you ensure structural consistency and avoid breaking model geometry when only a “fraction” is used?\n\nA more precise definition is needed to understand how fractional capacity is achieved.\n\n3: Merging multiple fine-tuned models is known to cause representational interference. This risk may increase with fractional merging.\n\n4: How do you ensure stability and avoid representation collapse when using fractional model portions?\n\n5: Do fractional models exhibit smooth performance scaling or sudden drops at certain fractional ratios?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1.The paper reframes model merging from a fixed single-target (1×) setup into a controllable continuous spectrum of model scales. This data-free formulation reflects real-world trade-offs between accuracy and resource budgets, turning a theoretical question into a deployable engineering solution.\n\n2.A block-wise greedy strategy decomposes Transformers into Attention, MLP, and other units, enabling fine-grained control over model capacity. The proposed pluggable meta-framework allows the merging primitive  to be instantiated with different data-free algorithms (e.g., TA, TIES, Consensus), demonstrating strong versatility and generality. \n\n3.Experimental results across ViT-B/32 and ViT-L/14 (covering 8 and 30 task settings), as well as T5/T0-3B under both PEFT and FFT configurations and multimodal scenarios, show consistent trends: performance improves sharply from 1× to 2× capacity and then saturates, confirming the framework’s stability and scalability across merging algorithms.", "weaknesses": "1.Although the proposed block-wise greedy merging strategy demonstrates strong empirical performance, the paper offers no theoretical justification or formal analysis (e.g., approximation guarantees or error bounds). Consequently, it remains unclear under what conditions this greedy selection strategy converges to an effective merging configuration, or what its worst-case behavior might be under large inter-task divergence.\n\n2.All experiments were conducted on models sharing identical architectures. The method’s feasibility and performance under heterogeneous or partially aligned architectures (e.g., differing scales or network structures) remain unexplored, despite such configurations being common in real-world multi-source settings. This assumption limits the method’s generalizability and real-world applicability.\n\n3.Although the paper reports partial results on merging time, inference latency, and reconstruction delay, the overall analysis remains insufficient. There is a lack of explicit complexity characterization for the block-level similarity computation and greedy matching procedure, as well as no quantitative comparison of memory and storage costs across different model scales and merging primitives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Xb2KEMU2m", "reviewer": "Reviewer_R12B", "summary": "This paper proposes FlexMerge, a data-free model merging framework enabling flexible merged models across sizes (including non-integers) via greedy block merging. It balances accuracy-storage, unifies merging algorithms. Experiments on vision/NLP (up to 30 tasks) show steep accuracy gains with modest size increases, shifting algorithm rankings, and efficiency.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.The paper is well organized and well written.\n\n2.The authors present a well-motivated approach .\n\n3.It conducts numerous experiments, validates the experimental results on models of various series and sizes, and covers a wide range of evaluation tasks.", "weaknesses": "1. **The necessity of the proposed architecture is not clearly justified.**\n   According to previous findings from *DARE*[1], *Twin-Merge*[2], and *Consensus*[3], only a small portion of task-specific parameters truly contribute to performance, and task information can be effectively compressed through pruning techniques (e.g., SVD compression or masking).\n   Given that the labels of the assumed dataset are known, the authors should compare *FlexMerge* with other methods under the same level of task information compression to demonstrate the superiority of their architecture.\n   For example, the authors could compress the task vectors in *TA* using SVD and dynamically select vectors based on task labels for a fair comparison.\n\n2. **In Figure 4(a), the performance of FlexMerge + TA decreases between 5 and 11 tasks.**\n   What causes this degradation? The paper should provide an explanation or analysis of this phenomenon.\n\n3. **In the OOD setting, task labels are not available.**\n   How does *FlexMerge* decide which branch to activate in this case? The paper should clarify the mechanism used for branch selection under the OOD scenario.\n\n[1]  Language models are super mario: Absorbing abilities from homologous models as a free lunch.\n\n[2] Twin-merging: Dynamic integration of modular expertise in model merging. \n\n[3] Localizing task information for improved model merging and compression.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MgO3Ru3xvy", "reviewer": "Reviewer_jaXQ", "summary": "This paper proposes a data-free model fusion framework called FlexMerge, aimed at addressing the trade-off between accuracy and model size in multi-task model fusion. FlexMerge supports generating fusion models of any size (including non-integer multiples) and can integrate various data-free fusion algorithms within a unified framework. Through experiments, the authors pointed out: moderately increasing the model size can bring significant accuracy improvements, and the performance rankings of different fusion algorithms are inconsistent at different sizes.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1: The author systematically studied the global trade-off between accuracy and size in model fusion for the first time, breaking through the limitations of traditional 'single-model fusion'.\nS2: The paper proposes a data-free framework supporting the fusion of model with non-integer sizes.\nS3: The author reveals that the phenomenon of \"algorithm performance re-ranking under different model sizes.", "weaknesses": "W1: Lack of insight. Although the experiments are thorough, there is a lack of theoretical explanation and further analysis on 'why certain algorithms perform better at larger scales'.\nW2: Lack of in-depth discussion on certain algorithms. Methods such as Consensus and EMR require storing masks or reconstructing them during inference. Although FlexMerge can mitigate this, its additional overhead is not discussed in depth in the paper.\nW3:Experiment: There is a lack of comparison with similar data-free methods on the same task.\nW4: Lack of theoretical analysis. The algorithm uses a greedy strategy to merge blocks, and no explanation of optimality is given.\nW5: Limited external validity. While the experimental suite is thorough within classification and homogeneous Transformer backbones, the evidence may not fully generalize to other settings. It would be helpful to include evaluations on detection/segmentation/generation tasks and on heterogeneous architectures to more convincingly demonstrate robustness across modalities and model families.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)