MASS: MoErging through Adaptive Subspace Selection
Authors: Donato Crisostomi
Alessandro Zirilli
Antonio Andrea Gargiulo
Maria Sofia Bucarelli
Simone Scardapane
Fabrizio Silvestri
Iacopo Masi
Emanuele Rodolà
Venue: ICLR 2026 Poster (2026)
Keywords: Model Merging
MoErging
Task Vectors
Multi-Task Learning
Deep Learning
Computer Vision
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Model merging has recently emerged as a lightweight alternative to ensembling, combining multiple fine-tuned models into a single set of parameters with no additional training overhead. Yet, existing merging methods fall short of matching the full accuracy of separately fine-tuned endpoints. We present MASS (MoErging through Adaptive Subspace Selection), a new approach that closes this gap by unifying multiple fine-tuned models while retaining near state-of-the-art performance across tasks. Building on the low-rank decomposition of per-task updates, MASS stores only the most salient singular components for each task and merges them into a shared model. At inference time, a non-parametric, data-free router identifies which subspace (or combination thereof) best explains an input's intermediate features and activates the corresponding task-specific block. This procedure is fully training-free and introduces only a two-pass inference overhead plus a ~2 storage factor compared to a single pretrained model, irrespective of the number of tasks. We evaluate MASS on CLIP-based image classification using ViT-B-16, ViT-B-32 and ViT-L-14 for benchmarks of 8, 14 and 20 tasks respectively, establishing a new state-of-the-art. Most notably, MASS recovers up to ~98% of the average accuracy of individual fine-tuned models, making it a practical alternative to ensembling at a fraction of the storage cost.

Published as a conference paper at ICLR 2026
MASS: MOERGING THROUGH ADAPTIVE SUBSPACE SELECTION
Donato Crisostomi* Alessandro Zirilli*!
Antonio Andrea Gargiulo'
Maria Sofia Bucarelli
Simone Scardapane
Fabrizio Silvestri' lacopo Masi!
Emanuele Rodola
Sapienza University of Rome
Paradigga *CNRS $i3s
(cri sostomi, zirilli, gargiulo, masi, rodola]adi aabola)adi.uiiroeee unwiraal...l.... it
momoe.. wmwoeescardaaaaeeec.m.. scardapane@uni romal. .it fbucarelli, fai lvestri Jēdi ag uni romal it
ABSTRACT
Model merging has recently emerged as a lightweight alternative to ensembling, combining multiple fine-tuned models into a single set of parameters with no ad- ditional training overhead. Yet, existing merging methods fall short of matching the full accuracy of separately fine-tuned endpoints. We present MASS (MoErg- ing through Aduptive Subspace Selection), new approach that closes this gap by unifying multiple fine-tuned models while retaining near lfte-of-theear perfor- mance across tasks. Building the low-rank decomposition of per-task updates, MASS stores only the most salient singular components for each task and merges them into shared model. At inference time, non- non-parametric, data-free router identifies which subspace (or combination thereof) best explains an input's inter- mediate features and activates the corresponding task-specific block. This pro- cedure is fully training- ffein and introduces only two-pass inference overhead plus a ~2x storage factor compared
single pretrained model, irrespective of the number of tasks. We evaluate MASS on open-vocabulary image classification using ViT-B-166 6, ViT-B-32 and ViT-I-14 for benchmarks of 8, 14 and 20 tasks respectively, establishing a new state-of-the-arr, Most notably, MASS recov- ers up to ~ 98% of the average accuracy individual fine-tuned models, making it practical alternative to ensembling at fraction of the storage cost.
https//gthhb.com/rrrosssom//a/
INTRODUCTION
In the early days of deep learning, the default practice train models entirely from scratch. With the rise of massive pretrained networks, research pivoted toward fine-tuning these backbones for specialized tasks (Devlin al., 2019; Tan al., 2018; Yosinski et al., 2014; Hu al., 2022: Radford et al., 2021). Nowadays, with the abundance of fine-tuned models on platforms like HuggingFace' we are witnessing shift toward no-tuning methods that leverage both pretrained foundations and diverse fine-tuned endpoints. Among these, model merging (Singh & Jaggi, 2020; Ainsworth et al., 2023: Ilharco al., 2023) has gained significant attention by combining multiple fine-tuned models into a single parameter set, eliminating the need for additional training or data.
important application of model merging is the combination of models fine-tuned on different tasks that share the same pretrained backbone. Early approaches such as Task Ar i thmet ic (Ilharco et al., 2023) and its extensions (Yadav et al., 2023; Yu et al., 2024; Daheim et al.; Wang et al., 2024; Ortiz-Jiménez et al., 2023; Huang et al., 2024; Akiba et al., 2025) define task vector as the difference between pretrained and fine-tuned weights, and build a multitask model by sum- ming these vectors to the base model. More recent work (Gargiulo al., 2025; Daniel et al., 2025) demonstrates that preserving the layer-wise structure of these updates leads to stronger results. In- stead of flattening the updates into vectors, methods such Task Singular Vectors (TSV) (Gargiulo et al., 2025) exploit their matrix form and uncover a clear low-rank structure, where only small number of singular vectors per task are needed recover most of the fine-tuned accuracy.
https:/huggiingfacccoo http://wwg.g.aaaoooooooooccchaae

---

Published as a conference paper at ICLR 2026
i1i
9 (X)
A BC
ABC
ABC
BCC
B
a
8t
Fabx)
Ott a
Pnc
Osce
8 MASS
Onc
Be(K)
BMT
pro
pre
Fine-tuning
Model Merging
MASS (ours)
Figure I: (left) Fine-tuning yields three separate models on tasks A, B and C. (middle) Model merging produces a single model incorporating the task vectors using a constant function of the input. (right) MASS stores the pretrained model Ore and the task singular vectors V across tasks. At test time, MASS adaptively performs merging using a routing mechanism that chooses appropriate task vectors for the input x, using thresholded gating function g(x). The gate is the residual between the activations of x and their projections onto the span of the right singular vectors Vi.
Despite the progress in performance, most merging methods remain limited by an unrealistic as- sumption: that the task identity is known inference time. We challenge this assumption im- practical and argue that under such setting, the optimal strategy would be compression. Indeed, the low-rank compression method TSV-C (Gargiulo al., 2025) achieves 99.5% normalized accuracy across all benchmarks while requiring only 2x storage and no additional inference overhead, effec- Exxture-of-Expert Merging (MoFrging) (Yadav et al., 2024) methods
tively solving the problem. such as SMILE (Tang et al., 2024a), WeMoE (Tang al., 2024b), and TwinMerging (Lu et al., 2024) go a step further by incorporating router into the merging process.
However, these methods still assume that the correct classification head is provided at test time, thereby inheriting the same unrealistic constraint and failing to exploit the full potential of routing- based approaches.
Key assumptions. In this work we assume that the task is not known at inference time, and that both the most suitable encoder subspaces and the corresponding classification head must be determined automatically. This setting is simultaneously more challenging and more realistic, enabling a single generalist model to handle all tasks encountered during fine-tuning without external supervision.
Contributions. We propose MASS, a novel MoErging method that dynamically selects the most
relevant tasks and corresponding label spaces via adaptive routing mechanism that conditions the merging process on the input itself (Fig. 1). MASS sidesteps the data and training required to train a router by leveraging novel weight-space router that identifies the most relevant task subspaces as defined by their task singular vectors. By selectively integrating these subspaces into the pretrained backbone and extending the routing decisions to the classification heads, MASS enables training- free and data-free merging process without relying on oracle knowledge of the task at hand.
We evaluate MASS on ViT-B-32, ViT-B-16, and ViT-L-14 across up to 20 vision tasks and 8 language tasks, demonstrating substantial gains over existing methods. For modest increase in computational cost (~ 2x forward passes) and storage (~ 2x parameter footprint regardless of the number of tasks), our method achieves up to 98% of the average accuracy of individual fine-tuned models while handling the full union of expert label spaces without oracle guidance at test time.
Wrapping up, our contribution is three-fold:
We introduce MASS. MoErging method that augments singular-vector-basse merging with adaptive routing, eliminating the need for test-time task knowledge.
We design projection- based router that operates without task data or additional tuning, making directly applicable to the merging setting.
2

---

Published as a conference paper at ICLR 2026
We present extensive experiments that establish new ofate-of-the-art results across bench- marks and further provide an interpretation of task singular vectors in text space.
We release our code, checkpoints, and all relevant logs for research purposes.
BACKGROUND
Task Vectors Task Arithmetic (TA) (Iharco et al., 2023) defines task vectors capturing the specific differences in model weights for individual tasks. Formally, the weights brr of a multi-task model for T tasks are computed by aggregating the task-specific weight differences, task vectors, OMT pre + a Di=1 Ta where Ppee is the set of pretrained model weights, or is scaling factor, and T; = 8R, Ppe is the task vector for task i, with On, being the fine-tuned weights for the task. Following Gargiulo et al. (2025), however, we consider these operations at the layer level. From a layer-wise perspective, task arithmetic can be rewritten as OuT off + a Ei ( where olie encodes the pretrained weights for layer E, and A 0 ofc2 the task-specific weight difference for task at layer When layer l has a matrix structure, corresponding A(9 is called the per-layer task matrix for task i. The layer index will be omitted for brevity.
Task Singular Vectors Given a task i, Gargiulo et al, (2025) consider the SVD of the correspond- ing task matrices A, generic layer A, V, They then perform low-rank approxima- tion the As and orthogonalize their singular vectors to reduce inter-task interference. In practice, this is equivalent to summing the top-k rank-one matrices for cach task, with an added orthogo- nalization step to prevent singular vectors belonging to different tasks from interfering. The full procedure is detailed in Algorithm (lines 10-20).
APPROACH
Our approach is best understood as pre-processing step followed by inference-time step. The former consists of a one-time merging procedure to obtain an encoder model 9Mr. as detailed in Algorithm 2. We refer to this the 'fixed' merging step, it is performed only once and mains independent of the input. During inference, ONT is used in a dynamic process, outlined in Algorithm 1, consisting of 4 steps:
(i) First pass: forward the input through bMr and extract its embedding z at a chosen layer E (ii) Routing: project zę onto the task subspaces, selecting those having lowest projection error;
(iii)
Adaptive merge: merge the selected task subspaces into Anta:
(iv) Second pass: classify the input using the final merged model OMr Opee a
3.1 FIXED MERGING
We begin with one-time merging step to produce model capable of task discrimination. This model provides the intermediate activations used by the router the first pass. For this, we use TSV-M (Gargiulo et al., 2025) due to its aubspace-aware aggregation. Although one could alterna- tively rely on the pretrained base, Table 3 shows that it leads to lower task classification accuracy.
3.2 INTEGRATING ROUTING
We extend the aggregation step in Section 2 to include routing mechanism. Given an input x, the merged model can be adaptively determined by:
T
& Σ ohuiofT
ONT = Opre + a Σ
1 (x)( Opee + a
Σ 1l (x)
(I)
J=1
where (x) a per-task gating function that adaptively selects which task subspaces to activate, and subsequently merge, depending on the input at hand.
3

---

Published as a conference paper at ICLR 2026
Algorithm Adaptive Merging Step
Require: Pretrained model weights Opee task-specific updates {A:) fixed merged model OMT,
top-k parameter k, threshold 77, task-specific classification heads {h+) sample
Ensure: Predicted class c*
1: Ze f Forwarddasss(Oma) x)
first pass
2: for i 1,.... do
3:
r: - Ze V; V. zell2
residual as per Section 3.2.1
4: end for
5: w <- softmax (-r)
6: Ω {i wy 77}
# Select tasks above threshold
7: Ω < (f2 w. k)
# Keep only top-k weighted tasks
8: Merge selected subspaces:
Aada Divnn Uj 2n V OMASS f- pre + a adda
9: Compute adaptive model:
Classification procedure
11: Z1.1 f- ForwardPass. MMASS x)
# Compute shared representation Evaluate each head # Highest logit across heads
2i 4- (2.
13: (i",") & arg max [][c]
(L...)(
14: return c*
Traditionally, however, routers require either task-specific data for non-parametric procedures such as nearest-neighbon based routing, or both data and training for parametric routers. This contrasts with the typical merging scenario, which does not assume access to the data used to train the endpoint models, as these could, in principle, be downloaded from any public model repository. We therefore introduce completely data- and tuning-free approach.
3.2.1 PROJECTION-BASEE BASED ROUTING
Given an input intermediate representation z, for prede- termined layer f, we want to identify which task subspace (or set of subspaces) is the most relevant. Concretely, one way to do this is to compute the Euclidean residual of Ze after projecting onto span(V,
2
V2
ZE
y
Zéproj V
Proj pen (ze)
ri
Figure 2: Projection of the activations Ze onto the span of TSVs V1, V2.
where Projyro (ze) vi (1) is the orthogonal
projection of 3e onto span(V ((0)) which yields the minimum-error (L2-optimal) reconstruction within that subspace (see Proposition D. At this point, the additive inverse of the residual vector r ro....,yy rr is normalized through a softmax to obtain the coefficients. The router then picks those exceeding a predetermined threshold n). limiting the selection to the top-k when more tasks surpass it. For details regarding the choice of the layer used to compute the residual, see Section 4.2.
3.2.2 ACCOUNTING FOR REDUNDANT DIRECTIONS
For projection-based routing to be effective, no task should overshadow the others. Consider, for example, three tasks: MNIST Ø. EMNIST 9, and KMNIST E. Being trained on very similar datasets covering the same classes, AMN and AEMN share large portion of their right-singular directions: span (v(() se span (V while KMNIST has some distinct directions (1( captur- ing more Japanese kana-like shapes. However, all three tasks may agree certain generic "black background, white glyph" features. Because MNIST and EMNIST partially reinforce these direc- tions (they both include them), the union of their subspaces can appear "wider" more dominant in that region of feature space. Consequently, for many test samples 2₃ with black backgrounds and centered shapes:
Proj VeNUVEN (z+)ll2 z, Proj Vie (zz)ll2-
Hence, the router sees a smaller residual for MNIST/EMNISTT ST, declaring those tasks more suitable even if the glyph belongs to KMNIST.

---

Published as a conference paper at ICLR 2026
During the fixed merging step, instead of aggregating all task matrices, we only keep those that sufficiently distinct. We first select a single task matrix as the imitial element of the merge set. Then, for each remaining task matrix, we determine whether to include it based its similarity to the matrices already in the set. A task matrix is added only its similarity to all previously accepted matrices remains below predefined threshold.
Formally, let (A......, Ao. } be the set of accepted updates at a given layer. When evaluating a new task update A., we flatten it d; vec(A;) and compute its cosine similarity sim(6;, sam. with each previously accepted update 6Rm-1 If
max sim(6,, 6am.) > E, IsmSr
where e is uper-spe threshold (e.g. E 0.3), we discard A; and do not merge it; else accept it. This ensures that highly similar task subspaces do not overshadow less common ones. This procedure is performed before merging, see lines 2-8 of Algorithm 2.
3.3 ADAPTIVE MERGING AND INFERENCE
With the router having selected a subset Ω of relevant tasks, we merge their subspaces via TSV-M (Gargiulo et al., 2025) into single model MASSS Crucially, unlike standard model merging, which assumes an oracle provides the correct task head at inference, we do not know the task in advance. Instead, after obtaining the shared representation ZL-1 Rdd from fASSS, we run each head h2
Rd + TRC, for every selected task E Ω:
(2I...)), 2, E RC:
We then pick the highest logit among all heads in Ω. Formally:
arg max ci',c)) (i.c) EΩx (1....C.)
2/[c].
In other words, we identify which head i* is most "confident" and select its predicted class e*. This procedure covers the unknown-taat ttsk scenario, allowing the model to determine both head and label space on a per-input basis.
3.4 RESIDUAL MINIMIZATION AS MAXIMUM A POSTERIORI ESTIMATION
The task selection process in MASS can viewed as a maximum a posteriori (MAP) estimation problem. If residuals follow an isotropic Gaussian, the likelihood of a feature vector given task decays exponentially with its squared l reconstruction error. Thus, choosing the task with min- imal residual is equivalent the MAP estimate. This view parallels probabilistic PCA (Tipping & Bishop, 1999), where squared reconstruction error minimization corresponds to maximum like- lihood estimation under the same Gaussian assumption. Residual-based selection is therefore sta- tistically optimal under simple, lcast-informative model that is particularly apt for MASS, which lacks training data to fit more complex distributions. The isotropic prior treats all directions equally, avoiding bias toward specific tasks.
Proposition 3.1. Let Ze E RRd be a feature vector, and for each task i, decompose it as
2, = V.V, Ze 6i E; (1 24.
Assume E, o²) and uniform prior over tasks: p(task i) 1/ for E f1,2,... K}..
Then the maximum posteriori estimate of the task reduces to
MAP arg max pltask il (x)) arg min ||e:
4 EXPERIMENTS
4.1 MERGING PERFORMANCE
Models, baselines and datasets We conduct our experiments on three versions of the CLIP (Rad- ford al., 2021) model, each equipped with different ViT (Dosovitskiy et al., 2021) visual
5

---

Published as a conference paper at ICLR 2026
ViT---32
VLT-B-16
VaU-1-14
Method
tooks
14 tasks
20
tacks
tasks
tasks
tasks
14 tacks
20 tasks
Bas
Zesoshot
48. (p4.n)
56.9
57 (y.5
61.9ga7m)
64 (1.9/
Pinetuned
90.3
89.0
92. (000
91.3kg00
91. 91.9
p00
94.0g00
Neight geight Averaging
67. (74.)))))
63.5)ra.01
64 A1/2
70.8pm
80 50...
78. B.a
Task Arit Arithmetio Ao
68.5 vi
64.0g72 9)
64.0gt
70.0
70.8gy7
69.0 09.0pps.0
84.4 4)
80.4 at
76.9(a1.
Fhod
Consensus TA
72. 61
70.3778.0
68 (68.5)
76.9(a1.7)
74.9ka..1
72. pps.0
85. Spp..
82 Opst
58.9
75V-M
20.2
78.6yss.o
85, 85.5yoa
81.4
78.8 (,s.5
91. ppo.)
88.8
87.5
Iso-C
82 (y1.v)
78.4gs..0)
73. 23.2
87.5 4)
79.8
75.8 75.9
92 92.0pe.2)
89 89.6
80.8
T80-CTS
92.0gsnon
80,6p90.4
77.0
88-7
84.1 coa
(077
92 (pa.t)
(07.4)
NeMot
86 86.8,mr.)
74.3ç22 s)
68 (p.22)
88.1gw
76.6 my
65, (ppus)
88. qu12)
72.3 at
65.0gs 9
50005
SMILE-1
s0.2
75.4
12.8
87.8
$1.7(sa.5)
79.5
91. 2p.2))
96.6
84.9
SHILE-2
94.4ys.
76.4(aa.1)
p4.1
80.0gm0))
82.7
80.4pn.0
02.0pma)
97.1
MASS
87.0
82 2)
90.6
87.8 87.8
81. Lyss.
92.9
90. 90.9
90 90.8,aa 8,
Table 1: Average absolute accuracy results on model merging benchmarks; subscript (in parenthe- ses) is the normalized average accuracy.
Method
CoLA
MNLI
MRPC
ONLI
QQP
RTE
SST-2
STS-B
Avg.
uine
83.4000)
(7.5
91,5
85.4moo
85.9 (5.9(
93.6(u00)
88.7
86, 86.4/100
Hell ggh Averag! ng
69.1 (9.1)
62.6(ya.1)
p9.4))
89.8
83.9
91.7
73.2
78. (91.))
Arithmet
(0.5
(7.8
78 4.....)))
92.5
(8.9
Ti Meeg-Mnggingg
y0.3
65.0
78. (00.
90. (:0..)
83.5
91.7
78. (8.3
79.9/ (2.55
Welooll
(2.5
799 79.0)04.77
51. (3..)
89. 19.3ggm (orr.)
699 6ojas
81. (pp
88 (.s..
82. 1 s:
76.8
SMILE-1
72.0ga..o)
84 2001.00
84.3 (..3po.))
91. 91.3
84.7 84.7
84, (pr.9)
93.3
87 1y
85.
SMILE-2
73,2,99 4)
84.2 84.2 m
85.0
91.3 w)
84.9 4)
(4.8
93.5 03
87.3gi 4t
85.5
MASS
74.1.0m 8)
83.2 5)
85.8
90.9 (0.9/...00
85.1 4)
$4.9 (s 5)
0402
(8.9
85.9 v..))
Table 2: Accuracy when merging fine-tuned models on the GLUE (Wang et al.) benchmark; Normalized average accuracy in subscript.
encoder: ViT-R-32, ViT-B-16, and ViT-I-14. As baselines, we compare against multiple training-free merging strategies, notably weight averaging, Task Arithmetic e (Ilharco et al., 2023), and Consensus Merging (Wang et al., 2024). For additional context, zero-shot perfor- mance serves null reference point, and the mean accuracy of individually fine-tuned models marks the upper bound on achievable performance. We evaluate on three collections of tasks, con- taining 8, 14, and 20 tasks, respectively. The latter is the most extensive setup considered in (Wang et al., 2024; Gargiulo et al., 2025; Daniel et al., 2025). We refer to Section B.3 for the specifics. We quantify results using both average absolute accuracy and average normalized accuracy.
MoErging results Table reports the average absolute accuracy and the corresponding normalized accuracy (sub- script, also in percentage) for each method, model size, and number of vision tasks. To adapt SMILE and WeMoE to our general setting, we employ a majority-voting based heuris- tic to route the classification head. Details in $B.6.
We first note that MASS sets a new state of the art for Mo- Erging across 8 out of benchmarks, with gains as high as ss 6% with respect to the best performing baseline. Fig- ure shows per-dataset breakdown of the results, indi- cating that the accuracy is consistently high throughout all the datasets and not skewed on a subset of easier ones. In terms of storage, MASS requires constant 2x parame- ter increase, whereas other MoErging baselines range from 2.5x up ~ 14x. A breakdown is provided in $B.1.
Ditudiliiiy
Pizesor
WEDALE
34AS5
Figure 3: Per-dataset accuracies.
We also report the accuracies obtained by fixed merging methods, f.e. those not employing a router. These are evaluated with the ground truth classification head, i.e. assuming the task is known at

---

Published as a conference paper at ICLR 2026
(a) Averaged across tasks for two models.
(b) Focusing on SUN397 and STL10 for a V
Figure 4: Per-layer task accuracies for ViT-B-3- 32 on the 20-task benchmark. Layers starting with 'A' indicate attention layers, while those starting with 'M' refer to MLPs.
inference time. While our evaluation setting is significantly more challenging, MASS still outper- forms these ones by consistent margin. In particular, when comparing with the TSV-M baseline it builds upon, we see that routing produces a R 5% increase in accuracy.
We further compare MASS with TwinMerging (Lu et al., 2024) on the 8-task benchmark us- ing 12, the only vision setting with reported results in their paper. Despite the lack of evaluation code for other setups, we find that MASS achieves 97.6% % normalized accuracy in this shared setting, exceeding the 95.3% reported for TwinMerging. Notably, the latter assumes or- acle knowledge of the correct classification head, whereas MASS selects both the merged experts and the output space at inference time. Despite operating in a more general setting, MASS still reaches a higher accuracy. We also report Table 2 the results on merging F1an-T5- Base models (Chung et al., 2024) finetuned the language tasks in GLUE (Wang al.). MASS still yields the best results, showing its benefits to be modality agnostic. MASS obtains the highest ab- solute accuracy on 5 out of the 9 GLUE tasks (CoLA, MRPC, QQP, RTE, SST-2, STS-B), while performing slightly below the best baseline on the remaining ones (MNLL, QNLI). We observe that the two tasks where MASS does not achieve the top score are the NLI-style benchmarks (MNLI and QNLI). This is consistent with these tasks requiring more broad semantic reasoning rather than localized feature shifts, possibly resulting in more diffuse and higher-rank task vectors compared to other GLUE tasks.
Batched inference In the main experiments, we evaluated our method in the most challenging scenario, where each sample may belong to a different task, foreing the router to operate per input. Yet, in many practical settings (e.g., batched requests from the same domain), several inputs share the same task. In such cases, can router-select a single merged model once per batch, clos- ing the gap almost entirely: as shown in Figure 5, our approach achieves mean normalized accuracy of at least 97% in 8 out of 9 settings, effectively matching individually fine-tuned mod- els. This indicates that while our per-sample approach is already effective, batching can further reduce overhead and significantly boost accuracy.
05.75
1-:0-
12
Nammllaani accuraty
Figure 5: Batched accuracy.
Taa by wnder Lade
Scaling the number of tasks We now study how MASS scales as the number of tasks increases. For this, we incor- porate 13 additional datasets and evaluate all merges from 2 up to 33 tasks. Fig. 6 shows that MASS maintains accuracy even on the largest task set. Interestingly, the accuracy is not monotonic function of the number of tasks. In several regions the performance even increases when adding more tasks, sug- gesting that scalability is driven more by the composition of the task set than by its cardinality alone. Overall, MASS main- tains high accuracy throughout the entire range of tasks, in clear contrast to the steep degradation observed with TA.
KARS (ymn)
Figure 6: Merging accuracy when scaling the number of tasks.
7

---

Published as a conference paper at ICLR 2026
4.2 CHOOSING A ROUTING LAYER
We now investigate the effect of routing layer selection on task accuracy. Figure 4a shows the task prediction accuracy obtained by routing at different layers for ViT-B-32 and ViT-B-16 architectures. Interestingly, the best layer is consistent across architectures, with ViT-B-32 and ViT-B-16 both achieving peak performance at layer 9, and MI.P layers exhibiting slightly better performance than the self-attention layers. However, it is immediately clear that the best layer varies significantly across tasks. fact, some layers exhibit a variance of up to 40% in accuracy across tasks. This can be better appreciated by looking at Figure 4b, where can see the per-layer task accuracies on STL10 and SUN397 for a WiT-B-3- 32. While both tasks can be accurately predicted with the best chosen layer l = 9, the former shows marked improvement in accuracy when routing at earlier layers = 3, 4, 5, while the latter benefits from routing at later layers & = 9, 10, 11 and results in poor performance in the earlier ones. This suggests that the best routing layer is task- dependent, and may encourage further research on adaptive ways to determine it. We report an analogous analysis for the ViT-L-14 model in the appendix.
4.3 COMPARISON WITH OTHER ROUTERS
We compare our router with two alternatives that differently balance accuracy, data, and compute.
Nearest Neighbor (nr.) first constructs a small support set of representative examples from each task's validation data. For each test sample, compare its intermediate representation z, to the stored representations of all support examples. Formally, we compute the cosine similarity to each support sample and select the nearest neighbor among all tasks, inferring the task identity from the task to which the support sample belongs. This method requires no additional parameters but assumes access to (and storage for) a curated batch of validation embeddings per task.
The second approach (mlp) fits an MLP over the union of validation sets from all tasks, with each example labeled by its originating task. After extracting zg, we train an MLP fo to predict the task via cross-entropy loss. Full architectural details are in the supplementary materials.
Results Table shows the average normalized accuracy obtained by MASS when varying routing strategy. The nn approach, which stores and compares with a small validation set from each task, gen- erally performs well but remains slightly below our best results. The MLP router achieves the highest accuracy overall, but it relies on a labeled validation set for training-a requirement that contradicts the core premise of merging methods, where access to original task data is often unavailable. This dependency makes the MLP less broadly applicable in realistic scenarios, such when downloading fine- tuned models from public hubs without any accompanying data.
NASS
VLT-I-14
tisks
20
nn
94.0
92.1
mip
98.9
99.5
98.3
projeu
99.1
97.7
91.9
pro
88.6
97.3
96.6
Table 3: Average normalized accuracy for different routers.
Focusing the projection- based router (proj), observe that starting from the TSV-M model (projişu-w) outperforms routing from the pretrained backbone (rro)ra=) when scaling the number of tasks to 20. This is further confirmed in Table 15, where the gap widens to x 10% accuracy for a ViT-B-32 32. This gap underscores the core key insight behind our method: TSV-M arranges each task's 's top singular vectors into distinct, orthogonal subspaces, all contained in the final merged model. Therefore, proirsv н only needs to measure how well each subspace reconstructs the activations, "finding back" the subspace that was originally embedded for each task. In other words, once TSV-M has embedded each task's directions into a single model, a projection is enough to pinpoint the correct subspace, requiring no labels or additional training.
4.4 INTERPRETING TASK SINGULAR VECTORS
Finally, we attempt to interpret the task singular vectors that are used for routing in MASS. Notably, prior work suggests that mid-to-late layer embeddings in CLIP-like models preserve semantic con- tent, implying that routing at this layer hinges on meaningful features (Gandelsman et al., 2024). To validate this, we interpret TSVs using the TEXTSPAN algorithm (Gandelsman et al., 2024; Basile et al., 2024). The latter iteratively identifies and removes the most influential text directions, reveal- ing which concepts best explain how the model's weight changes affect its representation. Impor-
8

---

Published as a conference paper at ICLR 2026
4-1 4, L21: IMAGE OF CAR
L214 CLOSE-UP OF TEXTURED SYNTHETIC MESH
B-32, L10: TMAGE
B-32, LIO: CLOSE-UP TEXTURED MESH
B-32, L3: AFRIAL VIEW OF A HAMLET
B-32, L33: TMAGE WITH A PENGUIN
(a) Cars
(b) DTD
L 14, L21: AN IMAGE OF THE NUMBER
14, L21: ABRIAL VIEW OF AN AGRICULTURAL RILLD
B-32, LIO: AN IMAGE OF THE NUMBER 8
B- B-3,,LO0 LIO: AERIAL VIEW OF FARMLAND
B-32, L3: PHOTO TAKEN SANTORINT
B-32, RURAL WINDMILL SILHOUBTTE
4
9
(c) MNIST
(d) EurosAT
Figure 7: Captions obtained by decoding task singular vectors as text as described in Section 4.4, accompanied by task representative images. Captions produced by the task singular vectors of pre- dictive layers reflect the task content, those obtained by non- non- non-predictive ones do not.
lantly, while TEXTSPAN was originally developed to analyze image embeddings, we employ it here to interpret singular vectors derived from weight updates.
Our results, illustrated in Figure 7, confirm that TSVs capture meaningful and interpretable visual- language associations. For example, the singular vectors associated with the Cars dataset (Krause al., 2013) strongly activate the description "IMAGE OF CAR", whereas those for the DTD dataset (Cimpoi et al., 2014) align closely with the phrase "CLOSE-UP OF A TEXTURED MESH"" Interest- ingly, across architectures (ViT-L-14, ViT-B-32), we find consistent textual concepts emerg- ing from similar singular vectors for the same tasks. For instance, the singular vectors for both ViT-L-14 at layer 21 and ViT-B-32 at layer 10 align with almost equal phrases. This con- sistency hints at the intriguing possibility of transferring semantic information across architectures using text embeddings as an interpretable common ground.
We further note that the interpretabiliiyy of these embeddings strongly depends the chosen routing layer. While mid-to-latt layers (e.g., layer 10 in ViT-B-32 and layer 21 in ViT-L-14) yield se- mantically meaningful descriptions aligned closely with each dataset's visual domain, earlier layers (e.g., layer in ViT-B-32) B2) produce irrelevant or misleading concepts such as "IMAGE WITH A PENGUIN" for the texture dataset (DTD) or "PHOTO TAKEN IN SANTORINI" for digit classification (MNIST). Such discrepancies suggest that early layers encode generic or low-level features, while mid-to-late layers become progressively specialized toward domain-specins -specffc semantic structures.
Overall, these analyses validate our routing strategy: the task singular vectors at mid-layer embed- dings capture precisely the semantic differences the router exploits to discriminate tasks. This also points to a deeper insight: the singular vectors derived from weight updates mirror the semantic structure of the data itself, highlighting a direct connection between weights and data.
RELATED WORK
Model Merging has emerged as lightweight alternative ensembling. Early work focused on aligning models trained from scratch with different random seeds by solving permutation and mode- connectivity issues (Frankle et al., 2020; Entezari al., 2022; Ainsworth et al., 2023; Crisostomi et al., 2025). More recent approaches instead merge multiple fine-tuned models derived from common backbone (Ilharco et al., 2023; Yadav et al., 2023; Yu et al., 2024; Wang et al., 2024; Gargiulo et al., 2025; Daniel et al., 2025). These methods typically treat task updates vectors and combine them through rescaling, pruning, or averaging, while newer work shows that respecting the layer-wise structure of updates significantly improves results (Gargiulo et al., 2025; Daniel et al., 2025). Our method builds directly on this line by adding adaptivity through input-driven routing, narrowing the performance gap between merged and fine-tuned models.

---

Published as a conference paper at ICLR 2026
MoErging. A parallel line of work incorporates routing into merging, often referred as "Mo- Erging" (He et al., 2023; Yadav et al., 2024; Tang et al., 2024a; Lu et al., 2024). In the LLM domain, routers are trained to select among LoRAs or adapters (Jang et al., 2023; Chronopoulou et al., 2023; Muqeeth et al., 2024), sometimes requiring additional supervision or fine-tuning. Closer to our setting, Trans fo roree? (Sun et al., 2025) introduces two-step routing pipeline but relies modified fine-tuning procedure, while SMILE (Tang et al., 2024a) proposes data- and training- free approach that leaves the pretrained backbone unchanged and adds task-specific low-rank updates at inference through layer-wise routing. In contrast, MASS embeds all updates into a single model, per- forms task selection once across layers, and deactivates irrelevant subspaces via a projection- baased router. Finally, Twi wwinMMrging ng (Lu et al., 2024) requires per-task labels to train its router, whereas MASS is fully training- and data-free. We defer further discussion of related work to Section
6 CONCLUSIONS
In this paper, we introduced MASS (MoErging through Adaptive Subspace Selection), method that aggregates low-rank task updates with adaptive routing, jointly selecting both encoder subspaces and classification heads without test-time supervision. To address the absence of per-task data in realistic merging scenarios, we designed a projection-based router that is entirely training- and data-free.
Our experiments show that MASS achieves state-of-the-arr performance, recovering nearly the full accuracy of fine-tuned experts at a fraction of their combined storage cost. Future work includes refining the router for more precise subspace selection and extending MASS to -utto--dist settings, where unseen tasks could be composed on the fly from existing singular vectors.
7 ACKNOWLEDGMENTS
This work is supported by the MUR FIS2 grant FIS-2023-00942 "NEXUS" (cup B53C25001030001), and partly by Sapienza University of Rome via the Seed of ERC grant "MINT.AI" (cup B83C25001040001). It was also supported by projects PNRR MUR PE0000013- FAIR under the MUR National Recovery and Resilience Plan funded by the European Union NexIGenerationEU, PRIN 2022 project 20227YET9B "AdVVent" CUP code B53D23012830006. and the project BEAT (Better dlEp leArning securiTy), a Sapienza University project.
This work has also been supported by the French government, through the 31A Côte d' Azur Invest- ments in the project managed by the National Research Agency (ANR) with the reference number ANR-23-IACL-0001.
We finally acknowledge ISCRA for awarding this project access to the LEONARDO supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CINECA (Italy).
REPRODUCIBILITY STATEMENT
All implementation details and hyperparameter settings required to reproduce our results are pro- vided in Sections 3, and in Appendix B. The code is provided the supplementary materials and will be publicly released along with all checkpoints and logs.
REFERENCES
M Israk Ahmed, Shahriyar Mahmud Mamun, and Asif Uz Zaman Asif. Denn-based vegetable image classification using transfer learning: A comparative study. 2021 5th International Conference on Computer, Communication and Signal Processing (ICCCSP), pp. 235-243 IEEE, 2021. 19
Sarder Iftekhar Ahmed, Muhammad Ibrahim, Md. Nadim, Md. Mizanur Rahman, Maria Mehjabin Shejunti, Taskeed Jabid, and Md. Sawkat Ali. Mangoleafbd: A comprehensive image dataset to classify diseased and healthy mango leaves. Data in Brief, 47:108941, 2023. ISSN 2352-3409. doi: http://canneeerccccceaa///////// com/ sc
108941 URL https: / www. www.scciieecddreett. 19
10

---

Published as a conference paper at ICLR 2026
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-baiin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Repre- sentations, ICLR 2023, Kigali. Rwanda, May 1-5, 2023, 2023. 1,9, 17
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 2025. ISSN 2522-5839. 1, 17
Lorenzo Basile. Valentino Maiorca, Luca Bortolussi, Emanuele Rodolå, and Francesco Locatello. Residual transformer alignment with spectral decomposition. arXiv preprint M.Ni::2411.002444 2024. 8
Joshua Belofsky. Token-level adaptation lora adapters for downstream task generalization. In Proceedings of the 2023 6th Artificial Intelligence and Cloud Computing Conference, pp. 168- 172, 2023. 17
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 Mining Discriminative Com- ponents with Random Forests. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 446-461, Cham, 2014. Springer International Publish- ing. ISBN 978-3-319-10599-4 doi: 10. 1...007/978.3331--10999999
19
CCHANG. Garbage classification, 2018. URL https: / www. www.we e. com/ ds / /81794 19
Feng Cheng, Ziyang Wang, Yi-Lin Sung, Yan-Bo Lin, Mohit Bansal, and Gedas Bertasius. Dam: Dynamic adapter merging for continual video qa learning. In 2025 JEEE/CVF Winter Conference IEEE Computer Society, 2025. 17
Applications of Computer Vision (WACV), pp.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote Sensing Image Scene Classification: Bench- mark and State of the Art. Proceedings of the IEEE, 105(10): 18110))1885-88833 October 2017. ISSN 1558-2256. doi: 10. 10000000...200072675559000 URL https: : /leeexploreee... ieee org Conference Name: Proceedings of the IEEE.
document/ /7 / 19
Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint arXiv:2302.07007. 2023. 10, 17
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-inetuunee lan- guage models. Journal of Machine Learning Research, 25(70): 2024. 7
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing Textures the Wild. In 2014 IEEE Conference on Computer Vision and Pattern Recogni- tion, pp. 3606-3666666 3613, Columbus, OH, USA, June 2014. IEEE. ISBN 978-1-4799-5118-5 doi: 10. 1109/CVPR.2014.46) URL https: eeexp ore. feee. wwtp://ooo/oooo/o/oo/oo//oooo 6909985. 9, 19
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep Learning for Classical Japanese Literature, November 2018. URL http://arxiv. borg/bbs 1812.011111 01 11778888 8. arXiv:18122 1ifi::181221 111.001788 [cs, stat]. 19
Adam Coates, Andrew Ng. and Honglak Lee. An Analysis of Single-Layer Networks in Unsu- pervised Feature Learning. In Proceedings of the Fourteenth International Conference on Ar- rificial Intelligence and Statistics, pp. 215-223, JMLR Workshop and Conference Proceedings, June 2011. URL wtt.thtss//pppccccinaae 1938-7228. 19
mlr.press.
coatescatesaaaeaaeeoooooeeoeeeee .tmml. ISSN:
Gregory Cohen, Saced Afshar, Jonathan Tapson, and André van Schaik. EMNIST: Extending
MNIST to handwritten letters. In 2017 International Joint Conference on Neural Nenworks (IJCNN), pp. 2921-2926, May 2017. doi: 10. 10.1099/CNNN2011.996211 URL https://
leeexplore ieee. 4407. 19
ISSN: 2161-
11

---

Published as a conference paper at ICLR 2026
Donato Crisostomi, Marco Fumero, Daniele Baieri, Florian Bernard, and Emanuele Rodolà. C Cycle-consistent multi-mode merging. In Advances in Neural Information Processing Systems, volume 37, 2025. 9, 17
Nico Daheim, Thomas Möllenhoff, Edoardo Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by amcertainty-basee gradient matching. In The Twelfth International Conference on Learning Representations. 1, 17
Marczak Daniel, Magistri Simone, Cygert Sebastian, Twardowski Barttomiej. D Bagdanov Andrew, and Joost van de Weijer. No task left behind: Isotropic model merging with common and task- specific subspaces. ArXiv preprint, 2025. 1, 6, 9, 17
MohammadReza Davari and Eugene Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks. In European Conference on Computer Vision. Springer, 2025. 17
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for compulational linguistics: human language technologies, volume (long and short papers), pp. 4171 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale, 2021. URL https: www.aaaivvv.oo org/ abs /201 00.10..00 11929.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022. 9, 17
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference Machine Learning, ICML,2020, 13-18 July 2020, Viriual Event, Proceedings of Machine Learning Research, 2020. 9, 17
Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition. In ICLR, 2024.
Antonio Andrea Gargiulo. Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà. Task singular vectors: Reducing task interference in model merging. Proc. CVPR, 2025. 3, 6, 9, 17, 18, 21, 22, 23
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dans. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, 2018. 17
Ian J. Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, Yingbo Zhou, Chetan Ramaiah, Fangxiang Feng. Ruifan Li, Xiaojie Wang, Dimitris Athanasakis, John Shawe-Taylor, Maxim Milakov, John Park, Radu Ionescu, Marius Popeseu, Cristian Grozea, James Bergstra, Jingjing Xie, Lukasz Romaszko, Bing Xu, Zhang Chuang. and Yoshua Bengio. Challenges in Represen- tation Learning: A Report on Three Machine Learning Contests. In Minho Lee, Akira Hirose, Zeng-Guang Hou, and Rhee Man Kil (cds.), Neural Information Processing, pp. 117-124, Berlin, Heidelberg, 2013. Springer. ISBN 978-3-642-42051-11 doi: 10.
19
Fidel A. Guerrero-Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differentiation. In IEFF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, 2023. 17
Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. Merging experts into one: Improving computational efficiency of mixture of experts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14685-1444 14691, 2023. 10, 17
12

---

Published as a conference paper at ICLR 2026
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification, 1EEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217- doi: 10. 2911000STAAS22299999993 /ttp//menn//777///2/amm/// 877855 Conference Name: IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 19
2226, July 2019. ISSN 2151-1535.
URL https:
//ieeexplore. i eee org/
Stefan Horoi, Albert Manuel Orozco Camacho, Eugene Belilovsky, and Guy Wolf. Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis. 2024. 17
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free high-performance model merging, 2024. 17, 18
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representutions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023. 3. 6, 9, 17,
18
Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. In International Conference Machine Learning, pp. 14702-14729. PMLR, 2023. 10, 17
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur, REPAIR: renor- malizing permuted activations for interpolation repair. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023. 17
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011. 19
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D Object Representations for Fine- Grained Categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 554-561, Sydney, Australia, December 2013. IEEE. ISBN 978 978-1-4499--302--77 doi: 10.1 109/ eeexpl orr.. eeee....................... /ttp://rror/ooomoo//////555 675.ee/4n..65555////// 9, 19
ICCVW.2013.77. URL htl.:t/p//// htt.ttt:p/eeeeeplllllec
Alex Krizhevsky and Geoffrey Hinton.
Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009. URL https://wii
coronto.edu.. -rri z/lea rninnnninnn-n few.tres----eeeeeeeeeeeeer 2009-0009RR- pdf. 19
Makerere AI Lab. Bean disease dataset, January 2020.
URL
com/
AI-Lab- makkerereeibbean/ 19
Y. Lecun, I.. Bottou, Y. Bengio, and P. Haffner. tradieet-hased learning applied to document recog- nition. Proceedings of the IEEE, 86(1 1):2)722222233 1998. doi: (0.1109/5.72691... 19
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024. 2, 7, 10, 17
Michael Matena and Colin Raffel. Merging models with fishhe-weighted averaging. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. 17
Tommaso Mencattini, Adrian Robert Minut, Donato Crisostomi, Andrea Santilli, and Emanuele Rodolà. Merge3: Efficient evolutionary merging on consumer-gras grade gpus. arXiv preprint arXiv:2502. 10436, 2025. 17
13

---

Published as a conference paper at ICLR 2026
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Dilan Görür, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In 9th Interna- tional Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. 17
Mohammed Mugeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among special- ized experts for zero-shot generalization. In International Conference Machine Learning, pp. 2024. 10, 17
Aviv Navon, Aviv Shamsian, Ethan Fetaya, Gal Chechik, Nadav Dym, and Haggai Maron. Equiv- ariant deep weight space alignment, 2023. 17
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http:://fflttt stanford / wwt.sennmbees//ipps..
housenumbers. .dd. 19
Maria-Elena Nilsback and Andrew Zisserman. Automated Flower Classification over Large Number of Classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Im- age Processing, pp. 729,729, December 2008. doi: 00.110//CVGIP.2008..44 URL https:
//ieeexplore. Leee. wwt.o/////ooo/o///no/...
19
Mihai Oltean. Fruits-360 dataset. https:/ bttps://wwwr wwwp fruits, 2017. Accessed: 2025-01-01. 19
Guillermo Ortiz-Jiménez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. 1, 17
Oleksiy Ostapenko, Zhan Su, Edoardo Ponti, Laurent Charlin, Nicolas Le Roux, Lucas Caccia, and Alessandro Sordoni. Towards modular lims by building and reusing library of loras. In International Conference on Machine Learning, pp. 38885-38904. PMLR, 2024. 17
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 3508,, June 2012. doi: N0.1109/CVPR.2012..666 document/ 62480992.
URL
ieee.ergggr.... org/
ISSN: 1063-6919. 19
Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Pe- ter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In Proceedings of the 8th ACM on Multi- media Systems Conference (MMSYS), pp. 164-169, 2017. doi: 10. 00.1145/3031187.088222
19
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 9748-8763. PMLR, 2021. 1,5
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In Advances in Neural In- formation Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. 1,
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment tree- bank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), Proceedings of the 2013 Conference Empirical Methods in Natural Language Process- ing. pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguisties. URL http://aaaaaallllg////o wwwg/o//11... 3
14

---

Published as a conference paper at ICLR 2026
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. In The 2011 Inter- national Joint Conference Neural Networks, Pp. 1453-1460, July 2011. doi: 10.1109/ LJCNN.201 160033395 URL. https:::p: eeexxp www.ppooe..... ieee. www.eeerreeeeeeeeeeeeeeeeeeeeeee http///cmmmm/t03333333
Parmumber=6033395. ISSN: 2161-44077 19
George Stoica, Daniel Bolya, Jakob Brandt Bjomer, Pratik Ramesh, Taylor Hearn, and Judy Hoff- man. Zipit! merging models from different tasks without training. In The Twelfth International Conference on Learning Representations, a. 17
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffmann Model merging with svd to tie the knots. In The Thirteenth International Conference on Learning Rep- resentations, b. 17
Qi Sun, Edoardo Cetin, and Yujin Tang. Transformer-suuared:: Self-adaptive Ilms. In The Thirteenth International Conference Learning Representations, 2025. 10, 17
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In Artificial Neural Networks and Machine Learning-ICANs 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27, pp. 270-279. Springer, 2018.
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization. In The Twelfth International Conference on Learning Representations. 17
Anke Tang, Li Shen, Yong Luo, Shuai Xie, Han Hu, Lefei Zhang, Bo Du, and Dacheng Tao. Smile: Zero-shot sparse mixture of low-rank experts construction from pre-trained foundation models. arXiv preprint
2024a. 2, 10, 17
Anke Tang, Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task mod- els via weight-ensembling mixture of experts. In International Conference on Machine Learning, pp. 477999 PMLR, 2024b. 2, 17
Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Sociery Series B: Statistical Methodology. 61 1-622, 1999.
Bastiaan S. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation Equiv- ariant CNNs for Digital Pathology. In Alejandro Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-López, and Gabor Fichtinger (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2018, pp. 210-218, Cham, 2018. Springer International Publish- ing. ISBN 978- 978-3-030-00934-2 doi: 10. 10.10979 00000/978---00000994444
24. 19
C. Wah, S. Branson, P. Welinder, P. Perona, and Belongie. Cub200. Technical Report CNS-TR- 2011-001, California Institute of Technology, 2011. 19
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. 6,
Kaihong Wang, Mona Jalal, Sankara Jefferson, Yi Zheng, Elaine O. Nsoesie, and Margrit Betke. Scraping social media photos posted in kenya and elsewhere to detect and analyze food types, 2019. URL https: httpss///aaxvvvaa wro. abs/ 909. 00134. 19
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In Proceedings of the st International Conference on Machine Learning, Proceedings of Machine Learning Research, 2024. 1, 6, 9, 17, 18
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models, 2021. 17
15

---

Published as a conference paper at ICLR 2026
Haixia Xiao. Weather phenomenon database (WEAPD), 2021. URL httpst///66/...g////1
2010/DVN/MBJOCE 19
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Bench- marking Machine Learning Algorithms, September 2017. URL http::haaxii..oo/ abs/ 1708. 07747. MrXiv:170.0077
[cs, stat]. 19
Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. SUN Database: Exploring Large Collection of Scene Categories. International Journal of Computer Vision, 119(1):3-2,, August 2016. ISSN 1573-1405. doi: 10.100775112233-04--07444 ///doi.org//// 1007/10000000 /07/s0126333 014 0748
URL https:
19
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. 1,9, 17
Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mo- hit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning, 2024. 2, 10, 17
Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang. and Dacheng Tao. Representation surgery for multi-task model merging. In Proceedings of the 41st Inter- national Conference on Machine Learning, Proceedings of Machine Learning Research, 2024a.
17
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo. Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, 2024b. 17
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014. 1
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Proceedings of the 41st Interna- tional Conference on Machine Learning, Proceedings of Machine Learning Research, 2024. 1, 9,
17
Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang. Kun Kuang, and Fei Wu. LoraRetriever: Input-aware LoRA retrieval and composition for mixed tasks in the wild. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Associa- tion for Computational Linguistics: ACL 2024, pp. 4447-4462, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1/.18 fin.g///...iiinngaa........ URL https: / http:///aaaaatthhaocc/e wrg/ Findings- acl. 263/ 17
Luca Zhou, Daniele Solombrino, Donato Crisostomi, Maria Sofia Bucarelli, Fabrizio Silvestri, and Emanuele Rodolà. Atm: Improving model merging by alternating tuning and merging, 2024.
URL htttttttt///aattta
abs/2411. 03055. 17
Åke Björck and Gene H. Golub. Numerical methods for computing angles between linear subspaces. Mathematics of Computation, 27(12)):57--594 1973. ISSN 00255718, 10886842. URL http: / /wwww jstor. wtg./ttbbeee2055566.. 25
16

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
A.1 MODEL MERGING
Model Merging has recently gained traction as a computationally efficient alternative to ensembling. Early methods, motivated by linear mode connectivity (Frankle et al., 2020; Entezari et al., 2022; Mirzadeh et al., 2021; Garipov et al., 2018), primarily aligned models trained with different opti- mization seeds. This was achieved by finding neuron permutations matching these ones before the aggregation (Ainsworth et al., 2023; Jordan et al., 2023; Crisostomi et al., 2025; Singh & Jaggi, 2020; Stoica et al., a; Guerrero-Petn et al., 2023; Navon et al., 2023; Horoi et al., 2024). More recent merging approaches aggregate instead multiple fine-tuned models derived from a shared pre- trained backbone (Ilharco et al.. 2023; Yadav et al., 2023; Yu et al., 2024; Matena & Raffel, 2022; Wortsman et al., 2021; Davari & Belilovsky, 2025; Wang et al., 2024; Zhou et al., 2024; Gargiulo al., 2025;
et al., 2023; Mencattini al., 2025; Huang et al., 2024; Daheim et al.; Stoica et al., b; Yang et al., 2024a; Tang et al.). These methods incorporate various strategies to improve the merging, such as finding the optimal combination of task vectors (Yang et al., 2024b), mitigating sign disagreement (Yadav et al., 2023), randomly dropping fraction of the updates (Yu et al., 2024), or employing evolutionary strategies (Akiba al., 2025; Mencattini et al., 2025). The most recent line of work considers task vectors at layer level, accounting this way for the natural structure of the layers and significantly improving the merging outcome (Stoica et al., b; Gargiulo et al., 2025; Daniel et al., 2025). Our approach builds upon the latter methodologies by introducing adaptivity through input-driven routing, significantly narrowing the performance gap between the finetuned endpoints and their resulting merge.
A.2 MOERGING
Model Merging with Mixture-of-Expere often referred to as "MoErging" (He et al., 2023; Yadav et al., 2024; Jang et al., 2023; Chronopoulou et al., 2023; Belofsky, 2023; Zhao et al., 2024; Muqeeth et al., 2024; Tang et al., 2024b; Ostapenko et al., 2024; Cheng et al., 2025; Tang et al., 2024a; Lu et al., 2024; Sun et al., 2025), explores how independently trained by a decentralized community-can be combined within a single adaptive model by dynamically selecting which expert(s) should handle given input. In the LLM domain, specialized modules (e.g.. LoRAs or adapters) are merged via parametric or data-driven routers that match the incom- ing prompt to the most relevant module (Jang al., 2023; Chronopoulou et al., 2023; Belofsky, 2023; Zhao et al., 2024; Muqeeth et al., 2024; Tang et al., 2024b; Ostapenko et al., 2024; Cheng et al., 2025); some approaches, such as PHATGOOSE (Muqeeth al., 2024), require fine-tuning additional routing parameters, whereas others, like enggeblinsembiig MoE (Tang et al., 2024b), may need train the router at test time. Sharing a similar two-pass pipeline, frrms (Sun et al., 2025) modifies the finetuning procedure to yield task-aligned singular vectors, allowing for expert routing at test time. Differently from the latter, MASS works on any independently fine- tuned models finetuned with no ad hoc routines. A similar strategy was independently introduced by SMILE (Tang et al., 2024a), which, like our method, enables data-free merging of multiple experts. However, SMILE leaves the pretrained backbone intact and selectively adds task-specific low-rank updates at inference, whereas MASS begins with a single model containing all updates and deac- tivates any irrelevant subspaces via router. The fact that both lines of work arrived at similar subspace-activation concept, despite differing motivations, highlights the versatility and broad ap- peal of such an approach. Lastly, Twi iMerging (Lu et al., 2024) similarly merges shared expert and multiple task-specific experts via gating function. However, TwinMer grgi ging relies on flat task arithmetic (Ilharco et al., 2023) and requires per-task labeled data train its router, reducing its applicability. In contrast, MASS operates in a fully data-free and training-free regime by design.
contributed
B ADDITIONAL DETAILS
We here describe the details required to implement and reproduce our results. The code is provided in the supplementary materials. In particular, Section B.2 describes implementation details, Sec- tion B.4 specifies the employed evaluation metrics, Section B.5 reports the employed architecture and Section C.2 specificies the hyperparameters and how they were chosen.
17

---

Published as a conference paper at ICLR 2026
ViT-3-32
ViT-B-16
ViT-L-14
Method
tasks
tasks
tasks
tasks
tasks
20 tasks
tasks
14 tasks
tasks
WeMoE
5.06
8.06
11.05
5.12
8.16
11.20
5.78
9.31
12.84
MoE
SMILE-1
2.07
1.62
2.55
1.47
2.18
SMTLE-2
3.05
4.60
6.14
3.09
4.67
6.24
2.59
3.78
4.96
MASS
2.00
2.00
2,00
2.00
2.00
2.00
2.00
2.00
2.00
Table 4: Relative parameters increase with respect to the base model.
B.1 STORAGE AND COMPUTE OVERHEAD
The 2x compute figure is approximation: run the backbone twice (fixed TSV-M (Gargiulo et al., 2025) pass + routed pass) and the router itself adds < 1% FLOPs. Since model-merging saves training and storage rather than inference time, this modest overhead is acceptable. MASS routes per sample, so each incurs this cost; however, batching over the same task can reduce it. For example, with 32 samples, routing costs only 1.06x forward equivalents. The storage overhead is exactly 2x: alongside the merged model, we store 1/T" TSVs per task. Summed across tasks and layers, these form a second full set of weights, effectively doubling storage. A full comparison with the overhead incurred by the baselines is provided in Table 4.
B.2 IMPLEMENTATION
We used the same model checkpoints Consensus TA (Wang et al., 2024), except for the one for the EMNIST dataset which we had to re-finetune due an inconsistency in image orientation between EMNIST and MNIST. Shortly, the torchvision' version yields rotated and flipped images, spuriously yielding extremely similar models (same classes, roughly same dataset statistics) that performed very poorly when interchanged. Simply re-rotating and flipping the EMNIST images to match the orientation of MNIST solves the issue. We further used a single classification head for STL10 and CIFARI0 0 due to their shared classes. The final head has the shared classes plus the two dataset-specific ones, i.e. monkey and frog.
B.3 BENCHMARKS AND DATASETS
The 8-task benchmark, introduced (Ilharco et al., 2023), comprises the following datasets: Cars, DTD, EuroSAT, GTSRB, MNTST, RESTSC45, SUN397, and SVHN. Moving 14 tasks, add CIFAR100, 00, STL10, Flowers100 02, OxfordIIITPet PCAM, and FER2013. The 20- task suite further includes EMNIST, CIFAR10 0, Foodi 01, FashionMNIST, RenderedSST2, and KMNIST. We provide the specific dataset details in Table 5. For our scalability study we further include 13 additional publicly available datasets to reach total of 33 tasks. These ad- ditional datasets are collected form the EMR-Merqing (Huang et al., 2024) thirty tasks bench- mark: Beans, CUB200, Dogs, flowerskaggle, Fruits360, Garbage, Intel Images, KenyanFood13 3, Kvasi rv2, Landscape, MangoLea fBD, Vegetables, and Wea Weather.
18

---

Published as a conference paper at ICLR 2026
Dataset
image size
# train
# val
# test
Cars (Krause et al., 2013)
varies
7330
814
8041
DID (Cimpoi et al., 2014)
varies
1692
188
1880
EuroSAT (Helber et al., 2019)
64 x 64
21600
2700
2700
GTSRB (Stallkamp al., 2011)
varies
23976
2664
12630
MNTST (Lecun et al., 1998)
28 x 28
55000
5000
10000
RESISC45 (Cheng et al., 2017)
256 x 256
17010
1890
6300
SUN397 (Xiao et al., 2016)
varies
17865
1985
19850
SVHN (Netzer et al., 2011)
32 x 32
68257
5000
26032
CIFAR100 (Krizhevsky & Hinton, 2009)
32 x 32
45000
5000
10000
STL10 (Coates et al., 2011)
96 x 96
4500
500
8000
Flowers102 (Nilsback & Zisserman, 2008)
varies
918
102
6149
OxfordIIITPet (Parkhi al., 2012)
varies
3312
368
3669
PCAM (Veeling et al., 2018)
96 x 96
257144
5000
32768
FER2013 (Goodfellow et al., 2013)
48 x 48
25839
2870
7178
EMNIST (Cohen et al., 2017)
28 x 28
235000
5000
40000
CIFAR10 (Krizhevsky & Hinton, 2009)
32 x 32
45000
5000
10000
Food10r 01 (Bossard et al., 2014)
512 x 512
70750
5000
25250
FashionMNISH (Xiao et al., 2017)
28 x 28
55000
5000
10000
RenderedsST2 (Socher et al., 2013)
varies
6228
692
1821
XMNIST (Clanuwat et al., 2018)
28 x 28
55000
5000
10000
Beans (Lab, 2020)
500 x 500
1030
133
128
CUB200 (Wah et al., 2011) 1)
varies 5999
0
5790
Dogs (Khosla et al., 2011)
varies 12000
0
5880
FlowersKaggle
varies
4242
942
Fruits360 (Oltean, 2017)
100 x 100
118783
0
39613
Garbage (CCHANG, 2018)
varies
2359
0
IntelImages
150 x 150
14000
3000
7000
KenyanFood13 (Wang et al., 2019)
varies 8,174
0
0
Kvasirv2 (Pogorelov et al., 2017)
varies
4000
0
Landscape
varies
1000
1500
500
MangoLeafBD (Ahmed al., 2023)
240 x 320
4000
0
0
Vegetables Ahmed et al. (2021)
varies
14700
3150
3150
Weather (Xiao, 2021)
256 256
6877
0
0
Table 5: Image sizes and numbers of train, validation, and test samples for all datasets considered. For datasets lacking validation or test split we used the 10% of the training set.
19

---

Published as a conference paper at ICLR 2026
2Maag
ue aus aus solluge
327 6 UUE DUE
Figure 8: vi T-I-14 per-layer task accuracies.
B.4 EVALUATION MEASURES
To account for differences in task difficulty, report both absolute and normalized accuracy in our results. The normalized accuracy serves as a relative performance measure by comparing the multi-task model's accuracy to that of individual fine-tunes models. computed as:
acc accuracy(Oft..1)
Normalized Accuracy
where 7 represents the total number of tasks, 0uT is the multi-task model, and ft, corresponds to the fine-tuned model for task f, By normalizing accuracy in this way, we ensure fairer comparison that accounts for variations in baseline task performance.
B.5 ARCHITECTURES
We use CLIP from the OpenClip library', using the three different versions described Table 6. For the router, we small two-layer MLP with hidden dimension of 1024. It accepts 512- dimensional embedding vector, applies a linear transformation, ReLU activation, and dropout with a probability of 0.5, and outputs logits corresponding to task selection probabilities.
Model
Layers
Hidden Dimension
Heads
Patch Size
Parameters
ViT-B-32
12
768
12
32x32
~66
ViT-B-16
12
768
12
16x16
~66
ViT-I-14
24
1024
16
14x14
~007
Table 6: Comparison of ViT-B BATT-B-33 B-32, ViT-V-1----- and ViT-L-1 architectures.
B.6 ADAPTING ORACLE MOERGING METHIODS
As extensively discussed, we argue that MoErging methods should not rely on an oracle head, since their pipelines implicitly assume that the task label is unknown at inference time (otherwise, merg- ing the correct task vector would be trivial). To respect this assumption, we introduced a routing procedure that exploits the routers already implemented in each method. For SMILE, we extracted the mode of the tokens at each layer and applied naïve majority-voting scheme across layers to select the head for each sample, which was then used for the final classification. Analogously, for WeMoE, we applied the same logic, implementing straightforward head-selection strategy that leverages the existing gates and coefficients. All results reported for MoE methods were obtained under this unified setting.
Radar charts on and 14-task benchmarks. section Section 4, we present comprehensive results for the approach using the 20 task benchmark. Figure 9 and Figure 11 respectively display the normalized accuracies for our method on and 14 tasks across all three model sizes (--3-B ViT-B-16 and ViT-L-14).. In both cases, the approach retains high fraction of each fine-tuned
hhtgs//ggtthhh.cc////ooonaannnne clip/
20

---

Published as a conference paper at ICLR 2026
Algorithm 2 Fixed Merging Step
Require: Pretrained model weights Pre. task-specific updates {A,) user-specified threshold Ensure: Fixed merged model weights ONT
1: Accounting for redundant directions (Section 3.2.2)
2: -M -0
3: for = L....., Todo do
4: 8; f vec(A;)
5: if [jx() sim(6;,6)) <E then
6: + (
7: end if
8: end for
9: Merging step using TSV-M (Gargiulo et al., 2025) the (Ai hem
10: for E M do
A; Ug V,T
U, t- U.
E, E.)) V
Vij.
13: end for
+ [ UZ2 ... [ū]]
15:
Σ + block diag dlig (L,, ....... Ex))
16:
V < [Vi V/2
Vr]
17:
U1 orthogonalize( U)
18:
VI + orthogonalizo V)
19:
A + EV
20:
ONT qu Å
21:
return ONT
VPB-2
VI-8-16
VI-4-44
Figure 9: Normalized task accuracies models - 32T and ViT-L-14 for the 8 tasks benchmark.
model's performance, with normalized accuracies often above 80-90%. Notably, the method scales gracefully the number of tasks increases from 8 to 14.
C ADDITIONAL EXPERIMENTS AND RESULTS
In this section, we provide detailed per-task and per-layer accuracy plots, along with further exam- ples of decoded task vectors, complement the results presented in the main paper.
C.I MORE EXPERIMENTS
Generalization to unseen tasks To evaluate how well the method behaves on tasks not included during merging, conducted an out-of-distribution experiment the 8-task benchmark. For each task turn, we removed from the merge set, merged the remaining seven tasks, and then evaluated on the held-out one. The Table reports the results, comparing MASS with Task Arithmetic. MASS
21

---

Published as a conference paper at ICLR 2026
outperforms the static merging baseline in all cases, this suggests that the task subspaces extracted from weight updates contain transferable structure that remains useful even when the target task was never seen during merging.
ViT-B-32
Method
DTD
MNIST
GTSRB
SVHN
EuroSAT
RESISC45
Cars
SUN397
TA MASS
(1,3
79.8
26.7(27.))
49.7
(1.0
36.9(877)
40.6
44.4
(4.2)
86.687.1
33.1(33.)
69.7
(0.9
46.4
49.5(62.0)
54.8(73.)
Table 7: Comparison of MASSand Task Arithmetic on 8 datasets using ViT-B-22 32. Main accuracy with normalized accuracy in subscript.
Scalability to Large Task Sets. Table summarizes our scala- bility evaluation presented in Fig. 6 providing exact numerical re- sults. Across all configurations, MASS maintains substantial per- formance margin over task arithmetic. Even at 33 tasks, our method retains gap of roughly 20 accuracy points over the baseline. Im- portantly, the method preserves efficiency: the overall computa- tional cost continues be dominated by two forward passes, which do not depend on the number of tasks.
VLT-B-32
1--20
n--25
#--30
i--3
0.72
0.63
0.64
MASS
0.91
0.85
0.83
0.83
Table 8: Scaling MASS to larger task sets.
The only term that grows with the number of tasks is the routing computation, each step projects intermediate activations onto the task subspaces via small matrix product, resulting in routing com- plexity that scales linearly with the number of tasks. This ensures that MASS continues to merge large task collections without degradation in compute efficiency.
Weighted vs Optimal TSV. Prior work (Gargiulo et al., 2025) has established that the optimal for TSV is 1.0 and remains invariant across a range of task counts (8-20 tasks). We implemented variant of TSV in which the left and right singular vectors are scaled by w softmax(-r). before the merging with using residuals norm as measure of contri- bution. The results indicate that the weighted TSV does not improve performance compared our original approach, con- firming the optimality of the value established in prior work.
Method
8 tasks
tasks
tasks
Weighted
93.5
90.1
86.4
96.5
Table 9: Comparison of weighted TSV and classic ViT-B-33 task counts.
TSV on across different
Adaptive Merging Ablation. We investigated the impact of singular vectors orthogonalization in our adaptive merging step in Table 10. The results show that applying orthogonalization restricted to the routed set does provide consistent benefit, but the gains are modest. This is expected: for small task sets, Task Arit Arittha hmetic and TSV already perform similarly, since interference between tasks limited and the dominant singular directions are relatively well separated. Orthogonalization becomes more useful as the number of tasks increases and subspace overlap grows, but it is not the primary driver of the method's ' performance.
ViT-B-32
ViT-B-16
ViT-L-14
Method
8 tasks
tasks
20 tasks
8 tasks
tasks
tasks
8 tasks
tasks
20 tasks
non-orthe
95.3
93.0
90.7
97.3
96.1
88.5
97.8
97.3
96.6
ortha
96.5
90.9
98.0
96.1
88.7
98.6
96.6
Table 10: Effect of orthogonalization in adaptive merging. Applying orthogonalization restricted the routed set consistently improves performance across ViT-B-32, ViT-B-16, and Vir-I-11
22

---

Published as a conference paper at ICLR 2026
Vi ViT-B-32
V-T-B-16
Method
tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 lasks
SMILEE (oracle head)
87.7
85.9
85.0
92.8
92.0
91.4
MASS (oracle head)
89.3
86.6
86.2
93.7
92.8
91.5
Table 11: Comparison of MASS and SMILE under the oracle head setting across ViT-B-32 and
ViT-B-11 6 for different task counts.
Oracle Head The novel experimental setting introduced in this paper considers the classification head not known priori, requiring router to select this alongside the optimal encoder. To ensure a comprehensive evaluation, we however also compared our approach against the best performing baseline LE also under the classical oracle head setting. Table shows the results. While the gap closes, MASS still outperforms SMILE across all benchmarks proving again its effectiveness.
C.2 HYPERPARAMETER SETTINGS
Following the recommendation of TSV (Gargiulo et al., 2025), we use a as single scaling factor with the suggested value of 1.0 for the TSV-M merging configurations in both Algorithm and 1 We Algorithm I. Consistent with TSV, the compression rate assigned to each task space is set to т. optimized the similarity threshold over the range {0.1, 0.2, .... 0.9) and determined the router's selection threshold 1y via Bayesian search within the interval [0.05, 0.5]. As illustrated in Figures 4a and 8, we identify the optimal projection layer by focusing on those revealing the highest task accuracy. Specifically, for ViT-B-32 32 and ViT-B-11 16 models, select the attention and MLP layers within the range {7, .... 113. while for the ViT-L-14 model, the chosen layers fall in the range (19, .... 23}. The temperature parameter for tuning the behavior of the softmax function at line 6 in Algorithm set to 1.
Sensitivity to Hyperparameters The merging co- efficient is set to 1 as in TSV-M (Gargiulo et al., 2025). Figure 10 shows how accuracy changes with routing threshold and lop-K. At low (0.05) and large K, too many tasks are merged, causing inter- ference. At high (2 0.3), only the top task is se- lected, making performance insensitive to K (it's ef- fectively argmax). The best accuracy occurs in a broad middle range, peaking at = 0.2, where the router balances selectivity and coverage. We also vary the cosine threshold ε used to discard similar task updates before merging. Due the high dimensionality of the As, large thresholds (e 2 0.4) retain all updates, leaving redundancy unaddressed (accuracy 93.5). Small ones (s <. 0.05) instead re- move even distinct directions, significantly harming accuracy (S 88.6). Intermediate values (e z 0.2) offer a robust filtering, improving performance (2 93.9) by suppressing redundancy.
0.05
94.6
94.1
89.5
85.2
Rouler Thrshood
0.1
94.3
96.6
94.8
96.1
0.15
94.7
96.4
95.9
3,
0.2
94.6
96.0
97.5
93.9
0.3
94.4
Max num tasks to select.
Figure 10: Hyperparameter sensitivity.
Layer-wise accuracies for individual datasets. Figures 12 and 13 show per-layer accuracies for ViT-B-32 on different subsets of the 8-task benchmark:
Figure 12 focuses on Cars, DTD, EuroSAT, and GTSRB.
Figure 13 displays results for MNIST, RESISC45, SUN397, and SVHN.
Again, the top perforrfonn layer is not shared across the tasks, confirming what we observed in the main paper.
23

---

Published as a conference paper at ICLR 2026
VIE
WDDE
20LE
Figure 11: Normalized task accuracies over models ViT B-T---32 32, ViT-B-ILBB ATT---16 16 and ViT L--14 for the 14 tasks benchmark.
(a) CARS.
(b) DTD.
EuroSAt.
GTSRB.
Figure 12: Per-layer task accuracies for ViT-B-32 on Cars, DTD, EuroSAT, and CTSRB.
(a) MN MNTST ST.
(b) RES TSC45.
(c) SUN397.
(d) SVHN.
Figure 13: Pay-layer task accuracies for ViT-B-32 MNIST, RESISC45, SUN397, and SVHN.
24

---

Published as a conference paper at ICLR 2026
Layer-wise accuracies for ViT-L-14. Figure reports average per-layer task accuracy for the larger ViT-L-14, showing that in this case the most predictive layer for routing is = 20. Since ViT-I-14 has 24 layers (compared to the 12 layers in ViT-B-32 and ViT-B-166 the most predictive layer is roughly at the same relative depth.
Visualizations for additional datasets. Following the approach in Section 4.4, Figure 14 shows examples of decoded task vectors for datasets like SVHN, GTSRB, SUN397, and RES RESISC45. Here, see textual prompts such as "An image of the number 4" or "Aerial view of an industrial area", which align with each dataset's distinct domain. This reaffirms that our singular vectors capture domain-specific transformations while preserving high-level semantic alignment to the pretrained model.
Replacing L2 with Mahalanobis distance. Alternatively the L2-based projection used for routing, one could employ a data-free Mahalanobis distance by whitening the task aasc-specific bases derived from the right singular vectors used for projection. Being all statis- ties computed solely from the learned bases, this would still pre- serve the data-free assumption.
ViT---32
tasks
14 casks
20 tasks
Mahalanohis
95.7
91.0
88.0
L.2 (Ours)
96.5
932
90.9
Table 12: Comparison of Ma- halanobis vs. L2 distance.
particular, given the right singular vectors (V; E R RRxP) Oi==0.....TT with R chosen such that R T D, concatenate and center them: V. =V bv, BV R.R and compute the covariance: Σ cov(V.) E DxD, The Mahalanobis distance between æ and is then:
d(x,v) = (x ((1--))T-- 1) y).
We report the corresponding results in Table 13. Whitening the bases and replacing L2 with Ma- halanobis does not improve performance; results are consistently slightly worse, indicating that the Fuclidean residual is sufficiently well-conditioned for routing mechanism.
Cosine similarity vs principal angles for redundancy filtering We implemented the proposed approach by computing the princi- pal angles between subspaces (PABS (Åke Björck & Golub, 1973)), pairs of task matrices A. For each candidate A., used the largest singular value (equivalently, the minimum cosine between the subspaces) as the similarity measure when deciding whether the update was already "represented" by previously accepted ones. As with the cosine-bases -bsed criterion, we performed small grid search over the threshold on the 8-task benchmark. We report the normal- ized accuracies below.
ViT B
8 tasks
tasks
20 tasks
PBAS Cosiue
96.8
92.7
86.8
96.5
93.2
90.9
Table 13: Comparison of Principal Angles Between Subspaces vs. Cosine.
The principal-angle criterion slightly outperforms cosine similarity on the 8-task benchmark, but loses accuracy on the larger 14- and 20-task settings. This suggests that while more geometrically principled, the principal-angle filtering is less robust when task diversity increases.
Per-task variances We provide in table Table 14 the task-wise mean and standard deviation for the MoErging experiments. Remarkably, MASS not only has a better average, but also a smaller standard deviation with respect the existing baselines.
D PROPOSITIONS AND PROOFS
Proposition D.I (Optimality of Orthogonal Projection). Let BRdk have orthonormal columns spanning a subspace S S Rd. and let E Rdd, Then the unique minimizer of l|a w|l3 over all
is W E
W V
25

---

Published as a conference paper at ICLR 2026
Mithod
VET 32
VES 1.6
VET 14
Walhot:
97.74 2.38
82.63 28.78
76.36 53.45
96.43 4.72
83.20 28.09
70.54 d: 12.51
94.20 a 7,50
72.61 di 85.64
61.41 #:22.1
02.14 5.02
84.50 12.63
60.
20.31
$4.00 5.78
89.50 + 10.18
80.75 4 16.29
96.78 2.90
92.78 + 10.75
90.52 13.72
83
90.56
90.
2.83
99.03
83,
91.15
Table 14: Average performance (mean + std) on 8-, 14-, and 20-task benchmarks for different VIT backbones.
MASS
ViT-B-32
ViT-B-16
+
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
nn
92.7
89.6
89.4
92.7
90.0
90.2
m1p
96.8
95.8
95.8
97.1
94.8
96.5
projere 98.2
88.6
79.4
98.7
92.0
81.2
pro projrsv-i
96.5
93.2
90.9
98.0
96.1
88.7
Table 15: Average normalized accuracy for different routers.
Proof. Any E can be written V a for some a R$ The problem
min a w
min la Va 1 a
w WES
has strietly convex objective, so its global minimizer is found by setting the gradient to zero. A short calculation shows
V
w aV a) VV a.
Uniqueness follows from the strict convexity, and |a W is necessarily the smallest possible distance in S. Equivalently, a a- is orthogonal to S, so further reduction in norm possible. O
Proposition D.2 ($3.1). Let 2z Rd be feature vector, and for each task i, decompose it
V.V, ze Ei.
e; = -1-VVVV =t-
Assume E; ~ (O, I)). Then the maximum posteriori estimate of the task reduces to MAP arg max pltask = zæ) arg min ||e.
Thus, under these assumptions, selecting the task with the smallest squared Euclidean residual is exactly equivalent to maximizing the posterior.
Proof. By assumption, p(ε.) (2 -/2 exp -xlel so logp(e;) x ||e Since V.V. ze is deterministic shift, the likelihood p(zz task = i) depends only on E,. With uniform prior over tasks,
MAP = arg max p(ze task i) = arg max maxp(e;) = arg min ||
Hence, minimizing the l residual is exactly equivalent maximizing the posterior.
26

---

Published as a conference paper at ICLR 2026
"AN IMAGE OF THE NUMBER 4"
"A BADGE"
2
80
SVEN
GTSRR
"AN IMAGE AN INTERIOR OF ROOM"
"AERIAL VIEW OF INDUSTRIAL AREA"
(c) SUN397
(d) RESISC45
Captions obtained by decoding task singular vectors as text for datasets SVHN GTSRB, SUN397, 7, and RES ISS4 as described in Section 4.4, accompanied by three representative images for each dataset.
Figure 14:
27


Reviews:
[{"review_id": "PSUqmkdx2R", "reviewer": "Reviewer_9j9B", "summary": "The paper introduces **MASS**, a novel approach for model merging. It targets the problem of merging multiple fine-tuned models into a single, shared model while retaining performance comparable to that of individually fine-tuned models, without the need for additional training or data. MASS leverages low-rank decomposition and adaptive routing mechanisms to select the most relevant task-specific subspaces for each input, making it an efficient solution for multitask learning scenarios. Experimental results show that MASS achieves near-state-of-the-art performance on various benchmarks while reducing storage requirements compared to traditional ensemble methods.", "questions": "1. How does the two-pass inference process impact latency, especially in time-critical applications?\n2. How can the method be generalized to handle tasks that were not part of the fine-tuning phase?\n3. As the number of tasks increases, how does the method maintain performance and efficiency?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. **Innovative Concept:** The adaptive routing mechanism for task-specific subspace selection and the use of low-rank decomposition is an interesting approach to reducing storage requirements in model merging.\n\n2. **Practical Application:** The method’s ability to merge models without additional data or retraining is a valuable contribution, especially for scenarios involving pre-trained models from public repositories.\n\n3. **Strong Experimental Results:** MASS performs well across multiple vision and language benchmarks, surpassing existing methods in accuracy and storage efficiency.\n\n4. **Efficiency:**  MASS performs well with a moderate increase in storage compared to a single model, providing a storage-efficient alternative to traditional ensembling methods.", "weaknesses": "1. **Generalization to Unseen Tasks:** The approach seems to focus on tasks seen during fine-tuning, but its ability to generalize to unseen tasks or tasks that require more nuanced adaptations is unclear. The task subspaces may not adequately cover unseen task distributions, affecting its robustness in dynamic or changing environments.\n\n2. **Scalability to Larger Task Sets:**  While MASS performs well on 8–20 tasks, its scalability to a larger number of tasks is unclear. As the task set increases, the complexity of the routing mechanism may grow exponentially, making the method less feasible for large-scale applications.\n\n3. **Inference Overhead:** The two-pass inference process introduces additional overhead, which could be problematic in real-time applications where latency is critical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wFB8K1ZpUd", "reviewer": "Reviewer_8SY3", "summary": "The paper proposes a training‑ and data‑free way to merge multiple fine‑tuned models into a single network that selects relevant encoder subspaces per input and also selects the appropriate classification head when the task is unknown at test time. Concretely, each per‑task weight update $\\Delta_i$ is SVD‑decomposed and truncated; at inference, an input’s mid‑layer features $z_\\ell$ are projected onto each task’s right‑singular subspace and the residual $r_i=|z_\\ell - V_i V_i^\\top z_\\ell|_2^2$ scores task relevance. The router keeps top‑K tasks above a softmax‑based threshold and adaptively merges the corresponding low‑rank updates; a second pass runs the selected heads and takes the max‑logit across them.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. It Introduces a projection‑based, training‑free router in weight space that operates on TSV‑organized task subspaces and extends routing to head selection when the task is unknown. The MAP view provides a statistical rationale. \n2. Algorithms and figures make the method easy to follow. \n3. It Provides constant 2× storage independent of task count.", "weaknesses": "1. Routing layer and thresholds are selected with labeled validation accuracy as Appendix C. This partially contradicts the “data‑free” framing and may inflate reported gains versus truly label‑free procedures. \n2. Alg. 1 uses $w=\\mathrm{softmax}(-r)$ only for selection; the merge $\\sum_{i\\in\\Omega}U_i\\Sigma_iV_i^\\top$ is uniform, which miss a straightforward weighted variant $\\sum w_i\\Delta_i$. An ablation could show whether weighting reduces interference or improves tail tasks. \n3. The fixed merge (Alg. 2) uses orthogonalized bases $U_\\perp,V_\\perp$. It is unclear whether the adaptive merge applies the same orthogonalization restricted to $\\Omega$. Clarify and ablate. \n4. MoE baselines are adapted with a majority‑vote head heuristic. Provide companion results with oracle heads (as in their original setting) and/or head‑logit calibration (e.g., temperature scaling) to separate routing vs. head‑selection effects. \n5. Euclidean residual assumes isotropy; unlabeled whitening/Mahalanobis distances on $z_\\ell$ could be stronger and still data‑free, which relaxes Proposition 3.1’s assumptions. So it would be better to report sensitivity. \n6. Cosine similarity of vec($\\Delta$) ignores subspace geometry. Principal angles between spans $(V_i)$ would be more principled. Provide ablation. \n7. No seeds/variance or CIs are given; wall‑clock overhead of two passes and multi‑head evaluation is not quantified; GLUE per‑task wins/losses are not detailed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rI9gwM38Ga", "reviewer": "Reviewer_eoaD", "summary": "This paper proposes applying model merging in a more challenging scenario, where the optimal encoder subspace and classification head for the current task are unknown. Under this setting, the authors introduce MASS, which predicts by selecting the most relevant series of encoders within the subspace. The method’s superiority over baselines is validated across tasks of varying scales.", "questions": "Please revise or clarify the description in Section 3.2.1.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors provide detailed experimental results for analysis.\n- The MASS method is simple yet effective.", "weaknesses": "- The description in Section 3.2.1 is somewhat unclear. Since minimizing $r_i$ yields the optimal $L_2$ projector, the proj in L191–192 should not be referred to as “the optimal $L_2$ projector.” Additionally, in line 195, what does “the residual” refer to—is it $r_i$?\n- My main concern lies in the reasonableness of the paper’s setting. \n    - Given that models have already been fine-tuned on each task, why is it necessary to assume that the task-optimal encoder and classification head are unknown? It is explicitly known which task each model has been fine-tuned on. \n    - The authors should impose stricter constraints on the models involved in merging to make the setting reasonable.\n    - Otherwise, the authors should clarify this point; otherwise, the foundation of the paper is difficult to justify.\n- Line 406 mentions “prior work suggests that mid-layer embeddings in CLIP-like models...”; a citation should be added here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3UOAGQlsRb", "reviewer": "Reviewer_JZ1o", "summary": "This paper introduces MASS (MoErging through Adaptive Subspace Selection), a novel model merging approach, and its core premise is that in practical applications, the task identity is often unknown at inference time. To tackle this more challenging and realistic scenario, MASS introduces an adaptive routing mechanism. This mechanism, which is free of additional training and data, dynamically selects the most relevant task subspace for each input sample by calculating the projection residuals of the input's features onto different task subspaces. Experimental results demonstrate that MASS achieves state-of-the-art performance on several benchmarks.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tA Critical and Practical Problem Setting: The paper astutely identifies and challenges an unrealistic assumption common in existing model merging literature: that task identity is known at inference time. By proposing a more realistic scenario where task identity is unknown, the authors elevate the practical relevance of their research and open up a more meaningful direction for the community. MASS provides a complete and effective end-to-end solution for this setting.\n2.\tA Novel and Efficient Training-Free Routing Mechanism: A major highlight of this work is its projection-based router. This process is entirely free of additional labeled data and training overhead, which aligns perfectly with the lightweight and training-free philosophy of model merging. And it performs on par with MLP-based routers that require additional data and training.", "weaknesses": "1.\tLimited Contribution: The paper's core framework is heavily built upon prior work, particularly Task Singular Vectors (TSV). Consequently, the core contribution of this paper can be viewed as the addition of an efficient, dynamic \"selector\" module on top of the TSV framework. While this selector is well-designed and effective, it functions more as a task scheduling or selection component rather than a fundamentally paradigm for “adaptive”  model merging. This makes the overall contribution of the paper feel somewhat incremental.\n2.\tInsufficient Experimental Scope: The paper's experiments are primarily focused on ViT-based vision classification tasks and NLU tasks. However, a more challenging and highly relevant application for model merging is on large language models (LLMs) performing difficult generative tasks (e.g., code generation, long-form text generation). For such tasks, task vectors often exhibit more complex, high-rank structures. Previous research[1, 2, 3] has questioned that simply relying on low-rank decomposition may not perform optimally when dealing with these high-rank task vectors. Given that MASS's performance is highly dependent on the effectiveness of TSV, its performance on LLMs and generative tasks remains a critical, unverified question. The absence of these experiments raises questions about the generalizability of MASS and its effectiveness on cutting-edge applications.\n[1] Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models\n[2] Knowledge Grafting of Large Language Models\n[3] Delta Decompression for MoE-based LLMs Compression", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)