SpectR: Dynamically Composing LM Experts with Spectral Routing
Authors: William Fleshman
Benjamin Van Durme
Venue: COLM 2025 (2025)
Keywords: MoE
routing
merging
LoRA
adapters
experts
TL;DR: SpectR is an approach for routing and merging existing LoRA models per-token and per-layer, without any additional training or data.

Abstract: Training large, general-purpose language models poses significant challenges. The growing availability of specialized *expert* models, fine-tuned from pretrained models for specific tasks or domains, offers a promising alternative. Leveraging the potential of these existing expert models in real-world applications requires effective methods to select or merge the models best suited for a given task. This paper introduces SpectR, an approach for dynamically composing expert models at each time step during inference. Notably, our method requires no additional training and enables flexible, token- and layer-wise model combinations. Our experimental results demonstrate that SpectR improves routing accuracy over alternative training-free methods, increasing task performance across expert domains.

Published as a conference paper at COLM 2025
SpectR: Dynamically Composing LM Experts with Spectral Routing
William Fleshman & Benjamin Van Durme Johns Hopkins University will. flls.flehmmnjhhu edu
Abstract
Training large, general-purpose language models poses significant chal- lenges. The growing availability of specialized expert models, fine-tuned from pretrained models for specific tasks or domains, offers a promising alternative. Leveraging the potential of these existing expert models in real-world applications requires effective methods to select or merge the models best suited for given task. This paper introduces SPECTR, an ap- proach for dynamically composing expert models at each time step during inference. Notably, our method requires additional training and enables flexible, token- and layer-wise model combinations. Our experimental re- sults demonstrate that SPECTR improves routing accuracy over alternative training- free methods, increasing task performance across expert domains.!
1 Introduction
Language models (LMs) have increasingly become more capable in recent years with model size being an important factor in scaling performance (Kolachina et al., 2012; Hestness et al., 2017; Kaplan et al., 2020; Hoffmann al., 2022). Unfortunately, large models are inherently more resource-intensive, motivating the development of efficiency-boosting strategies such as knowledge distillation (Hinton et al., 2015; Gou et al., 2021), quantization (Shen al., 2020; Kim et al., 2021; Wu et al., 2022; Dettmers & Zettlemoyer, 2023), and pruning (Fang et al., 2023; Ma et al., 2023). Architecting models as a Mixture-of- -xperts (MoEs) has also become popular for state-of-the-art open-source LMs such as the DeepSeek (DeepSeek-Al Al et al., 2024) Mixtral (Jiang et al., 2024), and Qwen (Qwen et al., 2025) model families. MoEs gain efficiency by learning to activate smaller subset of parameters for each input, reducing computation while allowing for larger model sizes (Jacobs et al., 1991; Fedus et al., 2022).
Simultaneously, parameter efficient fine-tuning (Mangrulkar et al., 2022) methods such as adapter-tuning (Bapna & Firat, 2019; Houlsby et al., 2019), prefix-tuning (Li & Liang, 2021), and prompt-tuning (Lester et al., 2021; Liu et al., 2022) have emerged to allow efficient customization of pretrained models for expert-level performance in new domains or tasks. Low-rank adaptation (LoRA) (Hu et al., 2022) is one of the most widely adopted adapter- tuning approaches, resulting in the proliferation of thousands of expert models openly available in repositories such as huggingface (Wolf al., 2020). Given the abundance of these experts, many strategies have been proposed for merging models to improve Setasttatggi
multi-task capability (liharco et al., 2023; Yadav et al., 2023; Yu et al., 2024; Stoica et al., 2025).
Model MoErging pairs these merging strategies with routing mechanisms akin to MoEs (Yadav et al., 2024). Unfortunately, the majority of existing MoErging methods require training data for learning to route (Pfeiffer et al., 2021; Shnitzer et al., 2023; Huang et al., 2024; Tang et al., 2024) or custom training procedures for enabling adapter compatibility or gathering activation statistics (Chronopoulou et al., 2023; Diao et al., 2023; Belofsky, 2024; Fleshman et al., 2024). These constraints motivate the development approaches which require no data and allow for the use of externally sourced LoRA experts.
training-free MoËrging
"ode available https: //gi thub. com/wf wf leshman/spectr

---

Published as a conference paper at COLM 2025
One training-free option is the use of j-routing, which forgoes selecting specific experts and instead routes every input to all models (Caccia et al., 2023; Ostapenko et al., 2024). While dense linear combinations of experts can be successful (Wortsman et al., 2022; Chronopoulou et al., 2023; Ilharco et al., 2023), merging several LoRAs is especially problematic due to interference among adapters (Ortiz-Jimenez et al., 2023; Tang et al., 2024; Stoica et al., 2025).
Ostapenko et al. (2024) proposed Arrow routing, which uses the singular value decomposi- tion of LoRA adapters for crafting a prototype vector per expert. These prototypes are then used to score input vectors by measuring the magnitude of their dot product, routing then to the top-k experts with the highest score. We recognize Arrow as a significant contribution, while developing an improvement we refer to Spectral Routing (SPECTR) that addresses limitations in existing training- methods such as Arrow (Figure 1). Specifically, we:
Identify and confirm weaknesses in existing training-free routing strategies;
Propose SPECTR, our approach designed to explicitly address these challenges;
Measure the impact of adapter rank and task similarity on routing effectiveness, demonstrating improved routing accuracies with SPECTR across 4 LMs;
Quantify the impact of different routing strategies on multi-task performance, with SPECTR increasing accuracy by up to 15% over alternatives; and
Discuss trade-offs between SPECTR and other training-free free methods.
X
Expert Library
Bi
B2
B
B
B1
Top-K(IIh,I)
ht
h;
h3
h
Spectral Alignment
A1
U.S.V² SVD(B,A))
Ai
A²
A
Bi U₁ 4i S.V?'
X
Figure 1: SPECTR uses the SVD to transform adapters into equivalent representations capable of measuring the compatibility of new inputs without relying on expert prototypes routing networks. Vectors (x) are projected into low-rank representations (h;) using the eigenvectors (A,) of each adapter covariance matrix. The length of these representations measures the alignment of the input with directions of maximum variation induced by the expert in the space of possible inputs. Routing continues for the top-k compatible experts.
2 Background and Motivation
2.1 Adapters Expert Models
Adapters enable parameter efficient fine-tuning (PEFT) of existing models to new tasks or domains by adding small number of trainable parameters to existing models (Bapna Firat, 2019; Houlsby et al., 2019). Low-rank adaptation (LoRA) is one of the most popular and effective adapter-tuning approaches, accomplished by adding a low-rank matrix product to each adapted layer (Hu et al., 2022). LoRA's popularity is partly due to technologies such as Mangrulkar et al. (2022)'s peft library and (Wolf et al., 2020)'s s huggingface model repository, resulting in wide availability of expert LoRA models trained for a diverse set of tasks and domains Brrüel-Gabrielssonn et al., 2024). In this work, we use LoRA adapters to derive several expert models, each capable of superior performance their specific task. We
2

---

Published as a conference paper at COLM 2025
explore methods for dynamically selecting and combining these experts at inference time with no prior knowledge of the applicability or compatibility of individual expert models.
Mixture of Experts (MoEs) efficiently scales the capacity of LLMs by only activating subset of the model parameters for each input (Jacobs et al., 1991; Fedus et al., 2022). These models are jointly trained without explicitly assigning individual tasks to each expert. The MoE approach has gained significant use in practice, resulting in multiple state-of-the-an language models in recent years (DeepSeek-Al et al., 2024; Jiang et al., 2024; Qwen et al., 2025). The success of MoE has motivated the development of techniques for mixing task- specific experts such as Ye et al. (2022)'s ' Task-level MoE and several adapter-based expert approaches (Pfeiffer et al., 2021; Wang et al., 2022; Caccia et al., 2023; Ponti et al., 2023; Fleshman et al., 2024; Huang et al., 2024; Zadouri et al., 2024). Unlike these existing works, we require zero data or training to select and combine our mixture of task-specifi experts.
Task-Arithmetic isolates parameters responsible for a specific task by taking the difference between model weights before and after fine-tuning on the task (Ilharco et al., 2023; Fleshman & Van Durme, 2024). Fleshman & Van Durme (2024) demonstrate that these differences can be decomposed into LoRA adapters using the singular value decomposition (SVD). We note that these arithmetic- adapters allow for arbitrary expert models to be used for adapter mixing, but our experiments in this work focus only on traditional LoRA models.
2.2 Adapter Selection and Merging
Model merging is an established technique for deriving a single multi-task model from individual task-specific models (Matena & Raffel, 2022; Wortsman et al., 2022; Stoica et al., 2024). One popular approach for model merging is to create a linear combination of the expert model weights (Wortsman et al., 2022; Chronopoulou et al., 2023; Ilharco et al., 2023; Fleshman et al., 2024). While simple, these weighted averages can be less effective when using LoRA experts, especially the number of experts grows (Tang et al., 2024; Stoica et al., 2025). Tang et al. (2024) hypothesizes that LoRA adapters are less disentangled than full-rank fine-tuned models, causing deleterious interference when combined. Chronopoulou (2023) and Fleshman et al. (2024) observed that LoRA averaging worked best when adapters for different tasks were equally initialized, an approach inapplicable when using experts from multiple sources. These challenges have led to many techniques for mitigating adapter interference (Ortiz-Jimenez et al., 2023; Yadav et al., 2023; Tang et al., 2024; Stoica et al., 2025; Yu et al., 2024). Ortiz-Jimenez et al. (2023) and Tang et al. (2024) suggest fine-tuning strategies to discourage entanglement during expert training. Yadav et al. (2023) propose TIES, which resolves adapter interference through
al.
election process during merging. Stoica et al. (2025)'s KnOTS method performs an SVD over the set of adapters and merges them in the new shared subspace. Yu et al. (2024)ss 's DARE randomly drops and rescales parameters to create sparse approximations of experts. Our approach is compatible with most post-training strategies such as DARE, KnOTS, or TIES, but we find simple averaging with sparse selection of experts reduces complexity and works well in our experiments.
Routing is the process of selecting relevant experts for a given task or query (Yadav et al., 2024). Yadav et al. (2024) refer to the combination of routing and merging as MoErging to highlight the similarity to MoEs while distinguishing methods using existing individual experts. We adopt their MoErging taxonomy which categorizes routing by the type of dataset used to learn routing, input and depth granularity, and the expert selection and aggregation mechanisms (Yadav et al., 2024). For example, j-routing does not require training data because it aggregates all experts as a uniform linear combination for all queries and model layers (Caccia et al., 2023; Ostapenko et al., 2024). In contrast, Chronopoulou et al. (2023) and Fleshman et al. (2024) perform clustering over dense representations of training data and route new queries to a mixture of experts represented by similar clusters. Our work focuses on situations where there is no data available for learning to route. Therefore, include j-routing as one of the appropriate baselines for our MoErging experiments.
3

---

Published as a conference paper at COLM 2025
2.3 Problem Setting
We training-free model MoErging methods using experts derived from standard LoRA training procedures (Hu et al., 2022; Ostapenko et al., 2024; Yadav et al., 2024). We assume access to T task-specific LoRAs for the same LM architecture without access to any data associated with their training. For a query qi related to task I, we would like to select and merge small number of experts (including the expert trained for f) capable of addressing ft- We choose this setting to enable the usage of large and distinctly trained expert repositories that may not include associated data. We use LoRA for parameter efficiency (Hu et al., 2022) and choose sparse expert selection to minimize interference while increasing the chances of selecting the correct expert under imperfect routing (Tang et al., 2024; Yadav et al., 2024).
2.4 Arrow Routing
Arrow routing (Ostapenko et al., 2024) is the only existing work covered by the MoErging taxonomy (Yadav et al., 2024) that meets the requirements of our problem setting. Ostapenko et al. (2024) interpret the routing matrix from a slandard MoE model as collection of expert prototypes and construct their own prototypes using singular value decomposition (SVD). Recall that a rank-r LoRA for task maps an input E Ri to output x* Rdout with:
x
Wx + B,A,x,
(1)
where W E Rou are the original layer weights, and Bj E Rdout and A E R are the low-rank LoRA matrices (Hu et al., 2022) The SVD of the LoRA product:
U4 V; = SVD(B,At),
(2)
produces the matrix containing the left singular vectors U4 Rdoutt, the diagonal matrix of singular values S; E IRrxr, and the matrix of right singular vectors Vi E R such that:
U.s.V, = -B,A+
The right singular vectors eigenvectors of the LoRA parameter covariance matrix (B,A,)² B,A; and represent orthogonal directions of maximum variation induced by the LoRA expert in the space of input vectors The top eigenvector Vt = Vi(:, satisfies the following equation over possible unit length input vectors:
Vt argmax |x||== ||
B/A|x|12-
(4)
For this reason, Ostapenko et al. (2024) use vi as the prototype for expert inputs similar to vi tend to produce activations with larger magnitudes. Let Pe TTxdn be the routing matrix for layer e with row Pt[t] = representing the prototype for expert t. Ostapenko et al. (2024) then route an input vector x of layer i to the top-k experts using:
experts = arg top-k((P:x)).
Ostapenko et al. (2024) use rank-4 LoRAs for their experiments, but we hypothesize that Arrow routing becomes less effective the rank of experts increases. As the rank increases, the top eigenvector will capture a smaller percentage of the overall variation induced by that expert. We test this hypothesis and create an improved routing mechanism that leverages the entire spectrum of the LoRA covariance matrix without storing additional prototypes.
3 Spectral Routing
Spectral Routing (SPECTR) is our approach for dynamic token- and layer-wise composition of LoRAs, enabling improved multi-task adaptation of a base model without explicitly leam- ing to route from data. SPECTR includes separate initialization and inference procedures.
LoRA applies fixed scalar the matrix product, which we absorb into for cleaner notation.
4

---

Published as a conference paper at COLM 2025
3.1 Spectral Alignment
We use an identical initialization procedure for each layer of the base model. Let BN and A be the current layer's 's LoRA parameters for expert 1. We use the SVD from Equation 2 to compute U,, St, and V, This can be done efficiently using low-rank or probabilistic algo- rithms (Halko al., 2011; Nakatsukasa, 2019). Following from Equation we reformulate the adapter parameters as:
(6) and discard the original parameters. Observe that A contains scaled eigenvectors of the covariance matrix, the first of which is the prototype used in Arrow routing (Ostapenko et al., 2024). Instead of storing separate prototypes, we capture all covariance structure directly in the new adapter representation. We repeat this process for each layer and adapter.
B U, and A S.V.",
3.2 Routing Procedure
Given set of aligned adapters (BB) (1),,42),.... (B7, A7) } and token vector input to the current layer x, we compute a low-rank representation h; for adapter using:
h A/x.
Observe that if A was rank-1, then h, = Py [f]x from Equation 5. SPECTR takes advantage of the full rank of A and computes h, directly from the aligned adapter parameters without requiring the separate prototype. We then compute a routing score for adapter using:
hi
(8)
which measures the length of h; in the subspace induced by adapter t. Similar to Equation we expect related vectors to maximize this score because the subspace represents directions of maximum variation for vectors used to train adapter We route the top-k experts with:
experts arg (0 (s:).
(9)
3.3 Merging Procedure
SPECTR flexible enough to allow for more advanced merging methods such DARE (Yu et al., 2024), TIES (Yadav et al., 2023), or KnOTS (Stoica et al., 2025), but we use the same linear merging procedures as our baselines for fair comparison. We discuss alternative merging options and considerations in Appendix A. Given the set of selected experts indexed 1,...,k, we uniformly average their low-rank representations from Equation 7:
h
(10)
Similarly, we uniformly merge the selected experts remaining LoRA parameters:
1
B
B
(11)
Like Ostapenko et al. (2024), a softmax of the routing scores could be used as an alternative to uniform weights. Finally, we compute the output of the current layer for token with:
x Wx + 8h
(12)
SPECTR performs these routing and merging procedures on per-token and per-layer basis, enabling model expressivity on the scale of traditional MoE models.
4 Experiments
We measure the effectiveness of SPECTR routing strategy, testing our hypothesis that it improves routing accuracy over Arrow's rank-l prototypes. We measure multi-task performance across different methods and explore the confounding impact of task similarity.

---

Published as a conference paper at COLM 2025
4.1 Models and Adapters
We replicate our experiments across four popular instruction-tuned LMs chosen for their diversity in parameter count: Gemma-2B' (Team et al., 2024), Phi-3.5B (Abdin et al., 2024), and Llama-3B* and -8B6 (Grattafiori et al., 2024). LoRA adapters (Hu et al., 2022) are trained for each model using the peft library (Mangrulkar al., 2022). We apply rank-8 adapters to all linear layers of each model. A main motivation for SPECTR is to allow the use of externally sourced adapters, so we use default hyperparameters instead of optimizing them for each routing method. We prompt each model using the corresponding template from the huggingface library (Wolf et al., 2020) and include our training details Appendix Keeping with Ostapenko et al. (2024), use k = 4 for Arrow and SPECTR top-k merging.
4.2 Datasets
We measure performance on several pub- lic datasets covering diverse set of tasks: agnews' cola (Warstadt et al., 2019), dbpe- dia (Auer et al., 2007), hellaswag (Zellers et al., 2019), mnli (Williams et al., 2018), mrpe (Dolan & Brockett, 2005), qnli (Ra- jpurkar et al., 2016), qqp, and sst2 (Socher et al., 2013). We include each model's out- of-sample accuracy before and after fine- tuning on the datasets in Appendix C.
sgrews-
0.40
8.22
0:37
0.25
0:41
0:33
0.25
0.36
091
0.40
0.26
0:47
0.31
9.62
0.44
0.38
6.50
abpeda
0.22
0.26
0:24
0.17
9:27
0.22
0.24
Tellowon
0:37
0.41
0.28
0:38
mml
0:25
0.31
0.28
0.28
mppc
0.41
0.42
0.42
0.32
0.04
911
0:31
0.25
0.28
45p
0.23
0.2
0:28
0:35
0.42
0.29
We measure the cosine similarity between the LoRA weights of different tasks as a proxy for task similarity. Figure displays the pairwise similarity scores averaged over the four LMs. We include the cosine scores for individual models in Appendix D.
629
obpaddo
Nelloooed
Figure 2: Pairwise cosine similarities between task adapters averaged across four models.
We compute an overall similarity score for each task by averaging its pairwise cosine scores (Figure 3). The score for each task is consistent across the four LMs, with a slight drop for Llama-8B likely due its increased dimensionality. The dbpedia, mnli, and qqp adapters are the least similar to other tasks, while sst2, cola, and mrpc the most similar.
Task Similarity
I1Gemma-2B
Llama-3B
0.4
In
Phi-3.5B
I Llama-8B
0.3
IL
0.2
dbpedia mnli
agnews qnli hellaswag sst2
cola
mrpc
Figure 3: The average cosine similarity between a specific task and other tasks.
4.3 Routing Performance
We first compare strategies by framing routing as classification problem. Arrow and
SPECTR both compute per-token routing scores at each layer. We label each token from an
"nttps://huggingff http://nnnnaaaccc/mmaaaamm////// Instruct
Pnttps://huggingfaae
-B---Intruuttt
htttp://grroos.....uupp.ittttu/p corpus of news ar articees.hhe html

---

Published as a conference paper at COLM 2025
agnews cola abpedia hellaswag mnii
mrpc
qnli
qap
sst2
Figure 4: Per-token routing distribution using SPECTR for sequence of tokens from dbpedia averaged across model layers. Darker areas indicate a higher average routing score.
out-of-sample sequence as belonging to the adapter trained on the corresponding dataset. We visualize the classification of an example sequence in Figure where routing scores have been averaged across all model layers. We note that perfect routing accuracy at the token level would be unexpected, as only a subset of tokens are consistent across examples from the same dataset. We therefore focus on the relative difference in accu- racies between the two approaches. We mark each token correct if the ground truth adapter is selected among the top-4 experts and record the routing accuracy in Table 1. SPECTR results an average top-4 accuracy gain of approximately 4 percentage points, with individual task gains of up 24 points. We include top-1 accuracies in Appendix E.
Routing Accuracy
60
We plot routing accuracies against each task similarity score in Figure 5 and see that SPECTR's largest accuracy gains occur for the more unique tasks such as dbpedia. SPECTR seems to systematically choose al- ternative adapters for the most similar tasks such as mrpe and cola, resulting in accura- cies below random chance. We will explore if these inaccuracies are consequential on downstream performance Section 4.5, or if SPECTR's choices still perform well due to selected adapters having been trained for similar tasks. Arrow's accuracy appears independent of task similarity, albeit at significantly lower average accuracy.
40
x
(R2 = 0.71)
(R2 0.08)
0.2
0.3
0.4
Task Similarity
Figure 5: SpectR is more accurate at rout- ing than Arrow except for datasets with higher task similarity. The black dashed line indicates random routing accuracy.
Gemma-2B
Llama-3B
Phi-3.5B
Llama-8B
7
/
agnews
51.7
58.5
54.1
65.9
54.5
62.9
52.5
61.2
cola
48.4
33.9
47.4
30.2
53.7
41.9
49.6
32.7
dbpedia
47.5
69.6
52.1
74.6
52.6
73.6
49.5
73.9
hswag
50.1
51.8
52.2
53.8
50.0
53.6
48.6
49.9
mnli
43.7
59.6
46.6
59.9
38,0
56.7
45.8
61.0
mrpc
47.5
28.9
45.8
29.6
60.1
46.9
44.7
26.9
qnli
47.3
52.5
50.6
57.1
45.4
53.6
52.0
58.6
qqp
46.5
64.7
43.6
61.0
41.5
53.6
45.6
65.2
sst2
51.8
50.0
54.2
50.0
53.1
53.9
58.5
60.0
AVG
48.3
52.2
49.6
53.6
49.9
55.2
49.6
54.4
Table 1: Top-4 routing accuracies. SPECTR(
outperforms Arrow (/) on all four models.
We acknowledge the token labels small, but keep them for curious readers.
7

---

Published as a conference paper at COLM 2025
4.4 Rank Routing Effectiveness
We recall our hypothesis that Arrow prototypes become less meaningful as the adapter rank increases, reducing the sensitivity of routing scores to anything other than the single direction of maximum variation for each task. We quantify this effect by measuring the ratio of the routing score for the ground truth adapter against the highest routing score among all adapters. A ratio of 1 indicates perfect token routing. We plot these ratios for both methods as function of adapter rank in Figure 6. The SPECTR routing scores from Equation 8 take advantage of the extra dimensions in the intermediate representation as the rank increases, leading to better discrimination between adapters. In contrast, Arrow uses the rank-1 prototype, which we confirm results decreased ratio scores at higher ranks.
4.5 Multi-Task Performance
Now that we have established SPECTR im- proves routing, we confirm the strategy results in good multi-task performance in practice. We compare the base instruction- tuned model with Arrow and SPECTR, and we include j-routing as an additional strong baseline. Following similar work, we
0.8
Routing Score
0.7
0.6
measure the normalized accuracy to control for the difficulty of the task (Yadav et al.,
0.5
2023; Ilharco et al., 2023; Stoica et al., 2025). Normalized accuracy divides the achieved performance by the accuracy of the oracle model fine-tuned for the specific task.
2
4
6
8
Adapter Rank
We report the normalized accuracies in Ta- ble All three routing methods outperform the baseline models and SPECTR results in the highest average accuracy in all cases.
Figure 6: SPECTR( becomes increasingly more effective than Arrow (/²) at selecting the correct adapter adapter rank increases.
The magnitude of individual performance differences varied across models and tasks, with SPECTR's largest gain over μ and Arrow being 15.1 and 10.7 percentage points respectively.
SPECTR performed well on tasks even when routing accuracy was low, suggesting that low routing accuracies were partially caused by different datasets having similar tasks. High task similarity resulted in good performance
though the ground truth adapter was used less often. Interestingly, all routing methods and LMs perform poorly on hellaswag, even though the individual adapters fine-tuned for that task do well (Appendix C). This could indicate task interference (Ortiz-Jimened et al., 2023; Tang et al., 2024), possibly caused by the simple linear merge used by all three methods in our experiments (Appendix A).
Trade-offs
While better routing and multi-task performance are desirable, there is no free lunch when choosing between methods (Wolpert & Macready, 1997). We discussed the impact of adapter rank in Section 4.4, and here highlight three additional dimensions to consider when choosing between training-free strategies: storage cost, VRAM usage, and interference.
Storage costs for j-routing and SPECTR are identical if the experts are kept isolated for circumstances such as access-control (Fleshman et al., 2024). Otherwise, j--outtnn can have zero overhead by merging all experts into the base model ahead time. SPECTR represents the experts in different form after spectral alignment (Section 3.1), but the resulting matrices are the same size and produce equivalent products. Arrow routing requires the same library but includes the additional overhead of storing prototype vectors for each expert at every layer in the model (Ostapenko et al., 2024). The additional overhead is equivalent to storing extra adapter for every 2r experts when using LoRAs of rank
8

---

Published as a conference paper at COLM 2025
agnews
cola
dbpedia
hswag
mnli
mrpe
qnli
sst2
AVG
Gem-2B
Base
86.2
92.7
88.4
38.3
62.2
90.4
73.2
84.9
95.0
79.0
87.6
93.2
95.4
33.1
69.9
89.1
60.8
90.1
96.9
79.6
87.9
92.3
95.2
29.8
53.2
91.9
53.1
91.4
96.6
76.8
88.2
91.3
96.8
29.7
69.8
90.8
63.8
92.0
96.7
79.9
Llam-3B
Base
79.7
65.3
86.1
39.0
38.0
63.3
72.9
53.5
88.3
65.1
84.3
90.7
94.1
53.0
55.6
86.4
83.2
82.9
95.7
80.7
85.3
88.2
53.9
49.7
84.6
83.8
90.4
94.4
80.6
84.8
89.4
96.1
52.4
55.3
87.4
83.3
91.9
95.7
81.8
Phi-3.5B
Base
78.9
95.0
65.6
80.6
94.1
60.8
86.5
94.0
83.3
84.8
96.2
97.3
69.3
86.3
94.0
59.2
91.9
95.7
86.1
85.0
96.2
97.7
73.9
88.2
92.5
56.9
92.1
95.9
86.5
86.0
96.2
98.2
74.3
88.3
94.0
55.9
92.4
95.8
86.8
Llam-8B
Base
88.1
84.5
94.0
52.4
57.3
91.9
81.3
75.9
93.4
79.9
93.3
87.4
99.1
41.5
73.0
91.8
90.2
94.5
97.7
85.3
90.2
93.5
97.0
56.4
71.0
93.8
89.0
91.4
97.2
86.6
88.4
93.2
98.0
56.6
72.2
93.8
90.2
92.7
97.3
86.9
Table 2: Multi-Task performance across models and methods.
VRAM requirements are lowest for μ-routing because inputs are processed by single merged adapter. Arrow and SPECTR require that adapters be loaded in GPU memory as selection and merging occur the fly for each token at each layer. Arrow takes advantage of its rank-1 prototypes to reduce computation in the routing step, while SPECTR requires all adapters to compute low-rank representations before selecting the top-k experts. For layer with hidden dimension Iı, routing with SPECTR requires the FLOPS equivalent to forward pass of the layer for every h/r experts. Arrow allows for lt experts at the same cost.
Interference between adapters is most likely when using the dense selection of y-routing (Caccia et al., 2023; Ostapenko al., 2024). Both Arrow and SPECTR help mitigate interfer- ence by using sparse routing and are compatible with additional mitigations such as DARE (Yu al., 2024), TIES (Yadav et al., 2023), or KnOTS (Stoica et al., 2025) if necessary.
These considerations can help practitioners choose strategy under various constraints. Our experiments demonstrate superior performance with SPECTR, but Arrow might be preferable for its increased GPU efficiency as long as the adapters are of low enough rank. y-routing remains good option for smaller adapter libraries where routing is unnecessary.
Conclusion
In this work, we studied training-fres model MoErging approaches for integrating externally trained LoRA experts into expressive models capable of performing well in multi-task environment. We identified limitations with existing strategies, such as interference and sensitivity to adapter rank. These challenges led to development of SPECTR, new ap- proach for per-token, per-layer routing, requiring zero additional data or training to employ. We conducted experimentation across diverse set of tasks and popular LMs, demonstrating the ability of SPECTR to leverage the extra dimensions of higher rank adapters for increased routing accuracy over alternatives. SPECTR improved routing accuracy by 4 percentage points on average, leading to individual increases in task performance of up to 15%. We also explored the impact of task similarity on routing accuracy, and we found the results for each dataset were consistent across all four LMs under evaluation. SPECTR achieved the highest accuracies on the more unique tasks. We found that the alternate adapters selected by SPECTR for highly similar tasks still performed well regardless of imprecise routing. Finally, we discussed trade-offs to consider when choosing a routing strategy, including storage costs, VRAM requirements, and potential adapter interference. Overall, hope the effectiveness of SPECTR will allow for broader use of the abundance of expert models available for composition and improve the multi-task capabilities and performance of LMs.

---

Published as a conference paper at COLM 2025
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, et al. Phi-3 technical report: highly capable language model locally on your phone, 2024.
URL https: http:////aaxiioooraa/////// 14219.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for web of open data. In Karl Aberer, Key-Sun Choi, Natasha Noy, Dean Allemang, Kyung-ll Lee, Lyndon Nixon, Jennifer Golbeck, Peter Mika, Diana Maynard, Riichiro Mizoguchi, Guus Schreiber, and Philippe Cudré-Maurove (eds.), The Semantic Web, pp. 722-735, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. ISBN 978-3-540-76298-0..
Ankur Bapna and Orhan Firat. Simple, scalable adaptation for neural machine translation. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 1538-1548, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/vv D19-1165. URL httpps::/aaaaatholl.ootto org//19-116//
Joshua Belofsky. Token-level adaattaao of lora adapters for downstream task generalization. In Proceedings of the 2023 6th Artificial Intelligence and Cloud Computing Conference, AICCC '23, pp. 168-172, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400716225. doi: 10.1145/3639992.36366 URL https: /tttps://doi.rrrr http:/oo..ogg//1.. 114533339995//3 3639615.
Rickard Babri-Gabriessson, Jiacheng Zhu, Onkar Bhardwaj,, Leshem Choshen, Kristjan Greenewald, Mikhail Yurochkin, and Justin Solomon. Compress then serve: Serving thousands of lora adapters with little overhead, 2024. URL httpp:://arxi..org/////a///////s 00066.
Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni. Multi-head adapter routing for cross-task generalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //ttt::///peeeeviee.. forum?2id=qchhB115H
Alexandra Chronopoulou, Matthew Peters, Alexander Fraser, and Jesse Dodge. Adapter- Soup: Weight averaging to improve generalization of pretrained language models. In Andreas Vlachos and Isabelle Augenstein (eds.), Findings of the Association for Compu- fational Linguistics: EACL 2023, pp. 2064-2063, Dubrovnik, Croatia, May 2023. Associ- ation for Computational Linguistics. doi: 10.18653 v1//vv1/// 20t.2222...nnn2sseeeee.. https://aclanthol//
URL
org/2023, findings-eacl. 153/.
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, et al. Deepseek-v3 technical report, 2024. URL https: http///axvvvvv.//eee wwwaarivvogg/aaaaaaaa /rgg/bbs/2412 19437.
10

---

Published as a conference paper at COLM 2025
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In Proceedings of the 40th International Conference on Machine Learning, ICML'23.
org,. 2023.
Shizhe Diao, Tianyang Xu, Ruijia Xu, Jiawei Wang, and Tong Zhang. Mixture-of-domain- adapters: Decoupling and injecting domain knowledge to pre-trained language models' memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers), Pp. 5113-5129, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653 /v1 2023.acl-laac 280. URL https: /ttpps://aclle https://aalannthllog// org/2023. acl-long.280) 280/
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL https: hatba:s///allathhologyy org/105-5002/.
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang- Depgraph: Towards any structural pruning, The IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
20233
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. 23(1), January 2022. ISSN 1532-4435.
William Fleshman and Benjamin Van Durme. Re-adapt: Reverse engineered adaptation of large language models, 2024. URL https://arxii arg/abs/24055 15007.
William Fleshman, Aleem Khan, Marc Marone, and Benjamin Van Durme. Adapterswap: Continuous training of Ilms with data removal and access-control guarantees. In Proceed- ings of Conference on Applied Machine Learning in Information Security (CAMLIS) 2024, 2024. URL https://arxiv.. wwwili..rg//bbbb org/abs/24/4 08417.
Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. Int. J. Comput. Vision, 1296))1789-1819, June 2021. ISSN 0920-5691. doi: 10.1007 (0.1007/s11266 1100//111122--0-15552244 URL https:///oi....co http:////o..r//////.// 1007/1001111 1211-1121221212525551
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris Mc- Connell, Christian Keller, Christophe Touret, Chunyang Wu, et al. The llama 3 herd of models, 2024. URL https: //arxiv
org/ wrw.oogg////....ooooooo...oooooe abs//407 21783.
N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilis- tic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2): 217-288, 2011. doi: 10.1137/09077/10 URL https: http://o..oorr/////
org/10. 100 1137/890771805
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017. URL https: http:///axxvvoo//aaa//
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015. URL http::://aaxxv.orrr http//ab//115..255555
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22 Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
11

---

Published as a conference paper at COLM 2025
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Larous- silhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: /ttp:://eeeeeeeeeee//eeee net/ forum?id-nZeWKeeFFY?
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic loRA composition. In First Conference on Language Modeling, 2024. URL https:
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ttp:://///peeevieew wwwprreveeeeeeeeffruuuee net/forumidd 6t0Kwf8-jrj.
Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3:79-87, 03 1991. doi: 00.112//nec..1991..1.1
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Suphia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https://arxiv.. org/abs/2401 040888
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: / 08361.
Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer-only bert quantization. International Conference on Machine Learning (Accepted), 2021.
Prasanth Kolachina, Nicola Cancedda, Marc Dymetman, and Sriram Venkatapathy. Pre- diction of learning curves in machine translation. In Haizhou Li, Chin-Yew Lin, Miles Osborne, Gary Geunbae Lee, and Jong C. Park (eds.), Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 22- 30, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https:
/ https////aaaattoogg...oooo wwwp://ooooyo.og11111/1/ 1003//
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-
prompt tuning. In tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 1...165//112222.emmlll emnnp- URL httpp://aclanttoo.gg. org/2021 emnlp-main. 243/
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597 Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ 11116533v1/2221111 22w.aaa.a.loooooggg....... 353. URL https: http//aaaannnooooooooooo or orooorg///////
acl-long.. 353/.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th
12

---

Published as a conference paper at COLM 2025
Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ 100//53/11/200// w1/2022.acl-short.. URL https: https://aalaanthology http://ooggyoor//2......
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in Neural Information Processing Systems, 2023.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: -atte-ofthhe-at parameter-efficiee fine-tuning methods. https: //github. hommhuuginggfane ace/peft, 2022.
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS 22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Yuji Nakatsukasa The low-rank eigenvalue problem, 2019. URL https: www.w.rg////./.... 1905. 11490.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Thirty-seventth Conference on Neural Information Processing Systems, 2023. URL 0A9f2jZDGW.
net net/forumiid=
Oleksiy Ostapenko, Zhan Su, Edoardo Ponti, Laurent Charlin, Nicolas Le Roux, Lucas Caccia, and Alessandro Sordoni. Towards modular LLMs by building and reusing library of loRAs. In Forty-first International Conference on Machine Learning, 2024. URL https://openreviiw. wwtpw/wwwwwwweeeteeeeeeee nett:/orrummittooFFeeeasass
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion: Non-destructive task composition for transfer learning. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487-503, Online, April 2021. Association for Computational Linguistics. doi: 10.18653 /v6653/vl// 2021 eacl-mmain. 39. URL https: http://caaanttoooyyyoooooooo org/ 2021 eacl-main.39/.
Edoardo Maria Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining parameter-efficient modules for task-level generalisation. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Pp. 687-702, Dubrovnik, Croatia, May 2023, Association for Computational Linguistics. doi: 10. 10.18653 /ttp:////222..cmmmaaa.aa.. URL https:// eacl-main.44
aclanthology. org.///3..aclmmminn.....
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL /nttps://arxitt http:///aaxvv...a/aabee arg/abs/2412 15115.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2392,, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1 /1///////1/11111111 1264. URL /ittts://aacaatt hwwpooooggyoggggo661 org/D16-1264)
Sheng Shen, Dong Zhen, Jiayu Ye, Linjian Ma, Zhewei Yao, Asghar Gholami, Michael Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. Proceedings of the AAAI Conference on Artificial Intelligence, 34:8815-8821, 04 2020. doi: 10.1609/ www.aaa.vv... maai.v3440066409.
13

---

Published as a conference paper at COLM 2025
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets, 2023. URL https://arxiv.orrrr http://wwww.rg///////g///////// 15789
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic com com compoitionnality over sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. org/D13-1170000
Association for Computational Linguistics. URL
George Stoica, Daniel Bolya, Jakob Brandt Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training, In The Twelfth International Conference on Learning Representations, 2024. URL Forum?id-LEYuukvdhhg
net/
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with SVD to tie the knots. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //pper////pereeiieww hwt.:/ee/foorroeee fet.:
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization. In The Twelfth International Conference on Learning Representations, 2024. URL https: //pp:://///eeeeeeww/ net/ forum??d==InnvvWAmm
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, et al. Gemma 2: Improving open language models at practical size, 2024. URL htttts://arxiv wwwg/bb///aae 00118.
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Has- san Awadallah, and Jianfeng Gao. AdaMix: Mixture-of-adaplations for parameter- efficient model tuning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pp. 5744-5760, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: (011655//1/202::mmppmmiii //aclanthology org/2022.0c ean.p....lppppmmmmmmmmmmmmp emnnp-maing 388/.
URL https:
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019. doi: 10.1162/tacl. a.00200. URL https::tp//. httpp::///aaaaatth http://aaaaaanlooyyyoooo orgoorgg////oo /01 9- 1040//
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp- 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: (0.18633/v1/NT 11011. URL httlhhttss//caaaaaee /wwpaaaannnnnnyoo..coc www.ooog////... 1101/
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-ae natural language processing, 2020. URL https: //axxvv. http/bbs/19100077//
14

---

Published as a conference paper at COLM 2025
D.H. Wolpert and W.G. Macready. No free lunch theorems for optimization. IEEE Transac- fions on Evolutionary Computation, 171)677-82, 1997. doi: 10.1109 4205////23358888../
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbllthh, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965-23998, PMLR, 17-23 Jul 2022. URL nttps://proocediings mlr.press/ v1 wt//wwrsttaa22222 html.
Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, and Yuxiong He. Extreme compression for pre-trained transformers made simple and efficient. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In Thirty-seventh Conference on Neural Infor- mation Processing Systems, 2023. URL https:://peenrviiiee httplppereiiee.eee/orrrmmm htt.:////fooommmxxxxxa/////////
Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning, 2024. URL https: //wxxv..ivivvv////////e////aaa arrg/bbs/24n 07057.
Qinyuan Ye, Juan Zha, and Xiang Ren. Eliciting and understanding cross-task skills with task-level mixture-of-experts. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2567- 2592, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/ /v1 1 2022 /ttp//2........sseeeeeea findings-emmll..... org/2022. feiz2fffininnseemee
URL https: /tttat//acllnttoologyy
189/-
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024. URL https://openrevierv wwt.e/oorrmmiiccff@aill.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. URL https:/ /ttp::///pe/eveeeeee
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-48000 Florence, Italy, July 2019. Association for Computational Linguistics. doi: (1118633/11/P19-4222 URL http://accaattoolloor///////////
A Alternate Merging Procedures
As mentioned in Section 3.3, we structure Equation 10 and Equation 11 to make comparisons with baselines more fair. However, merging the experts in two separate steps is known to cause issues when the adapters were initialized with different seeds (Chronopoulou et al., 2023; Fleshman et al., 2024), as the merging procedure is sensitive to the ordering of rows and columns in each matrix. The issue alleviated by merging the final result of each expert instead of doing separate merges:
k
8
B h
(13)
k
15

---

Published as a conference paper at COLM 2025
and using & in place of Bh in Equation 12. This removes the need for experts to have rows and columns and allows for seamless merging of adapters with different ranks, the resulting products will have the same dimensions. This simple average is also easily generalized to more sophisticated merging algorithms.
the same permutation
B LoRA Training
All adapters were trained on single Nvidia A100 GPU with 80GB of memory. We used rank-8 LoRAs with LoRA a 16, LoRA dropout of 0.05, and applied adapters to all linear layers. We used supervised fine-tuning and trained with batch size of 8 using a learning rate of le-4 with a constant learning rate scheduler.
C Adapter Performance
We computed the raw out-of-sample accuracy achieved by each model before and after fine-tuning the model on the in-sample portion of each dataset (Table 3).
Gemma-2B
Llama-3B
Phi-3.5B
Llama-8B
Base
Base
FT
Base
FT
Base
agnews
80.6
93.5
75.3
94.5
74.9
94.9
83.0
94.2
cola
79.0
85.2
55.5
85.0
81.6
85.9
73.7
87.2
dbpedia
87.5
99.0
85.2
99.0
93.6
99.2
93.2
99.1
hswag
34.4
89.9
34.2
87.6
60.1
91.6
48.3
92.2
mnli
54.8
88.1
33.8
88.9
72.8
90.3
51.5
89,9
mrpp
75.7
83.7
54.1
85.5
77.8
82.7
75.0
81.6
qnli
67.5
92.2
67.5
92.6
56.5
93.0
75.6
93.0
qqР
89.2
47.8
89.4
77.5
89.6
68.2
sst2
90.2
94.9
84.1
95.2
89.9
95.6
88.7
95.0
Table 3: Task Performances before and after fine-tuning task-specific adapter.
D Task Similarities
We display the cosine similarities between adapters for all models in Figure
E
Top-1 Routing
We show the top-1 routing accuracies on out-of-sample data in Table
Gemma-2B
Llama-3B
Phi-3.5B
Llama-8B
7
/
agnews
18.4
21.9
20.5
27.0
20.5
25.9
19.2
23.7
cola
14.0
7.5
13.2
7.0
14.7
11.1
14.2
7.1
dbpedia
16.1
36.0
19.7
39.6
20.0
41.6
16.6
38.5
hswag
17.2
17.8
18.7
19.2
19.3
21.5
16.3
15.7
mnli
11.0
20.3
12.8
21.6
10.5
21.3
12.5
22.1
mrpc
11.6
4.9
12.5
6.0
18.0
11.4
11.4
5.6
qnli
13.4
14.4
14.0
15.6
14.4
17.2
15.6
17.4
15.9
26.3
11.3
21.8
12.9
20.4
12.8
26.1
sst2
16.3
15.4
18.1
14.7
17.8
19.1
20.3
19.4
AVG
14.9
18.3
15.6
19.2
16.5
21.1
15.4
19.5
Table 4: Top-1 routing accuracies.
16

---

Published as a conference paper at COLM 2025
+gnews
egnews
cdl
0:41
001
0:41
dbpedia
0bpedis
hel lasnng
0:38
0.48
0.25
hel Tellaseeg
0.37
0.48
0.24
0:33
0.25
mmpc
0:42
0.00
0.28
0:45
0:33
mmpc
0:41
0.61
0.27
0:48
0:32
gnni
0:34
3:46
0.24
0:40
0:30
an/
0:33
0.44
0.22
0.38
0.28
44p
0.26
0.30
50t2
0:37
3.49
0.26
0.43
0:30
0.49
0.40
0.31
55t2
0:37
0.51
0.25
0.42
0.29
0.50
0.41
0.30
Cala
Unpedia
Nellowong
mtpc
Spmawe
Gela
dipedla
helloowwg
mppc
698
(a) Gemma-20
(b) Llama-3B
agnews
agnews
cola
0:43
0olA
0:37
0:25
0:18
aswag
0:38
3.47
0.26
helaswag
0:34
0.46
0.21
mn/i
0:26
0.32
0.18
0.29
0.21
0.29
mmpc
0:43
mxpc
0:37
6n/)
0:34
3:44
0.23
0.38
0:28
0.45
on/)
0.29
0.42
0.19
0:35
0.24
0.41
4PP
RPP
5582
0.38
0.53
0.26
0.42
0:30
0.52
0.40
0.32
55t2
0.32
0.47
0.20
0.40
0.26
0.46
0.37
6.27
ophews
dippntaa
Petlaseig
MB2
spnows
dipenaa
hetlaseig
490
(c) Phi-3,5B
(d) Llama-8B
Figure 7: Cosine Similarities for all models.
17


Reviews:
[{"review_id": "XVhEehvxrW", "reviewer": "Reviewer_N8tG", "summary": "This paper presents SpectR (Spectral Routing), a training-free approach during inference for dynamically composing (MoErging) multiple task-specific LoRA fine-tuned variants (experts) of the same base LM. The key innovation lies in using SVD of LoRA matrices to create low-rank spectral representations that, as the authors claim, can equivalently measure experts' input compatibility without requiring additional training data, or introducing prototypes as in Arrow routing. Based on $\\mathcal{L}^2$ norm of these representations as routing scores, SpectR then averages outputs of the top-k most compatible experts at both per-token and per-layer granularities.\n\nThe authors demonstrate improvements on SpectR's average routing accuracy and performance when compared with Arrow routing, and how the effectiveness of SpectR scales better with adapter rank compared to Arrow routing. The authors provide additional discussions with valuable insights into the relationship between different methods' performance and efficiency considerations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The technical approach in this paper is novel. Their use of full SVD spectrum instead of just the top eigenvector (as in Arrow routing) is a meaningful technical contribution that, as the authors demonstrate, can better leverages the rank structure of LoRA adapters. \n+ The experimental validation is substantial, with extensive fine-tuning and MoErging experiments across 4 different LLMs and 9 diverse tasks, demonstrating improvements of SpectR in routing accuracy and performance. \n+ The training-free nature of SpectR makes it highly applicable for real-world scenarios, and the authors also provide valuable discussions with insights that could facilitate practical deployments. \n+ The paper is well-written with clear structure and nice visual aids that effectively communicate their approach.", "reasons_to_reject": "### 1. Limited Scope and Generalizability\nThere is a **significant mismatch between the claims made and the currently presented contents**, which may mislead readers about the method's broader applicability and impact.\nThe paper's title and abstract claim to address \"dynamically composing LM experts\" and refer broadly to \"specialized expert models, fine-tuned from pretrained models,\" suggesting a general framework for expert composition. However, the methodology and experiments are entirely focused on LoRA adapters without exploration of generalizability to other forms of LM experts based on full model fine-tuning or other PEFT methods.  The authors should either reformulate the scope to directly address LoRA-based experts, or provide additional validation across different expert types to support the general claims.\n\n### 2. Insufficient Theoretical Foundation\nThe paper would benefit from deeper theoretical analysis of why the spectrum approach should work better than Arrow routing or any other potential designs. Current explanation for why full spectrum outperforms Arrow's rank-1 prototypes remains largely intuitive rather than rigorously grounded.  \n\n### 3. Marginal and Inconsistent Improvement\nAs is demonstrated in Table 1&2, SpectR's improvement against Arrow routing and $\\mu$-routing is rather marginal and task-dependent (as discussed, suffering from task similarities). Based on performance reported in Table 2, the improvement of SpectR against $\\mu$-routing baseline is not statistically significant (with p=0.12 from paired t-tests, and win rate only 19/36=53%), and the win rate against Arrow routing is still only 24/36. These brings concerning limitations to the practical significance of methods proposed.\n\n### 4. Missing Details\nThe paper lacks several critical information and experiment results that could undermine the core claims made in the paper. Please find the questions raised in the following \"Questions To Authors\" sections.", "questions_to_authors": "+ In Section 3.3 line 183 the authors suggest a **softmax of routing scores** could be used to make a weighted average of  selected Top-K experts, which is logically intuitive and has indeed been widely adopted in the routing process of modern MoE LLM architectures. This should have become an important design choice for SpectR, as distribute weights to experts with higher scores would easily outperform uniform distribution. Yet it is **never experimented / discussed** in the following sections. Could you elaborate on why the simple uniform averaging works well for merging selected experts? Have you experimented with weighted averaging based on routing scores? \n\n+ The paper provides **no ablation studies**. It would be more solid if the authors could assess the specific accuracy/performance gain to see the contribution of each components of the proposed methodology, for example, comparing the introduction of SVD over other simpler alternatives like a direct $\\mathcal{L}^2$ norm on the intermediate hidden state between LoRA matrices [1]. \n\n+ In Section 4.2, cosine similarities between LoRA matrices for different tasks are used as indicator of task similarities. However the paper fail to specify whether during their fine-tuning process, for a same base LLM, the **LoRA adapters for different tasks are always initialized with identical weights**. It is stated in Appendix A that a initialization with different seeds would bring trouble to merging matrices. Nevertheless, when initialization differs across tasks, the observed cosine similarities may reflect initialization artifacts rather than genuine task relationships, which may fundamentally undermines the task similarity analysis and its conclusions about SpectR's performance patterns in Sections 4.3 & 4.5. \n\n+ In Table 1&2, the results focused on top-4 routing, and in appendices only top-1 is provided. These **choices of top-k** appears arbitrary without justification or sensitivity analysis across different k values. Is there a principled way to determine the optimal k for a given set of experts? What happens when the number of available experts increases, does the routing accuracy degrade, and are there scalability concerns? Adding related experiments and demonstrate results in plots would bring valuable practical insights on this critical scaling factor for deploying SpectR.\n\n\n\n### References\n[1] Lv, Ang, Ruobing Xie, Yining Qian, Songhao Wu, Xingwu Sun, Zhanhui Kang, Di Wang, and Rui Yan. \"Autonomy-of-Experts Models.\" arXiv:2501.13074 (2025).", "extra_scores": {}}, {"review_id": "SjdSD3sRqi", "reviewer": "Reviewer_iGcM", "summary": "This paper addresses efficient deployment of LLMS through a training-free routing method—Spectral Routing (SPECTR)—for merging LoRA adapters without reliance on additional data or custom training procedures. The authors motivate their method by pointing out a key challenge of the existing method called Arrow Routing. By leveraging spectral properties of each adapter’s low-rank update, SPECTR constructs prototype vectors that more accurately capture expert relevance, resulting in consistent routing accuracy improvements over Arrow.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written with a good review of the existing methods and the motivation building up to the problem statement is easy to follow. The flow of the paper is also to the point. The problem statement is clearly motivated and the proposed method has been shown to outperform Arrow Routing which has a restriction on the top-eigenvector being chosen the element for the routing decision variable.\n\n1. The proposed method instead uses the right singular vector as the element for the routing decision. Since it is full rank, it circumvents the issue in Arrow.\n2. The results clearly show that on average, Spectr outperforms Arrow except for datasets with higher task similarity. This is a good finding.\n3. The method is fairly simple to follow and easy to implement.", "reasons_to_reject": "My main concern with the paper is two fold:\n\n1. Can the authors discuss when would be need more ranks in the Arrow method and why is rank-4 not enough. That is a key question that needs to be empirically studied. I can understand that higher rank is a disadvantage in Arrow, but do we need more?\n\n2. Following the above point, the authors need to do more robust experiments on showing how their method outperforms the Arrow routing on tasks. I see Section 4.4 discusses one example, but this study needs to be segregated for tasks. When is high rank necessary for Arrow, can we keep low rank and instead use larger topK?", "questions_to_authors": "1. Some discussion on whether this method can be extended to problems with lesser constraints. For e.g. one of the constraints in the problem statement is that we know that a query is related to task t, what there is not a direct query to task 1:1 matching but instead,  we have multiple tasks for each query and we need to increase top-k, will the method still outperform Arrow?\n\n2. Not a question, but a suggestion - a lot of the paper space is used up just to describe related work, motivation and introduction. You could reduce that space and instead focus more on expanding the experiments on the need for higher rank in LoRA adapters.", "extra_scores": {}}, {"review_id": "FHopv3UM5t", "reviewer": "Reviewer_Efhf", "summary": "This paper introduces SpectR, a training-free framework for enhancing multi-task performance in language models (LMs) through dynamic routing and merging mechanisms for task-specific LoRA modules during inference. Unlike existing approaches like Arrow routing, which relies on rank-1 prototypes derived from singular value decomposition (SVD) of LoRA parameters, SpectR leverages the full spectrum of the LoRA covariance matrix to compute routing scores. This enables token- and layer-wise expert selection while avoiding interference between adapters. Results show that SPECTR improves routing accuracy by 4 percentage points on average, leading to individual increases in task performance of up to 15%.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces SpectR, a training-free method that employs the full spectrum LoRA covariance matrices via SVD. It overcomes the limitations of rank-1 prototype-based methods (e.g., Arrow) and improves routing accuracy/task performance through enhanced input variation modeling.\n2. It explores the impact of adapter rank and task similarity on routing efficacy, advancing understanding of expert composition dynamics in LMs.\n3. The method is compatible with interference-mitigation techniques (e.g., DARE, TIES), enhancing robustness for large-scale expert libraries.", "reasons_to_reject": "1.\tThe computational cost of per-adapter/layer SVD decomposition remains unmeasured, with critical metrics (e.g., inference latency, memory footprint, scaling behavior across ranks/expert counts) absent, limiting practical feasibility analysis.\n2.\tSpectR’s performance with large-scale expert libraries (e.g., hundreds of adapters) and cross-domain tasks is untested, leaving scalability under unexplored scenarios unaddressed.\n3.\tExperiments are limited to text classification and inference tasks (e.g., MNLI, SST-2). Performance on diverse task types (e.g., generation, reasoning) remains unverified, raising questions about SpectR’s applicability beyond the tested domains.", "questions_to_authors": "It might be better to quantify practical trade-offs, including metrics like inference latency, VRAM usage, and scaling curves (experts vs. performance) to guide deployment strategies.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes SpectR, a training-free method that dynamically composes LoRA-based expert models using spectral properties of adapter matrices during inference. The approach improves upon Arrow routing in both routing accuracy and task performance. Reviewers appreciated the clear motivation, technical contributions, and empirical validation across models and tasks. However, all reviewers raised concerns about limited generalizability beyond LoRA, which limits the method's practical value. While the authors provide some justifications, some concerns remain. Overall I lean toward accepting the paper, but only marginally.
Decision: Accept