Scalable Multi-Task Low-Rank Model Adaptation
Authors: Zichen Tian
Antoine Ledent
Qianru Sun
Venue: ICLR 2026 Poster (2026)
Keywords: low rank adaptation
multi-task learning
mixture of experts
model adaptation
parameter efficient fine tuning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We identify why multi-task LoRA fails at scale and propose mtLoRA with spectral-aware regularization, block-level adaptation, and fine-grained routing, outperforming SOTA by 2.3% with 47% fewer parameters.

Abstract: Scaling multi-task low-rank adaptation (LoRA) to a large number of tasks induces catastrophic performance degradation, such as an accuracy drop from 88.2% to 2.0% on DOTA when scaling from 5 to 15 tasks. This failure is due to parameter and representation misalignment. We find that existing solutions, like regularization and dynamic routing, fail at scale because they are constrained by a fundamental trade-off: strengthening regularization to reduce inter-task conflict inadvertently suppresses the essential feature discrimination required for effective routing. In this work, we identify two root causes for this trade-off. First, uniform regularization disrupts inter-task knowledge sharing: shared underlying knowledge concentrates in high-SV components (89% alignment on Flanv2→BBH). Uniform regularization forces high-SV components to update in orthogonal directions, directly disrupting the shared knowledge. Second, Conflict Amplification: Applying LoRA at the component-level (*e.g.*, $W_q, W_v$) amplifies gradient conflicts; we show block-level adaptation reduces this conflict with only 50% parameters. Based on these insights, we propose mtLoRA, a scalable solution with three novel designs: 1) Spectral-Aware Regularization to selectively orthogonalize low-SV components while preserving high-SV shared knowledge, 2) Block-Level Adaptation to mitigate conflict amplification and largely improve parameter efficiency, and 3) Fine-Grained Routing using dimension-specific weights for superior expressive power. On four large-scale (15-25 tasks) vision (DOTA and iNat2018) and NLP (Dolly-15k and BBH) benchmarks, mtLoRA achieves 91.7%, 81.5%, 44.5% and 38.5% accuracy on DOTA, iNat2018, Dolly-15k and BBH respectively, outperforming the state-of-the-art by 2.3% on average while using 47% fewer parameters and 24% less training time.

Published as a conference paper at ICLR 2026
SCALABLE MULTI-TASK LOW-RANK MODEL ADAPTA- TION
Zichen Tian
Antoine Ledent Qianru Sun
Singapore Management University
z chen. Sthenntiame tian. ....an2002ee... 20.22@phccs smu edu. ag, (aledent, qianrusun] @ edu. .g
ABSTRACT
Scaling multi-task low-rank adaptation (LoRA) to large number of tasks in- duces catastrophic performance degradation, such as an accuracy drop from 88.2% 2.0% on DOTA when scaling from to 15 tasks. This failure due pa- rameter and representation misalignment. We find that existing solutions, like regularization and dynamic routing, fail at scale because they are constrained by fundamental trade-off:: strengthening regularization to reduce inter-task conflict inadvertently suppresses the essential feature discrimination required for effective routing. In this work, we identify two root causes for this trade-off. First, uniform regularization disrupts inter-task knowledge sharing: shared underlying knowl- edge concentrates in high-SV components (89% alignment Flanv2- Uniform regularization forces high-SV components to update in orthogonal direc- tions, directly disrupting the shared knowledge. Second, Conflict Amplification: Applying LoRA at the component-level (e.g.. W W.) amplifies gradient con- flicts; show block-level adaptation reduces this conflict by 76% with only 50% parameters. Based on these insights, we propose mtLoRA, a scalable so- lution with three novel designs: 1) Spectral-Aware Regularization to selectively orthogonalize low-SV components while preserving high-SV shared knowledge, 2) Block-Level Adaptation to mitigate conflict amplification and largely improve pa- rameter efficiency, and 3) Fine-Grained Routing using dimension-specifi weights for superior expressive power. On four large-scale (15-25 tasks) vision (DOTA and iNat2018) () and NLP (Dolly-15k and BBH) benchmarks, mtLoRA achieves 91.7%, 8 81.5%, 44.5% and 38.5% accuracy on DOTA, iNat2018, Dolly-15k and BBH respectively. outperforming the state-of-thee the--of-fhherrr- by 2.3% on average while using 47% fewer parameters and 24% less training time. Code available at https: ttt.://gttuuuuu.. com dwem//ommmmmmmmmmmmmm//m ICLR26 mtLoRA
INTRODUCTION
Low-Rank Adaptation (LoRA) (Hu ct al. 2021) has emerged the de-facto standard of Parameter- Efficient Fine- Tuning (PEFT) for pre-trained Visual Transformer (ViT) models, thanks to its minimal
trainable parameters, zero inference latency overhead, and modular deployment (He et al. 2022
Zhang et al. 2023 Dettmers el al. 2023 Han et al. 2024 Ge et al. 2025 Tian et al. 2025 Zhu et al.
2025
2024). Although LoRA achieves remarkable performance single-task adaptation ZZang et al.
2023 Liu et al. 2024 Tian et al. 2024b), real-world applications usually need scalable
tasti-task low-rank adaptation, i.e., using multiple task-specific LoRA modules (on top of one backbone model) to handle large number of tasks (15-25+) simultancously (Stoica et al., 2025 Wu
al. 2024a| Ma al., 2018). For instance, language models need to process multiple tasks (e.g.. mathematical reasoning, legal analysis, and ethical questions) concurrently (Hendryeks et al. 2020
Zhao et al. 2025). and vision models need to adapt across multiple spectrums (e.g.. optical and radar imagery) (Tian et al. 2024b). Training large foundation models from scratch for domain-specinn applications presents fundamental challenges, particularly in domains with limited data availability and severe data imbalance issues (Wang et al. (022),, further motivating parameter-efficient multi- task approaches. However, multi-task low-rank adaptation suffers from catastrophic performance degradation the number of tasks scales up (Tian et al. 2024a Wu et al. 2024a Stoica et al. 2025).

---

Published as a conference paper at ICLR 2026
38
2.70
Swoet Spot + 77%
85.4
Alggmmnt
2,5
Companertiani
Block-Lreel
BBH Accuracy
2.68
Percentage (50
80-
Impertanco
2,0
Confler
-2.66
Entrop
60-
54.0
Dendley
1.5
Confler sDaN
2.64
40
1.0
36.
Accursiyy 1
2.62
24.0
27.1
0.5
Routing Entropy 4
35
2.60
0.0
0.00
0.25
0.50
0.75
1.00
1op-10%
10-50%
50-100%
-0.7
-050
-0.15
0.00
0.25
0.50
6.75
Regularisation Strongth (A)
Singular Value components
Cosine Similarity
(A)
(B)
(C)
Figure 1: Motivating observations for our three novel designs. (A) Orthogonal regularization introduces trade-off between conflict reduction and routing uncertainty. Specifically, through orthogonal regularization, the model accuracy (blue eurve) peaks at X = 0.25 (+1.7%) but degrades at X = 1.0 (-1.8%), due to increased routing uncertainty (i.e., Routing Entropy in orange curve). (B) Shared knowledge concentrates high-SV components. Specifically, high-SV (top-20%, highlighted) shows 89% inter-task alignment and encodes 54% of total singular values, while low- SV (50-100) (00%) shows only 3% alignment with 22% of singular values (detailed in Sec. 4.1 This motivates spectral-aware regularization: preserve high-SV shared knowledge, only orthogonaliie low- SV components. (C) Block-level LoRA adaptation reduces gradient conflicts. Specifically, block- level adaptation achieves higher gradient alignment between tasks (measured by cosine similarity, -0.013+0. 169) as compared to component-leve adaptation ( 40.00544..201), accompanied by a +2. 12.1% % accuracy improvement (91.2% vs. 89.0% in Table 4.. See Sec. 4.3 for detailed experimental setups.
The core challenges are two kinds of misalignment: parameter misalignment and representation misalignment (Stoica et al. 2025 Han et al.) 2024). Specifically, parameter misalignment means different LoRA modules have conflicting weight updates (i.e., gradient of weights update in opposing directions). To address this, existing methods use regularizations to enforce orthogonality across LoRA parameters (liharco et al., 2022: Yadav et al. 2023 Yu et al. 2024a). Another challenge is representation misalignment, meaning that LoRA modules' output features divergent (Stoica et al. (025). Existing solutions use dynamic routing to weigh LoRA's output features, i.e., by sparse gating (e.g., select top-K activated LoRA) or soft routing (i.e., weighted combining all LoRA modules) (Wu et al. 20244] WWi et al.] [2025] Tian et al.) 2024a). However, these methods fail to scale to large numbers of tasks. An intuitive solution is to combine both approaches to leverage their complementary strengths. We find that, while this combination improves performance, it quickly reaches Pareto frontier: stronger regularization reduces gradient conflict but impairs routing effectiivness. Specifically, as shown in Figure [A), when training LoRA modules with dynamic routing on a multi-task NLP scenario (trained on 16 experts Flan-v2 and evaluated on BBH), increasing the regularization strength À initially improves the accuracy to 38.0% (+1.7% at A= 0.25). However, further strengthening regularization increases the routing uncertainty (i.e., routing entropy increases from 2.6 to 2.7) and causes accuracy to drop by 1.8%. This trade-off fundamentally limits scalability.
This raises a key question: why does this trade-off exist? We identify root causes stemming from how LoRA modules are treated and placed, respectively.
First, uniform regularization disrupts knowledge sharing across tasks. The key of multi-task learning is to share underlying knowledge (i.e., transferring inductive bias in (Caruana 1997) across tasks. However, uniform regularization disrupts this. We quantify the shared knowledge across tasks in [ Results show that the shared knowledge concentrates in high-SV components, i.e. high-SV (top-20%) contains 89% inter-task alignment. meanwhile encodes 54% of total singular values: while low-SV (50-100%) contains only 3% alignment with 22% of singular values (see 1.c Uniform regularization treats all spectral components equally, forcing orthogonality on high-SV and pushing them to update in different directions, which directly corrupts the knowledge sharing. This motivates our spectral-aware regularization, i.e., orthogonalize low-SV noise, while preserving high-SV shared knowledge. Second, applying LoRA to component-level matrices amplifies gradient conflicts. When multiple LoRA modules adapt individual weight matrices (e.g., Wg. W.) for different tasks, their gradients exhibit stronger misalignment. Figure CC) validates this: component-level adaptation yields an average gradient cosine similarity of -0.054=0.201 between task pairs, indicating substantial conflict. In contrast, block-level adaptation (applying LoRA to
2

---

Published as a conference paper at ICLR 2026
entire attention and FFN blocks) reduces this to 00.01 (76% reduced conflict). This improved gradient alignment leads to +2.2% accuracy gain (91.2% vs. 89.0%, 4
Given these insights, propose mt LoRA-a novel method designed for scalable multi-task low-rank adaptation. We introduce three key designs: 1) spectral- -awae regularization, 2) fine-grained routing, and 3) block-level adaptation. First, we design spectral-aware regularization. Our approach applies strong orthogonalization to low-SV components (empirically, identified as "noise") while preserving high-SV components (the "signal"). We achieve this through a weighting function w(() - exp(-o/o), where a is the singular value and ō is the average. For noisy low-SV components (i.e. < σ). the weight wr(r) approaches 1, enforeing strong orthogonality. For discriminative high-SV components (encoding shared underlying knowledge, i.e., o > σ). the weight w(r) approaches 0, preserving inter-task knowledge sharing. Second, we propose fine-grained routing. Unlike standard routing that assigns each LoRA a scalar weight (forcing uniform combination across all dimensions), we learn a router network to produce vector IT, E R2 to weigh each LoRA, where g is the number of groups. Each group contains d/g dimensions. This addresses the observed heterogeneous conflict pattern, allowing different feature subspaces to use different combinations of task experts. For example, for a complex prompt, "creativity" subspace can assign high weight to a brainstorming LoRA, while a "factual" subspace can simultaneously assign a high weight to a QA LoRA. Our fine-grained routing breaks the constraint of uniform combination and is stabilized by boad-balancing loss to prevent routing collapse. Third, we propose block-level adaptation. Rather than adapting individual matrices within a block, we apply the combined LoRA update as a parallel path that bypasses the block's 's internal computations. Consistent with Pre-LN architectures, for block F' and its LayerNorm LN, our adapted output hout hin (l,) + (h,N( )). The LoRA adapter A operates on the same normalized input as F, mitigating multiplicative gradient conflicts and ensuring architectural consistency.
We validate our LoRA on four large-scale multi-task benchmarks spanning both vision and language domains: DOTA (15 tasks), iNar2018 (25 tasks), Dolly-15k (16 tasks), and BBH (27 reasoning tasks). We make three key findings. 1) mt LoRA is more scalable. Existing methods face severe collapse when task number increases, For example, naive averaging leads to catastrophic DOTA (5-115 tasks) and 87.0%=0.33 on iNat2018 (1~1100 tasks in Supplementary Sec C.1). Our LoRA mitigates this collapse, achieving 64.0% average accuracy across all four benchmarks, outperforming HydraLoRA by 2.3% on average (Table 5 Ablation studies confirm 2) all three key designs contribute significantly. Specifically, block-level adaptation contributes +2.1% % with 50% fewer parameters (largest gain), spectral-aware regularization and fine-grained routing introduce consistent improvements, together improving from 61. 61.1% to 63.9% (+2.8% overall). The improvements are consistent across vision (+2.6% on DOTA/INa12088) and NLP (+2.9% on Dolly- 15k/BB,, Table Finally, we analyze performance across task difficulty levels and find 3) mtLoRA consistently outperforms SOTA across all difficulty levels. On BBH, we categorize 27 tasks by average accuracy: Easy (>50%), Medium (30-50%), and Hard (<30%). mtLoRA consistently outperforms SOTA across all levels: +1.6% on Easy, +3.5% on Medium, and +0.4% on Hard tasks (Table 6). demonstrating broad applicability across diverse task difficulties. Remarkably, these gains come with 4) improved parameter and training efficiency: due to the block-level adaptation design, our mt LoRA achieves +2.8% performance using only 47% parameters and 24% less training time (Section 4.5)
degradation: +2.0%%
Our contributions are three-fold. 1) We provide the first systematic analysis of why existing multi-task LoRA methods fail at scale. We identify that inter-task alignment (shared knowledge) concentrates in high-SV components; uniform regularization disrupts this, explaining the fundamental regularization- routing trade-off that prior work overlooked. 2) We make three key technical contributions in mechanistic understanding. Spectral-aware regularization selectively orthogonalizes low-SV noise while preserving high-SV shared knowledge. fine-graine routing assigns dimension-specifie weights instead of scalar weights, and block-level adaptation mitigates gradient conflict amplification while using 50% fewer parameters. 3) We demonstrate consistent improvements at large-scale (15-25 tasks) across both vision (DOTA, iNat2018) 88 and language (Dolly- BBH) benchmarks, achieving up to 2.8% absolute performance improvement over state-of-the-art while using 47% fewer parameters and 24% less training time, making scalable multi-task LoRA practical for real-world deployments.
3

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Multi-Task LoRA Adaptation. Multi-task low-rank adaptation (LoRA) aims to compose multiple low-rank updates (Hu et al. 2021) to handle various tasks, simultaneously (Caruana 1997p. The key challenge is the misalignment between low-rank updates (i.e., LoRA experts). Existing solutions can be categorised into regularization and routing methods, respectively tackling the parameter and representation misalignment.
1) Regularization methods address the parameter misalignment. Existing methods impose reg- ularization to enforce orthogonality across LoRA parameters (Ilharco et al.) [022] Yadav et al.) 2023 Yu al. 2024a). For instance, Task Arithmetic (Ilharco 2022) linearly combined task vectors; TIES Merging (Yadav et al. 2023) resolved sign conflicts through majority voting; and DARE (Yu et al.
2024a) applied stochastic masking to enforce sparsity. However, these methods
input-independent and ignore input dynamics.
2) Dynamic routing methods address representation misalignment. These methods typically learn networks to route LoRA experts with learned weights. For instance, hard gating networks (i.e., selecting the top-K LoRAs), and soft routing networks (i.e., combining all LoRA modules with weights). MoLE [Wu et al. 2024a) extended this to LoRA adaptation, introducing Top-K K routing and balancing losses prevent imbalanced expert selection. HydraLoRA (Tian et al.] (2024a) combined routing with an asymmetric LoRA structure (i.e., a single shared A, with multiple task-specific Bş). LoRAMoE forced some LoRA experts to maintain the foundation model's knowledge to protect against catastrophic forgetting. Recent work Hu et al. (2025) resolves task conflicts in representation space, but on hard- parametereteer shared backbones.
However, these approaches typically treat regularization and routing independent solutions. As preliminary study reveals, there a fundamental trade-off between regularization and routing, which hinders task scalability. Our work is the first identify and resolve this trade-off, enabling efficient and scalable multi-task low-rank adaptation.
Multi- Task LoRA Placement Strategies. In multi-task low-rank adaptation, prior works explored where to plug the LoRA modules into transformers. Ada-Merging Yang et al. (024) and MoLE (Wu et al. 2024a) found that uniform treatment across layers is suboptimal, so they assigned different weights for LoRAs in different layers. MTLoRA (Agiza et al. 2024) placed task-irrelevant modules at shallow layers and task-specific modules at deep layers in the network. MixLoRA (Wu et al. 2024b) only inserted LoRA into FFN blocks, avoiding attention layers completely. Hydral.oRA (Tian et al.) 2024a) applied LoRA only to Q and V projection layers (i.e., Wg and W.). However, all these methods apply LoRA to individual, component-level weight matrices (Wa. W.. or linear layers within the FFN block). In contrast, we apply LoRA at the block level, as a parallel adapter around attention and FFN blocks. This approach decouples the LoRA update path from the main block's internal computations, hence mitigating the amplification of gradient confficts.
3 METHOD
In this section, we detail our three designs in tackling the scalability challenge of multi-task low-rank adaptation. Firstly, we formulate this task, scalable multi-task low-rank adaptation, in Sec. 3.1 Then, detail the three novel designs in LcLoRA: spectral- aware regularization in Sec. 3.2 fine- grained routing in Sec. 3.3 and block-level adaptation in Sec. 3.4 We illustrate the overview of our architectural innovations Fig. 2
3.1 TASK FORMULATION
We formulate the challenge of scalable multi-task low-rank adaptation. Specifically, consider a frozen pretrained model with parameters W/(D) and a set of tasks, we introduce N low-rank updates [ to the model, where each A (xxx) A, is parameterized by down-projection A; E Rd and up-projection B, E Rdxr with rank & d (hidden dimension] [
During inference, the multi-task
'Hydra Tian et al. 2024a) shows that each low-rank update, especially B matrices, implicitly encodes task-specific knowledge.
4

---

Published as a conference paper at ICLR 2026
x Scalar multiplictionn
W;
ToRA
FFN Block
LoLo
MLP Router
Layer Norm
FFN Block
Scalar Routing
Attention
Matrix multiplication
MHA Block
mLLLOA
Mx
LayerNorm
MHA Block
B
MLP Router
Component-Level
Block-Level (Ours)
Fine-Grained (Ours)
(A)
(B)
Figure 2: The architectural innovations of mt LoRA. (A) Block-Level Adaptation. The LoRA update is computed in a parallel path that bypasses the block's internal non-linearities, mitigating gradient conffict amplification. This path takes the same LayerNorm output as the main block. (B) Fine-Grained Routing. Within the parallel path, a router (lightweight MLP) generates dimension- specific weight vectors to compose task experts, allowing different feature subspaces to use different LoRA combinations.
low-rank adaptation combines these low-rank updates:
f(x) fwio (x) + ( [m(s)-A((s)) A,())
(I)
where (x())) E IR are scalar routing weights. In this work, we focus on the scalable challenge, i.e., N scale-up to large number, that leads to severe conflicts between low-rank updates. Our three designs address this challenge.
3.2 SPECTRAL-AWAR REGULARIZATION
Motivation from HydraL.RRA's Structure. Our method builds upon Hydral.oRA's asymmetric structure (Tian et al. 2024a). where a shared down down-projection A captures task-agnostic representations while multiple up-projections B, encode task-specific information. Since A is shared, the task conflicts arise from the B; matrices. Importantly, since A? A2 = A ATB, B. B,A with shared A, orthogonality between B; and B, directly ensures orthogonality between full LoRA updates A, and A,. As discussed in the Introduction, shared knowledge concentrates in high-SV components, while low-SV shows minimal inter-task alignment. Standard orthogonal regularization forces orthogonality on all components equally, forcing tasks to update in different directions and disrupting knowledge sharing. To address this, we propose opectral-aaar regularization: only orthogonalize low-SV components, preserve high-SV shared knowledge. Concretely, for each B, E Rdxr, we apply SVD to obtain B, U,E.V. V with singular values (ax), and construct re- re-weighted matrices Bí that emphasize low-SV components via weighting function w(o) = exp(-a (op((a/o)) / where Ø is the mean singular value. Note that this weighting is continuous and adapts to each B, matrix-the percentile bands (e.g., top-20%) mentioned in our analysis (Fig. [T) are for visualization only, not fixed thresholds in the implementation. The spectral-aware loss is:
Lapoccral
Σ IIBB)) $C
This loss encourages orthogonality primarily among the low-SV components, while preserving the high-SV shared knowledge essential for multi-task learning.
3.3 FINE-GRAINED ROUTING
Unlike conventional dynamic routing that assigns one scalar weight per LoRA (s,, E R), we assign dimension-specifie weights. We partition the feature dimension d into g groups, where g denotes
5

---

Published as a conference paper at ICLR 2026
the number of groups (not group size). Our router network outputs weight vector II; R for each LoRA module using softmax normalization (soft routing), as shown in Fig. B). For example, g=1 corresponds to module-wiss routing (one scalar weight per LoRA), while 768 provides full dimension-level routing (one weight per dimension) 2
The low-rank update in Eq. becomes:
where @ denotes grouped clement-wise multiplication. This allows different feature subspaces to different LoRA combinations. To prevent routing collapse, where the router favors only a few experts, add load-balancing auxiliary loss Lalannc to encourage uniform distribution of routing weights across all experts (Wu et al. 2024a). The total loss is formulated as L = Lusk + A Espectaal + A2 balance.
3.4 BLOCK-LEVER ADAPTATION
Instead of adapting individual weight matrices (W,. q. Wo) inside blocks, we adapt LoRA at the block- level by adding it as parallel path after attention FFN blocks. As illustrated in Fig. [[), for frozen block (FF) (e.g., Multi-Head Attention FFN), the adapted output is:
='=+ ((F) (LN(z)) + A (LN(x)),
where A -E= II; A. is the combined low-rank update, and LN is the LayerNorn In this way, the LoRA update path is decoupled from the internal non non-linearities (e.g., Softmax) of the main blocks, hence mitigating the amplification of gradient conflict.
Why does block-level adaptation work? Compared with conventional LoRA, the block-level adap- tation avoids gradient conflicts in attention. Specifically, traditional LoRA is attached to linear layers (weight matrices W W.. W1, W2). The gradients will flow through the Softmax in attention, as shown in Figure ((A). This creates cross-token dependencies: changing attention to one token affects all other token positions. This effect amplifies task conflicts. For example, consider the input "The bank is steep". For finance tasks, the model needs high attention on phy tasks, the model needs "bank"->"river", These conflicting attention patterns interfere through Softmax. In traditional LoRA, updating B to increase decreases "bank"-"rivee" "riveer" attention due to Softmax normalization, as they compete for the same probability mass. Our block-level adaptation avoids this competition. The two adapters can add the "money" and "river" feature independently to "bank" representation.
For geogra-
attention automatically
4 EXPERIMENTS
We first provide an experimental setup in Sec 4. and then reveal the core challenge (Le., multi-
task collapse) preliminary study Sec 4.2 Given this understanding, we ablate the working
mechanisms of our three key designs in Sec 4.3 and perform an SOTA comparison in See 4.4 Additionally, we discuss the applicability and limitations of our designs Sec 4.5
4.1 EXPERIMENTAL SETUP
Benchmarks. We evaluate mt LoRA on four benchmarks: DOTA (Xia et al. 2018 (15 tasks),
iNat2018 (Van Horn et al. 2018) (25 tasks), D5lly-15k (Conover et al. 2023) (16 tasks) with
evaluation on MMLU, and lany2 (Longrre et al. 2023) subset with evaluation on Big Bench Hard (BBH) (Suzgun et al. 2022). We compare with of methods including HydraLoRA (Tian) et al. 2024a), MMo ooAA eWu et al. 2024a), and LoRAHub (Huang et al. 2003), with LoRA rank 6. Detailed experimental settings and compared methods' implementations provided in Appendix B.2
"The router outputs g weights per Lo For intermediate g (e.g., g=32), each weight is repeated d/g times to cover its dimension group
We apply LoRA after LayerNorm, consistent with the Pre-LayerNNorm (Pre-IN) architecture of VIT (Dosa vitskiy et al. 2020) and I.I Lmma22-B (Touvron al. 2023

---

Published as a conference paper at ICLR 2026
Implementation of Our Method. Our method is based on HydraLoRA (Tian al. 2024a) archi- tecture, where a single matrix A is shared across all tasks, and tasks are learned through diverse apply SVD to the task-specific B matrices pute Espestral for efficiency. This structure allows us to directly regularize spectrums of B, (instead of entire LoRA) to control task-specific conflicts. We apply SVD the task-specific B matrices once per epoch to compute the spectral-aware loss. The loss is Espectral = >iiei [I())) B where By U is temporary matrix constructed by re-weighting the singular values of B, U,E,V? with Eyg /w(ag) 0k where w(o) = exp( aa/o to penalize low-SV compo- nents. The router is a 2-layer MLP with output dimension N x where N is the task number and y is the number of groups. It takes the mean-pooled hidden states as input and applies softmax normalization to produce routing weights. For y > each weight is broadcast by repeating d/g times before element-wise multiplication with LoRA outputs. The total loss includes a load-balancing term Lblannce to prevent routing collapse. All LoRA modules are applied at the block level as parallel adapters, consistent with a Pre-LN architecture (such as AdaptFormer (Chen et al. 2022)).
B, matrices. Specifically,
per epoch to
4.2 CHALLENGE OF MULTI-TASK COLLAPSE
We quantitatively analyze the multi-task collapse in Appendix C.I Table S3 (due to page limit). Results show that performance degrades catastrophically across all datasets: 2.00% from 5 to 15 tasks on DOTA, 87.0%-0.55% from I to 100 tasks on iNat2018, and 46. 16.1%-~16.00 from 4 to 16 tasks on Dolly-15k. Meanwhile, the conflict score (see Appendix B.2 for calculating details) reaches 97. 97.9%, 99,7%, and 64.7%, respectively.
4.3 ABLATION STUDIES OF OUR METHOD
We conduct ablation studies reveal how three designs work. We answer three key questions:
Q1: How does spectral-aware regularization resolve the routing-regulaaization trade- off? Figure AA) shows that when training LoRA modules with dynamic routing large-scale multi-task scenario (Flan-v2 training evaluated on BBH, with 16 experts), strong orthogonal regularization (A 1.0) de- grades accuracy despite reducing conflicts. Why? Table [[ orthogonal regulariza- tion achieves only 20.5% on DOTA with- out routing but jumps to 89.8% with dy- namic routing. Without routing, all tasks contribute equally (uniform 1/N weight- ing), causing orthogonalized parameters to cancel out. Dynamic routing selectively activates task-relevant LoRAs, making thogonalization beneficial.
Table I: Orthogonal regularization requires dynamic routing. Without routing (uniform 1/N weighting), or- thogonal regularization achieves only 20.5% on DOTA; with dynamic routing, it improves to 89.8%. Sparsity regularization harms performance in both settings. All results in accuracy (%)f.
Method
DOTA
iNa12018
Avg.
Uniform Routing'
Hydral .oRA Loydral.oRA
18.0
8.5
13.3
+ Sparsity Reg.
16.5
7.2
11.9
However, even with routing, stronger regu- larization (A = 1.0) degrades performance by 1.8% compared to moderate regulariza- tion (A = 0.25). Figure A shows rout- ing uncertainty (entropy) increases from 2.62 to 2.67, indicating the router becomes less decisive. Furthermore, we observe that reaching the optimal point at À = 0.25 requires approximately 1.4x more training iterations com- pared no regularization, suggesting that naive orthogonality makes the optimization landscape more challenging. The core of multi-task learning is to share underlying knowledge across tasks. Uniform orthogonalization forces tasks to update in different directions, directly disrupting this knowledge sharing. This motivates spectral-aware regularization: preserve high-SV shared knowledge, only orthogonalize low-SV components. Figure [B) shows top-20% singular values have 89% inter-task alignment and encode 54% of total singular values (vs. 3% alignment and 22% for Nottom-50% Our spectral-as spectral-aware aware approach only orthogonalizes low-SV components, preserving the high-SV shared knowledge needed for effective routing. To empirically validate this selective effect, we visualize
+ Orthogonal Reg.
20.5
10.1
15.3
Dynamic Routing
HydraLoRA
89.0
78.3
83.8
Sparsity Reg.
87.9
77.2
82.6
+ Orthogonal Reg.
89.8
79.8
84.8
Unitorm weighting: 1/N per LoRA.
7

---

Published as a conference paper at ICLR 2026
Table Contribution of each key design, LoRA improves +2.8% over baseline with 47% fewer parameters (highlighted in blue ). Block-level adaptation contributes the most (+2.1%). Improve- ments are consistent across vision (+2.6% by average) and NLP (+2.9% % by average) benchmarks. All results average accuracy (%)1, reported with std across random seeds. Params shown as trainable parameters and % of LLaMA-2-7B. Wall-cloch time breakdown in Appendix C.6
Method
Black-Leve evel Adaptation
Spectral-Auare Regularization
Fine-Ciralned
Routing
Params
Time
Dolly-15k
Avg.
HydraLoRA
1.00x
41.6s1.0
(0.56%)
0.67x
91:2=0.a
70.9110
43.7a0.4
37.9
Lt (Ours)
37.7M (0.56%
0.70x
91.7+00
81.3+11
13.0+0.4
38. 1+0.3
63.8
8M (0.59%)
0.69%
80.9.0.5
80.2+0.7
11.100a
38.2+0.2
63.1
44.5.0.3
38.5.0.3
Table Ablation of block-level adaptation. Block-level achieves better performance with 50 % fewer parameters and 33% less wall-clock time. Specifically, Attn+FFN achieves 63. 1% average accuracy (+2.0% over component-level) with same parameter count (75.5M, 1.1% of foundation model). Notably, FFN alone uses only 50 % parameters (37.7M vs 75.5M) yet achieves 63.0% average (highlighted in blue All results in accuracy (% Params shown trainable parameters and % of LLaMA-2-7B. Wall-clock time breakdown in Appendix C.6
Adaptation Level
Params (%)
Time
DOTA
iNat2018
Dolly- 15k
BBH
Avg.
Component-Level
Wy W.
75.5M (1.11%)
1.00x
89.0
78.3
41.6
35.5
61.1
Block-Level
Attn
FFN
37.7M (0.56%)
0.67x
89.4
79.3
43.3
37.2
62.3
37.7M (0.56%)
0,67x
91.0
79.4
43.7
37.9
63.0
75.5M (1.11%)
0.85x
91.2
79.9
43.9
37.6
63.1
the SV spectrum before/after regularization in Figure components are suppressed x more (-6.0%) than high-SV components (-2.0%).
Appendix. Results confirm that low-SV
Q2: How does fine-grained routing exploit dimension-speciffe heterogeneity? Standard multi-tan routing assigns one scalar weight per LoRA module (5=1 group), forcing all dimensions to use the same task mixture. As discussed in Sec. 3.3 different feature dimensions encode different task attributes (e.g., creativity vs. accuracy). Coarse-grained routing (g=1) applies uniform weights to all dimensions, ignoring this het- erogeneity. In contrast, fine-grained routing (larger g) assigns dimension- specific weights to capture diverse at- tribute requirements.
Table 3: Ablation of routing granularity. Fine-grained routing balances performance and parameter efficiency. The g=2 already achieves +1 +1.5% with only +0.06% extra router parameters (highlighted in blue ), and g=32 achieves best performance (+2.2% on BBH). Results accuracy (%) Router A shown as extra param % of a.A.M-2-7B..
Table shows fine-grained routing (g=32) achieves 37.7% on BBH, out- performing module-wise routing (g=1, 35.5%) by +2.2%. The benefits vary by task type: reasoning tasks (BBH) benefit significantly (+2.2%). while cotpucctioo-oolowwin
Strategy
Dolly-15k
BBH
Avg.
Router A
Module-Wiie
Scalar
41.6
35.5
38.5
tasks (Dolly- 15k) show modest gains, suggest- ing reasoning tasks exhibit stronger dimension-speciific heterogeneity. No- tably, g=2 already provides +1.5% improvement with only +0.06% % ex-
Fine-Grained
2
41.6
37.0
39.3
+0.06%
8
41.3
36.6
39.0
+0.44%
Grouped
16
41.7
37.0
39.3
+0,93%
32
42.0
37.7
39,9
+1.93%
tra router parameters, offering strong select routing granularity based on their resource constraints: g=2 for minimal overhead, g=32 for maximum performance.
trade-off. In implementation, we
Q3: How does block-level adaptation reduce gradient conflicts?
As detailed in Sec. 3.4
8

---

Published as a conference paper at ICLR 2026
Table 5: Comparison with SOTA, mt LoRA achieves 64.0% average accuracy across four bench- marks (91.7% on DOTA, 81.5% on iNat2018, 44.5% on Dolly-15k, 38.5% on BBH), outperforming previous SOTA HydraLoRA by 2.3% on average. We compare with multiple SOTA methods includ- ing LoR,Hub, MMoELoRA, and HydraLoRA with identical experimental setup (rank 16, with identical experts numbers), Results obtained with rank r = 16. All results average accuracy (%
Method
DOTA
iNa12018
Dolly-15k
BBH
Avg.
LoRAHub (Huang ct al.
2023
88.9
80.2+1.6
42.0=0.3
34.9+0.4
61.5
MMoELoRA (Zadoun al.
2023
89.4
78.0+0.3
42.1
35.4
61.2
HydraLoRA TTan ct al.
2024a)
89.1+0.4
78.5+1.7
42.4
36.9+1.0
61.7
LoRA (Ours)
91.7++0.4
81.5+0.6
44.5+0.2
38.5=0.3
64.0
Implemented with optimal hyperpprrumtter search and BLC optimization.
component-level adaptation (W,, W.) suffers from gradient conflict amplification: gradients propa- gate through attention's 's Softmax, creating cross-token dependencies. Instead, our block-level adapta- tion learns a residual (h) that bypasses these internal non-linearities, decoupling the adaptation process from conflict-prone attention mechanics. Table 4 shows block-level adaptation (Attn+FFN) achieves 63.1% 1% average accuracy (+2.0% over component-level), revealing a consistent trend: compo- nent (61.1%) attention (62.3%) FFN (63.0%) Attn+FFN (63.1%) Notably, FFN alone uses only 50% parameters (37.7M vs 75.5M) yet achieves 63.0% average, demonstrating strong parameter efficiency.
Gradient misalignment. To verify that block-level adaptation actually reduces gradient conflicts, we measure gradient alignment between task pairs. We compare component-level adaptation (W;, W.) and block-level adaptation on LLaMA2-7B using Dolly-15k (16 tasks). For 5,000 iterations, we sample two tasks (A, B) and one data point from each. With the base model frozen, we compute loss and extract LoRA gradient vectors VW, and VWB via backward pass. We compute cosine similarity cos( (VW, A. VWB), where negative values indicate conflict. As shown in Figure TC). component- level adapters show mean cosine similarity of +..014 indicating substantial misalignment. In contrast, block-level adapters achieve -0.013 169. This represents a 76% reduction in the magnitude of average gradient conflict, creating easier optimization landscape. Additionally, we provide per-layer gradient correlation analysis in Appendix Section C,3 36% conflict reduction in later layers.
(Figure S3
showing up to
4.4 COMPARISON WITH STATE-OF-THE-AN
In this section, we compare the overall effectiveness with SOTA methods. Specifically, we compare with three SOTA multi-task low-rank adaptation approaches, HydraLoRA, MMoELoRA, LoRAHub, and analyze the contribution of each component through ablation. To ensure fair comparison, all compared methods use the identical experimental setup: same training hyperparameter configs (e.g.. rank r = 16), and same LoRA numbers per benchmark. The comparison results are shown in Tables and 2
Based on comprehensive comparison against multiple SOTA methods (LoRAHub, MMOELORA, and HydraLoRA) with identical experimental setup, we have three key findings: 1) Our method outperforms SOTA at scale. mt LoRA achieves 64.0% average accuracy across four benchmarks (91.7% on DOTA, 81.5% on iNat2018, 44.5% Dolly-15k, 38.5% BBH), outperforming previous SOTA HydraLoRA by 2.3% on average (Table 5) This demonstrates that our method effectively mitigates the multi-task collapse problem, enabling scalable adaptation a larger number of tasks. 2) Existing methods fail to scale beyond 15 tasks. LoRAHub and MMoELoRA achieve average accuracy around 61% % (61.5% and 61.2% respectively). despite being designed for smaller- scale scenarios with 5-8 tasks. Single LoRA achieves 94.5% on DOTA, establishing the upper bound without multi-task learning. This highlights the challenge of multi-task low-rank adaptation: existing methods cannot bridge the gap between single-task and multi-task performance. 3) Each component contributes substantially. Results show that all three designs are instrumental: Block- level adaptation contributes +2.1 2..1% 1% 50% fewer parameters (Table spectral-awart regularization and fine- mnn-grained routing contribute consistent improvements. The improvements are consistent across vision (+2.6%) and NI.P (+2.9%), together improving from 61.1 % 63.9% (+2.8% overall, Table

---

Published as a conference paper at ICLR 2026
Table Performance breakdown by task difficulty on BBH. Tasks' difficulty are categorized by average accuracy across methods: Easy (>50%), Medium (30-50%) Hard (<30%) (category details in Appendix C.5). mt LoRA achieves best across all difficulty levels, especially on Medium tasks (+3.5% over HydraLoRA, highlighted ). All results in accuracy (%)1. Per-task results in Table
Difficulty
LoRA
MMoELoRA
HydraLoRA
LtRA
Easy (7 tasks)
64.32
63.74
67.96
69.52
Medium (8 tasks)
37.82
40.82
37.55
41.01
Hard (12 tasks)
14.63
15.20
18.39
18.78
Overall (27 tasks)
34.38
35.37
36.92
38.52
4.5 DISCUSSION
A) Consistent Improvement Across Task Difficulty. To understand where mtLoRA excels, we categorize BBH's 27 tasks by difficulty based on average accuracy: Easy (>50%), Medium (30- 50%). and Hard (<30%). As shown in T mtLoRA consistently outperforms SOTA across all difficulty levels: +1.6% on Easy, +3.5% Medium, and +0.4% Hard tasks, demonstrating broad applicability rather than being limited to specific task regimes. Per-task breakdown results on BBH provided in Table S6 Appendix.
B) Domain Differences. Block-level adaptation universally improves both vision (+2. (%) and NL. (+2.3%) domains, while fine-grained routing shows dataset-dependent effects (-1.3% on DOTA), See Appendix C.4 for detailed analysis. This suggests that gradient conflict mitigation is a universal challenge, while dimension cemension-specific routing benefits depend on feature heterogeneity.
C) Computational Efficiency. As shown in Table 2 our mt LoRA achieves +2.8% performance improvement while simultaneously being more efficient: 47% fewer parameters and 24% faster training. Block-level adaptation is the key contributor, reducing training time by 33% (94.6 min -> 63.0 min) with 50% fewer parameters (75.5M -> 37.7M). Notably, FLOPss reduction is modest (0.85x-0.99% indicating that speedup primarily comes from improved GPU utilization rather than reduced computation. See Appendix for detailed breakdown.
D) Limitations and Future Work. Our design choices entail certain trade-offs. 1) We use shared matrix 'A' structure for efficiency, reducing SVD cost from (dd)) to Odd2). While efficient, designs (spectral-award regularization, fine- tne-grained routing, block-level adaptation) generalizable to standard LoRA arch (A, = B,A,) with minimal modification. 2) While our block- level adaptation targets Transformers, the underlying principle, i.e., bypassing conflict-amplifying non-linearities such as Softmax, is architecture-agnostic and extends to others like CNNs. We explore the applicability larger models (e.g., LLaMA2-13B) in the Supplementary. Finally, our working hypothesis, i.e., shared knowledge concentrates in high-SV while low-SV shows minimal inter-task alignment, empirically supported by our observations (Figure B)) and prior works (Yu al. 2024b 2025).
Its generalizability across other modalities warrants future investigation.
LLM Usage Claim. We employed Large Language Models (LLMs) for partial text polishing. listing related papers, and optimizing experimental deployment scripts. However, we confirm the core methodological innovations, critical code implementation, interpretation of results, and final manuscript verification remain authors' sole work.
5 CONCLUSION
We present LoRA, enabling stable, scalable multi-task adaptation by addressing the limitations of prior approaches. We identify that existing methods fail due to the eegularization-routin trade-off, rooted in spectral heterogeneity and gradient conflict amplification. Motivated by insights that inter- task alignment (shared knowledge) concentrates in high-SV components, and that component-level adaptation amplifies conflicts, we propose three designs: spectral- regularization, fine-graine grained routing, and block-level adaptation. Our approach achieves up to 2.8% improvement over SOTA while using 47% fewer parameters and 24% less training time across vision and NLP benchmarks, offering parameter-efficien,, compute-efficient, and robust path for scalable multi-task adaptation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
The authors gratefully acknowledge the support from the DSO research grant awarded by DSO National Laboratories, Singapore. This project also partially supported by the Ministry of Education, Singapore, under its Tier-I Academic Research Fund (No. 24-S1S-SMU-0400)

---

Published as a conference paper at ICLR 2026
REFERENCES
Ahmed Agiza, Marina Neseem, and Sherief Reda. Mtlora: Low-rank adaptation approach for efficient multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16196-16205, June 2024.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Shoufa Chen, Chongjian Ge, Zhan Tong. Jiangliu Wang, Yibing Song, Jue Wang. and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances in Neural Informa- tion Processing Systems, http///22..33333 2022. [ttht::: /ppp..pp....p/ppp/ www.ntiischolss..........e
org/CorpusID: 24909:44999888
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi. Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open
ttta http://aaa.......coooooo///////0 Instruction-tune 1lm, 2023. URL httpps///vee http://////wwtabbbb...ccc../oo//
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. Advances in neural information processing systems, 36: 16:88881111155 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Chendi Ge, Xin Wang. Zeyang Zhang, Hong Chen, Jiapei Fan, Longtao Huang, Hui Xue, and Wenwu Zhu. Dynamic mixture curriculum lora experts for continual multimodal instruction tuning. In Forty-second International Conference on Machine Learning. 2025.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor unified view of parameter-efficient transfer learning. In International Conference on Learning Representations. 2022.
and Graham Neubig. Towards
Dan Hendrycks, Collin Burn,, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009. 03300, 2020.
Dou Hu, Lingwei Wei, Wei Zhou, and Songlin Hu. Impartial multi-task representation learning via variance-invariant probabilistic decoding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
2wXi::2106.09685. 2021.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang. Chao Du, and Min Lin. Lorahub: Effi- cient cross-task generalization via dynamic lora composition. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089 2022.
Shih-Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 1631-22648. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi task learning with multi gate cexture-oo-experne In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1930-1939, Association for Computing Machinery, 2018. doi: 10.1 01553319819.320000
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots. ICLR, 2025.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung. Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprini urXiv:2210.09000 2022.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhooe Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 37: 9565-9584, 2024a.
Zichen Tian, Zhaozheng Chen, and Qianru Sun. Learning de-biased representations for remote- sensing imagery. In Advances in Neural Information Processing Systems, 2024b.
Zichen Tian, Yaoyao Liu, and Qianru Sun. Meta-learning hyperparameters for parameter efficient fine- tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288: 2023.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Fengyun Wang, Sicheng Yu, Jiawei Wu, Jinhui Tang, Hanwang Zhang, and Qianru Sun. 3d question answering via only 2d vision-language models. In Forty-second International Conference on Machine Learning, 2025.
Xuyang Wei, Chunlin Tian, and Li. Asymlora: Harmonizing data conflicts and commonalities in mllms, 2025.
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. 2024a.
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. In The Twelfth International Conference on Leurning Representations, 2024b.
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection aerial images. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 3974-3983, 2018.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES HES-menging:: Resolving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httpt:tt//// http:::::///ppeeviiieeeee n forum ? ddxxaxX3yyCyyC
Enneng Yang. Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, 2024.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models free lunch. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. -77775 PMLR, 21-27 Jul 2024a. URL https:://rroeee ngs. ml wwt..m...reessseem www.ppeeesss3..
html
13

---

Published as a conference paper at ICLR 2026
Wenlong Yu, Ruonan Liu, Dongyue Chen, and Qinghua Hu. Explainability enhanced object detection transformer with feature disentanglement. IEEE Transactions on Image Processing. 2024b.
Wenlong Yu, Qilong Wang, Chuang Liu, Dong Li, and Qinghua Hu. Coe: Chain-of-explanation via automatic visual concept circuit description and polysemanticity quantification. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4364-4374, 2025.
Ted Zadouri, Ahmet Ustün, Arash Ahmadian, Beyza Ermiş. Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. 2023.
URL htttt:://ax/xiii org/ ats///bs//2335 05444
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-effiieet fine- tuning. arXiv preprint arXiv:2303. /0512, 2023.
Kesen Zhao, Beier Zhu, Qianru Sun, and Hanwang Zhang. Unsupervised visual chain-of-thought reasoning via preference optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2303 2025.
Beier Zhu, Jiequan Cui, and Hanwang Zhang. Robust fine-tuning of zero-shot models via variance reduction. Advances in Neural Information Processing Systems, 37:76967-77990, 2024.
Beier Zhu, Jiequan Cui, Hanwang Zhang, and Chi Zhang. Project-probed probe-aggregate. Efficient fine- tuning for group robustness. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25487-25496, 2025.
14

---

Published as a conference paper at ICLR 2026
SUPPLEMENTARY MATERIAL
This supplementary material provides additional theoretical foundations, method implementation details, and extended experimental results that complement our main manuscript (references the main manuscript are shown in red).
Theoretical Foundations
Section A.1 provides mathematical justification for Gradient Conflict in attention mechanisms (Section 3.3).
Implementation Details
Section B.
illustrates the detailed architecture of LoRA (Figure 2).
provides detailed experimental setups, including dataset construction, metric defini- tions, and implementation details (Section 4.
Section B.2
Supplementary Experimental Results
Section C.
provides additional analysis on the multi-task collapse challenge (Section 4.2).
Section C.2
visualizes the effect of spectral-aware regularization (Section 4.3 Q1).
Section
provides per-layer gradient correlation analysis for block-level adaptation (Sec-
tion 4.3
Q3
Section analyzes domain differences between vision and NLP tasks (Section 4.5).
Section presents detailed per-task results and difficulty analysis on 27 BBH reasoning tasks
(Table 4)
Section C.6 provides computational efficiency analysis with wall-clock time breakdown (Sec- tion 4.5C Tables 3-4).
A THEORETICAL FOUNDATIONS
A.1 JUSTIFICATION OF GRADIENT CONFLICT
Mathematical Analysis of Gradient Conflict (Section 3.4).
Define gradient conflict as the ex-
pected negative cosine similarity between task gradients:
C = Ee [- cos
V B V
(5)
For traditional LoRA, the gradient includes the Softmax Jacobian JsM
VB JsM x V x (Agh)
(6)
where Jsm[i, (,, Sj) creates off off-diagonal coupling. This coupling amplifies conflicts: 4 Ann and t have localized differences, J/M spreads them across positions. Our residual adapter eliminates this amplification by bypassing JsM entirely.
B IMPLEMENTATION DETAILS
B. ARCHITECTURE DETAILS
Figure provides the detailed architecture of Lo (Figure 2). The architecture shows how our three key designs integrated into standard Transformer block. Specifically, LoRA modules are attached in parallel paths after each LayerNorm, bypassing the internal non-linearities of the frozen blocks to mitigate gradient conflict amplification.
B.2 EXPERIMENTAL SETUP (SECTION 4.1)
Benchmarking Datasets. We evaluate multi-task low-rank adaptation four benchmarks, i.e., DOTA (Xia et al.) 2018) (15 cross-domain tasks), iNat2018 (Van Horn et al., 2018) (25-100 fine- grained classification tasks), Dolly-15k (Conover et al. 2023) (16 instruction- followiinnn tasks), and
15

---

Published as a conference paper at ICLR 2026
FRN (MLP)
nILORA
Fine Grained Routing
LayerNorm
B
MLP Router
Multi-Head Attention
LILORA
LayerNorm
m=:LRA (Ours)
Transformer Block
Figure S1: Overall Architecture of mt Left: Transformer block consists of Multi-Head Attention and FFN (MLP) components, each preceded by LayerNorm. mt LoRA modules are attached in parallel paths after each LayerNorm. Right: The internal structure of LoRA shows the fine- ain-ggraied routing mechanism, where a Router MLP generates dimension-specife weights to dynamically compose task-specific experts.
Big Bench Hard (BBH) (Suzgun et al. 2022 For iNat2018, we partition fine-grained categories to construct 25 tasks with high visual similarity. For Dolly-15k, which contains only 5 tasks, we apply K-Means clustering on instruction embeddings to partition data into 16 semantically distinct clusters, treating each cluster as a task (Tian et al., 2024a). This evaluation protocol follows HydraLoRA's tasks, our method is further validated on three
setup (Tia et al.) 2024a). Besides, for naturally- benchmarks (DOTA, i BBH). For BBH, we evaluate the model's 's generalization capability on complex reasoning tasks. Specifically, we construct a multi-task training set from Flan-v2 (Longpre) jet al. 2 by sampling 30,000 examples evenly across 10 diverse task clusters (e.g.. commonsense reasoning, translation, QA). After multi-task training, we evaluate the model on all 27 BBH reasoning tasks using 3-shot in-context learning. This setup tests whether mt LoRA can effectively transfer capabilities from instruction tuning to challenging reasoning tasks.
Evaluation Metrics. To quantitatively analyze the challenges in multi-task low-rank adaptation, we define three key metrics. 1) Gradient Conffict Score: We measure the overall conflict between task pairs by computing the average cosine similarity of their respective LoRA gradient vectors, c,s, VW3), where more negative value indicates stronger conflict. Note that while this metric specifically detects destructive interference (opposing gradients), our spectral regularization (Fq. penalizes any non-orthogonality (absolute cosine) to minimize both conflict and redundancy. For the spectral analysis in Figure to precisely quantify inter-task alignment within a spectral band B, we (performed on converged models) use a welllla-vale-weieeeee score:
A(B) BBN -
ΣΣ
EoO Jo (u,, My,k)l in6l
where k E B indexes singular value positions, i. index LoRA modules, di, is the k-th singular value of module i's B; matrix, and u, is the corresponding left singular vector. High alignment in high-SV reflects shared knowledge across tasks. 2) Spectral Band Contribution: By definition of SVD, larger singular values correspond to principal directions of task-specific parameter updates, so higher fraction indicates more task-relevant information. To quantify task-specific information within spectral band (Figure B)), we compute the fraction of total singular values: Σ 0k/ / x 100%, where B denotes the spectral band and σ₁ are the singular values of the B, matrices. 3) Routing Uncertainty: To measure the router's decision-makinn makinn confidence for each input, we
calculate the average per-sample routing entropy: Ez [- - (x(x) log (1x)) where ((x) is the router's output distribution for a given sample x. We report routing uncertainty (entropy); lower values indicate more confident, decisive routing.
N
Implementation of Our Method. Our method is based on HydraLoRA (Tian et al. 2024a), where a single matrix A is shared across all tasks, and tasks are learned through diverse B, matrices. Since
16

---

Published as a conference paper at ICLR 2026
Table S1: Computational efficiency breakdown for main ablation (Table Block-level adaptation
reduces training time by 33% while using 50% fewer parameters. Full mt LoRA is 24% faster than HydraLoRA with 47% fewer parameters.
Method
Block
Spec.
FGR
Params
%
Time (min)
Rel. Time
F1.OPs
Rel. FLOPs
HydraLoRA
75.5M
1.11%
94.6
1.00x
4.73e17
1.00x
37.7M
0.56%
63.0
0.67x
4.70c17
0,99x
mcLoRA
37.7M
0.56%
66.1
0.70x
4.01c17
0.85x
39,8M
0.59%
65.5
0.69x
4.70e17
0,99x
39.8M
0.59%
72.1
0.76x
4.01e17
0.85x
Table S2: Computational efficiency breakdown for block-level ablation (Table 4 FFN-only achieves same efficiency Attn-only but better performance (63.0% vs 62.3%). Attn+FFN is 15%
faster than component-level despite same parameter count.
Configuration
Params
Time (min)
Rel. Time
FLOPs
Rel. FLOPs
Component (W.. W.)
75.5M
1.11%
94.6
1.00x
4.73c17
1.00x
Block Attn only
37.7M
0.56%
63.4
0.67x
4.70c17
0.99x
Block FFN only
37.7M
0.56%
63.0
0.67x
4.70e17
0.99x
Block Attn+FFN
75.5M
1.11%
80.3
0.85x
4.73e17
1.00x
ATA, BT B,A, orthogonality between B, and By (ie.. BT x 0) ensures orthogonality between entire LoRA updates A; and A,. This structure allows us directly regularize spectrums of B, (instead of entire LoRA) to control task-specific conflicts. We apply SVD to the task-specific B matrices per epoch to compute the spectral-aware loss. The loss Lsspccrall = Eier [I BJ 1
where B = V, is a temporary matrix constructed by re-weighting the singular values of /((o)).oR where w(ar exp(-o/o) penalize low-SV components.
B, = VV with Me The router is a 2-layer MLP with output dimension x g. where N is the task number and g is the number of groups. It takes the mean seaalp hidden states input and applies softmax normalization to produce routing weights. For > 1, each weight is broadcast by repeating d/g times before element-wise multiplication with LoRA outputs. The total loss includes load-balancing term Lialance to prevent routing collapse. All LoRA modules are applied at the block level as parallel adapters, consistent with Pre-LN architecture (such AdaptFormer (Chen al., 2022))
Implementation of Compared Methods. We compare with HydraLoRA (Tian et al. 2024a), MMoELoRA (Wu al. 2024a), and LoRAHub (Huang al. 2023). All compared methods use rank = 16 for fair comparison. For experiments involving varying regularization strengths (A), we perform hyperparameter search for the optimal learning rate for each ensure fair comparison.
C SUPPLEMENTARY EXPERIMENTAL RESULTS
C. ANALYSIS MULTI-TASK COLLAPSE
Table S3: Multi-task collapse increases with task numbers. Naive averaging degrades from 88.2% (5 tasks) to 2.0% (15 tasks) on DOTA, with conflict score reaching 97.9%. Single LoRA achieves 94.5% on DOTA, 87.0% on iNat2018. All results in accuracy (%)t.
DOTA
iNat2018
Dolly-15k
15
15
25
100
4
Single LoRA
94.5
94.5
94.5
87.0
87.0
87.0
87.0
45.5
45.5
45.5
Naive Averaging
88.2
12.0
2.0
3.5
1.0
0.5
0.3
46.1
40.5
16.0
Conflict Score
6.7
87.3
97.9
96.0
98.9
99.4
99.7
-1.5
10.9
64.7
17

---

Published as a conference paper at ICLR 2026
0.7
Magnitole We gli wist esplno)
1.0
we Spectral Rea
0.6
Mean Mean Singular Value
Spectral Rog
Magnitudee
Weight wio)
Reletive Change
M
0.4-
0.4
1
0.15
4.5%
0.2
0.10
0.2
0.1
Top 2016
20-50%
50-100%
Singular Value Index
Singular Value Endex
Singular
(A) Spectrum Reg Weight
(B) Per-Index Suppression
Per-Band Suppression
Figure S2: Spectral-aware regularization selectively suppresses low-SV components. (A) Higher weight on low-SV. The singular value magnitude a (blue) decays rapidly across indices. Our weighting function w(rr ) = exp(-o/ō) (orange) assigns higher regularization weights for low-SV components. (B) Per-index suppression. The relative change in SV magnitude after applying spectral regularization (A = 1.0). Low-SV components show stronger suppression. (C) Per-band suppression. Aggregated comparison across three bands: top-20% (-2.0%), 20-50% (-4.5%), and 50-100% (-6.0%).
Table S3 supplements multi-task collapse analysis (Section 4.2). Results show that naive averaging of experts leads to catastrophic performance degradation as expert numbers increase, accompanied by rising conflict scores.
C.2 SPECTRAL REGULARIZZTIN ANALYSIS
Figure S2 visualizes the effect of our spectral- regularization (Section 3.2). We analyze the singular value spectrum B matrices from 512 LoRA modules (across all layers and experts) trained on BBH with LLaMA-2-7B.
Key Observations. 1) Selective suppression: Low-SV components (50-100%) suppressed by 6.0%, while high-SV components (top-20%) are preserved with only 2.0% reduction. This 3x difference confirms that our weighting function w(o) = exp(-a/o selectively targets noise-prone subspaces. 2) Monotonic effect: The suppression increases monotonically from high-SV to low-SV (2.0% + 4.5% 6.0%), validating that our design preserves dek-discriminatiee directions while reducing interference in less informative subspaces. 3) Consistent with theory: This empirical result aligns with the Intrinsic Low-Rank trpothesis-high--e components encode principal task directions (signal), while low-SV components correspond to optimization noise.
C.3 PER-LAYER GRADIENT CORRELATION ANALYSIS
Figure $3 visualizes the per-layer pairwise gradient similarity between tasks, comparing component- level (HydralooRA, Wa (W,) and block-level (mtLoRA, FFN) adaptation (Section 4.3 Q3). We compute the average cosine similarity of task-pair gradients on lora_B parameters at each of the 32 transformer layers, using 9 task clusters from Flan-v2 with 30 samples per task.
Key Observations. 1) Block-level reduces overall conffict: Block-level adaptation achieves lower mean gradient similarity (0.540 0.579, -6.7%), indicating reduced inter-task gradient conflict. 2) Largest improvement in later layers: The most significant conflict reduction occurs in later layers Layer (-36%: 0.752 +0.322), Layer 31 (-37%: 1.824+0.516), and Layer 29 -9%: 0.742-00.511, This suggests that block-level adaptation is particularly effective at isolating task- specific updates in deeper representations. 3) Early layers show increased similarity: Conversely, early layers (0, 3, 4) exhibit higher gradient similarity under block-level adaptation, possibly because early layers encode more task- aggosticc features where block-level routing introduces shared gradient patterns. This layer-wise heterogeneity suggests that layer-specific adapter granularity- applying block-level adapters only to later layers while using component-lene adapters for early layers layers-could further optimize the conffict-performance trade-off, warranting future investigation.
18

---

Published as a conference paper at ICLR 2026
1.0
Mean Pairwise Gradient Simitarity
Componerr-Lewe
Blocedivvll
0.8
0.0
30
Index
(A) Per-Layer Trend
(better) higher
0.2
0.1
0.0
-0.1
10
15
20
25
Layer Index
Per-Layer Gradienc Similarity fference
Figure S3: Per-layer gradient similarity analysis. (A) Per-layer mean pairwise gradient cosine similarity for component-lever (red) and block-level (teal) adaptation. Error bands show standard deviation. (B) Per-layer difference (Block Component). Negative values (teal) indicate block-leve has lower gradient similarity (less conffict). Block-level shows up to 36% conflict reduction in later layers (24, 29, 31).
Table S4: Domain difference in NLP vs vision. Block-level adaptation improves both vision (DOTA: +2.2%, iNat2018: +1.6%) and NLP (Dolly-15k: +2.1%, BBH: +2.4%), while fine- fine-grained routing shows mixed effects: it degrades DOTA (-13% but improves iNat2018 (+0.3% and NLP (Dolly- 15k: +0.4%, BBH: +0.3%), indicating heterogeneous feature dimensions benefit from fine-grained routing, All results in accuracy (%)1.
Method
DOTA
iNat2018
Dolly-15k
BBH
Avg.
Block-Level
HydraLoRA
89.0
78.3
35.5
61.1
+ Block-Level Adaptation
91.2
79.9
43.7
37.9
63.2
Routing
Fine-Grained
89.9
80.2
38.2
63.1
C.4 DOMAIN DIFFERENCE ANALYSIS
Table S4 provides a detailed comparison of vision versus NLP domain performance (Section 4.5B). Block-level adaptation universally improves both vision (DOTA: +2.2%, iNat2018: +1.6%) and NLP (Dolly-15kk +2.1 %, BBH: +2.4%). However, fine-graine mme-grained routing shows mixed effects: it degrades DOTA (- 1.3%) but improves iNat2018 (+0.3%) and NLP (+0.4% avg). This indicates that datasets with heterogeneous feature dimensions benefit from fine- me--grieeed routing, while homogeneous visual features may not require dimension-speeiffc routing.
19

---

Published as a conference paper at ICLR 2026
C.5 BBH PERFORMANCE BREAKDOWN
Table S6 provides the detailed task performance on all 27 BBH tasks (Table Section 4.5A). mt LoRA achieves best results on 12 out 27 tasks, with notable improvements on logical deduction tasks (e.g., 7-object deduction: 28.3% vs 16.2% for HydraLoRAA, +12.1%)
Task Difficulty Categorization. We categorize 27 BBH tasks by average accuracy across all methods into three difficulty levels, as shown Table S3
Table S5: BBH task categorization by difficulty level.
Easy (>50%)
Medium (30-50%)
Hard (<30%)
formal fallacies
causal judgement snarks
logical deduction objects logical deduction objects penguins in table
boolcan expressions
movic recommendation
object counting
sports understanding
logical deduction objects date understanding
reasoning colored objects salient translation error tracking shuffled 5
hyperbaton
navigate
tracking shuffled
web of lies
disambiguation qa word sorting
tracking shuffled 7
dyck languages geometric shapes ruin names
temporal sequences multistep arithmetic
tasks
8 tasks
12 tasks
Table S6: Per-task performance on BBH. LoRA achieves best results on 12 out of 27 tasks. All results in accuracy (% Best per task in bold.
Task
LoRA
MMoELoRA
HydraLoRA
LtLo
formal fallacies
100.00
95.55
100.00
100.00
setrcn__epresssons
72.47
70.04
94.74
71.66
novie_recommendation
61.94
69.64
36.84
84.21
uoddrssundes
60.73
54.66
71.66
63.16
hyperbaton
48.18
48.18
64.37
59.51
navigate
56.28
57.89
57.49
57.49
web_of_lie
50.61
50.20
50.61
50.61
sausal_judgement
48.37
48.91
47.83
51.63
snarks
45.71
46.86
46.86
45.71
object counting
39.27
37.65
40.89
38,87
logical deduction three objects
37.65
43.32
34.01
47.37
date_ jitisunnddsttadingggg
36.03
38.46
34,82
34,41
htt___sfff___jjeeeeeee_ objects_3
32.79
36.44
35.63
34.82
disambiguation
31.98
41.30
26.72
46.15
word sorting
30.77
33.60
33.60
29.15
logical_dess dedulldeetiinn cecduction_fiv_obbiee
27.13
24.70
25.51
33.20
penguins in table
26.57
24.48
27.97
29.37
logical_deduction www.ddedudonnncmmm seven_ objects
20.24
17.81
16.19
28.34
reasoning_ade altpaaabblooreeeeeea colored objecss
19.03
19.43
20.65
22.27
salient translation enror detection
17.81
17.81
17.81
17.00
tracking shuffled objects_5
17.81
19.43
17.41
15.79
liakking_suffted_dopoooee _objects__sc__ tyck_languages
11.74
10.93
10.12
15.79
10.12
13.36
36.03
21.86
geometric_shee shapess
10.12
10.93
11.34
9.72
ruin names
8.10
9.72
23.48
22.67
temporal sequences
4.45
7.29
8.10
6,88
multistep arithmetic
2.43
6.48
6.07
2.43
Average
34.38
35.37
36.92
38.52
20

---

Published as a conference paper at ICLR 2026
C.6 COMPUTATIONAL EFFICIENCY ANALYSIS
We provide detailed computational efficiency analysis in Tables ST S2 (Section 4.5C), focusing on NLP benchmarks where all experiments were conducted the same hardware (2x GPU DDP, LLaMA-2-7B).
Main Ablation Efficiency. Table shows the parameter count and wall-clock training time. We have five key findings. 1) Block-level adaptation reduces training time by 33% (94.6 min 63.0
min) while using 50% fewer parameters (75.5M - 37.7M). 2) Spectral-aware regularization adds minimal overhead (+5% time. extra parameters). 3) Fine-grained routing adds modest overhead (+4% time, +5.6% parameters). 4) Full mt LoRA is still 24% faster than HydraLoRA baseline (72.1 min vs 94.6 min) with 47% fewer parameters. 5) FLOPs reduction is modest (0.85x-0.99x). This indicates that the wall-clock speedup primarily comes from improved GPU utilization (block-level avoids redundant routing computations multiple positions), not FLOPs reduction.
Bloek-Level Ablation Efficiency. Table S2 shows the efficiency comparison for different block- level configurations. Key findings: 1) Block-level (Attn FFN alone) uses 50% parameters (37.7M vs 75.5M) while achieving 33% faster training. 2) FFN-only is the most efficient: same parameters and time as Attn-only, but better performance (63.0% 62.3%). 3) Attn+FFN maintains efficiency advantage: 15% faster than component-level (80.3 min vs 94.6 min) despite same parameter count.
21


Reviews:
[{"review_id": "VwzJIXL6zZ", "reviewer": "Reviewer_M561", "summary": "The paper proposes mtLoRA, a \"plug-and-play\" solution to address catastrophic performance degradation in multi-task Low-Rank Adaptation. The authors argue that naively combining task-specific LoRA modules leads to \"misaligned feature subspaces\", and that existing solutions (regularization or routing) fail because they are applied independently and uniformly. The paper identifies two primary causes for this failure: 1) Spectral Heterogeneity, where LoRA's high singular value (SV) components encode both critical task-discriminative information and the majority of parameter conflicts, making uniform regularization suboptimal; and 2) Gradient Amplification, where attaching LoRA to individual $W_q/W_k$ matrices amplifies gradient conflicts through the attention mechanism's Softmax. Based on this diagnosis, mtLoRA introduces three components: 1) Spectral-Aware Regularization, which selectively applies strong orthogonality to low-SV (noise) components while preserving high-SV (discriminative) components; 2) Fine-Grained Routing, which uses a router network to produce dimension-specific (vector) weights for combining LoRA modules, rather than a single scalar weight; and 3) Block-Level Adaptation, which applies the LoRA module as a residual to an entire Transformer block, bypassing the internal Softmax conflicts. Empirically, mtLoRA is shown to provide significant gains over a strong baseline (HydraLoRA) on vision (DOTA, iNat2018) and language (Dolly-15k) benchmarks .", "questions": "1. Please resolve the contradiction in Section 4.2. Does larger $g$ mean finer or coarser routing? Please provide pseudocode for how the router's $N \\times (d/g)$ output is broadcast and multiplied with the LoRA update.\n\n2. Can you provide any empirical or theoretical justification for why the SVD of $B_i$ (the $d \\times r$ matrix) is a high-fidelity proxy for the spectral properties of the full $\\Delta_i = B_i A_i$ (the $d \\times d$ update)?\n\n3. What is the wall-clock training time overhead of enabling spectral-aware regularization (with its per-epoch SVD) and fine-grained routing (with its larger router network ) compared to the HydraLoRA baseline?\n\n4. Does mtLoRA's methodology require the asymmetric (shared $A$, task-specific $B$) structure of HydraLoRA? Or can the three components (spectral-reg, fine-routing, block-level) be applied to a standard multi-task setup with independent $B_i A_i$ modules?\n\n5. How does the full mtLoRA method compare against other SOTA methods from the literature, such as MoLE or TIES-Merging, on the iNat2018 or Dolly-15k benchmarks?\n\n6. In Table 1(C), does the \"Block\" level LoRA have the same number of trainable parameters as the \"Component\" level? Please clarify the parameter counts for these different attachment strategies.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The primary strength of the paper is its clear and empirically-supported diagnosis of why multi-task LoRA fails. The analyses in Table 1, showing the failure of uniform regularization (1A) , the spectral heterogeneity of LoRA (1B) , and the weakness of component-level attachment (1C), are persuasive and form a strong foundation for the method.\n\n2. The three proposed components of mtLoRA map directly and logically to the identified problems. Spectral-aware regularization is a clever fix for the issue identified in Table 1B. Block-level adaptation is an intuitive solution to the gradient amplification problem.\n\n3. The paper's component-level analysis is strong. Tables 5, 6, 7, and 8 clearly demonstrate the additive benefits of the components and their individual contributions (e.g., fine-grained routing on NLP and block-level adaptation on vision ).\n\n4. The method is presented as a \"plug-and-play\" add-on for existing methods like HydraLoRA, which increases its practical relevance. The evaluation across both vision and language domains demonstrates its general applicability.\n\n5. The finding that fine-grained routing is critical for NLP but harms performance in vision (Tables 7 & 8) is a valuable and interesting insight into the differing nature of representations in these domains.", "weaknesses": "# Major Concerns\n\n1. **Ambiguous Baseline Comparison:** The main results in Table 2 compare mtLoRA to \"HydraLoRA\" and \"Baseline\". The check-marked rows indicate that mtLoRA is built on top of HydraLoRA. This table is effectively an ablation study, not a SOTA comparison. It is unclear how mtLoRA stacks up against other, different SOTA multi-task methods mentioned in the related work, such as MoLE , TIES-Merging , or DARE. The claim of \"state-of-the-art results\"  is not fully substantiated against the broader field.\n\n2. **Confusing Implementation of Fine-Grained Routing:** The description of fine-grained routing is contradictory. Section 3.3 states weights are $\\Pi_i \\in \\mathbb{R}^{d/g}$. Section 4.2 states \"Larger g means finer grained routing\". However, $g$ is defined as \"group size\". Logically, a larger group size $g$ would mean coarser routing (e.g., $g=d$ would be module-wise). This is directly contradicted by the text. Furthermore, Table 5 lists \"Full $g=768$\". This implies $g=d=768$, which should be module-wise (the coarsest setting), but it's labeled \"Full\" (the finest) and performs best. This ambiguity is a major flaw in the paper's presentation and reproducibility.\n\n3. **Un-benchmarked Computational Overhead:** The spectral-aware regularization requires performing SVD on the $B$ matrices \"every epoch\". For a large number of tasks (e.g., $N=100$) and large models, this is a significant, non-trivial computational cost that is not benchmarked or discussed.\n\n4. **Under-justified SVD Proxy:** The method applies SVD to the $B_i$ matrix as a proxy for the full $\\Delta_i = B_i A_i$ update, justified only by cost. The spectral properties of $B_i$ and $B_i A_i$ are not the same, as the $A_i$ matrix performs a rank-bottleneck projection. This simplification is a potential threat to the method's soundness, and it is not theoretically or empirically justified why $B_i$'s singular vectors are a sufficient proxy.\n\n## Minor Concerns\n\n1. **Synthetic Task Creation:** The tasks for the NLP benchmark (Dolly-15k) are created by performing K-Means clustering on instruction embeddings. This is a synthetic setup. The method's performance may be sensitive to the number of clusters (N) or the quality of the clustering.\n\n2. **Missing Definition of HydraLoRA:** The paper states it builds on HydraLoRA and only briefly defines it in related work as having an \"asymmetric LoRA structure\". It is not clarified in the method section if mtLoRA _requires_ this asymmetric structure, or if it can be applied to standard LoRA modules.\n\n3. **Implementation of Spectral Loss:** Equation (3) defines the conceptual loss based on pairwise singular vector alignment. However, Section 4.2 describes a different implementation: computing weighted matrices $B_i'$ and then a Frobenius norm on their product. The paper should clarify if Eq. 3 is just conceptual and the (different) formulation in Sec 4.2 is the actual loss used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gdajc69Muy", "reviewer": "Reviewer_PCr8", "summary": "This paper addresses the misalignment problem in multi-task LoRA models. The authors observe that when multiple LoRA modules are trained independently for different tasks, feature subspaces become misaligned, causing interference and performance degradation. To overcome this, the paper proposes mtLoRA, which introduces three key innovations: spectral-aware regularization, fine-grained routing and block-level adaptation. Comprehensive experiments across vision (DOTA, iNat2018) and NLP (Dolly-15K) benchmarks show consistent gains, achieving up to 4.4% improvement over HydraLoRA.", "questions": "1. How does mtLoRA differ from distribution-alignment approaches (e.g., task covariance alignment)? Could spectral-aware regularization be interpreted as an implicit distribution alignment mechanism?\n\n2. The paper empirically splits singular values into top-10%, 10–50%, and 50–100% bands. Is this partition fixed or adaptive? Would learning the spectral mask end-to-end yield further improvement?\n\n3. What is the computational and memory cost of channel-wise routing for large models (e.g., LLaMA2-13B)? How does it scale compared with standard HydraLoRA?\n\n4. Have the authors tested the contribution of each design on distinct architectures? Does block-level adaptation generalize beyond ViTs and LLMs?\n\n5. Does mtLoRA preserve balanced performance across heterogeneous tasks (e.g., easy vs. hard or high-data vs. low-data tasks)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper identifies and formalizes spectral heterogeneity in multi-task LoRA modules—an overlooked cause of task conflict. The introduction of spectral-aware regularization and fine-grained routing is conceptually fresh and well-motivated by empirical analysis (e.g., SVD studies in Table 1).\n\n2. The methodology is well-grounded and clearly explained, combining theoretical motivation (singular value analysis, gradient interference) with solid empirical validation. The proposed masking and weighting schemes in spectral regularization are intuitive yet effective.\n\n3. The writing is structured and readable, with comprehensive ablations (Tables 4–9) that isolate each design’s contribution. The visualization of routing entropy and singular-value conflicts provides helpful interpretability.\n\n4. The method significantly improves multi-task adaptation robustness, especially in extreme task settings (up to 25–100 tasks). The generality across both vision and NLP domains enhances its impact.", "weaknesses": "1. While the empirical spectral analysis is convincing, the paper lacks a formal theoretical justification of why spectral-aware regularization specifically balances discrimination and conflict. A more rigorous connection between singular value magnitude and information content could strengthen the conceptual contribution.\n\n2. Although Experiments span domains, all are relatively moderate in scale and may not fully test scalability to large LLM adaptation or multi-domain real-world tasks.\n\n3. The paper does not cite or discuss related efforts that also attempt to align task representations, such as [1]. That work similarly mitigates task conflicts by aligning inter-task representation distributions (through variance regularization). A comparative discussion would help clarify how mtLoRA differs or complements such representation-alignment perspectives.\n- [1] Impartial Multi-Task Representation Learning via Variance-invariant Probabilistic Decoding. ACL 2025.\n\n\n4. The ablation on routing granularity (Table 5) could be more thorough by exploring computational trade-offs and visualizing per-dimension routing patterns.\n\n5. Since mtLoRA builds upon HydraLoRA with added modules, the parameter increase and training cost should be analyzed more explicitly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wz4OoiMGKF", "reviewer": "Reviewer_fc95", "summary": "In the paper 'Multi-Task Low Rank Model Adaptation' the authors identify two problems.\nParameter and representation misalignmen. Which lead to conflicting weight updates, which cause different Lora updates to pushe the model in oposite directions.\n\nAs a remedy the paper proposes\nspectrum aware regularization,\nblock level lora, and the use of a a router network\n\nThe proposed methods ar e tested on the\ndolly-15k, Dota and iNaturalist datasets.", "questions": "- Is it possible to plot singular value spectra, before and after tuning, with and without tuning, as well as pairwise correlations of adapter updates to support the claims made regarding the mechanisms which underpin the presented methods?\n\n- Is it possible to report standard deviation across across multiple seeds, for all experiments? This would make the paper more credible statistically.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The mask formulation is very elegant. It reminded me a lot of Thikonov regularization, which is a good idea.\n- The experiments agree with related work, orthogonalization, for example, has been shown to work well for fine-tuning in concurrent related work like https://arxiv.org/pdf/2507.13260 .\n- Experimentally the authors observe competitive or improved results.", "weaknesses": "- The variables appearing in the equations throughout the paper are often not properly defined. In equation (1) for example, dimensions and the domain, are missing.\n- The dataset choice is not well explained. Why did the authors choose the Dota dataset instead of VTAB-1k for example? I don't think this choice is explained in a convincing way. \n- While the paper presents improved numbers, it does not present evidence for the mechanism that the paper presents as an explanation for the efficiency of the presented methods. Which could have included plots of spectra or correlation plots of LORA adapters. \n\n#### Code\nOn my machine the anonymous code repository link did not work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UeCKHoBiHS", "reviewer": "Reviewer_sPqK", "summary": "This paper addresses performance degradation in multi-task LoRA fine-tuning caused by parameter and representation misalignment across tasks. It observes that high singular-value (SV) components are discriminative yet conflict-prone, while low-SV ones are mostly noise, and that inserting LoRA at fine-grained attention components amplifies gradient conflicts. To tackle this, the authors propose mtLoRA, which introduces three modules: spectral-aware regularization to orthogonalize low-SV components, fine-grained routing to weight LoRA dimensions adaptively, and block-level adaptation to mitigate attention coupling.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Simple Plug-and-play Module:** The three changes can be plugged into existing multi-task LoRA systems; the block-level residual adapter is especially practical.\n\n**Strong Empirical Results:** The full configuration (orthogonality + block-level scope + channel-wise routing) achieves the best results.", "weaknesses": "**Incremental Technical Novelty:** Each component (orthogonal regularization, routing, adapter placement) is known; the contribution is largely in diagnostics + integration. The spectral re-weighting uses SVD on LoRA factors with a monotone mask, and fine-grained routing is a natural extension of MoE-style gating—useful, but methodologically incremental.\n\n**More Analysis in Routing/regularization Interaction:** The work notes that stronger orthogonalization can harm routing discriminability (entropy rises), but practical guidance is thin (e.g., how to pick λacross datasets; robustness to mis-specification).\n\n**Compute and Memory Costs:** Fine-grained (dimension/group-wise) routing and per-epoch SVD add overhead; block-level adapters may increase parameters. There is no wall-clock/FLOPs/memory report to substantiate the claimed practicality at scale.\n\n**Ablation Depth:** Useful analyses exist (e.g., routing granularity; attach scope), but further breakdowns would strengthen claims: (i) how much each component contributes under very high N (e.g., 50–100 tasks) in both domains; (ii) failure modes where block-level adapters underperform (Table 7 suggests fine-grained routing can hurt in vision).\n\n**Routing Entropy vs. Accuracy:** Can you provide a practical recipe for tuning λthat balances conflict reduction and routing discriminability across datasets/backbones? Any automatic schedule?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)