ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation

## Abstract

Abstract Low-Rank Adaptation (LoRA) is widely adopted for downstream fine-tuning of foundation models due to its efficiency and zero additional inference cost. Many real-world applications require foundation models to specialize in several specific tasks simultaneously, motivating the need for efficient multi-task downstream adaptation.
To address this need, existing studies have primarily explored two directions: Model Merging with LoRA , which shows advantages in training-free scenarios but still lags behind multi-task training in overall performance; and MoE-based LoRA approaches, which improve multi-task learning performance but introduce routers that hinder the mergeability of LoRA parameters and incur considerable inference overhead, thereby limiting real-world deployment practicality.
To this end, we propose ThanoRA , a Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation framework that enables effective, efficient and unified multi-task downstream adaptation without introducing additional structure.
ThanoRA performs multi-task learning by tailoring subspace allocation at initialization and enforcing diversity preservation throughout training: it allocates varying dimensions to construct task-specific low-rank subspaces driven by inter-task heterogeneity, enabling fine-grained knowledge injection, while diversity-preserving regularization mitigates task interference and subspace collapse, thereby fully exploiting the low-rank capacity.
Extensive experiments across multimodal and text-only benchmarks under varying multi-task mixtures demonstrate that ThanoRA consistently outperforms strong baselines, surpassing even separate task-specific fine-tuning, while introducing no additional structures or inference overhead.
Our code will be publicly available at: https://github.com/LiangJian24/ThanoRA .

## 1 Introduction

With the rapid development of foundation models Bai et al. (2025); Wang et al. (2025); Grattafiori et al. (2024); Li et al. (2024a); Liu et al. (2023), their general capabilities across a wide range of tasks have been significantly enhanced. Nevertheless, they often require downstream adaptation to perform well on specific target tasks Hu et al. (2022); Huang et al. (2025). To address this, a variety of parameter-efficient fine-tuning (PEFT) methods have been proposed to adapt foundation models with minimal computational and storage overhead Houlsby et al. (2019); Lester et al. (2021); Liu et al. (2022); Li & Liang (2021); Hu et al. (2022). Among them, LoRA Hu et al. (2022) and its variants Meng et al. (2024); Liu et al. (2024b); Yang et al. (2024); Hayou et al. (2024) are particularly appealing due to their low training cost and the ability to merge adapted parameters into the base model without incurring inference overhead.
Although PEFT methods such as LoRA have shown competitive performance, recent studies suggest that they may encounter challenges on heterogeneous corpora involving multiple tasks or domains, where task conflicts and interference can hinder adaptation Tian et al. (2024). Meanwhile, real-world applications often require a single model to acquire several targeted capabilities simultaneously Li et al. (2024b); Liu et al. (2024a), motivating the need for efficient multi-task downstream adaptation.

Existing efforts toward multi-task efficient adaptation have primarily explored two directions. The first is Model Merging with LoRA (see [fig. 1](https://arxiv.org/html/2505.18640v2#S1.F1).a), where individually fine-tuned LoRA modules from different tasks are directly merged to form a multi-task model Stoica et al. (2025); Zeng et al. (2025); Zheng et al. (2025). This approach is attractive due to its training-free nature in combining arbitrary sets of tasks, but the lack of a shared optimization process limits its ability to resolve inter-task conflicts, resulting in inferior performance to multi-task training.
The second line of research combines LoRA with Mixture-of-Experts (MoE) (see [fig. 1](https://arxiv.org/html/2505.18640v2#S1.F1).b) architectures Li et al. (2024b); Shen et al. (2024); Zhao et al. (2025); Tian et al. (2024); Chen et al. (2024); Wu et al. (2024b). These methods employ routers to dynamically dispatch inputs to different LoRA experts, thereby alleviating conflicts and improving multi-task performance. Nevertheless, the reliance on routers prevents the adapted parameters from being merged back into the base model, leading to considerable inference overhead and extra storage requirements, thereby hindering real-world deployment.

To this end, we aim to develop an efficient multi-task low-rank adaptation framework better aligned with real-world deployment, enabling large foundation models to master the combination of targeted tasks without incurring additional inference or storage overhead.
This raises the following challenge:

Challenge I) How to effectively inject multi-task knowledge into the low-rank subspace?

Conducting multi-task training in a unified LoRA framework without additional non-mergeable structures requires tasks to share the same trainable parameter subspace, often leading to conflicts and performance degradation.
Thus, the key challenge is to compress multi-task knowledge into a shared low-rank space. To address this, we first note the following fact:

Tasks exhibit heterogeneous representational demands across layers of large foundation models.

Prior studies have shown that LoRA modules exhibit varying representational requirements across layers and modules, necessitating different rank configurations Zhang et al. (2023). In the context of multi-task learning, we further extend this conclusion by observing that different tasks present heterogeneous representational demands across different layers.
As illustrated in [fig. 1](https://arxiv.org/html/2505.18640v2#S1.F1).d, consider a concrete example: some tasks involve relatively simple visual content but require complex textual reasoning, whereas others display the opposite pattern. This heterogeneity results in divergent representational capacities required by the foundation model at different layers, as reflected in the accompanying bar chart. While this example illustrates visual–textual heterogeneity, real tasks exhibit multidimensional variations that further accentuate representational differences.
Consequently, applying a uniform adaptation strategy in multi-task training may fail to capture these complexity differences, leading to suboptimal resource allocation. This motivates us to allocate low-rank resources in a task-aware manner, aligning the representational capacity at each layer with the heterogeneous demands of different tasks.
To this end, we propose Heterogeneity-Aware Subspace Initialization (HASI). Specifically, task-specific priors are extracted based on instruction-previewed SVD, as similarly employed in Yang et al. (2024). We then estimate task heterogeneity via the spectral entropy of singular values, which serves as a proxy for the subspace complexity required by each task.
Based on the relative heterogeneity of each task, we partition the $R$-dimensional subspace spanned by the LoRA modules at each layer. Each task-specific subspace is then initialized with the truncated singular vectors corresponding to its allocated dimension, enabling fine-grained, layer-wise knowledge injection tailored to task complexity.

Figure: Figure 1: Comparison and Motivation. (a) LoRA Merging acquires multi-task knowledge without training but remains suboptimal compared to multi-task training. (b) MoE-based LoRA mitigates task conflicts via routers but compromises parameter mergeability. (c) Our method enables efficient multi-task adaptation through mergeable LoRA with task-specific subspace modeling. (d) Tasks show heterogeneous complexity across layers, motivating the allocation of task-specific subspaces to better match their representational demands.
Refer to caption: x1.png

While HASI enables a proper allocation of low-rank subspaces and task-aware knowledge injection at initialization, during multi-task training the parameter subspaces of different tasks still share gradient update signals. As a result, the initially disentangled task-specific subspaces may gradually lose their diversity, leading to subspace collapse (see [Sec. 3.2.2](https://arxiv.org/html/2505.18640v2#S3.SS2.SSS2) for definition). Thus, this raises the second key challenge:

Challenge II) How to mitigate low-rank subspace collapse during multi-task training?

To mitigate this, we further propose Subspace-Preserving Regularization (SPR), which enforces orthogonality among different task-specific subspaces during training while retaining a partially shared collaborative subspace for knowledge sharing. This orthogonality constraint enhances the expressiveness of the low-rank subspaces initialized by HASI and effectively alleviates the problem of subspace collapse (refer to [Sec. 4.3.3](https://arxiv.org/html/2505.18640v2#S4.SS3.SSS3) for experimental evidence).

Notably, HASI and SPR work synergistically:
HASI creates heterogeneous subspaces for task-aware knowledge injection at initialization, while SPR preserves their diversity throughout training, jointly enabling effective multi-task adaptation within a unified, mergeable LoRA framework.
Our contributions are summarized as follows:

- ❶
Revealing and Modeling Heterogeneous Representational Demands Across Layers.
We reveal that downstream tasks vary in complexity and characteristics across layers, motivating layer-wise heterogeneity modeling within a unified framework.
- ❷
Effective and Efficient Multi-Task Low-Rank Adaptation Framework.
We propose ThanoRA to address the challenges of task heterogeneity modeling and subspace collapse during multi-task training, enabling effective multi-task adaptation while preserving parameter mergeability and inference efficiency, supplementing the shortcomings of merging- and MoE-based methods.
- ❸
Comprehensive Empirical Validation. We conduct extensive experiments on multimodal and text-only benchmarks, consistently demonstrating robust performance improvements and efficiency advantages over strong baselines, surpassing even separate task-specific fine-tuning.

## 2 Related Work

### 2.1 Parameter-Efficient Fine-Tuning.

Recent advances in large foundation models have driven rapid progress Wang et al. (2025); Yang et al. (2025); Rong et al. (2025); Ye et al. (2025); Fang et al. (2025). However, with their ever-increasing scale, full fine-tuning is becoming prohibitively costly in computation and storage. Parameter-Efficient Fine-Tuning (PEFT) methods address this challenge by efficiently adapting large models with minimal trainable parameters Houlsby et al. (2019); Lester et al. (2021); Li & Liang (2021); Liu et al. (2022); Pfeiffer et al. (2020); Karimi Mahabadi et al. (2021); Wu et al. (2024a); Hu et al. (2022); Liang et al. (2025).
Among these, Low-Rank Adaptation (LoRA) Hu et al. (2022) is widely adopted due to its simplicity and zero additional inference cost, injecting trainable low-rank matrices into existing weight layers without modifying the model architecture.
Numerous extensions have been proposed to improve its performance.
DoRA Liu et al. (2024b) decomposes pretrained weights into magnitude and direction components to improve stability and adaptation capacity.
PiSSA Meng et al. (2024) accelerates convergence by initializing low-rank matrices with the principal components of the pretrained weights instead of random noise. Other approaches include dynamic rank allocation Zhang et al. (2023), vector-based random matrix adaptation Kopiczko et al. (2023), instruction-previewed initialization Yang et al. (2024).
These variants improve the capacity and convergence behavior of LoRA, without sacrificing its efficiency in both training and inference.

### 2.2 Multi-Task Learning.

Multi-task learning (MTL) aims to enhance generalization by jointly training on multiple tasks, enabling shared representations to improve knowledge integration Crawshaw (2020); Zhang & Yang (2021); Ban & Ji (2024); Agiza et al. (2024); Huang et al. (2023). Classical approaches include parameter sharing Wallingford et al. (2022); Ma et al. (2019), task-specific structure Guo et al. (2020); Sun et al. (2021), or regularization-based techniques Zhang & Yeung (2014); Yang & Hospedales (2016). With the rise of large foundation models, efficiently adapting them to diverse downstream tasks has become a central focus in recent MTL research.
Recent studies on parameter-efficient multi-task adaptation can be broadly divided into two categories:
(i) Model Merging with LoRA. These methods directly merge LoRA modules trained on individual tasks Stoica et al. (2025); Zeng et al. (2025); Zheng et al. (2025). While attractive for training-free scenarios, they generally lag behind multi-task training due to the absence of explicit inter-task optimization.
(ii) MoE-based LoRA. A growing number of works combine LoRA with Mixture-of-Experts (MoE) architectures Li et al. (2024b); Shen et al. (2024); Zhao et al. (2025); Tian et al. (2024); Chen et al. (2024); Wu et al. (2024b); Feng et al. (2024); Wang et al. (2024); Zhu et al. (2025). For example, MixLoRA Shen et al. (2024) mitigates task interference by conditionally constructing task-aware LoRA factors for task-specific routing, while HydraLoRA Tian et al. (2024) introduces asymmetric experts to improve multi-task adaptability. However, these approaches rely on non-mergeable routing structures, incurring considerable inference and additional storage overhead, thereby undermining the deployment efficiency of LoRA.
While LoRA model merging sacrifices overall performance and MoE-based LoRA designs compromise mergeability and efficiency, neither fully addresses the need for effective and practical multi-task adaptation.

## 3 Methodology

### 3.1 Preliminary

Low-Rank Adaptation.
LoRA introduces trainable low-rank matrices into pretrained models to achieve efficient fine-tuning with minimal parameter overhead. Specifically, for a linear projection $W\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$, LoRA reparameterizes it as:

$$ $W_{\text{LoRA}}=W+\Delta W=W+BA,$ (1) $$

where $A\in\mathbb{R}^{r\times d_{\text{in}}}$, $B\in\mathbb{R}^{d_{\text{out}}\times r}$, and $r\ll\min(d_{\text{in}},d_{\text{out}})$ is the rank of LoRA. This not only minimizes training overhead but also allows the updated weights to be seamlessly merged into the original model after training, which is a key advantage of LoRA for deployment efficiency.

Instruction-Previewed Initialization.
Following previous work Yang et al. (2024), we extract task-specific priors by forwarding a small batch (e.g., 500) of task-specific samples through the pretrained foundation model. For each linear layer, we collect the input activation $X\in\mathbb{R}^{d_{\text{in}}\times BL}$, compute the covariance $C=XX^{T}$, and derive a task-aware Singular Value Decomposition (SVD) as follows:

$$ $\widehat{W}=\text{SVD}(WC)C^{-1}=\sum_{i=1}^{R}\sigma_{i}\mathbf{u}_{i}\hat{\mathbf{v}}_{i}^{\top},$ (2) $$

yielding singular values $\{\sigma_{i}\}$ and corresponding vectors $\mathbf{u}_{i},\hat{\mathbf{v}}_{i}$ for knowledge extraction.

### 3.2 Proposed Method

To address the challenges of incorporating heterogeneous task knowledge into the low-rank subspace (Challenge [I](https://arxiv.org/html/2505.18640v2#Q1)) and mitigating subspace collapse (Challenge [II](https://arxiv.org/html/2505.18640v2#Q2)) in multi-task low-rank adaptation, we propose ThanoRA, a Task Heterogeneity-Aware framework that enables effective and efficient multi-task adaptation.

Figure: Figure 2: Overview of ThanoRA. (a) Heterogeneity-Aware Subspace Initialization via entropy-guided rank allocation; and (b) Subspace-Preserving Regularization with orthogonality constraints, which together enable multi-ability acquisition across downstream tasks. Please refer to [Sec. 3.2](https://arxiv.org/html/2505.18640v2#S3.SS2).
Refer to caption: x2.png

#### 3.2.1 Heterogeneity-Aware Subspace Initialization (HASI)

##### Motivation.

Real-world tasks often exhibit heterogeneity across various aspects, leading to divergent representational demands across layers of large foundation model. For example, as shown in [fig. 1](https://arxiv.org/html/2505.18640v2#S1.F1)(c), in multi-task fine-tuning of MLLM, DocVQA Mathew et al. (2021) task presents visually complex but textually simple inputs, whereas ScienceQA Lu et al. (2022) task features the opposite. This heterogeneity is further reflected in the normalized spectral entropy curves (computed via [eq. 3](https://arxiv.org/html/2505.18640v2#S3.E3)), where each task shows distinct complexity patterns across layers.
These observations align with prior findings that early layers of MLLMs focus on cross-modal fusion, while later layers predominantly process the fused semantic representations Huang et al. (2025); Zhang et al. (2024).
Notably, this example reflects only a single aspect of heterogeneity, while real-world tasks involve richer and multidimensional forms of variation. Such observations highlight the necessity of explicitly modeling heterogeneity in multi-task adaptation.

To capture such discrepancies, we construct task-specific low-rank subspaces aligned with layer-wise representational heterogeneity.
We first obtain a task-aware representation matrix $\widehat{W}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$ using instruction-previewed SVD, as defined in [eq. 2](https://arxiv.org/html/2505.18640v2#S3.E2). This captures the dominant semantic directions associated with the task-specific response to the pretrained model.
Then, to quantify the complexity of the representation required by each task at a given layer, we compute the spectral entropy of the singular values $\bm{\sigma}\in\mathbb{R}^{r_{\text{total}}}$ of $\widehat{W_{t}}$ for task $t$ as:

$$ $\mathcal{H}(\bm{\sigma})=-\sum_{i=1}^{R}p_{i}\log(p_{i}),\quad p_{i}=\frac{\sigma_{i}}{\sum_{j}\sigma_{j}},$ (3) $$

where $r_{\text{total}}$ denotes the total dimension budget for truncating the SVD of $\widehat{W_{t}}$. This quantifies the representational complexity of the subspace at each layer for a given task.

Furthermore, we compress multi-task knowledge into a shared low-rank space, which in LoRA is naturally constrained by the upper bound $r_{\text{total}}$. Under this fixed total dimension budget $r_{\text{total}}$, we allocate a portion $r_{t}^{\ell}$ of the LoRA rank to each task $t$ at layer $\ell$ proportionally to the normalized entropy across tasks:

$$ $r_{t}^{\ell}\propto\text{Softmax}\left(\frac{\mathcal{H}_{t}^{\ell}}{\tau}\right),$ (4) $$

where $\tau$ is a temperature hyperparameter controlling the sharpness of the allocation. To promote task cooperation, we reserve an additional cooperative subspace with rank $r_{\text{coop}}=\big\lfloor\frac{r_{\text{total}}}{T+1}\big\rfloor$, where $T$ denotes the number of tasks.

The remaining task-specific dimension budget $r_{\text{task}}=r_{\text{total}}-r_{\text{coop}}$ is then distributed by discretizing the soft allocations:

$$ $r_{t}^{\ell}=\max\left(r_{\min},\left\lfloor r_{\text{task}}\cdot\frac{\exp(\mathcal{H}_{t}^{\ell}/\tau)}{\sum_{j=1}^{T}\exp(\mathcal{H}_{j}^{\ell}/\tau)}\right\rfloor\right),$ (5) $$

where $r_{\min}$ ensures each task receives a minimum allocation to prevent vanishing ranks for certain tasks.

For each task $t$ and layer $\ell$, let $(\mathbf{U}_{t}^{\ell},\mathbf{V}_{t}^{\ell},\bm{\Sigma}_{t}^{\ell})$ be the truncated SVD components of the instruction-previewed weight estimation. We construct the initial LoRA matrices of each task by scaling the singular vectors with the square roots of their corresponding singular values:

$$ $\mathbf{B}_{t}^{\ell}=\mathbf{U}_{t}^{\ell}\cdot\left(\bm{\Sigma}_{t}^{\ell}\right)^{\frac{1}{2}},\quad\mathbf{A}_{t}^{\ell}=\left(\bm{\Sigma}_{t}^{\ell}\right)^{\frac{1}{2}}\cdot\mathbf{V}_{t}^{\ell}.$ (6) $$

Having obtained the task-specific components $\mathbf{B}_{t}^{\ell}$ and $\mathbf{A}_{t}^{\ell}$ for each task $t$, we define the corresponding low-rank initialization subspace at layer $\ell$ as $\mathbf{W}_{t}^{\ell}\triangleq\mathbf{B}_{t}^{\ell}\mathbf{A}_{t}^{\ell}.$
We aim to fuse the initialization subspaces of different tasks.
To this end, we realize the fusion via the following blockwise concatenation:

$$ $\mathbf{B}^{\ell}=\left[\mathbf{B}_{1}^{\ell},\cdots,\mathbf{B}_{T}^{\ell}\right],\mathbf{A}^{\ell}=\left[\mathbf{A}_{1}^{\ell};\cdots;\mathbf{A}_{T}^{\ell}\right],$ (7) $$

We demonstrate in the proposition below that this blockwise concatenation is effect-equivalent to the additive composition of the task-specific low-rank components.

###### Proposition 3.1 (Blockwise composition) .

Let $\mathbf{B}_{t}^{\ell}\in\mathbb{R}^{d_{\mathrm{out}}\times r_{t}}$ and $\mathbf{A}_{t}^{\ell}\in\mathbb{R}^{r_{t}\times d_{\mathrm{in}}}$ for tasks $t=1,\dots,T$, and define
$\mathbf{B}^{\ell}=[\mathbf{B}_{1}^{\ell},\dots,\mathbf{B}_{T}^{\ell}]\in\mathbb{R}^{d_{\mathrm{out}}\times r}$ and
$\mathbf{A}^{\ell}=[\mathbf{A}_{1}^{\ell};\dots;\mathbf{A}_{T}^{\ell}]\in\mathbb{R}^{r\times d_{\mathrm{in}}}$ with $r=\sum_{t}r_{t}$.
Then, for any input $\mathbf{X}\in\mathbb{R}^{d_{\mathrm{in}}\times n}$,

$$ $\mathbf{B}^{\ell}\mathbf{A}^{\ell}\mathbf{X}\;=\;\sum_{t=1}^{T}\mathbf{B}_{t}^{\ell}\mathbf{A}_{t}^{\ell}\mathbf{X}.$ (8) $$

###### Proof.

See Appendix [C.1](https://arxiv.org/html/2505.18640v2#A3.SS1).
∎

Thus, we fuse task-specific LoRA initialization subspaces from different tasks with heterogeneous ranks via blockwise concatenation, enabling knowledge injection into low-rank subspaces. Meanwhile, this property preserves the initialization parameter subspace of each task explicitly, facilitating the regularization of subsequent multi-task training.
We then concatenate all task-specific LoRA components along with the cooperative block:

$$ $\mathbf{B}^{\ell}=\left[\mathbf{B}_{1}^{\ell},\cdots,\mathbf{B}_{T}^{\ell},\mathbf{B}_{\text{coop}}^{\ell}\right],\mathbf{A}^{\ell}=\left[\mathbf{A}_{1}^{\ell};\cdots;\mathbf{A}_{T}^{\ell};\mathbf{A}_{\text{coop}}^{\ell}\right],$ (9) $$

where $r_{\text{coop}}=\big\lfloor\frac{r_{\text{total}}}{T+1}\big\rfloor$, $\mathbf{A}_{\text{coop}}^{\ell}\in\mathbb{R}^{r_{\text{coop}}\times d_{\text{in}}}$ is randomly initialized using Kaiming uniform distribution and
$\mathbf{B}_{\text{coop}}^{\ell}\in\mathbb{R}^{d_{\text{out}}\times r_{\text{coop}}}$ is initialized as zero. All components are further scaled by a global factor $\gamma$ to modulate the strength of knowledge injection.
Thus, we establish a task-aware, layer-specific low-rank parameterization that encodes heterogeneous priors into structurally decoupled subspaces, providing a strong starting point for collaborative multi-task adaptation.

###### Proposition 3.1 (Blockwise composition) .

###### Proof.

#### 3.2.2 Subspace-Preserving Regularization (SPR)

Motivation.
While HASI in [Sec. 3.2.1](https://arxiv.org/html/2505.18640v2#S3.SS2.SSS1) ensures task-specific subspace initialization,
receiving identical sample signals within the same low-rank subspace during multi-task training
can cause overlaps in their representation directions, thereby eroding the diversity of the initialized subspaces.
This results in an under-utilization of the low-rank parameter space, where performance fail to improve with larger ranks (see [fig. 5](https://arxiv.org/html/2505.18640v2#S4.F5)). We define this phenomenon as subspace collapse.
To address this, we aim to promote the diversity of the subspaces initialized by HASI during multi-task training.

Decomposed Orthogonality.
However, directly imposing orthogonality constraints on the task-specific parameter subspaces $\mathbf{B}_{t}^{\ell}\mathbf{A}_{t}^{\ell}$ incurs substantial computational and memory overhead, as it requires computing pairwise Frobenius inner products between the full update matrices at every layer, and the cost grows rapidly with the number of tasks. Please see Appendix [B](https://arxiv.org/html/2505.18640v2#A2) for additional discussion.
To mitigate this, we adopt a decomposed formulation by separately enforcing orthogonality on the LoRA factors $\mathbf{A}_{t}^{\ell}$ and $\mathbf{B}_{t}^{\ell}$, which serves as a sufficient condition for orthogonality of the full updates. This design significantly reduces the regularization overhead while maintaining effective subspace separation.
We formalize this condition with the following proposition.

###### Proposition 3.2 (Sufficient Condition for Orthogonality of LoRA Updates) .

Let $\mathbf{W}_{1}=\mathbf{B}_{1}\mathbf{A}_{1}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$ and $\mathbf{W}_{2}=\mathbf{B}_{2}\mathbf{A}_{2}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$. If either

$$ $\mathbf{B}_{1}^{\top}\mathbf{B}_{2}=\mathbf{0}\quad\text{or}\quad\mathbf{A}_{1}\mathbf{A}_{2}^{\top}=\mathbf{0},$ (10) $$

then $\langle\mathbf{W}_{1},\mathbf{W}_{2}\rangle_{F}=0$.

Proof. See Appendix [C.2](https://arxiv.org/html/2505.18640v2#A3.SS2)
∎

To enforce this principle during training, we introduce a subspace-diversifying regularization term that explicitly encourages diversity and prevents representation overlap between subspaces, thereby improving the utilization of the low-rank parameter space and mitigating subspace collapse.
For each layer $\ell$, the regularization loss is defined as:

$$ $\mathcal{L}_{\text{SPR}}^{\ell}=\sum_{t_{1}<t_{2}}\left\|\mathbf{B}_{t_{1}}^{\ell}{}^{\top}\mathbf{B}_{t_{2}}^{\ell}\right\|_{F}^{2}+\left\|\mathbf{A}_{t_{1}}^{\ell}\mathbf{A}_{t_{2}}^{\ell}{}^{\top}\right\|_{F}^{2},$ (11) $$

where the pairwise Frobenius norms measure the degree of inter-task subspace entanglement. Minimizing this term encourages representational diversity across tasks, thereby preserving the decoupled subspace structure established during initialization.

During training, this regularization is applied jointly with the supervised objective. The total training objective becomes:

$$ $\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{task}}+\lambda\sum_{\ell}\mathcal{L}_{\text{SPR}}^{\ell},$ (12) $$

where $\lambda$ is a hyperparameter controlling the strength of subspace regularization. This formulation allows the model to jointly optimize task performance and representational diversity, ensuring that the subspace knowledge injection introduced by HASI remains effective throughout training.

###### Proposition 3.2 (Sufficient Condition for Orthogonality of LoRA Updates) .

## 4 Experiments

**Table 1: Comparison on two-task mixtures across various multimodal and text-only datasets. Merge indicates whether the method can be merged back into original model for inference.**
|  | IconQA - ScienceQA | ChartQA - DocVQA | CSQA - OBQA |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | IconQA | ScienceQA | Avg | Norm | ChartQA | DocVQA | Avg | Norm | xCSQA. | .OBQAx | Avg | Norm | Merge |
| Mixture of 2 Tasks |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Individual FT | 77.53 | 87.51 | 82.52 | 100 | 33.00 | 30.64 | 31.82 | 100 | 74.53 | 76.20 | 75.37 | 100 | - |
| LoRA | 77.47 | 87.61 | 82.54 | 100.02 | 32.52 | 30.38 | 31.45 | 98.84 | 74.86 | 78.80 | 76.83 | 101.94 | ✓ |
| DoRA | 77.53 | 88.50 | 83.02 | 100.61 | 32.24 | 30.25 | 31.25 | 98.21 | 75.76 | 79.60 | 77.68 | 103.06 | ✓ |
| CorDA | 73.84 | 86.66 | 80.25 | 97.25 | 32.32 | 31.11 | 31.72 | 99.69 | 76.25 | 78.20 | 77.23 | 102.47 | ✓ |
| DARE | 70.33 | 84.10 | 77.22 | 93.58 | 27.36 | 27.91 | 27.64 | 86.86 | 72.56 | 70.00 | 71.28 | 94.57 | ✓ |
| KnOTS | 70.52 | 84.18 | 77.35 | 93.73 | 27.42 | 28.32 | 27.87 | 87.59 | 72.65 | 70.20 | 71.43 | 94.77 | ✓ |
| HydraLoRA${}_{\text{{(3B1A)}}}$ | 76.88 | 87.80 | 82.34 | 99.78 | 32.44 | 31.09 | 31.77 | 99.84 | 74.61 | 77.60 | 76.11 | 100.98 | ✗ |
| HydraLoRA${}_{\text{{(4B1A)}}}$ | 77.07 | 88.00 | 82.54 | 100.02 | 32.36 | 31.15 | 31.76 | 99.81 | 73.71 | 75.80 | 74.76 | 99.19 | ✗ |
| ThanoRA | 79.08 | 89.74 | 84.41 | 102.29 | 33.20 | 31.23 | 32.22 | 101.26 | 76.99 | 80.20 | 78.60 | 104.29 | ✓ |

### 4.1 Experimental Setup

Datasets and Architecture.
We evaluate our framework on multi-task settings covering both multimodal and text-only datasets, including three two-task mixtures (IconQA + ScienceQA, DocVQA + ChartQA, CSQA + OBQA) and a four-task mixture (IconQA, ScienceQA, DocVQA, ChartQA). Each dataset contributes 5,000 training samples. Detailed descriptions are provided in Appendix [A](https://arxiv.org/html/2505.18640v2#A1).

**Table 2: Comparison on four-task mixtures.**
|  | ChartQA - DocVQA - IconQA - ScienceQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | ChartQA | DocVQA | IconQA | SQA | Avg | Norm | Merge |
| Mixture of 4 Tasks |  |  |  |  |  |  |  |
| Individual FT | 33.00 | 30.64 | 77.53 | 87.51 | 57.17 | 100 | - |
| LoRA | 32.80 | 30.86 | 73.65 | 85.32 | 55.66 | 97.36 | ✓ |
| DoRA | 32.12 | 30.80 | 73.65 | 85.92 | 55.62 | 97.29 | ✓ |
| CorDA | 31.52 | 29.57 | 72.59 | 86.32 | 55.00 | 96.20 | ✓ |
| DARE | 23.52 | 25.41 | 55.15 | 74.32 | 44.60 | 78.01 | ✓ |
| KnOTS | 23.72 | 24.44 | 56.51 | 75.10 | 44.94 | 78.61 | ✓ |
| HydraLoRA${}_{\text{(3B1A)}}$ | 32.96 | 30.66 | 73.05 | 84.93 | 55.40 | 96.90 | ✗ |
| HydraLoRA${}_{\text{(4B1A)}}$ | 33.32 | 30.79 | 72.51 | 84.93 | 55.39 | 96.89 | ✗ |
| ThanoRA | 32.72 | 30.62 | 75.08 | 87.36 | 56.45 | 98.74 | ✓ |

Implementation Details.
All baselines are implemented within the LLaVA-1.5(^*^**[https://github.com/haotian-liu/LLaVA](https://github.com/haotian-liu/LLaVA)) framework. All PEFT methods are applied to every layer of the LLM component in LLaVA. The learning rate is set to $2e\!-\!5$ and the global batch size to 32. We train for 3 epochs on multimodal datasets and 1 epoch on the text-only mixture. Following previous work Yang et al. (2024), we sample a small part of samples (e.g. 500 per task) to initialize the task-specific subspaces in HASI.

Counterparts.
We compare ThanoRA against three representative categories of methods: ❶ Mergeable LoRA Variants , ❷ Model Merging, and ❸ MoE-based LoRA, including:
(a) Individual Fine-tune, where a separate LoRA module is trained for each task individually, serving as a strong baseline for multi-task adaptation;
(b) LoRA Hu et al. (2022), which updates parameters in a low-rank subspace for efficient adaptation;
(c) DoRA Liu et al. (2024b), which decouples low-rank direction and magnitude for improved performance;
(d) CorDA Yang et al. (2024), which previews task instructions to initialize LoRA modules based on knowledge priors;
(e) DARE Yu et al. (2024), a drop-and-rescale strategy that sparsifies delta weights for efficient multi-model fusion without retraining or GPU resources;
(f) KnOTS Stoica et al. (2025), a data-free LoRA merging method that leverages SVD to align task-specific updates.
and (g) HydraLoRA Tian et al. (2024), an asymmetric MoE-style LoRA framework that excels in handling complex multi-task scenarios. We evaluate its two official settings: 3B1A and 4B1A (i.e., three or four ${rank}{=}32$ B matrices with one $rank{=}32$ A matrix), both with parameter counts no less than standard LoRA with $rank{=}64$. All other baselines are configured with a default LoRA rank of 64 for fair comparison. For discussions under varying rank settings, please refer to [Sec. 4.3.3](https://arxiv.org/html/2505.18640v2#S4.SS3.SSS3).

### 4.2 Comparison to State-of-the-Arts

Performance Evaluation.
We evaluate ThanoRA across diverse multi-task settings, with results reported in [Tabs. 1](https://arxiv.org/html/2505.18640v2#S4.T1), [2](https://arxiv.org/html/2505.18640v2#S4.T2) and [5](https://arxiv.org/html/2505.18640v2#S4.T5). Here, “Avg” denotes the average performance across tasks, while “Norm” represents the percentage of Avg relative to individual fine-tuning performance, quantifying how closely multi-task learning approaches the strong baseline individual fine-tuning, following prior work Stoica et al. (2025).
Key observations are summarized:

i) Limited multi-task adaptability of existing methods.
Strong baselines (e.g., CorDA, DoRA) excel on text-only multi-task benchmarks but struggle on more complex multimodal ones; Training-free approaches (e.g., DARE, KnOTS) enable efficient LoRA model merging but still lag behind multi-task training in overall performance.

**Table 3: Ablation Study of Key Modules.**
|  |  | IconQA-ScienceQA | CSQA-OBQA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| HASI | SPR | IconQA | SQA | Avg | CSQA | OBQA | Avg |
| LoRA | 77.47 | 87.61 | 82.54 | 74.86 | 78.80 | 76.83 |  |
| ✓ |  | 78.28 | 89.04 | 83.66 | 76.82 | 79.00 | 77.91 |
| ✓ | ✓ | 79.08 | 89.74 | 84.41 | 76.99 | 80.20 | 78.60 |

ii) ThanoRA robustly improves multi-task adaptation. ThanoRA consistently achieves superior performance across diverse task combinations and benchmarks. Moreover, by enabling more effective knowledge injection into low-rank subspaces, it surpasses even single-task fine-tuning (see [Tab. 5](https://arxiv.org/html/2505.18640v2#S4.T5)), which serves as a strong multi-task training baseline. While counterparts such as HydraLoRA may improve performance on one task at the expense of another, ThanoRA consistently achieves better and more balanced gains across tasks in multi-task training.

**Table 4: Comparison of Inference Latency (s) and Throughput (it/s) across datasets. Arrows indicate changes relative to HydraLoRA${}_{\text{{(3B1A)}}}$. Accuracy denotes the average performance across tasks in the two-task mixture setting in [Tab. 1](https://arxiv.org/html/2505.18640v2#S4.T1).**
| Method | ChartQA | DocVQA | IconQA | ScienceQA | CSQA | OBQA | Accuracy |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Latency (s) (↓) |  |  |  |  |  |  |  |
| HydraLoRA${}_{\text{{(3B1A)}}}$ | 802 | 2095 | 1298 | 723 | 100 | 40 | 63.40 |
| HydraLoRA${}_{\text{{(4B1A)}}}$ | 885↑10.35% | 2342↑11.79% | 1443↑11.17% | 785↑8.58% | 109↑9.00% | 44↑10.00% | 63.02 |
| ThanoRA | 592↓26.18% | 1516↓27.64% | 1048↓19.26% | 489↓32.37% | 64↓36.00% | 26↓35.00% | 65.08 |
| Throughput (it/s) (↑) |  |  |  |  |  |  |  |
| HydraLoRA${}_{\text{{(3B1A)}}}$ | 3.14 | 2.55 | 4.86 | 5.86 | 12.07 | 12.18 | 63.40 |
| HydraLoRA${}_{\text{{(4B1A)}}}$ | 2.84 ↓9.55% | 2.28 ↓10.59% | 4.38 ↓9.98% | 5.40 ↓7.85% | 11.18 ↓7.37% | 10.95 ↓10.10% | 63.02 |
| ThanoRA | 4.24↑35.03% | 3.53↑38.43% | 6.03↑24.07% | 8.65↑47.61% | 18.79↑55.68% | 18.71↑53.61% | 65.08 |

**Table 5: Relative performance to Individual FT. Computed as (method / Individual FT) * 100.**
|  | IconQA - ScienceQA | ChartQA - DocVQA | CSQA - OBQA |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | IconQA | ScienceQA | Avg | ChartQA | DocVQA | Avg | CSQA | OBQA | Avg | Merge |
| KnOTS | 90.96 | 96.19 | 93.73 | 83.09 | 92.43 | 87.59 | 97.48 | 92.13 | 94.77 | ✓ |
| HydraLoRA${}_{\text{{(4B1A)}}}$ | 99.41 | 100.56 | 100.02 | 98.06 | 101.66 | 99.81 | 98.90 | 99.48 | 99.19 | ✗ |
| ThanoRA | 102.00 | 102.55 | 102.29 | 100.61 | 101.93 | 101.26 | 103.30 | 105.25 | 104.29 | ✓ |

Efficiency Analysis.
We compare ThanoRA with the MoE-based method HydraLoRA in terms of both inference efficiency and accuracy, as reported in [Tab. 4](https://arxiv.org/html/2505.18640v2#S4.T4). Compared with HydraLoRA${}_{\text{(3B1A)}}$, ThanoRA not only achieves consistently higher accuracy but also reduces latency by 20%–35% and improves throughput by 24%–56% across all tasks. The 4B1A variant of HydraLoRA further increases latency and decreases throughput around 10%, highlighting the scalability limitations of MoE-based LoRA. All evaluations are conducted on a single NVIDIA L20 GPU. Additional results in Appendix [D](https://arxiv.org/html/2505.18640v2#A4) further demonstrate the training efficiency of ThanoRA compared to baselines. These results underscore the practical advantage of mergeable LoRA designs like ThanoRA, particularly in latency-sensitive deployment scenarios such as on edge devices.

Overall, ThanoRA delivers a favorable balance between accuracy and efficiency, making it well-suited for real-world multi-task applications with both performance and deployment constraints.

### 4.3 Diagnostic Analysis

#### 4.3.1 Ablation Study.

Figure: Figure 3: Hyperparameter Study. Impact of regularization weight $\lambda$, Scale factor $\gamma$, and Temperature $\tau$ on accuracy in IconQA–ScienceQA mixture. ThanoRA consistently outperforms the LoRA baseline (gray dashed line) under almost all configurations. Please see Sec. [4.3.1](https://arxiv.org/html/2505.18640v2#S4.SS3.SSS1) for details.
Refer to caption: x3.png

[Tab. 3](https://arxiv.org/html/2505.18640v2#S4.T3) shows that HASI alone outperforms the LoRA baseline, while adding SPR further improves performance by mitigating subspace collapse, confirming their complementary roles. We further evaluate the impact of three key hyperparameters in [fig. 3](https://arxiv.org/html/2505.18640v2#S4.F3): the regularization weight $\lambda$, scaling factor $\gamma$, and entropy temperature $\tau$. Optimal values ($\lambda{=}10^{-4}$, $\gamma{=}5$, $\tau{=}0.2$) are used in main experiments.

#### 4.3.2 Visualization of Rank Allocation.

Fig. [4](https://arxiv.org/html/2505.18640v2#S4.F4) visualizes the layer-wise rank allocation of ThanoRA across task mixtures. In the four-task setting, DocVQA Mathew et al. (2021) receives the highest ranks in shallow layers, while IconQA Lu et al. (2021) consistently obtains the lowest, consistent with the relative complexity of their visual inputs (see Appendix [A](https://arxiv.org/html/2505.18640v2#A1)). This supports prior findings that shallow layers of MLLMs focus on cross-modal fusion Zhang et al. (2024). In contrast, deeper layers allocate higher ranks to ScienceQA Lu et al. (2022); Huang et al. (2025), reflecting its reliance on complex textual reasoning, consistent with the semantic role of later layers.

Figure: Figure 4: Layer-wise Rank Allocation. Lines indicate average allocated ranks per layer; shaded areas show rank range across blocks. Higher ranks in shallow layers for DocVQA and in deep layers for ScienceQA align with their respective visual and reasoning complexities. Please see [Sec. 4.3.2](https://arxiv.org/html/2505.18640v2#S4.SS3.SSS2).
Refer to caption: x4.png

#### 4.3.3 Experiments and Analysis on Different Rank.

Figure: Figure 5: Results Under Different Rank.
Refer to caption: x5.png

We evaluate the performance of ThanoRA under different LoRA ranks $r\in\{16,32,64,128\}$, as shown in [fig. 5](https://arxiv.org/html/2505.18640v2#S4.F5). Across all settings, both ThanoRA and its variant without SPR consistently outperform the LoRA baseline, demonstrating its robustness.
Notably, vanilla LoRA fails to benefit from increasing rank, indicating inefficient use and redundancy of the parameter space. In contrast, initializing with HASI yields the best performance at $r=32$, indicating improved rank efficiency through task-aware subspace construction. When further combined with SPR, performance continues to improve and peaks at $r=64$, confirming that SPR successfully mitigates subspace collapse and enables more effective knowledge injection across tasks.

## 5 Conclusion

In this paper, we propose ThanoRA, an effective and efficient Task Heterogeneity-Aware framework for multi-task low-rank adaptation without non-mergeable structures. By combining Heterogeneity-Aware Subspace Initialization (HASI) with Subspace-Preserving Regularization (SPR), ThanoRA enables fine-grained knowledge injection and prevents subspace collapse. Experiments on multimodal and text-only benchmarks show that it consistently outperforms strong baselines and even surpasses single-task fine-tuning, while maintaining the mergeability of LoRA and thus incurring no additional storage cost or inference latency.

## The Use of Large Language Models (LLMs)

We clarify that LLMs were used exclusively for language refinement—improving academic expression, coherence of arguments, and overall clarity. All core ideas, research designs, experiments, and conclusions are solely the work of the authors.

## Reproducibility

For reproducibility, the supplementary material includes the source code of the training and evaluation frameworks, while all experimental setups and training hyperparameters are detailed in [Sec. 4.1](https://arxiv.org/html/2505.18640v2#S4.SS1).

## Ethics and Social Impact

Our method enables efficient multi-task adaptation without introducing additional non-mergeable structures, preserving both training and inference efficiency. This design makes it particularly suitable for deployment in resource-constrained environments, such as edge devices or low-latency applications. By supporting multiple downstream capabilities with minimal overhead, our approach can facilitate broader accessibility to foundation models in domains like mobile health, assistive technologies, and real-time decision-making.

## References

- Agiza et al. (2024)
Ahmed Agiza, Marina Neseem, and Sherief Reda.
Mtlora: Low-rank adaptation approach for efficient multi-task learning.
In *CVPR*, 2024.
- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Ban & Ji (2024)
Hao Ban and Kaiyi Ji.
Fair resource allocation in multi-task learning.
*arXiv preprint arXiv:2402.15638*, 2024.
- Chen et al. (2024)
Shaoxiang Chen, Zequn Jie, and Lin Ma.
Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms.
*arXiv preprint arXiv:2401.16160*, 2024.
- Crawshaw (2020)
Michael Crawshaw.
Multi-task learning with deep neural networks: A survey.
*arXiv preprint arXiv:2009.09796*, 2020.
- Fang et al. (2025)
Yiyang Fang, Jian Liang, Wenke Huang, He Li, Kehua Su, and Mang Ye.
Catch your emotion: Sharpening emotion perception in multimodal large language models.
In *ICML*, 2025.
- Feng et al. (2024)
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang.
Mixture-of-loras: An efficient multitask tuning for large language models.
*arXiv preprint arXiv:2403.03432*, 2024.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Guo et al. (2020)
Pengsheng Guo, Chen-Yu Lee, and Daniel Ulbricht.
Learning to branch for multi-task learning.
In *ICML*, 2020.
- Hayou et al. (2024)
Soufiane Hayou, Nikhil Ghosh, and Bin Yu.
Lora+: Efficient low rank adaptation of large models.
*arXiv preprint arXiv:2402.12354*, 2024.
- Houlsby et al. (2019)
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.
Parameter-efficient transfer learning for nlp.
In *ICML*, 2019.
- Hu et al. (2022)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
In *ICLR*, 2022.
- Hu et al. (2024)
Jiajun Hu, Jian Zhang, Lei Qi, Yinghuan Shi, and Yang Gao.
Learn to preserve and diversify: Parameter-efficient group with orthogonal regularization for domain generalization.
In *ECCV*, 2024.
- Huang et al. (2023)
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin.
Lorahub: Efficient cross-task generalization via dynamic lora composition.
*arXiv preprint arXiv:2307.13269*, 2023.
- Huang et al. (2025)
Wenke Huang, Jian Liang, Xianda Guo, Yiyang Fang, Guancheng Wan, Xuankun Rong, Chi Wen, Zekun Shi, Qingyun Li, Didi Zhu, et al.
Keeping yourself is important in downstream tuning multimodal large language model.
*arXiv preprint arXiv:2503.04543*, 2025.
- Karimi Mahabadi et al. (2021)
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder.
Compacter: Efficient low-rank hypercomplex adapter layers.
*NeurIPS*, 2021.
- Kopiczko et al. (2023)
Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano.
Vera: Vector-based random matrix adaptation.
*arXiv preprint arXiv:2310.11454*, 2023.
- Lester et al. (2021)
Brian Lester, Rami Al-Rfou, and Noah Constant.
The power of scale for parameter-efficient prompt tuning.
*arXiv preprint arXiv:2104.08691*, 2021.
- Li et al. (2024a)
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li.
Llava-onevision: Easy visual task transfer.
*arXiv preprint arXiv:2408.03326*, 2024a.
- Li et al. (2024b)
Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, et al.
Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts.
*arXiv preprint arXiv:2404.15159*, 2024b.
- Li & Liang (2021)
Xiang Lisa Li and Percy Liang.
Prefix-tuning: Optimizing continuous prompts for generation.
*arXiv preprint arXiv:2101.00190*, 2021.
- Liang et al. (2025)
Jian Liang, Wenke Huang, Guancheng Wan, Qu Yang, and Mang Ye.
Lorasculpt: Sculpting lora for harmonizing general and specialized knowledge in multimodal large language models.
In *CVPR*, 2025.
- Liu et al. (2022)
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel.
Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.
*NeurIPS*, 2022.
- Liu et al. (2023)
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning.
In *CVPR*, 2023.
- Liu et al. (2024a)
Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng.
When moe meets llms: Parameter efficient fine-tuning for multi-task medical applications.
In *SIGIR*, 2024a.
- Liu et al. (2024b)
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen.
Dora: Weight-decomposed low-rank adaptation.
In *ICML*, 2024b.
- Lu et al. (2021)
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu.
Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning.
In *NeurIPS*, 2021.
- Lu et al. (2022)
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.
Learn to explain: Multimodal reasoning via thought chains for science question answering.
In *NeurIPS*, 2022.
- Ma et al. (2019)
Jiaqi Ma, Zhe Zhao, Jilin Chen, Ang Li, Lichan Hong, and Ed H Chi.
Snr: Sub-network routing for flexible parameter sharing in multi-task learning.
In *AAAI*, 2019.
- Masry et al. (2022)
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque.
Chartqa: A benchmark for question answering about charts with visual and logical reasoning.
*arXiv preprint arXiv:2203.10244*, 2022.
- Mathew et al. (2021)
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.
Docvqa: A dataset for vqa on document images.
In *WACV*, 2021.
- Meng et al. (2024)
Fanxu Meng, Zhaohui Wang, and Muhan Zhang.
Pissa: Principal singular values and singular vectors adaptation of large language models.
*NeurIPS*, 2024.
- Mihaylov et al. (2018)
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.
Can a suit of armor conduct electricity? a new dataset for open book question answering.
*arXiv preprint arXiv:1809.02789*, 2018.
- Pfeiffer et al. (2020)
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych.
Adapterfusion: Non-destructive task composition for transfer learning.
*arXiv preprint arXiv:2005.00247*, 2020.
- Rong et al. (2025)
Xuankun Rong, Wenke Huang, Jian Liang, Jinhe Bi, Xun Xiao, Yiming Li, Bo Du, and Mang Ye.
Backdoor cleaning without external guidance in mllm fine-tuning.
*NeurIPS*, 2025.
- Shen et al. (2024)
Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang.
Multimodal instruction tuning with conditional mixture of lora.
*arXiv preprint arXiv:2402.15896*, 2024.
- Stoica et al. (2025)
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman.
Model merging with svd to tie the knots.
*ICLR*, 2025.
- Sun et al. (2021)
Guolei Sun, Thomas Probst, Danda Pani Paudel, Nikola Popović, Menelaos Kanakis, Jagruti Patel, Dengxin Dai, and Luc Van Gool.
Task switching network for multi-task learning.
In *ICCV*, 2021.
- Talmor et al. (2018)
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.
Commonsenseqa: A question answering challenge targeting commonsense knowledge.
*arXiv preprint arXiv:1811.00937*, 2018.
- Tian et al. (2024)
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu.
Hydralora: An asymmetric lora architecture for efficient fine-tuning.
*NeurIPS*, 2024.
- Wallingford et al. (2022)
Matthew Wallingford, Hao Li, Alessandro Achille, Avinash Ravichandran, Charless Fowlkes, Rahul Bhotika, and Stefano Soatto.
Task adaptive parameter sharing for multi-task learning.
In *CVPR*, 2022.
- Wang et al. (2025)
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al.
Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.
*arXiv preprint arXiv:2508.18265*, 2025.
- Wang et al. (2023)
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang.
Orthogonal subspace learning for language model continual learning.
*arXiv preprint arXiv:2310.14152*, 2023.
- Wang et al. (2024)
Xujia Wang, Haiyan Zhao, Shuo Wang, Hanqing Wang, and Zhiyuan Liu.
Malora: Mixture of asymmetric low-rank adaptation for enhanced multi-task learning.
*arXiv preprint arXiv:2410.22782*, 2024.
- Wu et al. (2024a)
Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo.
Llama pro: Progressive llama with block expansion.
*arXiv preprint arXiv:2401.02415*, 2024a.
- Wu et al. (2024b)
Xun Wu, Shaohan Huang, and Furu Wei.
Mixture of lora experts.
*arXiv preprint arXiv:2404.13628*, 2024b.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Yang et al. (2024)
Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem.
Corda: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine-tuning.
*NeurIPS*, 2024.
- Yang & Hospedales (2016)
Yongxin Yang and Timothy M Hospedales.
Trace norm regularised deep multi-task learning.
*arXiv preprint arXiv:1606.04038*, 2016.
- Ye et al. (2025)
Mang Ye, Xuankun Rong, Wenke Huang, Bo Du, Nenghai Yu, and Dacheng Tao.
A survey of safety on large vision-language models: Attacks, defenses and evaluations.
*arXiv preprint arXiv:2502.14881*, 2025.
- Yu et al. (2024)
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li.
Language models are super mario: Absorbing abilities from homologous models as a free lunch.
In *ICML*, 2024.
- Zeng et al. (2025)
Fanhu Zeng, Haiyang Guo, Fei Zhu, Li Shen, and Hao Tang.
Parameter efficient merging for multimodal large language models with complementary parameter adaptation.
*NeurIPS*, 2025.
- Zhang et al. (2023)
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao.
Adalora: Adaptive budget allocation for parameter-efficient fine-tuning.
*arXiv preprint arXiv:2303.10512*, 2023.
- Zhang & Yang (2021)
Yu Zhang and Qiang Yang.
A survey on multi-task learning.
*TKDE*, 2021.
- Zhang & Yeung (2014)
Yu Zhang and Dit-Yan Yeung.
A regularization approach to learning task relationships in multitask learning.
*TKDD*, 2014.
- Zhang et al. (2024)
Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova.
Cross-modal information flow in multimodal large language models.
*arXiv preprint arXiv:2411.18620*, 2024.
- Zhao et al. (2025)
Ziyu Zhao, Yixiao Zhou, Didi Zhu, Tao Shen, Xuwu Wang, Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu, and Yu Cheng.
Each rank could be an expert: Single-ranked mixture of experts lora for multi-task learning.
*arXiv preprint arXiv:2501.15103*, 2025.
- Zheng et al. (2025)
Shenghe Zheng, Hongzhi Wang, Chenyu Huang, Xiaohui Wang, Tao Chen, Jiayuan Fan, Shuyue Hu, and Peng Ye.
Decouple and orthogonalize: A data-free framework for lora merging.
*arXiv preprint arXiv:2505.15875*, 2025.
- Zhu et al. (2025)
Didi Zhu, Yibing Song, Tao Shen, Ziyu Zhao, Jinluan Yang, Min Zhang, and Chao Wu.
Remedy: Recipe merging dynamics in large vision-language models.
In *ICLR*, 2025.

## Appendix

## Appendix A Dataset Details

Here we provide detailed descriptions of the six downstream datasets used in our experiments. These datasets cover diverse modalities and task types, including both multimodal benchmarks with distinct visual-textual characteristics and text-only reasoning benchmarks, allowing a comprehensive evaluation of multi-task adaptation under heterogeneous settings.

##### ScienceQA.

ScienceQA Lu et al. (2022) is a multimodal multiple-choice benchmark focused on science education, featuring questions grounded in both textual and visual contexts. We adopt 5,000 training and 2,017 test examples. Each sample presents a science question accompanied by both image-based and textual options, and the model is tasked with selecting the correct answer label (e.g., “A”, “B”). Accuracy is used as the primary evaluation metric.

##### IconQA.

IconQA Lu et al. (2021) targets abstract diagram understanding, challenging models to perform reasoning over symbolic and schematic visual inputs. We adopt the multiple-choice setting with 5,000 training and 6,316 test instances. Each question requires the model to identify the correct option by outputting the corresponding letter label. Evaluation is based on accuracy.

##### ChartQA.

ChartQA Masry et al. (2022) is a visual question answering benchmark based on bar and line charts, requiring models to interpret and reason over structured chart content. Each instance presents a chart and a natural language question, with the answer being a short free-form textual span (typically a word or phrase).
Relaxed Accuracy is used for evaluation, which considers a prediction correct if it approximately matches any reference answer under normalized comparison rules.

##### DocVQA.

DocVQA Mathew et al. (2021) is a visual question answering benchmark on document images, where models must extract and reason over text-rich visual content. Each instance comprises a document image and a natural language question, with the expected output being a short free-form textual answer.
ANLS (Average Normalized Levenshtein Similarity) is used as the evaluation metric, measuring string-level similarity between the predicted and reference answers while allowing for minor variations in phrasing or spelling.

##### CommonSenseQA (CSQA).

CommonSenseQA Talmor et al. (2018) is a text-only multiple-choice question answering benchmark designed to evaluate a model’s ability to perform commonsense reasoning. Each instance consists of a natural language question and five answer options, where only one is correct. The model is required to return the correct option label. We use 5,000 training and 1,221 test samples. Accuracy is used for evaluation.

##### OpenBookQA (OBQA).

OpenBookQA Mihaylov et al. (2018) is a text-only multiple-choice question answering benchmark designed to assess deeper understanding of scientific topics and language. Inspired by open-book exams, it requires models to combine a small set of provided science facts with broader commonsense and multi-hop reasoning. Each instance consists of a question with four answer options, and the model is expected to select the correct option. The dataset contains 5,000 training and 500 test samples. Accuracy is used as the evaluation metric.

## Appendix B Discussion

Several prior works have explored orthogonality in LoRA to enhance generalization, including PEGO Hu et al. (2024) for domain generalization and O-LoRA Wang et al. (2023) for continual learning. PEGO imposes orthogonality across full LoRA update matrices $\mathbf{W}_{t}=\mathbf{B}_{t}\mathbf{A}_{t}$ to promote feature diversity under distribution shift. Its regularization takes the form:

$$ $\mathcal{L}_{\text{diversify}}=\sum_{i=1}^{N}\sum_{j=i+1}^{N}\left\|(B_{i}A_{i})^{\top}(B_{j}A_{j})\right\|_{1},$ (13) $$

where $N$ is the number of tasks, and $B_{i}A_{i}$ denotes the LoRA update for task $i$. However, this formulation incurs substantial computational and memory costs due to full-rank pairwise projections at each layer. As shown in [Tab. 6](https://arxiv.org/html/2505.18640v2#A2.T6), this design quickly leads to out-of-memory (OOM) errors under modest settings.
To address this, we adopt a factorized formulation that independently enforces orthogonality over $\mathbf{A}_{t}$ and $\mathbf{B}_{t}$. Crucially, we prove (see [Prop. 3.2](https://arxiv.org/html/2505.18640v2#S3.Thmproposition2)) that this factor-wise constraint is a sufficient condition for the orthogonality of the resulting LoRA updates $\mathbf{B}_{t}\mathbf{A}_{t}$, enabling efficient subspace separation with significantly lower memory overhead, while preserving the training efficiency that is essential for scaling LoRA to larger models and datasets.

O-LoRA adopts an asymmetric regularization strategy by enforcing orthogonality only on the LoRA-A matrices. While this design reduces computational overhead, it is inherently incompatible with our SVD-guided subspace initialization (HASI), which jointly determines the task-specific subspaces through the coordinated initialization of both LoRA-A and LoRA-B matrices. Asymmetric regularization on only one factor disrupts this alignment, undermining the structural integrity established during initialization.

Notably, O-LoRA emphasizes knowledge isolation between LoRA modules across sequential tasks in continual learning. In contrast, our method imposes structural constraints within LoRA by regularizing subspaces across tasks, while still allowing unconstrained components to facilitate inter-task collaboration (see  [eq. 9](https://arxiv.org/html/2505.18640v2#S3.E9)). This distinction reflects different goals: isolating inter-task interference versus enabling coordinated multi-task adaptation within a mergeable framework.

**Table 6: Memory overhead (MB) analysis of LoRA orthogonal regularization. “QKV Only” denotes applying regularization to LoRA of $W_{Q},W_{K},W_{V}$ only; “All Layers” includes all LoRA-injected layers. OOM indicates out-of-memory on single 48GB device. LoRA rank is set to 64.**
|  | ThanoRA | PEGO Hu et al. (2024) |  |  |
| --- | --- | --- | --- | --- |
| Memory Overhead (MB) | QKV Only | All Layers | QKV Only | All Layers |
| 2 Tasks | 65.16 | 206.45 | 15360.23 | OOM |
| 4 Tasks | 237.39 | 751.36 | OOM | OOM |

## Appendix C Proof of Propositions

### C.1 Proof of Prop. 3.1

Proposition 3.1 (Blockwise composition).
Let $\mathbf{B}_{t}^{\ell}\in\mathbb{R}^{d_{\mathrm{out}}\times r_{t}}$ and $\mathbf{A}_{t}^{\ell}\in\mathbb{R}^{r_{t}\times d_{\mathrm{in}}}$ for tasks $t=1,\dots,T$, and define
$\mathbf{B}^{\ell}=[\mathbf{B}_{1}^{\ell},\dots,\mathbf{B}_{T}^{\ell}]\in\mathbb{R}^{d_{\mathrm{out}}\times r}$ and
$\mathbf{A}^{\ell}=[\mathbf{A}_{1}^{\ell};\dots;\mathbf{A}_{T}^{\ell}]\in\mathbb{R}^{r\times d_{\mathrm{in}}}$ with $r=\sum_{t}r_{t}$.
Then, for any input $\mathbf{X}\in\mathbb{R}^{d_{\mathrm{in}}\times n}$,

$$ $\mathbf{B}^{\ell}\mathbf{A}^{\ell}\mathbf{X}\;=\;\sum_{t=1}^{T}\mathbf{B}_{t}^{\ell}\mathbf{A}_{t}^{\ell}\mathbf{X}.$ (14) $$

###### Proof.

By block multiplication, we first compute

$$ $\mathbf{A}^{\ell}\mathbf{X}=\begin{bmatrix}\mathbf{A}_{1}^{\ell}\[-2.0pt] \vdots\[-2.0pt] \mathbf{A}_{T}^{\ell}\end{bmatrix}\mathbf{X}=\begin{bmatrix}\mathbf{A}_{1}^{\ell}\mathbf{X}\[-2.0pt] \vdots\[-2.0pt] \mathbf{A}_{T}^{\ell}\mathbf{X}\end{bmatrix}\in\mathbb{R}^{r\times n}.$ $$

Left-multiplying by $\mathbf{B}^{\ell}=[\mathbf{B}_{1}^{\ell},\dots,\mathbf{B}_{T}^{\ell}]\in\mathbb{R}^{d_{\mathrm{out}}\times r}$ yields

$$ $\mathbf{B}^{\ell}(\mathbf{A}^{\ell}\mathbf{X})=\big[\mathbf{B}_{1}^{\ell},\dots,\mathbf{B}_{T}^{\ell}\big]\begin{bmatrix}\mathbf{A}_{1}^{\ell}\mathbf{X}\[-2.0pt] \vdots\[-2.0pt] \mathbf{A}_{T}^{\ell}\mathbf{X}\end{bmatrix}=\sum_{t=1}^{T}\mathbf{B}_{t}^{\ell}(\mathbf{A}_{t}^{\ell}\mathbf{X})=\sum_{t=1}^{T}\mathbf{B}_{t}^{\ell}\mathbf{A}_{t}^{\ell}\mathbf{X},$ $$

∎

###### Proof.

### C.2 Proof of Prop. 3.2

Proposition 3.2 (Sufficient Condition for Orthogonality of LoRA Updates).
Let $\mathbf{W}_{1}=\mathbf{B}_{1}\mathbf{A}_{1}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$ and $\mathbf{W}_{2}=\mathbf{B}_{2}\mathbf{A}_{2}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$. If either

$$ $\mathbf{B}_{1}^{\top}\mathbf{B}_{2}=\mathbf{0}\quad\text{or}\quad\mathbf{A}_{1}\mathbf{A}_{2}^{\top}=\mathbf{0},$ (15) $$

then $\langle\mathbf{W}_{1},\mathbf{W}_{2}\rangle_{F}=0$.

###### Proof.

By definition of Frobenius inner product:

$$ $\langle\mathbf{W}_{1},\mathbf{W}_{2}\rangle_{F}=\mathrm{Tr}(\mathbf{W}_{1}^{\top}\mathbf{W}_{2})=\mathrm{Tr}(\mathbf{A}_{1}^{\top}\mathbf{B}_{1}^{\top}\mathbf{B}_{2}\mathbf{A}_{2}).$ (16) $$

We consider two cases:

(i) If $\mathbf{B}_{1}^{\top}\mathbf{B}_{2}=\mathbf{0}$, then:

$$ $\mathrm{Tr}(\mathbf{A}_{1}^{\top}(\mathbf{B}_{1}^{\top}\mathbf{B}_{2})\mathbf{A}_{2})=\mathrm{Tr}(\mathbf{A}_{1}^{\top}\cdot\mathbf{0}\cdot\mathbf{A}_{2})=0.$ $$

(ii) If $\mathbf{A}_{1}\mathbf{A}_{2}^{\top}=\mathbf{0}$, then using the cyclic property of the trace:

$$ $\mathrm{Tr}((\mathbf{A}_{1}^{\top}\mathbf{B}_{1}^{\top})(\mathbf{B}_{2}\mathbf{A}_{2}))=\mathrm{Tr}(\mathbf{B}_{2}\mathbf{A}_{2}\mathbf{A}_{1}^{\top}\mathbf{B}_{1}^{\top})=\mathrm{Tr}(\mathbf{B}_{2}(\mathbf{A}_{1}\mathbf{A}_{2}^{\top})^{\top}\mathbf{B}_{1}^{\top})=\mathrm{Tr}(\mathbf{B}_{2}\cdot 0\cdot\mathbf{B}_{1}^{\top})=0.$ $$

Hence, in either case, the Frobenius inner product vanishes.
∎

###### Proof.

## Appendix D Training Efficiency

In this section, we analyze the computational cost of ThanoRA and compare its overall training efficiency with MoE-based counterparts.

### D.1 One-Time Initialization Cost

Following [Sec. 3.2.1](https://arxiv.org/html/2505.18640v2#S3.SS2.SSS1), ThanoRA requires computing the inverse covariance $\mathbf{C}^{-1}$ and the singular value decomposition (SVD) of $\mathbf{W}\mathbf{C}$ to extract task-specific subspaces. Importantly, these computations are performed *only once at initialization* and thus introduce no additional overhead during training. The runtime of these operations is summarized in [Tab. 7](https://arxiv.org/html/2505.18640v2#A4.T7).

**Table 7: One-time initialization runtime.**
| Operation | Runtime (s) |
| --- | --- |
| $\mathbf{C}^{-1}$ | 249 |
| SVD | 708 |

As shown, the total cost is less than 1,000 seconds, which accounts for only 5.1% of the training time for 3 epochs and merely 1.5% for 10 epochs. This confirms that the initialization overhead is negligible compared with the overall training process.

### D.2 Overall Training Time and Performance

We further compare ThanoRA with HydraLoRA Tian et al. (2024) in terms of training time and accuracy. Results are summarized in [Tab. 8](https://arxiv.org/html/2505.18640v2#A4.T8). ThanoRA achieves better performance while requiring less training time, highlighting its advantage in both effectiveness and efficiency.

**Table 8: Training efficiency comparison. Total wall-clock training time and average accuracy.**
| Method | Training Time | Avg. Acc. (%) |
| --- | --- | --- |
| HydraLoRA | 6:26:50 | 55.40 |
| ThanoRA | 5:08:22 | 56.45 |

These results demonstrate that ThanoRA not only delivers superior accuracy but also reduces overall training time compared with MoE-based multi-task LoRA methods, underscoring its practical advantage in efficiency-sensitive scenarios.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. How does ThanoRA handle cases where tasks have vastly different dataset sizes or noise levels? Does entropy-based allocation scale in such imbalanced scenarios?

2. Could you provide quantitative overhead analysis of SPR for larger T (e.g., > 8 tasks)?

3. What advantages does the SPR principle of the proposed method offer compared to balancing multiple heterogeneous tasks through variational methods (i.e., [1], [2], [3])?
- [3] Variational multi-task learning with gumbel-softmax priors. NeurIPS 2021.

4. Could the HASI+SPR framework be extended to other PEFT paradigms (e.g., prefix-tuning or adapter fusion)?

5. Information Content: Have you analyzed whether HASI improves information retention per subspace (e.g., via mutual information or spectral energy retention metrics)?
soundness: 3
strengths: 1. The paper proposes an elegant and practically relevant way to handle task heterogeneity in multi-task LoRA adaptation, combining entropy-based rank allocation  and decomposed orthogonality regularization. The synergy between these two modules is well-motivated and experimentally validated. Unlike MoE-based methods, ThanoRA maintains full mergeability and zero inference overhead.

2. The methodology is mathematically sound and well-justified. The blockwise composition proposition and orthogonality condition are rigorously derived, while computational efficiency considerations demonstrate thoughtful design. Ablations on λ, γ, τ, and rank (r) reinforce the method’s robustness.

3. Strong and consistent performance gains on multi-task benchmarks demonstrate both generality and scalability.

4. Given the increasing industrial use of LoRA for multi-domain model adaptation, a mergeable and heterogeneity-aware multi-task approach without MoE routers is timely and impactful. ThanoRA has potential practical value for efficient adaptation of large language and multimodal models.
summary: This paper proposes ThanoRA, a task heterogeneity-aware low-rank adaptation framework for efficient and effective multi-task fine-tuning of large models. The method addresses limitations of LoRA-based multi-task adaptation where shared low-rank subspaces cause task interference and performance degradation, and MoE-based LoRA approaches that introduce inference overhead and prevent parameter merging. ThanoRA introduces two key components heterogeneity-aware subspace Initialization and subspace-preserving regularization. Experiments on multimodal and text-only multi-task benchmarks show that ThanoRA outperforms LoRA, DoRA, CorDA, HydraLoRA, and other strong baselines, achieving even higher accuracy than individual fine-tuning while maintaining mergeability and inference efficiency.
weaknesses: 1. The introduced information-theoretic calibration term is central to the method, yet its intuition and derivation could be elaborated further. Specifically, the paper states that the term encourages alignment by minimizing mutual information between task identity and normalized representations, but does not empirically analyze how much task identity leakage remains after optimization. A deeper interpretability or visualization study would strengthen the claim.

2. While ThanoRA presents a well-motivated and empirically effective approach for handling heterogeneous multi-task adaptation, its novelty and theoretical depth are limited when viewed in light of recent advances such as VIB-MTL [1] (VIB+UW) and VIP-MTL [2], which also addresses the heterogeneous multi-task issue. ThanoRA mitigates task heterogeneity primarily through feature-level normalization and an alignment regularizer, but it lacks a probabilistic or variance-invariant formulation that guarantees fairness and balance across tasks.
- [1] Multi-task variational information bottleneck. Arxiv 2020.
- [2] Impartial Multi-task Representation Learning via Variance-invariant Probabilistic Decoding. ACL 2025.

3. Absence of theoretical analysis on fairness or representation balance. While HASI and SPR intuitively mitigate subspace collapse, the paper lacks a formal information-theoretic or variance-decomposition justification showing that task representations remain unbiased or variance-balanced across tasks.

4. Limited exploration of large-scale or highly imbalanced task settings. Experiments are restricted to small-scale mixtures. It remains unclear whether HASI’s entropy-based allocation remains effective for dozens of tasks or severely unbalanced data distributions.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1.	The explanation that the entropy of the singular values of the incremental matrix reflects the task complexity seems somewhat intuitive — could the authors provide a more rigorous or analytical justification?

2.	What are the spectral entropy values for different tasks, and how many ranks are allocated accordingly? A case study or concrete examples would strengthen the analysis.

3.	After computing the required ranks for different tasks, how are these ranks actually applied to initialize the matrices? This implementation detail seems unclear in the paper and is a crucial step in the proposed method.

4.	While orthogonality constraints can encourage diversity in rank representations, how exactly does this mechanism address task heterogeneity?

5.	What are the current state-of-the-art (SoTA) methods being compared against? The description in Section 4.2 appears somewhat inaccurate.

6.	Regarding rank orthogonality, it would be helpful to include additional visualizations to verify subspace separability in the feature space.
soundness: 2
strengths: The paper introduces a spectral-entropy-based task complexity modeling method that adaptively allocates LoRA ranks for each task and layer, enabling automatic adjustment of subspace capacity in multi-task settings.

Through rank allocation visualization (Fig. 4) and layer-wise rank distribution across tasks (Fig. 5), the paper demonstrates the model’s ability to automatically align with semantic hierarchies, thereby enhancing the interpretability of the proposed approach.
summary: This paper proposes ThanoRA, a task-heterogeneity-aware multi-task low-rank adaptation method that leverages spectral-entropy-driven subspace initialization (HASI) and subspace-preserving regularization (SPR) to dynamically allocate LoRA ranks and mitigate subspace collapse, thereby enhancing both performance and stability in multi-task learning.
weaknesses: 	The design of the SPR regularization term remains largely empirical. Although Proposition 3.2 provides sufficient conditions, the paper does not clarify which form of orthogonality (on A or B) is more effective in practice. Moreover, enforcing orthogonality introduces substantial additional training time.

	The ablation study is not sufficiently comprehensive, lacking analyses on the shared cooperative subspace as well as comparisons with alternative rank allocation strategies (e.g., uniform allocation).

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: - In Proposition.3.1, the shape of $B_t^l$ should be $d_{out} \times r_t^l$ instead of $d_{out} \times r_t$?
- In Eq.3, does $\mathbf{\sigma}$ share the same dimension across different tasks and layers? If not, it would be better to have the subscript of layers or tasks. A follow-up question is, if the dimensions are different, is it appropriate to assign $\lfloor \frac{r_{total}}{T+1} \rfloor$ as the shared subspace?
- Why do we use $\lfloor \frac{r_{total}}{T+1} \rfloor$ as the cooperative subspace? Is there any specific reason or motivation? Using an average rank as the shared subspace is confusing to me.
- The initialization in Sec.3.2.1 requires the communication among tasks. This leads to two questions. First, under what situation would this setting be valid? Second, what is the benefit of this work beyond one-shot or few-shot FL, such as [1]?
- Is the rank in truncated SVD in Eq.6 the same as $r_t^l$?
- It seems [2] is related to this paper as they proposed an initialization as well, but the author did not discuss it. The scope of [2] is not that clear either, but maybe the author can rethink their work based on [2].
- Again, Eq.11 requires the communication among clients, which could be multi-round, and the author should discuss the application scenario where this assumption is valid.
- If I understand correctly, during the training phase, we will use the resulting LoRA blocks in Eq.9. If yes, then this will introduce computation or storage overhead. Also, the application scenario should be discussed since the LoRA blocks from all the tasks are kept. Maybe the author should discuss the conventional multi-task setting.
- Though the author studies the inference latency, I think the main cost is in the training phase.
- Not sure if the author will merge the learned LoRA modules after training, though they claim modules can be merged in Fig.1.
- I think DARE is usually used with **other** merging methods, and it is not clear what "DARE" means in Tab.2.

[1] Wang, Ziyao, et al. "One Communication Round is All It Needs for Federated Fine-Tuning Foundation Models." arXiv preprint arXiv:2412.04650 (2024).

[2] Zhang, Haobo, and Jiayu Zhou. "Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging." arXiv preprint arXiv:2505.22934 (2025).
soundness: 2
strengths: - The motivation of this paper is clear and good: Model merging is efficient during inference, but may lead to performance degradation, while MoE can maintain performance, but additional modules introduce extra storage and computation cost.
- The experiments are extensive. Multiple ablation studies are conducted to analyze the properties of the proposal.
summary: Current strategies for efficient multi-task setting include model merging and MoE. However, model merging may lead to performance degradation while MoE introduces extra deployment and computation cost. Thus, this paper proposed ThanoRA, which is a two-stage framework to address task heterogeneity. First, they allocate a shared subspace and task-specific subspaces. Second, they propose an orthogonality-aware loss function during training. Extensive experiments and ablation studies are conducted to validate their framework.
weaknesses: - The writing of this paper seems hasty and needs to be improved, and some notations are abused and unclear. Please see the questions.
- The proposed framework relies on some assumptions that I am unsure whether are valid in practice. Moreover, some strategies look too intuitive and unreasonable to me. Please see the questions.
- The scope of this paper is not clear to me and needs further explanation. Please see the questions.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: - Regarding SPR, its computational overhead increases quadratically with the number of tasks T ($O(T^2)$). Have the authors evaluated the training efficiency and feasibility of this method in scenarios with a larger number of tasks (e.g., $T>10$)? Are there any approximation or optimization methods to reduce this complexity?
- Could the authors provide more analysis or intuition about the role of the “Cooperative subspace”? For instance, what types of shared knowledge does this randomly initialized and unconstrained subspace eventually capture? How does its rank ($r_{coop}$) affect the final performance?
- In Figure 5, the performance of the baseline LoRA slightly decreases as the rank increases (from 16 to 128), which is explained as inefficient utilization of the parameter space. This is an interesting phenomenon. Are there any other optimization-related factors (such as learning rate strategies) that could lead to this result? Besides HASI and SPR, does ThanoRA have any other mechanisms to avoid similar performance degradation at higher ranks?
- HASI relies on a small number of samples (e.g., 500) for “instruction preview” to compute the SVD. Is the quality of this initialization sensitive to the selection of these preview samples? For example, if the distribution of the samples deviates from the entire training set, will it affect the final performance?
soundness: 3
strengths: - **Clear Motivation and Problem Definition**: The paper excellently dissects the inherent flaws of the two mainstream approaches in existing efficient multi-task fine-tuning methods (Model Merging vs. MoE). Based on this analysis, it presents a clear and valuable research objective: achieving superior multi-task learning performance while maintaining the mergeability of LoRA.
- **Novel and Elegantly Designed Method**: The two core components of ThanoRA, HASI and SPR, are highly innovative. HASI cleverly employs spectral entropy to quantify task complexity and guide hierarchical rank allocation, which is a novel idea. SPR addresses the subspace conflict issue through a computationally efficient decomposed orthogonality constraint, balancing effectiveness and efficiency. The overall methodological framework is designed quite elegantly.
summary: This paper proposes a novel framework named ThanoRA (Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation), which aims to address the limitations of existing parameter-efficient fine-tuning (PEFT) methods for multi-task learning in large foundation models. Current mainstream approaches, such as Model Merging, can combine tasks without training but often underperform compared to joint multi-task training. Meanwhile, methods based on Mixture of Experts (MoE) can improve performance but introduce non-mergeable routing structures, resulting in additional inference overhead.

ThanoRA achieves multi-task learning by customizing subspace allocation at initialization and enforcing diversity during training. It allocates different dimensions to construct task-specific low-rank subspaces driven by task heterogeneity, thereby enabling fine-grained knowledge injection. Additionally, through an orthogonality regularization term, it encourages the subspaces of different tasks to remain independent while preserving a shared collaborative subspace. This approach mitigates task interference and subspace collapse, fully leveraging the low-rank capacity.
weaknesses: - **Scalability to Large-Scale Task Numbers**: The experiments were primarily conducted in settings with 2 and 4 mixed tasks. The computational complexity of the SPR regularization term grows quadratically with the number of tasks T ($O(T^2)$). Although the overhead is manageable for a small number of tasks, it may become a training bottleneck when the number of tasks scales up to tens or more. The paper would be more convincing if it discussed the scalability of this method to larger task numbers or proposed potential optimization strategies.
- **Insufficient Analysis of the “Cooperative Subspace”**: The paper introduces a “Cooperative subspace” designed to facilitate knowledge sharing among tasks. However, it lacks in-depth analysis of what specific shared knowledge this subspace learns and how it interacts with task-specific subspaces. For instance, visualization or similarity analysis could be employed to explore its representational characteristics, making the method design more transparent and complete.
- **Hyperparameter Sensitivity and Selection**: The method introduces several key hyperparameters, such as the regularization weight $λ$, scaling factor $γ$, and temperature coefficient $τ$. From the hyperparameter tuning experiments shown in Figure 3, the model's performance exhibits some sensitivity to the values of these hyperparameters. Although the authors provided the optimal values used in the experiments, it would be helpful to offer principles or heuristic methods for selecting these hyperparameters based on task characteristics to enhance the method's practical usability. The sample size used for HASI initialization appears to be empirically chosen. However, in high-dimensional settings, if the number of samples used to estimate the activation covariance matrix is insufficient, the resulting estimates are often dominated by noise, affecting their stability and reliability. The authors are advised to supplement the paper with theoretical analysis or empirical results to support the choice of this sample size.


Decision: Reject