MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Models

## Abstract

Abstract Low-Rank Adaptation (LoRA) has emerged as one of the most widely used parameter-efficient fine-tuning (PEFT) methods for adapting large language models (LLMs) to downstream tasks. While highly effective in single-task settings, it struggles to efficiently leverage inter-task knowledge in complex multi-task learning scenarios, often requiring substantial task-specific data to achieve optimal performance. To address this limitation, we introduce MeTA-LoRA , a two-stage optimization framework that significantly improves data efficiency in multi-task adaptation. In the first stage, task-specific LoRA adapters are learned using only a few samples from each involved dataset, enabling rapid adaptation without large-scale supervision. In the second stage, the shared LoRA adapter is updated by aggregating gradients from multiple tasks to promote knowledge transfer across tasks, further reducing data usage by leveraging common patterns. In both multi-task learning and multilingual learning scenarios, our method matches or surpasses the performance of traditional full-data LoRA fine-tuning approaches, while using significantly less task-specific data.

## 1 Introduction

Large language models (LLMs) have transformed natural language processing by achieving state-of-the-art results on tasks from text generation to complex reasoning (brown2020language,; devlin2019bert,; chang2024survey,).
However, the sheer scale of these models, which often encompass billions of parameters, renders full-parameter fine-tuning prohibitively expensive in both computational and memory requirements, especially when adapting to multiple tasks simultaneously (han2024parameter,). As real-world applications increasingly demand multi-task capabilities, methods that reduce resource overhead while preserving performance have become critical. Parameter-efficient fine-tuning (PEFT) techniques (hu2022lora,; rebuffi2017adapters,; li2021prefix,; chang2024ba,), which add only lightweight adaptation modules to a frozen base model, offer a promising solution by slashing trainable parameters from hundreds of millions to mere thousands dramatically cutting GPU memory footprint and speeding up training without sacrificing modularity across tasks.

While PEFT approaches like LoRA (hu2022lora,) and its multi-task extensions (e.g., R-LoRA (liu2025rlora,), HydraLoRA (tian2024hydralora,)) deliver substantial efficiency gains in single-task scenarios, they still rely on large volumes of labeled data when scaled to many tasks resulting in poor data efficiency. For example, HydraLoRA reports that fine-tuning on just 43 tasks required over 320,000 in-domain examples to achieve satisfactory performance on downstream tasks (tian2024hydralora,). This heavy data demand undermines the very efficiency gains PEFT seeks to provide in multi-task settings and highlights the pressing need for methods that balance both parameter- and data-efficiency in large-scale multi-task adaptation.

Data‐efficient techniques like coreset selection (liu2023coreset1,; xia2024less,) or data pruning (azeemi2023dataprunig,) excel at trimming down data for a single task by homing in on the “most informative” examples, but in a multi‐task LLM scenario, this narrow focus comes at the expense of broader generalization across many objectives. By optimizing for task-specific highlights, these methods tend to under-represent the shared structures and cross-task patterns that are essential for robust performance on unseen or less frequent tasks (lin2024re,; chen2024molecular,).

Inspired by the principles of Meta-Learning, which emphasizes enabling models to “learn how to learn” (finn2017maml,), we propose a novel framework called MeTA-LoRA, specifically designed to address the data efficiency challenge in the fine-tuning process of LLMs within multi-task learning scenarios. To achieve this, we frame the fine-tuning process as an iterative optimization procedure consisting of two key stages: task-specific adaptation and meta-knowledge update. The task-specific adaptation stage allows for rapid per-task learning using only a small amount of data from the support set, enabling efficient adaptation to each task without requiring large datasets. Meanwhile, the meta-knowledge update stage aggregates insights from multiple tasks through a shared LoRA adapter, promoting the transfer of knowledge across tasks while minimizing data usage. Together, these two stages optimize the learning process, significantly enhancing data efficiency while preserving model performance. We demonstrate that MeTA-LoRA achieves competitive performance in both multi-task learning and multilingual learning scenarios while requiring significantly less task-specific data, showcasing its adaptability across a variety of tasks. Extensive ablation studies and analyses further demonstrate the necessity of the two-stage optimization framework.

In summary, the major contributions of this paper are outlined below.

- •
We propose MeTA-LoRA, a novel framework designed to enhance data efficiency for multi-task LoRA adaptation while preserving model performance.
- •
Comprehensive experimental evaluations validate the effectiveness of MeTA-LoRA in both multi-task learning and multilingual learning scenarios.
- •
Extensive ablation experiments and analyses demonstrate the necessity of the two-stage optimization framework.

## 2 Related Work

##### Parameter-Efficient Fine-tuning

As LLMs grow more powerful, fine-tuning them remains computationally intensive. This challenge has motivated the development of parameter-efficient fine-tuning (PEFT) methods, which aim to lower memory and storage demands during model adaptation. One representative approach is adapter tuning (rebuffi2017learning,; houlsby2019parameter,; sung2022vl,; stickland2019bert,), which introduces trainable layers into the existing model while keeping the original parameters frozen. Another line of PEFT research focuses on directly manipulating model activations through learnable vectors, with methods such as concatenation (liu2024gpt,; li2021prefix,; lester2021power,), multiplication, and addition.
Additionally, prompt-based tuning methods like prefix tuning (lester2021power,) and continuous prompt tuning (li2021prefix,; liu2021p,) replace discrete prompt engineering with trainable embeddings.
Beyond injecting new parameters, researchers have also explored sparse updates (sung2021training,; dey2024sparse,) and low-rank adaptation (LoRA) (hu2022lora,) as alternatives that modify only a small subset of the model’s parameters or its computational graph.

##### Lora Architecture on Multi-Task Learning

Multi-LoRA Architectures have emerged as a promising solution for adapting LLMs, such as LLaMA, in resource-constrained settings (touvron2023llama,). To further leverage the potential of LoRA, researchers have proposed multi-LoRA approaches that employ multiple low-rank adapters simultaneously. For instance, LoRAHub (huang2023lorahub,) trains multiple adapters and dynamically selects suitable combinations based on the domain at inference time, while MultiLoRA (wang2023multilora,) improves scalability by decomposing LoRA modules and introducing learnable scaling factors. To reduce resource usage, LoRAFit (zadouri2023pushing,) refines module structure, and LoRAMoE (dou2023loramoe,) incorporates a Mixture-of-Experts framework to protect pre-trained knowledge during instruction tuning.

##### Data-efficient strategies on LLM

Fine-tuning LLMs is computationally demanding, which has motivated the development of data-efficient strategies to reduce resource consumption without sacrificing performance (ding2023efficiency,; xu2024survey,). Among these, two prominent approaches are coreset selection (lin2024data,; xia2022moderate,; xia2024less,) and data pruning (marion2023less,; sorscher2022beyond,). These techniques highlight the promise of intelligent data selection for alleviating the computational burden of LLM fine-tuning, thereby improving accessibility in resource-constrained settings. Specifically, coreset selection focuses on identifying a representative subset of training data that can approximate the performance achieved with the full dataset (lin2024data,; xia2022moderate,; xia2024less,), while data pruning aims to eliminate less informative or redundant samples to streamline training (marion2023less,; sorscher2022beyond,).

## 3 MeTA-LoRA

In this section, we introduce MeTA-LoRA, a data-efficient LoRA-based architecture for fine-tuning, as illustrated in Figure [1](https://arxiv.org/html/2510.11598v1#S3.F1). The detailed fine-tuning procedure is provided in Algorithm [1](https://arxiv.org/html/2510.11598v1#alg1).

Figure: Figure 1: Architecture of MeTA-LoRA. During fine-tuning, MeTA-LoRA adopts a two-stage optimization framework comprising a task-specific adaptation stage and a meta-knowledge update stage. Starting from the initial shared LoRA parameters $\theta$, task-specific adapters are rapidly adapted using support set of each task, enabling efficient task-level specialization. In the second stage, the updated task-specific parameters ${\theta_{1},\theta_{2},\dots,\theta_{n}}$ are used to compute gradients from the corresponding query sets, which are then aggregated to update the shared LoRA adapter, enabling effective cross-task knowledge transfer. By iteratively alternating between these two stages, MeTA-LoRA enhances data efficiency in multi-task adaptation of LLMs.
Refer to caption: x1.png

### 3.1 Problem Formulation

Given a model $f$ with pre-trained weights $W_{0}$ and a set of tasks $\mathcal{T}=\{\mathcal{T}_{1},\mathcal{T}_{2},\dots,\mathcal{T}_{N}\}$, current LoRA-based methods for multi-task adaptation optimize:

$$ $\min_{\theta}\sum_{i=1}^{N}\mathbb{E}_{(x,y)\sim\mathcal{T}_{i}}[\mathcal{L}(f_{W_{0}+\Delta{\theta}}(x),y)]$ (1) $$

where $\theta=\{A,B\}$ denotes the shared LoRA adapter parameters across all $N$ tasks, and $\Delta{\theta}=BA$ approximates the accumulated gradient updates $\Delta{W_{0}}$. $\mathcal{T}_{i}$ denotes the data distribution of the $i$-th task. Different from the above paradigm, we propose a two-stage optimization framework that maintains multiple local LoRA adapters during the task-specific adaptation stage, alongside a global LoRA adapter updated in the meta-knowledge update stage, enabling the model to efficiently adapt to multiple tasks while minimizing the amount of data required for each task.

### 3.2 Fine-tuning

#### 3.2.1 Phase I: Task-Specific Adaptation

In each iteration, we randomly sample $n$ tasks $\mathcal{B}=\{\mathcal{T}_{1},\mathcal{T}_{2},\dots,\mathcal{T}_{n}\}$ from $\mathcal{T}$ with each task following the episodic formulation:

$$ $\mathcal{T}_{i}\coloneqq(\mathcal{S}_{i},\mathcal{Q}_{i})$ (2) $$

where $\mathcal{S}_{i}$ and $\mathcal{Q}_{i}$ are disjoint subsets randomly selected from task $\mathcal{T}_{i}$, traditionally referred to as the support set and the query set (finn2017maml,) with sizes $n_{i}^{s}$ and $n_{i}^{q}$, respectively.

For each task $\mathcal{T}_{i}\in\mathcal{B}$, the task‐specific adapter parameters $\theta_{i}$ are first initialized with the shared adapter parameters $\theta$ (i.e., $\theta_{i}\leftarrow\theta$). Subsequently, the model performs the gradient descent on $\theta_{i}$ for $k$ steps based on its corresponding support set, simulating the model’s rapid adaptation to a new task. One gradient update of task-specific (local) LoRA adapter parameters $\theta_{i}$ can be formulated as:

$$ $\theta_{i}\leftarrow\theta_{i}-\alpha\nabla_{\theta_{i}}\mathcal{L}_{\mathcal{S}_{i}}({\theta_{i}})$ (3) $$

where $\alpha$ is the learning rate of the adaptation stage, and the task-specific adaptation loss $\mathcal{L}_{\mathcal{S}_{i}}(\theta_{i})$ is:

$$ $\mathcal{L}_{\mathcal{S}_{i}}(\theta_{i})=\frac{1}{n_{i}^{s}}\sum_{(x,y)\sim\mathcal{S}_{i}}\mathcal{L}(f_{W_{0}+\Delta\theta_{i}}(x),y)$ (4) $$

Figure: Algorithm 1 MeTA-LoRA using the First-Order Approximation

#### 3.2.2 Phase II: Meta-Knowledge Update

After obtaining the adapted parameters $\theta_{i}$ for each task $\mathcal{T}_{i}$, the meta-update is performed on the shared (global) LoRA parameters $\theta$. To maintain computationally tractable with large models, we employ the first-order approximation of MAML, a common simplification that retains meta-learning signals while avoiding the prohibitive computational overhead of Hessian-vector products. In practice, this is implemented by detaching the computational graph of the task-specific adaptation, which prevents backpropagation through the task-specific update steps. The approximate meta-update rule, which averages the gradients from a batch of $n$ sampled tasks, is therefore formulated as:

$$ $\theta\leftarrow\theta-\beta\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta_{i}}\mathcal{L}_{\mathcal{Q}_{i}}(\theta_{i})$ (5) $$

where $\beta$ is the learning rate in the meta-stage, and the generalization loss $\mathcal{L}_{\mathcal{Q}_{i}}(\theta_{i})$ with respect to task $\mathcal{T}_{i}$ is computed on the corresponding query set using the adapted LoRA parameters $\theta_{i}$:

$$ $\mathcal{L}_{\mathcal{Q}_{i}}(\theta_{i})=\frac{1}{n_{i}^{q}}\sum_{(x,y)\sim\mathcal{Q}_{i}}\mathcal{L}(f_{W_{0}+\Delta\theta_{i}}(x),y)$ (6) $$

### 3.3 Inference

After fine-tuning, we obtain a single shared adapter that captures multi-task knowledge. It is worth emphasizing that the task-specific adapters are merely intermediate artifacts generated during the fine-tuning process. In the inference stage, the parameters of the shared adapter $\theta$ are seamlessly merged into the frozen pre-trained weights $W_{0}$, eliminating the need to activate or switch among task-specific adapters.

## 4 Experiments

In this section, we present a series of experiments to evaluate the effectiveness of MeTA-LoRA in both multi-task and multilingual learning scenarios. Additionally, we conduct ablation studies to examine the necessity of the proposed two-stage optimization framework, as well as the adaptability of MeTA-LoRA under more challenging multi-task configuration. Finally, we perform a sensitivity analysis to investigate the impact of the amount of fine-tuning data on overall performance.

### 4.1 Experiment Setting

#### 4.1.1 Multi-task Learning

In multi-task learning scenario, we consider two task configurations. First, we adopt the same experimental setting described in HydraLoRA (tian2024hydralora,). Additionally, we develop a five-task dataset comprising semantically and structurally diverse samples, to further evaluate the model’s capacity to handle varied tasks.

Flanv2 Setting

Dataset.
We fine-tune the model using a subset of the Flanv2 dataset (wei2021flan-v2,) that includes 43 tasks from both Natural Language Understanding (NLU) and Natural Language Generation (NLG), grouped into 10 distinct task clusters. The model’s performance is evaluated using the Big-Bench Hard (BBH) (suzgun2022bbh,) benchmark. More details of the dataset can be found in Appendix [A.1.1](https://arxiv.org/html/2510.11598v1#A1.SS1.SSS1).

Baselines.
We evaluate MeTA-LoRA against other LoRA-based methods designed for multi-task learning across multiple datasets: (1) Lorahub (huang2023lorahub,), which utilizes black-box optimization to learn weights for 20 randomly selected LoRAs for new tasks, applying weighted averaging without the need for gradient calculations; (2) LoRA MoE (liu2024moe,), which combines lightweight experts (LoRA) with a Mixture of Experts (MoE) architecture for high efficiency, enabling generalization to new tasks without prior knowledge; (3) HydraLoRA (tian2024hydralora,), which employs Multi-Head structure in conjunction with MoE to achieve a balance between parameter efficiency and training effectiveness.

Five-task Setting

Dataset.
We build a five-task dataset encompassing GSM8K (cobbe2021gsm8k,) for arithmetic reasoning, QQP (wang2017qqp,) and CosmosQA (huang2019cosmosqa,) for NLU, SiQA (sap2019socialiqa,) and PiQA (bisk2020piqa,) for commonsense reasoning. Then, we use the MMLU (Massive Multitask Language Understanding) (hendrycks2020mmlu,) benchmark to measure the world knowledge acquired during fine-tuning and problem solving ability of models. In addition, we include the BBH benchmark. Following common practice, we conduct 5-shot evaluation on MMLU and 3-shot evaluation on BBH.

Baselines.
Following prior studies (tian2024hydralora,; 2025-malora,), we adopt LLaMA2-7B and LLaMA2-13B as the backbone models. For comparison, we evaluate MeTA-LoRA against two representative baselines: (1) LoRA, a widely used approach for parameter-efficient fine-tuning; (2) HydraLoRA, a state-of-the-art method tailored for multi-task learning.

**Table 1: Performance of different methods on BBH (3-shot) with LLaMA2-7B and LLaMA2-13B fine-tuned on the subset of the Flanv2 dataset. MeTA-LoRA uses only 100 examples sampled from each task for fine-tuning, whereas the other methods except Base utilize the full dataset. * indicates results from tian2024hydralora.**
| Model Size | Base | LoRA | LoRAHub* | LoRA MoE* | HydraLoRA* | MeTA-LoRA |
| --- | --- | --- | --- | --- | --- | --- |
| 7B | 31.6 | 37.2 | 39.7 | 40.3 | 41.8 | 38.52 |
| 13B | 38.4 | 40.9 | 41.9 | 43.7 | 44.7 | 46.32 |

**Table 2: Performance comparison on the proposed five-task dataset. The base models are fine-tuned by MeTA-LoRA with only 50 examples randomly sampled from each task, and LoRA and HydraLoRA represent the full-data tuning schemes.**
| Method | MMLU | MMLU-math | BBH | AVG |
| --- | --- | --- | --- | --- |
| LLaMA2-7B |  |  |  |  |
| LoRA | 29.43 | 26.42 | 30.24 | 28.70 |
| HydraLoRA | 46.61 | 30.00 | 37.49 | 38.03 |
| MeTA-LoRA | 46.34 | 31.12 | 39.53 | 39.00 |
| LLaMA2-13B |  |  |  |  |
| LoRA | 39.99 | 26.08 | 44.84 | 36.97 |
| HydraLoRA | 54.24 | 30.88 | 44.67 | 43.26 |
| MeTA-LoRA | 54.18 | 30.97 | 46.12 | 43.76 |

#### 4.1.2 Multilingual Learning

In multilingual learning scenario, we fine-tune the LLaMA2-7B and LLaMA2-13B using Bactrian-X (li2023bactrian,), a comprehensive multilingual parallel dataset comprising 3.4 million instruction–response pairs across 52 languages. Bactrian-X is automatically constructed by translating instructions from Alpaca (taori2023alpaca,) and Dolly (conover2023dolly,) via the Google Translate API.2. In our evaluation, we compare MeTA-LoRA against two baselines: (1) the corresponding vanilla models; (2) the multilingual Bactrian models (BX), which are fine-tuned on the full Bactrian-X dataset. To probe the zero-shot language understanding capability of the different models and how much knowledge of they encode, we evaluate on the following four benchmarks:

- •
XCOPA (ponti2020xcopa,): a multilingual resource designed for causal commonsense reasoning, encompassing 11 languages from 11 families and several areas around the globe. The task requires selecting the correct subsequent sentence from two given options, based on cause and effect question types.
- •
XStoryCloze (lin2022XStoryCloze,): the professionally translated version of the English StoryCloze dataset (mostafazadeh2016corpus,) into 10 non-English languages. The task involves selecting one sentence as a plausible ending (closure) from two options, given a four-sentence story as the premise.
- •
XWinoGrad (XWinoGrad,; XWinoGrad_crosslingual,): a multilingual assessment dataset for commonsense reasoning, made up of Winograd Schema Challenge problems in six languages. The objective is to select the most plausible sentence from options that differ slightly.
- •
EXAMS (hardalov2020exams,): A multilingual multiple-choice question-answering dataset constructed from high school exam questions in 16 languages, covering a wide range of subjects including natural sciences (e.g., physics), social sciences (e.g., history), and humanities (e.g., philosophy).

#### 4.1.3 Hyper-parameter Settings

**Table 3: Statistics of different settings with respect to the number of tasks and the amount of samples used by the baselines and the proposed MeTA-LoRA.**
| Settings | Tasks | Baselines | Ours |
| --- | --- | --- | --- |
| Flanv2 | 43 | 325,783 | 4,300 |
| Five-task | 5 | 439,950 | 250 |
| Multilingual | 52 | 3,400,000 | 2,600 |
| Five-task variant | 5 | 487,075 | 250 |

For both multi-task learning and multilingual learning scenarios, we run MeTA-LoRA for 1,000 iterations. In each iteration, we randomly select $2$ tasks (i.e. $n=2$) for adaptation. For each task $i$, the size of the support set $n_{i}^{s}$ and the query set $n_{i}^{q}$ are both set to $8$. We perform $3$ steps of gradient descent on the support set to obtain task-specific LoRA adapters, using a learning rate of $5\times 10^{-6}$. During the meta-knowledge update phase, we employ AdamW optimizer with a learning rate of $2\times 10^{-6}$. A comprehensive comparison of the number of fine-tuning samples used by the baselines and our method under these settings is presented in Table [3](https://arxiv.org/html/2510.11598v1#S4.T3), and additional details regarding the hyper-parameter settings can be found in Appendix [A.1.2](https://arxiv.org/html/2510.11598v1#A1.SS1.SSS2).

### 4.2 Performance

#### 4.2.1 Performance on Multi-task Learning

Table [1](https://arxiv.org/html/2510.11598v1#S4.T1) and Table [2](https://arxiv.org/html/2510.11598v1#S4.T2) present the performance of various fine-tuned models evaluated on the different benchmarks, using a subset of the Flanv2 dataset and the proposed five-task dataset, respectively. We make several observations in more detail, and discuss them below.

- •
As shown in Table [1](https://arxiv.org/html/2510.11598v1#S4.T1), MeTA-LoRA consistently improves performance over both the base models and the models fine-tuned with standard LoRA. Notably, based on the LLaMA2-13B model, MeTA-LoRA achieves the highest BBH score of 46.32 among all LoRA-based baselines, despite involving only 100 examples per task into the fine-tuning process. The improvements highlight the effectiveness and data efficiency of the MeTA-LoRA framework.
- •
When a more diverse set of tasks is used for fine-tuning, as shown in Table [2](https://arxiv.org/html/2510.11598v1#S4.T2), MeTA-LoRA consistently outperforms both standard LoRA and HydraLoRA on LLaMA2-7B and LLaMA2-13B, even with only 50 examples per task. Overall, MeTA-LoRA surpasses HydraLoRA and LoRA by 0.97% and 10.3% on LLaMA2-7B, and by 0.50% and 6.79% on LLaMA2-13B, respectively, indicating that it can effectively capture task-specific knowledge and thereby enhances generalization across a broad range of tasks.

**Table 4: Averaged zero-shot accuracy for XCOPA, XStoryCloze, XWinograd, and EXAMS under different tuning schemes. MeTA-LoRA fine-tunes the base models with only 50 examples randomly sampled from each language, and BXLLaMA represents LoRA tuning on the full Bactrian-X dataset. * indicates results from li2023bactrian.**
| Model | XCOPA | XStoryCloze | XWinograd | EXAMS | AVG |
| --- | --- | --- | --- | --- | --- |
| LLaMA* (7B) | 50.22 | 57.03 | 57.96 | 28.20 | 48.35 |
| BXLLaMA* (7B) | 51.76 | 58.91 | 60.16 | 29.14 | 49.99 |
| MeTA-LoRA (7B) | 57.02 | 58.46 | 79.12 | 23.84 | 54.61 |
| LLaMA* (13B) | 51.04 | 57.88 | 52.97 | 30.41 | 48.08 |
| BXLLaMA* (13B) | 53.27 | 62.12 | 63.65 | 35.71 | 53.69 |
| MeTA-LoRA (13B) | 57.87 | 59.58 | 82.18 | 23.84 | 55.87 |

**Table 5: Performance comparison on the more challenging five-task dataset, where PiQA is replaced by WinoGrande. MeTA-LoRA fine-tunes LLaMA2-7B and LLaMA2-13B using 50 examples per task, while LoRA and HydraLoRA correspond to fine-tuning on the entire datatset.**
| Method | MMLU | MMLU-math | BBH | AVG |
| --- | --- | --- | --- | --- |
| LLaMA2-7B |  |  |  |  |
| LoRA | 39.03 | 29.16 | 32.01 | 33.40 |
| HydraLoRA | 47.19 | 30.98 | 36.34 | 38.17 |
| MeTA-LoRA | 46.50 | 32.15 | 39.07 | 39.24 |
| LLaMA2-13B |  |  |  |  |
| LoRA | 39.17 | 28.06 | 43.44 | 36.89 |
| HydraLoRA | 54.12 | 30.41 | 44.28 | 42.94 |
| MeTA-LoRA | 54.18 | 30.51 | 46.02 | 43.57 |

#### 4.2.2 Performance on Multilingual Learning

The average performance across all languages for XCOPA, XStoryCloze, XWinograd, and EXAMS is reported in Table [4](https://arxiv.org/html/2510.11598v1#S4.T4). Notably, even with only 50 randomly sampled examples per language, MeTA-LoRA achieves competitive or superior performance compared to the BX models, particularly on XCOPA and XWinograd. Moreover, it significantly outperforms the vanilla LLaMA2 models without multilingual adaptation on three out of four benchmarks. This substantial improvement in language understanding can be attributed to the MeTA-LoRA mechanism, which enables rapid adaptation across diverse languages using only a limited amount of data. By leveraging its two-stage optimization framework, MeTA-LoRA effectively captures both shared multilingual knowledge and language-specific patterns, thereby achieving strong data efficiency in multilingual adaptation.

Figure: (a) LLaMA2-7B, 50 examples per task.
Refer to caption: x2.png

### 4.3 Ablation Studies

RQ1: Do more challenging datasets better reveal the generation ability of models?

To further assess the effectiveness of MeTA-LoRA in more challenging multi-task learning scenarios, we construct a new five-task configuration by replacing PiQA with WinoGrande (sakaguchi2021winogrande,). Compared to PiQA, WinoGrande is a larger and more difficult commonsense reasoning dataset. It features cloze-style questions in which the model is required to select the correct option from two candidates to complete a given sentence.
We adopt LLaMA2-7B and LLaMA2-13B as base models. LoRA and HydraLoRA serve as the baselines by fine-tuning the base models on the full dataset, while MeTA-LoRA uses only 50 examples per task. The evaluation results on MMLU, MMLU-math, and BBH are summarized in Table [5](https://arxiv.org/html/2510.11598v1#S4.T5).

It can be observed that, compared to the standard five-task configuration (as shown in Table [2](https://arxiv.org/html/2510.11598v1#S4.T2)), fine-tuning LLaMA2-7B on the more challenging task sets using LoRA, HydraLoRA or MeTA-LoRA leads to a moderate improvement in the reasoning ability of the model. Specifically, LoRA, HydraLoRA and MeTA-LoRA achieve average performance gains of 4.70%, 0.14% and 0.24%, respectively. For LLaMA2-13B, although overall performance tends to saturate due to the strong inherent generalization of the model, MeTA-LoRA continues to substantially outperform LoRA and HydraLoRA across configurations, highlighting the robustness of MeTA-LoRA under more challenging task distributions and its scalability to larger models.

RQ2: Is the two-stage optimization framework necessary for the fine-tuning process?

To further understand the contribution of the task-specific adaptation stage in MeTA-LoRA, we conduct an ablation study, with the results presented in Figure [2](https://arxiv.org/html/2510.11598v1#S4.F2). Specifically, we investigate a variant, denoted as -STA, in which the task-specific adaptation stage is removed. In this variant, only the gradients computed from the query sets of multiple selected tasks are aggregated to update the shared LoRA adapters in each iteration.
Experiments are performed using two backbone models, LLaMA2-7B and LLaMA2-13B, under the proposed five-task learning configuration. For each task, either 50 or 100 examples are randomly selected for fine-tuning. In addition, both -STA and MeTA-LoRA are evaluated on three benchmarks: MMLU, the mathematics tasks of MMLU and BBH.

As shown in Figure [2](https://arxiv.org/html/2510.11598v1#S4.F2), the task-specific adaptation stage contributes positively to model performance across both model scales and data regimes, illustrating the necessity and effectiveness of the two-stage design. Notably, the performance gains introduced by the task-specific adaptation stage are more pronounced when using the smaller model (LLaMA2-7B) and the more limited amount of fine-tuning data (50 examples per task). In this case, MeTA-LoRA improves the average score by 1.07%, demonstrating its ability to capture more task-specific knowledge prior to meta-aggregation. Moreover, while the larger LLaMA2-13B model already exhibits strong generalization capabilities even with a simplified optimization process, the two-stage structure continues to deliver consistent performance benefits, confirming its robustness and scalability across model sizes.

**Table 6: Comparison of LoRA and MeTA-LoRA on the standard five-task learning scenario with varying amounts of fine-tuning data per task.**
| # Data / task | Method | MMLU | MMLU-math | BBH | AVG |
| --- | --- | --- | --- | --- | --- |
| LLaMA2-7B |  |  |  |  |  |
| 50 | LoRA | 46.60 | 32.60 | 31.06 | 36.75 |
|  | HydraLoRA | 46.21 | 29.42 | 37.30 | 37.64 |
|  | MeTA-LoRA | 46.34 | 31.12 | 39.53 | 39.00 |
| 100 | LoRA | 46.49 | 27.32 | 37.17 | 36.99 |
|  | HydraLoRA | 46.72 | 29.78 | 37.44 | 37.98 |
|  | MeTA-LoRA | 46.20 | 31.09 | 38.87 | 38.72 |
| LLaMA2-13B |  |  |  |  |  |
| 50 | LoRA | 54.17 | 30.47 | 45.89 | 43.51 |
|  | HydraLoRA | 52.82 | 31.22 | 34.80 | 39.61 |
|  | MeTA-LoRA | 54.18 | 30.97 | 46.12 | 43.76 |
| 100 | LoRA | 55.12 | 31.11 | 45.96 | 44.06 |
|  | HydraLoRA | 54.11 | 31.30 | 40.50 | 41.97 |
|  | MeTA-LoRA | 54.67 | 31.29 | 46.43 | 44.13 |

### 4.4 Parameter Analysis

RQ3: Does scaling the fine-tuning data always improve performance in multi-task learning?

As shown in Table [6](https://arxiv.org/html/2510.11598v1#S4.T6), we analyze how the scale of fine-tuning data impacts model performance under the standard five-task learning setting. Specifically, we fine-tune LLaMA2-7B and LLaMA2-13B using LoRA, HydraLoRA and MeTA-LoRA with either 50 or 100 examples per task. Subsequently, we evaluate the overall performance on MMLU, MMLU-math and BBH. Additional results on the more challenging variant of this setting are reported in Table [9](https://arxiv.org/html/2510.11598v1#A1.T9) of Appendix [A.3.2](https://arxiv.org/html/2510.11598v1#A1.SS3.SSS2).

Several insights can be drawn from the results. Overall, increasing the number of fine-tuning examples per task generally improves performance. Moreover, MeTA-LoRA consistently achieves higher average scores across model sizes and data regimes compared to both LoRA and HydraLoRA. For example, under the standard five-task setting with only 50 examples per task, MeTA-LoRA surpasses LoRA and HydraLoRA on LLaMA2-7B by 2.25% and 1.36% in average score, respectively. This advantage stems from its novel two-stage optimization framework, which enables strong gains in low-resource settings by extracting task-specific knowledge from only a few examples, while also maintaining stable improvements at larger data scales by mitigating task conflicts.

Interestingly, LoRA sometimes performs better with only 50 or 100 examples per task than with the full dataset. This counter-intuitive result arises from architectural limitations in multi-task learning. LoRA uses a single shared low-rank adapter that struggles to capture the heterogeneous requirements of diverse tasks. As the data scale grows, task-specific signals interfere with one another, leading to degraded performance. HydraLoRA alleviates this issue to some extent with its asymmetric design.

## 5 Conclusion

In this work, we introduce MeTA-LoRA, a simple yet effective two-stage optimization framework designed to enhance data efficiency in multi-task adaptation of LLMs. By explicitly decoupling task-specific adaptation and meta-knowledge aggregation, MeTA-LoRA is able to quickly adapt to individual tasks using only a few examples, while simultaneously promoting cross-task generalization through shared parameter updates. Comprehensive experiments across both multi-task and multilingual learning settings demonstrate MeTA-LoRA consistently matches or outperforms standard full-data LoRA fine-tuning, despite using significantly less task-specific supervision. These results highlight the potential of MeTA-LoRA as a practical and scalable solution for efficient fine-tuning in real-world low-resource and multi-task scenarios.

## References

- (1)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
Advances in neural information processing systems, 33:1877–1901, 2020.
- (2)
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language understanding.
In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019.
- (3)
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al.
A survey on evaluation of large language models.
ACM transactions on intelligent systems and technology, 15(3):1–45, 2024.
- (4)
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang.
Parameter-efficient fine-tuning for large models: A comprehensive survey.
arXiv preprint arXiv:2403.14608, 2024.
- (5)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
ICLR, 1(2):3, 2022.
- (6)
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.
Learning multiple visual domains with residual adapters.
Advances in neural information processing systems, 30, 2017.
- (7)
Xiang Lisa Li and Percy Liang.
Prefix-tuning: Optimizing continuous prompts for generation.
arXiv preprint arXiv:2101.00190, 2021.
- (8)
Yupeng Chang, Yi Chang, and Yuan Wu.
Ba-lora: Bias-alleviating low-rank adaptation to mitigate catastrophic inheritance in large language models.
arXiv preprint arXiv:2408.04556, 2024.
- (9)
Jinda Liu, Yi Chang, and Yuan Wu.
R-lora: Random initialization of multi-head lora for multi-task learning.
arXiv preprint arXiv:2502.15455, 2025.
- (10)
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu.
Hydralora: An asymmetric lora architecture for efficient fine-tuning.
Advances in Neural Information Processing Systems, 37:9565–9584, 2024.
- (11)
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He.
What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning.
arXiv preprint arXiv:2312.15685, 2023.
- (12)
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen.
Less: Selecting influential data for targeted instruction tuning.
arXiv preprint arXiv:2402.04333, 2024.
- (13)
Abdul Hameed Azeemi, Ihsan Qazi, and Agha Ali Raza.
Data pruning for efficient model pruning in neural machine translation.
In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 236–246, 2023.
- (14)
Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua.
Data-efficient fine-tuning for llm-based recommendation.
In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 365–374, 2024.
- (15)
Dingshuo Chen, Zhixun Li, Yuyan Ni, Guibin Zhang, Ding Wang, Qiang Liu, Shu Wu, Jeffrey Yu, and Liang Wang.
Beyond efficiency: Molecular data pruning for enhanced generalization.
Advances in Neural Information Processing Systems, 37:18036–18061, 2024.
- (16)
Chelsea Finn, Pieter Abbeel, and Sergey Levine.
Model-agnostic meta-learning for fast adaptation of deep networks.
In International conference on machine learning, pages 1126–1135. PMLR, 2017.
- (17)
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.
Learning multiple visual domains with residual adapters.
Advances in neural information processing systems, 30, 2017.
- (18)
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.
Parameter-efficient transfer learning for nlp.
In International conference on machine learning, pages 2790–2799. PMLR, 2019.
- (19)
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.
Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5227–5237, 2022.
- (20)
Asa Cooper Stickland and Iain Murray.
Bert and pals: Projected attention layers for efficient adaptation in multi-task learning.
In International Conference on Machine Learning, pages 5986–5995. PMLR, 2019.
- (21)
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.
Gpt understands, too.
AI Open, 5:208–215, 2024.
- (22)
Brian Lester, Rami Al-Rfou, and Noah Constant.
The power of scale for parameter-efficient prompt tuning.
arXiv preprint arXiv:2104.08691, 2021.
- (23)
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang.
P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.
arXiv preprint arXiv:2110.07602, 2021.
- (24)
Yi-Lin Sung, Varun Nair, and Colin A Raffel.
Training neural networks with fixed sparse masks.
Advances in Neural Information Processing Systems, 34:24193–24205, 2021.
- (25)
Nolan Dey, Shane Bergsma, and Joel Hestness.
Sparse maximal update parameterization: A holistic approach to sparse training dynamics.
arXiv preprint arXiv:2405.15743, 2024.
- (26)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023.
- (27)
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin.
Lorahub: Efficient cross-task generalization via dynamic lora composition.
arXiv preprint arXiv:2307.13269, 2023.
- (28)
Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang.
Multilora: Democratizing lora for better multi-task learning.
arXiv preprint arXiv:2311.11501, 2023.
- (29)
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker.
Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning.
arXiv preprint arXiv:2309.05444, 2023.
- (30)
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al.
Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment.
arXiv preprint arXiv:2312.09979, 4(7), 2023.
- (31)
Tianyu Ding, Tianyi Chen, Haidong Zhu, Jiachen Jiang, Yiqi Zhong, Jinxin Zhou, Guangzhi Wang, Zhihui Zhu, Ilya Zharkov, and Luming Liang.
The efficiency spectrum of large language models: An algorithmic survey.
arXiv preprint arXiv:2312.00678, 2023.
- (32)
Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al.
A survey of resource-efficient llm and multimodal foundation models.
arXiv preprint arXiv:2401.08092, 2024.
- (33)
Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua.
Data-efficient fine-tuning for llm-based recommendation.
In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 365–374, 2024.
- (34)
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu.
Moderate coreset: A universal method of data selection for real-world data-efficient deep learning.
In The Eleventh International Conference on Learning Representations, 2022.
- (35)
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker.
When less is more: Investigating data pruning for pretraining llms at scale.
arXiv preprint arXiv:2309.04564, 2023.
- (36)
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos.
Beyond neural scaling laws: beating power law scaling via data pruning.
Advances in Neural Information Processing Systems, 35:19523–19536, 2022.
- (37)
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.
Finetuned language models are zero-shot learners.
arXiv preprint arXiv:2109.01652, 2021.
- (38)
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al.
Challenging big-bench tasks and whether chain-of-thought can solve them.
arXiv preprint arXiv:2210.09261, 2022.
- (39)
Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng.
When moe meets llms: Parameter efficient fine-tuning for multi-task medical applications.
In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1104–1114, 2024.
- (40)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
Training verifiers to solve math word problems.
arXiv preprint arXiv:2110.14168, 2021.
- (41)
Zhiguo Wang, Wael Hamza, and Radu Florian.
Bilateral multi-perspective matching for natural language sentences.
arXiv preprint arXiv:1702.03814, 2017.
- (42)
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.
Cosmos qa: Machine reading comprehension with contextual commonsense reasoning.
arXiv preprint arXiv:1909.00277, 2019.
- (43)
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi.
Socialiqa: Commonsense reasoning about social interactions.
arXiv preprint arXiv:1904.09728, 2019.
- (44)
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.
Piqa: Reasoning about physical commonsense in natural language.
In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020.
- (45)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
arXiv preprint arXiv:2009.03300, 2020.
- (46)
Xujia Wang, Haiyan Zhao, Shuo Wang, Hanqing Wang, and Zhiyuan Liu.
MALoRA: Mixture of asymmetric low-rank adaptation for enhanced multi-task learning.
In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025, pages 5609–5626, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
- (47)
Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin.
Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation.
arXiv preprint arXiv:2305.15011, 2023.
- (48)
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.
Stanford alpaca: An instruction-following llama model, 2023.
- (49)
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin.
Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023.
- (50)
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen.
Xcopa: A multilingual dataset for causal commonsense reasoning.
arXiv preprint arXiv:2005.00333, 2020.
- (51)
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al.
Few-shot learning with multilingual generative language models.
In Proceedings of the 2022 conference on empirical methods in natural language processing, pages 9019–9052, 2022.
- (52)
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen.
A corpus and cloze evaluation for deeper understanding of commonsense stories.
In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, 2016.
- (53)
Alexey Tikhonov and Max Ryabinin.
It’s all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning.
arXiv preprint arXiv:2106.12066, 2021.
- (54)
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al.
Crosslingual generalization through multitask finetuning.
arXiv preprint arXiv:2211.01786, 2022.
- (55)
Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov.
Exams: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering.
arXiv preprint arXiv:2011.03080, 2020.
- (56)
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.
Winogrande: An adversarial winograd schema challenge at scale.
Communications of the ACM, 64(9):99–106, 2021.

## Appendix A Appendix

### A.1 Datasets and Hyper-parameters

#### A.1.1 Details for Flanv2 Setting

Following [10], we select a portion of the Flanv2 datasets covering Natural Language Understanding (NLU) and Natural Language Generation (NLG), which can be grouped into 10 distinct task clusters. Then we evaluate it with the Big-Bench Hard (BBH) benchmark.

We summarize the details of the used datasets as follows:

- 1.
Struct-to-Text Conversion: This task evaluates the capability to generate natural language descriptions from structured data inputs. We use the following datasets: (1) CommonGen; (2) DART; (3) E2ENLG; (4) WebNLG
- 2.
Translation: Translation involves converting text from one language to another, maintaining the original meaning and nuances. We use the following datasets: (1) En-Fr from WMT’14; (2) En-De, En-Tr, En-Ru, En-Fi, En-Ro from WMT’16; (3) En-Es from Paracrawl.
- 3.
Commonsense Reasoning: This involves assessing the ability to apply physical or scientific principles alongside common sense in reasoning tasks. We use the following datasets: (1) COPA; (2) HellaSwag; (3) PiQA; (4) StoryCloze.
- 4.
Sentiment Analysis: A fundamental task in natural language processing (NLP) that determines the sentiment polarity (positive or negative) of a given text. We use the following datasets: (1) IMDB; (2) Sentiment140; (3) SST-2; (4) Yelp.
- 5.
Paraphrase Detection: This task requires models to ascertain whether two sentences convey the same meaning, indicating semantic equivalence. We use the following datasets: (1) MRPC; (2) QQP; (3) Paws Wiki.
- 6.
Coreference Resolution: Involves identifying instances within a text that refer to the same entity, demonstrating an understanding of textual context. We use the following datasets: (1) DPR; (2) WSC273.
- 7.
Reading Comprehension: Assesses the capability to derive answers to questions from a provided text containing relevant information. We use the following datasets: (1) BoolQ; (2) DROP; (3) MultiRC; (4) OBQA; (5) SQuADv1; (6) SQuADv2.
- 8.
Reading Comprehension with Commonsense: Merges traditional reading comprehension skills with commonsense reasoning, requiring understanding beyond the explicit text. We use the following datasets: (1) CosmosQA; (2) ReCoRD.
- 9.
Natural Language Inference: Focuses on deducing the relationship between two sentences, determining if the second sentence logically follows from, contradicts, or is unrelated to the first sentence. We use the following datasets: (1) ANLI; (2) CB; (3) MNLI; (4) QNLI; (5) SNLI; (6) WNLI; (7) RTE.
- 10.
Closed-Book Question Answering: This task challenges models to answer questions about general knowledge without direct access to external information sources. We use the following datasets: (1) ARC; (2) NQ; (3) TriviaQA.

#### A.1.2 Hyper-parameter Settings

Details on hyperparameters used for MeTA-LoRA, LoRA and HydraLoRA are provided below.

MeTA-LoRA: For all experiments, we integrate adapter modules into every dense layer of the multi-head attention (namely $Q$, $K$, $V$, and $O$) in the selected LLMs. Also, we set the low-rank parameter $r$ to $16$.

LoRA: In all experiments, adapter modules are inserted into every dense layer of the multi-head attention components (namely $Q$, $K$, $V$, and $O$) in the selected LLMs, with the rank $r$ set to $16$. In addition, the learning rate is set to $2\times 10^{-4}$ and the batch size is set to $8$. For experiments on the full datasets, results are reported after a single epoch of fine-tuning. For parameter analysis with respect to the amount of fine-tuning data, LoRA fine-tunes the base models for 5 epochs when using 50 examples per task, and for 10 epochs when using 100 examples per task. Finally, we report the best results obtained on the benchmarks.

HydraLoRA: For experiments not covered in the original paper [10], we adopt the default configurations suggested by HydraLoRA and adjust the number of $B$ matrices (lora_nums) to match the number of tasks. Consistent with LoRA, full-dataset fine-tuning is performed for a single epoch. Under limited-data settings, the base models are fine-tuned for 5 epochs with 50 examples per task and for 10 epochs with 100 examples per task. Finally, we report the best results obtained in each setting.

### A.2 Training Efficiency

To further assess the efficiency of the proposed method, we measure the running time of different fine-tuning schemes under identical experimental environments. As presented in Table [7](https://arxiv.org/html/2510.11598v1#A1.T7), MeTA-LoRA substantially reduces training time compared to standard LoRA and advanced HydraLoRA, while maintaining competitive performance.

**Table 7: Comparison of running time across fine-tuning schemes. LoRA and HydraLoRA fine-tune the base models on the more challenging dataset for one epoch, while MeTA-LoRA performs 1,000 iterations using 100 samples from each task.**
| Model Size | LoRA | HydraLoRA | MeTA-LoRA |
| --- | --- | --- | --- |
| 7B | 17h | 37h | 3h |
| 13B | 29h | 57h | 5h |

### A.3 Parameter Analysis

**Table 8: Performance comparison on the standard five-task setting with respect to $\alpha$ and $\beta$.**
| Combination | $\alpha$ | $\beta$ | BBH |
| --- | --- | --- | --- |
| $c_{base}$(ours) | $5\times 10^{-6}$ | $2\times 10^{-6}$ | 39.53 |
| $c_{1}$ | $2\times 10^{-6}$ | $5\times 10^{-6}$ | 39.39 |
| $c_{2}$ | $5\times 10^{-5}$ | $2\times 10^{-5}$ | 37.90 |
| $c_{3}$ | $5\times 10^{-7}$ | $2\times 10^{-7}$ | 38.96 |

#### A.3.1 Impact of learning rates

To investigate the impact of learning rates, we fine-tuned LLaMA2-7B under the standard five-task learning setting using various $(\alpha,\beta)$ configurations. As presented in Table [8](https://arxiv.org/html/2510.11598v1#A1.T8), $c_{2}$ with higher learning rates and $c_{3}$ with lower learning rates both degrade overall performance, suggesting optimization instability and insufficient adaptation.

**Table 9: Comparative analysis of LoRA, HydraLoRA and META-LORA on the more challenging five-task learning scenario with varying amounts of fine-tuning data per task.**
| # Data / task | Method | MMLU | MMLU-math | BBH | AVG |
| --- | --- | --- | --- | --- | --- |
| LLaMA2-7B |  |  |  |  |  |
| 50 | LoRA | 47.04 | 31.32 | 34.90 | 37.75 |
|  | HydraLoRA | 46.30 | 30.17 | 36.64 | 37.70 |
|  | MeTA-LoRA | 46.50 | 32.15 | 39.07 | 39.24 |
| 100 | LoRA | 45.68 | 25.94 | 33.75 | 35.12 |
|  | HydraLoRA | 46.44 | 30.51 | 36.75 | 37.90 |
|  | MeTA-LoRA | 46.55 | 31.59 | 39.45 | 39.20 |
| LLaMA2-13B |  |  |  |  |  |
| 50 | LoRA | 54.14 | 31.69 | 46.18 | 44.00 |
|  | HydraLoRA | 53.18 | 31.83 | 36.25 | 40.42 |
|  | MeTA-LoRA | 54.18 | 30.51 | 46.02 | 43.57 |
| 100 | LoRA | 54.36 | 29.22 | 46.17 | 43.25 |
|  | HydraLoRA | 53.43 | 30.90 | 38.35 | 40.89 |
|  | MeTA-LoRA | 54.23 | 31.21 | 46.78 | 44.07 |

#### A.3.2 Impact of the fine-tuning data

As shown in Table [9](https://arxiv.org/html/2510.11598v1#A1.T9), we also analyze how the scale of fine-tuning data impacts model performance under the more challenging variant of the standard five-task setting. Specifically, we fine-tune LLaMA2-7B and LLaMA2-13B using LoRA, HydraLoRA and META-LORA with either 50 or 100 examples per task. Subsequently, we evaluate the overall performance on MMLU, MMLU-math, and BBH.

Consistent with the findings under the standard five-task setting, LoRA achieves better performance with limited data than with full-data, reflecting its inherent architectural limitations. When the difficulty of tasks increases, HydraLoRA struggles to capture task-specific knowledge in such low-resource regimes. In contrast, our method built upon a two-stage optimization framework can rapidly adapt in diverse multi-task scenarios, which further demonstrates its robustness and establishes its superiority over existing approaches.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 4
presentation: 2
questions: 1. On MoDULA: How does your approach differ fundamentally from MoDULA? Please provide a detailed comparison.
2. On meta-learning necessity: Why is the episodic meta-learning formulation necessary? Have you tried simpler alternatives like just training task-specific adapters independently and then learning to combine them?
3. On the approximation: What is the performance gap between your first-order approximation and full second-order MAML? This ablation is critical for understanding the trade-offs.
soundness: 2
strengths: 1. Clear motivation: The paper identifies a real problem that existing LoRA-based multi-task methods require substantial task-specific data, which limits their practical applicability.
2. Comprehensive experiments: The evaluation covers diverse task domains.
3. Data efficiency: The claimed reduction in data usage (using only 100 examples per task vs. full datasets) while maintaining competitive performance is noteworthy.
4. Well-structured presentation: The paper is generally well-written with clear methodology description and appropriate use of figures.
summary: This paper proposes META-LORA, a two-stage optimization framework for parameter-efficient multi-task fine-tuning of LLMs. The method consists of: (1) a task-specific adaptation stage that learns individual LoRA adapters using small amounts of data from each task, and (2) a meta-knowledge update stage that aggregates gradients across tasks to update a shared LoRA adapter. Experiments on multi-task and multilingual benchmarks demonstrate competitive performance with significantly reduced data requirements compared to traditional fine-tuning approaches.
weaknesses: 1. The "first-order approximation of MAML" (line 161) is mentioned but not adequately justified. How does this approximation affect performance compared to full MAML?
2. The episodic formulation with support/query sets (Eq. 2) is borrowed from meta-learning but the paper doesn't clearly explain why this is necessary for the multi-task LoRA scenario
3. Lack of ablation on key hyperparameters: k (adaptation steps), n (tasks per iteration), support/query set sizes
4. Missing crucial related work: The paper fails to cite and discuss "MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning", which presents a highly similar three-stage training paradigm combining universal and domain-specific experts. Beyond MoDULA, several other relevant works on multi-task PEFT and expert mixtures are not discussed:
(1) MoLoRA (Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning, ICLR2024)
(2) C-Poly (Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning, ICLR2024)

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. For Table 7, please report tokens processed, optimizer steps, and approximate FLOPs for each method to ensure apples-to-apples timing.
2. What drives the drop on EXAMS despite gains elsewhere? Provide per-language breakdowns and failure analyses.
soundness: 3
strengths: 1. The proposed algorithm is generally simple and implementable .
2. Data-efficiency with 50–100 examples per task/language while matching or exceeding full-data LoRA/HydraLoRA in several cases.
3. Ablations and scaling: -STA vs full META-LORA, and 50 vs 100 examples show Stage-I helps, and trends are consistent; LR sensitivity provided. 
4. Multilingual gains on XCOPA/XWinograd with tiny per-language budgets.
summary: The paper proposes META-LORA, a two-stage, data-efficient PEFT framework for multi-task (and multilingual) LLM fine-tuning. Stage I performs brief task-specific adaptation of a per-task LoRA adapter on a small support set; Stage II performs a meta-knowledge update by averaging gradients on query sets (a first-order MAML approximation) to update a single shared LoRA adapter that is used at inference. For the empirical experiments, the paper evaluates (i) a Flanv2-subset with BBH as the eval, (ii) a five-task mixture (GSM8K, QQP, CosmosQA, SiQA, PiQA/WinoGrande) with MMLU / MMLU-math / BBH, and (iii) multilingual fine-tuning on Bactrian-X (52 languages) with XCOPA, XStoryCloze, XWinograd, and EXAMS. Empirical results show that META-LORA generally surpasses LoRA and is competitive with (often better than) HydraLoRA despite using far less data (e.g., BBH wins for 13B on Flanv2; solid average gains on the five-task settings).
weaknesses: 1. The core comparisons concentrate on LoRA/HydraLoRA (plus LoRAHub/LoRA-MoE for Flanv2); stronger PEFT baselines (for example, MALoRA/R-LoRA, prompt/prefix/adapters with modern optimizers) are not included, limiting generality claims.
2. No CIs / multi-seed means on BBH/MMLU/etc.; several margins are small and may fall within variance.  
3. The runtime table (Meta-LORA 1000 iterations vs others’ epochs) lacks tokens/steps/FLOPs parity. Therefore the  wall-clock claims are difficult to interpret.  
4. Despite the fact that XCOPA/XWinograd improve, EXAMS drops markedly (e.g., 23.84 vs BX-LLaMA’s ~29–36), which deserves analysis.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1.	The paper claims that META-LoRA can alleviate gradient conflicts across multiple tasks. Could the authors provide quantitative evidence or analysis to support this claim?

2.	In the ablation study, the –STA variant appears conceptually similar to standard LoRA. Could the authors clarify whether they are effectively equivalent? Additionally, how exactly does META-LoRA’s two-stage paradigm aggregate gradients from different tasks during meta-updates?

3.	The paper describes HydraLoRA as the state-of-the-art (SoTA) approach in multi-task learning. This statement seems not entirely accurate, and the authors may need to provide justification or rephrase it more cautiously.

4.	Under the same amount of training data, how does META-LoRA’s two-stage iterative paradigm compare with standard fine-tuning methods (e.g., training for more epochs on the same small dataset)? A direct comparison could better highlight whether the improvement comes from the meta-learning mechanism itself or from additional training iterations.
soundness: 2
strengths: The method is simple yet effective, cleverly introducing the meta-learning (MAML) concept into the LoRA framework.
It achieves exceptionally high data efficiency, requiring only a few dozen samples per task to surpass the performance of both LoRA and HydraLoRA
summary: This paper proposes META-LoRA, a parameter-efficient fine-tuning method inspired by meta-learning.Through a two-stage optimization process—task-specific adaptation and meta-knowledge aggregation—it enables a single shared LoRA module to achieve efficient generalization across multi-task and multilingual scenarios.
weaknesses: 	The paper lacks intuitive motivation and theoretical insight — it is not sufficiently clear why the proposed two-stage optimization is designed this way or why it should work from a meta-learning perspective.

	The proposed method mainly focuses on improving data efficiency, whereas prior works like HydraLoRA and LoRAHub innovate at the architectural level. The paper would benefit from comparisons with other data-efficient fine-tuning methods beyond the LoRA family.

	All tasks are included in training, but there is no evaluation on unseen tasks to verify the model’s ability to rapidly adapt — which is one of the central promises of the meta-learning framework.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. If the final single shared adapter encounters task conflicts  in multi-task settings, could this lead to compromises between tasks? For example, if one task requires formal writing while another demands informal writing, the model might learn an average strategy. This could result in suboptimal performance on both tasks compared to methods (like HydraLoRA) that preserve specific parameters for different tasks. This conflict aspect requires discussion.


2. The meta-knowledge update relies on the average gradient across n tasks. If the total number of tasks N is large while n remains fixed at 2, model convergence to capture knowledge across all N tasks may become extremely slow or unstable. The paper does not explore performance when N increases.

3. The results of the ablation studies, shown in Fig. 2 , reveal that the task-specific adaptation (STA) stage  provides little to no improvement in model performance when the model is large (13B) or the dataset size is moderately large (100 examples), and even shows a decline in some cases, raising doubts about its effectiveness. Furthermore, even with 7B parameters and 50 examples, improvements are only observed for some datasets.
soundness: 2
strengths: 1. This work addresses the “data efficiency” challenge in fine-tuning large models for multi-task scenarios. This represents a widespread and critical challenge in real-world applications, where acquiring large volumes of high-quality labeled data is often prohibitively expensive.


2. The integration of meta-learning's “learning how to learn” philosophy with the LoRA framework constitutes a novel and intelligent approach. Decoupling the fine-tuning process into two distinct phases: task adaptation and meta-knowledge updating, is theoretically coherent and logically sound, offering fresh perspectives for addressing knowledge transfer challenges in multi-task learning.

3. META-LORA ultimately produces a single, shared LoRA adapter. This enables highly efficient model deployment and inference without requiring switching or combining multiple adapters for different tasks.
summary: This paper proposes a novel framework named META-LORA, designed to enhance the data efficiency of large language models during LoRA fine-tuning in multi-task scenarios through a two-stage meta-learning approach. The method first rapidly acquires task knowledge on a small number of samples through a Task-Specific Adaptation phase, followed by a Meta-Knowledge Update phase that aggregates gradients across multiple tasks to promote knowledge generalization. Experimental results demonstrate that META-LORA achieves performance comparable to or surpassing baseline methods fine-tuned on full datasets across multi-task and multilingual benchmarks, using only a minimal number of unique samples.

While the problem addressed is highly significant and the proposed approach innovative, its core conclusions rest on experimental comparisons that warrant further scrutiny. Furthermore, the method's complexity and robustness are not sufficiently validated through ablation experiments, leaving its advantages and applicability unclear.
weaknesses: 1. Parameters such as the learning rate (α) in the Task-Specific Adaptation phase , the learning rate (β) in the Meta-Knowledge Update phase , the adaptation steps (k) , the number of sampled tasks (n) , and the support/query set sizes (n_s, n_q)  are overly complex. Furthermore, the paper lacks sufficient ablation experiments  to justify these choices. Particularly concerning are the adaptation steps (k) and the number of sampled tasks (n).

2. For experiments using datasets of identical size in Tables 6 and 9 , META-LORA trained for 1,000 iterations  with 16 data points per batch (support set + query set), while the baseline trained for 5 or 10 epochs. This results in a significant disparity in both the number of backpropagation passes and the total training steps. This suggests a roughly 30-fold difference in gradient updates. Wouldn't this cause the baseline to underfit? The experiments here do not support the claim that “META-LORA achieves better results than the baseline on the same small dataset.”


3. The authors' contribution states “a novel framework designed to enhance data efficiency for multi-task LoRA adaptation ,” yet the baseline does not incorporate data-efficient strategies mentioned in the related work, such as coreset selection or data pruning.


Decision: Reject