MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models
Authors: Jingwei Xu
Junyu Lai
Yunpeng Huang
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
LoRA
Primary area: foundation or frontier models, including LLMs
TL;DR: A scalable and efficient LoRA embedding framework for LLMs, enhanced with novel MoE forward acceleration strategies that significantly boost inference speed.

Abstract: The pretrain+fine-tune paradigm is foundational for deploying large language models (LLMs) across various downstream applications. Within this framework, Low-Rank Adaptation (LoRA) stands out for its parameter-efficient fine-tuning (PEFT), producing numerous reusable task-specific LoRA adapters. However, this approach requires explicit task intention selection, posing challenges for autonomous task sensing and switching during inference with multiple existing LoRA adapters embedded in a single LLM. In this work, we introduce MeteoRA (Multiple-Tasks embedded LoRA), a scalable and efficient framework that reuses multiple task-specific LoRA adapters into the base LLM via a full-mode Mixture-of-Experts (MoE) architecture. This framework also includes novel MoE forward acceleration strategies to address the efficiency challenges of traditional MoE implementations. Our evaluation, using the LlaMA2-13B and LlaMA3-8B base models equipped with 28 existing LoRA adapters through MeteoRA, demonstrates equivalent performance with the traditional PEFT method. Moreover, the LLM equipped with MeteoRA achieves superior performance in handling composite tasks, effectively solving ten sequential problems in a single inference pass, thereby demonstrating the framework's enhanced capability for timely adapter switching.

Published as a conference paper at ICLR 2025
METEORA: MULTIPLE-TASKS EMBEDDED LORA FOR LARGE LANGUAGE MODELS
Jingwei Xul Junyu Lai'
Yumpeng Huang'
State Key Laboratory for Novel Software Technology. Nanjing University, China singweixxs www.. edu.en, {junyu. lai, hyp smail i.j... edu. cn
ABSTRACT
The pretrain*fine-tuee paradigm is foundational for deploying large language models (LLMs) across various downstream applications. Within this framework, Low-Rank Adaptation (LoRA) stands out for its parameter-efficient fine-tuning (PEFT), producing numerous reusable task-specific LoRA adapters. However, this approach requires explicit task intention selection, posing challenges for au- tonomous task sensing and switching during inference with multiple existing LoRA adapters embedded in single LLM. In this work, we introduce MeteoRA (Multiple-tasks embedded LoRA), scalable and efficient framework that reuses multiple task-specific LoRA adapters into the base LLM via a full-mode Mixture- of of-Exxssss (MoE) architecture. This framework also includes novel MoE forward acceleration strategies to address the efficiency challenges of traditional MoE implementations. Our evaluation, using the LlaMA2-14 13B and LlaMA3-8B base models equipped with 28 existing LoRA adapters through MeteoRA, demonstrates equivalent performance with the traditional PEFT method. Moreover, the LLM equipped with MeteoRA achieves superior performance in handling composite tasks, effectively solving ten sequential problems in a single inference pass, thereby demonstrating the framework's 's enhanced capability for timely adapter switching.
INTRODUCTION
Large language models (LLMs) have achieved significant advancement in modern intelligent appli- cations, excelling in tasks from language comprehension generation within the field of natural language processing (NLP) (Achiam et al. 2023 Touvron et al. (023). By applying the fine-tuning process to pretrained LLMs these models have demonstrated remarkable efficacy in handling domain- specific tasks. Examples include converting natural language text into SQL queries (Katsogiannis Meimarakis & Koutrika [023] Pourreza & Ratici (024)), utilizing LLMs as agents in diverse
interactive applications (Song et al. | Chen et al. 2023 Gupta & Kembhavi 2023). and develop- ing models tailored for specific domains, such as BloombergGPPT (Wu et al. 20236 for financial analysis and ChatLaw (Cui et al. 2023 for legal consulting.
This pretrain-fine-tue paradigm has catalyzed the development several parameter-efficient fine- tuning (PEFT) methods. Low-Rank Adaptation (LoRA) (Hu et al. 202T) stands out as noteworthy exemplar of PEFT, offering efficient fine-tuning by updating only the low-rank matrices while keeping the rest of base LLM's 's parameters unchanged. Once fine-tuned, these matrices, which consist of minimal number of parameters, are encapsulated as a LoRA adapter that can be readily deployed integrated with the base L.MM for enhanced functionality. To improve the capability of handling multiple tasks simultaneously, the scalability of deploying these fine-tuned LoRA adapters has been explored. Solutions such Huggingface PEFT (Mangruikar et al., 2022), S-LoRA (Sheng al.) 2023), and other variants have been developed to facilitate the simultaneous serving of numerous LoRA adapters on single base LLM, enhancing the model's 's adaptability and efficiency in diverse application environments.
Despite the success of LoRA in the pretrain-fine-ture paradigm, several challenges remain. When reusing existing LoRA adapters, primary challenge is the ability of multi-LoR, embedded LLMs to
"Corresooniigg author

---

Published as a conference paper at ICLR 2025
N
Besael Q Proj
Fowal: Proj
M
Proj
Linear Module
Proj
Attention Module
MLP Module
LoRA, LoRA,
Decoder Block
MeteoRA Module
Softmax
Module
Figure 1: Our proposed framework provides full-mode MoE architecture that directly reuses various off-the-shelf LoRA adapters, enhancing the LLM's ability to timely and autonomously activate appropriate adapters for the input. MeteoRA modules could be integrated into all basic linear layers of both Attention and MLP modules. With the MoE forward acceleration strategies, LLM equipped with MeteoRA could be capable of addressing tasks across a wide range of domains effectively.
autonomously and on-demand LoRA selection during inference, a process that should allow LLM to handle different tasks by activating the appropriate LoRA adapters without explicit user instructions. Furthermore, managing composite tasks that require timely switching between LoRA adapters presents difficulties, especially when these tasks involve multiple sub-problems each requiring specific adapter activation. Current approaches such as Huggingface PEFT and S-LoRA, while capable of serving multiple existing adapters simultancously. mainly focus on loading rather than autonomously activating adapters, thus requiring manual intervention. Similarly, current LoRA fusion methods such as LoRAHub (Huang al. 2023) and MoA (Feng et al., (024), although they integrate and merge knowledge from various adapters, are not suitably designed to fuse a wide range of existing LoRA adapters with such a limited MoE framework, and lack the evidence in effectively managing dynamic adapter switching during inference for composite tasks.
In this paper, we introduce novel multi-tasks embedded LoRA framework for LLMs to reuse existing LoRAs with the ability of autonomous task sensing and switching. The framework proposes MoE-style module called MeteoRA. Each MeteoRA module provides trainable Gating network with MoE forward acceleration strategies (overcome the efficiency issue in naive MoE, especially when number of experts much larger than 8) for all LoRAs' low-rank matrices in the linear layer. As shown in Figure [ the MeteoRA module is applicable for all kinds of layers in Transformer-bant based LLMs (Q, K, V, and O attention module and up-proj, gating for SiLU (Elfwing et al. 2018). and down-proj in MLP Module). Through fine-tuning all gates with minimal resources. MeteoRA effectively integrates the existing LoRA adapters into the base LLM model with the ability of system
autonomously on-demand LoRA selection, without the requirements of any explicit instructions. Furthermore, the presence of numerous gates" enhances the model with a full-mode MoE architecture, showing the capability timely LoRA switching, addressing composite tasks with only two-shot examples as illustrations for all inputs. Our empirical evaluations, which embedded 28 existing LoRA A adapters with MeteoRA to LlaMA2-13B-base and LlaMA3-8B--bsse, highlight the full-mode MoE capabilities and demonstrate a significant performance maintenance (e.g., MeteoRA based on LlaMA3-8B achieves only 0.4% accuracy loss when solving multiple-choice tasks.). This improvement is particularly notable in handling composite tasks, showcasing the efficacy of the MeteoRA framework. The primary contributions of MeteoRA are summarized as follows:
Scalable LoRA integration: MeteoRA framework for reusing existing LoRA adapters advances the LLM's capability of autonomous on-demand LoRA selection and switching.
MoE forward acceleration:
revealing efficiency issue of MoE and providing the forward acceleration strategies with new GPU kernel operators to achieve a 4x speedup in average while maintaining memory overhead.
For the LlaMA3-8B model, there are 224 MeteoRA modules in total, with each of the 32 decoder layers containing 7 gates (Q, K, O, up proj, gating, and down proj).
2

---

Published as a conference paper at ICLR 2025
Advanced performance: Evaluation shows superior performance in composite tasks when applying MeteoRA, thereby extending the practical utility of LLMs incorporating off-the- shelf LoRA adapters.
2 BACKGROUND
Low-Rank adaption. Low-Rank Adaptation (LoRA) (Hu et al. 2021 proposes a method to reduce the number of trainable parameters required for fine-tuning in downstream tasks. LoRA injects two trainable low-rank matrices E RRdxr and B E Tk* into each basic linear layer's weight matrix W E Rdx of the Transformer-basee LLM M. The matrix multiplication of A and B represents the updates A to the weight matrix W when fine-tuning the model. The LoRA adapter modifies the forward process of this layer as follows:
(I) where I E R. represents the input hidden states for any token, A, B first project it to the low-rank embedding space IR" and then map it back to the output space Rd. LoRA can be applied to seven types of linear layers in the Transformer: four in the self-attention module (W;, WN. W., and Wo) and three in the MLP module (W (Wup.proj- W gaating and Wanwn.proj). Training LoRA adapters straightforward. It continues to the optimization target of causal language modeling update LoRA's parameters while freezing the billions of parameters in the pretrained LLM M.
Obsse + Ao = wase + &AW base + ((x x) B) B)
Multi-task LoRA fusion. LoRA adapter is usually fine-tuned to specific downstream task. To enhance the capacity of LLMs in handling multiple tasks, two paradigms are utilized in practice. One approach is to fuse datasets from different tasks and then fine-tune single LoRA module on this combined dataset. However, Ling et al. (2024) points out the difficulty in learning all specialized knowledge of various domains in one LLM. The other approach leverages existing LoRA adapters as off-the-shelf components, directly merging these adapters into one base LLM. Current popular LoRA frameworks. such as PEFT (Mangrulkar al. 2022) and S-LoRA (Sheng et al. 2023). allow fusing multiple LoRA adapters. However, these frameworks must explicitly assign the active injected LoRAs, leaving an obvious disadvantage of lacking autonomous on-demand LoRA selection and timely LoRA switching during inference. Existing work, such LoRAHub (Huang et al.) (023). could combine multiple LoRA adapters without the explicit task intention given by humans. However, few-shot/in-context learning is required for AHR for every single downstream task.
Mixture-of-Experte MoE is machine learning paradigm that enhances model performance and efficiency by combining predictions from multiple specialized models, or experts. Introduced by Jacobs et al. (1991). MoE uses a gating network to assign input data to the most relevant experts dynamically. This approach leverages specialized knowledge from different experts, improving overall performance on diverse and complex tasks. Recent progress, particularly by /Saazeer et al. (2017), has demonstrated the effectiveness of MoE in large-scale neural networks. By using sparsely- gated MoEs, where only a subset of experts is activated for each input, computational efficiency is significantly increased without compromising model capacity. This has proven particularly useful in scaling Transformer based architectures for various applications, such as Mixtral (Jiang et al. 2024
GLaM (Du et al. 2022). DBRX (The Mosaic Research Team (024) and Grok-1 (xAl 2024).
THE PROPOSED METEORA
3.1 METEORA ARCHITECTURE
Given base LLM M and n existing LoRA adapters L2,, _,,) that have already been fine- tuned with the distinct tasks (D1, D2.... D.,) on M separately, our objective is to integrate the existing LoRA adapters into the base M via MeteoR framework, resulting in LoRA embedded model M Membed. Figure demmonsttttsss the MeteoRA module complemented to each basic linear layer in LLM. The reused LoRA adapters are off-the-shelf ones, available from open-source communities or have been fine-tuned for specific tasks, Each LoRA adapter L; contains set of low-rank matrices {A, B,). MeteoRA furnishes each basic linear layer with a wide MoE architecture to embed the low-rank matrices provided by LoRA adapters.
Figure shows the architecture of MeteoRA module. To embed n existing LoRA adapters, MeteoRA module leverages the MoF architecture by injecting trainable Gating network G Rd R
3

---

Published as a conference paper at ICLR 2025
Porameters requiro training
Directly reuse existing LoRA adapters
Base Layer from LLM
LORA
Network
LakA,
x
MeteoRA Module
Add
X
Mat-Mul
Forward
Module
Figure 2:
The architecture MeteoRA module with MoE-style LoRA embedding. MeteoRA directly reuses existing LoRA adapters without fine-tuning and only requires training the Gating network.
together with n existing pairs of {A, to M. By applying G(z), MeteoRA selects pairs of {A,, } with the top-k highest gated weights for each æ. It then proceeds with the forward pass as follows:
Obase + A0r(s) x wbsee + xAWr(a) W base
Σ (E7(2)
w. ((x x A.) B.)
where /(x: = {11, i2; i denotes the top-k LoRAs selected for each token æ, which may varies from one another every batch, and w; is the normalized weight of the selected LoRA L;- wy could be calculated as follows:
exp(Gi(xx)
softmax(G((x))
Σ (xp(G,(x))
jET(m)
where Gz( denotes the unnormalized gated logits for the i-th LoRAs. By doing this, the Gating network performs as routing strategy for selecting the appropriate LoRÁ adapters based on the layer's input. Each MeteoRA module contains Gating network, and the Gating networks from different MeteoRA modules make decisions based on their own inputs, the selection of LoRA adapters could be dynamically switched in the forward process of each MeteoRA module through all LLM's decoder blocks. MetcoRA also applies top-1 and top-k gating strategies as detailed in Appendix A.
3.2 LEARNING ALGORITHM
Training the injected MeteoRA modules adheres to the principles of fine-tuning LLM under autore- gressive language modeling tasks. Given that n pre-trained LoRA adapters, the training procedure for MeteoRA needs to maintain the parameters of the base LLM M and the LoRA adapters fixed. Since MeteoRA supports top-k experts (LoRAs) selection, we introduce the joint optimization that combines the loss of autoregressive language modeling Zin and all losses of Gating networks Cgate:
B
L C= Lim + BEgate = arg max Σ (log P(x; 2;-1:0) +
4 (h))
j=1k=1
where 3 is the hyper-parameter, i is the token index, L is the length of the language sequence represented as tokens. z; represents the token. The loss le,j is the cross-entropy loss for LoRA classification in one MeteoRA module. For a base M contained B decoder blocks with m MeteoRA modules in each decoder, Lyane sums the loss 4 (h) based on the corresponding hidden inputs h.
3.3 METEORA FORWARD ACCELERATION
The core component of the MeteoRA module is MoE architecture that incorporates n existing LoRA adapters. The classic MoE forward method, called loop-ortz oriigna,,, employs a foo- style of computation that processes only the tokens assigned to the i-th LoRA adapter in the i-th iteration,
4

---

Published as a conference paper at ICLR 2025
leading inefficiency especially when token-adapter assignments are sparse when X < (e.g., during decoding inference phase where s is fixed to 1), and resulting in up to a 10x slowdown compared to the single-lora forward in our experiments. To address this, introduce bnm- torch method which parallelizes the computation by performing batched matrix multiplications (BMM) (PyTorrc 2024) for all b x k tokens and adapters at once, represented by:
[A....... (ors)
Σ 1w....., Wbsk] @
11 ([I1...,,bbba Dbakk x [A......, A x
Bi Bissk)
6x8
bxexk
bxaxk
bxexk
bxoxk
This results in a 4x speedup over loop-original, thus only ~ 2.5x slower than the single-lora in most of our experiments (see Section 4.4p. However, bmm-ttoch achieves its great efficiency by temporarily allocating a locasd -sized space on the HBM for batched A, B, causing a potential out-of-memory risk for those tasks when either b or s quite large. To mitigate this, we further propose bmm-triton method with a custom GPU kernel implemented in Triton (Tillet et al. (019) to resolve this memory issue. our evaluation, bmm-triton achieves ~ 80% performance bmm--torch while maintaining the same low memory footprint as loop-original (see Section 4.4). This makes bmm-trrtonn more suitable solution for large-scale tasks, effectively balancing computational speed and memory efficiency. Thus, the two proposed acceleration methods could together boost the inference in practice, by using bem-triton in the prefill phase (where the sequence length s varies based on the input sequence) and bmm-torch in the decoding phase (where the sequence length 1). The details on the design of the hmm--rrttn kernel are provided in Appendix A.2
EVALUATION
We conduct experiments on individual and composite tasks as detailed in Section 4.1 For our base models, use two well-known LLMss, LJaMA2-13B (Touvrro al. 2023) and LJMMA3-8B (Meta 2024 The code and the models are available
4.1 EVALUATION SETTINGS
LoRA tasks and datasets. We select 28 tasks from well-known benchmarks for our experiment.
Specifically, our task set consists of 22 tasks from BigBench (bench authors (023). three non-English English translation tasks from Cews-Commentary (Tiedemann, (0022), and three widely utilized tasks: GSM8K (Cobbe et al. 202T), CNN/DailyMaii (See et al. and Alpaca (Taori et al. 2023). These 28 tasks span variety of NLP categories, such as contextual comprehension, conversational question answering, summarization, translation, mathematics, logical reasoning, and multilingual challenges. For detailed task descriptions, refer to Appendix A..
Metrics. We apply a zero-shot evaluation setting for all tasks, adding brief task descriptions for tasks such as CNN/Daily and the three translation tasks that not inherently include task descriptions. As for metrics, we accuracy for multiple-choice tasks and GSM8K while employing metrics such as BLEU, ROUGE-I, ROUGE-2, and ROUGE-L for other tasks.
Models. We use LlaMA2-138 13B and LlaMA3-8B the base LLMs for LoRA and MeteoRA adaption. Both LlaMA models are pretrained LLMs and do not include the process of instruction tuning. We train specific LoRA adapters for each task using their respective training sets. The process of training LoRA adapters could be offline or dismissed when off-the-shelf LoRA is accessible. Then, the Gating networks, which embed the adapters in the MeteoRA module, fine-tuned efficiently based on the balanced dataset containing 1,000 samples for each task. The Gating networks for 28 tasks take more than 10 hours reach the convergence with 4 H800 training via Accelerate (Gugger et al.) 2022). For scenarios where the training data for the original LoRA adapter is limited, we train Gating networks using 2 strategy, with only 100 and 5 samples accessible per task.
'The implementation code is accessible at pestps://githi www http://///uueeeeggggne/eeettcoot and the two MeteoRA embedded LLMs are available attttps://hugginggaccs NJJ.Depp:nngine htwpl//ppppiiii///teeeee..o n 1mmammm22 13b and https://huggiin face 20/ htt..o//eeeeeeeeiiinn.... MeteooAA1 11amo3
5

---

Published as a conference paper at ICLR 2025
OOMA
Lig..
Anede
Cavion Rondo
pad
O od
Srade
PEFT
LoRA-F
LORA-8
Avg Merge
TIES
DARE
Arrow
LoraHub
MeteoRA(T1-1k
Figure 3: Evaluation results on the 28 selected tasks. The results on the left are based on LlaMA2-3B. while those on the right are based on LlaMA3-8B. The MeteoRA performs similarly on most tasks, leading to high overlap between the two polygons in the radar graphs. For clarity, we only draw results from MeteoRA with top-1 strategy in the radar graphs. Detailed results for each individual task are available Appendix A.4
For baseline comparisons, we train one LoRA adapter (i.e., LoRA-F) using a mixed training set from all 28 tasks, and another LoRA adapter (i.e.. LoRA-B) with the balanced dataset designed for training the Gating network. We also use Huggingface PEFT (short in PEFT) loading all 28 LoRA adapters (same ones used for MeteoRA) with explicit LoRA activation information during evaluation reference model. Additionally, we include several LoRA merge methods for comparison, including: averaging 28 LoRA adapters (referred to as Avg Merge), TIES (Yadav al. 2024a). DARE (Yu et al. 2024). Arrow (Ostapenko al., 2024), and LoraHub (Huang et al. 2023).
All L adapters interact with seven linear layers in L.LMM'ss Decoder layer, configured with = 8, 16, and learning rate of 5e 5. Due to some tasks having small training sets, the batch size for fine-tuning set to 4. All our experiments were conducted on a GPU server with five H800 80G GPUs. Notice that carefully selected the training hyperparameters for the LoRA-F and LoRA-B to ensure that their performance on the 28 tasks would not be excessively incomparable.
Composite tasks. To evaluate the model's ability to sequentially solve composite tasks, construct three composite evaluation sets by serially concatenating independent tasks. These evaluation sets, referred composite-3, composite-5, and composite- consist of 5, and 10 tasks, respectively, each containing 200 samples. The samples in each composite-n set can be viewed as a single "paper sheet" created by concatenating the tasks in sequence. During evaluation, the entire "paper sheet" is input into the model, which is required to sequentially generate both the task number and the corresponding answer for each task the order presented. This setup tests the model's s ability to handle multiple tasks within a single input, maintaining coherence across the sequence. Temperature scaling involved Gating network. More details refer Figure Appendix A.5 and A.6
4.2
MAIN RESULTS
Figures demonstrates the performance of the MeteoRA models, LoRA-F, LoRA-B, 5 LoRA merge methods, and a reference model PEFT based on LlaMA2-13B and LlaMA3-SB, respectively, across the selected 28 tasks. Table shows the averaged scores in various matries for all methods.
The evaluation results indicate that, regardless the base LLM, the MeteoRA models utilizing the t strategy achieve performance very close to the reference model PEFT, while explicit LoRA activation/ddaaciivttoon is required in MeteoRA. Although LLMs with both LoRA-F and LoRA-B reach comparable performance several certain tasks, they exhibit significantly poorer outcomes

---

Published as a conference paper at ICLR 2025
Table I: Results of the 28 selected tasks on (LLMA2-13//LLMM3--333 base LLMs. TI and T2 represent the top-1 and top-2 strategies, while the subsequent numbers indicate the number of accessible samples per task for gate training. Our methods perform the best in most tasks. Notice that the task linguistics puzzles achieves significantly higher ROUGE scores on LlaMA3-8B base, disproportionately influencing the average ROUGE scores and resulting in slightly higher averages for LoRA-B. Excluding this outlier, our methods consistently lead in performance across the evaluation.
Model
Accuracy 0.762 / 0.817
BLEU 35.66 / 45.32
ROUGE-11
ROUGE-21
ROUGE-LA 0.316/03317
PEFT (reference)
0,340
0.163/0.164
LoRA-F
0.730 / 0.767
41.27 / 42.93
0.318 0.322
0.136/0.157
LoRA-B
0.666 / 0.750
37.98 / 38.47
0.314 / 0.343
0.128 / 0.117
0.288 / 0.321
Avg Merge
0.370 / 0.427
19.23 / 39.89
0.231 0.200
0.082/0.066
0.184 / 0.158
TIES
0.388 / 0,441
47.28 / 34.66
0.195 0.199 0.199
0.151 / 0.158
DARE
0.332 / 0,404
46.53 36.74
0.192 / 0.188
0.054/0.00
0.144 / 0.147
Arrow
0.569 / 0.647
41.03 / 29.93
0,281 0.283
0.123
0.234 / 0.242
LoraHub
0.307 / 0.235
13.43 / 10.11
0.158 / 0.141
0.049 0.035
0.124 / 0.104
MeteoRA (TI-1k)
0.755 / 0.811
36.73 / 45.64
0.336 / 0.338
0.160 0.158
0.313 / 0.314
MeteoRA (T2-1k)
0.748 / 0.806
38.97 / 44.98
0.336 0.337
0,161 / 0.158
0.314/02 0.311
MeteoRA (T2-100)
0.758 / 0.783
39.44 39.90
0.331 / 0.309
0.159 / 0.1199
0.281 / 0.256
MeteoRA (T2-5)
0.740 / 0.773
38.37 / 40.12
0.328 0.299
0.156 1.56/0.13 0.131
0.277 / 0.246
Table 2: The evaluation results of composite-n tasks. MeteoRA is marked in color on the side, while LoRA-B is in black on the right side. Refer to Appendix A.5
for a detailed explanation.
Metric
composite-3 3
composite-5
composite- 10
# Avg Attempt
2.95
3.00
4.63t
4.33
8.24f
6.07
Avg Correct
1.491
1.31
2.62t
2.42
3.75f
2.95
Avg BLEU
15.31
10.55
9.86t
9.41
8.85f
8.71
Avg ROUGE-1
0.195t
0.135
0.2211
0.219
0.2381
0.161
Avg ROUGE-2
0.052t
0.027
0.069
0.063
0.0591
0.043
Avg ROUGE-L
0.1821
0.128
0.207L
0.208
0.2091
0.123
others. Additionally, MeteoRA employing the top-2 strategy, despite occasionally showing greater capability loss compared to MeteoRA with top-1 strategy, occasionally outperforms PEFT with adapters trained directly on the individual tasks. This suggests that the Lim component in the loss function (Fquation 4 becomes influential in these cases, indicating a beneficial mix of LoRA adapters from various tasks for future study. For the MeteoRA (T2-100) and MeteoRA (T2-5), although their performance shows gap compared to MetcoRA Ik, they still outperform the baseline models on most metries. This demonstrates that the Gating network can still learn to effectively utilize existing LoRA adapters with only few examples.
4.3 COMPOSTE-N TASKS
The evaluation results for these three tasks are illustrated in Table 2 Notice that only LlaMA3-8B with the MeteoRA (top-2 strategy) and LoRA-B effectively address these composite-n tasks. Subsequent discussions will therefore focus exclusively on these two models. Although the MeteoRA model attempts slightly fewer questions than L-RA-B in composite-3 tasks, it correctly answers higher number of multiple-choice questions and achieves superior BLEU and ROUGE scores. As the task complexity increases to c 5 and composite-10, MetcoRA outperforms LoRA-B in almost all metries. For more details, refer to Appendix
To further validate the functionality of the Gating network in the MeteoRA block, we display the LoRA selection patterns in the inference process of a composite-3 sample in Figure 4 With the top-2 strategy, Gating network appropriately assigns greater weight to the corresponding LoRA adapters for the majority of the tokens, no matter in input or output. At the junctions of two adjacent tasks, the Gating network correctly performed the timely switching actions of LoRA adapters.
7

---

Published as a conference paper at ICLR 2025
Reample HinooNNST) The pingroph
arnnged there
lagioilly within goragaahh
bink... Which
andiise: The canlinal
chote:: blue kiluwat (alNNTT) [NST] Gives shott
which las Hore editing auschoggtsgto (aChoose approptite
Jaia The salco leftmost Inz Who first tedaa evente? (INSTT].nal Bunner. Robert
appropeitte
Example
(JNST])
Hilk's aher
Yes, Hiil'ss alter Nacler plyyics
Robert
Banner nuclest
Bold of plysics that autee
intercttonn ww..lee finshed secodd (nd How loep did through compeey 11a No. The mutimum with Reglish Charnel is
mudes their
in in[NNSS) The following pangraphes each descrite
Esample
theuss tike
S0mllss The miniuum witth of lowa 200 miles in iafllloaingg the chimpl shine:
genorie alowes to the quceiion below (NST The following paragaahe each dosnile of scves objects ananga in flned onler The staemennt are loginally
the socond from second from the right unchooo: The saven the second from the right (ENST] in a INST Given shont unswer aiong with its conceat, select
Composite task impur
the right. solloie: The hamminghed
Choose the approprizze question which has the given answer. (ANSST) a (IENST) Q Could the Allcs (.vventzess Wonderland" "ndd join Masonic Lodge uuAc (INSST). j The quail is the second from the right. inz Which of genghs kaon descendants was responsible for driving the mambiik from polestine"! n.. No. The main choracter of 'Alice's Adventures in
main choracter
Model output
Wonderfane" is Alicn, Women are not allowaed in join Masonic Lodgre
pelessio
virsieggage
other taak
Figure 4: An example of 3 task. We highlight the statistically dominant LoRA selected by MeteoRA in token level (decoded to words). The result shows that LLM with MeteoRA could achieve timely LoRA switching both phases of input understanding and output generation. The background color gets darker when Gating network assigns a higher weight value.
225
buseine
exiiinal
2.1
2.5
Figure 5: The overall root-of ronttmmee- of four forward pass designs on 28 different Big- Bench subtasks.
4.4 EFFICIENCY
To assess the efficiency of our novel forward pass designs using custom GPU kernel operators, we truncate batch_size x 10 samples from each test dataset of all 28 tasks. We evaluate these designs alongside four variants with the same hyperparameters: the upper-bound single-lora, the baseline toop-original, and two novel forward acceleration strategies based on bmm:: bmm-torch and bmm- triton, implemented by Py PyTorch and Triton respectively. Figure dispplas the histogram of the overall root-of-runtinee metric for each task and design. Additional evaluation is detailed in Appendix A.7
RELATED WORK
Multi-task fusion. Our proposed method falls into the field of LoRA adapter composition for multi-task fusion. The first category focuses on fusing the entire models. Researchers mainly study model ensembling and multi-task learning to achieve this goal. Existing works integrate the models under the setting of shared model architecture (Matena & Raffel 2022 Jin et al. 2022 Wu et al.) 2023al Yadav et al. 2024b). Others focus on merging models with various architectures or from different tasks. Both methods (Stoica al. 2023 Liu et al. 2022) try merge models that trained for various tasks without additional training. The second category is more concerned with
fusion in terms of the tasks. [Iharco al. 2022) proposes model editing method via task vectors.
8

---

Published as a conference paper at ICLR 2025
Sun al. (2022) leverage in-context learning with few-shots to enhance the performance of unseen tasks. However, these methods require multi-tase training or prior knowledge for the evaluation tasks. Our method embeds off off-the-shelf LoRA adapters with a Gate network in the MeteoRA module. None of the examples (zero-shot) are required for all individual tasks.
Fusion under MoE. In the context of tretrain-fine-tuee paradigm, PEFT becomes a common sense for developing Transformer- based LLM downstream applications. Directly fine-tuning on a fused dataset from various tasks is unable to achieve better performance [Ling et al. (2024). Some works focus on leveraging existing LoRA adapters as off-the-shelf components, integrating them directly into a base LLM. For example, PEFT (Mangrulkar al. 2022) and S-LoRA (Sheng et al.) 2023 are frameworks aiming to embed multiple LoRA adapters to one LLM. However, requiring explicit activation/deeactiittonn during usage. MixL (I. et al.) 2024) targets to a resource-efficient sparse MoE model, fine-tuning MoE on MLP module with the auxiliary load balance loss used in Mixtral (Jiang et al. 2024). Although MixLoRA supports LoRA adapters for the attention layer, the adapters are still dense models encompassed with the linear layers in the attention module. Others (Huang ct al. 2023 Yang et al., 2024, Feng et al. 2024] Chen et al. 2024] Wu et al. 2023c) propose LoRA fusion based on the concept of Mixture-of of expeets that enhance the model's s ability for cross-domain tasks. However, the methods mainly focus on fusing LoRA adapters to the FFN module or Q in the attention module. Our method could embed all kinds of LoRA adapters. By leveraging the full-mode MoE architecture, the LLM's capacity could be boosted with autonomous and timely LoRA switching, especially for solving composite tasks.
LIMITATIONS
LoRA adapter update. Although the Gating network within MeteoRA module is trained separately among the adapters, it is necessary to retrain or fine-tune the Gating network if some LoRA adapters are updated. The Gating network is trained using the hidden state as inputs, which are influenced by LoRA adapters in previous layers. Testing revealed that directly replacing some LoRA adapters with improved versions did not enhance performance on our test set. However, after retraining the MeteoRA modules, the LLM equipped with MeteoRA exhibited performance improvements. Technically, this issue may be related to the domain shift problem, where the Gating network is applied to another operational field the distribution shift. Employing statistical methods such as (Li et al., 2020, Krishnan & Tickoo, 2020) may help calibrate the output of the Gating network to produce more accurate logits and results.
Knowledge fusion tasks. Composite tasks, which involve a broad range of tasks, represent one type of complexity in terms of the scope of tasks. More challenging are tasks that require knowledge fusion across domains. To assess the capability of MeteoRA in knowledge fusion task, we construct a mathematics task by translating problems from GSM8K into a foreign language (e.g.. Italian), so that the LLM with MeteoRA must solve these foreign language GSM8K problems by leveraging knowledge from both GSMSk LoRA (trained on problems in English) and the foreign language LoRA (trained for Italian to English translation). Although MeteoRA successfully fuses the two LoRA adapters to address the math problems in a foreign language, it does not show superior performance compared to LLM equipped only with the GSM8K LoRA. We hypothesize that the base LLM's existing proficiency in the selected foreign language may render the additional adapter unnecessary. Future efforts could focus on constructing more suitably complex tasks where the required cross-domain knowledge is not already pre-trained into the base LLM.
MoE efficiency. Sparsely-gated MoE (Shazeer al., (017) offers computational efficiency advan- tages over dense MoE. However, the naive implementation of MoE forward originaa,,,,,, such as the SparseMoF in Mixtral (Jiang et al., 2024 Wolf et al. 2020), still encounters efficiency issues when the number of experts increases. In our evaluations, the runtime for inference can be up to t longer than that of single-lora when embedding 28 LoRA adapters into one LLM. With our proposed forward acceleration techniques bmm-torch and bmm-triton, we achieve a speedup of 4x compared to the poiioorig though this still falls short of the ideal upper bound (single-lora) Technically, it is extremely difficult to increase the inference speeds for MeteoRA when the number of embedded LoRA adapters increases. Future work could explore developing new operators in triton or CUDA to continuously enhance MoE acceleration in terms of memory efficiency.

---

Published as a conference paper at ICLR 2025
7 CONCLUSIONS
This paper presents a framework MeteoRA that achieves scalable multi-task LoRA embedding within LLMs, enhancing the existing LLMs with full-mode MoE architecture with forward acceleration strategies. LLMs equipped with MeteoRA enhance the ability to autonomously select the most pertinent LoRA adapters to generate appropriate responses. Moreover, its capability for timely LoRA switching leads to superior performance, particularly in sequentially solving composite tasks. Future work could explore the transformative potential of MeteoRA in multifaceted cobblem-solving scenarios, and inference efficiency by designing more efficient GPU kernel operators.
ACKNOWLEDGMENT
We are thankful to the anonymous reviewers for their helpful comments. This work is supported by Frontier Technologies R&D Program of Jiangsu (#BFF024459), the National Natural Science Foundation of China (Grants #62172199), and the Collaborative Innovation Center of Novel Software edu. cn) edu. the corresponding
Technology and Industrialization. Jingwei Xu (jingwei@njul
author.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXhv:2303.08774 2023.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
https:://ppereeve/ net wowumoiioosooor.... LtrruumiddduyLLLSBe...
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with Ilms: Fusing object-level vector modality for explainable autonomous driving. arXiv preprint 1rXiv:2310.01957, 2023.
Shaoxiang Chen, Zequn Jie, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning milms. arXiv preprint arXiv:2401. /6160, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 10.1618. 2021.
Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092. 2023.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixters In International Conference on Machine Learning, pp. 5547-5569. PMLR, 2022.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mfxture-ofllore loras: An efficient multitask tuning for large language models. arXiv preprint arXin:24033003332, 2024.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https: / http//gitti httpt/tttttttttttt///////ttthett com/ huggingfaceiceeeeccoccooogo...ne aacceeerate... 2022.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrrn Recognition, pp. 15953-14963 2023.
10

---

Published as a conference paper at ICLR 2025
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint NrXiv:2106.09685, 2021.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint wrXhv:2307.12269 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint 12Xi::2212.04089, 2022.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.0998/ 2022.
George Sooggiiann--Mimmaraaee and Georgia Koutrika. A survey on deep learning approaches for text-to-sql. The VLDB Journal, 32(4):905 2023.
Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus un- certainty optimization. Advances in Neural Information Processing Systems, 33: 1837-17--8,,8, 2020.
Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts, 2024.
Zenan Li, Xiaoxing Ma, Chang Xu, Jingwei Xu, Chun Cao, and Jian Lü. Operational calibra- tion: debugging confidence errors for dnns in the field. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, pp. 901-913, New York. NY, USA, 2020. Associa- doi: 10.1145/3368089.34000 3409666. URL
tion for Computing Machinery. ISBN 978 458174407743..
ht https:://ol. orooiiorgg/// 1000..11555500 01155/3388899 000.5688888.0009... 3409696
Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui. Xuchao Zhang, Tianjiao Zhao, Amit Panalkar, Dhagash Mehta, Stefano Pasquali, Wei Cheng, Haoyu Wang, Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White. Quanquan Gu, Jian Pei, Carl Yang, and Liang Zhao. Domain specialization as the key to make large language models disruptive: A comprehensive survey, 2024.
Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In International Conference Machine Learning, pp. 13857-13869, PMLR, 2022.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: ttate-of-the-art earameter-efficient fine-tuning methods. https: githhb com/ hwt.hhiggiggaa//petttt 2022.
Michael S Matena and Colin A Raffel. Merging models with fishhr-weighted averaging. Advances in Neural Information Processing Systems, 35:17703- 2022.
Inc Meta. Build the future of with meta llama 3. 2024. /tEps://1lance www.eeaa.....m com/
11 3/
11

---

Published as a conference paper at ICLR 2025
Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti, Laurent Charlin, Nicolas Le Roux, Matheus Pereira, Lucas Caccia, and Alessandro Sordoni. Towards modular llms by building and reusing a library of loras. arXiv preprint arXiv:2405.11157 2024.
Mohammadreza Pourreza and Davood Rafiei. Din-sql: Decomposed in-context learning text-to-sql with self-correction. Advances in Neural Information Processing Systems, 36, 2024.
PyTorch. toreh.bmm - pytorch 2.3 documentation. generated/ orch. brrn. html 2024. Accessed: 2024-05-23.
hwtp://www.....ooo///ooooos/s/// stabbl
Abigail See, Peter J. Liu, and Christopher D. Manning. Get the point: Summarization with pointer- generator networks. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10. 10.18653/11/17-11000 URL https: www. ac aclwes
org/ hxtp/anthologyyE17--1
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-exxprts layer. arXiv preprint 10XXvv/701..65..00 2017.
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:231 1.002885 2023.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2998-3009, 2023.
George Stoica, Daniel Bolya, Jakob Bjomer, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. arXiv preprint arXiv:2305.0300 2023.
Tianxiang Sun, Zhengfu He, Qin Zhu, Xipeng Qiu, and Xuanjing Huang. Multitask pre-training of modular prompt for chinese few-shot learning. arXiv preprint vrXiv:2210.075500 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An inslluction-foloowingg Ilama model. https: //g httpt/tttt.bt///gg/ com wwwo//ta...o htt///atttuuu 1ab/ sttanford_alpaeee 2023.
The Mosaic Research Team. Introducing DBRX: A New State-of-the-Ar Open LLM. https: :// art open 11m
www. dat auttabricke www.w///g/..ttt//////cooooo March 2024.
Jörg Tiedemann. Parallel data, tools and interfaces in OPUS. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Dogan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Eighth International Conference Language Resources and Evaluation (LREC') 12), pp. 2214-2218, Istanbul, Turkey, May 2012.
European Language Resources Association (ELRA). URL http: www. www.-conf.orgg/////oooooooo.
piooeedings//ree222/////////
Philippe Tillet, Hsiang- Kung, and David Cox. Triton: intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop Machine Learning and Programming Languages, pp. 10-19, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 23X2 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-Ae Natural Language Processing. pp. 38-45, Association for Computational Linguistics, October 2020. URL https: // wwwp//wwww.//././/////// ac web. org hww.bbbogggattthooooooooooooooo 2020 emn demos.
12

---

Published as a conference paper at ICLR 2025
Chengyue Wu, Teng Wang. Yixiao Ge. Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Luo. pi-tuning: Transferring multimodal foundation models with optimal multi-task interpolation. International Conference Machine Learning, pp. 37713 PMLR, 2023a.
Shijie Wu, Ozan Insoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhan- jan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. 2023b.
Xun Wu, Shaohan Huang, and Furu Wei. Mole: Mixture of lora experts. In The Twelfth International Conference Learning Representations, 2023c.
XAL. Grok-1.
2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models, Advances in Neural Information Processing Systems, 36, 2024a.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024b.
Shu Yang, Muhammad Asif Ali, Cheng-Long Wang. Lijie Hu, and Di Wang. Moral: Moe augmented lora for Iims' lifelong learning. arXiv preprint arXiv:2402. 12260, 2024.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning. 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 TOP-K STRATEGY
Top-1 strategy: When the Gating network is configured to select the LoRA adapter with the maximum logit, the forward process of MeteoRA detailed in Equation simplifies to the classical LoRA forward o = x base + (x x A,) x B, Thus, the weight w, calculated by the Gating network G; only contributes to the LoRA selection, but does not influence the token generation process, resulting in it being irrelevant to the loss Lim- Thus, training of Gating networks under the top-1 strategy could utilize the following truncated loss function Ltop-1:
Lrop1 arg max ΣΣΣ L (h)
(5)
Top-A strategy: With the top-k strategy set in the Gating network. MeteoRA computes the normalized weights wy for the h selected LoRA adapters. These weights participate in the computation of the losses of both Lim and Latte specified in Equation 2 Thus, the parameter updates for the Gating network derive from the losses associated with both LoRA classification and autoregressive token generation. Notice that the LoRA classification loss is only influenced by the LoRA adapter with the highest logit, whereas the backpropagation from the token loss affects the parameters in the Gating network responsible for all k selected LoRA adapters. Although these remaining k adapters lack direct supervision from the LoRA classification loss, the token generation loss contributes to enhanced robustness and the capacity for LoRA switching during generation.
A.2 DETAILS ON THE TAILLRRE TRITON KERNEL. FOR EFFICIENT METRORA A FORWARD
To address the memory copying problem caused by Py Torch indexing, we fuse the two bmm operations inside GPU kernel function implemented by Triton, which dynamically indexes the right pair of LoRA matrix (A,, B.) and load them from HBM to SRAM in each parallelized thread. Therefore, there is no need to explicitly allocate b S X pairs of (A,, B,) over the original n ones.
Another challenge is that Triton constraints all the dimensions for the matrix operators should be no less than 16, however, under the MeteoRA settings, this requirement can never satisfied since the first operator æ is vector, and also, the LoRA rank size may be less than 16 easily (c.g.. in all our experiments, we fix 8). Therefore, it is not that trivial to implement such kernel, unless using the simple masking strategy to meet the requirements with over 15x waste of I/O.
Algorithm 1 Pseudo Code for BMM-Triton Kernel Function
1: Prepare blockized X, A' with their masks M1, M2 before launching the kernel > I is the candidate LoRA index set
2: Load X, J, Mj, M2 from HBM to SRAM
3: Load A', B indexed by
4: OA' = X x A'
5: oA" - ((oA' @ M1) M2)
6: oB' oA" B
7: O < colsum[oF'
D. Compute column-wise sum
8: Store O back from SRAM to HBM
To both obey the dimension constraint and avoid too much waste, for the first bmm of a E R and A E dR we use blocking strategy to split the vector x along the hidden size dimension by m blocks, where m >= 16, In such case, the first operator becomes matrix X with shape (m, ) and also we have to split A along the first dimension to become a more square matrix A' with shape (á. r m). Notice that now the output of first bmm:: oAA' = X A' with shape (m, r x m) has relationship with the original one AA = x x with shape (1,r) r) as follows:
oA" (oAA' 3 Mi) x M2)
(6)
0.4 = colsum[oA")
14

---

Published as a conference paper at ICLR 2025
where M1 and M2 are two trivial 01 mask matrixs, each sized (m,rxm) and (rxm,r) respectively. So we can transform back to the right results by three additional negligible dot-produe with M1. matrix-proouett with M2, and colsum operations for the first bmm. For the second one, instead of directly using the right result oA, we can delay the colsum operation until we finish the second hmm, L.e. oA" with shape (m.r) and B with shape (r; h)) do matrix-produce operations to get the temporary result oB' with shape (m, ft), then apply colsum to get the final LoRA output O with shape (1, ht. Notably, on one hand, can avoid one more blocking operation for oA since oA" already meets the dimension constraint, on the other hand, if r 16, we can just simply utilize masking strategy since it is the inner dimension and small enough.
Overall, for the Triton kernel function, we offer the pseudo code as shown in Algorithm !
15

---

Published as a conference paper at ICLR 2025
A.3 INFORMATION ABOUT 28 TASKS
Table 3 shows the detailed information of the 28 selected tasks in the Section The name in parentheses is the abbreviation of the corresponding task. We use the original training sets from these tasks to fine-tune the LoRA adapters and Gating networks in MeteoRA modules. To achieve a balanced fine-tuning across the diverse task spectrum and ensure efficient training, we construct a balanced dataset by randomly sampling 1,000 samples from each task. This balanced dataset is then divided into training set with 25,200 samples (i.e., 900 samples for each task) and a validating set with 2,800 samples (i.e., 100 samples for each task) for fine-tuning In terms of the evaluation part, the performances are evaluated on each task's original test set.
Table 3: Details about the 28 selected tasks.
Task Name
Keywards
Description
Evallation Motrics
abstract naraaive undensianding (AbsNarr)
narrative understanding,
Given narnaive, choose the most related proverh. Accuracy
mulhiple choice
alpoca (ALPACA)
instraction-taniee toningg
Write appcopriste answers according to instrue-
BLEU, ROUGE
enn mail (CNNDM)
summarication
Given news articles, wrise the summanicasion.
ROUGE
contexusal parametric knowledge conflicts (ComParaKCC
contextual
Answer questions given the contextual informa-
Accuracy
answeringg
multigle
choice
oualgorithms (CSAlg)
algorishos, numericall sponse
Solve two common computer science tasks.
Accuracy
disti ga (DidIQAA
contextual answesing. comprehansion
question- reading
Pick the coorect answer span fromt the comeex given the disfluess question.
Accuncy
elementury juntuu (FlemMath)
mathematics
Answer mahhiple choice motheratical word proh- lems.
Accemey
epistemic reasoning (EpiRosson)
lngical reasoning, multi pia choice
Determne whether one senience entails the next.
Accuracy
fermal fwllacccess sty Villogimssneggsiion (FormPilll
Ingical ressoninng, multi-
Distinguish deduciively valid arguments from for- mal fallacies.
Accamey
ple choice. reasoning. multi- ple
goal step wikihuw (GSWIk)
Perform three subtasks step inference, goal inference, step urdering.
Accuracy
gsm8k (GSMBK)
mathematics
Solve the grade school math wond problems.
Accency
kagungg.deemiiiasiinn IT
multilingual, chuice
multiple
Given sentence, solect the language
Accemey
linguistics puzzles (LingPuzz)
logical reasoning, linguis-
Solve Rosens Stone style linguistics puzzles.
ROUGE
Iegical_deduurionn pleduction (Dsgui)
Iopical ressoninng multi-
Deduue order sequence objocts.
Accency
ple choice
news commeniary de (NewsDD)
muliilingual, translation maliilingual, translation muttiliguua, translation
Transate German sentences into English.
BLEU
news commentary es (NewsES)
Translate Spanish sentences into English.
BLEU
news.commeotary Jt (NewslI)
Transiate Italian sentences into English.
BLBU
object_couming (ObjCouni)
lugical reasoning
Questions that involve enumerating objects and asking the model to count them.
Ascenacy
pangappl.egnnnttiin segunentation (ParSSeg)
pangraph
segmenttiio, multilin- gual
Ideatify the sentences that end paragraph in a document.
Accuncy
Moy_ddnlog.mmm wwm. sifffeeerem (PlayDaag)
reading comprehension, multiple choice
Deternine if neurby lines in Shakespeare play were spoken by the some individull
Accemey
question selection (QuestSel)
reading comprehension,
Given answer along with its context, select the appropriate question which has the given
Accemacy
multiple choice
reasonngg_aoon.coo htapiig__hoonnoooed_eoooo (ColefRessen)
reading comprehension,
Answer extremaly simple questions about the oal- ors objects on surface.
Accemey
logical reasoning. multi-
ple choice
strategygo (StauQAA)
Ingical context-free answering
reasoning. question
Answer questious which roquired ressoning steps are implicit in the question.
BLEU, Accunccy
ROUGE
Iopical_chot (TopChot)
free response
Open-dommin response generation.
BLEU ROUGE
tracking oobjessloooheuss (TrokOOhi)
lugical reasoning, multi- ple choice
Determine the final positions given initial pesitions
Accenecy
and descrigtion of a sequence of swaps.
unit.conversion (UnitComv)
contextual question- answering. muthematies,
Perfomm various tasks relating to units, including
Accuracy
idmiiiicttio and convension.
mulliple choice
vituminc fact verification (VitaFact)
trathfulness, reading comprehassion, multiple
Identily whether a claim True Filse based Accuracy the given
choice
winoshy (WinoWhy)
consal reasoninng, multi-
Evaluate reasoning in answering Winograd Schema Challenge quesioos.
Accuncy
ple choice
16

---

Published as a conference paper at ICLR 2025
A.4 EXPERIMENTAL RESULTS OF 28 TASKS
T Table S Table and Table show the detailed evaluation results of different models on the 28 selected tasks. When drawing Figure 3 for tasks we use BLEU and ROUGE metrics, selected BLEU for news ccmmenttar _le, news_commentary.co. and news www.sscoommnryy..ccc while opting for ROUGE-L for the remaining tasks.
Table 4: Experimental results for tasks using accuracy as metric (LlaMA2-13B 13B base model).
Localith
Alohias
0.863
0.758
0.720
0.562
0.34g
0.190
0.788
0.278
0.855
0.860
0.500
0.965
OuPanKC
0.993
0.999
0.934
0424
0.579
0.554
0.236
0.514
0,999
0.999
0.599
0.995
0.48
0.113
0.525
0.306
0.307
0.675
0.661
(1301
0411
1730
0.208
0.312
0.369
0.564
0.794
0.225
01771
0.71
Tpikksoon
1.000
1880
0995
07367
8.90
03377
0.6S
0.211
LJB0
8999
1.800
1
homilll
01999
0.21
0510
0350
6410
0.995
0.95
1.000
0.70
GSMSK
0.425
D002
0052
0.155
0.20
0,357
LanglD
03574
0.728
0.512
0.35
0.40
0.25
0,455
0.233
0873
0351
0.507
0.445
LogDedio
0.720
0.53
0.680
0.350
0.5
0.35
0.587
0.473
0.13
0.17
0720
0.733
ObyCeuus
0.240
0.600
03300
0.45
0.735
0.725
0.720
Pay
0.699
0.649
0.650
D649
0.60
0649
0.640
0.265
0.64
0.649
0.69
0.649
Queatit
(1007
0.801
0.754
0.509
B517
0.56
BM37
0.241
0.077
0931
0.30
0.931
Calorloooon
(1950
0.000
0.980
5,00
0:00
0395
0/A00
0.11
0.900
0990
09535
0.810
Toctriny
0.181
0.188
D191
0100
0138
0.191
0.135
0.23
0.785
LunDaas
0.755
0.779
0.707
0.35
0,500
0274
0.534
0.308
0.727
0.735
0.729
0.604
VisFact
0.299
0.388
0.812
0.71
0.6
0.20
0.417
0.245
0.907
0.97
0.897
0.893
Winolile
0.802
0.757
0.767
0602
0.03
0838
0.055
0.797
0.767
0.795
Table 5: Experimental results for tasks using accuracy as metric (LlaMA3-88) 8B base model).
Bank Nams
PIPT oocforsoo))
LORALF
LUBA-BB
Aug LeitA
VIS
DARE
Armo
Lamaltsh
MelooRA
MezuKA (F2-14)
MazzBAA12-1000
ManriAA(T2-5)
Ahchaur
0.415
0,793
0.790
0.415
01122
01700
0.74R
CSAls
0.541
0.341
0.541
0.602
0.526
0.826
0.830
DidOA
0.700
0.680
0.555
0,374
0.386
0.371
0.504
0.197
0.706
0.711
0/666
6665
EleoMan
0.780
0.777
0.500
0,373
0.366
0.245
0.635
0.006
0.775
0.773
0.51
0.725
FornFFal
0.989
0.910
0.588
0.519
0.535
0.56
0.190
0,967
0951
0311
GSSii
0.955
0.911
0523
0.56
0.534
0.35
0.337
0551
01923
0.64
0395
GSSS5K
0.391
0.586
03568
0080
0.266
0.108
0112
0830
0.55
0.509
0311
C491
LangtD
0.449
0525
0.733
OkyCouzt
0.550
0.555
0.565
0.500
0.000
0.130
0.06
0.20
0.755
0.50
6255
6050
Punding
0.700
0.761
0.66
5044
0.050
1188
0.000
0.215
0335
0331
MayDag
0.633
0.69
0.000
0664
1666
0.49
0660
0.50
0111
Colertionee
(.99%
0.415
0.65
0.504
0.530
0191
0.718
0.80
0961
0915
0995
01716
0.307
RTT8
5.000
0611
0:538
1161
0.531
0.69
01670
0.45
6611
Track0oj
0.955
0.585
0.66
0:47
0105
0:135
0.171
0.93
0,5%
0985
0.95
ViuFact
0.903
0.530
0,67
0902
0350
WinWWhy
0.516
0.757
0.50
0336
0333
0.700
0.208
0.815
0.27
0.758
6.758
Averge
0.517
0.767
0.70
9427
0,441
0,404
0,647
0.25
0
0,806
0.753
0.73
17

---

Published as a conference paper at ICLR 2025
Table 6: Experimental results for tasks using BLEU and ROUGE as metrics (LlaMA2-13B base model). Task Name
Model
BLEU
ROUGE-1
ROUGE-2
ROUGE-L
PEFT (reference)
16.03
0.363
0.176
0.340
LoRA-F
23.96
0.302
0.140
0.283
LoRA-B
11.72
0.341
0.157
0,317
Avg LoRA
41.88
0.195
0.084
0.164
TIES
80.34
0.209
0.092
0.175
ALPACA
DARE
78.25
0.228
0.101
0,193
Arrow
24.62
0.271
0.128
0,230
LoraHub
0.00
0.240
0.117
0.206
MeteoRA (T1-1k)
28.83
0.350
0.166
0.329
MeteoRA (T2-1k)
24.12
0.349
0.162
0.327
MeteoRA (T2-100)
39.09
0.332
0.160
0,281
MeteoRA (T2-5)
12.49
0.306
0.140
0.256
PEFT (reference)
7.50
0.228
0.067
0.214
LoRA-F
15.69
0.241
0.076
0.227
LoRA-B
15.65
0.228
0.067
0,214
Avg LoRA
13.08
0.144
0,032
0.104
TIES
13.08
0.147
0.032
0.104
CNNDM
DARE
13.08
0.126
0.031
0.081
Arrow
17.42
0.173
0.043
0,122
LoraHub
4.77
0.141
0.030
0.104
MetcoRA (T1-1k)
7.50
0,229
0.069
0.216
MeteoRA (T2-1K)
5.57
0.230
0.070
0,217
MeteoRA (T2-100)
7.32
0.251
0.070
0.196
MeteoRA (T2-5)
7.77
0.254
0.073
0.199
PEFT (reference)
46.17
0.716
0.479
0.659
LoRA-F
62.23
0.649
0.365
0.582
LoRA-B
54.91
0,608
0.324
0,541
Avg Lora
36.72
0.531
0.233
0,441
TIES
49.14
0.405
0.117
0.308
LingPuzz
DARE
68.87
0.379
0.102
0.285
Arrow
56.23
0.643
0.365
0.562
LoraHub
0.00
0.172
0.057
0.131
MeteoRA (T1-1k)
68.34
0.717
0.478
0.661
MeteoRA (T2-1k)
46.17
0.713
0.476
0.655
MeteoRA (T2-100)
46.17
0.718
0.480
0.646
MeteoRA (T2-5)
57.47
0.716
0.474
0.646
PEFT (reference)
78.25
LoRA-F
78.25
LoRA-B
78.25
Avg Lora
3.38
TIES
86.48
NewsDE
DARE
86.48
Arrow
86.48
LoraHub
50.09
MeteoRA (T1-1k)
86.48
MeteoRA (T2-1k)
86.48
MeteoRA (T2-100)
86.48
MeteoRA (T2-5)
86.48
18

---

Published as a conference paper at ICLR 2025
Task Name
Model
BLEU
ROUGE-1
ROUGE-2 ROUGE-L
PEFT (reference)
70.05
LoRA-F
57.03
LoRA-B
81.54
Avg Lora
2.86
TIES
70.05
NewsES
DARE
46.27
Arrow
81.54
LoraHub
0.64
MeteoRA (TI-1k)
81.54
MeteoRA (T2-1k)
70.05
MeteoRA (T2-100)
70.05
MeteoRA (T2-5)
70.05
PEFT (reference)
39.04
LoRA-F
54.90
LoRA-B
40.08
Avg Lora
40.20
TIES
40.08
NewsIT
DARE
40.20
Arrow
36.92
LoraHub
40.08
MeteoRA (T1-1k)
39.04
MeteoRA (T2-1k)
39.04
MeteoRA (T2-100)
39.04
MeteoRA (T2-1k)
39.04
PEFT (reference)
15.72
0.237
0.064
0.222
LoRA-F
9.71
0.247
0.076
0.238
LoRA-B
11.90
0.249
0.073
0.236
Avg Lora
14.54
0.185
0.050
0,149
TIES
16.62
0.112
0.024
0.088
StratQA
DARE
16.62
0.123
0.026
0,095
Anrow
13.83
0,218
0.066
0.175
LoraHub
11.50
0.171
0.038
0.128
MeteoRA (T1-1k)
8.74
0.235
0.065
0.221
MeteoRA (T2-1k)
10.03
0.240
0.068
0.226
MeteoRA (T2-100)
13.95
0,222
0.063
0.172
MeteoRA (T2-5)
20.69
0.228
0.067
0.174
PEFT (reference)
12.50
0.157
0.027
0.146
LoRA-F
28.39
0.153
0.025
0.142
LoRA-B
9.78
0.143
0.021
0.134
Avg Lora
1.21
0.099
0.010
0.060
TIES
22.45
0.101
0.011
0.078
TopChat
DARE
22.48
0.103
0.011
0.065
Arrow
11.16
0.099
0.013
0.080
LoraHub
0.35
0.064
0.005
0.051
MeteoRA (T1-1k)
13.44
0.151
0.025
0.141
MetcoRA (T2-1k)
12.35
0.149
0.025
0.140
MeteoRA (T2-100)
13.44
0.132
0.023
0.108
MeteoRA (T2-5)
12.93
0.135
0.024
0.110
19

---

Published as a conference paper at ICLR 2025
Table Experimental results for tasks using BLEU and ROUGE metrics (LlaMA3-8B base
model).
Task Name
Model
BLEU
ROUGE-1
ROUGE-2
ROUGE-L
PEFT (reference)
24.72
0.376
0.190
0.353
LoRA-F
31.47
0.284
0.123
0.267
LoRA-B
29.27
0.358
0.175
0.335
Avg LoRA
73.49
0.206
0.089
0,172
TIES
73.49
0.214
0.092
0,181
ALPACA
DARE
73.49
0.230
0.099
0,192
Arrow
12.26
0.222
0.093
0,186
LoraHub
0.00
0.176
0.068
0.151
MeteoRA (T1-1k)
32.34
0.358
0.170
0.335
MeteoRA (T2-1k)
30.08
0.354
0.170
0.332
MeteoRA (T2-100)
31.19
0.317
0.147
0,266
MeteoRA (T2-5)
80.34
0.249
0.103
0,204
PEFT (reference)
11.93
0.231
0.069
0.218
LoRA-F
16.13
0.248
0.080
0.233
LoRA-B
13.27
0,233
0.070
0,218
Avg LoRA
21.07
0.168
0,039
0,121
TIES
18.07
0.154
0.037
0.109
CNNDM
DARE
4.67
0.137
0,032
0.096
Arrow
13.13
0.153
0.037
0,111
LoraHub
15.30
0.087
0.008
0.038
MetcoRA (T1-1k)
11.93
0,233
0.070
0.218
MeteoRA (T2-1K)
11.93
0.232
0.070
0,219
MeteoRA (T2-100)
21.11
0.205
0.054
0.146
MeteoRA (T2-5)
6.52
0.203
0.054
0.143
PEFT (reference)
44.12
0.785
0.589
0.734
LoRA-F
36.89
0.718
0.488
0.666
LoRA-B
37.10
0.743
0.519
0.689
Avg LoRA
28.87
0.421
0.134
0.331
TIES
34.17
0.432
0.134
0.339
LingPuzz
DARE
56.23
0.357
0.113
0,281
Arrow
59.00
0.721
0.505
0.659
LoraHub
39.28
0.245
0.063
0.184
MeteoRA (T1-1k)
41.72
0.695
0.451
0.636
MeteoRA (T2-1k)
41.72
0.696
0.448
0.639
MeteoRA (T2-100)
50.81
0.666
0.408
0.588
MeteoRA (T2-5)
46.17
0.655
0.394
0.580
PEFT (reference)
97.65
LoRA-F
78.25
LoRA-B
78.25
Avg LoRA
63.56
TIES
46.47
NewsDE
DARE
36.60
Arrow
37.36
LoraHub
11.87
MeteoRA (T1-1k)
86.48
MeteoRA (T2-1k)
86.48
MeteoRA (T2-100)
51.42
MeteoRA (T2-5)
86.48
20

---

Published as a conference paper at ICLR 2025
Task Name
Model
BLEU
ROUGE-1
ROUGE-2 ROUGE-L
PEFT (reference)
81.54
LoRA-F
81.54
LoRA-B
81.54
Avg LoRA
31.18
TIES
30.55
NewsES
DARE
17.61
Anrow
31.82
LoraHub
0.0
MeteoRA (TI-1k)
81.54
MeteoRA (T2-1k)
81.54
MeteoRA (T2-100)
81.54
MeteoRA (T2-5)
63.72
PEFT (reference)
54.90
LoRA-F
54.90
LoRA-B
38.54
Avg LoRA
38.54
TIES
37.48
NewsIT
DARE
52.21
Arrow
38.02
LoraHub
0.0
MeteoRA (T1-1k)
54.90
MeteoRA (T2-1k)
51,83
MeteoRA (T2-100)
35.22
MeteoRA (T2-5)
36.78
PEFT (reference)
10.58
0.249
0.077
0.236
LoRA-F
10.44
0.234
0.068
0.223
LoRA-B
10.58
0.243
0.071
0.230
Avg LoRA
38.80
0.112
0.024
0.089
TIES
10.90
0.102
0.022
0.082
StratQA
DARE
14.78
0.128
0.027
0,100
Anrow
12.19
0.206
0.057
0.165
LoraHub
14.35
0.147
0.033
0.116
MeteoRA (T1-1k)
10.58
0.252
0.076
0.239
MeteoRA (T2-1k)
10.58
0.250
0.077
0.239
MeteoRA (T2-100)
20.56
0,228
0.065
0.174
MeteoRA (T2-5)
11.67
0.213
0.055
0.162
PEFT (reference)
39.50
0.151
0.025
0.141
LoRA-F
33.82
0.150
0.024
0.140
LoRA-B
19.22
0.139
0.019
0.131
Avg LoRA
23.59
0.094
0.012
0.078
TIES
26.13
0.092
0.011
0.077
TopChat
DARE
38.31
0.086
0.008
0.066
Arrow
35.64
0.112
0.016
0.091
LoraHub
0.08
0.049
0.002
0.031
MeteoRA (T1-1k)
45.64
0.152
0.026
0.141
MetcoRA (T2-1k)
45.64
0.152
0.024
0.141
MeteoRA (T2-100)
27.36
0.129
0.021
0.107
MeteoRA (T2-5)
40.86
0.130
0.018
0.109
21

---

Published as a conference paper at ICLR 2025
A.5 Composite-n EVALUATION RESULTS DETAILS
The task construction method for the composite-n series is similar across different sets. Taking composite-10 an example, each sample in this test can be thought of "test sheet" containing 10 questions presented in sequence. During evaluation, this test sheet is provided as input to the LLM and ask to output the answers along with the corresponding question numbers in order. To ensure that the model is capable of answering these 10 questions, we select 10 tasks from the 28 selected tasks, ensuring diversity in knowledge domains and question formats. Each sample in the composite- 10 task is constructed by randomly sampling one instance from each of the 10 tasks (without repetition) and concatenating them sequence. However, given the limited capability the instruction following in the zero-shot setting, neither the MeteoRA models nor the models fine-tuned by LoRA achieve satisfactory results. Hence, we employ a 2-shot setting for evaluation on these composite-n tasks.
The evaluation metrics used for composite-n tasks are: average number of questions attempted, average number of multiple-choice questions answered correctly, and average BLEU, ROUGE scores for non-mullipleechoice questions.
Notice that in the composite-n tasks, when calculating the softmax values of the weights for the two LoRA adapters selected by the Gating network, we introduced a hyperparameter called temperature. The value of temperature needs be increased as the number of sub-tasks grows. Specifically, set the temperature values to 15, 20, and 30 for the three tasks, respectively.
Tables 9 and 10 present the detailed evaluation results for the composite-3, composite-5, and composite-10 tasks, respectively. Several important clarifications are necessary for interpreting these results:
1. The models required to generate both the corresponding question number and its answer. Any mismatch between the question number and the answer is therefore considered incorrect.
2. In the evaluation results, some BLEU scores are recorded as This occurs when the model generates mismatched question numbers and answers or provides extremely insufficient answers, resulting in an overall BLEU score.
3. For the task strategyga, which involves answering with either 'yes' or 'no' and providing reasoning steps, the accuracy metric specifically measures the correctness of the 'yes' or "no" response.
4. The reported ROUGE scores refer to the Fl-scores.
5. Samples that the lengths exceed 4,096 tokens are skipped in the evaluation process (we skip 13 samples total).
Table 8: The composite-3 evaluation results are presented in details with MeteoRA results on the left side and LoRA-B results on the right side of each metric column. dash ('-)) indicates that the corresponding metric was not applicable or included in the evaluation.
Sub-task Name
Accuracy!
BLEU
ROUGE-I1
ROUGE-21
ROUGE-LA
LogDeduc
0.5001
0.430
QuestSel
0.545]
0.630
StratQA
0.445t
0.250
15.31
10.55
0.195t
0.135
0.0521
0.027
0.1821
0.128
22

---

Published as a conference paper at ICLR 2025
Table 9: The composite-5 evaluation results are presented details with MeteoRA results on the left side and LoRA-B results on the right side of each metric column. A dash ('-') indicates that the corresponding metric was not applicable or included in the evaluation.
Sub-task Name
Accuracyt
BLEU
ROUGE-11
ROUGE-21
ROUGE-LA
LogDedue
0.500
0.500
QuestSel
0.620
0.770
AbsNarr
0.350
0.460
GSWiki
0.6501
0.410
StraiQA
0.4951
0.275
9.861
9.41 0.2211 0.219 0.0697 0.063 0.207] 0.208
Table 10: The composite-10 evaluation results are presented in details with MeteoRA results on the left side and LoRA-B results on the right side of each metric column, A dash ('-') indicates that the corresponding metric was not applicable included in the evaluation. Note that the 0.00 BLEU scores are caused by mismatch and too insufficient answers.
Sub-task Name
Accuracy!
BLEUt
ROUGE-I1
ROUGE-21
ROUGE-L
LogDeduc
0.5001
0.453
QuestSel
0.7031
0.688
AbsNarr
0.625
0.672
GSWiki
0.773
0.727
WinoWhy
0.422f
0.078
StratQA
0.4611
0.211
3,231
0.00
0,2251
0.106
0.0511
0.025
0,2107
0.099
DisAQQ
0.266t
0.117
NewsDE
14.781
14.54
ALPACA
0.000
8.17
0.2571
0.187
0.075
0.075
0.2411
0.167
LingPuzz
17.371
12.14
0.2331
0.189
0.0521
0.030
0.1761
0.103
A.6 FULL CONTENT OF THE ILLUSTARTED composite-n TASK
Below is the full content of composite-3 task, which includes 2-shot examples, the input (3 problems), and the model generated output, shown in Figure
Example 1 1 [ The following paragraphs each describe set of five objects arranged in fixed order. The statements are logically consistent within each paragraph. \n nnn a branch, there are five birds: cardinal, crow, falcon, robin, and blue jay. The robin cardinal. The cardinal is to the right of the blue jay. The blue jay is the second from the left. The crow the rightmost. Which choice is correct?' inchoiee: The cardinal is the leftmost, ncchiie:: The crow is the leftmost. ncchice: The falcon is the leftmost. \ nchoice: The robin is the leftmost |nchoice: The blue jay the leftmost. n n' n
 Given a short answer along with its context, select the most appropriate question which has the given short answer as its answer. \ n Here the short answer followed by the context. nnShort Answer: magazines and journals Contexti Tesla wrote number of books and articles for magazines and journals. Among his books are My Inventions: The Autobiography of Nikola Tesla, compiled and edited by Ben Johnston; The Fantastic Inventions of Nikola Tesla, compiled and edited by David Hatcher Childress; and The Tesla Papers. h.t.ss.,cchiiii ,nchoic:: Who was the first to post tesla's writings?! nchoice: ncchings? Who was in charge of editing tesla's autobiography nhhoose the appropriate question which has the given answer. [[NNST n[NST] Q: Can Hulk's s alter ego explain atomic events? ?' nA: 
 n', The falcon INST]Ann is the Jeftmost. n2. Who was the first post tesla's 's writings? n3. Yes. Hulk's alter ego Dr. Robert Bruce Banner. Dr. Robert Bruce Banner is a nuclear physicist. Nuclear physics is the field of physics that studies atomic nuclei and their constituents and interactions. \n), EExample 2 [nnNS[I The following paragraphs each describe set of three objects arranged in fixed order. The statements are logically consistent within each paragraph. \, nnn,nIn nIn a golf tournament, there were three golfers: Ana, Rob, and Joe. Joe finished above Ana. Rob finished above Joe. Which choice is correct?' nchoice: ncchocc? Ana finished second. nnchiie: Rob finished second. nccoice: Joe finished second. /n/INST]] n n
 Given short answer along with its context, select the most appropriate question which has the given short answer as its answer. (n' nHere is the short answer followed by the context. nhoortt Answer: months Context: It was until January 1518 that
the right of the
nl.
23

---

Published as a conference paper at ICLR 2025
friends of Luther translated the 95 Theses from Latin into German and printed and widely copied them, making the controversy one of the first in history to be aided by the printing press. Within two weeks, copies of the theses had spread throughout Germany; within two months, they had spread throughout Europe. nncoiie:: How long did the theses take to spread through europe?', i choice: How long did it take for the printing of the theses to spread through germany?' nChoose nnChooe the appropriate question which has the given answer. 
 n |
 Q: Can Iowa be hidden in the English Channel?'
.nA:: nA: 
 n nl. Joe finished second. n2. How long did the theses take spread through europe? ? n3. No. The maximum width of the English Channel is 150 miles. The minimum width of Lowa is 200 miles. |n . nFollowing the example above, generate answers to the questions below: | n
 The following paragraphs each describe set of seven objects
arranged a fixed order. The statements are logically consistent within each paragraph. nn'n branch, there are seven birds: cardinal, blue jay, robin, falcon, quail, hummingbird, and raven. The falcon is to the right of the quail. The robin is to the right of the hummingbird. The raven is to the left of the quail. The blue jay is the second from the left. The robin is the third from the left. The cardinal is the third from the right. Which choice is correct?' nchoice: The cardinal is the second from the right. nccoice: The blue jay is the second from the right. Ancooice: The robin is the second from the right. \ nchoice: The falcon is the second from the right. ) nchoice: The quail is the second from the right. nchoice: The hummingbird is the second from the right. Anchoic: The raven is the second from the right. in 
 n n [[[NST] Given short answer along with its context, select the most appropriate question which has the given short answer as its answer. \ Here is the short answer followed by the context. nnhortt Answer: Ghazan Khan Context: The invasions of Baghdad, Samarkand, Urgench, Kiev, Vladimir among others caused mass murders, such as when portions of southern Khuzestan were completely destroyed. His descendant Hulagu Khan destroyed much of Iran's northern part and sacked Baghdad although his forces were halted by the Mamluks of Egypt, but Hulagu's descendant Ghazan Khan would return to beat the Egyptian Mamluks right out of Levant, Palestine and even Gaza. According to the works of the Persian
correct?)nchoice:
historian Rashid-al-Din Hamadani, the Mongols killed more than 70,000 people in Merv and more than 190,000 in Nishapur. In 1237 Batu Khan, a grandson of Genghis Khan, launched an invasion into Kievan Rus". Over the course of three years, the Mongols destroyed and annihilated all of the major cities of Eastern Europe with the exceptions of Novgorod and Pskov. An choice: Which genghis khan descendant sacked baghdad?', n choice: Which of eastern europe's 's big cities survived the mongol invasion?' n choice: Which of genghis khan's 's descendants was responsible for driving the mamluks from palestine?,,nChonn
the appropriate question which has the given answer. // INST] n' [[INST) Q: Could the main character of " "Alice's Adventures in Wonderland" join a Masonic Lodge? nA: I/INST]'n nll The quail is the second from the right. n2. Which of genghis khan's 's descendants was responsible for driving the mamluks from palestine?\ n3. No. The main character of " "Alice's Adventures Wonderland" Alice. Women are not allowed to join Masonic Lodges.
regical-deducttinn
guestion
strategyqa
other task
24

---

Published as a conference paper at ICLR 2025
A.7 EFFICIENCY EVALUATION EXPERIMENTS ON DIFFERENT METEORA FORWARD PASS IMPLEMENTATIONS
In addition experiments pass design using randomly-generated pseudo data across various settings, including batch size (b), sequence length (s), gating weights top-k (k), LoRA rank size (r), number of LoRAs (l), maximum tokens to generate (g), input hidden dimension (h), and output hidden dimension (hout), Moreover, here we introduce new baseline, toop-speedu,, which improves upon loop-original by removing redundant or inefficient operations directly, acting like strong substitute for the original design.
28 selected tasks, we assess the efficiency of MeteoRA forward
As depicted in Figures / for memory efficiency and | for time efficiency, our bmm-torch design outperforms other implementations, boasting average speedup of ~ over loop-original. How- ever, its memory usage escalates with longer sequence lengths. contrast, bmm-triton maintains comparable memory footprint to the basclines while retaining 80% of the speedup achieved by bmm-ttrch, showcasing a balanced trade-off between time and space, as illustrated in Figure 6 for overall efficiency.
25

---

Published as a conference paper at ICLR 2025
Ivend aby reciece stedeatee
Jours
Inee eny red 000g33
Overall Efficiency Part-l
Sueeral skyf moal
argiters
2006
2000
2008
(b) Overall Efficiency Part-2
Figure 6: The overall efficiency evaluation curve displays the averaging runtime x memory footprint for each newly generated token (unit: ms GB / token).
26

---

Published as a conference paper at ICLR 2025
www.oiieeettii 25.22
Iran elt moselr meaile
Servoces
6688
Inatrsftyl E1 boutt110
010.8
Memory Efficiency Part-1
(neeudiy)) www.
poul
022
030
032
7.000
04
280
2013
(b) Memory Efficiency Part-2
Figure 7: The memory efficiency evaluation curve displays the averaging memory footprint for each newly generated token (unit: GB / token).
27

---

Published as a conference paper at ICLR 2025
12
03
255
aa) Time Efficiency Part-1
108
1000
(b) Time Efficiency Part-2
Figure The time efficiency evaluation curve displays averaging runtime for each newly generated token (unit: ms / token).
28


Reviews:
[{"review_id": "Koc5V0hNKN", "reviewer": "Reviewer_oZTQ", "summary": "This paper introduces MeteoRA, which automatically applies the appropriate LoRA adapters to a pre-trained LLM based on the current task.\nMeteoRA is an MoE-inspired approach in which a gating function selects the top-k LoRA adapters for each token in each input sequence.\nThe authors demonstrate that MeteoRA performs similarly to an LLM with a handpicked, in-domain LoRA adapter, and provide an efficient\nkernel implementation that addresses runtime concerns and memory overhead.", "questions": "**Questions**\n1. How does MeteoRA perform on out-of-distribution tasks (e.g., compared to baselines such as the base LLM, LoRA-F, and LoRA-B)?\n2. In section 3.3, what is $p_i$?\n3. For measuring forward-pass speed, what is the batch size?\n\n**Suggestions**\n1. Introduction should quantify benefit of MeteoRA beyond speedup (e.g., average accuracy increase).\n2. Fig 1: unclear where the MoE is located, and how experts are selected.\n3. Background: should cite other MoE-based LLMs, such as GLaM (preceded Mixtral), DBRX, and Grok.\n4. Section 3.3 needs more revisions for clarity. While I appreciate the explanation to motivate the kernel design, it took several reads to fully understand the problem with the `loop-original` method, why it is 10x slower, and how `bmm-torch` works.\n5. Figure 8 is hard to interpret. The font size is small and the colors/lines are difficult to distinguish due to small line width and shading. Such a key figure should be better-presented (i.e., bigger, clearer lines, easier to read).\n6. Figure 5: root-of-runtime is a strange (and potentially misnamed) evaluation metric. It would be better to report runtime directly.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. MeteoRA is a general approach to incorporate domain-specific knowledge from multiple LoRAs in a single model.\n2. Extensive evaluation which demonstrates that MeteoRA performs similarly to the PEFT reference implementation, which provides a reasonable upper-bound reference.\n3. The authors explain concerns about runtime and memory-efficiency. Based on this, the authors design, implement, and evaluate a CUDA kernel which addresses the concerns.", "weaknesses": "1. All LoRAs are stored in GPU memory, which limits the scalability of the approach. In contrast, S-LoRA (a LoRA serving system) scales to thousands of LoRA adapters by swapping LoRA weights to host memory. Proposing a target range for the # of LoRA adapters or a method to swap adapters to host memory could help address this concern.\n2. MeteoRA model is fine-tuned on a set of LoRAs and their target domains. Consequently, the approach does not efficiently integrate new LoRA adapters.\n3. Capability regression with $k=2$ indicates that LoRA adapters likely interfere with one another. Some discussion of how to mitigate interference or exploit $k=1$ for further speedups could address this weakness.\n4. On a few tasks, MeteoRA performs worse than the baselines (e.g., NewsIT, CNNDM, and TrackObj). An explanation of why this might be the case could help contextualize these results.\n5. No evaluation on how MeteoRA scales to larger batch sizes. It would be interesting to see the relationship between batch size and runtime/memory because larger batch sizes would access more adapters which could impact these metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VGLwEIpSTA", "reviewer": "Reviewer_v7tD", "summary": "This paper presents MeteoRA, a framework combining MoE and LoRA to enhance inference efficiency via forward acceleration. It analyzes PEFT methods related to user tasks, integrating multiple LoRA adapters for new tokens and identifying the top-k experts for processing. The authors introduce a batched matrix multiplication (bmm-torch) strategy to enable parallel processing of LoRAs, improving speed and efficiency over sequential methods. In summary, by merging MoE and bmm-torch, MeteoRA significantly accelerates token processing and enhances operational efficiency.", "questions": "The b$\\times$s tokens are treated as independent. However, there is concern about potential correlations among tokens. Knowledge across domains can be interrelated, and sentence meaning may depend on context. How do the authors address this issue? Could the assumption of independence negatively affect performance by ignoring relevant interdependencies?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "MeteoRA effectively implements a scalable integration of LoRA while adopting forward acceleration techniques during the inference phase, thereby enhancing the efficiency of the inference process.", "weaknesses": "- The paper is not very novel, given that using MoE for LoRA is an idea that has already been extensively explored [1,2,3]. It would be beneficial to clearly delineate how MeteoRA compares to and differs from the referenced LoRAMoE works. \n\n- The term \"reuse existing LoRA\" is misleading and unclear; it implies the need for offline training and does not introduce any innovation compared to other MoE methods.\n\n- While the bmm-torch method for parallel processing of LoRA adapters improves forward training, it may increase memory consumption. This approach requires larger memory allocations for concurrent processing, potentially offsetting time savings from reduced sequential processing. Please provide quantitative comparisons of memory usage and speed gains across different batch sizes or sequence lengths to clarify this trade-off.\n\n\n- MeteoRA is presented as an advancement over existing LoRA techniques, a direct comparison with LoRA MoE methods is missing. Such a comparison could underscore the performance superiority of the proposed method. Could the authors conduct and present a detailed comparative analysis with LoRA MoE [1,2,3] methods?\n\nMinior: \n\n-The font size in Figure 3 is too small to read.\n\nReferences:\n\n[1] When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications, SIGIR 2024.\n\n[2] Mixture of LoRA Experts, ICLR 2024.\n\n[3] Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qAIAczVt0T", "reviewer": "Reviewer_dk7r", "summary": "The paper introduces MeteoRA, a framework designed to enhance the deployment of multiple LoRA adapters in LLM through a Mixture-of-Experts (MoE) architecture. This approach aims to facilitate autonomous task sensing and dynamic adapter switching, improving efficiency in handling composite tasks.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The use of a full-mode MoE architecture to integrate multiple LoRA adapters is a novel contribution, potentially addressing limitations in existing methods like Huggingface PEFT and S-LoRA.\n- The proposed forward acceleration strategies address efficiency challenges in traditional MoE implementations, achieving significant speedups.", "weaknesses": "- It will be better to also compare with a model trained with MoE upcycling and discuss the benefit of the proposed method.\n- It should be a more detailed analysis of the triton operator, how it differ from methods like S-LoRA.\n- The legend in Figure 3 is too small", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x6RizNnGZ4", "reviewer": "Reviewer_4FYx", "summary": "This paper proposes MeteoRA to enable scalable multi-task LoRA embedding within LLMs. The key of MeteoRA is using gated MoE to automatically select the most pertinent LoRA adapters to generate appropriate responses. It also employs efficient GPU kernel operators for forward acceleration. Evaluation results show the effectiveness of MeteoRA.", "questions": "How to reduce the re-training costs when some LoRA adapters are updated? How many LoRA adapters can be supported by MeteoRA?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Using gated MoE for automatic selection of LoRA adapters\n+ Employing efficient GPU kernel operators for forward acceleration", "weaknesses": "- Re-training is required when some LoRA adapters are updated, making it hard to use in practice", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cZxKFjARi1", "reviewer": "Reviewer_4Xew", "summary": "The authors propose an automatic selection model for multi-task LoRA using a MoE arch, supporting both top-1 and top-k selection methods. In addition to constructing an automatic selection process using Gate Logits, they also utilize PyTorch's bmm operator for forward acceleration. The authors demonstrate the effectiveness of their proposed algorithm across multiple tasks by comparing it with existing multi-task LoRA methods as well as their own baselines, LoRA-F and LoRA-B.", "questions": "1. In the appendix, the authors mention, \"However, given the limited capability of the instruction following in the zero-shot setting, neither the MeteoRA models nor the models fine-tuned by LoRA achieve satisfactory results.\" Is there any supporting evidence for this statement? Also, why did the authors ultimately choose a 2-shot setting?\n2. According to Figure 3, MeteoRA's performance on the ParaSeg task is noticeably poor when using LlaMA2, but it shows significant improvement with LlaMA3. Could the authors provide an explanation for this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Originality\nSince the gate logits approach is primarily inspired by methods from Mixtral of Experts, the main novelty lies in the design of the loss function and the implementation of forward acceleration. This method can solve up to ten sequential problems in a single inference pass automatically, which demonstrates both the scalability and utility of the proposed solution.\n\n### Quality\nThe technical details appear correct, as the paper does not involve extensive mathematical derivations. The authors also introduce efficient acceleration strategies for MoE, which contribute meaningfully to improving the computational feasibility of such models.\n\n### Clarity\nThe paper is generally clear in presenting the proposed methodology and results.\n\n### Significance\nThe work addresses a significant challenge in the field of parameter-efficient fine-tuning (PEFT) by proposing a novel approach to autonomously manage and switch between multiple LoRA adapters embedded within a single LLM. The proposed framework is of considerable practical value for large-scale language models and downstream applications. The emphasis on practical deployment of the framework and the specific use of LoRA in a multi-task setting distinguishes this work from previous studies.", "weaknesses": "### Originality\nThe gate logits approach is primarily inspired by methods from Mixtral of Experts, and the paper could more clearly emphasize its origin and what it is (just a linear layer with softmax).\n\n### Clarity\nThe description of the gating network is insufficient. More details on the construction of composite-3/5/10 tasks should also be provided in the main text. More motivations should be included.\n\n### Writing Quality\nThe LaTeX formatting should be unified, such as using consistent notation for $o_{base}$ and $W_\\mathrm{base}$ in Equation (2). Additionally, consistent notation should be used in the formulas (e.g., whether the vectors are row or column vectors, and whether matrices operate on vectors from the left or the right). For instance, in Equation (2), the expression $o = xW_\\mathrm{base} + x\\Delta W_{I(x)}$ is used, while in Appendix A, $h = W_\\mathrm{base}x + B_iA_ix$ is used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)