MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models

Authors: Jingwei Xu, Junyu Lai, Yunpeng Huang
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, LoRA

Abstract:
The pretrain+fine-tune paradigm is foundational for deploying large language models (LLMs) across various downstream applications. Within this framework, Low-Rank Adaptation (LoRA) stands out for its parameter-efficient fine-tuning (PEFT), producing numerous reusable task-specific LoRA adapters. However, this approach requires explicit task intention selection, posing challenges for autonomous task sensing and switching during inference with multiple existing LoRA adapters embedded in a single LLM. In this work, we introduce MeteoRA (Multiple-Tasks embedded LoRA), a scalable and efficient framework that reuses multiple task-specific LoRA adapters into the base LLM via a full-mode Mixture-of-Experts (MoE) architecture. This framework also includes novel MoE forward acceleration strategies to address the efficiency challenges of traditional MoE implementations. Our evaluation, using the LlaMA2-13B and LlaMA3-8B base models equipped with 28 existing LoRA adapters through MeteoRA, demonstrates equivalent performance with the traditional PEFT method. Moreover, the LLM equipped with MeteoRA achieves superior performance in handling composite tasks, effectively solving ten sequential problems in a single inference pass, thereby demonstrating the framework's enhanced capability for timely adapter switching.

Decision: accept (poster)
Reviewer scores: 8, 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730294200307

thread_type:
official_review

<thread_object>
creator: reviewer_4xew

<content>
summary:
The authors propose an automatic selection model for multi-task LoRA using a MoE arch, supporting both top-1 and top-k selection methods. In addition to constructing an automatic selection process using Gate Logits, they also utilize PyTorch's bmm operator for forward acceleration. The authors demonstrate the effectiveness of their proposed algorithm across multiple tasks by comparing it with existing multi-task LoRA methods as well as their own baselines, LoRA-F and LoRA-B.

strengths:
### Originality
Since the gate logits approach is primarily inspired by methods from Mixtral of Experts, the main novelty lies in the design of the loss function and the implementation of forward acceleration. This method can solve up to ten sequential problems in a single inference pass automatically, which demonstrates both the scalability and utility of the proposed solution.

### Quality
The technical details appear correct, as the paper does not involve extensive mathematical derivations. The authors also introduce efficient acceleration strategies for MoE, which contribute meaningfully to improving the computational feasibility of such models.

### Clarity
The paper is generally clear in presenting the proposed methodology and results.

### Significance
The work addresses a significant challenge in the field of parameter-efficient fine-tuning (PEFT) by proposing a novel approach to autonomously manage and switch between multiple LoRA adapters embedded within a single LLM. The proposed framework is of considerable practical value for large-scale language models and downstream applications. The emphasis on practical deployment of the framework and the specific use of LoRA in a multi-task setting distinguishes this work from previous studies.

weaknesses:
### Originality
The gate logits approach is primarily inspired by methods from Mixtral of Experts, and the paper could more clearly emphasize its origin and what it is (just a linear layer with softmax).

### Clarity
The description of the gating network is insufficient. More details on the construction of composite-3/5/10 tasks should also be provided in the main text. More motivations should be included.

### Writing Quality
The LaTeX formatting should be unified, such as using consistent notation for $o_{base}$ and $W_\mathrm{base}$ in Equation (2). Additionally, consistent notation should be used in the formulas (e.g., whether the vectors are row or column vectors, and whether matrices operate on vectors from the left or the right). For instance, in Equation (2), the expression $o = xW_\mathrm{base} + x\Delta W_{I(x)}$ is used, while in Appendix A, $h = W_\mathrm{base}x + B_iA_ix$ is used.

questions:
1. In the appendix, the authors mention, "However, given the limited capability of the instruction following in the zero-shot setting, neither the MeteoRA models nor the models fine-tuned by LoRA achieve satisfactory results." Is there any supporting evidence for this statement? Also, why did the authors ultimately choose a 2-shot setting?
2. According to Figure 3, MeteoRA's performance on the ParaSeg task is noticeably poor when using LlaMA2, but it shows significant improvement with LlaMA3. Could the authors provide an explanation for this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730451644752

thread_type:
official_review

<thread_object>
creator: reviewer_4fyx

<content>
summary:
This paper proposes MeteoRA to enable scalable multi-task LoRA embedding within LLMs. The key of MeteoRA is using gated MoE to automatically select the most pertinent LoRA adapters to generate appropriate responses. It also employs efficient GPU kernel operators for forward acceleration. Evaluation results show the effectiveness of MeteoRA.

strengths:
+ Using gated MoE for automatic selection of LoRA adapters
+ Employing efficient GPU kernel operators for forward acceleration

weaknesses:
- Re-training is required when some LoRA adapters are updated, making it hard to use in practice

questions:
How to reduce the re-training costs when some LoRA adapters are updated? How many LoRA adapters can be supported by MeteoRA?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730454599231

thread_type:
official_review

<thread_object>
creator: reviewer_dk7r

<content>
summary:
The paper introduces MeteoRA, a framework designed to enhance the deployment of multiple LoRA adapters in LLM through a Mixture-of-Experts (MoE) architecture. This approach aims to facilitate autonomous task sensing and dynamic adapter switching, improving efficiency in handling composite tasks.

strengths:
- The use of a full-mode MoE architecture to integrate multiple LoRA adapters is a novel contribution, potentially addressing limitations in existing methods like Huggingface PEFT and S-LoRA.
- The proposed forward acceleration strategies address efficiency challenges in traditional MoE implementations, achieving significant speedups.

weaknesses:
- It will be better to also compare with a model trained with MoE upcycling and discuss the benefit of the proposed method.
- It should be a more detailed analysis of the triton operator, how it differ from methods like S-LoRA.
- The legend in Figure 3 is too small

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645023504

thread_type:
official_review

<thread_object>
creator: reviewer_v7td

<content>
summary:
This paper presents MeteoRA, a framework combining MoE and LoRA to enhance inference efficiency via forward acceleration. It analyzes PEFT methods related to user tasks, integrating multiple LoRA adapters for new tokens and identifying the top-k experts for processing. The authors introduce a batched matrix multiplication (bmm-torch) strategy to enable parallel processing of LoRAs, improving speed and efficiency over sequential methods. In summary, by merging MoE and bmm-torch, MeteoRA significantly accelerates token processing and enhances operational efficiency.

strengths:
MeteoRA effectively implements a scalable integration of LoRA while adopting forward acceleration techniques during the inference phase, thereby enhancing the efficiency of the inference process.

weaknesses:
- The paper is not very novel, given that using MoE for LoRA is an idea that has already been extensively explored [1,2,3]. It would be beneficial to clearly delineate how MeteoRA compares to and differs from the referenced LoRAMoE works. 

- The term "reuse existing LoRA" is misleading and unclear; it implies the need for offline training and does not introduce any innovation compared to other MoE methods.

- While the bmm-torch method for parallel processing of LoRA adapters improves forward training, it may increase memory consumption. This approach requires larger memory allocations for concurrent processing, potentially offsetting time savings from reduced sequential processing. Please provide quantitative comparisons of memory usage and speed gains across different batch sizes or sequence lengths to clarify this trade-off.


- MeteoRA is presented as an advancement over existing LoRA techniques, a direct comparison with LoRA MoE methods is missing. Such a comparison could underscore the performance superiority of the proposed method. Could the authors conduct and present a detailed comparative analysis with LoRA MoE [1,2,3] methods?

Minior: 

-The font size in Figure 3 is too small to read.

References:

[1] When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications, SIGIR 2024.

[2] Mixture of LoRA Experts, ICLR 2024.

[3] Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. ICLR 2024.

questions:
The b$\times$s tokens are treated as independent. However, there is concern about potential correlations among tokens. Knowledge across domains can be interrelated, and sentence meaning may depend on context. How do the authors address this issue? Could the assumption of independence negatively affect performance by ignoring relevant interdependencies?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709279532

thread_type:
official_review

<thread_object>
creator: reviewer_oztq

<content>
summary:
This paper introduces MeteoRA, which automatically applies the appropriate LoRA adapters to a pre-trained LLM based on the current task.
MeteoRA is an MoE-inspired approach in which a gating function selects the top-k LoRA adapters for each token in each input sequence.
The authors demonstrate that MeteoRA performs similarly to an LLM with a handpicked, in-domain LoRA adapter, and provide an efficient
kernel implementation that addresses runtime concerns and memory overhead.

strengths:
1. MeteoRA is a general approach to incorporate domain-specific knowledge from multiple LoRAs in a single model.
2. Extensive evaluation which demonstrates that MeteoRA performs similarly to the PEFT reference implementation, which provides a reasonable upper-bound reference.
3. The authors explain concerns about runtime and memory-efficiency. Based on this, the authors design, implement, and evaluate a CUDA kernel which addresses the concerns.

weaknesses:
1. All LoRAs are stored in GPU memory, which limits the scalability of the approach. In contrast, S-LoRA (a LoRA serving system) scales to thousands of LoRA adapters by swapping LoRA weights to host memory. Proposing a target range for the # of LoRA adapters or a method to swap adapters to host memory could help address this concern.
2. MeteoRA model is fine-tuned on a set of LoRAs and their target domains. Consequently, the approach does not efficiently integrate new LoRA adapters.
3. Capability regression with $k=2$ indicates that LoRA adapters likely interfere with one another. Some discussion of how to mitigate interference or exploit $k=1$ for further speedups could address this weakness.
4. On a few tasks, MeteoRA performs worse than the baselines (e.g., NewsIT, CNNDM, and TrackObj). An explanation of why this might be the case could help contextualize these results.
5. No evaluation on how MeteoRA scales to larger batch sizes. It would be interesting to see the relationship between batch size and runtime/memory because larger batch sizes would access more adapters which could impact these metrics.

questions:
**Questions**
1. How does MeteoRA perform on out-of-distribution tasks (e.g., compared to baselines such as the base LLM, LoRA-F, and LoRA-B)?
2. In section 3.3, what is $p_i$?
3. For measuring forward-pass speed, what is the batch size?

**Suggestions**
1. Introduction should quantify benefit of MeteoRA beyond speedup (e.g., average accuracy increase).
2. Fig 1: unclear where the MoE is located, and how experts are selected.
3. Background: should cite other MoE-based LLMs, such as GLaM (preceded Mixtral), DBRX, and Grok.
4. Section 3.3 needs more revisions for clarity. While I appreciate the explanation to motivate the kernel design, it took several reads to fully understand the problem with the `loop-original` method, why it is 10x slower, and how `bmm-torch` works.
5. Figure 8 is hard to interpret. The font size is small and the colors/lines are difficult to distinguish due to small line width and shading. Such a key figure should be better-presented (i.e., bigger, clearer lines, easier to read).
6. Figure 5: root-of-runtime is a strange (and potentially misnamed) evaluation metric. It would be better to report runtime directly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732722658385

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely appreciate the valuable suggestions provided by all the reviewers. We have revised the paper based on the suggestions and have submitted an updated version. We would be grateful to address any further questions or engage in any additional discussions, if needed.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a framework to incorporate domain-specific knowledge from multiple LoRAs in a single model, integrating multiple LoRA adapters for new tokens and identifying the top-k experts for processing.  A batched matrix multiplication (bmm-torch) strategy accelerates token processing. An evaluation using the LlaMA2-13B and LlaMA3-8B base models equipped with 28 existing LoRA adapters is shown to be performant compared to fine tuning, particularly in handling composite tasks.

Reviewers found the work sufficiently different from prior art, and were impressed by the set of experiments, all the way to an implementation of a CUDA kernel. Issues regarding scaling to multiple GPUs were brought up, and are left as future work. One reviewer remained concerned regarding novelty, but others were not as concerned.