LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild (2024)

Abstract: Low-Rank Adaptation (LoRA) provides an effective yet efficient solution for fine-tuning large language models (LLMs). The modular and plug-and-play nature of LoRA enables the integration of diverse domain-specific LoRAs to enhance the capabilities of LLMs. Previous research on exploiting multiple LoRAs either focuses on specific isolated downstream tasks or fixes the selection of LoRAs during training. However, in real-world scenarios, LLMs receive diverse prompts covering different tasks, and the pool of candidate LoRAs is often dynamically updated. To bridge this gap, we propose LoraRetriever, a retrieve-then-compose framework that adaptively retrieves and composes multiple LoRAs according to the input prompts. LoraRetriever contains three main components: firstly, identifying and retrieving LoRAs relevant to the given input; secondly, formulating strategies for effectively integrating the retrieved LoRAs; and thirdly, developing efficient batch inference to accommodate heterogeneous requests. Experimental results indicate that LoraRetriever consistently outperforms the baselines, highlighting its practical effectiveness and versatility. Our code is available at https://github.com/StyxXuan/LoraRetriever.

heading: 1 Introduction
text: Recently, large language models (LLMs) such as ChatGPT (Liu et al., 2023b) and Llama (Touvron et al., 2023) have shown notable successes in a range of fields (Hadi et al., 2023; Wang et al., 2023; Wu et al., 2023). Nevertheless, due to the prohibitively high computation costs for finetuning LLMs on specific domains, there is a growing shift towards Parameter-Efficient Fine-Tuning (PEFT) (Liu et al., 2022; Hu et al., 2023, 2021; Zhao et al., 2024), which only updates a small fraction of the model’s parameters or integrates new
∗Corresponding authors.
trainable parameters that augment the model’s capabilities. Within this sphere, Low-Rank Adaptation (LoRA) (Hu et al., 2021) stands out for its remarkable effectiveness, modularity, and plug-and-play capabilities. As AI communities like Huggingface and ModelScope witness an influx of LoRA parameters tailored for diverse tasks and domains, there is an increasing emphasis on employing various LoRA experts to provide a comprehensive service (Sheng et al., 2023).
Recent research has explored the integration of the mixture of expert (MoE; Jacobs et al. (1991); Jordan and Jacobs (1994)) with LoRAs (Wang et al., 2022; Liu et al., 2023a; Zadouri et al., 2023; Muqeeth et al., 2023; Anonymous, 2024). However, these methods lock in the selection of LoRAs while training, lacking the ability to dynamically update and scale in scenarios where the LoRA pool may consistently expand. LoRAhub (Huang
4447
et al., 2023) and AdapterSoup (Chronopoulou et al., 2023) explore composing LoRAs for specific downstream tasks. However, these two methods offer a one-size-fits-all solution for downstream tasks, overlooking the heterogeneous nature of diverse real-world requests.
To bridge these gaps, our paper explores the "mixed-task scenario" as exemplified by platforms like ChatGPT (Liu et al., 2023b) and Gemini (Team et al., 2023), wherein the nature of user requests encompasses diverse prompts covering different tasks. While LLMs present a unified solution for a broad spectrum of tasks, their performance can still falter in certain specialized areas(Liu et al., 2023a; Yang et al., 2023; Chai et al., 2024). This is where the integration of LoRAs becomes crucial. As shown in Fig.1, our vision encompasses a multi-LoRA serving framework capable of dynamic enhancement, continuously improving its functionality as new LoRA modules are added and updated. Through the plug-and-play capabilities of LoRA, the framework can provide personalized services for heterogeneous downstream requests.
In this paper, we introduce LoraRetriever, a retrieve-then-compose framework designed to exploit the plug-and-play nature of LoRA in mixedtask scenarios. Our framework consists of three key components: (1) Input-aware LoRA Retrieval: The first step of our framework is aligning user inputs with the corresponding LoRAs through sentence embeddings and is further refined by an instruction fine-tuning (Su et al., 2022; Asai et al., 2022) for effective LoRA retrieval. Through the retriever, we achieve a more flexible LoRA routing mechanism, whose training stage is disentangled from the training and inference of the LLM. (2) LoRA Composition: Our framework next employs two strategies for compositing the retrieved LoRAs in the first step. The Fusion of LoRAs averages multiple LoRAs’ parameters and constructs a singular comprehensive model for each input. The Mixture of LoRAs activates multiple LoRAs simultaneously and then averages the output of each submodule of the LoRAs. Compositing topk LoRAs increases the recall rate for the correct LoRA and improves the generalization of unseen tasks by integrating the LoRAs of similar tasks. (3) Batch Inference of Multiple LoRAs: Most previous work on the input-adaptive inference of LLMs does not support batch inference (Zhou et al., 2020; Chronopoulou et al., 2023). To tackle the challenge
of heterogeneous batched requests, we construct a unique LoRA mapping matrix for batch samples. This allows for tailored inferences through efficient matrix multiplication, ensuring each request activates its corresponding LoRAs while maintaining batch processing efficiency.
To assess the performance of LoraRetriever, we established a mixed-task evaluation benchmark comprising 48 LoRAs spanning a variety of natural language understanding and generation tasks. The experimental results underline the effectiveness of the proposed methods in serving both in-domain and out-of-domain downstream requests. Furthermore, the retrieval routing method exhibits a robust generalization capability: although the retriever is trained on just 40% of the tasks, it effectively retrieves the corresponding LoRAs for unseen tasks.

heading: 2 Related Work
text: Mixture of Experts. The Mixture of Experts (MoE) method combines various specialized submodules, guided by a gating network to tailor responses to different input types (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shen et al., 2023; Riquelme et al., 2021; Dou et al., 2023). Some work (Wang et al., 2022; Zadouri et al., 2023; Zhu et al., 2023; Liu et al., 2023a; Dou et al., 2023) focuses on using the MoE method for PEFT to achieve more effective and efficient model finetuning. Other work (Anonymous, 2024; Muqeeth et al., 2023) focuses on using MoE to coordinate existing LoRA experts without specifically training the experts’ parameters. These methods require training additional parameters for gating and are limited to a fixed number of LoRAs, making them unsuitable for complex and dynamic scenarios. Our method can be seen as gating through a retriever, hence achieving flexibility and generalization on unseen experts.
Adapter Merging. In addition to model ensembling through the MoE, there is an increasing focus on aggregating adapters from different domains through the method of Adapter Merging. AdapterSoup (Chronopoulou et al., 2023) aggregates different adapters in the parameter space, allowing large language models to adapt to new domains without additional training. LoRAhub (Huang et al., 2023) employs random sampling of LoRA parameters from various domains and tasks, followed by black-box optimization to learn the weights of different LoRA parameters without involving model
gradient backpropagation. These methods offer a one-size-fits-all solution for downstream tasks, which cannot be applied in the mixed-task scenario for providing personalized service.

heading: 3 Preliminaries
text: This section begins with a concise introduction to the Low-Rank Adaptation, followed by a detailed formalization of the mixed-task scenario.

heading: 3.1 Low-Rank Adaptation
text: Directly fine-tuning large language models with all parameters is computationally intensive and is not feasible in low-resource scenarios. Based on the idea that only a small number of low-rank parameters need to be fine-tuned for sufficient performance in new domains, Hu et al. (2021) proposed the LowRank Adaptation, where the LoRA module can be combined with the pre-trained parameters in parallel for efficient inference.
Specifically, given pre-trained weights W0 ∈ Rd×d of a sub-module of LLM, the LoRA adds an extra trainable weight matrix as W0 +∆W = W0 + BA, where ∆W can be decomposed into two smaller matrices B ∈ Rd×r and A ∈ Rr×d, where r stands for the rank of ∆W . The forward pass can be modified as follows:
x′ = W0x+∆Wx = W0x+BAx, (1)
where x ∈ Rd is the input and the x′ ∈ Rd denote the output.

heading: 3.2 Problem Formulation
text: In this part, we give a formal definition of the mixed-task scenario. Given an original LLM L, we have a set of k LoRAs, Φ = {ϕ1, ϕ2, · · · , ϕk}, on the shelf, where each LoRA ϕi is trained on its corresponding task Ti. The mixed task inputs can be formulated as Tmix = {x,∀x ∈ T1 ∨ T2 · · · ∨ Tk}, where ∨ stands for the logical disjunction operator.
Under the mixed-task scenario, given an input x ∈ Tmix without its task tag, the serving process can be written as:
y = F (g(Φ, x), x, θ), (2)
where θ denotes the original parameters of LLM, g(Φ, x) represents the input-aware LoRA retrieval process and returns a set of retrieved LoRAs Φi. F (Φi, xi, θ) depicts the LoRA composition process that integrates the retrieved LoRAs as a plugin to the original LLM.

heading: 4 LoraRetriever Framework
text: In this section, we describe the LoraRetriever framework as shown in Fig.2 for serving multiLoRAs in mixed-task scenarios. This framework contains three major components: the input-aware LoRA retrieval module (§4.1), the LoRA composition module (§4.2), and the batch inference strategy (§4.3).

heading: 4.1 Input-Aware LoRA Retrieval
text: Our goal is to construct a LoraRetriever tailored to effectively retrieve the corresponding LoRAs for each input in scenarios where LoRAs are dynamically updated. However, existing approaches fall short of accurately identifying LoRAs under such conditions. MoE-based methods (Anonymous, 2024; Muqeeth et al., 2023) struggle to generalize when new LoRAs are introduced due to the fixed selection of LoRAs established during router training. Retrieval methods like sentence embedding (Reimers and Gurevych, 2019; Ni et al., 2021) or task embedding (Achille et al., 2019; Zhou et al., 2022) fail to map both samples and LoRA into a shared embedding space, limiting their effectiveness in input-aware LoRA retrieval.
To achieve this goal, we propose to train a retriever via instruction fine-tuning (Su et al., 2022; Asai et al., 2022), namely LoraRetriever, which can retrieve suitable LoRAs from a massive LoRA pool for a given input sample. The fundamental concept behind LoraRetriever comprises two main steps: (i) First, to embed different task-specific LoRAs into embedding space for facilitating retrieval, we posit that each LoRA can be represented by some data points, which can be obtained by randomly choosing a dozen samples from the training dataset. Then we average their instruction embeddings to represent the embedding of each LoRA. (ii) To improve generalization for unseen LoRAs in LoRA retrieving, we train the retriever through instruction fine-tuning (Su et al., 2022; Wei et al., 2021) on a subset of all tasks. Training on a small subset of tasks is designed to simulate scenarios involving the integration of new LoRAs, thereby underscoring our method’s generalization abilities via instruction fine-tuning. These two strategies enable the effective use of limited data distributions for input-aware retrieval and can be generalized to unseen LoRAs.
Formally, with a sentence-embedding model E, input sequence x, and the instruction I for

heading: Top k Mixture
text: embedding purposes, the instructed embedding can be formulated as E(I ⊕ x), where ⊕ denotes the concatenation operation. In order to allow the embedding to capture the similarity between different tasks, the instruction is expressed as: "Represent the sentence for similar task retrieval" . Each LoRA module is embedded with m randomly selected domain-specific samples, expressed as E(ϕ) = 1m ∑m i=1E(I ⊕ xiϕ). This embedding method integrates both sample-wise and LoRA-module-wise embeddings, facilitating the calculation of similarity between an individual sample and a LoRA module. For measuring the similarity between LoRA module ϕ and the input sequence x, following (Ni et al., 2021), we leverage the cosine similarity between the LoraRetriever embeddings: s(x, ϕ, I) = cos(E(I ⊕ x), E(ϕ)).
To improve LoRA retrieval by the retriever and broaden its generalization to unseen LoRAs, we train the embedding model E through instruction fine-tuning on a small subset of tasks. To prevent the need to access new samples, we use previously employed samples for embedding LoRAs as our training data. Consider t distinct training tasks, represented as Ttrain = {T1, · · · , Tt}. Following Ni et al. (2021), the training dataset D comprises paired samples (xi, x+i ), where each xi is a sample from a task Ti ∈ Ttrain, and a positive sample x+i is randomly selected from the same task Ti. To complement each positive pair, we randomly
select p negative pairs (xi, x−ij) p j=1 , ensuring that x−ij is sourced from tasks outside of Ti, thereby x−ij /∈ Ti. The training process is achieved through a contrastive loss (Karpukhin et al., 2020; Izacard et al., 2021; Ni et al., 2021) defined as follows:
L = e s(xi,x
+ i ,I)/γ
es(xi,x + i ,I)/γ + ∑p j=1 e s(xi,x − ij ,I)/γ ,
where γ is the softmax temperature. During the LoRA retrieval phase, the top-k LoRAs are retrieved according to their similarity to the input x. This process can be formulated as follows:
g(xi,Φ) := Φi = TopK{s(ϕj , xi, I), ϕj ∈ Φ}.

heading: 4.2 LoRA Composition
text: After retrieving the top-k LoRAs Φi for an input xi, we proceed to integrate these LoRAs into the LLM with parameter θ. This integration is achieved by applying two different LoRA composition strategies: the Mixture of LoRAs and the Fusion of LoRAs.

heading: 4.2.1 Mixture of LoRAs
text: The mixture of LoRAs strategy involves the aggregation of the outputs of each submodule within the assembled LoRAs. Let us denote A = {A1, A2, . . . , An} and B = {B1, B2, . . . , Bn}
as the sets representing submodules within n LoRAs. For an input xi, the output derived from the mixture of LoRAs can be expressed as x′i = 1 n ∑n j=1BjAjxi, where x ′ i denotes the output. This process signifies the integration of each LoRA module’s output, effectively blending their contributions to form a unified output.

heading: 4.2.2 Fusion of LoRAs
text: In contrast to the Mixture method, which combines the output of different LoRAs, fusing the parameters of these LoRAs presents an alternative composition strategy.
Let the parameters of each LoRA ϕi be denoted by Θi. The parameter of the fused LoRA is then represented as Θfusion = 1k ∑k j=1Θj . This formulation allows the fused parameter to function akin to a single LoRA.

heading: 4.3 Batch Inference of Multiple LoRAs
text: Implementing batch inference in the presence of multiple LoRAs and diverse composition diagrams poses a significant technical challenge. To address this, we introduce a unique approach for batch inference. Our method involves processing a batch of samples denoted as X ∈ Rb×l×d, where b, l, and d denote the batch size, sequence length, and sample dimensionality, respectively. For each input xi and its retrieved LoRAs Φi within the same batch, we aggregate these LoRAs into a collective set denoted by ΦB. To ensure the uniqueness of ΦB, we eliminate duplicates, mindful of the possibility that retrieved LoRAs may overlap across different samples. The resulting set ΦB comprises p unique LoRAs, where p ≤ bk. For every sample xi, a p dimension mapping vector Mi is generated, which specifies the indices of its corresponding LoRAs within ΦB.
The LoRA mapping vectors are combined into a matrix M ∈ Rb×p. The parameters of a submodule in LoRA can be denoted as A and B, and are concatenated within the batched LoRAs ΦB to obtain A ∈ Rp×r×d and B ∈ Rp×d×r. The batch inference process of the mixture of LoRAs can be formulated as follows:
X ′ = M ◦ (B ◦A ◦X), (3)
where we denote the batched output of a layer of multiple LoRA as X ′ ∈ Rb×l×d and extend the symbol ◦ to denote potential broadcasting as Wen and Chaudhuri (2023). The batch inference process
of LoRA fusion can be formulated as
X ′ = (M ◦B)(M ◦A) ◦X. (4)
These strategies can be simply implemented by the einsum operation, and the PyTorch-style pseudocode is shown in Appendix.D.

heading: 5 Experiments
text: This section outlines the evaluation framework for assessing different approaches in mixed-task scenarios. Furthermore, a comprehensive analysis of the proposed LoraRetriever framework is presented.

heading: 5.1 Evaluation Framework
text: Base Model & LoRA Configuration. To test various methods in the mixed-task scenarios, we leverage Llama-2-{7b,13b} (Touvron et al., 2023) as the base models and train a range of LoRAs for a spectrum of tasks. We select a portion of the Flanv2 datasets (Wei et al., 2021) to train 48 LoRAs for a spectrum of tasks covering Natural Language Understanding (NLU) and Natural Language Generation (NLG). Following the categorization by Wei et al. (2021), these tasks can be grouped into 10 distinct task clusters. We train each LoRA according to the Alpaca (Taori et al., 2023) format and rank r, and the scaling hyperparameter α are set to 6 and 12, respectively. The details of the LoRAs training can be found in the Appendix. E.
Mixed Task Evaluation Dataset. For constructing the mixed-task dataset, we randomly chose 50 samples from the test set for each task used in training 48 LoRAs, subsequently mixing and shuffling these samples to form a unified dataset with 6000 data entries. Further details about these datasets are available in the Appendix.E.
Baseline Methods. We compared our method with the following baselines: (1) Mixture of Experts (Zhu et al., 2023; Zadouri et al., 2023; Liu et al., 2023a; Wang et al., 2022; Anonymous, 2024). (2) SMEAR (Muqeeth et al., 2023); (3) AdapterSoup (Chronopoulou et al., 2023); (4) LoRAhub (Huang et al., 2023). Specifically, we implement three variants of MoE. A detailed description of the baseline models can be found in Appendix.A, with their implementations presented in Appendix.F.
Implementation of LoraRetriever. To train the LoraRetriever, we continue to perform instruction
fine-tuning based on Instructor-xl (Su et al., 2022). The training data consisted of only 40% of the tasks used to train task-specific LoRAs, with each task represented by 20 samples randomly selected from its respective LoRA training set. In this process, we categorized samples from the same LoRA as positive examples, while those from different LoRAs were considered negative examples. Additionally, the three distinct strategies for LoRA composition are: (1) Selection, which involves choosing the highest-ranked (top-1) retrieved LoRA and applying it in a singular LoRA manner, and can be
viewed as a variant for Mixture and Fusion methods; (2) Mixture, which averaging the outputs of each submodule from the top-k retrieved LoRAs; and (3) Fusion, a method that averages the parameters of the top-k retrieved LoRAs. Throughout our experiments, k = 3 is established as the default setting.
Metrics. Following Wei et al. (2021), we assess the performance on the "Struct to Text" task using Rouge-{1, 2, L} and on the "Translation" tasks using BLEU. Additionally, for the NLU tasks, we evaluate the exact match accuracy of each method.

heading: 5.2 Main Results
text: The main results of the mixed-task evaluation are shown in Tab.1. We present the mean performance across each task cluster and additionally evaluate the LoraRetriever’s effectiveness in an out-of-
domain (OOD) setting. In the OOD configuration, we mask the corresponding LoRA for each sample, thereby inhibiting LoraRetriever from retrieving the ideal LoRA for each sample. In this way, we can assess the cross-task generalization capability of LoraRetriever. From the results, we have the following observations: (1) The proposed framework, LoRARetriever, which performs input-aware LoRA retrieval and composition, markedly surpasses other baselines focusing on specific downstream tasks. (2) Among them, the performance of Mixture and Selection is similar in IID scenarios, while Fusion’s performance is weaker compared to the other two methods. The reasons are as follows: (i) In the IID setting, LoraRetriever can achieve strong top-1 selection, leading to similar results between Selection and the Mixture; (ii) As different tasks are inherently heterogeneous, it is inferior to directly average top-k LoRA parameters in the Fusion. (3) In the OOD setting, the Mixture exceeds the performance of the Selection, and the performance of Fusion is similar to that of the Selection. The reasons can be as follows: (i) The selection cannot retrieve the associated LoRA for the input sample in the OOD setting, leading to a significant performance drop. (ii) The Mixture can fully leverage the capabilities of similar tasks to address OOD tasks, alleviating the performance drop. (4) The performance of the MoE and SMEAR methods is weaker than that of LoraRetriever. The limitation stems from the restricted capacity of these methods for adaptation and generalization to dynamically changing environments populated with diverse LoRAs, thereby diminishing their efficacy in mixed-task scenarios. (5) In mixed-task scenarios, although AdapterSoup uniformly searches for appropriate LoRAs for downstream tasks, the retrieved LoRAs fall short in personalization for each request, hindering their effectiveness for each specific task. (6) LoRAhub proves to be entirely ineffective in the mix-task scenario. First, the fusion of LoRAhub depends on randomly selected LoRAs, which may not be relevant. Second, the presence of heterogeneous tasks introduces conflicting parameter optimization directions, resulting in the total breakdown of parameter fusion.

heading: 5.3 Analysis
text: Performance of Retriever. We compare LoraRetriever with some popular off-the-shelf sentence embedding models in Huggingface and adopt
the model nomenclature following Wolf et al. (2020). To analyze the effect of the percentage of the tasks for training LoraRetriever, we trained three variants of LoraRetriever with different percentages. Tab.2 shows the performance of different retrieval models for retrieving relevant LoRAs. It is shown that guiding sentence embedding models with specific prompts leads to a performance improvement in retrieval compared to common retrieval models. After instruction fine-tuning, the retriever significantly enhanced the ability to retrieve corresponding LoRA based on the input. Conducting instruction fine-tuning on 40% of the tasks resulted in a 2.36% increase in top-1 accuracy and a 9.80% increase in top-3 accuracy. Training across all tasks achieved the largest improvement. To demonstrate the generalizability of the proposed framework when dealing with unseen LoRAs, we used a retriever trained on 40% of the tasks in the main experiment to simulate the scenario of dynamic updates to the LoRA pool that might occur while providing services with LoraRetriever.
Tab.3 shows the performance of LoraRetriever trained with different proportions of tasks for LoRA retrieval. It is observed that for the selection and mixture methods, training under 40% of the tasks has already seen significant improvements. The best performance is achieved when trained under all tasks, but the improvement compared to 40% is relatively small, which to some extent reflects the good generalization ability of instruction finetuning of LoraRetriever.
Fig.3 illustrates the similarity between task em-
beddings for different tasks through a heatmap, where tasks from the same task cluster are grouped in square brackets. It is shown that task embeddings within the same domain are more similar, indicating that the LoraRetriever embeddings can serve as task embeddings to characterize the similarities between different tasks and be applied for LoRA retrieval.

heading: Impact of the number of Retrieved LoRA.
text: Fig.4 (a) illustrates the performance of the number of retrieved LoRAs on the mean accuracy of the NLU tasks. The results indicate that as the number of retrieved LoRAs increases, the performance of the Mixture initially improves slightly but then stabilizes. In contrast, the Fusion shows a continuous decline in performance with an increasing number of LoRAs, which once again demonstrates that under the conditions of heterogeneous tasks, the simple averaging of parameters can compromise the original capabilities of the LoRAs. In particular, in the OOD setting, the performance of the Mixture improves significantly as the number of LoRAs increases, illustrating that in the absence of an ideal LoRA choice for a request, leveraging the capabilities of multiple LoRAs of similar tasks can effectively achieve cross-task generalization.

heading: Effectiveness of Batch Inference Strategy. To
text: evaluate the efficiency of our proposed batch inference strategy, we compared the throughput of different batch sizes. The throughput is defined as the number of both input and output tokens per second across all requests in the mixed-task benchmark. We specifically compared the computational efficiency with that of a single LoRA. Our evaluation encompassed the entire evaluation dataset, and we limited the generation to the first produced token to mitigate discrepancies caused by varying generation lengths across different methods. These experiments were conducted on an NVIDIA A100 GPU
(80GB) utilizing bfloat16 precision. As illustrated in Fig.4 (b), our batch inference strategy markedly improves the throughput of the framework, with a slight throughput reduction compared to a single LoRA. Notably, the Fusion outperforms the mixture strategy in throughput efficiency, attributed to its parameter averaging approach that circumvents the need for parallel computation across multiple LoRAs.
Showcases. We showcase the framework’s ability to adeptly integrate multiple LoRAs for synergistic problem-solving, as evidenced in Fig.5. We manually craft three problems in Fig.5, which cannot retrieve any single LoRA to solve these problems directly, necessitating the cooperation of existing LoRAs. Specifically, the first example requires LoraRetriever to integrate NLI and translation tasks’ capabilities. The retrieved LoRA wmt16-tren is utilized for comprehending Turkish, while glueqqp is applied to NLI tasks. In the second scenario, LoRAs are integrated for translating from German to French. Although there is no direct LoRA for German-to-French translation, the combined use of wmt16-deen for German-to-English and wmt14-enfr for English-to-French enables an effective German-to-French translation. The third scenario illustrates the fusion of distinct capabilities by combining Romanian translation with text generation: leveraging the wmt16-roen LoRA for Romanian comprehension and the common-gen LoRA for generating text, LoraRetriever successfully merges these diverse functionalities. This demonstration emphasizes the framework’s substantial ability to blend distinct LoRA capabilities, anticipating further exploration of capability fusion
of LoRAs as a future direction.

heading: 6 Conclusion
text: This paper investigates a new problem of serving multiple LoRAs with a dynamically updated LoRA pool for downstream heterogeneous requests. To this end, we introduce a framework named LoraRetriver to identify and retrieve the appropriate LoRAs based on a specific input. Subsequently, we focus on the composition of these retrieved LoRAs to ensure a tailored and practical application in real-world situations. We also propose an efficient batch inference strategy to accommodate batched requests. Subsequent experiments have also demonstrated the effectiveness of our proposed LoraRetriever.

heading: Limitation
text: While promising, there are still some drawbacks of LoraRetriever. (1) User data privacy issues. When users upload LoRA, we need to use a small amount of training data (10-20 pieces) to represent the distribution of the LoRA model. In privacysensitive scenarios, representation with data may not be feasible. Aligning LoRA parameters and sample distributions in the embedding space in a manner that respects data privacy presents a worthwhile direction for future exploration. (2) The proposed LoraRetriever framework is only suitable for multi-LoRA collaboration under the same model architecture. However, in reality, the model architecture chosen by the users themselves and the PEFT method are not necessarily the same, which is worth further research on how to design the corresponding collaborative mechanism for such scenarios.

heading: Ethics Statement
text: In this work, we take the open-sourced Llama2 (Touvron et al., 2023) model and Flan-v2 (Wei et al., 2021) dataset in the experiment. One potential ethical concern may arise when applying the proposed framework to real-world scenarios, as the data uploaded by LoRA contributors may contain user privacy. However, we believe this can be addressed by proper anonymization before uploading. Therefore, we believe our work will not pose a severe ethical concern.

heading: Acknowledgment
text: This work was supported by the National Natural Science Foundation of China (62441605, 62376243, 62037001, U20A20387), and the Starry Night Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-ZJU-SIAS0010).

heading: A Details of Baseline Methods
text: (1) Mixture of Experts (Zhu et al., 2023; Zadouri et al., 2023; Liu et al., 2023a; Wang et al., 2022; Anonymous, 2024). Many works have considered coordinating different adapters through MoE, and here we explored three distinct variants: one employing a soft mixture of experts and the other utilizing discrete routing (top1 and top3). Detailed
information on the training aspects of MoE methods is provided in the Appendix.F. (2) SMEAR (Muqeeth et al., 2023) introduces the concept of adaptive routing by performing a weighted average of different adapters’ parameters to utilize various experts effectively. For the MoE and SMEAR baselines, challenges arise in scaling due to training confined to a limited set of LoRAs. Consequently, we strategically selected a dedicated LoRA expert for each domain to specialize in router training. (3) AdapterSoup (Chronopoulou et al., 2023) uniformly selects the corresponding LoRAs for the entire downstream task, which lacks the ability to provide personalized service for diverse requests. (4) LoRAhub (Huang et al., 2023) enables blackbox optimization to learn the weights of various LoRA parameters, thereby facilitating weighted parameter averaging for specific downstream tasks. In our implementation, we conformed to the default setting, which entails randomly selecting 20 LoRAs from the available LoRA pool and performing weighted parameter averaging. For the MoE, SMEAR, and LoRAhub approaches, we selected 20 data samples from the training datasets of all tasks to serve as their training data.

heading: B Main Difference between LoraRetriever and Baseline Methods
text: As depicted in Fig.6, LoraRetriever distinguishes itself from baseline methods significantly. It utilizes
an off-the-shelf retriever for routing experts, contrasting with MoE’s approach, to achieve not only more flexibility but also enhanced generalization to unseen LoRAs. Moreover, unlike isolated taskfocused methods like LoRAhub and AdapterSoup, LoraRetriever is adept at adaptively managing heterogeneous downstream tasks.

heading: C Full Results
text: In Tab.4, we show the full results of the mixed-task scenario of all tasks.

heading: D PyTorch-style pseudocode for batch inference
text: The batch inference process can be easily achieved through a few lines of einsum operation. We show the PyTorch style pseudocode in Alg.1.
Algorithm 1: PyTorch-style pseudocode for batch inference # X: (b,l,d), M: (b,p)
# A: (p,r,d), B: (p,d,r)
# LoRA fusion computation
FA=torch.einsum(’bp,prd->brd’,M,A)
FB=torch.einsum(’bp,pdr->bdr’,M,B)
mid=torch.einsum(’bld,brd->blr’,X,FA)
res=torch.einsum(’blr,bdr->bld’,mid,FB)
# LoRA mixture computation
mid=torch.einsum(’bld,prd->blpr’,X,A)
mid=torch.einsum(’blpr,pdr->blpd’,mid,B)
res=torch.einsum(’blpd,bp->bld’,mid,M)

heading: E Details of Training and Evaluation datasets
text: We leverage a subset of flan-v2 datasets (Wei et al., 2021) as shown in Fig.7 for LoRA expert training and mixed-task dataset generation. We summarize the details of the used datasets as follows:
Struct-to-Text Conversion: This task evaluates the capability to generate natural language descriptions from structured data inputs. We use the following datasets: (1) CommonGen; (2) DART; (3) E2ENLG; (4) WebNLG;
Translation: Translation involves converting text from one language to another, maintaining the original meaning and nuances. We use the following datasets: (1) En-Fr from WMT’14; EnDe, En-Tr, En-Ru, En-Fi, En-Ro from WMT’16; (3) En-Es from Paracrawl.
Commonsense Reasoning: This involves assessing the ability to apply physical or scientific principles alongside common sense in reasoning tasks. We use the following datasets: (1) COPA, (2) HellaSwag, (3) PiQA, and (4) StoryCloze.
Sentiment Analysis: A fundamental task in natural language processing (NLP) that determines the sentiment polarity (positive or negative) of a given text. We use the following datasets: (1) IMDB, (2) Sentiment140, (3) SST-2, and (4) Yelp.
Closed-Book Question Answering: This task challenges models to answer questions about general knowledge without direct access to external information sources. We use the following datasets: (1) ARC, (2) NQ, and (3) TriviaQA.
Paraphrase Detection: This task requires models to ascertain whether two sentences convey the same meaning, indicating semantic equivalence. We use the following datasets: (1) MRPC, (2) QQP, and (3) Paws Wiki.
Coreference Resolution: Involves identifying instances within a text that refer to the same entity, demonstrating an understanding of textual context. We use the following datasets: (1) DPR and (2) WSC273.
Reading comprehension: Assesses the capability to derive answers to questions from a provided text containing relevant information. We use the following datasets: (1) BoolQ, (2) DROP, (3) MultiRC, (4) OBQA, (5) SQuADv1, (6) SQuADv2.
Reading Comprehension with Commonsense: Merges traditional reading comprehension skills with commonsense reasoning, requiring understanding beyond the explicit text. We use the following datasets: (1) CosmosQA; (2) ReCoRD.
Natural Language Inference: Focuses on deducing the relationship between two sentences, determining if the second sentence logically follows from, contradicts, or is unrelated to the first sentence. We use the following datasets: (1) ANLI, (2) CB; (3) MNLI; (4) QNLI; (5) SNLI; (6) WNLI; (7) RTE.
F Implementation Details of Baseline Methods

heading: F.1 MoE baselines
text: We use E to denote the LoRA expert and R to denote the router. The MoE methods can be expressed

heading: COMMONSENSE

heading: Translation

heading: Struct to Text

heading: COMMONSENSE

heading: Translation

heading: Struct to Text

heading: DRP
text: in the following way:
y = k∑
i=i
R(x)iEi(x). (5)
We implied two variants of the MoE routing mechanism. (1) Dense Gating. Following (Zadouri et al., 2023), the router network consists of a dense layer with trainable parameter Wg, and the gating score could be obtained through a softmax function by:
si = R(x)i = softmax(W T g x), (6)
(2)Sparse Gate. To maintain the sparsity while training, we leverage the Gumbel softmax trick as (Muqeeth et al., 2023; Nie et al., 2021), where the router can be written as:
R̂(x)i = (log(R(x)i) + gi)/τ∑k
i=1 exp((log(R(x)i) + gi)/τ) (7)
where gi ∼ Gumbel(0, 1) and τ is the temperature. Due to MoE not being easily scalable and arbitrarily adding new LORAs, we randomly selected a LoRA as an expert for each task cluster in the experiment and trained the corresponding Router’s parameters. We randomly selected 20 samples for each task during training to form a unified dataset for parameter training.

heading: F.2 SMEAR
text: SMEAR (Muqeeth et al., 2023) does not perform routing aggregation on the Adapter output but rather aggregates the Adapter at the parameter level. We adopt the same setting as the MoE methods, and the results could be calculated in the
following way:
ΘSMEAR = k∑
i=i
R(x)iΘi, (8)
where Θi denote the parameter of the LoRA-i.

heading: F.3 AdapterSoup
text: AdapterSoup (Chronopoulou et al., 2023), for new downstream tasks, retrieves the parameters that need to be involved in aggregation through sentence bert and performs weight-space averaging on these parameters to adapt to the new domain. We have uniformly retrieved 3 LoRAs for mixed-task to test their capabilities under mixed-task conditions.

heading: F.4 LoRAhub
text: LoRAhub (Huang et al., 2023) also aggregates 20 LoRAs randomly for new downstream tasks. In order to learn the weight of LoRA, a black-box optimization method is employed to learn the weight of each LoRA without calculating the gradients of the large model. It performs weighted averaging at the parameter level. Similar to the training process of MoE, we randomly selected 20 samples for each task to form a unified training dataset for black-box optimization.

heading: G More Related Works
text: Personalized LoRA serving. Sheng et al. (2023) propose S-LoRA to discuss serving thousands of concurrent LoRA. The framework targets scenarios in which multiple tasks must be handled simultaneously without compromising the efficiency of the base models. Wen and Chaudhuri (2023)
propose FLoRA, which enables efficient batching of diverse request types in low-rank adaptation (LoRA) of foundation models. These studies discuss how to deploy or train personalized LoRAs. However, these methods can only utilize a single user-specified LoRA during inference, failing to fully leverage the combination of LoRAs from different tasks. Moreover, the primary focus of these discussions is on computational strategies in GPUs and training strategies, which are orthogonal to the routing strategies with which we are concerned.

heading: Future Work
text: Aligning LoRA parameters and sample distributions in the embedding space in a manner that respects data privacy presents a worthwhile direction for future exploration. (2) The proposed LoraRetriever framework is only suitable for multi-LoRA collaboration under the same model architecture. However, in reality, the model architecture chosen by the users themselves and the PEFT method are not necessarily the same, which is worth further research on how to design the corresponding collaborative mechanism for such scenarios