LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 00:17:54

Abstract: Low-rank adaptation (LoRA) is often employed to fine-tune large language models (LLMs) for new tasks. This paper investigates LoRA composability for cross-task generalization and introduces LoraHub, a simple framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable performance on unseen tasks. With just a few examples from a new task, LoraHub can fluidly combine multiple LoRA modules, eliminating the need for human expertise and assumptions. Notably, the composition requires neither additional model parameters nor gradients. Empirical results on the Big-Bench Hard benchmark suggest that LoraHub, while not surpassing the performance of in-context learning, offers a notable performance-efficiency trade-off in few-shot scenarios by employing a significantly reduced number of tokens per example during inference. Notably, LoraHub establishes a better upper bound compared to in-context learning when paired with different demonstration examples, demonstrating its potential for future development. Our vision is to establish a platform for LoRA modules, empowering users to share their trained LoRA modules. This collaborative approach facilitates the seamless application of LoRA modules to novel tasks, contributing to an adaptive ecosystem.

Main paper: 
 introduction: 
 Recent progress in natural language processing (NLP) has been largely fueled by large language models (LLMs) such as OpenAI GPT (Brown et al., 2020) , Flan-T5 (Chung et al., 2022) , and LLaMA (Touvron et al., 2023) . These models demonstrate top-tier performance across different NLP tasks. However, their enormous parameter size presents issues regarding computational efficiency and memory usage during fine-tuning. To mitigate these challenges, Low-Rank Adaptation (LoRA) (Hu et al., 2022) has emerged as a parameter-efficient fine-tuning technique (Lester et al., 2021; He et al., 2022; An et al., 2022) . By reducing memory demands and computational costs, it speeds up LLM training. LoRA achieves this by freezing the base model parameters (that is, an LLM) and training a lightweight module, which regularly delivers high performance on target tasks.While prior research has targeted the efficiency enhancement facilitated by LoRA, there is a dearth of investigation into the inherent modularity and composability of LoRA modules. Typically, previous methods train LoRA modules to specialize in individual tasks. Yet, the intrinsic modularity of LoRA modules presents an intriguing research question: Would it be possible to compose LoRA modules to generalize to novel tasks in an efficient manner? In this paper, we tap into the potential of LoRA modularity for broad task generalization, going beyond single-task training to meticulously compose LoRA modules for malleable performance on unknown tasks. Crucially, our method enables an automatic assembling of LoRA modules, eliminating dependency on manual design or human expertise. With just a handful of examples from new tasks (e.g., 5), our approach can autonomously compose compatible LoRA modules without human intrusion. We do not make assumptions about which LoRA modules trained on particular tasks can be combined, allowing for flexibility in amalgamating any modules as long as they conform to the specification (e.g., using the same LLM). As our approach leverages several available LoRA modules, we refer to it as LoraHub and denote our learning method as LoraHub learning.To validate the efficiency of our proposed methods, we test our approaches using the widely recognized BBH benchmark with Flan-T5 (Chung et al., 2022) serving as the base LLM. The results Figure 1 : The illustration of zero-shot learning, few-shot in-context learning and few-shot LoraHub learning (ours). Note that the Compose procedure is conducted per task rather than per example. Our method achieves similar inference throughput as zero-shot learning, yet approaches the performance of in-context learning on the BIG-Bench Hard (BBH) benchmark.LORAHUBunderline the effectiveness of the LoRA module composition for unfamiliar tasks through a fewshot LoraHub learning process. Notably, our methodology achieves an average performance that closely matches that of few-shot in-context learning, while demonstrating a superior upper bound, particularly when using different demonstration examples. Additionally, our method substantially reduces the inference cost compared to in-context learning, eliminating the requirement of examples as inputs for the LLM. With fewer tokens per example during inference, our method significantly reduces computational overhead and enables faster responses. It aligns with a broader research trend, where recent studies are actively exploring approaches to reduce the number of input tokens (Zhou et al., 2023; Ge et al., 2023; Chevalier et al., 2023; Jiang et al., 2023a; Li et al., 2023; Jiang et al., 2023b) . Our learning procedure is also notable for its computational efficiency, using a gradientfree approach to obtain the coefficients of LoRA modules and requiring only a handful of inference steps for unseen tasks. For example, when applied to a new task in BBH, our methodology can deliver superior performance in less than a minute using a single A100 card.Importantly, LoraHub learning can feasibly be accomplished with a CPU-only machine, requiring proficiency solely for processing LLM inference. In our pursuit to democratize artificial intelligence, we are taking an important step forward by envisioning the establishment of the LoRA platform. The platform would serve as a marketplace where users can seamlessly share and access well-trained LoRA modules for diverse applications. LoRA providers have the flexibility to freely share or sell their modules on the platform without compromising data privacy. Users, equipped with CPU capability, can leverage trained LoRA modules contributed by others through automated distribution and composition algorithms. This platform not only cultivates a repository of reusable LoRA modules with a myriad of capabilities but also sets the stage for cooperative AI development. It empowers the community to collectively enrich the LLM's capabilities through dynamic LoRA composition. 
problem statement: 
 Large Language Models We assume that a large language model M θ is based on Transformer architecture (Vaswani et al., 2017) and has been pre-trained on a large-scale text corpus. The model architecture can be either encoder-decoder (Raffel et al., 2020) or decoder-only (Brown et al., 2020) . Also, M θ could also have been fine-tuned with a large set of instruction-following datasets such as Flan Colleciton (Longpre et al., 2023) and PromptSource (Bach et al., 2022) .In real-world situations, users often desire an LLM to perform novel tasks that it has not encountered before -an ability widely known as cross-task generalization. Generally, cross-task generalization falls into two categories: zero-shot learning (Mishra et al., 2022; Sanh et al., 2022; Chung et al., 2022; OpenAI, 2022; Lin et al., 2022) , which necessitates no labeled examples of the new task, and few-shot learning (Ye et al., 2021; Min et al., 2022) which demands a handful of labeled examples. Assume we have N distinct upstream tasks that the LLM has been trained on, denoted as T = {T 1 , ..., T N }. Our paper primarily focuses on the latter category, where for an unseen target task T ′ / ∈ T, users can only provide a limited set of labeled examples, Q. Our aim is to modify the model M θ to adapt it to task T ′ using only Q. An intuitive method would be to fine-tune the weights of M θ based on Q, yielding an updated model M ϕ with enhanced performance on T ′ . However, this approach is inefficient, time-consuming, and unstable when Q is small. LoRA Tuning LoRA (Hu et al., 2022) , a parameter-efficient fine-tuning method, facilitates the adaptation of LLMs using lightweight modules, eliminating the need for fine-tuning the entire  
methodology: 
 In this section, we provide an overview of our proposed method. We then explain the LoRA tuning procedure in detail. Last, we introduce the procedure of our LoraHub learning, which consists of the COMPOSE stage and the ADAPT stage.As depicted in Figure 2 , we initially train LoRA modules on a variety of upstream tasks. Specifically, for N distinct upstream tasks, we separately train N LoRA modules, each represented as m i for task T i ∈ T. Subsequently, for a new task T ′ / ∈ T, such as Boolean Expressions represented in Figure 2 , its examples Q are utilized to steer the LoraHub learning process. The LoraHub learning encapsulates two main phases: the COMPOSE phase and the ADAPT phase. In the COM-POSE phase, all available LoRA modules are combined into a single integrated module m, using {w 1 , w 2 , . . . , w N } as coefficients. Each w i is a scalar value that can take on positive or negative values, and the combination can be done in different ways. During the ADAPT phase, the combined LoRA module m is amalgamated with the LLM M θ , and its performance on few-shot examples from the new task T ′ is assessed. A gradient-free algorithm is subsequently deployed to update w, enhancing m's performance (e.g., loss) on the few-shot examples Q. Finally, after iterating through K steps, the optimum performing LoRA module is applied to the LLM M θ , yielding the final LLM M ϕ = LoRA(M θ , m). This serves as an effectively adjusted model for the unseen task T ′ , which will then be deployed and not updated anymore.LoRA effectively minimizes the number of trainable parameters through the process of decomposing the attention weight matrix update of the LLM, denoted as W 0 ∈ R d×k , into low-rank matrices. In more specific terms, LoRA exhibits the updated weight matrix in the form W 0 + δW = W 0 + AB, where A ∈ R d×r and B ∈ R r×k are trainable low-rank matrices with rank r, a dimension significantly smaller than those of d and k. In this context, the product AB defines the LoRA module m, as previously elaborated. By leveraging the low-rank decomposition, LoRA substantially reduces the number of trainable parameters needed to adapt the weights of LLMs duriing fine-tuning.Within the COMPOSE stage, we implement an element-wise method to combine LoRA modules. This process integrates the corresponding parameters of the LoRA modules, requiring the modules being combined to have the same rank r to properly align the structures. Given that m i = A i B i , the combined LoRA module m can be obtained by:m = (w 1 A 1 + w 2 A 2 + • • • + w N A N )(w 1 B 1 + w 2 B 2 + • • • + w N B N ).(1)Notbly, as we show in Sec. 5, combining too many LoRA modules at once can expand the search space exponentially, which may destabilize the LoraHub learning process and prevent optimal performance. To mitigate this, we employ random selection to prune the candidate space, and more advanced pre-filtering algorithms could be explored in the future.During the ADAPT stage, our goal is to modify the coefficients w to boost the model's performace on the examples from an unseen task. One might think of using gradient descent to optimize w, following standard backpropagation methods. However, this approach demands constructing a hypernetwork for all LoRA modules, similar to differentiable architecture search methods (Zhang et al., 2019) . Constructing these hypernetworks demands for substantial GPU memory and time, posing a challenge. Given that w consists of a relatively small number of parameters, we opted for gradient-free methods for optimization instead of gradient descent.Inspired by previous work (Sun et al., 2022) , we utilize a black-box optimization technique to find the optimal w. The optimization process is steered by the cross-entropy loss, setting the goal to locate the best set {w 1 , w 2 , . . . , w N } that reduces the loss L on the few-shot examples Q. Furthermore, we incorporate L1 regularization to penalize the sum of the absolute values of w, helping to prevent obtaining extreme values. Consequently, the final objective of LoraHub is to minimizeL + α • N i=1 |w i |,where α serves as a hyperparameter. In terms of the gradient-free method, we leverage Shiwa, a combinatorial optimization approach (Liu et al., 2020) . Shiwa offers a variety of algorithms and chooses the most suitable optimization algorithm for different circumstances. In most of the forthcoming experimental setups, we primarily employ the Covariance Matrix Adaptive Evolution Strategies (CMA-ES) (Hansen & Ostermeier, 1996) . CMA-ES, as a stochastic and population-based optimization algorithm, offers versatility in addressing a broad spectrum of optimization challenges. It dynamically adjusts a search distribution, which is defined by a covariance matrix. During each iteration, CMA-ES systematically updates both the mean and covariance of this distribution to optimize the target function. In our application, we employ this algorithm to mold the search space for w. Ultimately, we use it to identify the optimal w by evaluating their performance on the few-shot examples from an unseen task. 
experimental results: 
 In this section, we provide details on our main experiments. First, we give an overview of the experimental setup and implementation details. Next, we present our findings along with the results.Large Language Model In our main experiments, we employ FLAN-T5 (Chung et al., 2022) , particularly FLAN-T5-large, as the base LLM. The model has shown impressive abilities to perform zero-shot and few-shot learning.Candidate LoRA Modules Our methodology requires a compendium of LoRA modules trained on preceding tasks. For parity with FLAN, we adopt the tasks utilized to instruct FLAN-T5, thereby Table 1 : Experimental results of zero-shot learning (Zero), few-shot in-context learning (ICL), IA3 fine-tuning (IA3), LoRA tuning (LoRA), full fine-tuning (FFT) and our proposed few-shot LoraHub learning (LoraHub) on the BBH benchmark with FLAN-T5-large as the base LLM. We denote algorithmic tasks with the superscript § following previous work (Wu et al., 2023) . Note that we employ three runs, each leveraging different 5-shot examples per task, as demonstrations for all few-shot methods. The average performance of all methods is reported below, and the best performance of each few-shot method can be found in the Appendix A. Dataset and evaluation Our method is evaluated using the Big-Bench Hard (BBH) benchmark, a well-established standard that consists of multiple-choice questions from a variety of domains. The benchmark consists of 27 different tasks, which are regarded to be challenging for language models. For all tasks, we employ the exact match (EM) as our evaluation metric.Baseline Setup To enhance the demonstration of our method's performance, we expanded our comparisons beyond the zero-shot and in-context learning settings. We specifically chose three representative gradient-based methods for comparison: full fine-tuning (FFT), LoRA tuning (LoRA), and IA3 fine-tuning (IA3) (Liu et al., 2022) . For all gradient-based methods, for a fair comparsion, we train for 40 epochs on the same three runs of 5 examples employed in our methods. In the case of FFT, a learning rate of 3e-5 is employed, whereas for IA3 and LoRA, we adopt a learning rate of 2e-4. We report the performance of each method on the test set at the end of training (averaged over three runs) without any model selection to avoid potential selection bias.As shown in Table 1 , our experimental results demonstarte the superior efficacy of our method in comparison to zero-shot learning while closely resembling the performance of in-context learning (ICL) in few-shot scenarios. This observation is derived from an average performance of three runs, each leveraging different few-shot examples. Importantly, our model utilizes an equivalent number of tokens as the zero-shot method, notably fewer than the count used by ICL. Although occasional performance fluctuations, our method consistently outperforms zero-shot learning in most tasks. In the era of LLMs, the input length is directly proportional to the inference cost, and thus LoraHub's ability to economize on input tokens while approaching the peak performance grows increasingly significant. Moreover, as shown in Appendix Table 8 , the upper bound performance of our method across these runs can surpass ICL on 18 tasks, demonstrating its potential for future development.Even when compared to certain gradient-based optimization methods, our approach consistently demonstrates competitive performance. For example, as depicted in Table 1 , our method exhibits a notable improvement of 3.1% on average in contrast to the promising IA3 method. Nevertheless, we acknowledge that our approach still falls behind LoRA tuning and full fine-tuning, especially in tasks that exhibit significant deviation from the upstream task. Taking Dyck Languages as an example, both LoraHub and ICL achieve only an average performance of nearly 1.0% on these tasks, while LoRA and FFT methods showcase impressive results with only 5 examples.LoraHub addresses the challenge of reducing inference costs by eliminating the need for processing additional tokens, resulting in a noticeable reduction in overall inference expenses. However, it introduces an inherent cost during the ADAPT stage, necessitating extra inference steps, such as the 40 steps employed in our experiments. This introduces a trade-off between choosing the ICL approach and LoraHub, with the decision typically hinging on the nature of the situation.For one-time ad-hoc tasks, the ICL approach should be more pragmatic due to LoraHub's additional inference step costs. In such scenarios, where immediate, single-use solutions are preferred, the simplicity and efficiency of ICL might outweigh the benefits of potential savings offered by LoraHub. Conversely, for recurring or similar tasks, LoraHub emerges as a compelling option. Despite the added inference step cost, LoraHub's ability to efficiently handle repetitive tasks, often occurring thousands of times, while concurrently reducing overall expenses, positions it as a viable option in such kind of situations.In summary, our intention is not to replace ICL, but to present LoraHub as a complementary strategy with performance-efficiency trade-offs. Thus, we encourage a careful consideration of specific use cases and requirements when choosing between ICL and LoraHub, recognizing that the optimal solution may vary based on the nature and frequency of the tasks at hand. 
experimental analysis: 
 In this section, we thoroughly examine the characteristics of our proposed method and uncover several insightful findings. If not specified, we use FLAN-T5-large for all analysis.Which LoRA modules are most effective for BBH tasks?We hypothesized that the amalgamation of LoRA modules could incorporate skills and insights from a variety of specific tasks. To evaluate this, we examined the extent of influence a single LoRA module had amongst all tasks from the BBH benchmark. We measured the impact of each isolated task by calculating the average absolute weight. The top five modules, presented in Table 2, were found to have substantial influence, as indicated by their maximum average weights, which suggested that they were notably more effective in cross-task transfer. Remarkably, a common feature among these top five modules was their association with tasks requiring reading comprehension and reasoning skills-attributes indicative of higher cognitive complexity. However, it is worth noting that none of the modules exhibited consistent improvement across all BBH tasks, as reflected in their average performance on all BBH tasks, which did not show a significant improvement compared to the original FLAN-T5-large, except for the Rank 2. The results underscore the advantages of composing diverse modules in LoraHub.How effective is the gradient-free optimization method?To assess the effectiveness of our gradient-free optimization method in correctly identifying the most suitable LoRA module for a given downstream task, we carried out an empirical study using the WikiTableQuestions (Pasupat & Liang, 2015) (WTQ) dataset. We strategically included a LoRA module that was specifically trained on the WTQ dataset into our pool of LoRA candidate modules, which originally stemmed from tasks exclusive to the Flan Collection. Subsequently, we designated WTQ as the targeted downstream task and computed the weights consistent with the methods employed in LoraHub learning. As an end result, the WTQ-specific LoRA module was awarded the highest weight, exemplifying the algorithm's success in recognizing it as the most relevant. Moreover, the combined LoRA module demonstrated marginal superiority over the WTQ LoRA module. This underscores the claim that the gradient-free optimization method has the ability to proficiently select the optimal upstream LoRA module for an unseen task.Can LoraHub work well on non-instruction-tuning models?In previous investigations, we primarily focused on models with zero-shot capabilities that were trained with instruction tuning. However, for models like T5 without zero-shot abilities, where training has a larger effect on parameters, it was unclear if LoraHub could still effectively manage and improve them. Our experiments show that although these models perform worse than FLAN-T5, LoraHub learning can still enable them to effectively generlize to unseen tasks. See Appendix B for more details.Will the rank of LoRA modules impact the performance of LoraHub learning?The parameter rank plays a crucial role in the LoRA framework, directly influencing the number of trainable parameters utilized during LoRA tuning. This prompts an intriguing question: does the variation in rank values influence the outcomes observed within the LoraHub learning? Our analysis indicates that, for FLAN-T5, the choice of rank has minimal impact. However, for T5, it still exerts some influence. Empirical findings reveal that, in comparison to rank values of 4 or 64, a rank value of 16 consistently demonstrates superior performance across different runs, both in terms of average and optimal values. Additional results are available in Appendix B.Does more LoRA modules lead to better results?In our main experiments, we randomly selected 20 LoRA modules for LoraHub learning. Therefore, we conducted experiments to investigate the effect of using different numbers of LoRA modules. The results demonstrate that as we increased the number of LoRA modules, the variance in performance increased. However, the maximum achievable performance also improved. More analysis on the variance and the detailed results can be found in Appendix G.Does composing LoRA modules extend beyond the single module's benefits?We acknowledge the investigation of cross-task performance in prior work (Jang et al., 2023) , which delved into the capabilities of LoRA and proposed a novel method centered around LoRA module retrieval. In order to ensure a fair comparison, we conducted an experiment where we designed a LoRA retrieval mechanism based on the loss derived from few-shot examples. Specifically, we ranked all LoRA module candidates according to this loss and evaluated the best candidate on the test set of the unseen task. As depicted in Table 3 , the performance of LoRA retrieval is notably impressive, positioning it as a strong baseline. However, in comparison to LoraHub, the performance of LoRA retrieval is relatively less favorable 
related work: 
 Model merging Our method substantially draws on the concept of LoRA module composition, and thus, aligns with the significant thread of research in model merging. This research focus is broadly categorized based on the ultimate objectives of model merging.The first category focuses on merging entire models, and the goal is to combine individually trained models to approximate the performance benefits of model ensembling or multi-task learning. Prior works such as Matena & Raffel (2021) and Jin et al. (2023) operated under the assumption of shared model architectures. Matena & Raffel (2021) amalgamates models by approximating Gaussian posterior distributions garnered from Fisher information, while Jin et al. (2023) merges models steered by weights that minimize the differences in prediction. Another approach is merging models with different architectures. For instance, Ainsworth et al. (2023) configures weights of different models prior to their merger. Following this objective, Stoica et al. (2023) merges models operating on varying tasks by identifying common features, without requiring additional training. Unlike these works, our work focuses on merging models to enable cross-task generalization.The 2023) concentrates on amalgamating experts in mixture of experts models; However, these methods generally necessitate multi-task training or human prior on module selection for the downstream task. In contrast, our method does not impose any special training requirements and simply employs vanilla LoRA tuning. Additionally, the module selection for downstream tasks is entirely data-driven without human prior knowledge. This design gives the advantage of easily adding new LoRA modules for reuse, allowing our method to flexibly scale up the number of LoRA module candidates in the future.The Mixture of Experts (MoE) is an ensemble method, often visualized as a collection of sub-modules, or "experts", each specializing in processing different types of input data. Each expert in this system is controlled by a unique gating network, activated based on the distinct nature of the input data. For every token in these input sequences, this network identifies and engages the most suitable experts to process the data. As a result, the performance is superior compared to relying on a single, generic model for all types of input. This technique has proven instrumental in numerous domains, such as natural language processing and computer vision (Jacobs et al., 1991; Shazeer et al., 2017; Du et al., 2022; Zhang et al., 2022; crumb, 2023) . Our methodology displays similarities to MoE, wherein upstream-trained LoRA modules can be aligned with MoE's expert design. A noteworthy distinguishing factor is that our approach mechanism does not require any specialized manipulation of LoRAs during training while facilitating dynamic LoRA module assembly at any scale, each pre-tuned to different tasks. In contrast, MoE mandates a predetermined count of experts during both the training and testing phases. Recent studies on the interrelation between MoE and instruction tuning have demonstrated that the simultaneous application of both approaches enhances the effectiveness of each individually (Shen et al., 2023) .Cross-Task generalization Recent advancements like CrossFit (Ye et al., 2021) , ExT5 (Aribandi et al., 2022) , FLAN (Wei et al., 2022) , T0 (Sanh et al., 2022) , InstructGPT (Ouyang et al., 2022), and ReCross (Lin et al., 2022) have been striving to foster a vastly multi-task model's generalization across different tasks, very much aligned with the objectives of our research. Among this cohort, the connections of CrossFit and ReCross with LoraHub are particularly noteworthy. The CrossFit framework (Ye et al., 2021) mandates a minimal number of labeled examples of the target task for few-shot fine-tuning. However, its limitation lies in the application of task names as hard prefixes in templates, posing challenges in the task's generalization. On the other hand, while ReCross mitigates the need for labels in few-shot examples for retrieval, it necessitates a fine-tuning process using the retrieved data. This procedure appears time-consuming when compared to LoraHub's approach. Through the deployment of few-shot labeled examples and a gradient-free optimization process, LoraHub facilitates an iterative update of weights to compose the LoRA modules. The resultant method is more efficient and cost-effective relative to previous work. Overall, LoraHub offers a more practical and viable solution to the optimization process. 
limitations & future work: 
 Pre-Filtering of LoRA Module Candidates While our method is successful in identifying and weighting relevant aspects from seen tasks to enhance unseen task performance, relying entirely on the model to perform this search can lead to increased computational demands and potentially unstable results. Incorporating a pre-filtering step to select only pertinent LoRA modules could expedite and refine performance. Identifying an effective selection strategy warrants further study.Method Applicability to Decoder-Only Models All experiments for this study were executed using the encoder-decoder architecture. We aspire to extrapolate this method to decoder-only models such as GPT (Brown et al., 2020) , aiming to determine its applicability in such contexts.Exploring Superior Optimization Methods The use of a genetic algorithm for optimization in this study raises the question of whether better optimization approaches exist that could provide superior gradient-free optimization with limited examples. Although the current method has shown adequate performance, there is still room for improvement. 
conclusion: 
 In this work, we have introduced LoraHub, a strategic framework for composing LoRA modules trained on diverse tasks in order to achieve adaptable performance on new tasks. Our approach enables the fluid combination of multiple LoRA modules using just a few examples from a novel task, without requiring additional model parameters or human expertise. The empirical results on the BBH benchmark demonstrate that LoraHub can effectively match the performance of in-context learning in few-shot scenarios, removing the need for in-context examples during inference. Overall, our work shows the promise of strategic LoRA composability for rapidly adapting LLMs to diverse tasks. By fostering reuse and combination of LoRA modules, we can work towards more general and adaptable LLMs while minimizing training costs. 
 
 Appendix: As shown in Table 8 , compared to gradient-based parameter-efficient training methods like LoRA and IA3, our approach demonstrates superior performance in terms of best results over experimental runs. While it exhibits a noticeable lag behind the fully fine-tuning (FFT) method, which updates all parameters during training, this observation suggests that our proposed method has a promising upper limit. We anticipate that future research efforts can contribute to accelerating the optimization speed and further enhancing the efficacy of our approach.Table 4 : Experimental results of several few-shot methods, including in-context learning (ICL), IA3 fine-tuning (IA3), LoRA tuning (LoRA), full fine-tuning (FFT) and our LoraHub learning (LoraHub) on the BBH benchmark with FLAN-T5-large as the base LLM. We denote algorithmic tasks with the superscript § following previous work (Wu et al., 2023) . Note that we use 5 examples per task as the demonstration for all methods. The best (best) performance is reported as the maximum value obtained across three runs. In order to enhance the robustness of LoraHub, we explored a straightforward approach in the selection of LoRA module candidates. Specifically, we first identified 20 LoRA module candidates with the lowest loss on the few-shot examples. Our findings indicate a slight improvement in overall performance after applying the pre-filtering startegy. Since the primary instability in our approach arises from the selection of LoRA candidates. This method involves choosing a fixed set of LoRA candidates to ensure the stability of our approach. In our study, we found that certain LoRA modules tend to have a strong influence when combined into merged LoRAs. We're interested in evaluating how well BBH performs on the top five tasks related to these LoRAs. The results indicate that these top LoRAs perform similarly or even worse than zero-shot in most cases. Only one of them stands out as significantly better than zero-shot. However, it's worth noting that this performance is not as impressive as Lorahub. These findings support the idea that the merging process can improve overall performance. We implemented LoRA tuning using the Huggingface PEFT library (Mangrulkar et al., 2022) , with the rank being set as 16. The gradient-free method was implemented using the open-source Nevergrad optimization library (Rapin & Teytaud, 2018) , with a constraint that the absolute value of LoRA weights should not exceed 1.5. Originally, all coefficients of LoRA modules were set at zero.In our standard settings, we set the maximum number of iterations K as 40. The same 5 examples were used during our LoraHub learning and the few-shot in-context learning. The hyperparameter α is set as 0.05. Regarding the hyperparameters for training candidate LoRA modules, we maintained consistency across all modules, setting the batch size at 64, the learning rate at 1e -4, and the number of training epochs at 10.As shown in Figure 3 , with an increase in the number of LoRA module candidates, there is a corresponding increase in the performance variance. Based on our in-depth analysis, the primary source of variance is not related to gradient-free optimization algorithms but rather associated with the LoRA candidate modules. In other words, once the candidates are determined, random seeds have minimal impact on the final performance. Hence, we posit that the observed instability primarily arises from the inherent challenge of balancing the quantity and quality of the LoRA module candidates.In this section, we omitted the threshold in our implementation, and the results are summarized in Table 9 . Our observations indicate that the removal of the threshold had minimal impact on the majority of tasks, underscoring the robustness of the gradient-free optimization algorithm itself in most cases. The algorithm efficiently identified reasonable ranges even without specific upper and lower bounds. However, three tasks, namely Date Understanding, Disambiguation and Hyperbaton, exhibited notable effects. The resulting performance decline led to an average decrease of 1.2% compared to the setting with threshold. This highlights the significance of establishing a reasonable threshold to mitigate extreme scenarios. We accessed these publicly available tasks via huggingface.co/datasets/conceptofmind/FLAN 


Decision: The paper proposes a simple and intuitive framework for combining LoRA models by applying zero-order optimization on weights of these LORA additions to minimize loss.
The problem is important, but the solution is not particularly novel or surprising, yet the reviewers liked the paper. I therefore recommend acceptance pending space.

Reviews: review: paper_summary: paper_summary: The paper operates in the task generalization setting, where information (parameters) from seen tasks are used to efficiently generalize to downstream, unseen tasks. This task generalization setting have long been explored in the past but remains (imho) a relevant and interesting problem. Many related papers exist in the literature: the most related papers in this setting are (amongst others) SpoT and Polytropon. The novelty of this paper with respect to these papers is rather small: the authors use LoRAs instead of soft-prompts (as in SPoT) (minor), but use gradient free optimization for task adaptation (major novelty?). It is not clear to me however what is exactly the advantage of gradient-free optimization, i.e. the combination weights and adapters parameters themselves might be very cheaply optimized via gradient-descent on the downstream examples. The paper makes some nice ablations in the appendix and goes a little bit towards selecting best modules to combine or filtering modules. I think some more work in this direction might give a bit more novelty to this paper. I think in its current status this is quite a borderline paper.
strength_weakness: strength_weakness: - Novelty is somewhat limited with respect to existing works (MHR, SPoT, Polytropon)
- Performance on BBH is quite low (uses T5 model), so it might be difficult to get insights from ablations
- The justification for using gradient-free optimization appears a bit weak, I would love to see comparisons to gradient-based optimization of the LoRA weights (performance vs time)
questions: questions: You might fix the few typos in the appendix:

NON-INSTRCUTION-TUNED -> Non-Instruction Tuned
Comparsion -> Comparison
paper_summary: paper_summary: This paper investigates the issue of efficiently combining multiple LoRAs for effective cross-task generalization. The proposed method, LoraHub, dynamically integrates multiple LoRA modules—each trained on distinct tasks—to enhance performance on new, unseen tasks. LoraHub is designed to efficiently integrate these modules without requiring human intervention, additional parameters, or gradients. Further experiments conducted using the BBH benchmark demonstrate that LoraHub can emulate in-context learning in few-shot scenarios, effectively eliminating the need for in-context examples during inference. Additionally, this approach encourages community-driven model sharing, which could significantly boost progress in general intelligence applications and the deployment of large language models in production environments.
strength_weakness: strength_weakness: 1.	The experimental results significantly lag behind other baselines, particularly when using many tasks' LoRAs with few-shot samples, where the model's performance remains weaker than that of in-context learning. Furthermore, the model substantially underperforms compared to other gradient-based methods. The authors need to consider more carefully the model's applicability and potential use cases.
2.	LoraHub randomly selects 20 LoRAs from the LoRA pool for combination, which seems arbitrary. This approach risks selecting LoRAs significantly mismatched to the downstream task while overlooking those from similar tasks, thus degrading model performance. Recent studies [1-2] have successfully employed a retriever mechanism to identify LoRAs that are more analogous to the downstream task for composition. Incorporating similar technology could refine LoraHub's selection process, making it more effective.
3.	The authors point out that gradient-based methods are resource-intensive. Therefore, it is crucial to quantify the computational resource consumption of different methods in the experiments to highlight LoraHub's advantages. This comparison was neglected in the conducted experiments.

[1] Chronopoulou, Alexandra, et al. "Adaptersoup: Weight averaging to improve generalization of pretrained language models." arXiv preprint arXiv:2302.07027 (2023).

[2] Zhao, Ziyu, et al. "LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild." arXiv preprint arXiv:2402.09997 (2024).
questions: questions: See reasons to reject.
paper_summary: paper_summary: This paper introduces LoraHub a framework to compose multiple LoRA modules to improve performance on unseen task. LoraHub involves two main steps compose and adapt. Compose involves composing different LoRA modules by introducing weights to weigh the contribution of each module and adapt involves finding the optimal weights so as to get good performance on unseen task. To optimize this the paper uses CMA-ES gradient free optimizaton framework to find the optimal weights using the few shot samples. The paper also claims that they can get performance close to ICL by using the LoraHub approach on BBH tasks.
strength_weakness: strength_weakness: - Decent gap between performance of ICL and LoraHub (on average by 3 points on different datasets reported in the paper).
questions: questions: - In the main text of the paper the authors used randomly selected 20 LoRA modules as a pool of modules for composing them. Is there any reason why not always use all available LoRA modules for all experiments? As the number of LoRa modules increases is there any impact on computational time to find optimal $w_i$
paper_summary: paper_summary: The authors explore how to exploit and leverage lora modules to be effective for few-shot learning without the additional inference cost of updating the prompt. They do so by tuning lora modules through compose and adapt iterations to adjust the coefficients of the module without any gradient updates, allowing the method to be lightweight and scale to any task at hand allowing for cross-task generalization. They demonstrate that their proposed method is competitive with few-shot ICL setups and only slightly lagging behind lora fine-tuning.
strength_weakness: strength_weakness: - The LORA Hub setup is not a clear win over few-shot ICL, while it improves the latency of inference this comes at the overhead of adjusting the lora module co-efficients, which otherwise would have required hand authoring a couple of samples
questions: questions: - Have you explored how we could mitigate the number of adjust iterations by adding some few-shot examples to compensate like a hybrid approach that requires fewer hand-authored samples but can also leverage the LORA modules
- Have you quantified the amount of time it takes to author high quality few-shot samples and how LORA modules compare to that? It might make more a more convincing case

Comments: comment: Thanks for the clarification, I’d like to leave my score as is