LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
Authors: Chengsong Huang
Qian Liu
Bill Yuchen Lin
Tianyu Pang
Chao Du
Min Lin
Venue: COLM (2024)
Keywords: Parameter Efficient Tuning, Cross-Task Generalization, Model Merging
TL;DR: We present LoraHub, a framework for easily combining multiple LoRA modules to achieve similar performance as in few-shot in-context learning.

Abstract: Low-rank adaptation (LoRA) is often employed to fine-tune large language models (LLMs) for new tasks. This paper investigates LoRA composability for cross-task generalization and introduces LoraHub, a simple framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable performance on unseen tasks. With just a few examples from a new task, LoraHub can fluidly combine multiple LoRA modules, eliminating the need for human expertise and assumptions. Notably, the composition requires neither additional model parameters nor gradients. Empirical results on the Big-Bench Hard benchmark suggest that LoraHub, while not surpassing the performance of in-context learning, offers a notable performance-efficiency trade-off in few-shot scenarios by employing a significantly reduced number of tokens per example during inference. Notably, LoraHub establishes a better upper bound compared to in-context learning when paired with different demonstration examples, demonstrating its potential for future development. Our vision is to establish a platform for LoRA modules, empowering users to share their trained LoRA modules. This collaborative approach facilitates the seamless application of LoRA modules to novel tasks, contributing to an adaptive ecosystem.

Published as a conference paper at COLM 2024
LoraHub: Efficient Cross-Task Generalization via Dy- namic LoRA Composition
Chengsong Huang Qian Liut*, Bill Yuchen Lin Tianyu Pang*, Chao Du Min Lint
tSea Al Lab, Singapore
$ Washington University St. Louis, MO, USA
O Allen Institute for AI, Seattle, WA, USA
Abstract
Low-rank adaptations (LoRA) are often employed to fine-tune large lan- guage models (LLMs) for new tasks. This paper investigates LoRA com- posability for cross-task generalization and introduces LoraHub, simple framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable perfor- mance on unseen tasks. With just few examples from a new task, Lo- raHub can fluidly combine multiple LoRA modules, eliminating the need for human expertise and assumptions. Notably, the composition requires neither additional model parameters gradients. Empirical results on the Big-Bench Hard benchmark suggest that LoraHub, while not surpass- ing the performance of in-context learning, offers notable performance- efficiency trade-off in few-shot scenarios by employing significantly re- duced number of tokens per example during inference. Notably, LoraHub establishes better upper bound compared to in-context learning when paired with different demonstration examples, demonstrating its poten- tial for future development. Our vision is to establish platform for LoRA modules, empowering users to share their trained LoRA modules. This collaborative approach facilitates the seamless application of LoRA mod- ules to novel tasks, contributing to an adaptive ecosystem. Our code available at gi thub. com/sail- lalllaallllllaaaaeeeeeeeeeeeaaae ss//lrrahu, and all the pre-trained LoRA modules are released at huggingf ace. ct/coo/olaaaaa co/lorahub
1
Introduction
zero-shot learning
(a)
(b)
(c)
few-shot in-context learning (b)
Compose
few-shot LoraHub learning (c)
LLM
LLM
LLM
LORAHUE
888
37.5
27.0
BiG-Bench Hard Performance
Unseen Task Examples
Unseen Task Test Input
Figure 1: The illustration of zero-shot learning, few-shot in-context learning and few-shot LoraHub learning (ours). Note that the Compose procedure is conducted per task rather than per example. Our method achieves similar inference throughput zero-shot learn- ing, yet approaches the performance of in-context learning on the BIG-Bench Hard (BBH) benchmark.
Recent progress in natural language processing (NLP) has been largely fueled by large lan- guage models (LLMs) such as OpenAl GPT (Brown et al., 2020), FLAN-T5 (Chung et al., 2022), and LLaMA (Touvron et al., 2023). These models demonstrate top-tier performance
"The first three authors contributed equally to this work. Correspondence to Qian Liu at Liugian@sea.com

---

Published as a conference paper at COLM 2024
across different NLP tasks. However, their enormous parameter size presents issues garding computational efficiency and memory usage during fine-tuning. To mitigate these challenges, Low-Rank Adaptation (LoRA) (Hu et al., 2022) has emerged efficient fine-tuning technique (Lester et al., 2021; He et al., 2022; An et al., 2022). By reduc- ing memory demands and computational costs, it speeds up LLM training. LoRA achieves this by freezing the base model parameters (that is, an LLM) and training a lightweight module, which regularly delivers high performance on target tasks.
parameter-
While prior research has targeted the efficiency enhancement facilitated by LoRA, there is dearth of investigation into the inherent modularity and composability of LoRA modules. Typically, previous methods train LoRA modules to specialize in individual tasks. Yet, the intrinsic modularity of LoRA modules presents an intriguing research question: Would it be possible to compose LoRA modules to generalize to novel tasks in an efficient manner? In this paper, we tap into the potential of LoRA modularity for broad task generalization, going beyond single-task training to meticulously compose LoRA modules for malleable performance on unknown tasks. Crucially, our method enables an automatic assembling of LoRA modules, eliminating dependency on manual design or human expertise. With just a handful of examples from new tasks (e.g., 5), our approach can autonomously compose compatible LoRA modules without human intrusion. We do not make assumptions about which LoRA modules trained on particular tasks can be combined, allowing for flexibility in amalgamating any modules as long as they conform to the specification (e.g., using the same LLM). As our approach leverages several available LoRA modules, we refer to it as LoraHub and denote our learning method as LoraHub learning.
To validate the efficiency of our proposed methods, we test our approaches using the widely recognized BBH benchmark with FLAN-T5 (Chung al., 2022) serving as the base LLM. The results underline the effectiveness of the LoRA module composition for unfamiliar tasks through a few-shot LoraHub learning process. Notably, our methodology achieves an average performance that closely matches that of few-shot in-context learning, while demonstrating a superior upper bound, particularly when using different demon- stration examples. Additionally, our method substantially reduces the inference cost com- pared to in-context learning, eliminating the requirement of examples as inputs for the LLM. With fewer tokens per example during inference, computational overhead and enables faster responses. It aligns with a broader research trend, where recent studies are actively exploring approaches to reduce the number of in- put tokens (Zhou et al., 2023; Ge et al., 2023; Chevalier et al., 2023; Jiang et al., 2023a; Li et al., 2023; Jiang et al., 2023b). Our learning procedure is also notable for its computational efficiency, using a gradient-free approach to obtain the coefficients of LoRA modules and requiring only a handful of inference steps for unseen tasks. For example, when applied to a new task in BBH, our methodology can deliver superior performance in less than
method significantly reduces
minute using a single A100 card.
Importantly, LoraHub learning can feasibly be accomplished with CPU-only machine, requiring proficiency solely for processing LLM inference. In our pursuit to democratize artificial intelligence, we are taking an important step forward by envisioning the establish- ment of the LoRA platform. The platform would serve as a marketplace where users can seamlessly share and access well-trained LoRA modules for diverse applications. LoRA providers have the flexibility to freely share or sell their modules on the platform without compromising data privacy. Users, equipped with CPU capability, can leverage trained LoRA modules contributed by others through automated distribution and composition al- gorithms. This platform not only cultivates repository of reusable LoRA modules with myriad of capabilities but also sets the stage for cooperative AI development. It empow- ers the community collectively enrich the LLM's capabilities through dynamic LoRA composition.
2 Problem Statement
Large Language Models We assume that a large language model Mg is based on Trans- former architecture (Vaswani et al., 2017) and has been pre-trained large-scale text cor-
2

---

Published as a conference paper at COLM 2024
pus. The model architecture can be either encoder-decoder (Raffel et al., 2020) decoder- only (Brown et al., 2020). Also, Ma could also have been fine-tuned with a large set of instruction-follooinng datasets such as Flan Colleciton (Longpre et al., 2023) and Prompt- Source (Bach et al., 2022).
Cross-Task Generalization In real-world situations, users often desire an LLM to per- form novel tasks that it has not encountered before - an ability widely known cross-lask generalization. Generally, cross-task generalization falls into two categories: zero-shot learn- ing (Mishra et al., 2022; Sanh et al., 2022; Chung et al., 2022; OpenAl, 2022; Lin et al., 2022), which necessitates no labeled examples of the new task, and few-shot learning (Ye et al., 2021; Min et al., 2022) which demands handful of labeled examples. Assume N distinct upstream tasks that the LLM has been trained on, denoted as ' = [TI, TN]. Our paper primarily focuses on the latter category, where for unseen target task T & T, users can only provide a limited set of labeled examples, Q. Our aim is to modify the model Ma adapt to task T' using only Q. An intuitive method would be fine-tune the weights of Mg based on Q, yielding an updated model Mp with enhanced performance on T'. However, this approach is inefficient, time-consuming, and unstable when Q is small.
have
LoRA Tuning LoRA is a parameter-efficient fine-tuning method (Hu et al., 2022), facil- itates the adaptation of LLMs using lightweight modules, eliminating the need for fine- tuning the entire weights. LoRA tuning involves keeping the original model weights frozen while introducing trainable low-rank decomposition matrices as adapter modules into each layer of the model. Compared to the base LLM, this module possesses signif- icantly fewer trainable parameters, paving the way for rapid adaptation using minimal examples. As such, LoRA tuning presents resource-efficient technique to quickly adapt LLMs for new tasks with restricted training data. However, traditional LoRA methods pri- marily concentrate training and testing within the same tasks (Gema et al., 2023), rather than venturing into few-shot cross-tast sassk generalization.
3 Methodology
In this section, we provide an overview of our proposed method. We then explain the LoRA tuning procedure in detail. Last, we introduce the procedure of our LoraHub learn- ing, which consists of the COMPOSE stage and the ADAPT stage.
3.1 Method Overview
As depicted in Figure 2, we initially train LoRA modules variety of upstream tasks. Specifically, for N distinct upstream tasks, we separately train N LoRA modules, each rep- resented as m; for task T; T. Subsequently, for a new task T' T, such as Boolean Ex- pressions represented in Figure 2, its examples Q are utilized to steer the LoraH learning process. The LoraHlub learning encapsulates two main phases: the COMPOSE phase and the ADAPT phase. In the COMPOSE phase, all available LoRA modules are combined into single integrated module m, using { [w, W2,..., w coefficients. Each w; is scalar value that can take on positive or negative values, and the combination can be done in different ways. During the ADAPT phase, the combined LoRA module Ait is amalgamated with the LLM Ma, and its performance on few-shot examples from the new task T' is as- sessed. gradient-free algorithm subsequently deployed to update w, enhancing m's performance (e.g., loss) on the few-shot examples Q. Finally, after iterating through K steps, the optimum performing LoRA module is applied to the LIM Ma, yielding the final LLM M3 = LoRA(M, m). This serves as an effectively adjusted model for the unseen task T', which will then be deployed and not updated anymore.
3.2 LoRA tuning on upstream tasks
LoRA effectively minimizes the number of trainable parameters through the process of decomposing the attention weight matrix update of the LLM, denoted as Wo E dd into
3

---

Published as a conference paper at COLM 2024
Commonsense
W
W2
W3
W4
Wy
Reasoning
X
Natural Language
Inference
Question Answering
K
ADAPT
COMPOSE
result Boolein expression:
LLM X
Question
evaluate
(TTrue) and
Generation
(True ) False
Expressions
LoRA Tuning on Upstream Tasks
LoraHub Learning for Unseen Tasks
Figure 2: Our method encompasses two stages: the COMPOSE stage and the ADAPT stage. During the COMPOSE stage, existing LoRA modules are integrated into one unified mod- ule, employing set of coefficients, denoted w. In the ADAPT stage, the combined LoRA module is evaluated on few examples from the unseen task. Subsequently, gradient-free algorithm applied to refine w. After executing K iterations, highly adapted combined LoRA module is produced, which can be incorporated with the LLM to perform the in- tended task.
low-rank matrices. In more specific terms, LoRA exhibits the updated weight matrix in the form Wo 5W Wo AB, where Råxr and B Ra* are trainable low-rank matrices with rank a dimension significantly smaller than those of d and k. In this context, the product AB defines the LoRA module m, as previously elaborated. By leveraging the low-rank decompositin needed to adapt the weights of LLMs duriing fine-tuning.
LoRA substantially reduces the number of trainable parameters
3.3 COMPOSE: Element-wise composition of LoRA modules
Within the COMPOSE stage, we implement an element-wise method to combine LoRA modules. This process integrates the corresponding parameters of the LoRA modules, requiring the modules being combined to have the same rank r to properly align the struc- tures. Given that m; = A,B; the combined LoRA module i can be obtained by:
n = (w) A1 + W2/2 + ... + AN)N)(W) Bi + W2B2 + ... + WN)
(1)
Notbly, as we show in Sec. 5, combining too many LoRA modules at once can expand the search space exponentially, which may destabilize the LoraHub learning process and prevent optimal performance, To mitigate this, we employ random selection to prune the candidate space, and more advanced pre-filtering algorithms could be explored in the fu- ture.
3.4 ADAPT: Weight optimization via gradient-free methods
During the ADAPT stage, our goal is to modify the coefficients w boost the model's per- formace the examples from unseen task. One might think of using gradient descent to optimize w, following standard backpropagation methods. However, this approach de- mands constructing hypernetwork for all LoRA modules, similar to differentiable archi- tecture search methods (Zhang et al., 2019). Constructing these hypernetworks demands for substantial GPU memory and time, posing challenge. Given that w consists of rel- atively small number of parameters, we opted for gradient-free methods for optimization instead of gradient descent.
Inspired by previous work (Sun et al., 2022), we utilize a black-box optimization technique to find the optimal w. The optimization process is steered by the cross-entropy loss, setting the goal to locate the best set [ ......, w that reduces the loss L on the few-shot examples Q. Furthermore, we incorporate L1 regularization to penalize the sum of the absolute values of helping to prevent obtaining extreme values. Consequently, the final objective of HuraHub is to minimize NN [ww), where serves hyperparameter.

---

Published as a conference paper at COLM 2024
In terms of the gradient-free method, we leverage Shiwa, combinatorial optimization ap- proach (Liu et al., 2020). Shiwa offers a variety of algorithms and chooses the most suitable optimization algorithm for different circumstances. In most of the forthcoming experi- mental setups, we primarily employ the Covariance Matrix Adaptive Evolution Strategies (CMA-ES) (Hansen & Ostermeier, 1996). CMA-ES, optimization algorithm, offers versatility in addressing a broad spectrum of optimization challenges. It dynamically adjusts a search distribution, which is defined by covariance matrix. During each iteration, CMA-ES systematically updates both the mean and covari- ance of this distribution to optimize the target function. In our application, we employ this algorithm to mold the search space for w. Ultimately, we use it to identify the optimal w by evaluating their performance on the few-shot examples from an unseen task.
stochastic and population-based
Experimental Results
In this section, we provide details on our main experiments. First, we give an overview of the experimental setup and implementation details. Next, we present our findings along with the results.
4.1 Experimental setup
Large Language Model In our main experiments, we employ FLAN-T5 (Chung et al., 2022), particularly FLAN-I5-large, the base LLM. The model has shown impressive abil- ities to perform zero-shot and few-shot learning.
Candidate LoRA Modules Our methodology requires compendium of LoRA modules trained on preceding tasks. For parity with FLAN, we adopt the tasks utilized to instruct FLAN-T5, thereby incorporating nearly 200 distinct tasks and their corresponding instruc- tions. Following this, we trained several LoRA modules as potential candidates. During each experimental sequence, we randomly select 20 LoRA modules from them the can- didate for our LoraHub learning.
Dataset and evaluation Our method is evaluated using the Big-Bench Hard (BBH) bench- mark, a well-established standard that consists of multiple-choice questions from a variety of domains. The benchmark consists of 27 different tasks, which are regarded to be chal- lenging for language models. For all tasks, we employ the exact match (EM) as our evalu- ation metric.
Baseline Setup To enhance the demonstration of our method's performance, we ex- panded our comparisons beyond the zero-shot and in-context learning settings. We specif- ically chose three representative gradient-based methods for comparison: full fine-tuning (FFT), LoRA tuning (LoRA) (Hu al., 2022), and IA3 fine-tuning (IA3) (Liu al., 2022). For all gradient-based methods, for fair comparsion, we train for 40 epochs on the same three runs examples employed
our methods. In the case of FFT, learning rate of 3e-5 is employed, whereas for IA3 and LoRA, we adopt learning rate of 2e-4. We report the performance of each method on the test set at the end training (averaged over three runs) without any model selection to avoid potential selection bias.
4.2 Main results
As shown in Table 1, our experimental results demonstarte the superior efficacy of our method in comparison to zero-shot learning while closely resembling the performance of in-context learning (ICL) in few-shot scenarios. This observation is derived from aver- age performance of three runs, each leveraging different few-shot examples. Importantly, our model utilizes equivalent number of tokens the zero-shot method, notably fewer than the count used by ICL. Although occasional performance fluctuations, our method consistently outperforms zero-shot learning in most tasks. In the era of LLMs, the input

---

Published as a conference paper at COLM 2024
Table Experimental results of zero-shot learning (Zero), few-shot in-context learning (ICL), IA3 fine-tuning (IA3), LoRA tuning (LoRA), full fine-tuning (FFT) and our pro- posed few-shot LoraHub learning (LoraHub) on the BBH benchmark with FLAN-T5-large as the base LLM. We denote algorithmic tasks with the superscript $ following previous work (Wu et al., 2023b). Note that we employ three runs, each leveraging different 5-shot examples per task, as demonstrations for all few-shot methods. The average performance of all methods is reported below, and the best performance of each few-shot method can be found in the Appendix B.
Task
Zero
ICLavg
IA3avg
LoRAavg
FFT avg
LoraHubavg
Boolean Expressions
54.0
59.6
56.2
56.0
62.2
55.5
Causal Judgement
57.5
59.4
60.2
55.6
57.5
54.3
Date Understanding
15.3
20.4
20.0
35.8
59.3
32.9
Disambiguation
0.0
69.1
68.2
45.2
Dyck Languages
1.3
0.9
4.2
22.2
19.5
1.0
Formal Fallacies
51.3
55.3
51.5
53,6
54.0
52.8
Geometric Shapes
6.7
19.6
14.7
24
31.1
7.4
Hyperbaton
71.8
49.3
55.3
77.3
62.8
Logical Deduction
21.3
39.1
32.7
40.0
42.2
36.1
(five objects)
Logical Deduction
12.7
40.7
33.8
37.3
44.9
36.8
(seven objects)
Logical Deduction
0.0
51.6
8.5
53.6
52.9
45.7
(three objects)
Movie Recommendation
62.7
55.8
61.8
51.5
66.0
55.3
Multistep Arithmetic
0.7
0.7
0.7
0.2
0.0
0.4
Navigate
47.3
45.3
48.0
48.0
47.1
Object Counting
34.7
32.4
35.1
38.7
35.6
33.7
Penguins in a Table
43.5
41.3
45.0
36.2
31.9
35,9
Reasoning about Colored Objects
32.0
40.2
40.7
39,6
37.6
40.0
Ruin Names
23.3
19.3
37.8
61.3
24.4
Salient Translation Error Detection
37.3
47.3
37.1
16.0
16.2
36.0
Snarks
50.0
54.2
53.9
55.6
66.7
56.9
Sports Understanding
56.0
54.7
55.1
56.5
54.0
56.7
Temporal Sequences
16.7
25.1
18.2
25.1
37.8
18.2
Tracking Shuffled Objects
12.0
12.0
12.0
13.8
16.9
12.3
(five objects)
Tracking Shuffled Objects
6.7
6.7
6.7
10.0
9.8
7.7
(seven objects)
Tracking Shuffled Objects
24.7
31.1
30.7
30.9
32.0
29.2
(three objects)
Web of Lies Word Sorting
54.0
53.8
54.2
52.7
48.2
50.1
1.3
0.5
1.3
4.9
4.9
1.1
Avg Performance Per Task
27.0
37.3
31.6
37.7
42.1
34.7
Avg Tokens Per Example
111.6
597.8
111.6
111.6
111.6
111.6
Gradient-based Training
No
No
Yes
Yes
length is directly proportional to the inference cost, and thus LoraHub's ability to econ- omize on input tokens while approaching the peak performance grows increasingly sig- nificant. Moreover, as shown in Appendix Table 4, the upper bound performance of our method across these runs can surpass ICL on 18 tasks, demonstrating its potential for fu- ture development.
Even when compared to certain gradient-based optimization methods, our approach con- sistently demonstrates competitive performance. For example, as depicted in Table 1, our method exhibits notable improvement of 3.1% on average in contrast to the promising IA3 method. Nevertheless, we acknowledge that our approach still falls behind LoRA tuning and full fine-tuning, especially tasks that exhibit significant deviation from the upstream task. Taking Dyck Languages example, both LoraHub and ICL achieve

---

Published as a conference paper at COLM 2024
only average performance of nearly 1.0% on these tasks, while LoRA and FFT methods showcase impressive results with only examples.
4.3 Discussion
LoraHub addresses the challenge of reducing inference costs by eliminating the need for processing additional tokens, resulting in noticeable reduction in overall inference penses. However, it introduces an inherent cost during the ADAPT stage, necessitating extra inference steps, such the 40 steps employed in our experiments. This introduces a trade-off between choosing the ICL approach and LoraHub, with the decision typically hinging on the nature of the situation.
For one-time ad-hoc tasks, the ICL approach should be more pragmatic due to LoraHub's additional inference step costs. In such scenarios, where immediate, single-use solutions are preferred, the simplicity and efficiency of ICL might outweigh the benefits of potential savings offered by LoraHHb. Conversely, for recurring or similar tasks, LoraHub emerges as compelling option. Despite the added inference step cost, LoraHub's 's ability to ef- ficiently handle repetitive tasks, often occurring thousands of times, while concurrently reducing overall expenses, positions it as viable option in such kind of situations.
In summary, our intention is not to replace ICL, but to present LoraHub as a complemen- trade-offs. Thus, we encourage careful consid-
tary strategy with performance- eration of specific use cases and requirements when choosing between ICL and LoraHub, recognizing that the optimal solution may vary based on the nature and frequency of the tasks at hand.
5 Experimental Analysis
In this section, we thoroughly examine the characteristics of our proposed method and uncover several insightful findings. If not specified, we use FLAN-15-large for all analysis.
Does composing LoRA modules extend beyond the single module's benefits?
We acknowledge the investiga- tion of cross-task performance in prior work (Jang et al., 2023), which delved into the capabilities of LoRA and proposed a novel method centered around LoRA module retrieval. In order to en- fair comparison, we con- ducted an experiment where we designed a LoRA retrieval mechanism based on the loss derived from few-shot examples. Specifically, we ranked all LoRA module candidates according to this loss and evaluated the best candidate on the test of the unseen task. As depicted in Table 2, the performance of LoRA retrieval is notably impressive, positioning it as strong baseline. However, in comparison LoraHub, the performance of LoRA retrieval is relatively less favorable
Table 2:
The average performance of various methods across all tasks in the benchmark BBH.
LoRA Retrieval
LoraHub avg.
LoraHub best
31.7
34.7
41.2
How effective is the gradient-free optimization method?
To assess the effectiveness of our gradient-free optimization method in correctly identi- fying the most suitable LoRA module for given downstream task, we carried out an empirical study using the WikiitaleeQuestions (Pasupat & Liang, 2015) (WTQ) dataset. We strategically included LoRA module that was specifically trained on the WTQ dataset into our pool of LoRA candidate modules, which originally stemmed from tasks exclusive to the Flan Collection. Subsequently, we designated WTQ as the targeted downstream task and computed the weights consistent with the methods employed in LoraHub learning. As an end result, the WTQ-specific LoRA module was awarded the highest weight,
7

---

Published as a conference paper at COLM 2024
emplifying the algorithm's success in recognizing it as the most relevant. Moreover, the combined LoRA module demonstrated marginal superiority over the WTQ LoRA module. This underscores the claim that the gradient-free optimization method has the ability to proficiently select the optimal upstream LoRA module for an unseen task.
Can LoraHub work well on non-instruction-tuning models?
In previous investigations, we primarily focused on models with zero-shot capabilities that were trained with instruction tuning. However, for models like T5 without zero-shot abili- ties, where training has a larger effect on parameters, it was unclear if LoraHub could still effectively manage and improve them. Our experiments show that although these mod- els perform worse than FLAN-15, LoraHub learning can still enable them to effectively generlize unseen tasks. See Appendix C for more details.
Will the rank of LoRA modules impact the performance of LoraHub learning?
The parameter rank plays crucial role in the LoRA framework, directly influencing the number of trainable parameters utilized during LoRA tuning. This prompts an intriguing question: does the variation in rank values influence the outcomes observed within the Lo- raHub learning? Our analysis indicates that, for FLAN-T5, the choice of rank has minimal impact. However, for T5, it still exerts some influence. Empirical findings reveal that, in comparison to rank values of 4 or 64, a rank value of 16 consistently demonstrates superior performance across different runs, both in terms of average and optimal values. Additional results are available in Appendix C.
Does more LoRA modules lead to better results?
In our main experiments, randomly selected 20 LoRA modules for LoraHub learning. Therefore, we conducted experiments to investigate the effect of using different numbers of LoRA modules. The results demonstrate that modules, the variance in performance increased. However, the maximum achievable per- formance also improved. More analysis on the variance and the detailed results can be found in Appendix H.
increased the number of LoRA
How much computational resource can be saved?
We follow to the memory test settings from the LoRA-FA (Zhang et al., 2023b) study for an accurate benchmark. In this context, full fine-tuning required about 40GB of memory, whereas LoRA fine-tuning used around 34GB. Remarkably, Loralub only utilized about 5GB of memory, illustrating its efficiency due to the inference-only mode, which eliminates the need for storing gradients and optimization states.
Related work
Model Merging Our method substantially draws on the concept of LoRA module com- position, and thus, aligns with the significant thread of research in model merging. This research focus broadly categorized based on the ultimate objectives of model merging.
The first category focuses on merging entire models, and the goal is to combine individ- ually trained models to approximate the performance benefits of model ensembling or multi-task learning. Prior works (Matena & Raffel, 2021; Jin et al., 2023; Yadav et al., 2023; Wu et al., 2023a) operated under the assumption of shared model architectures. For ex- ample, Matena & Raffel (2021) amalgamates models by approximating Gaussian posterior distributions garnered from Fisher information, while Yadav et al. (2023) merges models via resolving model interferences. Another approach is merging models with different ar- chitectures. For instance, Ainsworth et al. (2023) configures weights of different models prior to their merger. Following this objective, Stoica et al. (2023) merges models operating
8

---

Published as a conference paper at COLM 2024
varying tasks by identifying common features, without requiring additional training. Unlike these works, our work focuses on merging models for better cross-task generaliza- tion.
The second category most closely aligns with our research, stemming from a shared mo- tivation of module composition. Various scholars have made advances in this line of re- search: Kingetsu et al. (2021) decomposes and recomposes modules on the basis of their functionality; Ilharco et al. (2023) proposes modulating model behavior using task vectors; Lv et al. (2023) amalgamates parameter-efficien modules weighted according to task simi- larity; Zhang et al. (2023a) crafts modules by employing specific arithmetic operations; Sun et al. (2023) improves few-shot performance of unseen tasks by multi-task pre-training of prompts; Chronopoulou et al. (2023) averages adapter weights intended for transfer; Ponti et al. (2023) focuses on jointly learning adapters and a routing function that allocates skills to each task; and Muqeeth et al. (2023) concentrates on amalgamating experts in mixture of experts models; However, these methods generally necessitate multi-task training or hu- man prior on module selection for the downstream task. In contrast, our method does not impose any special training requirements and simply employs vanilla LoRA tuning. Addi- tionally, the module selection for downstream tasks is entirely data-driven without human prior knowledge. This design gives the advantage of easily adding new LoRA modules for reuse, allowing our method to flexibly scale up the number of LoRA module candidates in the future.
Mixture of Experts The Mixture of Experts (MoE) is an ensemble method, often visual- ized as a collection of sub-modules, or "experts", each specializing in processing different types of input data. Each expert in this system is controlled by a unique gating network, activated based on the distinct nature of the input data. For every token in these input se- quences, this network identifies and engages the most suitable experts to process the data. As a result, the performance is superior compared to relying on a single, generic model for all types of input. This technique has proven instrumental in numerous domains, such as natural language processing and computer vision (Jacobs et al., 1991; Shazeer et al., 2017; Du et al., 2022; Zhang et al., 2022; Wang et al., 2022; crumb, 2023). Our methodology dis- plays similarities to MoE, wherein upstream-tran trained LoRA modules can be aligned with MoE's expert design. A noteworthy distinguishing factor is that our approach mechanism does not require any specialized manipulation of LoRAs during training while facilitat- ing dynamic LoRA module assembly at any scale, each pre-tuned contrast, MoE mandates predetermined count of experts during both the training and testing phases. Recent studies on the interrelation between MoE and instruction tuning have demonstrated that the simultaneous application of both approaches enhances the ef- fectiveness of each individually (Shen et al., 2023).
different tasks. In
Cross-Task generalization Recent advancements like CrossFit (Ye et al., 2021), ExT5 (Aribandi et al., 2022), FLAN (Wei et al., 2022), TO (Sanh et al., 2022), Instruct- GPT (Ouyang et al., 2022), and ReCross (Lin et al., 2022) have been striving to foster vastly multi-task model's generalization across different tasks, very much aligned with the objectives of our research. Among this cohort, the connections of CrossFit and ReCross with LoraHub are particularly noteworthy. The CrossFit framework (Ye et al., 2021) man- dates a minimal number of labeled examples of the target task for few-shot fine-tuning. However, its limitation lies in the application of task names as hard prefixes in templates, posing challenges in the task's generalization. On the other hand, while ReCross mitigates the need for labels in few-shot examples for retrieval, it necessitates a fine-tuning process using the retrieved data. This procedure appears time-consuming when compared to Lo- raHub's 's approach. Through the deployment of few-shot labeled examples and gradient- free optimization process, LoraHub facilitates an iterative update of weights to compose the LoRA modules. The resultant method is more efficient and cost-effective relative to previous work. Overall, LoraHub offers a more practical and viable solution to the opti- mization process.

---

Published as a conference paper at COLM 2024
7 Conclusion
In this work, have introduced LoraHub, a strategic framework for composing LoRA modules trained on diverse tasks in order to achieve adaptable performance on new tasks. Our approach enables the fluid combination of multiple LoRA modules using just few examples from a novel task, without requiring additional model parameters or human expertise. The empirical results on the BBH benchmark demonstrate that LoraHub can effectively match the performance of in-context learning in few-shot scenarios, removing the need for in-context examples during inference. Overall, our work shows the promise of strategic LoRA composability for rapidly adapting LLMs to diverse tasks. By fostering reuse and combination of LoRA modules, we can work towards more general and adapt- able LLMs while minimizing training costs.
Reproducibility Statement
The authors have made great efforts to ensure the reproducibility of the empirical results reported in this were described in detail in Section 4.1. Secondly, the codes and script for reproduce the result will be opensource after accepted. Second, the source code implementing the pro- posed method and experiments will be made publicly available paper. Third, pre-trained LoRA modules from this work along with their configuration files and weights will be shared. These allow reproduction without retraining the LoRA modules, enabling quick testing and verification.
Firstly, the experiment settings, evaluation metrics, and datasets
upon acceptance of the
References
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023.
Shengnan An, Yifei Li, Zeqi Lin, Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng, and Jian-Guang Lou. Input-tuning: Adapting unfamiliar inputs to frozen pre- trained models. ArXiv preprint, 2022.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Prakash Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-tast scaling for transfer learning. In Proc. of ICLR, 2022.
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. PromptSource: An integrated development environment and repository for natural lan- guage prompts. In Proc. of ACL, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Sys- tems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
10

---

Published as a conference paper at COLM 2024
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. CoRR, abs/ 2205.14555 2023. doi: 10.48550/ ARXS5/032X...305 14788. URL https://doi. org/1...//.. 48550/ arriv. 2305. 14788.
Alexandra Chronopoulou, Matthew Peters, Alexander Fraser, and Jesse Dodge. Adapter- Soup: Weight averaging to improve generalization of pretrained language models. In Findings of the Association for Computaatonnl Linguistics: EACL 2023, 2023.
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, An- drew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling cnttrructoo-finntueen language models. ArXiu preprint, 2022.
crumb.
Llama-2,
mixutre of lora.
https://crumbss hww.umbyyy.ddduu.... com/
1lama- aatnaa--mmoorrr--f0044555
2023.
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fe- dus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstemn, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, Proceedings of Machine Learning Research, 2022.
of language models with mixture-of-expeet
Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in large language model. CoRR, abs/2307.06945 2023. doi: 10. 48550/ARXIV.2307.00
2..40.99... URL https://dof http:://do..org/10... 48555//rr.v. 06945.
Aryo Pradipta Gema, Luke Daines, Pasquale Minervini, and Beatrice Alex. Parameter- efficient fine-tuning of llama for the clinical domain. ArXiv preprint, 2023.
Nikolaus Hansen and Andreas Ostermeier. Adapting arbitrary normal mutation distri- butions in evolution strategies: the covariance matrix adaptation. Proceedings of IEEE International Conference on Ecolutionary Computation, 1996.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpand Kirk-arkkatrik, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. In Proc. of ICLR, 2022.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In Proc. of ICLR, 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh Interna- tional Conference on Learning Representations, 2023.
Robert A. Jacobs, Michael I. Jordan, Steven Nowlan, and Geoffrey E, Hinton. Adaptive mixtures of local experts. Neural Computation, 1991.
Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. In International Conference on Machine Learning, 2023.
URL /ttps://api. semanticscholar wwt.ciccoo....ooooorousssseeeeee 100C0rpps11:22627777.. 256627673.
Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LImlingua: Com- pressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, December 2023a. URL https: //tps//////iii///// www.iii..g//aaa.... org//bb//2100007.66
11

---

Published as a conference paper at COLM 2024
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing Ilms in long context scenarios via prompt compression. CoRR, abs/ abs/2330.06839, 2023b. doi: 10.48550/ ARXIV:2310.06999, URL https://doi. http///oooor/o////////// wtt.10...5500//rkkii 48550/arXiv. 2310. 06839.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng, Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023.
Hiroaki Kingetsu, Kenichi Kobayashi, and Taiji Suzuki. Neural network module decom- position and recomposition. ArXiv preprint, 2021.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proc. of EMNLP, 2021.
Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, Association for Computational Linguistics, December 2023. URL https: /ttpss////xxv...// www.oogg//bbbbeeeee 23t///2111..001....
Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. Unsupervised cross-task generalization via retrieval augmentation. In NeurlPS, 2022.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in- context learning. ArXiv, abs52205.05638, 2022. URL https: /tatps/:/apii. semanti semanticscholarr org/
20r99SID:24469328.2
Jialin Liu, A. Moreau, Mike Preuss, Baptiste Rozière, Jérémy Rapin, Fabien Teytaud, and Olivier Teytaud. Versatile black-box optimization. Proceedings of the 2020 Genetic and Evolutionary Computation Conference, 2020.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023.
Xingtai Lv, Ning Ding, Yujia Qin, Zhiyuan Liu, and Maosong Sun. Parameter-efficient
weight ensembling facilitates task-level knowledge transfer. In Annual Meeting of the Association for Computational Linguistics, 2023.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://gitie thub. com/ huggingface/pefft. 2022.
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. Arxiv preprint, 2021.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task gen- eralization via natural language crowdsourcing instructions. In Proc. of ACL, 2022.
Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adap- tive routing. ArXiv preprint, 2023.
OpenAI. ChatGPT. 2022. URL hwtph:ttp////eeeee http::////ppaaa..ooooooo//aaagg/
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to fol- low instructions with human feedback. ArXiv preprint, 2022.
12

---

Published as a conference paper at COLM 2024
Panupong Pasupat and Percy Liang. Compositional semantic parsing semi-structured tables. In Proc. of ACL, 2015.
Edoardo Maria Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining parameter-efficient modules for task-level generalisation. In Proceedings of the 17th Con- ference of the European Chapler of the Association for Computational Linguistics, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 2020.
J. Rapin and O. Teytaud. Nevergrad A gradient-free optimization platform. https://
GitHub. htt////ccooooeeeere////eeerrrrre 2018.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In Proc. of ICLR, 2022.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparselyy mixture- of-experts layer. In Proc. of IČLR, 2017.
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. Mixture-of-experts meets instruction tuning: winning combination for large language models, 2023.
George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffman. Zipit! merg- ing models from different tasks without training. ar arviv, 2023.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tun- ing for -anguiag-moddel-s----seeet
In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, Proceedings of Machine Learning Research, 2022.
Tianxiang Sun, Zhengfu He, Qin Zhu, Xipeng Qiu, and Xuanjing Huang. Multitask pre- training of modular prompt for Chinese few-shot learning. In Proc. of ACL, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv preprint, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. Vish- wanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017.
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. AdaMix: Mixture-of-adappatioon for parameter-efficient model tuning. In Proc. of EMNLP, 2022.
13

---

Published as a conference paper at COLM 2024
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc Le. Finetuned language models are zero-shot learners. In Proc. of ICLR, 2022.
Chengyue Wu, Teng Wang, Yixiao Ge, Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Luo. N-tuning: Transferring multimodal foundation models with optimal multi-task inter- polation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Ma- chine Learning Research, pp. 37713-37727. PMLR, 2023a. URL https: /ptp//eeeoocc....e//eepepo mlr. press/v202/ wu23t. html.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David S. Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. CoRR, abs/ 2303.17564 2023b. doi: 10.48550/ 23Xiv.2333.
17564. URL https://doo /ttp:////.o..././////////// 48550/ aato///rxi.../ 2303. 17564.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES- merging: Resolving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openrevien wtt.eewwwwwwewtnnniiiii.ee xtaX3WyCj1.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. CrossFit: A few-shot learning challenge for cross-task generalization in NLP. In Proc. of EMNLP, 2021.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural archi- tecture search. In Proc. of ICLR, 2019.
Fan Zhang, Duyu Tang, Yong Dai, Cong Zhou, Shuangzhi Wu, and Shuming Shi. Skillnet- nlu: A sparsely activated model for general-purpose natural language understanding, 2022.
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operations. ArXiv preprint, 2023a.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, ArXiv, Obs//200.003003, 2023b. URL hnttps://pp. semanti tescooicc orgp/oorpus1:::26882555
Wangchunshu Zhou, Yuchen Eleanor Jiang, Ryan Cotterell, and Mrinmaya Sachan. Ef- ficient prompting via dynamic in-context learning. CoRR, abs/2305.11177,, 2023, doi: 10.45550/ARKX1222055.111 11170. URL ht https::///t /ttp:://dooooooo
/ttp://.oo/..../5..///////////// htgg//..4550//rx..220. 11170.
14

---

Published as a conference paper at COLM 2024
Table 3: The top five beneficial LoRA modules for BBH tasks and their associated upstream tasks, the average weight values and the average performance on all BBH tasks.
Rank
Dataset: Task
Weight
Perf
Task Description
1
WIQA: Last Process
0.72
28.1
Identifying the last step of given process.
2
RACE: Is this the Right Answer
0.68
30.8
Determining if given answer is correct.
3
WIQA: First Process
0.63
28.1
Identifying the first step of a given process.
Answering question created by an adversarial model-in-the-loog
4
AdversarialQA: BiDAF
0.61
25.1
Answering question based on information extracted from the web.
5
WebQuestions: What the Answer
0.58
27.0
A
More Analysis
Which LoRA modules are most effective for BBH tasks?
We hypothesized that the amalgamation of LoRA modules could incorporate skills and insights from a variety of specific tasks. To evaluate this, we examined the extent of influ- ence single LoRA module had amongst all tasks from the BBH benchmark. We measured the impact of each isolated task by calculating the average absolute weight. The top five modules, presented in Table 3, were found to have substantial influence, as indicated by their maximum average weights, which suggested that they were notably more effective in cross-task transfer. Remarkably, a common feature among these top five modules was their association with tasks requiring reading comprehension and reasoning skills-attribine attributes indicative of higher cognitive complexity. However, it is worth noting that none of the modules exhibited consistent improvement across all BBH tasks, as reflected in their av- erage performance on all BBH tasks, which did not show a significant improvement com- pared to the original FLAN-T5-large, except for the Rank 2. The results underscore the advantages of composing diverse modules in LoraHub.
How effective is the gradient-free optimization method?
To assess the effectiveness of our gradient-free optimization method in correctly identi- fying the most suitable LoRA module for given downstream task, we carried out an empirical study using the WikiTableeQesstions (Pasupat Liang, 2015) (WTQ) dataset. We strategically included LoRA module that was specifically trained on the WTQ dataset into our pool of LoRA candidate modules, which originally stemmed from tasks exclusive to the Flan Collection. Subsequently, we designated WTQ as the targeted downstream task and computed the weights consistent with the methods employed in LoraHub learning. As end result, the WTQ-specific LoRA module was awarded the highest weight, emplifying the algorithm's success in recognizing it as the most relevant. Moreover, the combined LoRA module demonstrated marginal superiority over the WTQ LoRA module. This underscores the claim that the gradient-free optimization method has the ability to proficiently select the optimal upstream LoRA module for an unseen task.
B Result of Best Results
As shown Table 4, compared to gradient-based parameter-efficient training methods like LoRA and IA3, approach demonstrates superior performance in terms of best results it over experimental runs. exhiibis noticeable lag behind the fully fine-tuning (FFT) method, which updates all parameters during training, this observation suggests that our proposed method has promising upper limit. We anticipate that future research efforts can contribute to accelerating the optimization speed and further enhancing the efficacy of our approach.
15

---

Published as a conference paper at COLM 2024
Table 4: Experimental results of several few-shot methods, including in-context learning (ICL), IA3 fine-tuning (IA3), LoRA tuning (LoRA), full fine-tuning (FFT) and our LoraHub learning (LoraHub) on the BBH benchmark with FLAN-T5-large as the base LLM. We de- note algorithmic tasks with the superscript Ş following previous work (Wu et al., 2023b). Note that we use 5 examples per task as the demonstration for all methods. The best (best) performance reported as the maximum value obtained across three runs.
Task
ICI best
IA3
LoRAbest
FFT best
LoraHubpest
Boolean Expressions
62.7
58.0
60.7
65.3
60.7
Causal Judgement
59.8
62.1
57.5
60.9
63.2
Date Understanding
21.3
20.7
40.7
67.3
45.3
Disambiguation
69.3
0.0
68.7
70.7
68.0
Dyck Languages
2.0
4.7
25.3
33.3
2.7
Formal Fallacies
59.3
52.0
56.7
56.0
59.3
Geometric Shapes
20.0
15.3
28.7
39.3
18.7
Hyperbaton
72.7
49.3
57.3
82.0
72.7
Logical S
39.3
32.7
41.3
43.3
40.0
(five objects)
Logical Deduction (seven objects)
42.0
34.0
42.7
46.0
Logical Deduction
52.7
8.7
56.7
60.7
52.7
(three objects)
Movie Recommendation
56.7
62.0
64.5
70.7
62.0
Multistep Arithmetic
0.7
0.7
0.7
0.0
1.3
Navigate
46.7
47.3
5U.7
50.0
51.3
Object Counting
34.7
35.3
42.0
38.0
36.7
Penguins in Table
43.5
45.7
41.3
37.0
47.8
Reasoning about Colored Objects
41.3
41.3
40.7
38.7
44.7
Ruin Names
20.7
25.3
42.0
66.0
Salient Translation Error Detection
48.0
37.3
17.3
21.3
42.7
Snarks
55.1
56.4
59,0
69.2
61.5
Sports Understanding
56.7
55.3
58.7
58.7
62.7
Temporal Sequences
26.7
18.7
31.3
48.7
21.3
Tracking Shuffled Objects
12.0
12.0
16.0
20.0
(five objects)
Tracking Shuffled Objects
6.7
6.7
12.0
10.0
15.3
(seven objects)
Tracking Shuffled Objects
31.3
30.7
32.0
36.0
31.3
(three objects)
Web of Lies Word Sorting
54.0
54.7
55.3
54.0
57.3
0.7
1.3
5.3
6.0
1.3
Best Performance (Average)
38.4
32.1
40.9
46.2
41.2
16

---

Published as a conference paper at COLM 2024
C Result of non-instrcution-tuned models
Table 5: Comparsion among different ranks for few-shot LoraHub learning with the back- bone 15-large (Raffel et al., 2020) on the BBH benchmark. Note that the 15-large model achieved 0.0% on all tasks under the zero-shot setting except Dyck Languages, where it scored 0.67%.
Task Rank ++
4avg
best
16avg
16best
64avg
64best
Boolean Expressions
52.13
57.33
50.67
58.00
47.47
58.00
Causal Judgement
52.41
55.17
49.66
54.02
50.80
54.02
Date Understanding
0.40
2.00
14.40
29.33
4.53
10.00
Disambiguation
10.00
31.33
26.93
42.00
1.73
4.67
Dyck Languages
0.40
0.67
0.40
0.67
0.40
2.00
Formal Fallacies
48.40
54.00
46.93
51.33
46.93
50.00
Geometric Shapes
0.00
0.00
6.53
32.67
1.47
7.33
Hyperbaton
30.13
50.00
39.07
57.33
32.93
48.00
Logical Deduction
5.20
14.67
8.80
19.33
1.33
6.67
(five objects)
Logical Deduction
6.40
17.33
9.33
19.33
3.47
16.00
(seven objects)
Logical Deduction
14.40
32.00
21.73
34.67
6.93
15.33
(three objects)
Movie Recommendation
7.07
18.67
7.87
22.00
1.20
6.00
Multistep Arithmetic two
0.00
0.00
0.00
0.00
0.00
0.00
Navigate
49.60
54.67
52.27
56.67
49.87
52.00
Object Counting
7.20
18.00
16.00
21.33
13.73
26.67
Penguins in a Table
6.52
13.04
10.43
17.39
0.43
2.17
Reasoning about Colored Objects
6.27
10.00
5.07
16.67
0.53
2.67
Ruin Names
7.73
13.33
13.20
28.00
5.73
15.33
Salient Translation Error Detection
0.00
0.00
1.73
8.67
0.00
0.00
Snarks
21.28
42.31
49.49
60.26
16.15
38.46
Sports Understanding
46.53
58.67
46.80
58.67
46.53
58.67
Temporal Sequences
3.07
13.33
6.53
26.67
2.40
12.00
Tracking Shuffled Objects
5.20
14.00
4.13
9.33
0.13
0.67
(five objects)
Tracking Shuffled Objects
2.67
10.00
2.80
14.00
3.20
8.00
(seven objects)
Tracking Shuffled Objects
3.73
17.33
16.27
34.67
5.87
26.67
(three objects)
Web of Lies Word Sorting
48.53
54.00
54.00
56.00
54.67
57.33
0.40
0.67
0.13
0.67
0.00
0.00
Average Performance per Task
16.14
24.17
20.78
30.73
14.76
21.43
17

---

Published as a conference paper at COLM 2024
D Result of larger model
Table 6: Experimental results zero-shot learning (Zero) and our few-shot LoraHub learn ing (LoraHub) the BBH benchmark with FLAN-T5-xI the base LLM. Note that use 5 examples per task as the demonstration for both ICL and LoraHub. The average (avg) per- formance of LoraHub is computed over 5 runs with different random seeds, while the best (best) performance is reported as the maximum value obtained across these runs. We can see the trend of the results are similar to FLAN-T5-large.
Task
Zero
LoraHub avg
LoraHub best
Boolean Expressions
52.0
58.7
63.3
Causal Judgement
62.1
53.8
59.8
Date Understanding
38.0
37.6
38.0
Disambiguation Qa
0.0
20.5
54.7
Dyck Languages
1.3
0.9
2.0
Formal Fallacies
56.0
56.0
56.0
Geometric Shapes
8.7
17.5
28.0
Hyperbaton
45.3
53.5
56.7
Logical Deduction
1.3
42.7
48.7
(five objects)
Logical Deduction
8.7
44.3
50.0
(seven objects)
Logical Deduction
0.7
56.4
61.3
(three objects)
Movie Recommendation
2.0
62.8
66.0
Multistep Arithmetic Two
0.0
0.4
0.7
Navigate
50.7
50.7
50.7
Object Counting
39.3
40.7
48.0
Penguins In Table
17.4
40.9
45.7
Reasoning About Colored Objects
46.7
47.3
50.7
Ruin Names
18.0
35.6
44.7
Salient Translation Error Detection
44.7
45.1
48.7
Snarks
60.3
60.8
61.5
Sports Understanding
56.7
51.3
53.3
Temporal Sequences
21.3
21.5
22.0
Tracking Shuffled Objects
3.3
9.9
13.3
(five objects)
Tracking Shuffled Objects
5.3
7.3
8.7
(seven objects)
Tracking Shuffled Objects
7.3
21.7
31.3
(three objects)
Web Of Lies Word Sorting
54.7
47.1
48.7
1.3
1.5
2.0
Average Performance per Task
25.8
36.5
41.3
18

---

Published as a conference paper at COLM 2024
E Improving the Robustness of LoraHub
In order to enhance the robustness of LoraHub, we explored straightforward approach in the selection of LoRA module candidates. Specifically, we first identified 20 LoRA module candidates with the lowest loss on the few-shot examples. Our findings indicate slight improvement in overall performance after applying the pre-fitteingg startegy. Since the primary instability in our approach arises from the selection of LoRA candidates. This method involves choosing fixed set of LoRA candidates to ensure the stability of our approach.
Table The experimental results of loss-based pre-filtering.
Task
LoraHubavg
LoraHubfittr
Boolean Expressions
55.5
60.00
Causal Judgement
54.3
52.9
Date Understanding
32.9
33.3
Disambiguation
45.2
62.7
Dyck Languages
1.0
0.0
Formal Fallacies
52.8
54.0
Geometric Shapes
7.4
4.0
Hyperbaton
62.8
64.0
Logical Seduction
36.1
37.3
(five objects)
Logical Deduction (seven objects)
36.8
22.0
Logical Deduction
45.7
56.0
(three objects)
Movie Recommendation
55.3
68.0
Multistep Arithmetic
0.4
0.7
Navigate
47.1
49.3
Object Counting
33.7
38.7
Penguins in Table
35.9
37.0
Reasoning about Colored Objects
40.0
33.3
Ruin Names
24.4
22.0
Salient Translation Error Detection
36.0
24.0
Snarks
56.9
52.66
Sports Understanding
56.7
58.0
Temporal Sequences
18.2
27.3
Tracking Shuffled Objects
12.3
11.3
(five objects)
Tracking Shuffled Objects
7.7
8.0
(seven objects)
Tracking Shuffled Objects
29.2
32.7
(three objects)
Web of Lies
50.1
46.0
Word Sorting
1.1
1.3
Avg Performance Per Task
34.7
35.4
19

---

Published as a conference paper at COLM 2024
F Performance on General Important Task
In our research, we have identified specific LoRA modules that exhibit significant impact when integrated into merged LoRAs. Our focus lies in assessing the performance of the top five task-related LoRAs on the BBH benchmark. The results indicate that these top LoRAs perform similarly or even worse than zero-shot in most cases. Only one of them stands out as significantly better than zero-shot. However, it's worth noting that this performance is not as impressive as Lorahub. These findings support the idea that the merging process can improve overall performance.
Table 8: Detailed experimental results of top five LoRA modules shown in Table on BBH tasks.
lask
WIQA: Last
RACE: Right
WIQA: First
ADQA
WebQA
Boolean Expressions
52.67
58.00
52.67
54.67
53.33
Causal Judgemeen
55.17
63,22
55.17
57.47
57.47
Date Understanding
17.33
19.33
17.33
16.67
15.33
Disambiguation
0.00
0.00
0.00
0.00
0.00
Dyck Languages
0.67
0.67
0.67
1.33
1.33
Formal Fallacies
51.33
51.33
51.33
51.33
51.33
Geometric Shapes
8.00
13.33
8.00
6.67
7.33
Hyperbaton
16.67
44.00
16.67
1.33
6.00
Logical Deduction
23.33
28.00
23.33
19.33
20.67
(five objects)
Logical Deduction (seven objects)
22.00
26.00
22.00
10.67
12.00
Logical SeductionS
0.67
9.33
0.67
0.00
0.00
(three objects)
Movie Recommendation
63.33
62.67
63.33
56.67
63.33
Multistep Arithmetic
0.67
0.67
0.67
0.67
0.67
Navigate
47.33
50.00
47.33
47.33
47.33
Object Counting
34.67
34.00
34.67
35.33
35.33
Penguins in a Table
45.65
41.30
45.65
39.13
43.48
Reasoning about Colored Objects
40.00
37.33
40.00
31.33
30.67
Ruin Names
22.00
21.33
22.00
17.33
22.67
Salient Translation Error Detection
36.67
34.67
36.67
32.67
37.33
Snarks
52.56
55.13
52.56
47.44
52.56
Sports Understanding
56.00
58.67
56.00
55.33
55.33
Temporal Sequences
16.67
17.33
16.67
12.67
17.33
Tracking Shuffled Objects
(five objects)
12.00
12.00
12.00
10.67
12.00
Tracking Shuffled Objects
6.67
6.67
6.67
6.67
6.67
(seven objects)
Tracking Shuffled Objects
20.67
30.67
20.67
10.67
25.33
(three objects)
Web of Lies
54.67
54.00
54.67
54.00
54.00
Word Sorting
1.33
1.33
1.33
1.33
1.33
Avg Performance per Task
28.10
30.78
28.10
25.14
27.04
A FLAN-I5-large
1.10
3.78
1.10
-1.86
0.04
20

---

Published as a conference paper at COLM 2024
booicon bosilconccgreessiog expressions
causal judgement
sate understanding
drammiguutton q3
Syck languoges
20
shapes
Pap
Ingiral
60
35-
20
10
20
16
120
56
102
50
logial deduction thres
modie recom mendation multstup artihmatic two
navigade
obaet counting
6.6
20.0
0.6
41.5
(5.0
0.0
25
30
120
52
120
10
120
20
50
102
50
100
Figure 3: The influence of number of LoRA modules on 15 tasks from BBH, and each box is obtained from 5 separate runs. The horizontal axis shows the number of LoRA modules to be composed in LoraHub learning.
G Implementation details
We implemented LoRA tuning using the Huggingface PEFT library (Mangrulkar et al., 2022), with the rank being set 16. The gradient-free method was implemented using the open-source Nevergrad optimization library (Rapin & Teytaud, 2018), with a constraint that the absolute value of LoRA weights should not exceed 1.5. Originally, all coefficients of LoRA modules were set at zero.
In our standard settings, we set the maximum number of iterations K as 40. The same examples were used during our LoraHub learning and the few-shot in-context learning. The hyperparameter a is set as 0.05. Regarding the hyperparameters for training candidate LoRA modules, we maintained consistency across all modules, setting the batch size at 64, the learning rate at le - 4, and the number of training epochs at 10.
H Influence of Number of LoRA modules
As shown in Figure 3, with an increase in the number of LoRA module candidates, there corresponding increase in the performance variance. Based on our in-depth analysis, the primary source of variance is not related to gradient-free optimization algorithms but rather associated with the LoRA candidate modules. In other words, once the candidates are determined, random seeds have minimal impact on the final performance. Hence, we posit that the observed instability primarily arises from the inherent challenge of balancing the quantity and quality of the LoRA module candidates.
I The Impact of Threshold
In this section, we omitted the threshold in our implementation, and the results are summa- rized in Table 9. Our observations indicate that the removal of the threshold had minimal impact on the majority of tasks, underscoring the robustness of the gradient-free optimiza- tion algorithm itself most cases. The algorithm efficiently identified reasonable ranges even without specific upper and lower bounds. However, three tasks, namely Date Under- standing, Disambiguation and Hyperbaton, exhibited notable effects. The resulting perfor- mance decline led to average decrease of 1 1.2% compared to the setting with threshold.
21

---

Published as a conference paper at COLM 2024
This highlights the significance of establishing reasonable threshold to mitigate extreme scenarios.
Table 9: The comparsion between LoraHub and LoraHub without threshold.
Task
LoraHubavg with threshold
LoraHubayg without threshold
Boolean Expressions
55.5
54.0
Causal Judgement
54.3
54.8
Date Understanding
32.9
17.7
Disambiguation
45.2
40.6
Dyck Languages
1.0
1.1
Formal Fallacies
52.8
51.7
Geometric Shapes
7.4
6.7
Hyperbaton
62.8
55.5
Logical Deduction
36.1
36.5
(five objects)
Logical Deduction (seven objects)
36.8
35.6
Logical Deduction
45.7
49.9
(three objects)
Movie Recommendation
55.3
59.3
Multistep Arithmetic
0.4
0.7
Navigate
47.1
47.6
Object Counting
33.7
34.7
Penguins in a Table
35.9
33.8
Reasoning about Colored Objects
40.0
37,9
Ruin Names
24.4
24.0
Salient Translation Error Detection
36.0
37.1
Snarks
56.9
51.6
Sports Understanding
56.7
55.9
lemporal Sequences
18.2
16.7
Tracking Shuffled Objects (five objects)
12.3
12.3
Tracking Shuffled Objects
7.7
8.5
(seven objects)
Tracking Shuffled Objects
29.2
29.8
(three objects)
Web of Lies Word Sorting
50.1
50.3
1.1
1.3
Avg Performance Per Task
34.7
33.5
22


Reviews:
[{"review_id": "7jqAFRsEFs", "reviewer": "Reviewer_dLQP", "summary": "The paper operates in the task generalization setting, where information (parameters) from seen tasks are used to efficiently generalize to downstream, unseen tasks. This task generalization setting have long been explored in the past but remains (imho) a relevant and interesting problem. Many related papers exist in the literature: the most related papers in this setting are (amongst others) SpoT and Polytropon. The novelty of this paper with respect to these papers is rather small: the authors use LoRAs instead of soft-prompts (as in SPoT) (minor), but use gradient free optimization for task adaptation (major novelty?). It is not clear to me however what is exactly the advantage of gradient-free optimization, i.e. the combination weights and adapters parameters themselves might be very cheaply optimized via gradient-descent on the downstream examples. The paper makes some nice ablations in the appendix and goes a little bit towards selecting best modules to combine or filtering modules. I think some more work in this direction might give a bit more novelty to this paper. I think in its current status this is quite a borderline paper.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Novelty is somewhat limited with respect to existing works (MHR, SPoT, Polytropon)\n\nThank you for highlighting these related works. We appreciate your insights and will ensure to include these references in the revised version. LoraHub employs a unique gray-box method that uses only model output logits, unlike the gradient methods applicable mainly to open-source models in previous work. Furthermore, LoraHub functions solely during inference without requiring fine-tuning, facilitating efficient learning on CPU-only machines.\n\n> Performance on BBH is quite low (uses T5 model), so it might be difficult to get insights from ablations\n\nThank you for your comments on the BBH metric performance. We recognize the concerns about the challenges in deriving insights from ablation studies with lower performance on the T5 model. However, our choice of the FLAN-T5 models as the foundation for our experiments is deliberate and strategic.\n\nThe FLAN-T5 series is highly popular on HuggingFace, known for its proficiency in zero-shot and few-shot learning scenarios and robust problem-solving capabilities. These traits make it an excellent baseline for demonstrating the effectiveness of our methods, particularly in terms of learning efficiency and adaptability.\n\nUsing FLAN-T5 also ensures a fair comparison in zero-shot scenarios. Our integration of LoRA modules, trained solely on the FLAN collection, allows us to maintain consistent conditions across experiments. This approach prevents any potential bias that might arise from introducing additional training data, which could unfairly benefit our method.\n\nMoreover, we have included results from FLAN-T5-XL, a 3B model, in the Appendix to further validate the generalizability of LoraHub across model scales.\n\n> The justification for using gradient-free optimization appears a bit weak, I would love to see comparisons to gradient-based optimization of the LoRA weights\n\nThank you for your comment on the justification for using gradient-free optimization. We understand your interest in seeing a direct comparison between gradient-free and gradient-based optimization of the LoRA weights. Due to the constraints of our timeline, we were unable to include this specific comparison in the current version of our research. However, we acknowledge the value such a comparison would add to our findings and will incorporate the analysis into the revised version.\n\nThank you for reminding and we will fix these typos in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clear and very easy to read\n- Operates in a very well-defined setting and therefore makes a focused contribution\n- Ablations are nice\n- This paper has already had quite some impact in democratizing the \"parameter reuse\" paradigm in the community (been in arxiv since a while)", "reasons_to_reject": "- Novelty is somewhat limited with respect to existing works (MHR, SPoT, Polytropon)\n- Performance on BBH is quite low (uses T5 model), so it might be difficult to get insights from ablations\n- The justification for using gradient-free optimization appears a bit weak, I would love to see comparisons to gradient-based optimization of the LoRA weights (performance vs time)", "questions_to_authors": "You might fix the few typos in the appendix:\n\nNON-INSTRCUTION-TUNED -> Non-Instruction Tuned\nComparsion -> Comparison", "extra_scores": {}}, {"review_id": "R88WwIaBAK", "reviewer": "Reviewer_7Jht", "summary": "This paper investigates the issue of efficiently combining multiple LoRAs for effective cross-task generalization. The proposed method, LoraHub, dynamically integrates multiple LoRA modules—each trained on distinct tasks—to enhance performance on new, unseen tasks. LoraHub is designed to efficiently integrate these modules without requiring human intervention, additional parameters, or gradients. Further experiments conducted using the BBH benchmark demonstrate that LoraHub can emulate in-context learning in few-shot scenarios, effectively eliminating the need for in-context examples during inference. Additionally, this approach encourages community-driven model sharing, which could significantly boost progress in general intelligence applications and the deployment of large language models in production environments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Model's applicability and potential use cases\n\nThank you for pointing out the performance differences between LoraHub and other methods like ICL and gradient-based methods. We acknowledge the concerns regarding observed performance gaps in our experiments.\n\nLoraHub offers distinct advantages with its gray-box approach that requires only model output logits, avoiding the need for access to model weights and extensive GPU memory. This allows LoraHub to function efficiently in inference-only mode, ideal for CPU-only environments, simplifying deployment and reducing computational demands. Furthermore, as documented in Appendix B, LoraHub achieves a peak performance of 41.2 on the BBH benchmark, exceeding ICL's 38.4 and all other baselines except full fine-tuning, underscoring its potential superior performance.\n\nYour feedback is invaluable in our ongoing efforts to refine LoraHub, enhancing its performance and extending its practical applications to meet real-world needs.\n\n> Retriever mechanism\n\nWe appreciate your critical insights regarding our LoRA module selection method in LoraHub. We understand your concerns about the apparent arbitrariness of our current selection process and its potential impact on downstream task performance.\n\nIn response, we have introduced $\\rm LoraHub_{filter}$ in Appendix E. This new approach enhances the selection process by identifying the top 20 LoRA module candidates that exhibit the lowest loss on a set of few-shot examples, helping to standardize and improve selection accuracy. Our tests indicate an improvement in average performance from 34.7 to 35.4, demonstrating the benefits of a more sophisticated retriever mechanism for module selection.\n\n> Comparison of resource requirement\n\nThank you for emphasizing the importance of quantifying computational resource consumption in our studies. Our research primarily targets memory usage, as time requirements can vary with different epochs, dataset sizes, and GPU specifications.\n\nWe follow to the memory test settings from the LoRA-FA study for an accurate benchmark. In this context, full fine-tuning required about **40GB** of memory, whereas LoRA fine-tuning used around **34GB**. Remarkably, LoraHub only utilized about **5GB** of memory, illustrating its efficiency due to the inference-only mode, which eliminates the need for storing gradients and optimization states. This comparison will be included in the revised version to clearly highlight LoraHub's resource efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper introduces a novel method for achieving effective cross-task generalization with few-shot samples by reusing and combining existing LoRA modules for black-box optimization.\n2.\tExperimental results on the BBH benchmark indicate that LoraHub's performance rivals that of in-context learning, while processing fewer tokens and eliminating the need for gradients.\n3.\tThe writing is clear and easy to follow.\n4.\tThe authors have open-sourced a collection of diverse LoRAs, greatly benefiting further research within the community.", "reasons_to_reject": "1.\tThe experimental results significantly lag behind other baselines, particularly when using many tasks' LoRAs with few-shot samples, where the model's performance remains weaker than that of in-context learning. Furthermore, the model substantially underperforms compared to other gradient-based methods. The authors need to consider more carefully the model's applicability and potential use cases.\n2.\tLoraHub randomly selects 20 LoRAs from the LoRA pool for combination, which seems arbitrary. This approach risks selecting LoRAs significantly mismatched to the downstream task while overlooking those from similar tasks, thus degrading model performance. Recent studies [1-2] have successfully employed a retriever mechanism to identify LoRAs that are more analogous to the downstream task for composition. Incorporating similar technology could refine LoraHub's selection process, making it more effective.\n3.\tThe authors point out that gradient-based methods are resource-intensive. Therefore, it is crucial to quantify the computational resource consumption of different methods in the experiments to highlight LoraHub's advantages. This comparison was neglected in the conducted experiments.\n\n[1] Chronopoulou, Alexandra, et al. \"Adaptersoup: Weight averaging to improve generalization of pretrained language models.\" arXiv preprint arXiv:2302.07027 (2023).\n\n[2] Zhao, Ziyu, et al. \"LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild.\" arXiv preprint arXiv:2402.09997 (2024).", "questions_to_authors": "See reasons to reject.", "extra_scores": {}}, {"review_id": "X9aFzb61hT", "reviewer": "Reviewer_3rz1", "summary": "This paper introduces LoraHub a framework to compose multiple LoRA modules to improve performance on unseen task. LoraHub involves two main steps compose and adapt. Compose involves composing different LoRA modules by introducing weights to weigh the contribution of each module and adapt involves finding the optimal weights so as to get good performance on unseen task. To optimize this the paper uses CMA-ES gradient free optimizaton framework to find the optimal weights using the few shot samples. The paper also claims that they can get performance close to ICL by using the LoraHub approach on BBH tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Gap between performance of ICL and LoraHub\n\nThank you for your feedback regarding the performance gap between LoraHub and ICL. We acknowledge concerns about the current performance of our model.\n\nOur results show a gap compared to ICL, which we attribute primarily to our current implementation strategy rather than to intrinsic limitations of the LoraHub model. We are confident that refining our module composition algorithms can significantly narrow this gap. In Appendix B, we demonstrate LoraHub's potential, achieving a score of 41.2 on the BBH, surpassing ICL's 38.4. This illustrates LoraHub's capabilities and its potential to excel under certain conditions.\n\nLoraHub also provides significant advantages in terms of inference latency, crucial for applications requiring quick response times. LoraHub enhance efficiency by reducing the number of input tokens needed, which is vital in latency-sensitive environments.\n\nWe are committed to the ongoing enhancement of LoraHub and believe it has considerable potential across various applications. The insights from these initial challenges are invaluable, aiding in the enhancement of LoraHub's applicability and performance across a broader task spectrum. We value this opportunity to further refine and optimize LoraHub, ensuring it reaches its full potential in practical settings.\n\n> The number of LoRA modules \n\nWe adopted this strategy primarily for computational efficiency. Increasing the number of LoRA modules enhances the dimensionality of the optimization problem, significantly affecting the time needed to find an optimal configuration. To manage complexity and balance performance with computational feasibility, we chose to use a subset of 20 modules.\n\nThis method is supported by our findings in Appendix H, where we investigated the impact of increasing the number of optimization steps relative to the number of modules. These experiments indicated that although it is feasible to scale up the number of modules, this requires more optimization time, emphasizing the computational trade-offs involved.\n\nUsing a fixed number of modules also demonstrates the stability and robustness of our approach under controlled conditions, essential for evaluating the effectiveness of the LoraHub method in practical applications. This strategy allows us to systematically assess the impact of module selection and composition without the overwhelming computational overhead associated with a larger set of variables.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Good framework for composing LoRA modules and an interesting research direction, it can help to reuse lot of compute i.e. different people can upload their models and users can choose to compose them using LoRAHub to solve different downstream tasks.\n- Results on diverse baselines including full fine tuning, LORA retrieval and in-context learning.\n- Good set of ablation studies on non-instruction tuned models as well.\n- Inference cost is lower in comparison to in context learning as we don't need to feed in examples every time we prompt the model.", "reasons_to_reject": "- Decent gap between performance of ICL and LoraHub (on average by 3 points on different datasets reported in the paper).", "questions_to_authors": "- In the main text of the paper the authors used randomly selected 20 LoRA modules as a pool of modules for composing them. Is there any reason why not always use all available LoRA modules for all experiments? As the number of LoRa modules increases is there any impact on computational time to find optimal $w_i$", "extra_scores": {}}, {"review_id": "oTQgasLcuv", "reviewer": "Reviewer_YvRA", "summary": "The authors explore how to exploit and leverage lora modules to be effective for few-shot learning without the additional inference cost of updating the prompt. They do so by tuning lora modules through compose and adapt iterations to adjust the coefficients of the module without any gradient updates, allowing the method to be lightweight and scale to any task at hand allowing for cross-task generalization. They demonstrate that their proposed method is competitive with few-shot ICL setups and only slightly lagging behind lora fine-tuning.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "> The decision between ICL and LoraHub\n\nThank you for your insights regarding the trade-offs between LoraHub and ICL. Your observations are critical for understanding the practical implications of these methodologies.\n\nLoraHub is designed to significantly reduce inference costs by eliminating the need for additional input tokens, which decreases overall computational expenses. However, this efficiency comes with the necessity to adjust LoRA module coefficients during the Adapt stage, introducing extra inference steps.\n\nThis presents a fundamental trade-off: for one-time or ad-hoc tasks, ICL may be more suitable due to its simplicity and immediacy, which are advantageous in scenarios that require quick solutions. On the other hand, LoraHub excels in repetitive or frequently recurring tasks. Despite the initial overhead, it can offer considerable long-term efficiency gains. The minimized need for continual input processing makes LoraHub a cost-effective choice for applications handling many similar tasks, justifying the initial setup costs over time.\n\nUltimately, the decision between LoraHub and ICL hinges on the specific needs and frequency of the tasks. Each method provides distinct advantages depending on the operational context and objectives.\n\n> Hybrid approach\n\nThank you for your suggestion about incorporating few-shot examples to reduce adjustment iterations for the LoRA modules. We have not explored this hybrid approach in depth yet, but we find it promising for enhancing model efficiency. We plan to investigate this strategy in the revised version. Your insight is greatly appreciated, and we look forward to exploring its potential benefits.\n\n> Cost comparison\n\nThank you for highlighting the significance of time efficiency in creating high-quality few-shot examples and comparing them with LoRA modules. \n\nIn our experiments, authoring five high-quality few-shot examples requires about 10 seconds for the entire LoraHub learning process on A100 cards. We plan to include these timing details in our analysis to provide a thorough comparison of how LoRA modules may enhance process efficiency.\n\nAdditionally, regarding memory usage, LoraHub shows considerable efficiency, using only about **5GB** of memory with the FLAN-T5-Large model, compared to **34GB** required for LoRA fine-tuning. This data underscores the benefits of integrating LoraHub, notably in reducing memory overhead and improving processing times.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is very well written with sufficient technical details and the authors have done a great job of anticipating most questions from reviewers in Section 5\n- The proposed method is a novel use of LORA modules and demonstrates competitive performance with Few-Shot ICL setups with significantly lower inference times, however coming at an overhead of tuning the modules \n- The circumventing of gradient updates through only tuning the coefficients also makes this a lightweight and scalable approach\n- This is the first approach to the best of my knowledge to leverage lora for cross-task generalization", "reasons_to_reject": "- The LORA Hub setup is not a clear win over few-shot ICL, while it improves the latency of inference this comes at the overhead of adjusting the lora module co-efficients, which otherwise would have required hand authoring a couple of samples", "questions_to_authors": "- Have you explored how we could mitigate the number of adjust iterations by adding some few-shot examples to compensate like a hybrid approach that requires fewer hand-authored samples but can also leverage the LORA modules\n- Have you quantified the amount of time it takes to author high quality few-shot samples and how LORA modules compare to that? It might make more a more convincing case", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a simple and intuitive framework for combining LoRA models by applying zero-order optimization on weights of these LORA additions to minimize loss.
The problem is important, but the solution is not particularly novel or surprising, yet the reviewers liked the paper. I therefore recommend acceptance pending space.
Decision: Accept