HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
Authors: Chunlin Tian
Zhan Shi
Zhijiang Guo
Li Li
Cheng-zhong Xu
Venue: NeurIPS 2024 oral (2024)
Keywords: Large Language Models
Efficient Fine-Tuning
Asymmetric Structure
Primary area: generative_models

Abstract: Adapting Large Language Models (LLMs) to new tasks through fine-tuning has been made more efficient by the introduction of Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA. However, these methods often underperform compared to full fine-tuning, particularly in scenarios involving complex datasets. This issue becomes even more pronounced in complex domains, highlighting the need for improved PEFT approaches that can achieve better performance. Through a series of experiments, we have uncovered two critical insights that shed light on the training and parameter inefficiency of LoRA. Building on these insights, we have developed HydraLoRA, a LoRA framework with an asymmetric structure that eliminates the need for domain expertise. Our experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even those that rely on domain knowledge during the training and inference phases. Our anonymous codes are submitted with the paper and will be publicly available. Code is available: https://github.com/Clin0212/HydraLoRA.

HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
Chunlin Tian' University of Macau yc274020um.ed edu.mo
Zhan Shi University of Texas at Austin 2shi178cs utexas.
Zhijiang Guo University of Cambridge zg2830cam ac uk
Li University of Macau 11ili@um. edu.mo
Chengzhong Xu University of Macau czxučum. edu, mo
Abstract
Adapting Large Language Models (LLMs) new tasks through fine-tuning has been made more efficient by the introduction of Parameter- Efficient Fine- Tunn-ngnig (PEFT) techniques, such LoRA. However, these methods often underperform compared to full fine-tuning, particularly in scenarios involving complex datasets. This issue becomes even more pronounced complex domains, highlighting the need for improved PEFT approaches that can achieve better performance. Through series of experiments, we have uncovered two critical insights that shed light on the training and parameter inefficiency of LoRA. Building on these insights, we have developed HydraLoRA, LoRA framework with an asymmetric structure that eliminates the need for domain expertise. Our experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even those that rely on domain knowledge during the training and inference phases. Code is available.
Introduction
are notably powerful, yet their training involves substantial expense. Adapting a single LLM multiple downstream applications via fine- tuning has emerged as prevalent method to cater to specific domain needs, balancing performance with practicality. This approach, however, faces a significant challenge due to extensive memory and computational resources required for full finettunigg (FFT), i.c., fine-tuning all billions of parameters. A solution to this has been the development of more selective adaptation techniques, involving modifying only portion of the parameters or integrating external modules designed for new tasks. Key methodologies this sphere include LoRA 18] Adaptors 337 17 31. and many
Large Language Models (LLMs; [0
48 32 33
other variants 25 24
9
14 53) all part of what be generally termed Parameter- Efffccient
Fine-tuning (PEFT). PEFT strategies are characterized by freezing the backbone model parameters while only minimal number of task-specific parameters introduced and fine-tuned. This method substantially boosts efficiency in the phases of fine-tuning and subsequent deployment, marking a significant advancement in the practical of L.LMs.
While fine-tuning a small subset of parameters offers a streamlined approach for domain adaptation, it's well-recognized that model performance is closely tied to the number of parameters involved [22) This intrinsic characteristic of methods like LoRA often results in them falling short the FFT baseline, which updates all parameters, thereby creating trade-off between efficiency and model quality. This issue compromised quality in low parameter setting becomes even more pronounced in target domains characterized by complex sub-domains and diverse tasks. This situation presents compelling research question:
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

B
B
B
B3
B
B2
B3
4
2
A3
A
A 1 (c)
(a)
(b)
Figure Illustration LoRA architecture changes in HydruLoRA. Only the tunable parameters shown in this Figure. (a) LoRA architecture with matrix A to achieve low rank and matrix B to recover. (b) under the same parameter count, a monolithic LoRA is split into multiple smaller A and B matrices avoid training interference. (c) based on (b), HydraLoRA has an asymmetric structure that has a shared A matrix and multiple B matrices.
What is the optimal architecture that can deliver superior model performance while still capitalizing on the efficiency benefits of a reduced parameter footprint?
In our research, we carry out a series of exploratory experiments, applying LoRA to the LLaMA2 model to adapt it to new domain encompassing multiple downstream tasks. As shown in Figure Ta). LoRA adds trainable pairs of rank decomposition matrices and B in addition to existing weight matrices. Our in-depth analysis of LoRA's mechanics yields several insightful observations and leads to the formulation of key hypotheses. First, rather than employing single LoRA for the entire domain, it proves more effective to deploy multiple, smaller LoRA heads, each dedicated to a specific downstream task (see Figure This suggests that domain or task interference might harmfully impact the training process. We further hypothesize that this interference originates from "intrinsic components" " sub-domains or distinct tasks potentially unknown even to domain experts. parameters
Additionally, upon visualizing the parameters of LoRA, discern pattern: predominantly learn the commonalities across all data, while others focus on the unique aspects of each intrinsic component. From these observations, posit that optimal LoRA architecture should embody an explicit, asymmetric structure.
Building upon the observations, we propose an improved end-to-end LoRA framework, which we refer to as HydraLoRA. From the architecture perspective, unlike LoRA's symmetric structure, HydraLoRA has an asymmetric structure that has shared A matrix and multiple B matrices (see Figure [().. The shared matrix used by all samples for parameter efficiency. During the fine-tuning phase, HydraLoRA is designed to auto-identify "intrinsic components" and segregate training samples into distinct B matrices. During the inference phase, HydraLoRA leverages multiple B matrices using Mixture-of-Exper completely eliminates the need for human expertise and assumptions, showing better performance than using domain knowledge to guide the fine-tuning process.
(MoE; 200 40)) manner. Unlike prior work, HydraLoRA
2
Background and Motivation
2.1 LoRA Basics
LoRA 18 achieves comparable performances fine-tuning on many benchmarks by freezing the pre-trained model weights Wo and inserting trainable rank decomposition matrices into each layer of the pre-trained model. particular, for each layer, LoRA uses two sequential low-rank matrices A and B to fit the residual weights for adaptation. The forward computation is written as follows:
y/ y A Wox + BAr
where E R is the output and the E R k denotes the input. B E R dxr A E R rsk with r & min(d,k). Normally matrix B initialized with zeroes and matrix is initialized with Kaiming Uniform [15] to force A y 0 at the beginning.

---

2.2 LoRA's Practical Dilemma
Parameter count has a clear impact on the performance of neural models [22 331. Yet, Parameter-
Efficient Fine-tuning (PEFT) methods, such Adapter [17] and prefix-tuning [25]. focus fine- tuning a limited set of parameters. These approaches present practical dilemma: while restricting the number of tuned parameters is essential for training efficiency, it hinders the model's 's ability to learn from diverse datasets, This trade-off becomes particularly evident when considering corpus heterogeneity (2]. Figure 2 reveals a notable performance disparity between PEFT techniques and full fine-tuning (FFT), with the gap widening in scenarios involving a more diverse or heterogeneous training corpus.
Table 1: Performance instruction tuning with Dolly- 15KK (8) evaluated with MMLU [16] with different ranks. For LoRA (Split) decomposes high- rank LoRA modules into smaller, equivalent low- rank components (r x n). n is the number of LoRAs, denotes the rank of each LoRA.
Performance
Increased Gap
Full Porameter Fine-Tunie TreeiTunin Parameter-Efficn Effficient Fine- Tuning
Corpus Heterogeneity
Schemes
MMLU
Parameter
Figure 2: Performance impact of corpus heterogeneity on full fine-tuning vs. parameter-effiieen fine-tuning. Heterogeneity signifies the diversity within the dataset, often leading interference due its varied content and style (2). crrameter-efficieee approaches are particularly sensitive, suffering greater performance losses in hetero- gencous cases.
LoRA
8
43.22
0.062
LoRA
16 1
45.45
0.124
LoRA
32 x 1
46.59
0.248
LoRA (Split)
2
46.82
0.248
LoRA (Split)
8xx
46.94
0.248
LoRA (Split)
46.83
0.248
2.3 Observations
In this work, we aim for a PEFT approach that strikes better balance between maximizing the learning capability for heterogeneous data and minimizing the number of parameters involved. A key goal is ensure that our enhanced technique exhibits robust generalization across unseen tasks, independent of any prior task-specific knowledge. To achieve our objectives, we focus on LoRA and conduct series of experiments as Table to gain a deeper understanding of its mechanisms. Our methodology involves leveraging data from diverse tasks within a domain, and training distinct LoRA heads for each domain, leading to our first observation:
Observation With the same parameter count, rather than employing a single LoRA for the entire domain dataset, proves more effective to deploy multiple, smaller LoRA heads, each dedicated specific downstream task.
This suggests that interference among tasks might harmfully impact the training process. Furthermore, we posit that this interference is NOT exclusive to this explicit multi-task training. This interference could happen in any training setting since all datasets inherently consist of multiple implicit intrinsic components, such as sub-domains or tasks within a domain that is even unknown to domain experts. To better understand how multiple LoRA heads mitigate the interference among intrinsic components, in Figure 3 we employ the t-SNE technique (49 to visualize the parameters of matrix and B across all heads. This analysis yields another critical observation:
Observation II: When multiple LoRA heads are trained individually on different data, the parameters of matrix A from different heads tend to converge, while those of matrix B are distinguishable.
In detail, the parameters of matrix A across all heads exhibit high degree similarity, leading to their overlaps in the figure. Conversely, the parameters of matrix B from different heads are distinct and easily distinguishable. We posit that this divergence an artifact of the initialization schemes, with matrix A inclined toward capturing commonalities across domains, while matrix B adapts to Nomain-specitfe diversities. The distinction between matrix A and B offers valuable insights for enhancing both parameter efficiency and effectiveness. From an efficiency standpoint, our hypothesis suggests that the parameters of matrix could potentially be shared across multiple heads, thereby reducing redundancy. Regarding effectiveness, since the parameters of matrix B of different heads
3

---

SNE Visualication of BA Matrix Diff. LoRAs
SNE Virnsizaction of A Mascix DIE. LollAs
LSNE Visualiaatin B Matrix Dutt LottAs
SUM
SUM
SUM
QA
50
IR
ALL
00
30
gille
1.00
1000
100
-50
50
100
100 -50
100
-50
50
Figure 3: Breakdown analysis of LoRA modules. Compare line-tuned LoRA modules of D5KKy-15K 8 with thrce subtasks of D5K including "summarization (SumJ", "closed QA (QAJ" and "information extraction (IE)" using t-SNE. Consider LLaMA2-7B (random seed=42), which contains 32 decoder layers, corresponding to 32 adaptive modules. Each module consists of (0: q_proj of A, 1: q_proj of B. 2: v_proj of A, 3: v_proj of B) submodules. This makes total of 32 x 4 submodules. Left displays all submodules. Center shows all even submodules, i.c. the A matrix. Right represents all odd submodules, i.e. the matrix. It can seen that the differences in the fine-tuned LoRA modules for different tasks arise mainly from the B matrix.
are dispersed, suggesting that using single head adapt multiple domains might be less effective than using individual heads for each domain, which minimizes the interference between domains.
Building upon observations, we propose optimized LoRA architecture designed to enhance costeefffctiveeeess. In this architecture, we share the parameters of A matrix across various sub- domains tasks to improve parameter efficiency, while deploying multiple B matrices, each tailored to handle different intrinsic components. This design allows for a more effective adaptation to the specific characteristics of each component. While these intrinsic components can be manually identified using prior knowledge of the training data, we also introduce end-to-end methods using Mixture-of--Epeete (MoEs) (21). which will be detailed in the methodology section. This automatic approach facilitates flexibility and applicability, particularly in scenarios where prior knowledge is limited unavailable.
HydraLoRA
In this section, we introduce the proposed Hydral an asymmetric LoRA architecture for efficient fine-tuning, illustrated in Figure After that, we show the workflow of HydraL as Figure
3.1 Asymmetric LoRA architecture
The LoRA method updates two low-rank matrices A and B, and uses AB as the change of a pretrained and frozen weight Wi of linear layer as shown in Eq. [1] The integral parameters fine-tuned for the whole corpus in the original LoRA, which causes difficulty in learning the various knowledge aspects. Drawing from a detailed breakdown analysis of LoRA, potential solution is segment the entire LoRA into "Hydra" structured LoRA variants, that is, characterized by central shared matrix A and several distinct matrices B, fostering blend of shared knowledge and specialized functionalities. As Figure I HydraLoRA is to fine-tune LoRAs to achieve robust performance without redundancy, thereby benefiting the entire heterogeneous corpus. The asymmetric LoRA architecture can be formulated as:
W Wo A W
N
W
W. B,A
The matries B, Raxxr and shared A E R The hyper-parameter N denotes the number of B matrices. The term ui modulates these contribution weights for head B.
3.2 Workflow of HydraLoRA
Figure 4 illustrates the workflow of HydraL.RR. Initially, Hydral delves into the adaptive identification and initialization of LoRA modules within heterogeneous corpus, aligning them with

---

A. Fine- Tuning
Inference
Mixed-task Input
Initializattin
B1
B2
B
Attention
Identify intrinsic
Heterogeneous Corpus
components
LN
B
8
B A8, y
Initialized HydroLoRA
A0
A8g
Router
FNN
LN
2. Tuning
B
B
B
Inference
Pretrained Weights
Routing LoRA Marge
Segregate somples intrinsic
Router
components by
Trained HydraLoRA
Output
Figure 4: Architecture and workflow of HydraLoRA. During the fine-tuning stage, HydraLoRA first adaptively intrinsic components without specific domain knowledge. then employs a
identifies and initializes trainable MoE router that treats each intrinsic component expert automatically segregate training samples into intrinsic components for fine-tuning. During the inference stage, HydraLoRA merges multiple B matrices flexibly and dynamically through trained router.
task relevance through the application of k-means or developer-specified size. Subsequently, we propose a Mixture-of- (MoE) framework that handles B matrices as expert adapters to ensure computational efficiency throughout the fine-tuning (Section [32.1] and inference (Section (3.2.2) stages by freezing the rest of the LLM parameters. During inference, it flexibly and dynamically merges multiple matrices through the MoE router.
3.2.1 Fine-tuning
Motivated by Mixture-of Efpextsss (MoEs; 20 40)), where experts are selectively activated by a gating mechanism (Router) in response to different imputs. In HydraLoRA, substitute each expert with lightweight LoRA adapter. During fine-tuning, while weights of LLMs remain frozen, the experts and router layers trained from scratch. In order achieve unified approach to the distinct forward processes of multiple matrices, we define a set of experts, denoted as (Ei...., EN). ), to learn the updated matrix AW. As HydraLoRA fine-tunes the experts using the heterogeneous corpus, the shared matrix A inherently captures collaborative knowledge to augment satra-gains, and different matrices B foster knowledge modularity to mitigate fine-tuning inter-offsets. Based on this structure, the forward process of HydraLoRA expressed as:
= Wor E,A (Mob)
where N denotes the number of experts, i.e., B matrices. To regulate these contributions, we introduce gate function (router network) commonly consisting of a dense layer with trainable weights (transformation matrix) W, E R followed by a softmax function which takes an intermediate token representation æ as input and combines the output of each expert based on the gating scores (wi,...,wy) W:)):
Wi softmax W x) (Router)
3.2.2 Inference
During inference, HydraLoRA merges adapters by enabling routing computation based on the input. Specifically, since matrices B operate as linear functions, we initially compute a weighted average of the experts. Following this, we apply PEFT transformation using the combined expertise. The HydraL.oRA significantly enhances training efficiency through extremely parameter-efficient MoE formulation. Additionally, the intrinsic structural modularity of Hydral facilitates rapid recovery and merging of the trained parameters during inference, leading substantial memory savings.
5

---

Table Comparative performance different tuning schemes across multiple benchmarks on a single domain. 8-shot for GSM8K, zero-shot for others. #B refers to the average B matrix number.
Human Eval
Schemes
MMLU
Medical
Law
GSM8K
%Param
#A
#B
P@ 10
LLMM2-7B
48
38.88
35.98
33.51
13.10
20.34
10.38
Full Fine-Turee Turning
49.91
46.78
46.08
20.24
32.93
25.70
100
Prompt Tuning
39.91
37.59
35.02
13.66
21.55
13.18
0.001
P-Tuning(256)
41.11
39.81
36.72
13.60
0.193
Prefix Tuming
41.78
40.28
36.54
13.23
22.56
16.89
0.077
(IA)* 27
40.45
37.12
35.25
13.54
23.17
13,98
0.009
AdaLoRA 651
44.32
42.83
39.36
14.81
23.78
19.51
0.093
(oRAirm8)
43.22
41.59
37.85
15.67
22.95
18.24
0.062
LoRA(r=16)
45.45
43.10
39,
16.71
25.60
20.32
0.124
46.59
44.32
40.81
17.12
25.89
20.67
0.248
1
LoRA-Split Sppiitppittssss
46.94
45.28
41.35
18.20
26.85
21.92
0.248
4
4
HydraLoRA
47.22
45.71
42.18
18.31
27.43
22.27
0.124
4
Experiments
In this section, we detail the principal experiments. We begin with an overview of the experimental setup and implementation intricacies. Following this, we share our findings and offer a succinet interpretation.
4.1 Experiment Setting
Dataset and Benchmarks To explore the properties and commonalities of the LoRA asymmetric structure, we conduct experiments on both single and multiple domains to evaluate the effectiveness of HydraLoRA for profiling intrinsic components. Single domain. 1) General:
fine-tune with
the general instruction tuning coabbicks--ooly--e [8] for generic language capability and evaluate with MMLU [16]. 2) Medical: fine-tune with GenMedGPT and clinic-10k from ChatDoctor [26 for medicine applications and evaluate medical tasks in MMLU. 3) Law: we fine-tune with two legal instruction tuning datasets lawyer-Instruch' [1] and US-Terms 4 then evaluate with law tasks in for mathematical reasoning and evaluate with test set of GSM8K. 5) Code: fine-tune with CodeAlpaca [5] for code generation and evaluate with HumanEval 66. Multi-task domain. We select portion of the Flanv2 [51] datasets covering Natural Language Understanding (NLU) and Natural Language Generation (NLG), which can be grouped into 10 distinct task clusters. Then we evaluate it with the Big-Bench Hard (BBH) benchmark. A detailed description of the benchmarks can be found in Appendix A.
MMLU. 4) Math: we fine-tune with the training split of GSM8K
45
Baselines First, we compare HydraLoRA with different PEFT methods on single datasets: 1) Full fune-tuning; 2) Prompt Tuming (24) 3) P-Tuning (2); Prefix Tuning (25): IA (27]: 6)
AdaLoRA [55]. Second, we extend the experiments exploring HydraLORA on multiple datasets compared with more weighted average methods: 1) Lorahub [19] employs black-box optimization to learn weights of 20 randomly selected LoRAs for new tasks, using weighted averaging without needing gradient calculations. 2) LoRA MoE 52) combines lightweight experts (LoRA) with MoE architecture for high efficiency, generalizing to new tasks without prior knowledge. A detailed description of the baseline models can be found in Appendix
4.2 Overall Performance
The experimental results of HydraLoRA and the competing baselines are presented in Table with single domain and Table with the mixed task domain. The evaluation of diverse tasks demonstrates that HydraLoRA consistently outperforms all other schemes. The performances rooted in LoRA outperform those of conventional PEFT methodologies. Compared to the default single LoRA configuration (rank=8), the Hydra architecture, enriched by the integration of several B matrices, effectively addresses the inherent conflicts among intrinsic components of the corpus. Furthermore. with equivalent parameters (rank=16), the model shows superior performance, confirming the ef-

---

Table 3: Comparative performance of different tuning schemes, including base model (Base), LoRA tuning L), LoralHub learning, multi-LoRA tuning with MoE inference (LoRA MoE) and our proposed HydraLoRA learning across task domain on the BBH benchmark with LLaMA2-7B, LLaMA2- 13B as the base LLM (3-shot).
Metries
Base [48
LoRA [18
Lorahub [19
LoRA MoE [52]
HydraLoRA
7B
31.6
36.8
39.7
40.3
41.5
Performance
13B
38.4
40.1
41.9
43.7
44.1
of A/B for training of A/B inference
1/1
48/48
48/48
1/10
1/1
20/20
48/48
1/10
Params
0.062
1.240
2.976
0.341
fectiveness of the adopted parameters. Based on Table 2 and Table questions that confirm the aforementioned observations.
3 we propose three research
RQ1: Is it more effective to use multiple smaller LoRA heads for specific tasks rather than one single LoRA for the entire domain dataset, given the same parameter count? Comparing the high-dimensional LoRA configuration with = 32 against a segmented version using LoRA-Split, variant introduced by HydraLoRA, which divides the model into four distinct components each with = 8. That is, multiple vanilla LoRAs directly utilized to capture the differences between data. We observe a noteworthy trend in the performance across a variety of tasks as detailed in Table It illustrates the superior performance of I LoRA-Split in comparison the traditional LoRA approach, across all the evaluated scenarios. This enhancement in performance is a strong indication of the detrimental impact that task interference can have on the training process. By segregating the tasks into discrete components, LoRA-Splic effectively minimizes the conflict and interference between tasks, thereby promoting a more efficient and focused environment.
The concept of LoRA-Splic hinges on the construction of different intrinsic component compositions, employing LoRA as foundational technique to strategically mitigate the interference conflict. This architectural innovation has proven to be a pivotal factor in enhancing model performance. However, it's important note that while LoR L-RA-Split marks a significant advancement in model efficiency and task handling, it also introduces a certain level of parameter redundancy. The segmented approach of LoRA-Split inevitably leads to increase in the overall model parameters, which can be manifold in comparison to the traditional, singular LoRA model. This increase in parameters, while contributing the model's robustness and capability to handle multiple tasks simultancously, also poses new challenges in terms of computational resources and model optimization.
RQ2: Will multiple LoRA heads, individually trained on different data, improve efficiency by distinguishing matrix B parameters? We evaluated the Hydra structure LoRA HydraLoRA that is characterized by a shared LoRA A matrix, while maintaining distinct B matrices that are trained separately. This configuration was meticulously compared with both the standard LoRA and the LoRA-Split approaches, emphasizing efficiency parameters.
According to the results presented in Table 2, unlike split which straightforwardly adopts multiple vanilla LoRAs, Hydral-oRA adopts an asymmetric LoRA structure that not only improves parameter efficiency by separating the uses of A matrix for commonalities and B matrices for diversities with notably smaller adapter parameter set, but also employs a trainable router to improve the composition of multiple B matrices that outperforms the LoRA-Split approach. This finding significant suggests that HydraLoRA not only enhances performance efficiency but also boosts overall system effectiveness. This may be driven by 1) different B matrices capturing different features of the intrinnntiisic knowledge, mitigating mutual interferences, and avoiding performance offsets. 2) Module A maintains the collaborative knowledge by taking the strengths of each and integrating them to improve the model performance.
RQ3: How does HydraLoRA fare against other merge methods in complex, multi-task domains, considering scalability and robustness? While we hypothesize that the asymmetry is mainly rooted in the different initialization methods of A and B matrices, it possible that this behavior varies on different model architectures and datasets. Recent work confirms similar empirical observations [54 [3]
To the best of our ability, extended the experiments exploring HydraLoRA on multiple
7

---

Energy (KWW)
LoRA(--8
2.5
LoRAir-10)
Latency (b)
2.0
LoMA(r-32)
1.5
LollA-8pli4keh)
1.0
HydnL.RA
0.5
0.0
CPU
GPU
RAM
R-5
R-16
R-32
HydraLoRA
Figure Energy consumption and latency during fine-tuning with different LoRA approaches (fine-tuning LLaMA2-7B with GSM-8K).
30.0
Performance (%)
47.5
36,9447222
wif
45.28 45.2846.71
45.71
45.0
Hydra
12.81
HydraLLoRA
42.5
1:50
37.85
37.5
17.12
35.0
Mmlu
Medical
Law
Figure 6: Comparative performance of ablation study for Hydral across multiple benchmarks.
datasets. LoRA MoE and their variants typically aim tackling multi-tasks by employing multiple independent LoRAs. This makes them suitable for handling various domains. However, for a single dataset like ours, "default" MoE method might not be optimal. HydraLoRA addresses this by constructing asymmetric structures and utilizing multiple B matrices to capture the specific nuances within the single dataset. The effectiveness of this approach demonstrated by the experimental results in Table
4.3 Energy and Throughput Analysis
RQ4: How does the "Hydra" structure in Hydral.RRA enhance system efficiency, particularly in reducing training energy consumption and latency? We evaluate the system efficiency of Hydral from two perspectives: training energy consumption and latency. The following experi- ments were executed on a GPU infrastructure consisting of 4 NVIDIA A40 GPUs and CPU powered by an Intel(R) Xeon(R) Gold 6330 CPU clocked at 2. 2.00GHz. Power consumption measurements were recorded using CodeCarbon (34]. Figure 5 shows the results of various fine-tuning approaches for GSM-SK using the LLaMA2-7B 7B model. we can see that HydraLoRA effectively speeds up the training process 1.96x and reduces 49.6% energy cost compared to LoRA (rank=32). While the energy consumption and latency of LoRA-Split exceeds the LoRA (rank=32). This is for the reason that HydraLoRA jointly considers inherent knowledge modularity and collaboration, which utilizes the "Hydra" structure with a shared A matrix and different matrix. In this way, it only employs r training overhead but expands performance enhancement of more than rank=32. Overall, this experiment demonstrates the parameter effectiveness of HydraloRA.
4.4 Ablation Study
What impact do the MoE architecture and the gate function have on the fine-tuning process? To delve deeper into understanding the contributions of each component in HydraLoRA. we present the results of our ablation study in Figure|6) The variant w/o MoE (essentially reverts to LoRA) excludes the MoE architecture. Similarly, the w/o gate variant employs uniform expert weights bypassing the gate function. The w/o hydra adopts multiple vanilla LoRAs in a straightforward way. Figure indicates indiiatess that the full HydruL model outperforms its variants, showing that both the MoE architecture and gate function significantly contribute to its effectiveness across various language understanding domains.
RQ6:

---

Number of Clusters
47.4
47.2
Performancee
46.
46
46
DBSCAN
46.
Figure 8: The results of experiments for hyper- parameters number of clusters.
Figure 7: Number of clusters generated different approaches including developer-specifie (static), k-means, and DBSCAN.
4.5
Hyper-parameter Analysis
RQ7: How do the number of intrinsic component of HydraLoRA influence performance out- comes? As Figure Saaure@Sshwwn we conduct comprehensive and meticulous analysis by fine-tuning the Dolly-15K model on the LLaMA2-7B dataset and subsequently evaluating its performance on the MMLU benchmark to rigorously examine the impact of variations in the intrinsic component, symbolized by the variable N, on the model's overall performance. Empirically we find that the number of clusters not a sensitive parameter for HydraLoRA, with a wide range of reasonable number N of clusters (e.g. 2 to 4) performing decently well in all settings in our experiments. Specifi- cally, the performance loss of
vs. the optimal N = is only 0.42%. Meanwhile, as illustrated in Figure 7 we employ three distinct methods to generate the number of corpus clusters 15-fold, and the results demonstrate that the k-means 30 yields comparable outcomes with DBSCAN [39]. Therefore, based on this observation, we choose k-means because it is simple but effective, more sophisticated hyperparameter search approaches (e.g. DBSCAN, parameter sweep and Bayesian optimization) will be unnecessarily costly. It's noteworthy that HydraLoRA is adeptly designed to orchestrate its components in a way that it can automatically calibrate and navigate toward the optimal performance configuration across various parameters. This intelligent auto-tuning is achieved through the application of the k-means clustering algorithm. This strategic component orchestration not only enhances performance but also ensures more efficient and effective utilization of resources. underpinning the model's capability to adapt and perform efficiently in a dynamic computational environment.
Related work
Parameter-Fffiiiiet Fine-tuning LLMs are becoming increasingly powerful, but fine-tuning them often requires significant computational resources. This has spurred research parameter- efficient fine-tuning (PEFT) techniques that reduce memory and storage costs during adaptation. One prominent PEFT approach is adapters [17 37]. It introduces new, trainable dense layers within
the existing model, keeping the original parameters frozen. This concept has proven successful across various domains [35
adapter compactness involve constructing parameter matrices using Kronecker products of low-rank matrices [31] Another PEFT strategy directly manipulates activations with learned vectors. This can be achieved through concatenation (29 25 [4]], multiplication (IA3; (27)). or addition (BitFit; (53]). Prefix-tunpi -tuiingnngg (25) and prompt-tuning 24 are noteworthy examples that fine-tune continuous prompts instead of designing discrete ones 19. Interestingly, a study suggests that many PEFT methods can be viewed as form of adapter, providing unified perspective [14) Beyond adding new parameters or altering the computational graph, researchers also explore sparse 12 46 or low-rank updates (LoRA; [18).
42 43 (56). Further improvements
Multi-LoRA Architecture LoRA has notably garnered increasing interest recently, becoming standard approach for adapting LLMs such as LLaMA 47] 48] under limited computational resources. Recognizing its potential, researchers have delved deeper, exploring the benefits of employing multiple LoRAs. LoraHub [19] takes this multi-LoRA approach by training several adapters and strategically picking combinations based on the domain during inference. Meanwhile, MultiLoRA (50) focuses on horizontal scaling, aiming to reduce LoRA's parameter dependence. This involves splitting LoRA modules along the rank dimension and introducing learnable scaling factors for enhanced expressiveness. Addressing scaling challenges from a different angle, the
mixture of LoRA concept is further proposed [52] This mitigates resource consumption when

---

scaling instruction-tuned LLMs. Recognizing the potential for conflict during instruction tuning, LoRAMoE [T] leverages the Mixture-of-ENee -xperts (MoEs; (20] structure to safeguard the pre- trained LLM's knowledge from excessive corruption by instruction data. Similarly, MOELORA [28) incorporates a MoE. framework into I.MMs, thereby improving their multitasking capabilities in the medical domain. Shifting the focus to the system perspective, S-LoRA [1]] provides framework for efficiently serving multiple LoRA adapters. Unlike previous methods that relied on choosing LoRA combinations based on their training domains, HydraLoRA breaks free from the dependence on domain knowledge during inference. Additionally, HHydraLoRA' 's asymmetric structure further enhances parameter efficiency compared to existing symmetric approaches.
6 Conclusion
In this work, we start by conducting exploratory experiments applying the LoRA technique to LLaMA2, aiming to adapt
new domain across various tasks. This study unveils the limitations of employing single LoRA for the entire domain, highlighting the detrimental effects of domain interference, In response, we introduce a novel architecture HydraLoRA that features an asymmetric structure with a shared matrix for all samples and distinct matrices for each intrinsic component. This design improves domain adaptation by selectively focusing on distinct components, enhancing both fine-tuning and inference efficiency. Our research highlights the importance of balancing learning capabilities for diverse datasets against the need for a lean model, offering a viable pathway for improving LLMs with minimal parameter growth. More discussion about limitation and broader impacts are available Appendix D and
Acknowledgments
This research received support from MYRG-GGRG2033-0211--TSS---- Research Grant of the University of Macau (SRG2022-00100-100SS Chunlin Tian and Zhan Shi contributed equally this work. For correspondence, please contact Dr. Li Li (lili@um.edu or Dr. ChengZhong Xu (czxu@ ud.edu.mo).
and the Start-up
References
[1] Alignment-Laab-A. Lawyer-instruct, 2024.
[2] Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H. Ezzeldin, Qingfeng Liu, Kee-Bong Song. Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. CoRR, 0bs/2300.06522 2023.
[3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[4] Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Katz, and Anders Søgaard. LeXFiles and LegaILAMA: Facilitating English multinational legal language model development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15513-15535, Toronto, Canada, July 2023. Association for Computational Linguistics.
[5] Sahil Chaudhary, Code alpaca: An instruction-followingg llama model for code generation. tttps:///gtthz
sahi 1aaaaa110001111/eeaapppaa 2023.
[6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul
10

---

Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hehgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107. abs/2107.03374, 2021.
[7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 14168.. 2021.
[8] Mike Conover, Matt Hayes, Ankit Mathur, Xiangrui Meng. Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, et al. Free dolly: Introducing the world's first truly open instruction-tuned 1lm, 2023.
[9] Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforce- ment learning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3369-3391, Association for Computational Linguistics, 2022.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics, 2019.
[11] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. CoRR, abs/2312.09977 2023.
[12] Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing, ACLJJCCLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages Linguistics, 2021.
Association for Computational
[13] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lorat: Efficient low rank adaptation of large models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[14] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Kirkpiiikp,,,ick, and Graham Neubig. To- wards a unified view of parameter-efficient transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net, 2022.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 1026-1034. IEEE Computer Society, 2015.
[16] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. CoRR, wtwp/009.00300 2020.
11

---

[17] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficien transfer learning for NL.. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, Califormia, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR, 2019.
[18] Edward J. Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen- -Zu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April
25-29, 2022. OpenReview.net, 2022.
[19] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Elficient cross-task generalization via dynamic lora composition. CoRR, abs/2307.13269, 2023.
[20] Robert A. Jacobs, Michael I.
Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Comput., 3(1):79-87, 1991.
[21] Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Comput., 6(2):181-214 1994.
[22] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08366 2020.
[23] David J Ketchen and Christopher I. Shook. The application of cluster analysis in strategic management research: an analysis and critique. Strategic management journal, (16):441-5588 1996.
[24] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-effiiiin prompt tuning. In Faaic-Franc Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 1044-3059... Association for Computational Linguistics, 2021
[25] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume Long Papers), Virtual Event, August 1-6, 2021, pages Linguistics, 2021.
Association for Computational
[26] Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (lama) using medical domain knowledge. Cureus, 15(6), 2023.
[27] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[28) Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications. CoRR, 1is/2310.18339) 2023.
[29] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. CoRR, 1is/2103.11388 2021.
[30] Stuart P. Lloyd, Least squares quantization in PCM. IEFE Trans. Inf. Theory, 28(2): 1982.
12

---

[31] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021,
NeurlPS 2021, December 6-14 14, 2021, virtual, pages 1022-1033 2021.
[32] OpenAl. ChatGPT, 2022.
[33] OpenAl. GPT-4 Technical Report. CoRR, abs/2303.08774, 2023.
[34] David A. Patterson, Joseph Gonzalez, Quoe V. Le, Chen Liang, Lluis-Mique' Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. CoRR, 0os/2104.10350, 2021.
[35] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pages 487-503 Association for Computational Linguistics, 2021.
[36] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine I Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67 2020.
[37] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Leaninn multiple visual domains with residual adapters. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 506-516, 2017.
[38] Gerard Salton and Chris Buckley. Terg-teighting approaches in automatic text retrieval. Inf. Process. Manag., 24(5):513-52) 1988.
[39] Erich Schubert, Jörg Sander, Martin Ester, Hans-Peter Kriegel, and Xiaowei Xu. DBSCAN revisited, revisited: Why and how you should (still) use DBSCAN, ACM Trans. Database Syst., 42(3):19:1-19:21, 2017.
[40] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[41] Ying Sheng. Shiyi Cao. Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang. Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and lon Stoica. S-lora: Serving thousands of concurrent lora adapters. CoRR, abs/231 1. .03285, 2023.
[42] Asa Cooper Stickland and lain Murray. BERT and pals: Projected attention layers for efficient adaptation in multi-task learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5986-59869 5995. PMLR, 2019.
[43] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VL-ADAPTER: parameter-efffcient transfer learning for -idion-and-language tasks. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleuns, LA, USA, June 18-24, 2022, pages 5217-5227. IEEE, 2022.
[44] Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wort- man Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Confer- ence on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14 14, 2021, virtual, pages 24193-24205 2021.
13

---

[45] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challeng- ing big-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan I.. Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACI. 2023, Toronto, Canada, July 9-14, 2023, pages 13003-13051. Association for Computational Linguistics, 2023.
[46] Chunlin Tian, Xinpeng Qin, and Li. Greenllm: Towards efficient large language model via energy-aware pruning. In 32nd IEEE/ACM International Symposium Quality of Service, IWQoS 2024, Guangzhou, China, June 19-21, 2024, pages 1-2. IEEE, 2024.
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, 0etp22022.1991.. 2023.
[48] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodrigue7, Robert Stojnie, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307 009888, 2023.
[49] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911). 2008
[50] Yiming Wang. Yu Lin, Xiaodong Zeng, and Guannan Zhang. Multilora: Democratizing lora for better multi-task learning. CoRR, abs/2311. 1.1101,,, 2023.
[51] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[52] Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. CoRR, abs/2309.0544 005444 2023.
[53] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine- tuning for transformer- masked language-modett In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computationall Linguisties (Volume Short Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1-9. Association for Computational Linguistics, 2022.
[54] Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memoryy-fficient low-rank adaptation for large language models fine-tuning. CoRR, abs/2308.0333 2023.
[55] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[56] Han Zhou, Xingchen Wan, Ivan Vulie, and Anna Korhonen. Autopeft: Automatic configuration search for parameter-efficient fine-tuning. CoRR, abs/2301.12132. 2023.
14

---

A Datasets and Baselines
A.1 Datasets
Single Domain
1. General: we fine-tune with the general instruction tuning databricks-dolll--5kk for generic language capability and evaluate with MMLU.
2. Medical: we fine-tune with GenMedGPT and clinic- dinic-1Ok from ChatDoctor for medicine applications and evaluate medical tasks in MMLU including three related tasks: "clinical knowledge". "professional medicine" and "college medicine".
3. Law: fine-tune with two legal instruction tuning datasets Lawyer-Instruct Instruct and US-Terms then evaluate with law tasks in MMLU including two related tasks: "professional law" and "international law".
4. Math: we fine-tune with the training split of GSM8K for mathematical reasoning and evaluate with test set of GSMSK.
5. Code: we fine-tune with CodeAlpaca for code generation and evaluate with HvmanEval.
well for complex mixed nulti-task/domain, we select portion of the Flanv2 datasets covering Natural Language Understanding (NLU) and Natural Language Generation (NLG), which can be grouped into 10 distinct task clusters. Then we evaluate with the Big- Big-Bench Hard (BBH) benchmark.
Multi-task Domain
We summarize the details of the used datasets follows:
1. Struct-to-Text Conversion: This task evaluates the capability to generate natural language descriptions from structured data inputs. We use the following datasets: (1) CommonGen;
(2) DART; (3) E2ENLG; (4) WebNLG;
2. Translation: Translation involves converting text from one language another, maintaining the original meaning and nuances. We use the following datasets: (1) En-Fr from WMT" 14;
EnDe, En-Tr, En-Ru, En-Fi, En-Ro from WMT' 16; (3) En-Es from Paracrawl.
3. Commonsense Reasoning: This involves assessing the ability apply physical or scientific principles alongside common sense in reasoning tasks. We use the following datasets: (1)
COPA, (2) HellaSwag, (3) PiQA, and (4) StoryCloze.
4. Sentiment Analysis: A fundamental task in natural language processing (NLP) that deter- mines the sentiment polarity (positive or negative) of given text. use the following datasets: (1) IMDB, (2) Sentiment 100, (3) SST-2, and (4) Yelp. information sources. We
use the following datasets: (1) ARC, (2) NQ, and (3) TriviaQA.
5. Paraphrase Detection: This task requires models ascertain whether two sentences convey the same meaning, indicating semantic equivalence. We use the following datasets: (1)
MRPC, (2) QQP, and (3) Paws Wiki.
6. Coreference Resolution: Involves identifying instances within text that refer to the same entity, demonstrating an understanding of textual context. We use the following datasets: (1)
DPR and (2) WSC273.
7. Reading comprehension: Assesses the capability to derive answers to questions from a provided text containing relevant information. We use the following datasets: (1) BoolQ, (2)
DROP, (3) MultiRC, (4) OBQA, (5) SQuADv1, (6) SQuADv2.
8. Reading Comprehension with Commonsense:
Merges traditional reading comprehension skills with commonsense reasoning, requiring understanding beyond the explicit text. We use the following datasets: (1) CosmosQA: (2) ReCoRD.
9. Natural Language Inference: Focuses deducing the relationship between sentences, determining if the second sentence logically follows from, contradicts, or is unrelated to the first sentence. We use the following datasets: (1) ANLI, (2) CB; (3) MNLI; (4) QNLI; (5)
SNLI: (6) WNLI; (7) RTE.
Blosed-Book Question Answering: This task challenges models to answer questions about general knowledge without direct access to external
15

---

A.2 Baselines
PEFT methods
1. Full Fine-tuning is the default strategy for adaptation. During fine-tuning, the model is initialized with pretrained weights and biases, and model parameters undergo gradient updates.
2. Prompt Tuning adds task-specific prompts to the input, and these prompt parameters are updated independently the pretrained model parameters which are frozen.
3. P-Tuning adds trainable prompt embeddings to the input that is optimized by prompt encoder to find a better prompt, eliminating the need to manually design prompts. The prompt tokens be added anywhere in the input sequence, and P-Tuning also introduces anchor tokens for improving performance.
4. Prefix Tuning prefixes a series of task-specific vectors to the input sequence that can be learned while keeping the pretrained model frozen. The prefix parameters inserted in all the model layers.
5. IA3 enhances efficiency by infusing learned vectors into transformer architectures, drasti- cally cutting trainable parameters while preserving performance and minimizing inference latency.
6. AdaLoRA a method for optimizing the number of trainable parameters to assign to weight matrices and layers, unlike LoRA, which distributes parameters evenly across all modules. More parameters are budgeted for important weight matrices and layers while less important ones receive fewer parameters.
Multiple LoRA weighted average methods
1. LoRA MoE. A collection of n parameterized experts, denoted as Ej..., Ens is orchestrated by a router network R. This network features dense layer with adjustable weights W from Th xx A softmax function then processes an intermediate token representation æ, yielding gating scores $1,..., .... 8n that determine the weighted contribution of each expert's s output:
R(x); softmax(W""
(Router)
Subsequently, the overall output y is synthesized by aggregating the experts" outputs, each modulated by its respective gating score:
y
E((r) (MoE)
(6)
This results a dynamic allocation of the model's capacity, enabling specialized processing by experts directed by the router's gating mechanism.
2. LoraHub aggregates 20 LoRAs random for new downstream tasks. To master the weight each LoRA, it utilizes a black-box optimization technique, bypassing the need for gradient calculations of the large model. This process involves weighted averaging at the parameter level. Mirroring the MoE training approach, we select 20 random samples for each task, creating cohesive training dataset optimized through this black-box method.
B Initialization via k-means
In the case of considering heterogeneous corpora, it is crucial to select the appropriate number N of matrix B, ensure consistent performance and minimize unnecessary computational overhead. This choice is usually closely related to the training corpus. In this work, we propose initializing OY modules via k-means 300 algorithm for adaptive initialization. Specifically, k-means utilized to process the heterogeneous corpus to identify the best-fit taxonomy of the corpus, i.e., the optimal N. First, we extract key features from corpus by applying the Term Frequency- Inverse Document Frequency (TF-IDF: (38)) algorithm and transform the textual information into numerical
16

---

feature vectors. We integrate the elbow method (23] to determine the optimal value of N. Initially, N cluster centers are randomly selected for preliminary clustering as Eq. followed by updating the cluster centers to accurately reflect the data within each cluster as Eq. 8 where Cj is the cluster center to which data point X, is assigned and .( is the Euclidean distance function. Sj is the set of data points in the j-th cluster.
C2 = argmin d(X,, Cÿ)
(7)
C2
C |S X;ES,
X.
(8)
By analyzing the relationship between the sum of squares of errors (SSE) and different N values, we observe that SSE decreases as increases. Identifying the elbow point on the SSE curve where the rate of decrease in SSE slows down is crucial. The elbow point represents the optimal N value, beyond which increasing the number of clusters does not significantly enhance performance, thereby achieving an ideal balance between model complexity and performance.
C LoRA Breakdown
t SNE Vistalization of DA Motrix Diff. LoRAs
LSNE Vinaliaaie
A-Malii LoRAs
I-SNE B-Milri LoRAA
LoRA
LoKLA LoRA 2
10d4
LoRA
of toge
Taltà
LaRA GSARKK
LoltA
GaMaR
GSMEK
0
(a) Compare tuned LoRA modules of GSMSK 7 with its subsets using T-SNE, We employ the Independent and Identically Distributed (IID) segmentation scheme divide GSM8K into three subsets and fine-tune them using different LoRAs.
SNN Visaaitation of Matrix DEt, LottAs
I-NE Vienalisation of A-Matrie Difl. LoRAA
5-NN Viuniiration of R-Matric Dilt. LoRAs
1061 dulgis
LaRA
LaRA
30
LeRA
20
LARA/
EaltA
LoRA 3
USMES
CSMBB
10
-20
20
-10
10 201
90
-00 20
20
-20 -10
30
200
300
(b) Specific tasks.
Figure 9: Breakdown analysis of LoRA modules. Compare fine-tuned LoRA modules of GSM-
8K (7) with its subsets using T-SNE. We employ the Independent and Identically Distributed (IID) segmentation scheme to divide GSMSK into three subsets and fine-tune them using different LoRAs. Consider LLaMA2-7B (random seed=42), which contains 32 decoder layers, corresponding to 32 adaptive modules. Each module consists of [0: A_ 1: q_proj_BB _,j_ 2: v_proj_A, 3: v_proj_B) submodules. This makes a total of 32 submodules. (a.b) left displays all submodules. (a.b) center shows all even submodules, i.c. the A-matrix. (a.b) right represents all odd submodules, i.c. the B-matrix. It can be seen that the differences in the fine-tuned LoRA modules for different tasks arise mainly from the B matrix.
D Limitation
In this study, we concentrate on well-established PEFT techniques, particularly LoRA. However, we have not yet investigated other configurations, such as prompt-tuning and adapters, which are
17

---

t-SNE Visualization of Matrix Diff. LoRAs
SUM
QA
50
ALI.
40
-50
10
100
1000
-00
50
100
Figure 10: Breakdown analysis of LoRA modules (Dolly-15K) with its subsets using T-SNE on different layer.
reserved for future exploration. Our current evaluation is limited to fine-tuning, while an examination of the efficacy of these techniques during the pre-training phase remains subject for subsequent research. Another potential limitation stems from the training data itself. In multi-task scenarios, extreme conditions-suee as contaminated or adversarial data-can significantly impair performance due to aggregation issues. The heterogeneity across tasks, including variations in language, task type, and domain, may reduce the effectiveness of shared knowledge, thereby diminishing its overall impact. This challenge is not exclusive to HydraLoRA but is common all multi-task frameworks. Addressing these concerns may involve implementing robustness -eahancin-c measures (e.g., data sanitization, robust aggregation, and anomaly detection) and integrating eevacc-preserving technologies, such as homomorphic encryption, differential privacy, and blockchain.
E Broader Impacts
Positive Societal Impacts The proposed HydraLoRA framework, with its asymmetric structure and parameter-efffiieet fine-tuning approach, has the potential to make LLMs more accessible and efficient. This could democratize AI, enabling more researchers, developers, and organizations to leverage the power of LLMs for various applications, ultimately driving innovation and progress. Moreover, by effectively addressing the challenge of domain or task interference, HydraLoRA could significantly enhance the performance of LLMs in complex, multi-task domains. This could lead to more accurate and reliable AT-powered tools and services areas like healthcare, education, and finance, ultimately improving the quality of life for many people. Lastly, the parameter-efficieet approach of Hydral.oRA could help reduce the computational resources required for training and fine-tuning LLMs, thereby lessening the environmental impact of Al.
Negative Societal Impacts As with any Al technology, there are potential negative societal impacts to consider. The risk of misuse is a significant concern, as HydraLoRA could be used for malicious purposes, such as creating more sophisticated and convincing deepfakes or spreading misinformation and propaganda. Additionally, the increased efficiency and accessibility of AI brought about by HydraLoRA could lead to job displacement become capable of performing tasks traditionally done by humans. Lastly, the use of LLMs in various applications could potentially lead to privacy and security issues, especially if these models are used to process generate sensitive information. The proposed HydraLoRA framework, while not directly related to these issues, could inadvertently contribute to them by making easier to deploy LLMs in various applications.
certain sectors, as Al-powered tools and services
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction sections, we articulate the motivation behind HydraLoRA, highlight its differences from previous work, and outline the contributions of this paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations in the Appendix D
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yss]
Justification: We provide detailed explanation of the parameter usage and will release the source code ensure reproducibility.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide a detailed explanation of parameter usage and will release the source code.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The full details are provided in the appendix and supplemental material (code).
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Experimental results are tested multiple times to ensure stability and reliability.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
19

---

Justification: Compute resources are thoroughly described and evaluated in both the experi- mental setup and the experimental results sections.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in this paper fully conforms to the NeurIPS Code of Ethics in every respect.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive societal impacts and negative societal impacts of the work performed in Appendix
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper cites the original paper that produced the code package or dataset.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper relates the details the code part of the submission.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
20


Reviews:
[{"review_id": "iDPzXJxbek", "reviewer": "Reviewer_LBHQ", "summary": "This paper tackles the challenge of efficiently adapting large language models to new tasks. The authors highlight the limitations of current techniques like LoRA, which, while parameter-efficient, struggle with diverse data.\n\nThrough a series of experiments, they discover that using multiple, task-specific LoRA modules improves performance but introduces redundancy. They further observe that within these multiple modules, certain parameters consistently learn common knowledge while others specialize in individual tasks.\n\nBased on these findings, they introduce HydraLoRA which utilizes an asymmetric LoRA structure. A single, shared matrix captures the common knowledge identified in their analysis, while multiple smaller matrices, one per task, handle specialized adaptations. This design maximizes learning from diverse data while minimizing redundancy.\n\nRather than depending on pre-defined task information, HydraLoRA employs a Mixture-of-Experts approach to dynamically route data during training and combine expert outputs during inference.\n\nExperimental results across multiple benchmarks demonstrate HydraLoRA consistently outperforming other efficient fine-tuning methods, including those using MoE. The authors further emphasize HydraLoRA's practical advantages by analyzing its energy consumption and latency.", "questions": "* The shared \"A\" matrix effectively captures common knowledge in your experiments. However, how would HydraLoRA perform on datasets with more disparate domains or tasks where this notion of shared knowledge might be weaker or less well-defined?\n\n* Did you experiment with other routing techniques, such as top-k routing, during your exploration of HydraLoRA's design? If so, could you elaborate on the performance implications of these different routing strategies and what led you to choose your current approach?", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Motivation and Design:\n* The paper excels at connecting its experimental findings to the proposed architecture. Specifically:\n    * The authors use t-SNE visualizations to analyze the parameter distributions of LoRA modules trained on different data subsets. This approach reveals a clear pattern: the \"A\" matrices of these modules tend to converge, indicating common knowledge acquisition, while the \"B\" matrices remain distinct, suggesting they specialize in task-specific features. This key finding highlights the inherent asymmetric nature of knowledge representation within LoRA and provides the foundation for HydraLoRA's design.\n    * Building upon this insight, the authors demonstrate that splitting a single LoRA into multiple, smaller ones, each trained on a different data subset (LoRA-Split), leads to significant performance improvements. This is evident in tasks like MMLU, Medical, and Law, where LoRA-Split consistently outperforms a single, large LoRA with the same parameter budget. These results suggest that intrinsic dataset differences can hinder the performance of a monolithic LoRA, and splitting helps mitigate this by allowing for specialized adaptation to those inherent data variations.\n\nEvaluation:\n  * Comparisons against a wide spectrum of PEFT methods, from traditional techniques like Prompt Tuning and P-tuning to more recent ones like AdaLoRA and (IA)3, provide a comprehensive picture of HydraLoRA's effectiveness.\n  * Significant Improvement over LoRA MoE: The direct comparison with LoRA MoE is a key strength in my opinion. While both methods utilize MoE, HydraLoRA consistently demonstrates superior performance. This highlights the effectiveness of HydraLoRA's shared \"A\" matrix in capturing common knowledge and its advantage over using entirely separate LoRA modules. These gains are evident in both accuracy improvements and reduced parameter count, as shown in the BBH benchmark results.\n  * Thorough Ablations: Authors present extensive ablation studies to capture the impact of various For example, comparing HydraLoRA to a variant with uniform expert weights (\"w/o Gate\") demonstrates the crucial role of the gating mechanism in selectively applying expert knowledge. This level of detail, presented across multiple benchmarks, strengthens the paper's conclusions and provides a deeper understanding of HydraLoRA's inner workings.", "weaknesses": "* While the shared \"A\" matrix in HydraLoRA appears effective for the tested benchmarks, the paper could benefit from exploring potential limitations of this design choice. Investigating performance on datasets with very different domains or tasks, where the notion of shared knowledge might be less applicable, would strengthen the claims about its generalizability.\n\n* The paper would be more convincing with a comparison against a LoRA-Split baseline that uses existing domain knowledge. For example, on a multi-task dataset, directly comparing HydraLoRA against splitting LoRAs by task labels would provide valuable insights into the trade-offs between automatic routing and a more informed, but potentially manual, approach.\n\n* The paper covers a wide variety of necessary aspects, but the presentation could be more streamlined and easy to read. For example, placing the comparison with MoE-based methods and the discussion about the shared \"A\" matrix's advantages earlier in the paper would have made this paper more appealing to readers. This would also emphasize HydraLoRA's unique strengths more effectively.\n\n* A deeper analysis of the MoE router's behavior would have been really interesting. Exploring aspects like its complexity, influence on overall latency, and potential routing biases could provide a more complete picture of its role within HydraLoRA.\n\n*  It's surprising that the authors mention the increased training iterations required by HydraLoRA (1-2 times more than typical PEFT) only within the limitations section. It would have been interesting to explore this nuance further or at least call it out in one of the main sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:\n\n> W1 & Q1: Exploring potential limitations of this design.\n\nThanks for the insightful question. The limitations may primarily stem from the training data. Particularly, in multi-task, extreme conditions such as contaminated or adversarial data can severely impair performance due to aggregation. The heterogeneity between tasks—differences in language, task type, and domain—means that the shared knowledge might be weaker or less noteworthy.  Importantly, this challenge is not unique to HydraLoRA but is common across all multi-task frameworks. Robustness enhancement (such as data sanitization, robust aggregation, and anomaly detection) and privacy-enhancing technologies (like homomorphic encryption, differential privacy, and blockchain) may be potential solutions.\n\n> W2: Automatic routing v.s manual predefined tasks.\n\nIn Table 3, the LoRA MoE baselines [1,2] utilize existing domain knowledge (manual) to train multiple LoRA units, whereas HydraLoRA employs automatic routing. The results indicate that HydraLoRA uses **fewer parameters and performs better in downstream tasks**. This suggests potential coupling relationships between tasks, aligning closely with real-world conditions where we cannot anticipate the domains needing fine-tuning. \nMoreover, Section 4.5 reveals that the **number of clusters K is not a sensitive parameter** for HydraLoRA. It demonstrates the efficiency and robustness of HydraLoRA.\n\n> W3: More streamlined presentation.\n\nThanks for your constructive comment. We will revise the paper based on your suggestions in the updated version.\n\n> W4 & Q2: More MoE discussion.\n\nThanks for your constructive comment.  We add more experiments with the same setting with Table 3, to explore how the number of experts (B matrices) during the HydraLoRA inference pipeline influences performance. As shown in the Table below, we find that an increase in the number of B matrices generally leads to enhanced performance in downstream tasks. \n\nIn practice, **user requests may belong to different tasks, while a single request potentially involves mixed tasks**. This improvement can be attributed to the expanded configuration space afforded by additional LoRA modules, which allows for a more fine-grained and tailored adaptation to the diverse and mixed-task inputs encountered in the benchmark.\n\n| Methods | Base | Top-1  | Top-3 | HydraLoRA |\n|:---:|:---:|:---:|:---:|:---:|\n| **BBH**  | 31.6   | 35.4 | 38.6 | 41.5   |\n\nTable: Sensitivity analysis of the number of B matrices. \"Base\" means vanilla Llama2-7B, \"Top-1\" means selecting the highest-ranked (top-1) B matrix, and \"Top-3\" means selecting three highest-ranked (top-3) B matrices.\n\n> W5：Overhead with other PEFT.\n\nWhile the vanilla LoRA method incurs higher computational overhead compared to other PEFT approaches, it also **delivers significant performance improvements**. HydraLoRA, an adaptation of LoRA, enhances downstream task performance with the same parameter settings (rank=16), as demonstrated in Table 2. \n\nMoreover, as Figure 5 illustrates, HydraLoRA **cuts energy consumption by 50%** compared to Split-LoRA, which uses multiple LoRA modules. This underscores HydraLoRA's efficiency and **eco-friendly**. However, LoRA's carbon footprint is negligible compared to full-parameter tuning, emphasizing its environmental and computational advantages [3]. Meanwhile, **fine-tuning is a one-time event, but inference overhead is crucial**. As fellows, HydraLoRA boosts performance with minimal additional overhead. \n\n|  |Latency(s)|Energy(Wh)|MMLU(%)|\n|---|:---:|:---:|:---:|\n|**LLaMA2-7B**|90.21|72.72|38.88|\n|**+Prompt Tuning**|91.78(+1.57)|73.53(+0.81)|39.91(+1.03)|\n|**+P-Tuning**|91.3(+1.13)|73.87(+1.15)|41.11(+2.23)|\n|**+Prefix Tuning**|92.52(+2.31)|74.21(+1.49)|41.78(+2.90)|\n|**+LoRA (r=8)**|92.28(+2.07)|73.95(+1.23)|43.22(4.34)|\n|**+HydraLoRA (r=8)**|92.86(+2.65)|74.25(+1.53)|47.22(+8.34)|\n\nTable:  latency and energy consumption during inference using Llama2-7B with different PEFT methods, evaluated on the WikiText2 dataset using a single NVIDIA A40 GPU.\n\nReferences:\n\n[1] Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. ICLR 2024.\n\n[2] Lorahub: Efficient cross-task generalization via dynamic lora composition, COLM 2024\n\n[3] [Carbon Footprint of LLM Fine Tuning — A Case Study](https://towardsdatascience.com/carbon-footprint-of-llm-fine-tuning-a-case-study-7703afc716a9).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bxI6XVK4zH", "reviewer": "Reviewer_6P2g", "summary": "The paper introduces HydraLoRA, a PEFT (Parameter-Efficient Fine-Tuning) architecture designed to improve the efficiency and performance of fine-tuning large language models (LLMs). HydraLoRA's main contribution lies in its asymmetric structure, which employs a shared matrix (A) for commonalities across tasks and multiple distinct matrices (B) for task-specific adaptations. The paper claims that this approach mitigates task interference and enhances parameter efficiency without requiring domain expertise.", "questions": "- what are the inference speed gains compared to other PEFT methods?\n- what is the actual training overhead compared to other PEFT methods?", "limitations": "The authors discuss that HydraLoRA is computationally demanding, primarily due to the necessity of fine-tuning large-scale language models. It incurs a higher training expenditure than conventional PEFT methods, attributed to the employment of multiple adapter copies.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The idea of an asymmetric LoRA architecture that splits the parameter matrices into shared and task-specific components is a somewhat novel approach aimed at addressing the inefficiencies in traditional symmetric PEFT methods.\n- The paper includes a variety of experiments across different domains, including general language tasks, medical, legal, mathematical reasoning, and code generation. This wide scope provides a robust evaluation of HydraLoRA's potential benefits.\n- HydraLoRA is compared with several existing PEFT methods such as Prompt Tuning, P-Tuning, Prefix Tuning, and AdaLoRA, providing a comprehensive view of its performance relative to state-of-the-art techniques.", "weaknesses": "- Many sections of the paper are vague and lack sufficient detail. For example, the exact observations of how the shared matrix (A) and distinct matrices (B) interact and are optimized is not clearly explained. This makes it difficult to fully understand the proposed method. For example, lines 97 to 105 explain Figure 3, but it’s confusing to read the center and right subfigures. The center subfigure shows A matrix has fewer clusters and the heads are more distinct, but the text says the opposite (B is more distinct) right subfigure shows B is more clustered and not easily distinguishable. The workflow section 3.2 is scattered and difficult to follow. Key components of HydraLoRA, such as the structure of the matrices and the routing mechanism, are not described cohesively. The figures provided do not effectively clarify these components.\n\n- The idea of using MoE and LoRA adapters to implement multiple B matrices is very similar to Mixture of LoRA Experts (https://openreview.net/forum?id=uWvKBCYh4S, ICLR 2024), but not discussed and compared. The difference is probably the rank size selection.\n\n- The empirical results are incremental, table 2 shows most results compared to LoRA are within 1% improvements, e.g. Compared with LoRA-Split or r=32, HydraLoRA does use half trainable parameters, but unclear how much inference efficiency gains it achieves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below：\n\n> W1：Clarify asymmetric structure and workflow. \n\n- ***Asymmetric structure***:  Figure 3 presents the post-fine-tuning characteristics of the LoRA module within Llama-7B across four different tasks, analyzing the same submodules.  Figure 3a displays the total 4x128 submodules across four LoRAs. Figure 3b illustrates the breakdown of the A matrices (even-numbered), **the same submodules  ( same index) overlap significantly, making them indistinguishable**. Conversely, Figure 3c shows the B matrix breakdown (odd-numbered) where, after training on different tasks, **the same submodules demonstrate distinct differences, facilitating clear differentiation**. This analysis substantiates the HydraLoRA approach of sharing the \"A\" module and training distinct \"B\" modules to perfectly couple the **synergies and differences** across tasks.\n\n- ***Workflow***. For Section 3.2, the HydraLoRA fine-tuning first involves categorizing datasets to initialize the number of B matrices, essentially constructing asymmetric structures. Subsequently, these B matrices serve as the experts of MoE (Eq. 3). We'd like to clarify that **HydraLoRA goes beyond a simple MoE for the PEFT approaches**. Our core focus is on **better understanding and analyzing the LoRA structure** (line 35), which delivers superior model performance while maintaining the efficiency benefits of a reduced parameter footprint.\n\n> W2：Novelty of HydraLoRA & Compared with a LoRA MoE Work: MOLE [1].\n\n- **Novelty**. We'd like to clarify that HydraLoRA represents an asymmetric **architecture** enhancement of the vanilla LoRA, while existing LoRA MoE approaches [1,2,3,4] serve as LoRA **frameworks** for multi-tasks. **MoE plays a secondary role** in HydraLoRA. We leverage it as a method to aggregate these asymmetric B-matrix modules. Thus, **HydraLoRA architecture** can be **seamlessly adapted into** existing enhancements to the LoRA MoE framework, further extending its capabilities and effectiveness. \n\n- **Compared with MOLE [1]**.  As the reviewer mentioned, we had noticed the LoRA MoE work MOLE [1] before, but since it is **NOT open-sourced** (https://github.com/yushuiwx/MoLE/issues). To be fair, in Table 3, we compare HydraLoRA with another similar LoRA MoE work ICLR 2024 [2] and COLM 2024 [3]. Meanwhile, we have attempted to reproduce MOLE [1], which is not a guaranteed fair comparison. The results are as follows, MOLE underperformances [2, 3], and  HydraLoRA still achieves better performance, which **further proves the strong adaptability and efficiency of HydraLoRA**.\n\n|Llama2-7B|Base|Lorahub [3]|LoRA MoE [2]|MOLE [1] |HydraLoRA|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|**BBH**|31.6|39.7|40.3|37.4|41.5|\n\n\n> W3：Results are incremental.\nHydraLoRA achieves superior performance on downstream tasks with fewer parameters. Specifically,\n- Compared with LoRA (r=8), HydraLoRA (r=8) demonstrates a **performance gain of over 5%**, as shown in Table 2; \n- Compared with strategies that employ multiple LoRAs directly for Mixture of Experts (Table 3) and LoRA (r=32) (Table 2), HydraLoRA enhances efficiency by sharing the \"A\" module to capture task synergies and training distinct \"B\" modules to recognize task differences. Consequently, HydraLoRA significantly **reduces about 88.5% of parameters** compared to existing methods [2].\n\n> Q1: Comparison of inference speed.\n\nFor inference, the speed is primarily influenced by the base model. Since the parameters of PEFT modules constitute a small fraction of the total model parameters (ranging from 0.001% to 0.248% as shown in Table 2), the inference latency differences among various PEFT methods are minimal. \n\nThe following presents the latency and energy consumption during inference using Llama2-7B with different PEFT methods, evaluated on the WikiText2 dataset using a single NVIDIA A40 GPU. The results show nearly equal energy consumption and latency, but HydraLoRA exhibits the highest model performance.\n\n|  |Latency(s)|Energy(Wh)|MMLU(%)|\n|---|:---:|:---:|:---:|\n|**LLaMA2-7B**|90.21|72.72|38.88|\n|**+Prompt Tuning**|91.78(+1.57)|73.53(+0.81)|39.91(+1.03)|\n|**+P-Tuning**|91.3(+1.13)|73.87(+1.15)|41.11(+2.23)|\n|**+Prefix Tuning**|92.52(+2.31)|74.21(+1.49)|41.78(+2.90)|\n|**+LoRA (r=8)**|92.28(+2.07)|73.95(+1.23)|43.22(4.34)|\n|**+HydraLoRA (r=8)**|92.86(+2.65)|74.25(+1.53)|47.22(+8.34)|\n\n> Q2：Comparison of training overhead.\n\n- ***Compared with LoRA variants and LoRA MoE methods***.\nHydraLoRA not only enhances performance with the same parameters LoRA variants (rank=16) as shown in Table 2, but it also demonstrates **substantial parameter reductions** compared with LoRA MoE— **reducing 88.5% compared to [2] and 72.5% to [3]** as shown in Table 3. Moreover, as Figure 5 illustrates, HydraLoRA **cuts energy consumption by 50%** compared to Split-LoRA, which uses multiple LoRA modules. This underscores HydraLoRA's efficiency and system-friendly.\n\n- ***Compared with other PEFT***. While the vanilla LoRA method involves a higher computational overhead than other PEFT strategies, it offers significant performance gains, as shown in Table 2.  However, LoRA's carbon footprint is **negligible** compared to full-parameter tuning, emphasizing its environmental and computational advantages[4].  Meanwhile, the **fine-tuning is a one-time event**, but inference overhead is crucial. As noted earlier, HydraLoRA boosts performance with minimal additional overhead.\n\nReferences:\n\n[1] Mixture of LoRA Experts, ICLR 2024.\n\n[2] Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. ICLR 2024.\n\n[3] Lorahub: Efficient cross-task generalization via dynamic lora composition, COLM 2024\n\n[4] [Carbon Footprint of LLM Fine Tuning — A Case Study](https://towardsdatascience.com/carbon-footprint-of-llm-fine-tuning-a-case-study-7703afc716a9).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fw4X502sJg", "reviewer": "Reviewer_dwiN", "summary": "The paper presents HydraLoRA, an innovative and asymmetric Low-Rank Adaptation (LoRA) framework designed to enhance the efficiency of fine-tuning Large Language Models (LLMs) for specific tasks. The authors identify inefficiencies in the original LoRA approach, particularly its underperformance in complex domains, and propose HydraLoRA to address these issues.", "questions": "1. How does the asymmetric structure of HydraLoRA impact the interpretability of the model, and can the authors provide insights into how different components of the model contribute to the final predictions?\n\n2. The paper uses k-means for initialization. How sensitive are the results to the choice of initialization method, and how does this impact the overall performance?", "limitations": "1. The paper uses k-means for initialization, but it is not clear how sensitive the model's performance is to the choice of initialization method\n\n2. The use of multiple adapter copies in HydraLoRA leads to higher training costs compared to conventional PEFT methods.\n\n3. The asymmetric structure of HydraLoRA may introduce complexity in terms of model interpretability.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Improved Efficiency: The framework requires no domain expertise and outperforms other Parameter-Efficient Fine-Tuning (PEFT) methods, including those that use domain knowledge during training and inference.\n\nGeneralizability: The framework shows robust generalization across unseen tasks without relying on prior task-specific knowledge, making it a versatile solution for adapting LLMs to various domains.\n\nResource Optimization: HydraLoRA is designed to be parameter-efficient, which not only improves performance but also reduces the computational resources required for training and deployment of LLMs.", "weaknesses": "HydraLoRA is more computationally intensive than conventional Parameter-Efficient Fine-Tuning (PEFT) methods due to the use of multiple adapter copies.\n\nHydraLoRA It requires more training iterations, which can be 1 to 2 times higher than typical PEFT methods, affecting the environmental footprint of model training.\n\nThe study primarily examines LoRA and does not test additional configurations like prompt-tuning and adapter layers, limiting the scope of the findings.\n\nThe method's practical effectiveness in real-world applications outside the experimental setup is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below：\n\n> W1& Limitation2: HydraLoRA has multiple adapter copies.\n\nThe reason for multiple \"B\" modules is that, in practice, downstream tasks are often complex and **multi-task**. Traditional PEFT  methods typically focus on optimizing for a single task. Tuning a single LoRA to serve all tasks without considering the task differences can lead to reduced performance.  \n\nCurrent approaches [1,2,3,4] often **train multiple LoRA modules for multiple tasks, similarly overlooking the task synergies**. In contrast, HydraLoRA, by sharing the \"A\" module and training distinct \"B\" modules, perfectly couples these two features, leading to **superior performance**. Moreover, by sharing the \"A\" module, HydraLoRA significantly **reduces the parameter count to just 11.5% of that of existing methods** [2], as detailed in Table 3. Nonetheless, the additional computation introduced is negligible compared to the parameters of LLMs themselves, as shown in Table 2 where HydraLoRA accounts for only 0.124% of the total parameters.\n\n> W2: Environmental footprint of model training.\n\nWhile the vanilla LoRA method incurs higher computational overhead compared to other PEFT approaches, it also **delivers significant performance improvements**. HydraLoRA, an adaptation of LoRA, enhances downstream task performance with the same parameter settings (rank=16), as demonstrated in Table 2. \n\nMoreover, as Figure 5 illustrates, HydraLoRA **cuts energy consumption by 50% compared to Split-LoRA**, which uses multiple LoRA modules. This underscores HydraLoRA's efficiency and its eco-friendly nature. Additionally, the **carbon footprint of fine-tuning LoRA is effectively negligible when contrasted with full-parameter tuning**, highlighting its environmental and computational benefits [5].\n\n> W3: Primarily on LoRA, not test other PEFT configurations.\n\nOur core focus is on **better understanding and analyzing the LoRA structure** (line 35). We first perform a thorough analysis of the LoRA structure, showing that the asymmetry  (Figure 3) is primarily due to the different initialization methods of the A and B matrices. However, this characteristic may not be directly transferable to other PEFT methods. We appreciate your suggestion and will consider how similar explorations might be applied to other PEFT techniques.\n\n\n> W4: Outside the experimental setup is not discussed.\n\nWe have validated HydraLoRA on representative **single-domain** datasets in General, Medical, Law, Math, and Code (line166 - line175), as well as on the **multi-domain** dataset Flanv2, which covers 10 distinct task clusters (line175 - line178), effectively simulating common scenarios. We hope this addresses the reviewer's question and we are willing to answer more questions about the setup.\n\n\n> Q1&Limitation3: HydraLoRA's asymmetric structure interpretability.\n\nOur analysis of the LoRA module breakdown (Figure 3) revealed asymmetrical properties of the A-B modules: post-training, the **A module shows similarities across tasks, whereas the B module exhibits distinct differences**. This observation aligns with the synergies and differences encountered in downstream multi-task learning with LLMs. Consequently, we have refined the existing LoRA structure and introduced the HydraLoRA asymmetric architecture (Figure 1.C). In this design, the A module captures the commonalities of knowledge, while the B module captures specific characteristics. We hope this addresses the reviewer's question. Could the reviewer please clarify what is meant by “interpretability of the model”? We apologize for any confusion.\n\n> Q2&Limitation1: Initialization of K-means.\n\nAs discussed in Section 4.5, we find that the number k of clusters is **NOT a sensitive parameter** for HydraLoRA with a wide range of reasonable number k of clusters performing decently well in all settings in our experiments (Figure 8). We also compare K-means with sophisticated hyperparameter search approaches and find that K-means is simple but effective (Figure 7).\n\nReferences:\n\n[1] Mixture of LoRA Experts, ICLR 2024.\n\n[2] Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. ICLR 2024.\n\n[3] Lorahub: Efficient cross-task generalization via dynamic lora composition, COLM 2024.\n\n[4] When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications, SIGIR 2024.\n\n[5] [Carbon Footprint of LLM Fine Tuning — A Case Study.](https://towardsdatascience.com/carbon-footprint-of-llm-fine-tuning-a-case-study-7703afc716a9)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wJSkGsx1Qt", "reviewer": "Reviewer_vvtw", "summary": "This paper proposes two improvements to LoRA geared towards heterogeneous corpora on which LoRA underperforms full fine-tuning. First, it proposes training a number of smaller Lora heads (Ai,Bi) (Lora-Split) rather than a single head which improves performance while preserving the overall number of parameters. Second, the paper proposes an improvement over Lora-Split - called HydraLora - which reduces the number of parameters by sharing 'A' matrices across domains while allowing Bi's to vary across domains. This variant uses Mixture-of-Experts strategy for training/inference and improves performance over Lora with fewer parameters.", "questions": "* L54: 'autonomous' -> 'automatic'\n* Fig 2: How is corpus heterogeneity measured?\n* Table 1: How is Lora split trained - Is each Lora head trained on examples from a specific domain? If so, what are the domains? Are these domains naturally occurring in the corpus? or were they inferred by 'k-means clustering'?\n* Table 3: What is the performance of Lora-split?\n* L193: 'With equivalent parameters (rank=16), …'  - this is unclear since Table 2 reports performance of HydraLora with rank=8.\n* Figure 7: what is the x-axis?\n* L259: How does the variant without MoE work?", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* Proposes a new method called HydraLoRA that improves performance over LoRA on heterogeneous corpora with fewer parameters (originality)\n* HydraLoRA does not require domain expertise either at training or inference (originality).\n* HydraLoRA improves training speed by around 2X relative to LoRA (significance)\n* Reports ablations showing what components matter in the final model (Quality)\n* Proposed method is likely to be employed by a number of researchers who are working on datasets which exhibit heterogeneity (Significance)\n* The paper presents an observation based on a tsne analysis whereby 'A' matrices from Lora heads are similar across domains while 'B' matrices vary. This is a really useful form of visualization that could be used by researchers working with LoRA (Significance)", "weaknesses": "* The proposed method still underperforms full fine-tuning.\n* It looks like inference using HydraLoRA routes each example to all experts i.e. B matrices and then computes a weighted average. The paper does not provide an ablation where only one of the B matrices (argmax of the gating score) is used at inference time, which may further reduce inference cost.\n* There are some details in the paper which are not clear. See questions below.\n\nThanks to the authors for addressing many of these issues in the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below：\n\n> W1: HydraLoRA still underperforms full fine-tuning.\n\n- ***HydraLoRA is more efficient***. HydraLoRA offers the advantage of low training overhead, allowing LLMs to adapt to specific domain tasks more efficiently than Full Fine-Tuning (FFT).  Although HydraLoRA may not match FFT in model performance, as depicted in Figure 2, FFT adjusts all parameters (as shown in Table 2, **FFT modifies tunable parameters about 800 times higher than HydraLoRA**), which better captures downstream task features but also incurs substantial costs that may be **prohibitive for end-users**.  **FFT cannot construct efficient multi-head structures** like LoRA.  Meanwhile, compared to other PEFT methods, **HydraLoRA minimizes this performance gap with FFT**, as shown in Table 2.\n\n- ***HydraLoRA is more robust and adaptive***. As the downstream tasks dynamically evolve, the overhead of re-running the FFT process is significant. However, **HydraLoRA easily adapts to the changes** due to its plug-and-play and asymmetric architecture.\n\n> W2: Ablation where only one of the B matrices.\n\nThanks for your constructive comment.  We add more experiments with the same setting as Table 3, to explore how the number of experts (B matrices) during the HydraLoRA inference pipeline influences performance. \nAs shown in the table below, we find that an increase in the number of B matrices generally leads to enhanced performance in downstream tasks. In practice, **user requests may belong to different tasks, while a single request potentially involves mixed tasks**. This improvement can be attributed to the expanded configuration space afforded by additional LoRA modules, which allows for a more fine-grained and tailored adaptation to the diverse and mixed-task inputs encountered in the benchmark.\n\n| Methods | Base | Top-1  | Top-3 | HydraLoRA |\n|:---:|:---:|:---:|:---:|:---:|\n| **BBH**  | 31.6   | 35.4 | 38.6 | 41.5   |\n\nTable: Sensitivity analysis of the number of B matrices. ‘Base’ means vanilla Llama2-7B, Top-1 means selecting the highest-ranked (top-1) B matrix, and Top-3 means selecting three highest-ranked (top-3) B matrices.\n\n> Q1: Typos.\n\nThanks for pointing out them. We will correct all the typos in the updated version.\n\n> Q2: How is corpus heterogeneity measured?\n\nHeterogeneity signifies the diversity within the dataset. To visualize the diversity, we adopt similarity between task embeddings for different tasks. We place an example heatmap figure in the overall rebuttal pdf.\n\n> Q3: How does lora_split in Table 1 classify the data?\n\nTo simulate real-world scenarios, we cannot know in advance the domains of data that require fine-tuning. Therefore, **Split-LoRA, a baseline we proposed**, performs k-means clustering on the data and then fine-tunes it for different categories. This approach underscores the importance of exploiting asymmetry in HydraLoRA.\n\n> Q4: The performance of Split_LoRA in Table 3.\n\nFor a single dataset, no existing studies have discussed multi-LoRA fine-tuning methods, prompting us to introduce the LoRA-Split variant. In contrast, Table 3 focuses on multi-task\n scenarios, where numerous methods[1,2,3,4] already exist. Therefore, we directly compared our approach with established LoRA MoE methods [1,2].\n\n> Q4: L193 “With equivalent parameters (rank=16)”\n\nIn Table 2, HydraLoRA (r=8) refers to each A/B matrices with a rank of 8, yet the total parameter count is equivalent to a single LoRA module with a rank of 16, due to multiple B matrices. Meanwhile, HydraLoRA demonstrates superior performance, further highlighting its efficiency.\n\n> Q5: What is the x-axis in Figure 7?\n\nFigure 7 displays the dataset classification results for different methods, with the x-axis representing the number of repeated experiments, aiming to provide more representative results through a 15-fold experiment as mentioned on line 274.\n\nReferences:\n\n[1] Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. ICLR 2024.\n\n[2] Lorahub: Efficient cross-task generalization via dynamic lora composition, COLM 2024.\nReferences:\n\n[3] Mixture of LoRA Experts, ICLR 2024.\n\n[4] When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications, SIGIR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear PCs, SAC, AC, and Reviewers:

We sincerely appreciate your thoughtful review and insightful comments, we have tried our best to address your concerns one by one in the correspondence rebuttal sessions. If our responses address your concerns, we would be grateful if you could consider raising your final rating to a higher score.

Attached is a PDF containing the *task embedding similarity heatmap*, supplementing Question 2 posed by **Reviewer vvtw**.

Wishing you all the best,

Sincerely,

Authors

Meta-review: This paper presents a architecture improvement to LoRA that brings in compute efficiency and also quality improvements. Basically, the authors split the Lora heads first into multiple heads while keeping the number of parameters the same, and then within the split heads they use the same A matrix for the multiple heads thus sharing more parameters while introducing a MoE architecture. The experimental setup is robust and some nice qualitative analysis is provided for the results.
Decision: Accept (oral)