Title: HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Models

Efficient Fine-Tuning

Asymmetric Structure
Abstract: Adapting Large Language Models (LLMs) to new tasks through fine-tuning has been made more efficient by the introduction of Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA. However, these methods often underperform compared to full fine-tuning, particularly in scenarios involving complex datasets. This issue becomes even more pronounced in complex domains, highlighting the need for improved PEFT approaches that can achieve better performance. Through a series of experiments, we have uncovered two critical insights that shed light on the training and parameter inefficiency of LoRA. Building on these insights, we have developed HydraLoRA, a LoRA framework with an asymmetric structure that eliminates the need for domain expertise. Our experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even those that rely on domain knowledge during the training and inference phases. Our anonymous codes are submitted with the paper and will be publicly available. Code is available: https://github.com/Clin0212/HydraLoRA.

HydraLoRA : An Asymmetric LoRA Architecture for Efficient Fine-Tuning

Abstract

Adapting Large Language Models (LLMs) to new tasks through fine-tuning has been made more efficient by the introduction of Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA. However, these methods often underperform compared to full fine-tuning, particularly in scenarios involving complex datasets. This issue becomes even more pronounced in complex domains, highlighting the need for improved PEFT approaches that can achieve better performance. Through a series of experiments, we have uncovered two critical insights that shed light on the training and parameter inefficiency of LoRA. Building on these insights, we have developed HydraLoRA , a LoRA framework with an asymmetric structure that eliminates the need for domain expertise. Our experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even those that rely on domain knowledge during the training and inference phases. Code is available.

1 Introduction

Large Language Models (LLMs; [10, 3, 36, 47, 48, 32, 33]) are notably powerful, yet their training involves substantial expense. Adapting a single LLM for multiple downstream applications via finetuning has emerged as a prevalent method to cater to specific domain needs, balancing performance with practicality. This approach, however, faces a significant challenge due to the extensive memory and computational resources required for full fine-tuning (FFT), i.e., fine-tuning all billions of parameters. A solution to this has been the development of more selective adaptation techniques, involving modifying only a portion of the parameters or integrating external modules designed for new tasks. Key methodologies in this sphere include LoRA [18], Adaptors [37, 17, 31], and many other variants [25, 24, 9, 14, 53], all part of what can be generally termed as Parameter-Efficient Fine-tuning (PEFT). PEFT strategies are characterized by freezing the backbone model parameters while only a minimal number of task-specific parameters are introduced and fine-tuned. This method substantially boosts efficiency in the phases of fine-tuning and subsequent deployment, marking a significant advancement in the practical use of LLMs.

While fine-tuning a small subset of parameters offers a streamlined approach for domain adaptation, it's well-recognized that model performance is closely tied to the number of parameters involved [22]. This intrinsic characteristic of methods like LoRA often results in them falling short of the FFT baseline, which updates all parameters, thereby creating a trade-off between efficiency and model quality. This issue of compromised quality in a low-parameter setting becomes even more pronounced in target domains characterized by complex sub-domains and diverse tasks. This situation presents a compelling research question:

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

...

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

...

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

'

(a)

(b)

(c)

(C)  AutoLoRA: Split-fusion mul

Tasks

Auto-Router

What is the optimal architecture that can deliver superior model performance while still capitalizing on the efficiency benefits of a reduced parameter footprint?

GLYPH(cmap:d835)

GLYPH(cmap:d835)

In our research, we carry out a series of exploratory experiments, applying LoRA to the LLaMA2 [48] model to adapt it to a new domain encompassing multiple downstream tasks. As shown in Figure 1(a), LoRA adds trainable pairs of rank decomposition matrices A and B in addition to existing weight matrices. Our in-depth analysis of LoRA's mechanics yields several insightful observations and leads to the formulation of key hypotheses. First, rather than employing a single LoRA for the entire domain, it proves more effective to deploy multiple, smaller LoRA heads, each dedicated to a specific downstream task (see Figure 1(b)). This suggests that domain or task interference might harmfully impact the training process. We further hypothesize that this interference originates from 'intrinsic components' -sub-domains or distinct tasks-potentially unknown even to domain experts. Additionally, upon visualizing the parameters of LoRA, we discern a pattern: some parameters predominantly learn the commonalities across all data, while others focus on the unique aspects of each intrinsic component. From these observations, we posit that an optimal LoRA architecture should embody an explicit, asymmetric structure.

Building upon the observations, we propose an improved end-to-end LoRA framework, which we refer to as HydraLoRA . From the architecture perspective, unlike LoRA's symmetric structure, HydraLoRA has an asymmetric structure that has a shared A matrix and multiple B matrices (see Figure 1(c)). The shared A matrix is used by all samples for parameter efficiency. During the fine-tuning phase, HydraLoRA is designed to auto-identify 'intrinsic components' and segregate training samples into distinct B matrices. During the inference phase, HydraLoRA leverages multiple B matrices using Mixture-of-Experts (MoE; [20, 40]) manner. Unlike prior work, HydraLoRA completely eliminates the need for human expertise and assumptions, showing better performance than using domain knowledge to guide the fine-tuning process.

2 Background and Motivation

2.1 LoRA Basics

LoRA [18] achieves comparable performances to fine-tuning on many benchmarks by freezing the pre-trained model weights W 0 and inserting trainable rank decomposition matrices into each layer of the pre-trained model. In particular, for each layer, LoRA uses two sequential low-rank matrices A and B to fit the residual weights for adaptation. The forward computation is written as follows:

where y ∈ R d is the output and the x ∈ R k denotes the input. B ∈ R d×r , A ∈ R r×k with r ≪ min ( d, k ) . Normally matrix B is initialized with zeroes and matrix A is initialized with Kaiming Uniform [15] to force ∆ y = 0 at the beginning.

GLYPH(cmap:d835)

GLYPH(cmap:d835)

A

GLYPH(cmap:d835)

GLYPH(cmap:d835)

A

...

Shared

...

Auto Ini

...

2.2 LoRA's Practical Dilemma

Parameter count has a clear impact on the performance of neural models [22, 33]. Yet, ParameterEfficient Fine-tuning (PEFT) methods, such as Adapter [17] and prefix-tuning [25], focus on finetuning a limited set of parameters. These approaches present a practical dilemma: while restricting the number of tuned parameters is essential for training efficiency, it hinders the model's ability to learn from diverse datasets. This trade-off becomes particularly evident when considering corpus heterogeneity [2]. Figure 2 reveals a notable performance disparity between PEFT techniques and full fine-tuning (FFT), with the gap widening in scenarios involving a more diverse or heterogeneous training corpus.

Corpus  Heterogeneity

Performance

Increased Gap

Full Parameter Fine-Tuning

Parameter-Efficient Fine-Tuning

2.3 Observations

In this work, we aim for a PEFT approach that strikes a better balance between maximizing the learning capability for heterogeneous data and minimizing the number of parameters involved. A key goal is to ensure that our enhanced technique exhibits robust generalization across unseen tasks, independent of any prior task-specific knowledge. To achieve our objectives, we focus on LoRA and conduct a series of experiments as Table 1 to gain a deeper understanding of its mechanisms. Our methodology involves leveraging data from diverse tasks within a domain, and training distinct LoRA heads for each domain, leading to our first observation:

Observation I : With the same parameter count, rather than employing a single LoRA for the entire domain dataset, it proves more effective to deploy multiple, smaller LoRA heads, each dedicated to a specific downstream task.

This suggests that interference among tasks might harmfully impact the training process. Furthermore, we posit that this interference is NOT exclusive to this explicit multi-task training. This interference could happen in any training setting since all datasets inherently consist of multiple implicit intrinsic components , such as sub-domains or tasks within a domain that is even unknown to domain experts. To better understand how multiple LoRA heads mitigate the interference among intrinsic components, in Figure 3, we employ the t-SNE technique [49] to visualize the parameters of matrix A and B across all heads. This analysis yields another critical observation:

Observation II : When multiple LoRA heads are trained individually on different data, the parameters of matrix A from different heads tend to converge, while those of matrix B are distinguishable.

In detail, the parameters of matrix A across all heads exhibit a high degree of similarity, leading to their overlaps in the figure. Conversely, the parameters of matrix B from different heads are distinct and easily distinguishable. We posit that this divergence is an artifact of the initialization schemes, with matrix A inclined toward capturing commonalities across domains, while matrix B adapts to domain-specific diversities. The distinction between matrix A and B offers valuable insights for enhancing both parameter efficiency and effectiveness. From an efficiency standpoint, our hypothesis suggests that the parameters of matrix A could potentially be shared across multiple heads, thereby reducing redundancy. Regarding effectiveness, since the parameters of matrix B of different heads

100

50

0

50

100

-

100

-

50

0

50

100

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

t-SNE Visualization of BA-Matrix Diff. LoRAs

S

UM

QA

IE

ALL

100

50

0

50

100

-

100

-

50

0

50

100

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

32

34

36

38

40

42

44

46

48

50

52

54

56

58

60

62

64

66

68

70

72

74

76

78

80

82

84

86

88

90

92

94

96

98

100

102

104

106

108

110

112

114

116

118

120

122

124

126

t-SNE Visualization of A-Matrix Diff. LoRAs

SUM

QA

IE

ALL

50

0

50

-

75

-

50

-

25

0

25

50

75

1

3

5

7

9

11

13

15

17

19

21

23

25

27

29

31

33

35

37

39

41

43

45

47

49

51

53

55

57

59

61

63

65

67

69

71

73

75

77

79

81

83

85

87

89

91

93

95

97

99

101

103

105

107

109

111 113

117 115

119

121

123

125

127

t-SNE Visualization of B-Matrix Diff. LoRAs

SUM

QA

IE

ALL

-

-

-

-

-

are dispersed, suggesting that using a single head to adapt to multiple domains might be less effective than using individual heads for each domain, which minimizes the interference between domains.

Building upon our observations, we propose an optimized LoRA architecture designed to enhance cost-effectiveness. In this architecture, we share the parameters of A matrix across various subdomains or tasks to improve parameter efficiency, while deploying multiple B matrices, each tailored to handle different intrinsic components. This design allows for a more effective adaptation to the specific characteristics of each component. While these intrinsic components can be manually identified using prior knowledge of the training data, we also introduce end-to-end methods using Mixture-of-Experts (MoEs) [21], which will be detailed in the methodology section. This automatic approach facilitates flexibility and applicability, particularly in scenarios where prior knowledge is limited or unavailable.

3 HydraLoRA

In this section, we introduce the proposed HydraLoRA , an asymmetric LoRA architecture for efficient fine-tuning, as illustrated in Figure 1. After that, we show the workflow of HydraLoRA as Figure 4.

3.1 Asymmetric LoRA architecture

The LoRA method updates two low-rank matrices A and B , and uses AB as the change of a pretrained and frozen weight W 0 of a linear layer as shown in Eq. 1. The integral parameters are fine-tuned for the whole corpus in the original LoRA, which causes difficulty in learning the various knowledge aspects. Drawing from a detailed breakdown analysis of LoRA, a potential solution is to segment the entire LoRA into 'Hydra' structured LoRA variants, that is, characterized by a central shared matrix A and several distinct matrices B , fostering a blend of shared knowledge and specialized functionalities. As Figure 1, HydraLoRA is to fine-tune LoRAs to achieve robust performance without redundancy, thereby benefiting the entire heterogeneous corpus. The asymmetric LoRA architecture can be formulated as:

The matrics B i ∈ R d × r and shared A ∈ R r × k . The hyper-parameter N denotes the number of B matrices. The term ω i modulates these contribution weights for head B i .

3.2 Workflow of HydraLoRA

Figure 4 illustrates the workflow of HydraLoRA . Initially, HydraLoRA delves into the adaptive identification and initialization of LoRA modules within a heterogeneous corpus, aligning them with

...

...

Attention

LN

FNN

LN

Pretrained

Weights

GLYPH(cmap:d835)

∆GLYPH(cmap:d835)

GLYPH(cmap:d835)

∆GLYPH(cmap:d835)

GLYPH(cmap:d835)

∆GLYPH(cmap:d835)

GLYPH(cmap:d835)

Router

A

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Output

Mixed-task Input

Trained HydraLoRA

1. Initialization

Segregate training

samples to intrinsic

components by MoE

3. Inference

B. Inference

2. Tuning

Routing LoRA Merge

...

Router

Heterogeneous

Corpus

Identify intrinsic

component

s

Initialized HydraLoRA

task relevance through the application of k -means or developer-specified size. Subsequently, we propose a Mixture-of-Experts (MoE) framework that handles B matrices as expert adapters to ensure computational efficiency throughout the fine-tuning (Section 3.2.1) and inference (Section 3.2.2) stages by freezing the rest of the LLM parameters. During inference, it flexibly and dynamically merges multiple B matrices through the MoE router.

3.2.1 Fine-tuning

Motivated by Mixture-of-Experts (MoEs; [20, 40]), where experts are selectively activated by a gating mechanism (Router) in response to different inputs. In HydraLoRA , we substitute each expert with a lightweight LoRA adapter. During fine-tuning, while weights of LLMs remain frozen, the experts and router layers are trained from scratch. In order to achieve a unified approach to the distinct forward processes of multiple B matrices, we define a set of experts, denoted as ( E 1 , . . . , E N ) , to learn the updated matrix ∆ W . As HydraLoRA fine-tunes the experts using the heterogeneous corpus, the shared matrix A inherently captures collaborative knowledge to augment intra-gains, and different matrices B foster knowledge modularity to mitigate fine-tuning inter-offsets. Based on this structure, the forward process of HydraLoRA is expressed as:

where N denotes the number of experts, i.e., B matrices.To regulate these contributions, we introduce a gate function (router network) commonly consisting of a dense layer with trainable weights (transformation matrix) W g ∈ R r × N followed by a softmax function which takes an intermediate token representation x as input and combines the output of each expert based on the gating scores ( ω 1 , . . . , ω N ) :

3.2.2 Inference

During inference, HydraLoRA merges adapters by enabling routing computation based on the input. Specifically, since matrices B operate as linear functions, we initially compute a weighted average of the experts. Following this, we apply a PEFT transformation using the combined expertise. The HydraLoRA significantly enhances training efficiency through an extremely parameter-efficient MoE formulation. Additionally, the intrinsic structural modularity of HydraLoRA facilitates rapid recovery and merging of the trained parameters during inference, leading to substantial memory savings.

4 Experiments

In this section, we detail the principal experiments. We begin with an overview of the experimental setup and implementation intricacies. Following this, we share our findings and offer a succinct interpretation.

4.1 Experiment Setting

Dataset and Benchmarks To explore the properties and commonalities of the LoRA asymmetric structure, we conduct experiments on both single and multiple domains to evaluate the effectiveness of HydraLoRA for profiling intrinsic components. · Single domain. 1) General : we fine-tune with the general instruction tuning databricks-dolly-15k [8] for generic language capability and evaluate with MMLU [16]. 2) Medical : we fine-tune with GenMedGPT and clinic-10k from ChatDoctor [26] for medicine applications and evaluate medical tasks in MMLU. 3) Law : we fine-tune with two legal instruction tuning datasets Lawyer-Instruct [1] and US-Terms [4] then evaluate with law tasks in MMLU. 4) Math : we fine-tune with the training split of GSM8K [7] for mathematical reasoning and evaluate with test set of GSM8K. 5) Code : we fine-tune with CodeAlpaca [5] for code generation and evaluate with HumanEval [6]. · Multi-task domain. We select a portion of the Flanv2 [51] datasets covering Natural Language Understanding (NLU) and Natural Language Generation (NLG), which can be grouped into 10 distinct task clusters. Then we evaluate it with the Big-Bench Hard (BBH) [45] benchmark. A detailed description of the benchmarks can be found in Appendix A.1.

Baselines · First, we compare HydraLoRA with different PEFT methods on single datasets: 1) Full fine-tuning ; 2) Prompt Tuning [24]; 3) P-Tuning [29]; 4) Prefix Tuning [25]; 5) IA 3 [27]; 6) AdaLoRA [55]. · Second, we extend the experiments exploring HydraLoRA on multiple datasets compared with more weighted average methods: 1) Lorahub [19] employs black-box optimization to learn weights of 20 randomly selected LoRAs for new tasks, using weighted averaging without needing gradient calculations. 2) LoRA MoE [52] combines lightweight experts (LoRA) with MoE architecture for high efficiency, generalizing to new tasks without prior knowledge. A detailed description of the baseline models can be found in Appendix A.2.

4.2 Overall Performance

The experimental results of HydraLoRA and the competing baselines are presented in Table 2 with a single domain and Table 3 with the mixed task domain. The evaluation of diverse tasks demonstrates that HydraLoRA consistently outperforms all other schemes. The performances rooted in LoRA outperform those of conventional PEFT methodologies. Compared to the default single LoRA configuration (rank=8), the Hydra architecture, enriched by the integration of several B matrices, effectively addresses the inherent conflicts among intrinsic components of the corpus. Furthermore, with equivalent parameters (rank=16), the model shows superior performance, confirming the ef-

fectiveness of the adopted parameters. Based on Table 2 and Table 3, we propose three research questions that confirm the aforementioned observations.

RQ1: Is it more effective to use multiple smaller LoRA heads for specific tasks rather than one single LoRA for the entire domain dataset, given the same parameter count? Comparing the high-dimensional LoRA configuration with r = 32 against a segmented version using LoRA-Split, a variant introduced by HydraLoRA, which divides the model into four distinct components each with r = 8 . That is, multiple vanilla LoRAs are directly utilized to capture the differences between data. We observe a noteworthy trend in the performance across a variety of tasks as detailed in Table 2. It illustrates the superior performance of LoRA-Split in comparison to the traditional LoRA approach, across all the evaluated scenarios. This enhancement in performance is a strong indication of the detrimental impact that task interference can have on the training process. By segregating the tasks into discrete components, LoRA-Split effectively minimizes the conflict and interference between tasks, thereby promoting a more efficient and focused environment.

The concept of LoRA-Split hinges on the construction of different intrinsic component compositions, employing LoRA as a foundational technique to strategically mitigate the interference conflict. This architectural innovation has proven to be a pivotal factor in enhancing model performance. However, it's important to note that while LoRA-Split marks a significant advancement in model efficiency and task handling, it also introduces a certain level of parameter redundancy. The segmented approach of LoRA-Split inevitably leads to an increase in the overall model parameters, which can be manifold in comparison to the traditional, singular LoRA model. This increase in parameters, while contributing to the model's robustness and capability to handle multiple tasks simultaneously, also poses new challenges in terms of computational resources and model optimization.

RQ2: Will multiple LoRA heads, individually trained on different data, improve efficiency by distinguishing matrix B parameters? We evaluated the Hydra structure LoRA HydraLoRA that is characterized by a shared LoRA A matrix, while maintaining distinct B matrices that are trained separately. This configuration was meticulously compared with both the standard LoRA and the LoRA-Split approaches, emphasizing efficiency parameters.

According to the results presented in Table 2, unlike split which straightforwardly adopts multiple vanilla LoRAs, HydraLoRA adopts an asymmetric LoRA structure that not only improves parameter efficiency by separating the uses of A matrix for commonalities and B matrices for diversities with a notably smaller adapter parameter set, but also employs a trainable router to improve the composition of multiple B matrices that outperforms the LoRA-Split approach. This finding is significant as it suggests that HydraLoRA not only enhances performance efficiency but also boosts overall system effectiveness. This may be driven by 1) different B matrices capturing different features of the data-intrinsic knowledge, mitigating mutual interferences, and avoiding performance offsets. 2) Module A maintains the collaborative knowledge by taking the strengths of each and integrating them to improve the model performance.

RQ3: How does HydraLoRA fare against other merge methods in complex, multi-task domains, considering scalability and robustness? While we hypothesize that the asymmetry is mainly rooted in the different initialization methods of A and B matrices, it is possible that this behavior varies on different model architectures and datasets. Recent work confirms similar empirical observations [54, 13]. To the best of our ability, we extended the experiments exploring HydraLoRA on multiple





















Energy (KWh)

















 













Latency (h)







35.0

37.5

40.0

42.5

45.0

47.5

50.0

Performance (%)

43.22

41.59

37.85

42.81

40.92

37.12

46.94

45.28

41.35

47.22

45.71

42.18

w/o Gate

HydraLoRA

datasets. LoRA MoE and their variants typically aim at tackling multi-tasks by employing multiple independent LoRAs. This makes them suitable for handling various domains. However, for a single dataset like ours, a 'default' MoE method might not be optimal. HydraLoRA addresses this by constructing asymmetric structures and utilizing multiple B matrices to capture the specific nuances within the single dataset. The effectiveness of this approach is demonstrated by the experimental results in Table 3.

4.3 Energy and Throughput Analysis

RQ4: How does the 'Hydra' structure in HydraLoRA enhance system efficiency, particularly in reducing training energy consumption and latency? We evaluate the system efficiency of HydraLoRA from two perspectives: training energy consumption and latency. The following experiments were executed on a GPU infrastructure consisting of 4 NVIDIA A40 GPUs and a CPU powered by an Intel(R) Xeon(R) Gold 6330 CPU clocked at 2.00GHz. Power consumption measurements were recorded using CodeCarbon [34]. Figure 5 shows the results of various fine-tuning approaches for GSM-8K using the LLaMA2-7B model. we can see that HydraLoRA effectively speeds up the training process 1 . 96 × and reduces 49.6% energy cost compared to LoRA (rank=32). While the energy consumption and latency of LoRA-Split exceeds the LoRA (rank=32). This is for the reason that HydraLoRA jointly considers inherent knowledge modularity and collaboration, which utilizes the 'Hydra' structure with a shared A matrix and different B matrix. In this way, it only employs rank=16 training overhead but expands to a performance enhancement of more than rank=32. Overall, this experiment demonstrates the parameter effectiveness of HydraLoRA .

4.4 Ablation Study

RQ6: What impact do the MoE architecture and the gate function have on the fine-tuning process? To delve deeper into understanding the contributions of each component in HydraLoRA . we present the results of our ablation study in Figure 6. The variant w/o MoE (essentially reverts to LoRA) excludes the MoE architecture. Similarly, the w/o gate variant employs uniform expert weights bypassing the gate function. The w/o hydra adopts multiple vanilla LoRAs in a straightforward way. Figure 6 indicates that the full HydraLoRA model outperforms its variants, showing that both the MoE architecture and gate function significantly contribute to its effectiveness across various language understanding domains.





























Number of Clusters







4.5 Hyper-parameter Analysis

RQ7: How do the number of intrinsic component of HydraLoRA influence performance outcomes? As Figure 8 shown, we conduct a comprehensive and meticulous analysis by fine-tuning the Dolly-15K model on the LLaMA2-7B dataset and subsequently evaluating its performance on the MMLU benchmark to rigorously examine the impact of variations in the intrinsic component, symbolized by the variable N , on the model's overall performance. Empirically we find that the number N of clusters is not a sensitive parameter for HydraLoRA , with a wide range of reasonable number N of clusters (e.g. 2 to 4) performing decently well in all settings in our experiments. Specifically, the performance loss of N = 3 vs. the optimal N = 4 is only 0.42%. Meanwhile, as illustrated in Figure 7, we employ three distinct methods to generate the number of corpus clusters 15-fold, and the results demonstrate that the k-means [30] yields comparable outcomes with DBSCAN [39]. Therefore, based on this observation, we choose k-means because it is simple but effective, more sophisticated hyperparameter search approaches (e.g. DBSCAN, parameter sweep and Bayesian optimization) will be unnecessarily costly. It's noteworthy that HydraLoRA is adeptly designed to orchestrate its components in a way that it can automatically calibrate and navigate toward the optimal performance configuration across various parameters. This intelligent auto-tuning is achieved through the application of the k-means clustering algorithm. This strategic component orchestration not only enhances performance but also ensures a more efficient and effective utilization of resources, underpinning the model's capability to adapt and perform efficiently in a dynamic computational environment.

5 Related work

Parameter-Efficient Fine-tuning LLMs are becoming increasingly powerful, but fine-tuning them often requires significant computational resources. This has spurred research on parameterefficient fine-tuning (PEFT) techniques that reduce memory and storage costs during adaptation. One prominent PEFT approach is adapters [17, 37]. It introduces new, trainable dense layers within the existing model, keeping the original parameters frozen. This concept has proven successful across various domains [35, 42, 43, 56]. Further improvements on adapter compactness involve constructing parameter matrices using Kronecker products of low-rank matrices [31]. Another PEFT strategy directly manipulates activations with learned vectors. This can be achieved through concatenation [29, 25, 24], multiplication (IA 3 ; [27]), or addition (BitFit; [53]). Prefix-tuning [25] and prompt-tuning [24] are noteworthy examples that fine-tune continuous prompts instead of designing discrete ones [9]. Interestingly, a study suggests that many PEFT methods can be viewed as a form of adapter, providing a unified perspective [14]. Beyond adding new parameters or altering the computational graph, researchers also explore sparse [12, 44, 46] or low-rank updates (LoRA; [18]).

Multi-LoRA Architecture LoRA has notably garnered increasing interest recently, becoming a standard approach for adapting LLMs such as LLaMA [47, 48] under limited computational resources. Recognizing its potential, researchers have delved deeper, exploring the benefits of employing multiple LoRAs. LoraHub [19] takes this multi-LoRA approach by training several adapters and strategically picking combinations based on the domain during inference. Meanwhile, MultiLoRA [50] focuses on horizontal scaling, aiming to reduce LoRA's parameter dependence. This involves splitting LoRA modules along the rank dimension and introducing learnable scaling factors for enhanced expressiveness. Addressing scaling challenges from a different angle, the mixture of LoRA concept is further proposed [52]. This mitigates resource consumption when

N



























Performance (%)

scaling instruction-tuned LLMs. Recognizing the potential for conflict during instruction tuning, LoRAMoE [11] leverages the Mixture-of-Experts (MoEs; [20]) structure to safeguard the pretrained LLM's knowledge from excessive corruption by instruction data. Similarly, MOELoRA [28] incorporates a MoE framework into LLMs, thereby improving their multitasking capabilities in the medical domain. Shifting the focus to the system perspective, S-LoRA [41] provides a framework for efficiently serving multiple LoRA adapters. Unlike previous methods that relied on choosing LoRA combinations based on their training domains, HydraLoRA breaks free from the dependence on domain knowledge during inference. Additionally, HydraLoRA 's asymmetric structure further enhances parameter efficiency compared to existing symmetric approaches.

6 Conclusion

In this work, we start by conducting exploratory experiments applying the LoRA technique to LLaMA2, aiming to adapt it to a new domain across various tasks. This study unveils the limitations of employing a single LoRA for the entire domain, highlighting the detrimental effects of domain interference. In response, we introduce a novel architecture HydraLoRA that features an asymmetric structure with a shared matrix for all samples and distinct matrices for each intrinsic component. This design improves domain adaptation by selectively focusing on distinct components, enhancing both fine-tuning and inference efficiency. Our research highlights the importance of balancing learning capabilities for diverse datasets against the need for a lean model, offering a viable pathway for improving LLMs with minimal parameter growth. More discussion about limitation and broader impacts are available in Appendix D and E.

7 Acknowledgments

References

Alignment-Lab-AI. Lawyer-instruct, 2024.

Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H. Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. CoRR , abs/2308.06522, 2023.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Katz, and Anders Søgaard. LeXFiles and LegalLAMA: Facilitating English multinational legal language model development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 15513-15535, Toronto, Canada, July 2023. Association for Computational Linguistics.

Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca , 2023.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul

Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR , abs/2107.03374, 2021.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR , abs/2110.14168, 2021.

Mike Conover, Matt Hayes, Ankit Mathur, Xiangrui Meng, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, et al. Free dolly: Introducing the world's first truly open instruction-tuned llm, 2023.

Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. Rlprompt: Optimizing discrete text prompts with reinforcement learning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 3369-3391. Association for Computational Linguistics, 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 4171-4186. Association for Computational Linguistics, 2019.

Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. CoRR , abs/2312.09979, 2023.

Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021 , pages 4884-4896. Association for Computational Linguistics, 2021.

Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024.

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015 , pages 1026-1034. IEEE Computer Society, 2015.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. CoRR , abs/2009.03300, 2020.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 2790-2799. PMLR, 2019.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. CoRR , abs/2307.13269, 2023.

Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Comput. , 3(1):79-87, 1991.

Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Comput. , 6(2):181-214, 1994.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR , abs/2001.08361, 2020.

David J Ketchen and Christopher L Shook. The application of cluster analysis in strategic management research: an analysis and critique. Strategic management journal , 17(6):441-458, 1996.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 3045-3059. Association for Computational Linguistics, 2021.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021 , pages 4582-4597. Association for Computational Linguistics, 2021.

Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus , 15(6), 2023.

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022 , 2022.

Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications. CoRR , abs/2310.18339, 2023.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. CoRR , abs/2103.10385, 2021.

Stuart P. Lloyd. Least squares quantization in PCM. IEEE Trans. Inf. Theory , 28(2):129-136, 1982.

Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 1022-1035, 2021.

OpenAI. ChatGPT, 2022.

OpenAI. GPT-4 Technical Report. CoRR , abs/2303.08774, 2023.

David A. Patterson, Joseph Gonzalez, Quoc V. Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. CoRR , abs/2104.10350, 2021.

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021 , pages 487-503. Association for Computational Linguistics, 2021.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21:140:1-140:67, 2020.

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 506-516, 2017.

Gerard Salton and Chris Buckley. Term-weighting approaches in automatic text retrieval. Inf. Process. Manag. , 24(5):513-523, 1988.

Erich Schubert, Jörg Sander, Martin Ester, Hans-Peter Kriegel, and Xiaowei Xu. DBSCAN revisited, revisited: Why and how you should (still) use DBSCAN. ACM Trans. Database Syst. , 42(3):19:1-19:21, 2017.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017.

Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. S-lora: Serving thousands of concurrent lora adapters. CoRR , abs/2311.03285, 2023.

Asa Cooper Stickland and Iain Murray. BERT and pals: Projected attention layers for efficient adaptation in multi-task learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 5986-5995. PMLR, 2019.

Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VL-ADAPTER: parameter-efficient transfer learning for vision-and-language tasks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 5217-5227. IEEE, 2022.

Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 24193-24205, 2021.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023 , pages 13003-13051. Association for Computational Linguistics, 2023.

Chunlin Tian, Xinpeng Qin, and Li Li. Greenllm: Towards efficient large language model via energy-aware pruning. In 32nd IEEE/ACM International Symposium on Quality of Service, IWQoS 2024, Guangzhou, China, June 19-21, 2024 , pages 1-2. IEEE, 2024.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR , abs/2302.13971, 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR , abs/2307.09288, 2023.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang. Multilora: Democratizing lora for better multi-task learning. CoRR , abs/2311.11501, 2023.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. CoRR , abs/2309.05444, 2023.

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient finetuning for transformer-based masked language-models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 , pages 1-9. Association for Computational Linguistics, 2022.

Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. CoRR , abs/2308.03303, 2023.

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

Han Zhou, Xingchen Wan, Ivan Vulic, and Anna Korhonen. Autopeft: Automatic configuration search for parameter-efficient fine-tuning. CoRR , abs/2301.12132, 2023.

A Datasets and Baselines

A.1 Datasets

Single Domain

General : we fine-tune with the general instruction tuning databricks-dolly-15k for generic language capability and evaluate with MMLU.

Medical : we fine-tune with GenMedGPT and clinic-10k from ChatDoctor for medicine applications and evaluate medical tasks in MMLU including three related tasks: 'clinical knowledge', 'professional medicine' and 'college medicine'.

Law : we fine-tune with two legal instruction tuning datasets Lawyer-Instruct and US-Terms then evaluate with law tasks in MMLU including two related tasks: 'professional law' and 'international law'.

Math : we fine-tune with the training split of GSM8K for mathematical reasoning and evaluate with test set of GSM8K.

Code : we fine-tune with CodeAlpaca for code generation and evaluate with HumanEval.

Multi-task Domain As well for complex mixed multi-task/domain, we select a portion of the Flanv2 datasets covering Natural Language Understanding (NLU) and Natural Language Generation (NLG), which can be grouped into 10 distinct task clusters. Then we evaluate it with the Big-Bench Hard (BBH) benchmark.

We summarize the details of the used datasets as follows:

Struct-to-Text Conversion : This task evaluates the capability to generate natural language descriptions from structured data inputs. We use the following datasets: (1) CommonGen; (2) DART; (3) E2ENLG; (4) WebNLG;

Translation : Translation involves converting text from one language to another, maintaining the original meaning and nuances. We use the following datasets: (1) En-Fr from WMT'14; EnDe, En-Tr, En-Ru, En-Fi, En-Ro from WMT'16; (3) En-Es from Paracrawl.

Commonsense Reasoning : This involves assessing the ability to apply physical or scientific principles alongside common sense in reasoning tasks. We use the following datasets: (1) COPA, (2) HellaSwag, (3) PiQA, and (4) StoryCloze.

Sentiment Analysis : A fundamental task in natural language processing (NLP) that determines the sentiment polarity (positive or negative) of a given text. We use the following datasets: (1) IMDB, (2) Sentiment140, (3) SST-2, and (4) Yelp. information sources. We use the following datasets: (1) ARC, (2) NQ, and (3) TriviaQA.

Paraphrase Detection : This task requires models to ascertain whether two sentences convey the same meaning, indicating semantic equivalence. We use the following datasets: (1) MRPC, (2) QQP, and (3) Paws Wiki.

Coreference Resolution : Involves identifying instances within a text that refer to the same entity, demonstrating an understanding of textual context. We use the following datasets: (1) DPR and (2) WSC273.

Reading comprehension : Assesses the capability to derive answers to questions from a provided text containing relevant information. We use the following datasets: (1) BoolQ, (2) DROP, (3) MultiRC, (4) OBQA, (5) SQuADv1, (6) SQuADv2.

Reading Comprehension with Commonsense : Merges traditional reading comprehension skills with commonsense reasoning, requiring understanding beyond the explicit text. We use the following datasets: (1) CosmosQA; (2) ReCoRD.

Natural Language Inference : Focuses on deducing the relationship between two sentences, determining if the second sentence logically follows from, contradicts, or is unrelated to the first sentence. We use the following datasets: (1) ANLI, (2) CB; (3) MNLI; (4) QNLI; (5) SNLI; (6) WNLI; (7) RTE.

Closed-Book Question Answering : This task challenges models to answer questions about general knowledge without direct access to external

A.2 Baselines

PEFT methods

Full Fine-tuning is the default strategy for adaptation. During fine-tuning, the model is initialized with pretrained weights and biases, and all model parameters undergo gradient updates.

Prompt Tuning adds task-specific prompts to the input, and these prompt parameters are updated independently of the pretrained model parameters which are frozen.

P-Tuning adds trainable prompt embeddings to the input that is optimized by a prompt encoder to find a better prompt, eliminating the need to manually design prompts. The prompt tokens can be added anywhere in the input sequence, and P-Tuning also introduces anchor tokens for improving performance.

Prefix Tuning prefixes a series of task-specific vectors to the input sequence that can be learned while keeping the pretrained model frozen. The prefix parameters are inserted in all of the model layers.

IA3 enhances efficiency by infusing learned vectors into transformer architectures, drastically cutting trainable parameters while preserving performance and minimizing inference latency.

AdaLoRA is a method for optimizing the number of trainable parameters to assign to weight matrices and layers, unlike LoRA, which distributes parameters evenly across all modules. More parameters are budgeted for important weight matrices and layers while less important ones receive fewer parameters.

Multiple LoRA weighted average methods

LoRA MoE . A collection of n parameterized experts, denoted as E 1 , ..., E n , is orchestrated by a router network R . This network features a dense layer with adjustable weights W g from R d m × n . A softmax function then processes an intermediate token representation x , yielding gating scores s 1 , ..., s n that determine the weighted contribution of each expert's output:

Subsequently, the overall output y is synthesized by aggregating the experts' outputs, each modulated by its respective gating score:

This results in a dynamic allocation of the model's capacity, enabling specialized processing by experts as directed by the router's gating mechanism.

LoraHub aggregates 20 LoRAs at random for new downstream tasks. To master the weight of each LoRA, it utilizes a black-box optimization technique, bypassing the need for gradient calculations of the large model. This process involves weighted averaging at the parameter level. Mirroring the MoE training approach, we select 20 random samples for each task, creating a cohesive training dataset optimized through this black-box method.

B Initialization via k-means

In the case of considering heterogeneous corpora, it is crucial to select the appropriate number N of matrix B, to ensure consistent performance and minimize unnecessary computational overhead. This choice is usually closely related to the training corpus. In this work, we propose initializing HydraLoRA modules via k -means [30] algorithm for adaptive initialization. Specifically, k -means is utilized to process the heterogeneous corpus to identify the best-fit taxonomy of the corpus, i.e., the optimal N . First, we extract key features from the corpus by applying the Term Frequency-Inverse Document Frequency (TF-IDF; [38]) algorithm and transform the textual information into numerical

feature vectors. We integrate the elbow method [23] to determine the optimal value of N . Initially, N cluster centers are randomly selected for preliminary clustering as Eq. 7, followed by updating the cluster centers to accurately reflect the data within each cluster as Eq. 8. where C j is the cluster center to which data point X i is assigned and d ( · , · ) is the Euclidean distance function. S j is the set of data points in the j -th cluster.

By analyzing the relationship between the sum of squares of errors (SSE) and different N values, we observe that SSE decreases as N increases. Identifying the elbow point on the SSE curve - where the rate of decrease in SSE slows down - is crucial. The elbow point represents the optimal N value, beyond which increasing the number of clusters does not significantly enhance performance, thereby achieving an ideal balance between model complexity and performance.

C LoRA Breakdown

40

20

0

20

40

-

20

0

20

40

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118 119

120

121

122

123

124

125

126

127

t-SNE

Visualization

of

BA-Matrix

Diff.

LoRAs

LoRA 1

LoRA 2

LoRA 3

GSM

8K

40

20

0

20

40

-

20

0

20

40

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

32

34

36

38

40

42

44

46

48

50

52

54

56

58

60

62

64

66

68

70

72

74

76

78

80

82

84

86

88

90

92

94

96

98

100

102

104

106

108

110

112

114

116

118

120

122

124

126

t-SNE

Visualization

of

A-Matrix

Diff.

LoRAs

LoRA 1

LoRA

2

LoRA 3

GSM

8K

20

10

0

10

20

30

-

10

0

10

20

30

1

3

5

7

9

11

13

15

17

19

21

23

25

27

29

31

33

35

37

39

41

43

45 47

49

51

53

55

57

59

61

63

65

67

69

71

73

75

77

79

81

83

85

87

89

91

93

95

97

99

101

103

105

107

109

111

113

115

117

119

121

123

125

t-SNE

Visualization

of

B-Matrix

Diff.

LoRAs

LoRA 1

LoRA

2

LoRA 3

GSM

8K

-

-

-

-

-

-

-

40

-

20

0

20

40

-

20

0

20

40

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118 119

120

121

122

123

124

125

126

127

t-SNE

Visualization

of

BA-Matrix

Diff.

LoRAs

LoRA 1

LoRA 2

LoRA 3

GSM

8K

-

40

-

20

0

20

40

-

20

0

20

40

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

32

34

36

38

40

42

44

46

48

50

52

54

56

58

60

62

64

66

68

70

72

74

76

78

80

82

84

86

88

90

92

94

96

98

100

102

104

106

108

110

112

114

116

118

120

122

124

126

t-SNE

Visualization

of

A-Matrix

Diff.

LoRAs

LoRA 1

LoRA

2

LoRA 3

GSM

8K

-

20

-

10

0

10

20

30

-

10

0

10

20

30

1

3

5

7

9

11

13

15

17

19

21

23

25

27

29

31

33

35

37

39

41

43

45 47

49

51

53

55

57

59

61

63

65

67

69

71

73

75

77

79

81

83

85

87

89

91

93

95

97

99

101

103

105

107

109

111

113

115

117

119

121

123

125

t-SNE

Visualization

of

B-Matrix

Diff.

LoRAs

LoRA 1

LoRA

2

LoRA 3

GSM

8K

(b) Specific tasks.

D Limitation

In this study, we concentrate on well-established PEFT techniques, particularly LoRA. However, we have not yet investigated other configurations, such as prompt-tuning and adapters, which are

100

50

0

50

100

-

100

-

50

0

50

100

110

38

4

4

118

91

91

91

91

87

87

8

7

87

33

33

33

33

43

43

43

43

t-SNE

Visualization

of

Matrix

Diff.

LoRAs

S

UM

QA

ALL

IE

-

-

reserved for future exploration. Our current evaluation is limited to fine-tuning, while an examination of the efficacy of these techniques during the pre-training phase remains a subject for subsequent research. Another potential limitation stems from the training data itself. In multi-task scenarios, extreme conditions-such as contaminated or adversarial data-can significantly impair performance due to aggregation issues. The heterogeneity across tasks, including variations in language, task type, and domain, may reduce the effectiveness of shared knowledge, thereby diminishing its overall impact. This challenge is not exclusive to HydraLoRA but is common to all multi-task frameworks. Addressing these concerns may involve implementing robustness-enhancing measures (e.g., data sanitization, robust aggregation, and anomaly detection) and integrating privacy-preserving technologies, such as homomorphic encryption, differential privacy, and blockchain.

E Broader Impacts

Positive Societal Impacts The proposed HydraLoRA framework, with its asymmetric structure and parameter-efficient fine-tuning approach, has the potential to make LLMs more accessible and efficient. This could democratize AI, enabling more researchers, developers, and organizations to leverage the power of LLMs for various applications, ultimately driving innovation and progress. Moreover, by effectively addressing the challenge of domain or task interference, HydraLoRA could significantly enhance the performance of LLMs in complex, multi-task domains. This could lead to more accurate and reliable AI-powered tools and services in areas like healthcare, education, and finance, ultimately improving the quality of life for many people. Lastly, the parameter-efficient approach of HydraLoRA could help reduce the computational resources required for training and fine-tuning LLMs, thereby lessening the environmental impact of AI.

Negative Societal Impacts As with any AI technology, there are potential negative societal impacts to consider. The risk of misuse is a significant concern, as HydraLoRA could be used for malicious purposes, such as creating more sophisticated and convincing deepfakes or spreading misinformation and propaganda. Additionally, the increased efficiency and accessibility of AI brought about by HydraLoRA could lead to job displacement in certain sectors, as AI-powered tools and services become capable of performing tasks traditionally done by humans. Lastly, the use of LLMs in various applications could potentially lead to privacy and security issues, especially if these models are used to process or generate sensitive information. The proposed HydraLoRA framework, while not directly related to these issues, could inadvertently contribute to them by making it easier to deploy LLMs in various applications.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction sections, we articulate the motivation behind HydraLoRA , highlight its differences from previous work, and outline the contributions of this paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations in the Appendix D.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a detailed explanation of the parameter usage and will release the source code to ensure reproducibility.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a detailed explanation of parameter usage and will release the source code.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The full details are provided in the appendix and supplemental material (code).

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Experimental results are tested multiple times to ensure stability and reliability.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute resources are thoroughly described and evaluated in both the experimental setup and the experimental results sections.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper fully conforms to the NeurIPS Code of Ethics in every respect.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential positive societal impacts and negative societal impacts of the work performed in Appendix E.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper cites the original paper that produced the code package or dataset.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: This paper relates the details of the code as part of the submission.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two improvements to LoRA geared towards heterogeneous corpora on which LoRA underperforms full fine-tuning. First, it proposes training a number of smaller Lora heads (Ai,Bi) (Lora-Split) rather than a single head which improves performance while preserving the overall number of parameters. Second, the paper proposes an improvement over Lora-Split - called HydraLora - which reduces the number of parameters by sharing 'A' matrices across domains while allowing Bi's to vary across domains. This variant uses Mixture-of-Experts strategy for training/inference and improves performance over Lora with fewer parameters.\n\nSTRENGTHS:\n* Proposes a new method called HydraLoRA that improves performance over LoRA on heterogeneous corpora with fewer parameters (originality)\n* HydraLoRA does not require domain expertise either at training or inference (originality).\n* HydraLoRA improves training speed by around 2X relative to LoRA (significance)\n* Reports ablations showing what components matter in the final model (Quality)\n* Proposed method is likely to be employed by a number of researchers who are working on datasets which exhibit heterogeneity (Significance)\n* The paper presents an observation based on a tsne analysis whereby 'A' matrices from Lora heads are similar across domains while 'B' matrices vary. This is a really useful form of visualization that could be used by researchers working with LoRA (Significance)\n\nWEAKNESSES:\n* The proposed method still underperforms full fine-tuning.\n* It looks like inference using HydraLoRA routes each example to all experts i.e. B matrices and then computes a weighted average. The paper does not provide an ablation where only one of the B matrices (argmax of the gating score) is used at inference time, which may further reduce inference cost.\n* There are some details in the paper which are not clear. See questions below.\n\nThanks to the authors for addressing many of these issues in the rebuttal.\n\nQUESTIONS:\n* L54: 'autonomous' -> 'automatic'\n* Fig 2: How is corpus heterogeneity measured?\n* Table 1: How is Lora split trained - Is each Lora head trained on examples from a specific domain? If so, what are the domains? Are these domains naturally occurring in the corpus? or were they inferred by 'k-means clustering'?\n* Table 3: What is the performance of Lora-split?\n* L193: 'With equivalent parameters (rank=16), …'  - this is unclear since Table 2 reports performance of HydraLora with rank=8.\n* Figure 7: what is the x-axis?\n* L259: How does the variant without MoE work?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents HydraLoRA, an innovative and asymmetric Low-Rank Adaptation (LoRA) framework designed to enhance the efficiency of fine-tuning Large Language Models (LLMs) for specific tasks. The authors identify inefficiencies in the original LoRA approach, particularly its underperformance in complex domains, and propose HydraLoRA to address these issues.\n\nSTRENGTHS:\nImproved Efficiency: The framework requires no domain expertise and outperforms other Parameter-Efficient Fine-Tuning (PEFT) methods, including those that use domain knowledge during training and inference.\n\nGeneralizability: The framework shows robust generalization across unseen tasks without relying on prior task-specific knowledge, making it a versatile solution for adapting LLMs to various domains.\n\nResource Optimization: HydraLoRA is designed to be parameter-efficient, which not only improves performance but also reduces the computational resources required for training and deployment of LLMs.\n\nWEAKNESSES:\nHydraLoRA is more computationally intensive than conventional Parameter-Efficient Fine-Tuning (PEFT) methods due to the use of multiple adapter copies.\n\nHydraLoRA It requires more training iterations, which can be 1 to 2 times higher than typical PEFT methods, affecting the environmental footprint of model training.\n\nThe study primarily examines LoRA and does not test additional configurations like prompt-tuning and adapter layers, limiting the scope of the findings.\n\nThe method's practical effectiveness in real-world applications outside the experimental setup is not discussed.\n\nQUESTIONS:\n1. How does the asymmetric structure of HydraLoRA impact the interpretability of the model, and can the authors provide insights into how different components of the model contribute to the final predictions?\n\n2. The paper uses k-means for initialization. How sensitive are the results to the choice of initialization method, and how does this impact the overall performance?\n\nLIMITATIONS:\n1. The paper uses k-means for initialization, but it is not clear how sensitive the model's performance is to the choice of initialization method\n\n2. The use of multiple adapter copies in HydraLoRA leads to higher training costs compared to conventional PEFT methods.\n\n3. The asymmetric structure of HydraLoRA may introduce complexity in terms of model interpretability.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces HydraLoRA, a PEFT (Parameter-Efficient Fine-Tuning) architecture designed to improve the efficiency and performance of fine-tuning large language models (LLMs). HydraLoRA's main contribution lies in its asymmetric structure, which employs a shared matrix (A) for commonalities across tasks and multiple distinct matrices (B) for task-specific adaptations. The paper claims that this approach mitigates task interference and enhances parameter efficiency without requiring domain expertise.\n\nSTRENGTHS:\n- The idea of an asymmetric LoRA architecture that splits the parameter matrices into shared and task-specific components is a somewhat novel approach aimed at addressing the inefficiencies in traditional symmetric PEFT methods.\n- The paper includes a variety of experiments across different domains, including general language tasks, medical, legal, mathematical reasoning, and code generation. This wide scope provides a robust evaluation of HydraLoRA's potential benefits.\n- HydraLoRA is compared with several existing PEFT methods such as Prompt Tuning, P-Tuning, Prefix Tuning, and AdaLoRA, providing a comprehensive view of its performance relative to state-of-the-art techniques.\n\nWEAKNESSES:\n- Many sections of the paper are vague and lack sufficient detail. For example, the exact observations of how the shared matrix (A) and distinct matrices (B) interact and are optimized is not clearly explained. This makes it difficult to fully understand the proposed method. For example, lines 97 to 105 explain Figure 3, but it’s confusing to read the center and right subfigures. The center subfigure shows A matrix has fewer clusters and the heads are more distinct, but the text says the opposite (B is more distinct) right subfigure shows B is more clustered and not easily distinguishable. The workflow section 3.2 is scattered and difficult to follow. Key components of HydraLoRA, such as the structure of the matrices and the routing mechanism, are not described cohesively. The figures provided do not effectively clarify these components.\n\n- The idea of using MoE and LoRA adapters to implement multiple B matrices is very similar to Mixture of LoRA Experts (https://openreview.net/forum?id=uWvKBCYh4S, ICLR 2024), but not discussed and compared. The difference is probably the rank size selection.\n\n- The empirical results are incremental, table 2 shows most results compared to LoRA are within 1% improvements, e.g. Compared with LoRA-Split or r=32, HydraLoRA does use half trainable parameters, but unclear how much inference efficiency gains it achieves.\n\nQUESTIONS:\n- what are the inference speed gains compared to other PEFT methods?\n- what is the actual training overhead compared to other PEFT methods?\n\nLIMITATIONS:\nThe authors discuss that HydraLoRA is computationally demanding, primarily due to the necessity of fine-tuning large-scale language models. It incurs a higher training expenditure than conventional PEFT methods, attributed to the employment of multiple adapter copies.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge of efficiently adapting large language models to new tasks. The authors highlight the limitations of current techniques like LoRA, which, while parameter-efficient, struggle with diverse data.\n\nThrough a series of experiments, they discover that using multiple, task-specific LoRA modules improves performance but introduces redundancy. They further observe that within these multiple modules, certain parameters consistently learn common knowledge while others specialize in individual tasks.\n\nBased on these findings, they introduce HydraLoRA which utilizes an asymmetric LoRA structure. A single, shared matrix captures the common knowledge identified in their analysis, while multiple smaller matrices, one per task, handle specialized adaptations. This design maximizes learning from diverse data while minimizing redundancy.\n\nRather than depending on pre-defined task information, HydraLoRA employs a Mixture-of-Experts approach to dynamically route data during training and combine expert outputs during inference.\n\nExperimental results across multiple benchmarks demonstrate HydraLoRA consistently outperforming other efficient fine-tuning methods, including those using MoE. The authors further emphasize HydraLoRA's practical advantages by analyzing its energy consumption and latency.\n\nSTRENGTHS:\nMotivation and Design:\n* The paper excels at connecting its experimental findings to the proposed architecture. Specifically:\n    * The authors use t-SNE visualizations to analyze the parameter distributions of LoRA modules trained on different data subsets. This approach reveals a clear pattern: the \"A\" matrices of these modules tend to converge, indicating common knowledge acquisition, while the \"B\" matrices remain distinct, suggesting they specialize in task-specific features. This key finding highlights the inherent asymmetric nature of knowledge representation within LoRA and provides the foundation for HydraLoRA's design.\n    * Building upon this insight, the authors demonstrate that splitting a single LoRA into multiple, smaller ones, each trained on a different data subset (LoRA-Split), leads to significant performance improvements. This is evident in tasks like MMLU, Medical, and Law, where LoRA-Split consistently outperforms a single, large LoRA with the same parameter budget. These results suggest that intrinsic dataset differences can hinder the performance of a monolithic LoRA, and splitting helps mitigate this by allowing for specialized adaptation to those inherent data variations.\n\nEvaluation:\n  * Comparisons against a wide spectrum of PEFT methods, from traditional techniques like Prompt Tuning and P-tuning to more recent ones like AdaLoRA and (IA)3, provide a comprehensive picture of HydraLoRA's effectiveness.\n  * Significant Improvement over LoRA MoE: The direct comparison with LoRA MoE is a key strength in my opinion. While both methods utilize MoE, HydraLoRA consistently demonstrates superior performance. This highlights the effectiveness of HydraLoRA's shared \"A\" matrix in capturing common knowledge and its advantage over using entirely separate LoRA modules. These gains are evident in both accuracy improvements and reduced parameter count, as shown in the BBH benchmark results.\n  * Thorough Ablations: Authors present extensive ablation studies to capture the impact of various For example, comparing HydraLoRA to a variant with uniform expert weights (\"w/o Gate\") demonstrates the crucial role of the gating mechanism in selectively applying expert knowledge. This level of detail, presented across multiple benchmarks, strengthens the paper's conclusions and provides a deeper understanding of HydraLoRA's inner workings.\n\nWEAKNESSES:\n* While the shared \"A\" matrix in HydraLoRA appears effective for the tested benchmarks, the paper could benefit from exploring potential limitations of this design choice. Investigating performance on datasets with very different domains or tasks, where the notion of shared knowledge might be less applicable, would strengthen the claims about its generalizability.\n\n* The paper would be more convincing with a comparison against a LoRA-Split baseline that uses existing domain knowledge. For example, on a multi-task dataset, directly comparing HydraLoRA against splitting LoRAs by task labels would provide valuable insights into the trade-offs between automatic routing and a more informed, but potentially manual, approach.\n\n* The paper covers a wide variety of necessary aspects, but the presentation could be more streamlined and easy to read. For example, placing the comparison with MoE-based methods and the discussion about the shared \"A\" matrix's advantages earlier in the paper would have made this paper more appealing to readers. This would also emphasize HydraLoRA's unique strengths more effectively.\n\n* A deeper analysis of the MoE router's behavior would have been really interesting. Exploring aspects like its complexity, influence on overall latency, and potential routing biases could provide a more complete picture of its role within HydraLoRA.\n\n*  It's surprising that the authors mention the increased training iterations required by HydraLoRA (1-2 times more than typical PEFT) only within the limitations section. It would have been interesting to explore this nuance further or at least call it out in one of the main sections.\n\nQUESTIONS:\n* The shared \"A\" matrix effectively captures common knowledge in your experiments. However, how would HydraLoRA perform on datasets with more disparate domains or tasks where this notion of shared knowledge might be weaker or less well-defined?\n\n* Did you experiment with other routing techniques, such as top-k routing, during your exploration of HydraLoRA's design? If so, could you elaborate on the performance implications of these different routing strategies and what led you to choose your current approach?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}