HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models
Authors: Qiushi Huang
Tom Ko
Zhan Zhuang
Lilian Tang
Yu Zhang
Venue: ICLR 2025 Oral (2025)
Keywords: Parametric-efficient fine-tuning
Large Language Model
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: We propose Hadamard High-Rank Adaptation (HiRA), a parameter-efficient fine-tuning (PEFT) method that enhances the adaptability of Large Language Models (LLMs). While Low-rank Adaptation (LoRA) is widely used to reduce resource demands, its low-rank updates may limit its expressiveness for new tasks. HiRA addresses this by using a Hadamard product to retain high-rank update parameters, improving the model capacity. Empirically, HiRA outperforms LoRA and its variants on several tasks, with extensive ablation studies validating its effectiveness. Our code is available at https://github.com/hqsiswiliam/hira.

Published as a conference paper at ICLR 2025
HIRA: PARAMETER-ERHICHE HADAMARD HIGH- RANK ADAPTATION FOR LARGE LANGUAGE MODELS
Qiushi Huang!.2 :2; Tom Ko¹, Zhan Zhuang 1,, Lilian Tang', Yu Zhang' 'Southern University of Science and Technology. University of Surrey DanteDance, 'City University of Hong Kong
{qiushi huang. cs, tomkocse, yu. zhang. ust f@gmail com, h. taangesurreyy... ac. uk, 122500 63 630mail. sustech. wwatteechheu..coom edu.cr
ABSTRACT
We propose Hadamard High-Rant Adaptation (HiRA), a parameter-efficient fine- tuning (PEFT) method that enhances the adaptability of Large Language Models (LLMs). While Low-rank Adaptation (LoRA) is widely used to reduce resource demands, its low-rank updates may limit its expressiveness for new tasks. HiRA addresses this by using a Hadamard product to retain high-rank update parameters, improving the model capacity. Empirically, HiRA outperforms LoRA and its vari- ants on several tasks, with extensive ablation studies validating its effectiveness. Our code is available at atltttss:/github com/ hasiswsws.....................hm wwwwwwwwwwihhi. am/hira
INTRODUCTION
Recent advancements pre- trainedied Large Language Models (LLMs) Touvroon et al. 2023 Zhang et al. 2022 AAchiam al. 2023) have significantly enhanced performance across various natural language processing tasks. Traditionally, adapting those LLMSs to specific tasks required full fine- tuning, wherein all model parameters are updated. However, due to the massive number of param- eters in those LLMs, full fine-tuning becomes computationally prohibitive, especially in resource- constrained environments.
To address this challenge, parameter-efficien fine-tuning (PEFT) methods have been developed to adapt LLMs by updating only small subset of parameters. Building on this approach, several recent studies (Lester et al. [ Liu et al. [002 Hu et al. 2021 Liu et al. (2024) have introduced methods that maintain the integrity of the original architecture by freezing the majority of the model parameters and introducing updates to a limited set. Notably, LoRA (Hu et al.) (021) exemplifies PEFT by integrating a low-rank matrix decomposition into the update AW L2 where L E Rdxr and L2 E x are low-rank matrices with the rank at most This technique significantly reduces computational costs required compared to updating the full-rank parameter matrix W.
However, previous studies (Jiang al., 2024 Liu al. 2023
2024 Zhuang et al. 2024) have
shown that LoRA and most of its variants (Lialin et al. 2023) Hayou et al. 2024) do not perform well when applied to complex tasks, such commonsense reasoning that requires training on single dataset but evaluating across multiple sub-tasks. One potential reason for LoRA's limitations in these scenarios that its update matrix, AW, which is derived from the multiplication of low- rank matrices L1 and Lg, is confined to maximum rank of r. Consequently, although AW is a d k matrix, its rank cannot exceed r, which may limit the expressiveness of AW, particularly for more complex tasks. A natural solution to this issue is to raise the rank of the update parameter matrix to increase its capability. However, due to resource constraints. we still hope to follow the PEFT strategy. This gives rise to our research question: "Is it possible to achieve higher-rank adaptation for L.LMs under the PEFT strategy?"
To answer this question, in this paper, we propose a Hadamard high-Rank Adaptation (HiRA) for LLMs. The central innovation of HiRA is to express the update parameter matrix AW as the Hadamard product (a.k.a. elementwise product) of the original parameter matrix in the LLM and
Work done as an intern at ByteDance.
1 "oorrsponniig author.

---

Published as a conference paper at ICLR 2025
low-rank matrix achieve high-rank adaptation, thus increasing its rank and also expressiveness. Due to property of the Hadamard product that Rank(Op + 02) S (ank((O1) x Rank((2) for two matrices O and O2 with an equal size (Million 2007), where @ denotes the Hadamard product and (- gives the rank of a matrix, the Hadamard product could have a higher rank even though one of the two matrices in the Hadamard product is low-rank.
As an illustration, where the experimental setting is de- tailed in Appendix A.II Figure shows that the average rank of update parameters in HiRA is much higher than that in LoRA (i.e., 2837 vs. 32), which demonstrates that the proposed HiRA could possess high-rank update pa- rameter matrices while keeping the same number of train- able parameters as LoRA. The increased rank of AW in HiRA could enhance its expressiveness, as demonstrated in our experimental section.
2800
AW Rank
500
200
HiRA: RanklAWi
LoRA: RanklaW)
200
MORA: Rank(aW)
Through comprehensive experiments, we demonstrate that HiRA significantly outperforms LoRA and its vari- ants, showcasing the effectiveness of the proposed HiRA. Extensive ablation studies further elucidate the impact of different components in the HiRA, confirming its advan- tages and practical utility.
30
Figure 1: Rank comparison of update parameters AW among LoRA, MoRA, and the proposed HiRA.
In summary, the contributions of this paper are as follows.
We are the first to apply the Hadamard product to the parameter-efficient adaptation of LLMs.
Based on the Hadamard product, we propose HiRA, novel high-rank adaptation strategy for LLMs. The proposed HiRA method significantly increases the rank of update parameter matrices, enhancing the model's expressive power and adaptability.
We conduct extensive experiments and analyses to demonstrate the effectiveness of HiRA.
RELATED WORKS
Low-rank adaptation (LoRA). PEFT methods aim efficiently adapt LLMs to downstream tasks by training small subset of additional parameters while keeping most pretrained weights frozen. Current prevalent PEFT methods can be broadly categorized into three main types, with low-rank adaptation being major category. LoRA (Hu et al., 2021) factorizes the update weight into two low-rank matrices, allowing seamless integration into the original model without modi- fying its architecture increasing inference overhead. As extension, DoRA (Liu et al.) 2024) decomposes weight updates into magnitude and direction, updating the directional component in the manner LoRA. MoRA (Jiang al., 2024) compresses inputs via predefined functions, transforms them via a square "higher-rank" matrix, and then decompresses them to achieve a higher- rank adaptation. Different from LoRA which relies on the low-rank structure of the update weight, HiRA aims to learn the update weight with a high rank to enhance the expressive power. However, unlike MoRA, which relies on complex static compression and decompression functions that can complicate weight merging. HiRA merges seamlessly into the pretrained model, just as LoRA does.
Recently, FLoRA (Wen & Chaudhuri 2024) uses the Hadamard product to implement an example- specific adapter, enabling efficient batching, while PACE (Ni et al. (024) applies element- wise mul- tiplicative noise improve generalization cost-efficiently. In contrast, HiRA employs the Hadamard product to achieve higher-rank adaptation (as empirically demonstrated in Figures and 7 with more in in-depth analysis, which makes them significantly different.
Other PEFT methods. The second category comprises prompt-based methods, which integrate extra trainable virtual tokens into the input of LLMs and focus exclusively training those tokens. Representative methods include Prompt Tuning (Lester et al. 2021). which introduces a series of virtual tokens for task task-specific adaptations at the initial layer, and P- P-Tuning (Liu et al. (022). which adds virtual tokens at every layer instead of the initial layer. Although prompt-bases methods add a negligible number of trainable parameters into the input, they notably sensitive to initialization
2

---

Published as a conference paper at ICLR 2025
LoRA
Rank(AW) S
W
Wo
+
AW
decompose
L1
L2
HIRA
Rank(AW) s. not
Hadamard Product
decompose
W
=
W
+
AW
A
O
W
Figure 3: An illustration of the proposed HiRA method in comparison to the LoRA method. Wu et al. 2024). Morcover, due to the quadratic computational complexity of transformer archi- tectures Vaswann et al., 2017). prompt-based approaches could increase computational costs during the length of the prompt. The third category includes adapter-based methods. Those methods insert trainable modules, such as adapter layers, into the original frozen LLMs. Typical methods include Adapters (Houlsby et al. (209)) and Compacter (Karimi Mahabadì et al. 202T) that add linear layers to LLMs. Parallel adapters (He et al. (20IT) integrate adapters in parallel for performance enhancement. Those methods modify the architecture of original models during training and inference, potentially increasing overhead compared to the original LLMs.
inference proportionally
MOTIVATIONS
Effective fine-tuning of LLMs requires a careful balance between model expressiveness and computational effi- ciency. Hence, existing studies use LoRA with lower ranks (e.g., 32 or 64). Such setting achieves good per- formance on some tasks. For more complex tasks such as commonsense reasoning, we find that LoRA with higher ranks can significantly enhance performance of Llama- 3-8B as illustrated in Figure 2 Our results indicate that especially for complex tasks, updates model parame- ters with a higher rank could be helpful to achieve better performance. This observation complements existing re- search by highlighting scenarios where high-rank adapta- tions offer significant advantages.
Pemomancc
80
BaslQ
Halas
Whn0
OS0A
Average
32
128
256
512
1024
Figure 2: Performance of Llama-3-8B
commonsense reasoning using dif- ferent LoRA configurations.
However, increasing the rank in LoRA results height- ened computational demands, which conflicts with the objectives of PEFT, and it becomes more difficult to train, often leading to issues like gradient explosion, as detailed in Appendix A.3 Therefore, there is a need for a method that enables high- rank adaptation without incurring additional computational burdens. The proposed HiRA method addresses this need, as introduced in the next section.
4 METHODOLOGY
4.1 RANK ANALYSIS
A limitation of LoRA and its variants relying the product between two low-rank matrices is that the maximum achievable rank of the update parameter is inherently constrained. Wo E d denotes the original parameter matrix and its rank is denoted by ru, where ru min(d, k). The update parameter matrix AW in LoRA assumed to be the product of L, E ddxr and L2 e grxk where is much smaller than d and k. i.e., AW Ln Due the property of the rank, the

---

Published as a conference paper at ICLR 2025
maximum rank of AW is Thus, the low-rank property of AW may limit its capabiliiy to capture high-rank updates. As a result, such a low-rank update parameter may limit the rank of the final tuned parameter denoted by W' (i.e., W Wo + AW) since
Rank(Wo + AW) = Ran + Rank(AW) S To +r.
where the first equality holds due to the property of the rank function. Therefore, W has a maxi- mum rank of min(min(d, k),, ro + r). Consequently, the low-rank property of AW may limit the expressiveness of W". To address this, we propose the HiRA method to learn AW with higher rank under the PEFT strategy, which could enhance expressiveness and performance.
4.2 ENHANCING THE RANK VIA HADAMARD PRODUCT
The Hadamard product of two matrices P and Q with the same size gives matrix O satisfying Oij = Pijfij. where Pij. Pij. and Ozj denote the (i, entry in P, Q, and O, respectively. So the Hadamard product is also known the elementwise product between two matrices. nice property of the Hadamard product is that
Rank P @ Q) S Pank(P) x Rank(Q),
where denotes the Hadamard product. According to the inequality T we can see that the max- imal achievable rank of the Hadamard product of two matrices is upper-bounded by the product of their ranks. When P and Q have appropriate sizes to make matrix multiplication feasible, we have
Rank(PQ) < min(Rank(P)) Rankk) Rank(Q)).
Compared the inequalities and (2, we see that the upper- bound of the rank of the Hadamard product much larger than that of the matrix multiplication even when P or Q or both have a low rank. Note that the update parameter in LoRA relies the matrix multiplication of two low- rank matrices and inequality (2) implies that the update parameter in LoRA is low-rank. From this perspective of the upper-bound of the rank, the Hadamard product could help improve that.
The upper-bound of the rank may not precisely reflect the actual rank, but it gives the possible maximal rank of the Hadamard product. To the best of our knowledge, the lower-bound analysis of the rank for the Hadamard product is only for P and Q with special structures (e.g., positive semidefinite matrices (Hom Yang 2020)). Empirically we find that the Hadamard product could enhance the rank as demonstrated in Figures and 7
4.3 FORMULATION
Built on the Hadamard product, we give a general formulation for the update parameter matrix as
AW @ Whi:
(3)
where Wai the trainable parameter and R is fixed matrix. Based inequality I we obtain
Rank(AW) Rank(R) (R) x Rank(Wnaa.
(4)
When R has a high rank, inequality 4 suggests that AW can also achieve a high rank, potentially exceeding min k). To ensure parameter efficiency, we restrict Was to be low-rank, that is,
Wh. AB,
where Rd B E Rrxk and r is much smaller than min (d, k). This decomposition defined in Eq. (5) indicates that Whi has maximum rank of r, confirming its low-rank nature.
Based on Eqs. and 5 it is easy to see that when R is chosen to be a matrix of all ones, Eq. (3) gives the formulation of the update parameter matrix in LoRA, making LoRA a special case of HiRA. However, this setting of R is not so informative as it does not utilize any information in the pre-trained LLMs. Instead, in HiRA, we use the frozen parameter Wo in LLMs be R since Wo could contain useful information of LLMs. By combining all the above considerations together, we can obtain the update parameter for the proposed HiRA method
AW
Wo (AB).
4

---

Published as a conference paper at ICLR 2025
Based on the inequality 4 the rank of AW defined in Eq. (6) is upper-bounded by for, allowing for potentially high rank, as illustrated in Figure 3 Here, r is set equal to the dimension of LoRA, ensuring that the number of trainable parameters remains same and meets the requirements of PEFT.
During the training, Wo remains frozen, while and B serve as trainable parameters to facilitate the model updating. For linear layer h Wor, HiRA modifies the forward pass of this layer as
h W x = Wo + (Wo C (AB))z.
The calculation of W yields a computational complexity Odrk + dk) O(drk), which is equiv- alent to that in LoRA. To ensure that the initial value of update parameters will not modify the original LLMs, require the initial value of update parameters to be zero matrices. To achieve that, the initial values for Wh could be zero matrix. Under this requirement, A is initialized to be matrices, while B is initialized with the kaiming initialization (He et al. 2015).
4.4 EFFICIENT MODEL. ADAPTATION FOR INFERENCE
During production deployment, HiRA facilitates efficient inference by pre-computing and merging the update parameters into Wo to form W Wo + Wo (AB). This enables LLMs switch between tasks swiftly, as the original parameters can be recovered by through element- division by AB + Then, the LLM can be adapted to new tasks using HiRA. Notably, integrating the update parameters directly into Wo eliminates computational overhead during inference and avoids additional latency commonly associated with other PEFT techniques like Prompt Tuning and P- Tuning. Moreover, MoRA introduces complex mapping functions to compress the input into relatively high dimension and then decompress back, which cannot be easily merged into the original parameters in LLMs only if the function mappings in the compression and decompression can be represented by a transformation matrix and will incur additionally computational overhead.
4.5 RELATIONS WITH INTRINSIC DIMENSIONALITY
Previous studies (Li et al., 2018 Aghajanyan et al. 2021) show that LLMs have a low intrinsic dimensionality, meaning that only small subset of trainable parameters is necessary for effective fine-tuning. While LoRA (Hu et al. 202T suggests low rank for the update parameter matrix AW, our findings complement this by showing that the rank of AW can be enhanced under the low intrinsic dimensionality of Wh. based on the Hadamard product. It is crucial differentiate between the intrinsic dimensionality and the rank of AW; although low intrinsic dimensionality implies that only a few parameters need to fine-tuned, does not inherently mandate low rank. Our method highlights that increasing the rank of AW is beneficial for enhancing model flexibility and performance, even within a compact parameter space low intrinsic dimensionality.
4.6 THE EXPRESSIVE POWER OF HIRA
In this section, analyze the expressive power of HiRA, building on Zeng & Lee (024). which originally examined the expressive power of LoRA. In HiRA, given the pre-trained weight matrix Wo. the updated weight defined as Wo + Wo @ (AB). We denote by E the optimal update and measure the expressive power by the minimal difference between the updated weight and E. For LoRA with rank r, the minimum difference is equal to the (r + 1)-th largest singular value, denoted by 0r+1 (E). In Theorem we analyze the expressive power of HiRA.
Theorem 1. (The Expressive Power of HiRA) Consider the optimal parameter update E and the HiRA update with the form Wo O Whi where the runk of Whs is lexs than v. Then we have
min W...)a
|[Wo @ Whi
F S 0rt1 (E Wo) [Woll2
where 2 denotes the element-wise division.
According to Theorem [ the expressive power of HiRA depends on E 2 Wo rather than E alone in LoRA. A detailed proof and analysis are provided in Appendix B The key distinction between HiRA and LoRA lies in the role of Wo. whose high-rank properties tained information. In this context, W, in HiRA serves a dual role: it confines and facilitates the adaptation. While the information in Wo limits the flexibility of the update matrix, preventing it
inherently tied to the
5

---

Published as a conference paper at ICLR 2025
from achieving unconstrained high-rank matrix, the pre-trained knowledge embedded in Wo also aids the adaptation process, allowing for more efficient fine-tuning.
4.7 GRADIENT ANALYSIS
The distinction between
HiRA and LoRA also lies in how their gradients interact with the pre- trained weight matrix, Wn- Let y' Wor AWr and y represent the predicted and true labels. For simplicity, we consider a linear neural network with the mean squared error loss L.
In LoRA, the gradients are computed as Z BJ (y v')(==T) and &c az
(y -0((=x))
which are independent of Wo. In contrast, the gradients in HiRA are given by
dc 0A = B (Wo @ ((v - -')-))
dc
aB = (Wo ② ((v - (()-)(((x-))) A.
This reveals that HiRA could leverage the information encoded in Wo to guide the adaptation. Thus, HiRA can potentially enhance performance when the pre-trained model has already captured pat- terns relevant to downstream tasks.
5
EXPERIMENT
In this section, we conduct experiments three tasks to evaluate the proposed HiRA method.
5.1 DATASETS
Commonsense Reasoning. We utilize eight sub-tasks with predefined training and testing datasets
(Hu al. 2023 combining 170,420 query-answer pairs for fine-tuning LLMs and selecting 120
random entries as a validation set. The sub-tasks include BoolQ (Clark et al. 2019) (yes/no QA),
PIQA (Bisk et al., 2020) (physical commonsense), SIQA (Sap et (0019) (social reasoning). Hel- laSwag (Zellers et al. 2019) (commonsense NLI), WinoGrande (Sakaguchi et al., (021) (fill-in-
the-blank), ARC-c and ARC-e (Clark et al., 2018) (multiple-choice science questions), and OBQA Mihaylov et al. 2018) (multi-step reasoning). Table I1 presents the dataset statistics.
Open-domain Dialogue Generation. We use the ConvA12 dataset (Dinan et al. 2019). including 17,878 training and 1,000 testing multi-turn conversations. Each dialogue features persona profiles of 4-5 descriptive sentences and conversational history. Following (Liu al.) 2020; Song et al.) we adopt self-persona setting, revealing only the speaker's
2021 Huang et al. persona.
2023b
2024).
Mathematical Reasoning. For this task, employ MetaMath YYu al. 2023) as the training corpus and GSMSK (Cobbe et al. 202I) as the test dataset.
5.2 EXPERIMENTAL SETTINGS
Evaluation Metric. To assess the performance on the commonsense reasoning datasets, we employ accuracy the primary metric for each sub-task. For each test instance, the language models decode answers from the provided queries. We then search for the presence of specific answer keywords (e.g.. "true" "false" for BoolQ). The first occurrence of the keyword is recorded as the model's response. If no relevant keywords are identified, the model is considered to have failed to correctly answer the commonsense reasoning question. This method allows us to consistently evaluate the performance of model responses across all eight sub-tasks, and is adopted by (Hu et al.) [023] Liu et al. (024). For the CONVAI2 dataset, we use the BLEU (Papineni et al., (002) and BERT Score (Zhang et al. (019) as the evaluation metrics. For the mathematical reasoning task, we utilize the accuracy as the evaluation metric.
Baseline Methods. We compare the HiRA with prompt-based methods including Prompt Tuning (Lester et al. 202T) and P-tuning (Liu et al. 2022), and low-rank adaptation methods including
LoRA (Hu et al. 2)21). DoRA (Liu et al. 2024), and MoRA (Jiang al. 2024). Our experiments use the Llama-2. 1- 2-B (Touvron et al.) 2023) and -amm-3-8B (Dubey et al. 2024) open- LLMs.
https://gi thu.. com AGI wwt../////iii.neeeeerreeeeeeeeee http://hw...ookkee//o/eeee/////a

---

Published as a conference paper at ICLR 2025
Table I: Accuracy comparison among various PEFT methods on commonsense reasoning datasets. Results for ChatGPT, LoRA, and DoRA are sourced from (Liu et al. 2024). The best performance within each LLM is indicated in bold, while the second best performance is highlighted in underline.
Model
Method
Params (%)
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
CharGPT
73.10
85.40
68.50
79.90
89.80
74,80
78.50
66.10
77.01
Prompt Tuning
0.0012
55.93
12.35
30.50
6.06
8.63
9.40
6.91
40.57
21.29
P-Tuning
0.7428
58.75
36.02
0.20
0.17
1.98
0.80
0.01
0.00
12.24
LoRA 32)
0.8256
79.90
79.50
64.70
79.80
81.00
83.60
82.60
77.61
Llama-2-7B
DoRA (r 32)
0.8256
71,80
83.70
76.00
68.20
83.70
82,40
89.10
82.60
79.69
MoRA (r 32)
0.8241
80.79
79.53
71.42
85.31
72,46
HIRA (r 16)
0.4128
69.82
80,20
78.20
71.35
85.90
81.00
86.99
83.43
79.61
HIRA 32)
0.8256
83.35
79.53
73.81
86.74
84.60
Prompt Tuning
0.0010
$6.85
45.05
36.13
31.57
32.74
29.20
14.01
50.12
36,96
P. Tuning
0.6240
59.97
11.64
8.19
7.42
8.63
9.50
1.77
37.65
18.
LoRA 32)
0.7002
70.80
85.20
79.90
71.20
84.20
79.00
91.70
84.30
80.79
Elama-3-88
DoRA (r 32)
0.7002
74.60
89.30
79.90
80.40
90.50
85.80
95.50
85.60 $5.60
85,20
MoRA (r 32;
0.6997
74.28
87.43
80.71
79.61
91.16
85.60
43.53
86.74
78.63
HIRA (r = 16)
0.3513
73.85
89. 12
81.06
82.59
93.06
87.40
94,85
86.74
86.08
HiRA 32)
0.7002
75.40
89.70
81.15
82.90
93.27
87.70
86.72
Implementation Details. Following the identical training setup to (Liu ct al. 2024) except learning rate adjustments, we implement HiRA on the Llama-2-7B and Llama-3-8B models with r 16 and r = 32, respectively. The Adam optimizer (Loshchilov & Hutter (019) is employed with a learning rate 0,001, which warms up for 100 steps. For the commonsense reasoning dataset, we fine-tune LLMs for 3 epochs, with evaluations at every 80 step to select the best checkpoint based on the validation set. We place LoRA, DoRA, MoRA and HiRA on the query, key, value weights, and two linear layers (i.e., down and up projection) in attention modules. To ensure fair comparisons among LoRA, DoRA, MoRA, and HiRA, we maintain the same or comparable numbers of trainable parameters. For Prompt Tuning and P-Tuning, which inherently involve fewer trainable parameters due to their reliance on prefix soft prompts, we adjust accordingly to keep the number of trainable parameters comparable. The detail can be found in Appendix A..] HiRA evaluated over 5 runs with different random seeds. Experiments on the CONVAI2 dataset use training epoch, while the mathematical reasoning task uses 2 epochs, which is the only difference from the settings above.
5.3 RESULTS ON COMMONSENSE REASONING TASKS
As shown in Table [ HiRA consistently outperforms all baseline methods in terms of average accu- racy across both the Llama-2-7B and Llama-3-8B models. For r = 32, HiRA achieves an average accuracy of 81.42% for the Llama-2-7B model, surpassing the best baseline, DoRA, which records 79.69%. In the case of the Llama-3-8B model, HiRA shows a significant improvement in terms of the average accuracy (86.72% 85.20% over the pest-peforrmig baseline (i.c., DoRA). These results underscore HiRA's effectiveness in leveraging the Hadamard product to enhance the model capacity and performance within the PEFT strategy.
In contrast, for r = 16, HiRA achieves an average accuracy of 79.61% % for Llama-2-1 2-7B model, utilizing half the number of trainable parameters compared to LoRA and DoRA while achieving comparable or even performance. Similarly, for the Llama-3-8B model, HiRA with r = 16 achieves an impressive average accuracy of 86.08%. This demonstrates HiRA's capability deliver strong performance while maintaining lower intrinsic dimensionality.
5.4 RESULTS ON CONVERSATIONAL TASK
According to the results on the CONVAI2 dataset as shown in Table 2. HiRA consistently outper- forms all baseline methods across all comparison metrics. Specifically, HiRA with r = 32 achieves the highest average score of 47.80%, closely followed by HiRA with r = 16, which also performs exceptionally well with an average score of 47.79% despite using only half of the number of train- able parameters in LoRA and its variants. Both configurations which exhibit similar results with average scores of 46.62% and 46.59% respectively. MoRA, while not as strong as LoRA in this task, still outperforms Prompt Tuning. Those results further substan- tiate the superiority of HiRA in both parameter efficiency and performance across various tasks.
HiRA surpass DoRA and LoRA,
7

---

Published as a conference paper at ICLR 2025
Table 2: Results on the CONVAI2 dataset, where BERT FI, BERT-R, and BERT-P denote the FI, Precision, and Recall based the BERT score, respectively.
Model
Method
Params (%)
BLEU
BERT FI
BERT-R
BERT-P
Meteor
R-L
Average
Prompt Tuning
0.0012
0.04
72.44
77.38
68.23
0.80
0.80
36.62
P-Tuning
0.7428
0.60
83.29
83.33
83.28
15.11
12.36
46.33
MoRA 32)
0.8241
1.09
84.09
84.65
83.59
10.97
9.57
45.66
Llama-2-7B
LoRA (r 32)
0,8256
1.82
84.41
84.71
84.16
11.38
10.55
46.17
DoRA (r = 32)
0.8256
1.73
84.18
84.61
83.81
11.25
10.41
46.00
HiRA 16)
0.4128
2.56
83.97
84.12
83.86
13.35
12.58
46.74
HiRA 32)
0.8256
2.70
84.86
84.98
84.77
13.56
12.80
47.28
Prompt Tuning
0.0012
1.45
82,99
82.99
83,05
14.72
13.13
46.39
P-Tuning
0.7428
1.50
81.52
81.07
82.01
15,49
13.55
45.86
MoRA 32)
0.8241
1.60
84.22
84.06
84.43
12.37
11.19
46.31
Llama-3-8B
LoRA (r 32)
0,8256
2.26
84,32
84,00
84.67
12.51
11.77
46.59
DoRA (r = 32)
0,8256
2.29
84,32
84.06
84.62
12.63
11.78
46.62
HiRA == 16)
0.4128
3.32
84.84
84.41
85.30
14.93
13.94
47.79
HiRA -22)
0.8256
3.41
84.40
85.25
14.87
14.05
47.80
5.5 RESULTS ON MATHEMATICAL REASONING TASKS
We evaluate the performance of HiRA on mathematical reasoning tasks using the MetaMath dataset for training and the GSM8K benchmark for evaluation. As shown in Table 3 HiRA significantly outperforms baseline meth- ods, achieving an accuracy of 70.81%. %. This represents notable improvement over LoRA (65.89%). DoRA (66.12%) 12%), and MoRA (67.98 (77). Even with fewer train- able parameters (r 16), HiRA remains competitive, achieving 67.63%. On the Llama-2-7B model, HiRA delivers strong results, achieving 46.85% substantial increase compared to LoRA (15.16%) and its variants. These results demonstrate HiRA's superior ability adapt complex mathematical reasoning, tributed to its high-rank updates, which enhance the model's expressive power while maintaining parameter efficiency.
Table 3: Results on mathematical rea-
soning tasks.
Method
Trainable
Prompt Tuning
0.0012
15.62
P-Tuning
0.7428
LoRA (r 32)
0,7002
65.89
Llama-3-8B
DoRA (r 32)
0.7002
66.12
MoRA 32)
0.6997
HiRA 16)
0.4128
67.63
32)
ABLATION STUDIES
6.1 SINGULAR VALUE ANALYSIS OF FULL FINE-TUNING, LORA AND HIRA
To further analyze the advantages of HiRA, we calculate the singular values of update parameter matrix in full fine-tuning (FFT), LoRA, and HiRA. In Figure present the number of singular
W sows
FFT
300
2800
LoRA 1=28
1000
20c
1K0
MoRA
Maree Louer 300
Macee Liwer 32)
Nosel Liger 32
Motel Liyer 013
Hotel Liper 325
Figure 4: Count of singular values exceeding 0.005 across layers for FFT, LoRA, MoRA, and HiRA.
values exceeding 0.005 for each layer. FFT produces diverse singular value distributions across dif- ferent layers, reflecting the difference in the ease of updating layers for fine-tuning. Notably, HiRA exhibits similar trends to those FFT, indicating higher rank is crucial for effective optimization. In contrast, the inherently low-rank structure of LoRA may limit its expressive power.
presentss the sum the squared singular values per layer, which corresponds to the squared Frobenius norm. The results indicate that MoRA A and LoRA with rank=32 yields substantially larger singular values, as reported in (Lialin et al., 2024). Those large singular values suggest that its updates make strong adjustments in certain directions on the pre-trained model to swiftly adapt to new tasks. However, excessively large singular values (i.e.. the spectral norm) may increase the risk of overfitting, consequently impairing the generalization ability (Bartlett al. 2017). Moreover,
8

---

Published as a conference paper at ICLR 2025
FFT
LORA r-32
LoRAr=128
27
LoRAr=512
MoRA
HIRA
Lager 201
Layar 201
Nocnl Layarty-23 22)
Nodal Layariy 231
Model Layeriy
Figure 5: Sum of squared singular values across layers for FFT, LoRA, MoRA, and HiRA.
overly large singular values can induce gradient explosion or vanishing issues, thereby compromis- ing the stability and convergence rate of subsequent training (Pennington et al. 2017). In contrast, HiRA tends to maintain smaller singular values as FFT did.
6.2 IMPACT
MODEL PERFORMANCE
Figure 6 illustrates the impact of the parameter dimen- sion r to the model performance across multiple com- monsense reasoning tasks. As increases from 32, there is a consistent and notable improvement in perfor- mance, with the average accuracy rising from 83.16% to 86.72%. This trend underscores the importance of higher-rank structure in enhancing the model ability to generalize across diverse tasks. Notably, tasks such as PIQA and HellaSwag exhibit significant gains, with PIQA improving by 2.37% as increases, highlighting the critical role of tasks that require advanced rea- soning capabilities. Interestingly.
95
90
Perfomance
ARC-C
PIQA
ABC
WnoG
SIDA
Average
16
20
24
28
32
Figure 6: Performance of HiRA across tasks when r increases.
at 8, HiRA still outperforms LoRA with r = 32, demonstrating the superiority of HiRA to efficiently leverage a smaller number of trainable parameters achieve good performance, making HiRA a compelling choice even under constrained resources.
6.3 IMPACT OF DIFFERENT CHOICES OF
Table 4: Performance comparison between different choices of R defined Eq. ( HiRArand denotes a variant using randomly initialized R instead of Wo. HiRA denotes the vanilla version where I equals Wo. Both methods are tuned on the Llama-3-8B model.
Model
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HcllaS
WinoG
Average
HiRA
75.40
89.70
81.15
82.90
93.27
88.32
95.36
87.70
86.72
HiRApand
62.17
50.38
33.62
26.62
26.39
26.40
25.06
50.36
37.63
In this section, we explore the impact of different choices for R in Eq. 3 on performance in commonsense reasoning tasks. Specifically, compare with variant, HiRArand- where R is randomly generated from a uniform distribution [0. 1] before the training and remains fixed. As shown in Table 4 both methods learning rate, and training epochs, yet HiRA significantly outperforms HiRApand- This highlights the effectiveness of using Wn as R. Moreover, discusses in Section 4.4 using W, for R aids in recovering Wo from the merged parameters W' given A and B, whereas HiRAyand requires storing separately achieve that, which could lead additional storage costs.
identical training protocols by utilizing the same optimizer,
6.4 RANK ANALYSIS
In this section, we compare the average rank of the update parameter matrix AW over layers for HiRA, LoRA, and MoRA, which have comparable numbers of trainable parameters. As shown in Figure HiRA possess AW with much higher ranks than LoRA and MoRA, indicating that HiRA can achieve high-rank adaptation under the PEFT strategy via the Hadamard product. Notably, as the layer goes deeper, the rank of AW first increases and then fluctuates, which indicates that deeper layers may need a higher-rank AW to adapt to new tasks. Overall, HiRA attains higher-rank AW's across all layers, which correlates with improved performance detailed in Table

---

Published as a conference paper at ICLR 2025
Table 5: Performance of the Llama-3-4 3-BB model with HiRA integrated into various components.
Component
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
FC. QKV
75.40
89.70
81.15
82.90
93.27
88.32
95.36
87.70
86.72
FC
73.30
89.45
81.17
82.51
92.89
87.60
94.82
86,58
86.04
QV
73.09
88.85
81.06
80.38
92.68
86.20
94.37
85.87
85.31
QKV
72.26
89.23
80.30
80.89
92.97
84.80
94.37
86.19
85.13
QK
71.38
87.49
78.92
80.80
91.29
83.00
93.68
84.37
83.86
67.46
87.60
79.48
79.01
91.20
82.20
92.91
83.58
82.93
Q 68.59
86.51
77.64
79.61
90.99
81.20
92.68
81.93
82.39
K
68.20
86.07
77.74
80.03
90.82
80.00
92.29
81.37
82.07
6.5 ANALYSIS ON PLACEMENT OF HIRA IN TRANSFORMERS
analyze the effectiveness of HiRA when applied to any subset of weight matrices within the Transformer, including fully connected (FC) layers and query (Q). key (K), and value (V) in attention module, across multiple commonsense reasoning tasks. The results indicate that applying HiRA to the FC layers and the combination of QKV matrices yields the best performance across most tasks. In contrast, applying HiRA individual components leads to inferior performance. Hence, we choose to apply HiRA on both FC layers and the combined QKV matrices.
As shown in Table
200
RankgAwy
HIRA: Rank(sw) RanktaWi LoRA: RankloM)
2000
150
Layer
Figure 7: Average rank of AW for HiRA, MoRA, and LoRA tuned on the Llama-3-8B.
Table The results of the HiLoRA, where the Llama-3-88 model is used. The best performance is shown in bold, while the second best performance is highlighted in underline.
Model
BoolQ
PIQA
STQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
LoRA 32)
70.80
85.20
79.90
71.20
84.20
79.00
91.70
84.30
80.79
HiRA (r 32)
75.40
89.70
81.15
82.90
93.27
88.32
95.36
87.70
86.72
HiLoRA (ri 2,,2 30)
73.76
84.98
80.60
74.57
88.38
84.60
88.68
86.27
82.73
HiLoRA =4,r2 28)
74.22
90.26
82.80
82.25
93.48
89.80
96.13
88.48
87.18
HiLoRA -8,72 =44)
73.52
89.77
82.65
83.36
93.81
89.60
96.04
89.03
87.22
HiLoRA 16, 16)
74.77
89.88
81.73
83.28
92.93
88.20
95.52
88.08
86.80
HiLORA -20. 12)
75.84
90.42
82.14
84.22
93.73
87.80
96.45
89.27
87.48
HiLoRA (r 24.r2 8)
74.89
89.61
81.68
82.85
92.93
86.20
95.68
89,34
86.65
HiLoRA (r1 = 28, 4)
73.64
88.96
81.06
83.11
93.14
87.80
95.51
88.71
86.49
HiLoRA (11 30,72 2)
73.46
89.50
83.06
82.34
93.10
89.00
95.86
88.87
86.90
6.6 COMBINATION WITH LORA
In this section, we explore the combination of HiRA with LoRA to investigate the impact of integrat- ing both techniques into the performance. Specifically, such combination, which is called HiLoRA in this section for clarity, formulates the update parameter matrix AW Wo @ AB + L L2. where gdxr, E IR') xk L1 E [Rdxrs, and L2 E Rra becomes O, HiLoRA degenerates to HiRA, and if r1 equals O, HiLoRA becomes LoRA.
easy to see that when T2
To compare HiLoRA with HiRA and LoRA fairly, we ensure the same numbers trainable pa- rameters by setting in HiRA and LoRA to ri 72. Table presents the results of HiLoRA with various configurations on 71 and r2. The results indicate that increasing i' generally improves per- formance. HiLoRA with r1 = 20 and r2 12 achieves the highest average score, excelling in tasks like HellaSwag and ARC-c. This suggests that higher intrinsic dimension for HiRA is preferable over LoRA in HiLoRA, which demonstrates the usefulness of HiRA that could have larger capac- ity. Moreover, most configuration of 71 and perform comparably or better than HiRA and LoRA with 32, indicating that the combination of LoRA and HiRA is promising for fine-tuning LLMs.
CONCLUSION
In this paper, we introduced HiRA, novel high-rank adaptation that maintains comparable numbers of trainable parameters while enhancing the rank of update parameters. HiRA offers a cost-effective alternative to LoRA, providing similar benefits but without additional inference overhead. Extensive experiments demonstrate the effectiveness of the HiRA method. In our future work, we are interested in applying HiRA more applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by National Key R&D Program of China 20222D016033 and NSFC key grant under grant no. 62136005.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 23ix:2333308774 2023.
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tumi finettning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7319-7328, Online, August 2021. Association for Computational Linguis- tics. doi: 10. 00.18633/v1/022 acl-ooo...l-o URL https: http://aaaaattttttco/o org.2200010.0/111.... acl-long: 568
Peter L Bartlett, Dylan Foster, and Matus Telgarsky. Spectrally-normaliii margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Christopher Clark, Kenton Lee, Ming-Wos Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring surprising difficulty of natural yes/no questions. Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- Minneapolis, Min-
guage Technologies, Volume (Long and Short Papers), pp. nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19- 1300. URL
fetts://acclntte wow.ohoooy........o..o....oo org/No.g... 1300
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05453 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint urXiv:2110.1 14168. 2021.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Ur- banek, Douwe Kiela, Arthur D. Szlam, lulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan Black, Alexander I. Rudnicky, Jason Williams, Joelle Pineau, Mikhail S. Burtsev, and Jason We- ston. The second conversational intelligence challenge (convai2). arXiv:1902.00098. 2019.
Abhimanyu Dubey, Abhinav Jaubri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint 4rXiv:2407.2 2 1783, 2024.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, 1(3):211-218, 1936.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large
models. ArXiv, abs/2402. 12354, 2024. URL https:: wttpppp..stttttccsccoooooooooooo orogg//
CoxpusID:267750122
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kinkpatik,, and Graham Neubig. Towards a unified view of parameter-efffiient transfer learning. arXiv preprint arXiv:21 10.04366, 2021.
11

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-10344 2015.
Roger A Horn and Charles R Johnson. Topics in matrix analysis. Cambridge university press, 1994.
Roger A. Horn and Zai Yang. Rank of a Hadamard product. Linear Algebra and Its Applications, 591:87798, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International conference machine learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint S.Niv:2006006688 2021.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. LLM-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5254-5276, Singapore, December 2023. Association for Computational Linguistics. doi: (0.18653/v1/2023..mm htg.:223emmmmmmmae main. 319. URL htt.pps:/aclaattholtt
org 20223 emn in. 319.
Qiushi Huang, Shuai Fu, Xubo Liu, Wenwu Wang, Tom Ko, Yu Zhang, and Lilian Tang. Learning retrieval augmentation for personalized dialogue generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 2523-2540, Singapore, December 2023a. Association for Computational Linguisties. doi: 2023. em..pmmnppmmm. 154
www.:02...emlmmmmmmmm mainl 154. URL https::/aclaaanhoooggy orgg/
Qiushi Huang, Yu Zhang, Tom Ko, Xubo Liu, Bo Wu, Wenwu Wang, and H Tang. Personalized dialogue generation with persona-adaptive attention. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11):2916-12299
Jun. 2023b. doi: 10.1609/aaai 1.7.11.26518, URL
httpss:/ojs aaai. org/ index. wwipkk.pppppAA... CaappAAAA/aartiic httpp//A//aariil///e////////////
Qiushi Huang, Xubo Liu, Tom Ko, Bo Wu, Wenwu Wang, Yu Zhang, and Lilian Tang. Selec- tive prompting tuning for personalized conversations with LLMs. In Lun-Wei Ku, Andre Mar- tins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL. 2024, pp. 16212-16226, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 10. 10.11655//1/2224.immm http://///.....iiiaaaa..... 2024. findings ac1. 9597
findings--al..959,
URL https: hatass///caathhooog... org/
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, et al. Mora: High-rank updating for earfmeter-efficient fine- tuning. arXiv preprint arXiv:2405. /2..::2005.//300, 2024.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems, pp. 1022- 1035, 2021.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for efffiener-efficcent prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 1665://1/222.eemlpp-mo org. eme ma i n. 243
URL
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Mcasuring the intrinsic dimension of objective landscapes. arXiv preprint trXiv:1804 08838, 2018
Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira. and Anna Rumshisky. Relora: High-rank training through low-rank updates. In International Conference on Learning Representationss 2023. URL httts:t:: //api http:/pp...a.a.....mm...mmmmaa.. semant wawchlccllacoooccco ccc.choll....oogooooosssst//SSS.
12

---

Published as a conference paper at ICLR 2025
Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. ReloRA: High- rank training through low-rank updates. In The Twelfth International Conference on Learning Representations, 2024.
Mingjie Liu, Grodor-Dumitre Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktanoglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang. Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu. Sumit Jain, Brucek Khailany, Kishor Kunal, Xiaowei Li, Hao Liu, Stuart F. Oberman, Sujeet Omar, Sreedhar Pratty, Jonathan Raiman, Ambar Sarkar, Zhengjiang Shao, Hanfei Sun, Pratik Suthar, Varun Tej. Kaizhe Xu, and Haoxin Ren. Chipnemo: Domain-adapter Ilms for chip design. ArXiv, abs/2311.00 1.00176,, 2023. URL https://api. sww.nttistnttttcnccccc.aeee
Qian Liu, Yihong Chen, Bei Chen, Jian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang. You impress me: Dialogue generation via mutual persona perception. Proceedings of the 58th Annual Meeting of the Association for Computatioonal Linguistics, pp. 1417-1427, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/w1/2020..ce main.131. URL org/ acl.aaan.mmmmmmm333
Shih- Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adapta- tion. ArXiv, abs/2402.09355 2024. URL. https:
icscholar. org/
CorpusID:2676577
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.). Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, Dublin, Ireland, 122.633/1/2222.all-eel
May 2022. Association for Computational Linguistics. doi:
URL
oww.tthhooooooooo/o/ooo/ wotp:oooooooo//2....lla--a----a. short.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Leurning Representations, 2019. URL
net/ forum?rummmmmmm did
Bkg6RiCqY?
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsuji (eds.). Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 2381-2391, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v//D18--1336 URL
org/D1 1260
Elizabeth Million. The hadamard product. Course Notes, 3(6 2007.
Yao Ni, Shan Zhang, and Piotr Koniusz. PACE: marrying the generalization of -fffceter-cfficcent fine-tuning with consistency regularization. In The Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. BRLhttt:: httt::://operreiiew.rr
LoLhTPT
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Association for Computational Linguistics, pp. 311-318, USA, 2002. Association for Computational Linguussttiiinniiiiiiiiiii Linguistics. doi: 10.3 10.3115/10730883077333 URL https:
//doi.org/10) 1ttp:/oor/11..11/107730030 1073135
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep leamn ing through dynamical isometry: theory and practice. Advances neural information processing systems, 30, 2017.
Olivier Roy and Martin Vetterli.
The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pp. 606-610. IEFF, 2007.
13

---

Published as a conference paper at ICLR 2025
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9))99- -009 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Common- sense reasoning about social interactions. arXiv preprint UrXiv:1904.09728, 2019.
Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. Lora vs full fine- tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21228, 2024.
Haoyu Song. Yan Wang, Kaiyan Zhang, Weinan Zhang, and Ting Liu. Bob: Bert over bert for
training persona-base based dialogue models from limited personalized data. Association for Com- putational Linguistics, 2021.
Nathan Srebro and Tommi Jaakkola. Weighted low-rank approximations. In Proceedings of the 20th international conference on machine learning (ICML-03), pp. 720-727, 2003.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
Yeming Wen and Swarat Chaudhuri. Batched low-rank adaptation of foundation models. In The Twelfth International Conference on Learning Representations, 2024. URL https:
openreview net/ Lttee/orrnndddaaaa11100
Junda Wu, Tong Yu, Rui Wang, Zhao Song, Ruiyi Zhang, Handong Zhao, Chaochao Lu, Shuai Li, and Ricardo Henao. Infoprompt: Information-thcoreiii soft prompt tuning for natural language understanding. Advances in Neural Information Processing Systems, 36, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1rX5v:/905.0785. 2019.
Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. The Twelfth International Conference Learning Representations, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint MoXXv:19044096755 2019.
Zhan Zhuang, Yulong Zhang, Xuehao Wang, Jiangang Lu, Ying Wei, and Yu Zhang. Time- varying loRA: Towards effective cross-domain fine-tuning diffusion models. In The Thirty- eighth Annual Conference Neural Information Processing Systems, 2024. URL https: 7/ //openeeview httt:/orrrmmm---00022222
14

---

Published as a conference paper at ICLR 2025
A
TRAINING DETAILS
A.1 TRAINING HYPERPARAMETERS
Table outlines hyperparameters used for tuning the Llama-2-7B and Llama-3-8B models with HiRA across two tasks: commonsense reasoning and CONVAI2. Both tasks utilize the same hyper- parameter set, with the primary distinction being the number of epochs; the commonsense reasoning task runs for three epochs, whereas CONVAI2 runs for a single epoch. Each experiment conducted separately, employing varying for a single run each model. The best results are chosen based on the validation loss. For bascline methods, the hyperparameters also reuse the above configurations. Different PEFT methods could have different hyperparameters to be set for comparable or the same number of trainable parameters.
Table 7: Hyperparameters for HiRA.
Hyperparameter
Value
Optimizer
Adam
Weight Decay
0
Base Model
[Llama-2-7B, Llama-3-8B]
Learning Rate
[0.0001, 0.0002]
[2, 4, 8, 16, 24, 28, 30)
Warm Up
100 steps
Batch Size
32
Target Modules
q proj, k proj. v v-proj. up- proj, down proj Every 80 steps
Evaluation Steps
Table 8: The statistics of the CONVAI2 dataset.
Data Split Utterances
Dialogues
Personas
Train 131,438 Test 7,801
17,878
1,155
1,000
100
A.2 STATISTICS OF THE CONVAI2 DATASET
We conduct experiments using the ConvA12 dataset (Dinan et al. 2019). a benchmark for open- domain dialogue generation tasks. As shown in Table this dataset consists of 17,878 training and 1,000 testing multi-turn conversations collected from crowdworkers. Each dialogue features persona profiles that provide four to five sentences describing each speaker's background, along with the conversational history between the two interlocutors. In our experiments, by following (Liu cct al.) 2020 Song et al., 2021 Huang et al. 2023b a) we utilize self-person setting, where only the speaking interlocutor's ' persona is revealed, while the persona of the other remains undisclosed.
A.3 EXPERIMENTAL SETTINGS FOR LoRA WITH DIFFERENT RANKS
In this section, we detail the experimental settings used for fine-tuning the Llama-3-8B model with various LoRA configurations. The specific hyperparameters, including the rank (r), learning rate (Ir), and batch size, are outlined in Table 9
Table lists the specific configurations used for each LoRA rank. Higher ranks, such as 1024, required a lower learning rate of 1.00e 06 and a smaller batch size of to prevent issues like gradient explosion. In contrast, lower ranks like 32 could be trained with higher learning rate of 1.00e 04 and a larger batch size of 72.
Table 10 provides the general hyperparameters that were consistent across all experiments, ensuring a fair comparison between different configurations. These settings were designed to optimize the fine-tuning process for the Llama-3-8P model while maintaining computational efficiency.
15

---

Published as a conference paper at ICLR 2025
Table 9: LoRA Configurations with Different Ranks, Learning Rates, and Batch Sizes.
LoRA Config
Learning Rate (Ir)
Batch Size
r=1024
1.00c-06
8
r=512
1.00e-06
16
r=256
1.00e-05
32
1=128
5.00e-05
72
r=32
1.00c-04
72
Table 10: General Hyperparameters Used Across All Experiments.
Hyperparameter
Value
Optimizer
AdamW
Weight Decay
0
Base Model
Llama-3-8B
Warm Up
100 steps
Target Modules
q proj. proj, v proj, up proj. down proj Every 80 steps
Evaluation Steps
A.4 TRAINING COST
The computational cost for training the commonsense reasoning task requires 14 GPU hours over 3 epochs on Nvidia-A10k 80G GPU on Llama-3-8B, while the CONVAI2 task requires GPU hours for single epoch under the HiRA (r 32) Nvidia-A100 80G Llama-3-8B.
A.5 STATISTICS OF COMMONSENSE REASONING DATASET
As illustrated in Table 11 the training set comprises mix of eight sub-tasks totalling 300 entries, while the validation set contains random selection of 120 entries. The test dataset also covers these eight sub-tasks and is evaluated using single trained model.
Table 11: The detailed statisties of commonsense reasoning datasets.
Dataset
Data Number
Type
Train
170.300
Mixed
Validation
120
Mixed
Test
BoolQ
3,270
Yes/No
PIQA
1,830
Option
SIQA
1,954
Option
HellaSwag
10,042
Option
WinoGrande
1,267
Option
ARC-e
2,376
Option
ARC-c
1.172
Option
OBQA
500
Option
A.6 GPU MEMORY CONSUMPTION AND RUNNING TIME ANALYSIS
This section presents comparative analysis of GPU memory consumption and running time be- tween the LoRA method and the proposed HiRA configurations under similar settings of trainable parameters. The experiments conducted using the LLaMA-3-8B model.
16

---

Published as a conference paper at ICLR 2025
Table 12: Comparison of GPU Memory Consumption and Running Time
Configuration
GRAM (GB)
Training Hours
LoRA (r 32)
65.48
15 hours 0 minutes 56 seconds
HiRA (r 32)
61.49
14 hours 09 minutes 32 seconds
EXPERIMENTAL SETUP
Model: LLaMA-3-8B
Batch Size: 72
Epochs: 3
Data Size: 170k entries
LoRA Rank: 32
HiRA Rank: 32
B PROOF OF HIRA'S EXPRESSIVE POWER
In this section, we give the details proof of the expressive power of HiRA in comparison to LoRA. We begin by introducing the Eckart- Young-Mirsky Theorem (Eckart & Young 1936), which pro- vides the optimal low-rank approximation of matrix. We will refer to this theorem as Lemma [ Lemma 1. (Eckart- Houng-Mirsky Theorem) The best rank-r approximation of matrix W in the spectral norm is given by the (r + 1)-th largest singular value, ie..
min W:Rank(15 (W) <T
[
W|]
or+ (W).
According to Theorem 5.5.1 in (Horn & Johnson
1994), we have the following Lemma 2
Lemma 2. For matrices A and B, the Hadamard product satisfies || @ BlI2 < || |
Theorem 1. (The Expressive Power of HiRA) Consider the optimal parameter update E and the HiRA update with the form Wo O Whis where the rank of Whs is less than Then we have
min Wa.Baan War
W @ Wh
F. S Gr+1 @ Wo) [Woll2.
where 2 denotes the element-wise division.
Proof. The proof proceeds as follows
min Wo @ Wha Ell, min Ws.
Wo (Whs E Wo)
min
Wolla (W - - E Wo)
Wh:
- E 2 Wo) [Woll
In this analysis, we assume that Wo contains no zeros and use to denote the element-wise division. The optimization objective minw,,, [W @ (Wh: Ø Wo)l|, equivalent to the weighted low- rank approximation problem, which is NP-hard (Srebro & Jaakkola, 2003). As a result, we can analyze it by applying the upper bound provided by Lemma This indicate that HiRA's expressive power is linked to the singular value of E Ø Wo.
Notably, HiRA can exhibit greater expressive power than LoRA, particularly since the updated weights correspond the original weights. For instance, consider the following matrices
2 1
[4 2] 1 2
00.5 0.5
F
W
Fow
1 1
In this toy example, find that Fr41 (E @ W) 0. while 0 (E) 1. Thus have 0r (E Ø
W) ||]]] < ar-1(F)
17

---

Published as a conference paper at ICLR 2025
C COMPARISON OF EFFECTIVE RANK WITH DIFFERENT METHODS
conducted additional experiments to analyze the effective rank (Roy Vetterli 2007) of differ- ent fine-tuning methods, as presented in Figure
1000
49.0
IFT
LoRA
20:00
20:0
1=RA
100
LoRA
HIRA
25
20
20
15
15
25
Figure Effective rank across layers for FFT, LORA, MoRA, and HiRA.
Unlike simply calculating the rank counting the number of singular values above given thresh- old, the effective rank accounts for the full singular value spectrum. For matrix A of size M x N with [M, the effective rank is defined follows:
erank(A) ()) = exp (-2 Pk log
Pk
|
where denotes the k-th largest singular value and
- is the I --nrr of the singular values.
From the results, we can observe that the update matrix obtained by FFT achieves the highest fective rank. For the query component, MoRA shows behavior closer to FFT, while for the value component-and especially for the projection layer (up and down components)- better. Conversely, even LoRA with rank=512 offers relatively lower effective rank. This analysis follows similar approach to that (Shuttleworth et al. 2024h.
exhibits
D EVALUATION ON TRANSFER LEARNING TASKS
We trained HiRA and LoRA separately on eight commonsense reasoning tasks and evaluated their performance on these tasks. As shown in the Figure 9 HiRA demonstrates strong transferability across tasks, achieving high scores on related tasks. This indicates its ability to generalize effectively beyond the specific task it was trained on.
Tardeesraii Exiiii VSL Stwedl
Twedinaall anthas BA PeNil BHN
V1A
605
10
Figure 9: Transferability analysis for (a) LoRA and (b) HiRA. Each represents the performance of fine-tuned model trained on a specific task dataset when evaluated across eight different tasks.
In comparison, LoRA exhibits more pronounced overfitting. While performs well on its training task (e.g., bool scores 71.44 on boolq), its performance drops significantly other tasks. In
18

---

Published as a conference paper at ICLR 2025
contrast, HiRA maintains respectable transferability, on siqa, 70.20 on obqa, and 58.56 on winog. outperforming LoRA in these cases. HiRA also excels at harder tasks requiring (winog). On the other hand, LoRA performs relatively better on simpler tasks like BoolQ and PIQA, but its overfitting tendency causes significant drops in performance unrelated tasks.
with arce-hira, which achieves 64.23
complex reasoning, such HellaSwag (hellas) and WinoGrande
In summary. HiRA shows superior generalization and less prone to overfitting compared to LoRA. Evaluating HiRA on transfer learning tasks could further validate its robustness and adaptability.
E
IMPACT OF MORE CHOICES OF R
E. REDUCING THE RANK OF R
To validate the correlation between the rank of AW and expressiveness, we conducted an addi- tional experiment by reducing the rank of R on LLaMA-3-SB across commonsense reasoning tasks. Specifically, R is derived from Wo. the pretrained weight matrix, using SVD decomposition, We retained only the top-u singular values and the corresponding singular vectors to form reduced- rank version, Wo.... The results, presented in Table T3, show that increasing the rank of R generally improves the performance across tasks.
Table 13:
Performance comparison between R with different ranks based on Wo-
Model
Rank
BoolQ
PIQA
SIQA
ARC-e
ARC-e
OBQA
HellaS
WinoGi
Average
R
Wo.j32)
32
73.67
88.63
80.04
80.97
93.18
86.00
94.31
86.58
85.42
Wo.8-128
128
74.16
89.23
82.04
80.89
93.10
87.20
95.11
87.45
86.
Wo.jor
512
74.34
89.88
81.99
82,25
93.86
88.20
95.38
87.61
86.69
Wo (Ours)
2852
75.40
89.70
81.15
82.90
93.27
88.32
95.36
87.70
86.72
E.2 USING Wo 'S IN ADJACENT LAYERS AS R.
We experimented with smoothed version of Wo by incorporating information from adjacent layers. Specifically, the smoothed Wo (R W smooth) was computed as the average of the upper, current, and lower layers for the same transformer component. The results, summarized in the Table [4] show that the smoothed Wo achieves an average score 83.71, slightly lower than the original Wo, which achieves 86.72. While smoothing provides competitive performance, the original Wo retains a clear advantage, likely due to its direct alignment with the pretrained weights, preserving task-specific expressiveness and parameter efficiency.
Table 14:
Performance comparison between smoothed Wo and Wo used as R.
Model
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
W smooth
72.02
83.84
79.84
81.23
90.24
85.60
92.07
84.85 87.70
83.71
Wo (Ours)
75.40
89.70
81.15
82.90
93.27
88.32
95.36
86.72
F EXPERIMENTAL EVIDENCE ON HIRA'S EXPRESSIVE POWER
Theorem establishes an upper bound on the approximation error of HiRA, which proportional to 0r+1 (E @ Wo), in contrast to the 0,41 (EE in LoRA. In this section, we use experimental evidence to prove the advantages of 0 E 2 Wo).
Since the optimal update matrix E comes from a complex distribution and is not predetermined, directly proving this inequality challenging. We approximate the optimal update matrix E through the update matrix of full fine-tuning (FFT), denoted by E,, such that:
E AW = Wm Wo.
where Wm denotes the weights of the fully fine-tuned models.
19

---

Published as a conference paper at ICLR 2025
Directly comparing the singular values of E 2 Wo and E may be influenced by the norm of Wo. To mitigate this, we compute the normalized singular values for comparison. Additionally, we analyze the effective rank as described in Appendix C to provide further insights.
W/
W4
W Wanan
0.00
E We
6.4
0.50
Peudd Layer (lw521
Mode:
Mooto Laver
Moddd (rm321
Mould Loyir (tw521
Figure 10: Normalized singular values of layers for E and E @ Wo-
W
W
W.
Wsp
Wetes
1000
1X
2500
2006
2007
2520
Eo W.
2000
100
350
15
25
Model
Figure 1I: Effective rank across layers for E and E 2 Wo-
The singular values of E @ Wo exhibit a faster rate of decline compared to those of E, as demon- normalized singular values to account for the influence
strated in Figure TO This comparison of Wo's s scaling. Furthermore, Figure 11 demonstrates that the effective rank of E. s Wo is lower than that of E. Those findings indicate that Ø Wo aligns more effectively with low-rank approx- imations compared to F, supporting the conclusion that HiRA can leverage the Hadamard product fully utilize the knowledge embedded in the full-rank Wo.
20


Reviews:
[{"review_id": "6bzEN3yi0y", "reviewer": "Reviewer_P2fT", "summary": "The paper proposes Hadamard High-Rank Adaptation (HiRA), a parameter-efficient fine-tuning (PEFT) method for LLMs, to address limitations of Low-Rank Adaptation (LoRA) for complex tasks. The authors argue that while LoRA's expressiveness can be limited by its low-rank nature, which may restrict the model's ability to adapt to complex tasks. HiRA addresses this limitation by applying a Hadamard product between the low-rank update matrices and the frozen pretrained weights. The authors claim that this method effectively creates high-rank updates and enhance the expressiveness of the LLMs without significantly increasing the number of trainable parameters.", "questions": "1. Is there a way to gain insights that using HiRA's Hadamard product is a better choice than other high-rank methods.\n2. Can the authors justify that the singular value patterns of HiRA closely correlated to expressiveness properties that benefit tasks? Perhaps a detailed comparison of HiRA and MoRA in this context might help.\n3. Are there attributes of HiRA other than high-rank strucutre that may contribute to its improved performance? Again, an in-depth comparison of HiRA and MoRA might help clarify this question.\n4. Is HiRA prone to overfitting? Would evaluating HiRA on, say, transfer learning tasks help address this question?\n5. Could you provide quantitative evidence of HiRA’s computational efficiency, such as memory usage, training time, and inference latency?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a creative way of using the Hadamard product to achieve high-rank adaptation within PEFT. This approach is an important contribution to address the common expressiveness limitation of LoRA-like PEFT methods. The authors show HiRA's consistent improvements over LoRA and other baselines which demonstrate that HiRA not only theoretically sounds but also practically effective. The ablation studies, such as singular value distribution analysis, show how HiRA's parameter dynamics behaves somewhat similarly to those of full fine-tuning and provide insights into the understanding of HiRA's expressiveness. Furthermore, the paper also compares HiRA not only to LoRA but also to another high-rank adaptation method called MoRA to justify the claim that its Hadamard product approach has unique advantages in achieving high-rank adaptability. Finally, HiRA seems to be lightweight and computationally efficient, which is attractive for practical applications.", "weaknesses": "First, my major concern is the connection between high rank, expressiveness and improved performance. Although the paper attempts to address this connection through ablation studies and performance improvements in experiments, it lacks a rigorous explanation that directly links high rank to expressiveness and explains why this is essential for improved performance in complex tasks. In other words, without showing how the observed rank impacts various task complexities, it's unclear if performance gains are solely due to the high rank, not other factors. For example, it's possible that the similarity in singular values of HiRA compared to FFT is due to an artifact of the data or tasks rather than an inherent property of high-rank adaptations.\n\nSecond, the paper lacks an in-depth analysis to show unique advantages of HiRA's high-rank nature. In particular, it's unclear if the improved performance of HiRA over MoRA is due solely to high-rank structure of Hadamard product, not other aspects of HiRA. Without detailed comparison of HiRA with other high-rank approaches such as MoRA, for example, showing distinctive singular value patterns of HiRA compared to those of MoRA, it's not easy to see unique benefits of HiRA's high-rank structure.\n\nFinally, the paper provides limited insights into HiRA's generalization ability. In particular, while a high-rank structure might improve adaptability, it could also be potentially overfitting. The paper would benefit from evaluation of HiRA on, say, tasks requiring transfer learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eJmjkanOHi", "reviewer": "Reviewer_q3zD", "summary": "This paper proposes the use of the Hadamard product in the new technique HiRA to parametrize _high-rank_ updates to the base model, based on LoRA, while using the same parameter count. Empirically HiRA outperforms LoRA on various tasks. The authors also provided analyses showing 1) the updates in HiRA are indeed high rank and 2) prominent singular values of HiRA trend more like full fine-tuning than LoRA. Finally the authors look into combining LoRA and HiRA as HiLoRA.", "questions": "Did the authors consider alternative parametrization of the right operand of the Hadamard product? For example instead of W_0 maybe 'smooth' it with adjacent layers / other attention components within the same layer?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear presentation that empathizes the simple insight that the Hadamard product yields high-rank matrices. Extensive experiments. Convincing ablation studies.", "weaknesses": "More theoretical analysis would be useful. For example, more clarification on why Theorem 1 is important will help readers a lot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rTwcs7tlPw", "reviewer": "Reviewer_m6FX", "summary": "This paper introduces Hadamard High-Rank Adaptation (HiRA), a novel parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs). HiRA aims to enhance the adaptability of LLMs by using a Hadamard product to retain high-rank update parameters, addressing the limitations of low-rank adaptation methods like LoRA. The authors demonstrate HiRA's effectiveness through extensive experiments and ablation studies, showing improved performance across various tasks compared to LoRA and its variants.", "questions": "1. How does HiRA perform on very large language models (e.g., models with hundreds of billions of parameters)? Are there any scaling challenges?\n\n2. Have you explored the potential of combining HiRA with other PEFT techniques? Could this lead to even better performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a new approach to PEFT by applying the Hadamard product to LLM adaptation, which is an original contribution to the field. Compared with LoRA, this method, HiRA, addresses a key limitation of LoRA by enabling high-rank updates, potentially improving the model's capacity to adapt to new tasks.\n\n2. This method maintains a comparable number of trainable parameters to LoRA while enhancing performance, offering a cost-effective solution without additional inference overhead.\n\n3. The paper offers a theoretical basis for the proposed method, and provides experimental results and ablation studies to demonstrate the effectiveness of the method.", "weaknesses": "The paper focuses primarily on comparison with LoRA and its variants. It would be beneficial to see comparisons with a broader range of PEFT methods. And it's unclear how well it generalizes across different model architectures or scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)