HiLoRA: High-frequency-augmented Low-Rank Adaptation

Authors: Hyowon Wi, Noseong Park
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Large Language Models, LoRA, Frequency, Catastrophic Forgetting

Abstract:
As large language models (LLMs) have demonstrated remarkable performance, parameter-efficient fine-tuning (PEFT) has emerged as an important paradigm. As a solution, low-rank adaptation (LoRA) freezes the pre-trained weights and introduces small learnable adapters instead of fine-tuning the full set of parameters. However, LoRA suffers from $\textit{catastrophic forgetting}$, where pre-trained knowledge is overwhlemed and forgotten as new information is learned. One cause of this issue is $\textit{implicit regularization}$, where deep learning models tend to favor more generalized solutions. This tendency leads to a significant increase in the largest singular values of the weights, which correspond to low-frequency components. To address this problem, we propose an advanced LoRA that balances the retention of pre-trained knowledge with the learning of new information. Since fine-tuning involves learning fine-grained details, which correspond to high-frequency information, we designed HiLoRA, a method that injects learnable high-frequency components into the pre-trained model. By leveraging the parameterized SVD and constraining singular values to appropriate levels, HiLoRA adapts to new tasks by focusing on the high-frequency domain with minimal change from the pre-trained weights. To evaluate the effectiveness of HiLoRA, we conduct extensive experiments on natural language understanding and question answering tasks. The results show that HiLoRA not only improves performance but also effectively retains pre-trained knowledge compared to baseline models.

Decision: reject
Reviewer scores: 6, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1729367205059

thread_type:
official_review

<thread_object>
creator: reviewer_k5bg

<content>
summary:
The paper presents a method to mitigate the catastrophic forgetting of LoRA, by restricting the adaptation matrix to have singular values clamped to an upper bound.

strengths:
* The authors tackle an important and meaningful topic that would be of interest to the community. PEFT is gaining more and more attention as the size of language models continues to grow.
* The results demonstrate that HiLoRA forgets less than LoRA, and some of its variants.

weaknesses:
* Line 190, authors should explain more why depth of 2 or greater results in a separation of values.
* Equation (6), line 244, I believe the shape of U should be d_1xr, Sigma should be rxr and V should be rxd2, unlike what is written.
* Tables 1 and 2 better include the model name in the table description. The two tables use a different model. 
* Table 1 shows that AdaLoRA has a lower average result than LoRA (On Roberta). However, in the original AdaLoRA paper that compared the two models on Deberta, AdaLora got better results. On other papers that compared the two on Roberta, AdaLora got better results. Are the results taken from other papers or calculated by the authors?
* Line 354, word Indicating should be with a small letter.
* Figure 4: In tables 1 and 2, authors compared 5 variants, and here only 4. AdaLoRA was omitted. The readers are interested to know the ‘forgetting’ property of AdaLoRA.

questions:
* Equation (4), I can’t find the definition of i. Maybe it should appear as the index of the singular value on the left side, instead of n.
* Figure 4: The title says MRPC, however, in line 464 it says STS-B.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730175577743

thread_type:
official_review

<thread_object>
creator: reviewer_mcnz

<content>
summary:
This paper addresses an intriguing issue—mitigating forgetting during fine-tuning of neural networks. It introduces a method focused on the fine-tuning of high-frequency components of pre-trained weights, claiming measurable improvements.

strengths:
1.	This paper proposes a new sight for fine-tuning and tries to solve the forget problem in fine-tuning.
2.	It does reduce the accuracy loss on the pre-trained task.

weaknesses:
1.The architecture diagrams in Figure 2 are unclear. To improve clarity, the implementation details of HiLoRA should be included in the main paper instead of relegated to the appendix.

2.The HiLoRA design lacks novelty.

3.The paper mentions, "U and V can be initialized with random r singular vectors of W0, or with U initialized to zero and V with a random Gaussian initialization." What initialization strategy was used in your experiments?

4.A deeper analysis of the relationship between the pre-trained task and the fine-tuning task in relation to the forgetting method would be beneficial. If the pre-trained and fine-tuning tasks are similar, does the forgetting problem still occur?

5.A scaling-up experiment, such as fine-tuning LLaMA-2-7B with Meta-Math and evaluating it on GSM8K, as in PiSSA, would be insightful.

6.Does this method remain effective when the rank increases?

7.Equations (6) and (7) should be introduced within Algorithm 1.

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639456750

thread_type:
official_review

<thread_object>
creator: reviewer_vq3k

<content>
summary:
In this paper, the author proposes a new method to address the problem of catastrophic forgetting in LoRA fine-tuning. They suggest learning the adapter $\Delta W$ with small eigenvalues and validate their method across various downstream tasks.

strengths:
1.	This paper describes the problem and method in detail.
2.	The authors validate the effectiveness on various downstream datasets.

weaknesses:
1.	The definition of "frequency" in the paper is confusing and meaningless; low frequency and high frequency merely refer to the value of the eigenvalues. If the largest eigenvalue corresponds to pre-trained knowledge, why does an increase in the largest singular value during fine-tuning, as shown in Figure 1(a), lead to a decline in performance on the pre-training task? From Figure 1(a), it appears that they are either negatively correlated or unrelated.
2.	The novelty is limited. It is very similar to, including the implementation in Figure 7 and Equation (8). The only difference is that the $\Sigma$ matrix is learnable and clipped according to Equation (7). Additionally, in Table 2, HiLoRA performs worse than AdaLoRA, and the authors do not address the issue of forgetting in AdaLoRA.
3.	The motivation for investigating catastrophic forgetting in LoRA requires further explanation. When employing LoRA, my primary concern is its performance on specific downstream tasks. If I need to tackle multiple different tasks, I would prefer to use a general LLM or load various LoRA adapters through MultiLoRA.
4.	More comparisons are needed. The authors should include the results of PiSSA and MiLoRA in Table 2, and it needs to compare with methods such as DoRA, rsLoRA, and LoRA+. Furthermore, experiments should be conducted on larger LLM models, such as LLaMA.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705716165

thread_type:
official_review

<thread_object>
creator: reviewer_36is

<content>
summary:
In this paper, the authors proposed a new parameter-efficient fine-tuning (PEFT) approach with learnable high-frequency components while  constraining singular values to appropriate levels. Multiple experiments and ablation studies were conducted to show the effectiveness of the proposed method.

strengths:
1. The motivation and formulation of the proposed HiLoRA makes sense and is technically sound to me.
2. The authors conducted extensive experiments on multiple datasets and showed improved performance over several baseline methods.
3. Ablation studies and sensitivity analysis were also conducted to show the effectiveness of the proposed approach.
4. Writing is good and easy to follow.

weaknesses:
1. The performance improvement seems very small as shown in table 1 and 2 and sometimes even worse than baseline methods.
2. Are there any principles or rule or thumb for setting those hyper-parameters for different tasks/models?

questions:
Overall, I think the HiLoRA proposed in this paper is novel and beneficial to the community. Please refer to the weakness section to prepare rebuttals on the performance and hyper-parameter setting.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732105417031

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers

comment:
We thank all reviewers for their valuable time and insightful comments. We leave key general remarks here and will respond to each review below.

**1. Comparison with other state-of-the-art baselines. (Reviewer 36is, vQ3k)**

we have added state-of-the-art baselines on the GLUE task with 4 random seeds. Due to time constraints, we have included these baselines only for the following six datasets, and we plan to continue updating the tables for other two remaining datasets.

| Method | SST2 |CoLA | QNLI | RTE | MRPC | STS-B |
| --- | --- | --- | --- | --- | --- | --- |
| DoRA [1] | 92.32 |64.29 | 92.90 | 81.68 | 89.52 | 90.95 |
| LoRA+ [2] | 93.98 |63.63 | 92.72 | 81.23 | 89.03 | 90.87 |
| rsLoRA [3] | 95.04 |64.17 | 92.59 | 79.78 | 89.40 | 90.84 |
| HiLoRA | **95.10** | **64.66** | **93.12** | **82.85** | **90.20** | **91.16** |

**2. Catastrophic forgetting in AdaLoRA. (Reviewer vQ3k, K5bg)**

We measured the catastrophic forgetting phenomenon in AdaLoRA using the MRPC and STS-B datasets in the GLUE task. Specifically, we measured the largest singular value and Frobenius norm of the difference between the pre-trained model and the fine-tuned model. Also, we evaluated the accuracy and the evaluation loss on the pre-trained task. For each dataset, the metrics were measured every 5 epochs during the fine-tuning of AdaLoRA, and the metrics at the point where AdaLoRA and HiLoRA achieved their best accuracy are also reported, respectively. The results are presented in the table below.

- MRPC dataset

| Epoch | 5  | 10 | 15 | 20 | 25 | 30 | Best (AdaLoRA) | Best (HiLoRA) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Largest singular value | 2.0177 | 2.0069 | 2.0016 | 2.0009 | 2.0004 | 2.0001 | 2.0177 | 0.9358 |
| Frobenius norm | 2.0201 | 2.0050 | 2.0014 | 2.0011 | 2.0005 | 2.0001 | 2.0201 | 0.9284 |
| Accuray on pre-trained task  | 25.578 | 14.879 | 8.962 | 16.794 | 10.591 | 11.343 | 25.58 | 32.00 |
| Evaluation loss on pre-trained task | 5.231 | 6.696 | 7.617 | 6.178 | 7.103 | 7.007 | 5.2308 | 4.3496 |

- STS-B dataset

| Epoch | 5 | 10 | 15 | 20 | 25 |  Best (AdaLoRA) | Best (HiLoRA) |
|  --- | --- | --- | --- | --- | --- | --- | --- |
| Largest singular value | 2.0064 | 2.0034 | 2.0056 | 2.0021 | 2.0001 | 2.0001 | 0.9351 |
| Frobenius norm | 2.0087 | 2.0030 | 2.0065 | 2.0013 | 2.0001 | 2.0001 | 0.9312 |
| Accuray on pre-trained task  | 33.526 | 34.118 | 29.61 | 28.19 | 28.485 | 28.49 | 43.72 |
| Evaluation loss on pre-trained task | 3.984 | 3.951 | 4.426 | 4.566 | 4.525 | 4.5248 | 3.1785 |

According to the tables, AdaLoRA also experiences catastrophic forgetting as its fine-tuning progresses. The Frobenius norm increases from 0 to 2 in the early fine-tuning phase, with the performance on the pre-trained task decreases. Even at its peak performance during fine-tuning, the model still exhibits low performance on the pre-trained task.

This can be attributed to the lack of consideration for frequency components of adapters, leading to a tendency for learning the low-frequency components while forgetting the pre-trained information. In contrast, the proposed model regularizes the frequency components in the adapter, injecting the new knowledge into high-frequency components during fine-tuning. As a result, the proposed model retains pre-trained information more effectively.

**3. Validation of the scalability of the proposed method through large-scale experiments. (Reviewer vQ3k, K5bg)**

We conduct the experiments for the commonsense reasoning task on LLaMA-7B. Following [4], we amalgamate the training datasets from 8 sub-tasks to create the final training dataset, and conduct evaluations on the individual testing dataset for each task. The results are provided in below table.


| LLaMA-7B | BoolQ | PIQA  | SIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | QBQA  | Avg.  |
|----------|-------|-------|-------|-----------|------------|-------|-------|-------|-------|
| LoRA     | 68.9  | 80.7  | 77.4  | 78.1      | 78.8       | 77.8  | 61.3  | 74.8  | 74.7  |
| HiLoRA | 62.2 | 82.7 | 78.3 | 81.0 | 80.9 | 83.3 | 66.8 | 78.6 | **76.7** |

HiLoRA demonstrates improved performance over LoRA on average in large-scale models, highlighting its stability and effectiveness while maintaining strong results across diverse downstream tasks.


>[1] Liu, Shih-yang, et al. "DoRA: Weight-Decomposed Low-Rank Adaptation." Forty-first International Conference on Machine Learning.
>
>[2] Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. "Lora+: Efficient low rank adaptation of large models." preprint, 2023.
>
>[3] Kalajdzievski, Damjan. "A rank stabilization scaling factor for fine-tuning with lora."  preprint, 2023.
>
>[4] Hu et al. "LLM-adapters: An adapter family for parameter-efficient fine-tuning of large language models." preprint, 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732294143435

thread_type:
official_comment

<thread_object>
creator: area_chair_ufhb

<content>
title:
Interactive Discussions

comment:
Dear Reviewers,

Thank you for your efforts in reviewing this paper. We highly encourage you to participate in interactive discussions with the authors before November 26, fostering a more dynamic exchange of ideas rather than a one-sided rebuttal.

Please feel free to share your thoughts and engage with the authors at your earliest convenience.

Thank you for your collaboration.

Best regards,
ICLR 2025 Area Chair
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission proposes a simple method to address catastrophic forgetting in LoRA fine-tuning. The method focuses on learning adapters with small eigenvalues, with its performance validated across various downstream tasks.

Strengths: Simple method, promising performance.

Weaknesses: Limited novelty, over-claimed forgetting mitigation.

After reviewing all the feedback (2 positive and 2 negative reviews) and the rebuttals, the Area Chair concludes that the submission does not meet the ICLR standard (borderline on the reject side) mainly due to its lack of novelty compared to existing LoRA fine-tuning methods.