GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection

Authors: Xutao Liao, Shaohui Li, Yuhui Xu, Zhi Li, Yu LIU, You He
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, parameter-efficient fine-tuning, low-rank

Abstract:
Recent low-rank training methods, such as GaLore, have significantly reduced the memory required to optimize large language models (LLMs). However, these methods often suffer from time-consuming low-rank projection estimations. In particular, the singular value decomposition (SVD) in GaLore can consume more than 80\% of the total training time. To address this issue, we propose GaLore$+$, which uses cross-head low-rank projection to reduce the substantial time consumption in estimating low-rank projections for multi-head attention. In addition, we employ randomized subspace iteration to achieve fast SVD. To further enhance performance, we propose sparsely coded residuals to reduce the errors caused by low-rank approximation on the first- and second-order moments of the optimizers and weight updates. We evaluate GaLore$+$ on arithmetic reasoning and natural language generation datasets. Our experiments demonstrate that GaLore$+$ delivers superior performance while achieving approximately $4\times$ fine-tuning speed compared to vanilla GaLore.

Decision: reject
Reviewer scores: 6, 3, 3, 6

Reviews and discussion:
<thread>
creation date:
1730124828607

thread_type:
official_review

<thread_object>
creator: reviewer_r2k7

<content>
summary:
The paper introduces an improved version of GaLore, significantly reducing training time and memory usage. The authors first proposed the cross-head low-rank projection, which obtained the low-ranking gradient projection via the SVD of a randomly selected head rather than that of all heads. They further proposed a mechanism to rectify the first, second moments and the weight updates to reduce the error induced by the low-rank approximation.

strengths:
- GaLore+ showcases its superiority in the arithmetic reasoning task.
- The authors successfully propose a training-accelerating version of Galore, which will reduce both the training resources and improve performance.

weaknesses:
- The empirical results on the natural language generation tasks show insignificant improvements of GaLore+ over LoRa. 
- In Line 50, the authors mentioned that the errors in Equation 6 are lower than those in Equation 4, as depicted in Figure 2b. This is because Equation 6 treats all attention heads collectively, thereby not accurately capturing the structure of each individual attention head. Does a lower error indicate a better scenario? Is the error calculated for each head individually or across all heads?
- How does GaLore+ determine the head used for computing low-ranking gradient projection? Based on my understanding, it is selected randomly. Is there any theoretical or empirical rationale behind this choice? If not, at the very least, conducting an ablation study on various head selection strategies is essential.

**Minor errors**: 

- Tebble in Line 401.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690347415

thread_type:
official_review

<thread_object>
creator: reviewer_zudp

<content>
summary:
The paper presents GaLore+, a new parameter-efficient finetuning (PEFT) technique that extends previous work on GaLore to enhance its time-efficiency as well as downstream performance. To reduce the cost of GaLore taken to compute SVD factors on weight gradients, GaLore+ proposes cross-head low-rank projection which uses a single attention head to compute its low-rank projection, and apply the same projection across all heads. Instead of deterministic SVD, GaLore+ also uses randomized SVD via subspace iterations. Lastly, inspired by previous work on low-rank plus sparse approximators, GaLore+ adds a sparsely coded residual to the low-rank factors, approximating the true gradients better than with only low-rank factors. Experiments using the Llama2-7B model on arithmetic reasoning and natural language generation datasets show that GaLore+ significantly reduces the cost of finetuning in both time and memory.

strengths:
- [S1] **Clear motivation and interesting problem setup.** Parameter-efficient finetuning is a widely used technique for adapting pretrained LLMs, and the techniques introduced in this paper would be a good addition to the LLM finetuning toolkit. The motivation is also crystal clear with preliminary experimental results presented early in the paper: to reduce the time-consumption and enhance performance of GaLore.

weaknesses:
- [W1] **Weak novelty and contribution.** The techniques newly incorporated into GaLore+, which include cross-head low-rank projection, randomized SVD, and sparse residuals, are all essentially from existing work, which significantly weakens the overall novelty of this work. The techniques are also discussed and demonstrated solely under the scope of applying gradient projections (i.e. vanilla GaLore) and thus the contributions are too shallow, and does not seem sufficient to constitute a conference paper at ICLR.
- [W2] **Lacking experimentation.** Even though the paper is mostly empirical, the experimental setup is narrow, covering two arithmetic reasoning datasets and one natural language generation dataset. Considering that GaLore+ can be used essentially anywhere when finetuning pretrained Transformers, it would be great to add (1) more datasets and pretrained models as well as (2) more PEFT baselines (other than LoRA and GaLore) to fully argue that the gain in time-efficiency as well as downstream performance leads to competitive results.
- [W3] **Weak intuition behind cross-head low-rank projection.** The discussion on the cross-head similarity for simplified SVD (Section 4.1.1) is a bit hard to follow. Specifically, how can it be the case that "the gradient of different heads within the same layer are similar" (Lines 236-237), but treating all attention heads as a whole "fails to capture the structure of each individual attention head accurately" (Lines 250-251)?  Shouldn't the hypothesis essentially allow us to treat all attention heads as a whole?

questions:
- [Q1] For Figure 2b, is GaLore+ (in blue) using each $i$-th head to approximate the gradient of the $i$-th head? How can it be that GaLore+ is achieving lower RMSE on all heads, when the rank-$r$ SVD used by vanilla GaLore is known to provide an optimal rank-$r$ approximation in Frobenius norm?
- [Q2] How many random seeds were used for main results in Tables 1 and 2? What were the standard deviations on different performance metrics?
- [Q3] When using randomized subspace iteration to perform fast SVD, how many subspace iterations does GaLore+ use? 
- [Q4] Based on Figure 3, it seems the sparsely coded residual does not have a significant impact on downstream performance. Could we compare LoRA and GaLore vs. GaLore+ without sparsely coded residuals to better assess the performance gains?
- Typo in Line 401: Teble 1 $\rightarrow$ Table 1
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721973761

thread_type:
official_review

<thread_object>
creator: reviewer_ykpf

<content>
summary:
GaLore+ is proposed as an enhancement to GaLore, addressing the computational bottleneck of SVD which consumes over 80% of training time in the original method. The improved method combines cross-head low-rank projection and randomized subspace iteration for faster SVD computation, while introducing sparsely coded residuals to minimize approximation errors in optimizer moments. Experimental results show that GaLore+ achieves 4x faster fine-tuning speed compared to vanilla GaLore while maintaining or improving performance on arithmetic reasoning and natural language generation tasks.

strengths:
This paper presents an upgraded version of GaLore that employs randomized SVD for acceleration and uses Residual of Moments to compensate for potential accuracy loss from the randomized approach.

weaknesses:
While the core idea is straightforward and easy to understand, the paper's primary weakness lies in insufficient experiment, making it difficult for reviewers to properly evaluate the effectiveness of these enhancements. 

- Counter-intuitive results require explanation. While the method aims to accelerate training through randomized SVD and use the sparsely coded residuals to reduce the approximation errors, it's fundamentally still an SVD computation. Theoretically, the performance upper bound should be the exact SVD computation. Although speed improvements are understandable, the significant performance gains over exact SVD computation (GaLore) are unexpected and need explanation

- The experimental methodology raises several concerns. The authors only examine cases with large rank values, despite evidence showing that LoRA typically performs best with r=8~32 during Supervised Fine-Tuning (SFT). Additionally, the single-epoch training approach may result in underfitting, potentially skewing the results. The effectiveness of the Sparsely Coded Residual method also remains uncertain, as there's insufficient evidence to show whether the improvements come from this specific component or could be achieved simply through the combination of random SVD and GaLore.

- Another  limitation of this work is the insufficient comparison with relevant baselines. Given that the paper's main contribution centers on accelerating PEFT training, it should include comparisons with other efficient methods like DoRA and FFT LoRA that achieve similar goals with reduced parameter counts and faster training speeds. 

[1] DoRA: Weight-Decomposed Low-Rank Adaptation

[2] Parameter-Efficient Fine-Tuning with Discrete Fourier Transform

questions:
To strengthen the paper, the authors should expand their experimental evaluation to include various rank settings, extended training epochs, comprehensive ablation studies, and detailed comparisons with modern PEFT methods. This would provide a more complete and convincing demonstration of their method's advantages.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731294936593

thread_type:
official_review

<thread_object>
creator: reviewer_hiny

<content>
summary:
This paper introduces GaLore+, an improved version of the GaLore method for memory-efficient fine-tuning of large language models. GaLore+ tackles the time bottleneck in GaLore by utilizing cross-head low-rank projections and randomized subspace iteration for faster singular value decomposition. Additionally, it introduces sparsely coded residuals to minimize errors from low-rank approximation. Experiments on arithmetic reasoning (GSM8k, d MAWPS datasets) and natural language generation (E2E) tasks show that GaLore+ achieves significantly faster fine-tuning (around 4x) while maintaining superior performance compared to the original GaLore.

strengths:
1) the analysis (Figure 2) nicely identifies the issue and motivates the paper
2) the proposed method is sound and the results seems convincing.

weaknesses:
1) the evaluation datasets in experiment study are limited. Eval metrics for NLG(E2E) are not suitable for generative task evaluation. 
2) The sensitivity of Hparams (rank, low data regime) is not well studied.

questions:
1) Given LoRA's established effectiveness in low-data fine-tuning scenarios, how does the proposed method perform under similar low-rank, data-constrained conditions?
2) Do the reported performance gains, measured using LLM-based evaluation metrics on NLG datasets, translate to demonstrable improvements in generated output quality?
3) To further assess the method's capabilities, consider expanding the evaluation to include a wider range of generative tasks and datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732364740921

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all the reviewers for their valuable feedback to improve the quality of our manuscript. We have updated the manuscript with the revised contents highlighted in blue. The main changes are summarized as below.

**Low-rank projections in GaLore $+$**

- During the rebuttal period, we found the mistake of Figure 2b. We previously mistakenly calculated separate projection matrices for each attention head and used these matrices to compute the approximation error for each head. Thus, the approximation error is lower than that of exact SVD. We have addressed this mistake and revised the manuscript. Moreover, we have carefully checked our codes for the main experimental results and there were no mistakes.

- We have revised the descriptions on the low-rank projection to improve the presentation.

**Extended experiments**

- We have conducted additional experiments with low-rank settings ($r=32$) to investigate the effectiveness of GaLore $+$ under low-rank conditions. 
- We have included two additional natural language generation (NLG) datasets (*i.e.*, CNN/DM and XSum), two foundation models (*i.e.*, LLaMA2-13B, LLaMA3-8B), and a state-of-the-art PEFT method (*i.e.*, DoRA) as the baseline, in evaluation. 
- We have included an ablation study on the sparsely coded residual with multiple random seeds for convincing results.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The work aims to improve Low-Rank Adaptation for LLMs by using cross-head low-rank projection, and a more efficient alternative to SVD.
Unfortunately, several concerns remained about the paper. First, on the statistical significance and rigor of the experimental results, including only relatively high ranks instead of lower ranks often used in practice, performance improvements often fell within one standard deviations, and there was unclarity if the sparsely coded residual component actually contributed meaningful improvements. Second, we found it inadequately explained how GaLore+ could outperform vanilla GaLore in the cases without sparsely coded residuals, where the methods would then coincide.

We hope the detailed feedback helps to strengthen the paper for a future occasion.