RandLoRA: Full rank parameter-efficient fine-tuning of large models

Authors: Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran, Cristian Rodriguez-Opazo, Anton van den Hengel, Ehsan Abbasnejad
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: parameter-efficient; finetuning; CLIP; vision-language; full rank;

Abstract:
Low-Rank Adaptation (LoRA) and its variants have shown impressive results in reducing the number of trainable parameters and memory requirements of large transformer networks while maintaining fine-tuning performance. The low-rank nature of the weight update inherently limits the representation power of fine-tuned models, however, thus potentially compromising performance on complex tasks. This raises a critical question: when a performance gap between LoRA and standard fine-tuning is observed, is it due to the reduced number of train-
able parameters or the rank deficiency? This paper aims to answer this question by introducing RandLoRA, a parameter-efficient method that performs full-rank updates using a learned linear combinations of low-rank, non-trainable random matrices. Our method limits the number of trainable parameters by restricting optimization to diagonal scaling matrices applied to the fixed random matrices. This allows us to effectively overcome the low-rank limitations while maintaining parameter and memory efficiency during training. Through extensive experimen-
tation across vision, language, and vision-language benchmarks, we systematically evaluate the limitations of LoRA and existing random basis methods. Our findings reveal that full-rank updates are beneficial across vision and language tasks individually, and even more so for vision-language tasks, where RandLoRA significantly reduces—and sometimes eliminates—the performance gap between standard fine-tuning and LoRA, demonstrating its efficacy.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730363853107

thread_type:
official_review

<thread_object>
creator: reviewer_zgk2

<content>
summary:
This paper introduces RandLoRA, a novel method for parameter-efficient fine-tuning (PEFT) of large pre-trained models. By leveraging learned linear combinations of low-rank, non-trainable random matrices, RandLoRA enables full-rank updates, which significantly enhance the adaptability and efficiency of fine-tuning processes. The method strategically limits the number of trainable parameters by optimizing diagonal scaling matrices, which are applied to the fixed random bases, thus maintaining a low parameter count and minimal memory usage during training.

strengths:
(1): The manuscript is well-crafted with a clear and logical progression of ideas.
(2): Visual aids like figures and tables are effectively used to illustrate key points and compare performance metrics clearly.
(3): The extensive experiments across various tasks and architectures demonstrate the method's effectiveness and adaptability.

weaknesses:
(1): Lines 86-89: The phenomenon of performance saturation as the rank of LoRA increases is well-known in the field (This has already been explained in the VeRA paper.). I suggest that this point be rephrased or discussed within the context of known literature to maintain the integrity of the paper.
(2): While the method is promising in terms of parameter efficiency and memory usage, its practicality is challenged by the substantially increased training times on the Llama3B model. A more thorough investigation into the computational trade-offs and possible optimizations to reduce training times would benefit the study and its broader applicability.

questions:
(1): Lines 77-80: The paper claims that RandLoRA consistently outperforms LoRA across the same parameter counts. However, based on Figure 1(a) and 1(b), it appears that RandLoRA surpasses LoRA only when LoRA begins to overfit as the parameter count increases. I recommend the authors to qualify their statements to reflect that RandLoRA's superiority emerges prominently under conditions of LoRA’s overfitting.
(2): In the related work section of this paper, the authors have omitted some significant recent advancements in LoRA modifications. For example: SVFT, HydraLoRA, PISSA, LoRA-XS, FLoRA, etc. The inclusion of these advancements is essential for enriching the research background and understanding the current research progress in this field. While DoRA is mentioned in the related work, it is not compared with RandLoRA in the experimental section. I recommend that the authors consider such comparisons in future work. This would not only enhance the persuasiveness of the paper but also better showcase the advantages and distinct characteristics of RandLoRA among the plethora of methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730608429812

thread_type:
official_review

<thread_object>
creator: reviewer_rgkh

<content>
summary:
This paper proposes RandLoRA, a new method to address the limitations of LoRA in complex tasks. RandLoRA overcomes the low-rank constraint of LoRA by learning a combination of random low-rank basis matrices to achieve full-rank updates, and trend-off a balance between parameter efficiency and model performance. However, the paper needs to be further strengthened in several aspects. Overall, the paper is novel, but there is room for improvement in experimental and theoretical.

strengths:
1. RandLoRA proposes a full-rank optimization strategy based on random low-rank matrix combinations, which helped the limitations of LoRA in complex tasks, especially the problem that its low-rank matrix cannot fully capture the complexity distribution of the task.
2. In the case of limited parameters, RandLoRA shows higher performance than LoRA, especially in vision-language tasks, showing that the method has certain parameter efficiency.
3. The paper provides novel ideas for the fine-tuning of large models, while reducing computing resource consumption and memory usage, while improving the performance of the model on specific tasks.
4. This paper well analysis the rationale behind the effectiveness of the proposed method.

weaknesses:
1. The paper's derivation of RandLoRA is based on SVD and random basis matrix combination, but the theoretical rigour is still insufficient. The derivation assumes that the basis matrix obeys a specific random distribution (such as Gaussian or uniform distribution), which is difficult to strictly guarantee in practice. In addition, the combination of random basis matrices may cause stability problems in large-scale training. It is recommended to conduct experiments on models with larger parameter amounts to verify the robustness of the method.
2. Theorem 4.1 proposed in the paper gives the approximation error bound of RandLoRA, but does not explain in detail how to control the size of the error in practical applications, especially as the model size increases, whether the error will accumulate, which may affect its approximation effect.
3. The introduction of sparse matrices is intended to reduce computational complexity, but the impact of sparse matrices on the full-rank approximation effect has not been fully demonstrated. Although Table 3 shows the experimental effect of sparse matrices in RandLoRA, the paper does not explore the theoretical impact of sparse matrices in full-rank approximation in depth, and it is recommended to add analysis in this regard.
4. The comparative experiment of the paper selected LoRA, NoLA, VeRA and other parameter efficient fine-tuning methods, but did not include full parameter fine-tuning as a control. It may not be sufficient to select only LoRA as the main benchmark. It is recommended to supplement the full parameter fine-tuning results to fully evaluate the advantages and disadvantages of RandLoRA.
5. RandLoRA has relatively small improvements in visual tasks, but its effect in visual-language tasks is significantly enhanced. It may be related to the complexity of the task and the characteristics of multimodal data?
6. The impact of different configurations of RandLoRA (such as the sparsity of the random basis matrix and the distribution selection of the basis matrix) on the effect deserves further study. It is recommended to add ablation experiments on factors such as the basis matrix generation method and parameter scale to more comprehensively reveal the performance influencing factors of RandLoRA.
7. Although RandLoRA performs well on small-scale parameter models, its effectiveness in larger-scale models (such as LLaMA 70B and LlaVA 32B) has not been verified. It is recommended to conduct experiments on larger-scale models.

questions:
see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719210627

thread_type:
official_review

<thread_object>
creator: reviewer_ufjr

<content>
summary:
The paper proposes RandLoRA for parameter-efficient fine-tuning for vision and language models. The authors start from analyzing the drawbacks of traditional low rank adaptation methods(LoRA) and argue that the importance of non-essential ranks during adaptation. RandLoRA shows better performance than existing methods on fine-tuning CLIP models on image classification and fine-tuning LLMs on 8 commonsense reasoning tasks.

strengths:
- The presentation is clear and easy to understand
- The proposed RandLoRA's convergence has been theoretically proved
- Various experiments on different tasks and models are done

weaknesses:
- Limited technical novelty. What is the main difference between VeRA and RandLoRA? There is a fairly similar update formulation in VeRA, e.g. two frozen low-rank matrices and two trainable small matrices.
- Lack of some important experiments for further verification. Most competitors, e.g. VeRA and LoRA, in the paper are proposed for language models and language tasks. To confirm the superiority of RandLoRA, the authors should directly compare the performance between RandLoRA and former competitors on standard language tasks, e.g. GLUE and E2E used in VeRA.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721336836

thread_type:
official_review

<thread_object>
creator: reviewer_uanm

<content>
summary:
This paper introduces RandLoRA, a method designed for efficient parameter tuning of both visual and linguistic models. The researchers discuss the shortcomings of conventional low-rank adaptation techniques, known as LoRA, and highlight the significance of non-critical ranks in the adaptation process. As a result, compared with traditional LoRA, RandLoRA earns better performance with fewer trainable parameters. The convergence of RandLoRA is discussed in detail. Extensive experiments verify its effectiveness on vision and language tasks.

strengths:
+ RandLoRA is proposed to approximate low-rank updates under a clear motivation about the importance of non-critical ranks.
+ Multiple scales of models are selected as baselines, and RandLoRA can lead to good improvement in most situations.
+ The paper is well-written and easy to follow.

weaknesses:
- The motivation here mainly focus on how to approximate and improve low-rank adaptation methods like LoRA. The conclusion is to use full-rank updates and thus the authors propose RandLoRA. However, RandLoRA also outperforms full fine-tuning in various tasks like image classification. How to explain this experimental result? Why we can earn improvement by approximating low-rank updates to full-rank updates over both LoRA and full fine-tuning?
- Some important baselines are missing. For example, in the field of tuning CLIP on image classification tasks, many state-of-the-art methods use prompt-based tuning methods, e.g. PromptSRC (ICCV'23)[a], DePT (CVPR'24)[b] instead of LoRA. Such kind of parameter-efficient fine-tuning methods should also be discussed and compared with, given that the most-related works VeRA and LoRA are not initially proposed for image classification tasks.

[a] Self-regulating Prompts: Foundational Model Adaptation without Forgetting, https://arxiv.org/abs/2307.06948)
[b] DePT: Decoupled Prompt Tuning, https://arxiv.org/abs/2309.07439

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732095588536

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their thoughtful comments and feedback on the limitations they evidenced in our work. We have given exhaustive answers to every questions to elicit an insightful discussion to allow us to address concerns and satisfy the acceptance threshold. We have updated the paper draft with the proposed changes and highlighted them in a blue color to facilitate review. 

A large amount of additional experiments have been requested to further test the limits of RandLoRA or to add further comparisons with related work. We will do our best to report experimental results as they come in and hopefully in time given our academic resources. 

Note that we have performed extensive experiments in our paper across multiple tasks and datasets: vision in section 5.2 and appendix D.2, vision-language in section 5.3 and appendix D.1, and commonsense-reasoning in section 5.3 and appendix D.3; where we ablate both number of trainable parameters of the PEFT methods and network sizes. These tests extensively report on RandLoRA's strengths and weaknesses and we believe give reliable trends toward generalization to larger architectures and other tasks.

In the meantime, please let us know if you require further clarifications or have additional concerns you would like addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732771939065

thread_type:
official_comment

<thread_object>
creator: area_chair_s6jv

<content>
title:
Reviewers, please kindly respond

comment:
Dear Reviewers,

If you have not responded to author's rebuttal, please kindly do so as soon as possible. The deadline is Dec 2, but the authors can potentially further clarify questions if you respond earlier. Thanks!

Best, AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: a new PEFT method that uses parameters to combine random matrices as weight updates; it demonstrates ranks are important, and with minimal parameters, it can achieve competitive performance on various tasks.

Strengths: simple yet effective full-rank PEFT method; extensive experiments on different modalities; clear presentation; outperforms LoRA.

Weaknesses: increased training time; some missing baselines; no large-scale models (>30B).

Reason for decisions: demonstrated the importance of rank; a simple and effective method; all reviewers are leaning positive.