RandLoRA: Full rank parameter-efficient fine-tuning of large models
Authors: Paul Albert
Frederic Z. Zhang
Hemanth Saratchandran
Cristian Rodriguez-Opazo
Anton van den Hengel
Ehsan Abbasnejad
Venue: ICLR 2025 Poster (2025)
Keywords: parameter-efficient; finetuning; CLIP; vision-language; full rank;
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a full rank alternative to LoRA to finetune large models

Abstract: Low-Rank Adaptation (LoRA) and its variants have shown impressive results in reducing the number of trainable parameters and memory requirements of large transformer networks while maintaining fine-tuning performance. The low-rank nature of the weight update inherently limits the representation power of fine-tuned models, however, thus potentially compromising performance on complex tasks. This raises a critical question: when a performance gap between LoRA and standard fine-tuning is observed, is it due to the reduced number of train-
able parameters or the rank deficiency? This paper aims to answer this question by introducing RandLoRA, a parameter-efficient method that performs full-rank updates using a learned linear combinations of low-rank, non-trainable random matrices. Our method limits the number of trainable parameters by restricting optimization to diagonal scaling matrices applied to the fixed random matrices. This allows us to effectively overcome the low-rank limitations while maintaining parameter and memory efficiency during training. Through extensive experimen-
tation across vision, language, and vision-language benchmarks, we systematically evaluate the limitations of LoRA and existing random basis methods. Our findings reveal that full-rank updates are beneficial across vision and language tasks individually, and even more so for vision-language tasks, where RandLoRA significantly reduces—and sometimes eliminates—the performance gap between standard fine-tuning and LoRA, demonstrating its efficacy.

Published as a conference paper at ICLR 2025
RANDLORA: FULL-RANK PARAMETER-EFFICIEN FINE- TUNING OF LARGE MODELS
Paul Albert Frederic Z. Zhang
Hemanth Saratchandran
Cristian Rodriguez-Opazo Anton van den Hengel
Ehsan Abbasnejad
Australian Institute for Machine Learning The University of Adelaide
[firstname. htt.://mmme.aattttttaaaaaaaeaaee edu.
ABSTRACT
Low-Rank Adaptation (LoRA) and its variants have shown impressive results in reducing the number of trainable parameters and memory requirements of large transformer networks while maintaining fine-tuning performance. The low-rank nature of the weight update inherently limits the representation power fine- tuned models, however, thus potentially compromising performance complex tasks. This raises critical question: when a performance gap between LoRA and standard fine-tuning is observed, is it due to the reduced number of train- able parameters or the rank deficiency? This paper aims to answer this question by introducing RandLoRA, parameter-efficient method that performs full-rank updates using a learned linear combinations of low-rank, non-trainable random matrices. Our method limits the number of trainable parameters by restricting optimization to diagonal scaling matrices applied to the fixed random matrices. This allows us effectively overcome the low-rank limitations while maintaining parameter and memory efficiency during training. Through extensive experimen- tation across vision, language, and vision-language benchmarks, we systemati- cally evaluate the limitations of LoRA and existing random basis methods. Our findings reveal that full-rank updates are beneficial across vision and language tasks individually, and even more so for vision-language tasks, where RandLoRA significantly reduces and sometimes eliminates the performance gap between standard fine-tuning and LoRA, demonstrating its efficacy.
INTRODUCTION
Large pre-trained models that leverage broad data have demonstrated significantly improved gen- eralization capabilities and remarkable versatility across diverse tasks. However, the resultant high parameter count also leads to a significant increase in the computational resources required to fine- tune such models downstream tasks. To tackle this issue, parameter-efficient fine-tuning (PEFT) approaches such as low-rank adaptation (LoRA) (Hu et al. 2022), draw inspiration from the low intrinsic dimensionality of pre-trained models (Li al. 2018 Aghajanyan al., (222)) and char- acterize the weight updates as the product of two low-rank matrices, substantially reducing the number of trainable parameters and memory requirements during training. This formulation leads to an adaptable number of trainable parameters, as one modifies the rank of the matrices, providing great flexibility under various resource constraints.
In spite of the strong performance of LoRAs in parameter-efficcent settings, our investigation un- covers an accuracy plateau, wherein an increase of rank and thus learnable parameters fail to bridge the accuracy gap with standard fine-tuning. These undesirable scaling properties (Kopiczko et al.) 2024) raise questions about the inherent limitations imposed by the low-rank structure, particularly when tackling complex tasks that benefit from larger parameter counts. This issue would ideally be addressed by introducing full-rank updates while maintaining the parameter-efficienny To this end, propose RandLoRA, PEFT method that leverages set of iinearl-independee in the form of non- -o--rriiablle low-rank matrices. By solely learning scaling coefficients for the linear combination of the random low-rank bases, method achieves full-rank updates, while maintain-
random bases

---

Published as a conference paper at ICLR 2025
911
86
Avg. Accuracy (5)
Avg. Accuracy (%)
85.5
Arg. Accuracy
84
90
RandLoRA
22
RundLoRA
84.6
KandLoKA
LoRA
LoRA
LoRA
80
10
10
10
10
0.4
0.8
Trainalle parameters
Trainable parameters
Trainable parameters
-10
(a) DinoV2
(b) CLIP
(c) LLama3-8B
Figure 1: LoRA becomes limited by the rank of its update. We train D2no'2 and CLIP classify 21 image datasets and LLama3-8F 3BB to solve 8 commonsense reasoning tasks.
ing low memory usage. As a result, RandLoRA strikes a balance between parameter efficiency and full-rank updates, allowing for more flexible and effective fine-tuning.
Through extensive experimentation, we empirically demonstrate the limitations of the low-rank for- mulation in LoRA, particularly on vision-language tasks, and show how RandLoRA can improve performance under similar parameter budget. Figure [ summarizes our findings across pure vi- sion (DinoV2), vision-language (CLIP) and commonsense reasoning (LLama3-8B), where increas- ing LoRA's parameter count has highly diminishing returns. We find that RandLoRA A outperforms LoRA as the parameter budget expands, while remaining parameter efficient thanks to its full-rank update strategy. We conclude our investigation with insightful discussion the distinctive char- acteristics of RandLoRA where our analysis reveals that, in contrast to LoRA, RandLoRA yields activation patterns in deeper layers that closely align with those obtained through full fine-tuning. Furthermore, our visualization of the loss landscape reveals that the local minima reached by Rand- LoRA is often closer that reached by standard fine-tuning, and it always leads to lower loss than LoRA A for an equal parameter count. Additionally, we explore the integration of sparse random bases, where initial findings highlight that sparse bases preserves the performance of RandLoRA. This suggests promising avenues to further reduce memory and computational requirements when training large transformer models, without compromising model performance.
Our contributions are summarized as:
1. We investigate the interplay between rank and number of trainable parameters when fine- tuning large pre-trained models, highlighting the limitations of LoRA in improving perfor- mance when larger ranks are required.
2. We propose RandLoRA, novel parameter- efficcient fine-tuning (PEFT) strategy based on random basis combinations, enabling full-rank updates without memory overhead over LoRA.
3. We rigorously assess RandLoRA across diverse pre-trained architectures and tasks, span- ning pure vision and vision-language image classification to commonsense reasoning, demonstrating its versatility and effectiveness.
RELATED WORK
2.1 LOw RANK ADAPTATION OF LARGE MODELS
Low Rank Adaptation (LoRA) of large language models has revolutionized the fine-tuning paradigm, enabling memory-constrained adaptation to specialist tasks and democratizing access to larger models. Initially introduced by (Hu et al. 2022). LoRA leverages the observation that weight updates during mie-tuning converge to suitable performances without necessitating full rank updates. By factorizing weight updates into the product two low rank matrices, LoRA achieves memory-efficient solution for adapting large models. Moreover, once the low rank matrices
2

---

Published as a conference paper at ICLR 2025
merged into the original weight matrix size, latency is present during inference. Several improve- ments have been proposed to build upon LoRA's success.
LoRAs (DoRA) (Liu et al. 2024) proposes to improve convergence by decomposing LoRA updates into magnitude and direction components. AdaLoRA (Zhang et al. (023) and Autol (Zhang et al. 2024c), utilize specialized metrics meta-learning propose rank-adapted LoRA formulations that dynamically adjust the rank to suit every layer's is need. Other improvements include initialization strategies for the low rank matrices using the truncated SVD of the pre-trained weights and where the whole decom- position is fine-tuned as in Pissa (Meng et al. 2024) or where only the singular value matrix is as in SVFT (Lingam et al. 2024) or LORA-XS (Balazy al., (024). Further improvements are proposed in HydraLoRA (Tian et al. 2024) where the scaling-up matrix of the low rank decomposition is split into multiple ones with a routing layer added to select the contribution of each head. This for- mulation enhances multi-task learning at the cost of losing the merging capabilities of LoRA in the pre-trained weight at test-time. These advancements collectively enhance the efficiency of LoRA, solidifying its position cornerstone of large language model fine-tuning.
2.2
PARAMETER-
DHAMETEE-EEFICIENT EFFICIENT FINE-TUNING (PEFT) USING RANDOM BASES
Recent research has focused on further reducing the trainable parameter count of LoRA, a crucial aspect for low-shot applications where minimizing trainable parameters can prevent overfitting and enhance generalization. A promising direction involves utilizing random bases combinations, where randomly generated matrices are combined using a limited number of trainable parameters to esti- mate a weight update.
PRANC (Nooralinejad et al.) 20233 pioneered the random base strategy by learning weighted averaged of random matrices through propaaarpa..tion. PRANC's solution averages multiple full size weight matrices for each layer, leading to high memory consumption. To address this, the authors generate random bases on the fly during forward and backward passes using fixed seed random number generator, reducing memory usage to that of the largest trained layer in the network at the cost of training latency.
Building upon PRANC, NOLA (Koohpayegani al. 2024) introduces improved algorithm where random bases are estimated as the product of two low-rank random matrices, each weighed using learnable scalar and summed before matrix multiplication. This approach effectively ap- proximates a rank LoRA with significantly fewer trainable parameters and largely reduces memory consumption during training over PRANC.
Concurretty,, VeRA (Kopiczko et al. 2024) proposed an alternative strategy utilizing a single high- rank random matrix (typically 256 or 1024), instead of summing multiple rank I matrices as in NoLA, VeRA also employs a scaling strategy of random bases distinct from NoLA, detailed in section 4 which relates to our approach. Both NOLA and VeRA achieve comparable performance to LoRA in few-shot fine-tuning scenarios while training substantially fewer parameters.
2.3 ALTERNATIVE STRATEGIES FOR PARAMETER-EFFICIES FNNE-TNNING
We report here alternatives to weight tuning for parameter-efficient adaptation, specifically fo- cusing on prompt tuning. Context Optimization (CoOP) (Zhou et al., 2022b) introduced Iearnable context vectors for CLIP class names, later generalized to instance-specific prompts in Conditional CoOP (CoCoOP) (Zhou et al.] 2022a). Recent prompt tuning methods, like DePT (Zhang et al., 2024b) and PromptSRC (Khattak et al., 2023b), emphasize knowledge preservation by isolating shared subspaces or regularizing prompts. While parameter-effiiient prompt tuning can struggle with generalization beyond few-shot settings (Han al. (2024) and may less effective than LoRA as data increases (Zanella & Ben Ayed] (024)) We therefore consider prompt tuning orthogonal to weight-tuning for the scope of this paper and exclude from direct RandLoRA comparisons except for early results found in Appendix B.3
MOTIVATIONS
Our literature review reveals that research on improving LoRA is focused on reducing the number of trainable parameters further, either through adaptable ranks or by using fixed shared low rank
3

---

Published as a conference paper at ICLR 2025
projection matrices. When looking at moderate to larger parameter budgets however LoRA remains highly competitive.
We identify that early research has convincingly demonstrated the promise of random basis combi- nations as a parameter-efficient strategy for large models, particularly in few-shot scenarios. Two approaches have emerged, each representing distinet paradigm. VeRA advocates for unique ran- dom base with large rank, while NoLA proposes to average a large number of random bases with small ranks. Both approaches report performance comparable LoRA in few-shot scenarios while converging on a significantly reduced number of trainable parameters. However, as we will demon- strate, this reduction comes at the cost of limited performance when venturing beyond few-shot learning, limiting the scalability of these algorithms.
Finally, we report that LoRA predicated on the assumption that low-rank updates suffice for fine- tuning large models. We aim in this paper to question the universality of this hypothesis, exploring scenarios where full rank alternatives may be necessary. The fundamental question follows: is parameter efficiency achieved through low-rank approximation limited by (I) the low-rank nature of the update or (2) by the low parameter count. Can parameter-efficient full rank updates provide more accurate solution This paper aims to address these questions, exploring the balance between parameter efficiency and low-rank fine-tuning of large transformer models, and shedding light on the limitations of existing approaches.
4 RANDLORA PARAMETER EFFICIEET FINE-TUNING WITH FULL RANK
4.1 WEIGHT UPDATES AS A SUM OF LOW-RANK MATRICES
Let Wo E Dxd be a weight matrix of a large pre-trained model. Fine-tuning aims to find an appropriate AW Dxd, such that the fine-tuned weights Wo + AW lead to an adapted model, tailored to a specific downstream task. Without loss of generality, let us assume d < D. The motivation behind RandLoRA stems from the singular value decomposition (SVD) of AW, i.e., AW = UEVT, where U E R Dxd Σ E Rddd V E Rdxd This decomposition can be written as the sum of the product of rank-one matrices, follows
AW
u.v.
where u, and v, denote the columns of U and V, respectively. We suggest that in this context, low- rank updates such as LoRAs can be characterized as approximation of the few largest singular values while the rest of the information in AW being discarded. To better illustrate this point, let us denote the rank of LoRA by and for brevity of exposition, assume d is divisible by r. We rewrite equation] T as a sum of the product of rank-r matrices, as follows
AW Σ U,E,V j=1
where - Zierj u. and where d/r. This formulation reveals how LoRA mod- els the approximates the first low-rank partition U,E, VTT and implicitly assumes Σド= V x O. We however argue that the remaining n - 1 terms can play a crucial role when capturing more complex task- task-specific variations that require larger deviations from the pre-trained weight Wy.
4.2 PARAMETER EFFICIENT APPROXIMATION OF LOW-RANK MATRICES
Approximating more terms in the decomposition of AW using LoRA's formulation quickly be- comes parameter inefficient, culminating to Dd+ d2 parameters for a full rank d in place of the orig- inal Dd parameters of AW. To perform full-rank updates while maintaining parameter-efficienccy. propose instead to approximate each term of AW in equation 2 using low-rank random bases where only scaling coefficients are learned,
AW = Σ BAA
(3)
F=1
4

---

Published as a conference paper at ICLR 2025
where Bj E R ?xr and A, E Rexd non-trainable, random matrices. The two learnable diagonal scaling matrices, A, IR' and r Id are unique each of the terms and fulfill com- plementary roles to improve the approximation. We aim for A,I' transform the input features into an low-dimensional space (rank-r), A, to scale the compressed features which are then transformed back into the desired output space by B Since J operates on the column space of Aj and is unique each A,. expressivity but reducing memory consumption. With a shared A, we formulate the update as
unique shared matrix A E Ir across all terms without loss of
AW E BAA
To achieve full-rank update, set d/r. leading "d +))+ / d learnable param- eters. Note that unlike LoRA, the number of learnable parameters is inversely proportional to the rank the random bases in RandLoRA, increasing the rank of the bases leads reduction in trainable parameters while maintaining full rank. In summary, RandLoRA trades-off approximation accuracy for scope, sacrificing more precise representation of the individual SVD elements of A to capture a larger portion of its singular value decomposition.
4.3 CONVERGENCE ANALYSIS
In this section, we present a theorem showing that weight updates using RandLoRA is an accurate approximation of general matrices under certain theoretical conditions.
Theorem 4.1. Let W be a fixed D x matrix, with D > and rank(W)) d. Fix 1 S n S d, such that d = nr. The matrix W can be factorized using SVD
= -=E
(5)
where U, Rxr, V₁ E Rexxd are partitions of the left and right singular vectors, and E.j Rixr contains singular values. For each 1 Sj S. let B, denote random D X matrix whose entries are drawn ii.d from either a Gaussian or uniform distribution, A₁ denotes an r x d matrix whose entries are drawn similarly, Aj is diagonal x matrix and I diagonal d x d matrix drawn similarly. Assume
(6) for each 1 jj S n for some 0 < Then we have that with probability 1 that each ByA has full rank and
[ - B Se
W Σ
ByAJA,T,
Sn.
For details the proof of theorem 4.1 please refer to appendix D.I
Theorem 4.1 is premised on AyA, being a good approximation for the r-truncated singular value of AW. which is shown to be true empirically in VeRA (Kopiczko et al.) (024) for example. We show in this case that AW can be accurately approximated as Σー A, motivating BandLoRA' formulation. In contrast, since the best approximation rank-r LoRA can achieve the r-truncated SVD of W, then by Eckart- Young-Mirsky theorem, the Frobenius norm of the difference between W and low-rank adaptation BA is lower bounded as follows
| - BAI 2
W
Σ 1=1
Σ
of
(8)
F 1=++
We conclude that while LoRA's rank approximation is limited by the sum of the last d - squared singular values of W, RandLoRA does not present this low bound and is only limited by how close (e) can B,A A,T, approximate length-r segments of the SVD of W.
'The formulation of our method is similar to that of VeRA Kopiczko et al. 2024), which will be discussed
in detail in section 6.5
5

---

Published as a conference paper at ICLR 2025
960
80
Avg. Accuracy (%)
86
78
Arg. Accuracy
Avg. Accuracy
76
74
72
82
70
66
RandLoRAA
RandLoRA8
RanILaRRA (18.000)
NoLA (4.2G)
78
(21.2)
(20.2))
VeRA (4.16)
VeRA (21.70)
256 (20 (0)
LoRA32 (4.3G)
LoRA32 (21.8))
54
LoRA32 (20.2G)
FT (4.903
FT (24.903)
FT (22.10)
60
74
50
2
16
50% 100%
2
16
50% 100%
2
50% 100%
Shots
Shots
Shots
(a) ViT-B/32
(b) VIT-L/14
(c) D2
Figure 2: Tuning CLIP and D2 vision encoders for image classification. Accuracy averaged over 21 datasets. We additionally report max GPU VRAM usage during training.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
We conduct a comprehensive comparison with three state-of-the-art approaches: LoRA (Hu et al. 2022). NoLA IKoohpayegani et al. 2024), and VeRA (Kopiczko et al. 2024). We perform hyper- parameter search to identify optimal settings for LoRÁ, NoLA, VeRA, and RandLoRA to ensure a fair comparison. More details about the experimental settings be found in appendix C Addi- tional experiments on the General Language Understanding Evaluation (GLUE) (Wang et al. (019)) and End-to-end (E2E) Novikova al. (2017) natural language generation benchmarks well as further comparison with prompt-tuning algorithms are available in appendix B
5.2 VISION: DINOV2 AND CLIP'S VISION BACKBONE
We evaluate tnn-ttunig vision backbones for image classification using pre pre-trained ViT-B/14 Di- no (Oquab et al. 2023) and ViT-B/32, ViT-L/14 CLIP (Radford et al. (021) vision only back- bones. We fine-tune on datasets (Appendix C. Table and evaluate [1. 2, 4, 16)-shot learning and performance with 50% and 100% training data.
We compare RandLoRA To LoRA rank 32 where RandLoRA's rank is adjusted to match LoRA's parameters, and include VeRA and NoLA as random base alternatives. We fine-tune the vision backbones and learn linear classifiers for DinoV2, or use frozen CLIP language embeddings for classification. Results displayed in Figure 2 where also report VRAM usage, detailed results are available in Appendix E.2
We find that LoRA exhibits smaller accuracy gap with standard fine-tuning (FT) on D2noV2 than CLIP. With equal parameters, Randl improves over LoRA, bridging the FT gap in both cases. We believe that LoRA's success the Dino backbone is partly explained by its training objective (see Section 6.1). RandLoRA demonstrates LoRA's rank limitation for CLIP architectures and the benefit of full-rank updates in matching FT performance. VeRA and NoLA are efficient in few-shot settings but become limited with more data.
5.3 VISION-LANGUAGE: CLIP
We extend in this section our experimental setting to fine-tuning CLIP-like transformer architec- tures on classification datasets where contrary to section 5.2 both the language and vision encoders of CLIP are trained. We add ImageNet (Krizhevsky et al. (0122) 2) to the dataset pool to scale up to 22 classification datasets. To assess the effectiveness of RandLoRA compared to LoRA models of varying sizes, we consider three variants of pre-trained CLIPs from the open-clip repository (Cherti)
et al.
2023): VIT-B/32 (151M parameters), VIT-L/144 (428M parameters) and ViT-H/14 (IB pa-

---

Published as a conference paper at ICLR 2025
84
90
80
Avg. Accunicy (36)
Avg. Accuricy (3)
Avg. Accuncy (%)
76
86
72
62
68
RandLoRA6 (6.6G)
RanLLoRAS (21.7)
82
RandLoRA (38.2G)
NoLA (6.8G)
(23.())
NoLA
VeRA 256 (6.8G)
VeRA (23.
VeRA 1024 (39.5G)
LoRA32 (6.83)
LoRA32 (23.20)
LoRA32 (39.7G)
FT (8.5G)
FT (27.8G)
78
FT (57.5))
74
2
50% 100%
16 Shots
50% 100%
2
100%
Shots
Shots
(a) ViT-B/32
(b) VIT-L/14
(c) ViT-H/14
Figure 3: Tuning CLIP's vision and language encoders for image classification. Accuracy averaged over 22 datasets. We additionally report max GPU VRAM usage during training.
rameters). We scale the rank of the random bases in RandLoRA in the same way section 5.2 to maintain a number parameters comparable to a rank 32 LoRA: GandLoRA-(6.8,100 for ViT- (B/32,L/14,H/144 respectively.
summary of results available in Figure D with detailed results being available appendix E. Because fine-tuning vision-language architectures such CLIP is harder optimization problem, we observe the existence of a larger performance gap between full fine-tuning and LoRA than for pure vision, which confirm is not bridged by increasing the rank of LoRA (see Figure D. This suggests that increasing parameter count is not enough, pointing towards the rank of the update as the possible limit the performance of LoRA. When running RandLoRA with the same amount of trainable parameters, we observe that the gap with fine-tuning is bridged. When compared with
NoLA and VeRA we come to the same conclusions as section [5.2 although VeRA is this time much more competitive for larger data budgets, hinting towards the importance of high ranks for finetun- ing CLIP-like vision language architectures. We also report that our base sharing strategy allows Randl A to decrease VRAM usage over LoRA which can be relevant for large architectures such as ViT-H/14.
5.4 COMMONSENSE REASONING
We evaluate RandLoRA for fine-tuning LLMs on eight commonsense reasoning tasks (see Ap- pendix C.4). We fine-tune Qwen2 (0.5B), Phi3 (3B), and Llama3 (8B) models and assess data effi ciency by training on both a 170,000-sample full dataset and a 15,000-sample subset, following Hu al. (023)).
Table I compares RandLoRA to LoRA, VeRA, and NoLA, We test two LoRA ranks: rank-16 ("Ef- ficient") and rank-32 ("Performant"). We then scale RandLoRA the same lower amount of pa- rameters to ensure fair comparison. Detailed results are found in Appendix 15
RandLoRA performs competitively with, and sometimes surpasses, LoRA. Phi3's strong zero-shot abilities enable VeRA and NoLA to achieve strong results despite fewer parameters. Conversely, Qwen2 and I.aaa3 require more adaptation, challenging VeRA and NoLA to match LoRA's perfor- mance. The 15k-sample regime can lead to overfitting when scaling trainable parameters for LoRA and Ran decreasing performance even with dropout regularization. When training on the full 170k samples, RandLoRA consistently outperforms LoRA. Results comparing with DoRA (Liu) jet al. 2024) for LLama3 only are available in Table 6 in the appendix where Lan outper- forms both DoRA and LoRA for larger parameter budgets, while DoRA and LoRA competitive at "Efficient" budgets. We conclude RandLoRA is a compelling alternative to LoRA and DoRA for L.LM fine-tuning, especially with larger datasets and parameter budgets.
7

---

Published as a conference paper at ICLR 2025
Table Parameter-efficient fine-tuning of Large Language Models (LLMs). Results averaged commonsense reasoning tasks. We bold the best accuracy between parameeer-equuivalent RandLoRA and LoRA configurations.
LoRA
RandLoRA
Network
Size
ZeroShot
NoL.
VeRA
Efficient
Performant
Efficient
Performant
15k
5.2
42.6
48.1
53.2
52.3
53.5
52.9
Qwen2-0.5b
170k
5.2
47.4
51.8
57.4
57.3
57.7
57.9
15k
65.4
80.4
78.6
81.8
80.3
81.7
82.3
Phi3-3b
170k
65.4
82.3
81.4
84.6
85.0
84.7
85.2
15k
27.0
76.9
77.1
82.7
83.1
81.0
81.3
LLama3-8b
170k
27.0
81.2
81.7
84.4
85.2
84.6
85.6
Figure 4: How close do RandLoRA and LoRA get to standard fine-tuning We compare CKA scores of RandLoRA and LoRA with fine-tuned activations (top) and the mode connectivity in the loss landscape of UCF101 (bottom)
1.0
1.0
0.9
0.8
<0.8
0.7
0.6
LoRA
RandLoRA
0.6
RandLoRA
0.5
10
20
30
10
20
30
Layers
Layers
(a) CKA with fine- tuning CLIP
(c) CKA with fine-tuning Dino V2
RandLoRA
0.92
RandLoRA
0.9
0.005
0.88
0.004
0.003
0.000
0.78
LoRA
DORA
0.76
0.74
(b) Loss landscape CLIP
(d) Loss landscape Di
DISCUSSION
6.1 SIMILARITIES WITH FINE-TUNING: ACTIVATIONS
We evaluate activation similarity to assess LoRA and RandLoRA's ability to mimic fine-tuned model activations. Using the Centered Kernel Alignment (CKA) (Komnblih et al. 2019) metric, we mea- sure the similarity between activations of LoRA, RandLoRA, and a fully fine-tuned model. This protocol assesses how well each method captures dataset-specific activation patterns. Figure 4a shows CKA scores for self-attention and MLP layers in CLIP and DinoV2 vision backbones. av- eraged over datasets where RandLoRA imrpoves over LoRA. For CLIP, LoRA's CKA decreases in deeper layers, losing alignment with fine-tuned activations. RandLoRA, with equal parameters, matches LoRA's early layer alignment but improves upon it in deeper layers. This CKA drop for LoRA in deeper layers is absent in V2, explaining LoRA's near-identical accuracy to fine- tuning on Dino V2. This difference likely arises from training objectives: DinoV2's visual objective creates classification-rean features needing minimal weight adjustments, thus low-rank LoRA A suf-
8

---

Published as a conference paper at ICLR 2025
Table Fine-tuning CLIP or LLama3 using Rand- LoRA different random distributions or base sparsity.
Table 2: Ablation the rank of the up-
dates. The same amount of trainable pa- rameters used in all methods.
Model
Sparsity
Accuracy
uniform
0%
85.98
CLIP-ViT-B/32 normal
0%
85.61
Method
Rank
Accuracy
CLIP-ViT-B/32
binary
0%
85.52
LoRA
32
83.74
ELIP-ViT-B/32
66%
85,43
RandLoRRA-
32
83.62
CLIP-VIT B/32
93%
85.57
RandLoRAAb
384
85.32
CLIP-VIT
98%
84.35
RandLoRA-6
768
85.98
CLIP-VIT-B/32
99%
83.34
LLama3-8h
0%
85.59
LLama3-8b
66%
85,42
fices. CLIP's multimodal objective, however, demands higher ranks for effective adaptation vision tasks.
6.2 SIMILARITIES WITH FINE-TUNING: LOSS LANDSCAPE
We analyze loss landscape connectivity for models fine-tuned with standard fine-tuning, LoRA, and RandLoRA. We visualize 2D loss landscape plane by positioning LoRA, RandLoRA, and fine-tuning models at (0,0), (1,0), and (0.5,1) respectively. For each point (x, on this plane, we interpolate model weights by solving for coefficients or (where Σ 0g 1) and evaluate the interpolated model's loss on 5% training subset. Figure 4b shows that for CLIP, RandLoRA reaches deeper loss minima than LoRA, often with a low-loss path to the fine-tuning optimum, and despite training the same parameter count. For V2, all optima reside in shared low-loss basin, with LoRA already close to fine-tuning, reflecting LoRA's strong performance on this task. These visualizations reinforce LoRA's low rank particularly limiting for complex tasks, and demonstrate RandLoRA's 's ability to achieve deeper minima than LoRA with equal parameters due to full-rank updates. Appendix A provides 3D visualizations for additional datasets.
6.3 FURTHER STUDIES ON FULL VS LOW RANK FINE-TUNING OF CLIP
We investigate whether RandLoRA's CLIP performance advantage over LoRA stems from better SVD approximation or its full-rank capability. We ablate RandLoRA with two rank-controlled variants. Ran-L restricts the update rank to r by averaging bases before multiplication:
AW B.A, (ΣN A.D.). RandLoRA-b uses half-rank updates by setting N rank (an(/AW/rr2 and adjusting base rank to maintain parameter count parity with RandLoRRAF- accuracy on 100%
All variants train the same parameters, only update rank varies. of 22 datasets for CLIP ViT-B/32. Results show that higher update rank correlates with better performance, given equal parameter counts. This supports the importance of large rank updates, particularly for CLIP fine-tuning.
6.4 SPARSE RANDOM MATRICES
We propose to investigate using sparse random matrices for improved memory and computational efficiency, drawing inspiration from random projection literature and the Cohnson-Lindenstre lemma (Lindenstrauss & Johnson 1984). We adopt the sparse construction from Bingham & Man- nila 2001 and Li et al. (2006), where matrix elements are {-1, 0, 1} with probabilities
5, }
(s E 22, DD for W E R Dxd), followed by normalization. Appendix (.6 why this formu- lation preserves full rank. Table Sshows experimental results using these sparse bases in RandLoRA. We explore sparsity ratios s {2, 6, vD, 100, 200), achieving sparsity levels from 66 to 99%. Con- sistent with Li et al. (2006). the recommended sparsity levels (vD) yield performance comparable dense matrices, theoretically reducing memory and compute. However, higher sparsity can de-

---

Published as a conference paper at ICLR 2025
grade accuracy, suggesting potential for optimized RandLoRA variants using compute-optimine sparse random bases.
6.5 SUMMARY DIFFERENCES WITH RELATED RANDOM BASES ALGORITHMS
Prior work like VeRA (Kopickko et al. 2024) and NoLA Koohpayegami et al. 2024) utilizes random bases for parameter-effiient fine-tuning. However, unlike VeRA and NoI.A which approximate a low-rank LoRA update, RandLoRA aims to approximate the full-rank weight update. It could be argued that VeRA approximates only the first block in a decomposition of W, whereas RandLoRA approximates all blocks. Thus, while VeRA and NoLA improve parameter-cfficieney while main- taining low-rank updates, RandLoRA addresses cases requiring full-rank updates. Furthermore, Equation equation Settrionooeiideeeess the flexibilitty in RandLoRA's ' parameter count, ranging from VeRA's parameter efficiency (r = rank(W)) to full fine-tuning parameters (r = 1) while maintaining full- rank.
6.6 LIMITATIONS
Despite Randl LoRA's 's effectiveness, we identify three key limitations for future research.
First, RandLoRA introduces computational overhead in weight update calculations, increasing train- ing time for larger models (Appendix C.6.I). We however evidence room for improvement using ternary sparse bases in Section 6.4 Future work should explore matmul-free matrix combinations using these ternary sparse bases. Efficient implementations could replace costly matrix products with simple aggregations, eliminating floating- point arithmetic (Li et al.) 2006). and accelerating RaA training time pending the development of optimized CUDA kernels (Zhu et al. (024).
Second, exploring non-random, optimal bases B, and A could improve convergence and efficiency by further reducing in equation equation [6] Discovering such bases, potentially through experi- ments decomposition of pre-trained weights (Balazy al. 2024 Meng al., 2024), is promis- ing research direction to enhance RandLoRA.
Third, hybrid approaches combining LoRA and RandLoRA warrant investigation. LoRA could estimate the dominant SVD components of W, while Randl captures the remaining spectral information efficiently. Despite challenges in harmonizing training objectives, starting point would use RandLoRA to refine a LoRA when convergence is insufficient. Addressing these limitations will further improve RandLoRA's potential for efficient full-rank fine-tuning.
7 CONCLUSION
This paper introduces RandLoRA, a method achieving parameter efficiency and low memory cost while enabling full rank model updates. Our findings underscore the critical importance full- rank updates when fine-tuning pre-trained architectures and observe that our approach surpasses LoRA's performance for an equal parameter count, highlighting the value of full-rank updates in large model fine-tuning. Through extensive experiments across diverse tasks demonstrated the efficacy of our method. While RandLoRA incurs additional computational overhead due to random basis multiplications, memory consumption remains contained and provide venues for reducing this compute in practice. As a results, RandLoRA offers a viable alternative to LoRA for fine-tuning large pre-trained models on consumer-grade grade hardware. Our results have significant implications for efficient and effective model adaptation, prompting for future research in scalable and versatile full-rank fine-tuning techniques.
ACKNOWLEDGMENTS
This research is funded in part by the Australian Government through the Australian Research Coun- cil (Project DP240103278), and the Centre of Augmented Reasoning at the Australian Institute for Machine Learning, established by grant from the Department of Education. This work is also supported by supercomputing resources provided by the Phoenix HPC service at the University of Adelaide.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing, pp. 7319-7322 Association for Computational Linguistics, Aug 2021.
URL /tttps://ccanttiinn
ottg/////1.0 acl-long 568
Klaudia Balazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adapta- tion with extremely small number parameters. arXiv preprint arXiv:2405. 17604, 2024.
Ella Bingham and Heikki Mannila. Random projection in dimensionality reduction: applications to image and text data. In International Conference Knowledge Discovery and Data mining (ACM SIGKDD), 2001.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on Artificial Intelligence (AAAI), 2020.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman -mage learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Christopher Clark, Kenton Lee, Ming-War Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXiv:1905.10044 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.0545:
2018.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint 240.::2403.14600 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA; Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR), 2022.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.0193: 2023.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: and Pattern Recognition (CVPR), 2023a.
prompt learning. In IEEE/CVF Conference on Computer Vision
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- -ssun Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In /EEE/CVF International Conference on Computer Vision (ICCV), 2023b.
Soroush Abbasi Koohpayegani. KL Navaneet, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsi- avash. NOLA: Compressing LoRA using Linear Combination Random Basis. In International Conference on Learning Representations (ICLR), 2024.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. In International Conference on Learning Representations (ICLR), 2024.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning (ICML), 2019.
A. Krizhevsky, Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (NeurIPS), 2012.

---

Published as a conference paper at ICLR 2025
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the instrinsic di-
mension of objective landscapes. In ICLR, Vancouver, Canada, 30 Apr-3 May 2018. URL pittps:/cpperreee
Ping Li, Trevor J Hastie, and Kenneth Church. Very sparse random projections. In ACM SIGKDD international conference on Knowledge discovery and data mining, 2006.
W Johnson J Lindenstrauss and J Johnson. Extensions of lipschitz maps into hilbert space. Con- temp. Math, 1984.
Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bojchevski, and Sujay Sanghavi. SVFT: Parameter-Effiee Fine-Tuning with Singular Vectors. In International Conference on Machine Learning Workshops (ICMLW), 2024.
Shih- Yang Liu, Chien- Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Interna- tional Conference on Machine Learning (ICML), 2024.
Y Liu, M Ott, N Goyal, J Du, M Joshi, D Chen, O Levy, M Lewis. L Zettlemoyer, and V Stoyanov. RoBERTa: robustly optimized BERT pretraining approach. arXiv: /6692, 2019.
Fanxu Meng. Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: 1809.087899 2018.
Parsa Nooralinejad, Ali Abbasi, Soroush Abbasi Koohpayegani, Kossar Pourahmadi Mcibodi, Rana Muhammad Shahroz. Khan, Soheil Kolouri, and Hamed Pirsiavash. Pranc: Pseudo random net- works for compacting deep models. In IEEE/CVF International Conference Computer Vision (ICCV), 2023.
Jekaterina Novikova, Ondfej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, 2017.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu. Vasu Sharma, Shang-Wer Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nico- las Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqu: Common- sense reasoning about social interactions. arXiv preprint arXiv:/904.097. 2019.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
12

---

Published as a conference paper at ICLR 2025
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-tast benchmark and analysis platform for natural language understanding. In Inter- national Conference Learning Representations (ICLR), 2019.
Maxime Zanella and Ismail Ben Ayed. Low-Rank Few-Shot Adaptation of Vision-Language Mod- els. In JEEE/CVE Conference on Computer Vision and Pultern Recognition (CVPR), 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv: /9t::07035000 2019.
Frederic Z Zhang, Paul Albert, Cristian Rodriguez-Opaze Anton van den Hengel, and Ehsan Ab- basnejad. Knowledge Composition using Task Vectors with Learned Anisotropic Sealing. In Advances in Neural Information Processing Systems (NeurlPS), 2024a.
Ji Zhang, Shihan Wu, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Dept: Decoupled prompt tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024b.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adal.oRA: Adaptive budget allocation for parameter- efficient fine-tuning. International Conference on Learning Representations (ICLR), 2023.
Ruiyi Zhang. Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. AutoLoRA: Automati- cally Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning. arXiv preprint arXiv:2403.091 13, 2024c.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 2022b.
Rui-Jie Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Peng Zhou, and Jason K Eshraghian. Scalable MatMul-free Language Modeling. arXiv preprint arXiv:2406.02558,, 2024.
13


Reviews:
[{"review_id": "XFHnMEyUVE", "reviewer": "Reviewer_uANM", "summary": "This paper introduces RandLoRA, a method designed for efficient parameter tuning of both visual and linguistic models. The researchers discuss the shortcomings of conventional low-rank adaptation techniques, known as LoRA, and highlight the significance of non-critical ranks in the adaptation process. As a result, compared with traditional LoRA, RandLoRA earns better performance with fewer trainable parameters. The convergence of RandLoRA is discussed in detail. Extensive experiments verify its effectiveness on vision and language tasks.", "questions": "Please see weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ RandLoRA is proposed to approximate low-rank updates under a clear motivation about the importance of non-critical ranks.\n+ Multiple scales of models are selected as baselines, and RandLoRA can lead to good improvement in most situations.\n+ The paper is well-written and easy to follow.", "weaknesses": "- The motivation here mainly focus on how to approximate and improve low-rank adaptation methods like LoRA. The conclusion is to use full-rank updates and thus the authors propose RandLoRA. However, RandLoRA also outperforms full fine-tuning in various tasks like image classification. How to explain this experimental result? Why we can earn improvement by approximating low-rank updates to full-rank updates over both LoRA and full fine-tuning?\n- Some important baselines are missing. For example, in the field of tuning CLIP on image classification tasks, many state-of-the-art methods use prompt-based tuning methods, e.g. PromptSRC (ICCV'23)[a], DePT (CVPR'24)[b] instead of LoRA. Such kind of parameter-efficient fine-tuning methods should also be discussed and compared with, given that the most-related works VeRA and LoRA are not initially proposed for image classification tasks.\n\n[a] Self-regulating Prompts: Foundational Model Adaptation without Forgetting, https://arxiv.org/abs/2307.06948)\n[b] DePT: Decoupled Prompt Tuning, https://arxiv.org/abs/2309.07439", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nOUtqvlWaM", "reviewer": "Reviewer_ufJr", "summary": "The paper proposes RandLoRA for parameter-efficient fine-tuning for vision and language models. The authors start from analyzing the drawbacks of traditional low rank adaptation methods(LoRA) and argue that the importance of non-essential ranks during adaptation. RandLoRA shows better performance than existing methods on fine-tuning CLIP models on image classification and fine-tuning LLMs on 8 commonsense reasoning tasks.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The presentation is clear and easy to understand\n- The proposed RandLoRA's convergence has been theoretically proved\n- Various experiments on different tasks and models are done", "weaknesses": "- Limited technical novelty. What is the main difference between VeRA and RandLoRA? There is a fairly similar update formulation in VeRA, e.g. two frozen low-rank matrices and two trainable small matrices.\n- Lack of some important experiments for further verification. Most competitors, e.g. VeRA and LoRA, in the paper are proposed for language models and language tasks. To confirm the superiority of RandLoRA, the authors should directly compare the performance between RandLoRA and former competitors on standard language tasks, e.g. GLUE and E2E used in VeRA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v2NByGa59p", "reviewer": "Reviewer_rgkh", "summary": "This paper proposes RandLoRA, a new method to address the limitations of LoRA in complex tasks. RandLoRA overcomes the low-rank constraint of LoRA by learning a combination of random low-rank basis matrices to achieve full-rank updates, and trend-off a balance between parameter efficiency and model performance. However, the paper needs to be further strengthened in several aspects. Overall, the paper is novel, but there is room for improvement in experimental and theoretical.", "questions": "see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. RandLoRA proposes a full-rank optimization strategy based on random low-rank matrix combinations, which helped the limitations of LoRA in complex tasks, especially the problem that its low-rank matrix cannot fully capture the complexity distribution of the task.\n2. In the case of limited parameters, RandLoRA shows higher performance than LoRA, especially in vision-language tasks, showing that the method has certain parameter efficiency.\n3. The paper provides novel ideas for the fine-tuning of large models, while reducing computing resource consumption and memory usage, while improving the performance of the model on specific tasks.\n4. This paper well analysis the rationale behind the effectiveness of the proposed method.", "weaknesses": "1. The paper's derivation of RandLoRA is based on SVD and random basis matrix combination, but the theoretical rigour is still insufficient. The derivation assumes that the basis matrix obeys a specific random distribution (such as Gaussian or uniform distribution), which is difficult to strictly guarantee in practice. In addition, the combination of random basis matrices may cause stability problems in large-scale training. It is recommended to conduct experiments on models with larger parameter amounts to verify the robustness of the method.\n2. Theorem 4.1 proposed in the paper gives the approximation error bound of RandLoRA, but does not explain in detail how to control the size of the error in practical applications, especially as the model size increases, whether the error will accumulate, which may affect its approximation effect.\n3. The introduction of sparse matrices is intended to reduce computational complexity, but the impact of sparse matrices on the full-rank approximation effect has not been fully demonstrated. Although Table 3 shows the experimental effect of sparse matrices in RandLoRA, the paper does not explore the theoretical impact of sparse matrices in full-rank approximation in depth, and it is recommended to add analysis in this regard.\n4. The comparative experiment of the paper selected LoRA, NoLA, VeRA and other parameter efficient fine-tuning methods, but did not include full parameter fine-tuning as a control. It may not be sufficient to select only LoRA as the main benchmark. It is recommended to supplement the full parameter fine-tuning results to fully evaluate the advantages and disadvantages of RandLoRA.\n5. RandLoRA has relatively small improvements in visual tasks, but its effect in visual-language tasks is significantly enhanced. It may be related to the complexity of the task and the characteristics of multimodal data?\n6. The impact of different configurations of RandLoRA (such as the sparsity of the random basis matrix and the distribution selection of the basis matrix) on the effect deserves further study. It is recommended to add ablation experiments on factors such as the basis matrix generation method and parameter scale to more comprehensively reveal the performance influencing factors of RandLoRA.\n7. Although RandLoRA performs well on small-scale parameter models, its effectiveness in larger-scale models (such as LLaMA 70B and LlaVA 32B) has not been verified. It is recommended to conduct experiments on larger-scale models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XsEN9zbW8u", "reviewer": "Reviewer_ZGK2", "summary": "This paper introduces RandLoRA, a novel method for parameter-efficient fine-tuning (PEFT) of large pre-trained models. By leveraging learned linear combinations of low-rank, non-trainable random matrices, RandLoRA enables full-rank updates, which significantly enhance the adaptability and efficiency of fine-tuning processes. The method strategically limits the number of trainable parameters by optimizing diagonal scaling matrices, which are applied to the fixed random bases, thus maintaining a low parameter count and minimal memory usage during training.", "questions": "(1): Lines 77-80: The paper claims that RandLoRA consistently outperforms LoRA across the same parameter counts. However, based on Figure 1(a) and 1(b), it appears that RandLoRA surpasses LoRA only when LoRA begins to overfit as the parameter count increases. I recommend the authors to qualify their statements to reflect that RandLoRA's superiority emerges prominently under conditions of LoRA’s overfitting.\n(2): In the related work section of this paper, the authors have omitted some significant recent advancements in LoRA modifications. For example: SVFT, HydraLoRA, PISSA, LoRA-XS, FLoRA, etc. The inclusion of these advancements is essential for enriching the research background and understanding the current research progress in this field. While DoRA is mentioned in the related work, it is not compared with RandLoRA in the experimental section. I recommend that the authors consider such comparisons in future work. This would not only enhance the persuasiveness of the paper but also better showcase the advantages and distinct characteristics of RandLoRA among the plethora of methods.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1): The manuscript is well-crafted with a clear and logical progression of ideas.\n(2): Visual aids like figures and tables are effectively used to illustrate key points and compare performance metrics clearly.\n(3): The extensive experiments across various tasks and architectures demonstrate the method's effectiveness and adaptability.", "weaknesses": "(1): Lines 86-89: The phenomenon of performance saturation as the rank of LoRA increases is well-known in the field (This has already been explained in the VeRA paper.). I suggest that this point be rephrased or discussed within the context of known literature to maintain the integrity of the paper.\n(2): While the method is promising in terms of parameter efficiency and memory usage, its practicality is challenged by the substantially increased training times on the Llama3B model. A more thorough investigation into the computational trade-offs and possible optimizations to reduce training times would benefit the study and its broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)