VeRA: Vector-based Random Matrix Adaptation
Authors: Dawid Jan Kopiczko
Tijmen Blankevoort
Yuki M Asano
Venue: ICLR 2024 poster (2024)
Keywords: Parameter-efficient fine-tuning
Transfer learning
Low-rank
NLP
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Low-rank adapation (LoRA) is a popular method that reduces the number of trainable parameters when finetuning large language models, but still faces acute storage challenges when scaling to even larger models or deploying numerous per-user or per-task adapted models. In this work, we present Vector-based Random Matrix Adaptation (VeRA), which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance. It achieves this by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead. We demonstrate its effectiveness on the GLUE and E2E benchmarks, image classification tasks, and show its application in instruction-tuning of 7B and 13B language models. Website: https://dkopi.github.io/vera

Published as a conference paper at ICLR 2024
VERA: VECTOR-BASED RANDOM MATRIX ADAPTATION
Dawid J. Kopiczko*t QUVA Lab University of Amsterdam
Tijmen Blankevoort
Yuki M. Asano
Qualcomm Al Research*
QUVA Lab University of Amsterdam
ABSTRACT
Low-rank adapation (LoRA) a popular method that reduces the number of train- able parameters when finetuning large language models, but still faces acute stor- age challenges when scaling to even larger models or deploying numerous per- user per-task adapted models. In this work, we present Vector-based Random Matrix Adaptation (VeRA which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance. It achieves this by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead. We demonstrate its effectiveness on the GLUE and E2E benchmarks, image classification tasks, and show its application in intructiion-uningg of 7B and 13B language models.
INTRODUCTION
In the era of increasingly large and complex language models, the challenge of efficient adaptation for specific tasks has become more important than ever. While these models provide powerful capabilities, their extensive memory requirements pose significant bottleneck, particularly when adapting them for personalized use. Consider, for example, a cloud-based operating system assistant that continuously learns from and adapts individual store multiple checkpoints of finetuned models for each user rapidly escalates the required storage, even more so when multiple tasks come into play.
behaviors and feedback. The need to
The situation is further exacerbated when we look at the state-of- state-of-the-art models like GPT-4 (OpenAl 2023). Finetuning techniques like LoRA (Hu et al. 2022), while effective, still introduce consider- able memory overhead. As illustrative example, applying LoRA with rank of 16 to the query and value layers GPT-3 (Brown et al.) 2020) would demand at least 288MB of memory, if stored in singe-precision at million finetuned weights, e.g., one per user, that would amount to 275TB.
Given the recent proliferation of language models and their deployment in personalized assistants, edge devices, and similar applications, efficient adaptation methods are paramount. We believe there is untapped potential for even more efficient approaches. Previous work (Aghajanyan et al.) 2021) pointed out the low intrinsic dimensionality of pretrained models' features. These studies reported numbers much lower than the trainable parameters used in LoRA, suggesting there is room for improvement.
In parallel to this, recent research has shown the surprising effectiveness of models utilizing random weights and projections (Peng al. 2021 Ramanujan et al.) 2020 Lu al. 2022 Schrimpf al. 2021| Frankle et al. 2021). Such models serve as the basis of our proposed solution, Vector-based Random Matrix Adaptation (VeRA), which minimizes the number of trainable parameters intro- duced during finetuning by reparametrizing the weights matrices. Specifically, we employ "scaling vectors" to adapt pair of frozen random matrices shared between layers. With this approach, many more versions of the model can reside in the limited memory of a single GPU.
*dj kopi c com
I Datasetts were solely dowaloaded and evaluated by the University of Amsterdam.
"Qualcomm AI Research is an initiative of Qualcomm Technologics, Inc.
Website: httpt:tt/////. // wwt.a/.pppiii.i.bb.oo... github. fo/vera/

---

Published as a conference paper at ICLR 2024
In summary, main contributions follows:
We introduce novel finetuning method with no additional inference time cost. Our method further reduces the number of trainable parameters compared to the state-of-the-art LoRA method, while yielding comparable results.
We compare our approach with LoRA and other parameter-effiiien adaptation methods on the natural language understanding (GLUE) and natural language generation (F2F) bench- marks, and compare against LoRA on cestruciionfollowiin and image classification tasks.
We perform an ablation study to better understand the individual components of our method and their effects on performance.
RELATED WORK
Low-Rank Adaptation (LoRA). LoRA offers an innovative solution to the computational chal- lenges posed by the finetuning of large pretrained language models. Introduced by Hu et al. (2022). the method employs low-rank matrices to approximate the weight changes during finetuning, ef- fectively reducing the number of parameters that need to trained. Among its advantages, LoRA significantly lowers the hardware barrier for finetuning by reducing the need for gradient calcula- tion and optimizer state maintenance for most parameters. It can also work with quantized model weights (Dettmers al., 2023), reducing the requirements even further. Furthermore, LoRA mod- ules are easily swappable, making task-switching efficient and less eesourcce-ntensivve Importantly, and different to adapter-bace based finetuning approaches (Houlsby al. 2019 Lin et al. 2020 Pfeiffer et al. 2021 Rücklé et al. (021). LoRA incurs no additional inference time cost when deployed, as the trainable matrices can be merged with the frozen weights.
Based on this, AdaLoRA (Zhang et al. (023b) extends the LoRA method, introducing dynamic rank adjustment for the low-rank matrices during finetuning. The core idea is to optimally distribute the parameter budget by selectively pruning less important components of the matrices based on an importance metric.
Parameter Efficiency in Existing Methods While methods such LoRA have shown significant improvements in finetuning performance, they still require considerable amount of trainable pa- rameters. According to Aghajanyan et al. (202T), the upper bound for intrinsic dimensions is much smaller than what is typically utilized in such methods. For instance, the dgd for RoBERTapes is reported to be 896, whereas authors of the LoRA paper reported using 0.3M trainable parameters for this model, suggesting that the parameter count could be reduced further.
Although LoRL takes steps in this direction by dynamically allocating parameters to more crit- ical layers, posit that different approach could achieve substantial parameter reduction, while tolerating marginal performance degradation. This sets the stage for the method we introduce in the following section.
Random Models and Projections. The concept of using random matrices and projections for model efficiency is supported by multiple strands of research. Frankle & Carbin (2019) identified that randomly-initializen neural networks contain subnetworks that capable of reaching high per- formance when trained. Meanwhile, Ramanujan et al. (2020) revealed that there exist subnetworks that can achieve impressive results even in the absence of training. Aghajanyan al. (202T) showed that training only small number of parameters, randomly projected back into the full space, could achieve 90% of the pala-pare model performance. |Ruiz al.) (2023) introduced a parameter- efficient finetuning method for personalization of text-to-image models, utilising random frozen matrices inside LoRA. Other works (Lu al., [002]] Schrimpf et al. 2021] Frankle al. 2021) have shown that frozen, randomly initialized models, with small sections finetuned, can perform surprisingly well.
Thh smallest dimension d that provides satisfactory solution, which is 90% of the full training metric, as defined by [i al. (0088)
2

---

Published as a conference paper at ICLR 2024
dum
dim
0
B 0
B . o
shared across layers
Pretrained Weights
Pretrained Weights
W R dxd
W E Buxd
1
shared across layers
A = (0)
A N (0,
X
frozen
dimgn
dimin
trainable
Figure I: Schematic comparison of LoRA (left) and VeRA (right). LoRA updates the weights matrix W by training the low-rank matrices A and B, with intermediate rank In VeRA these matrices frozen, shared across all layers, and adapted with trainable vectors d and b, substantially reducing the number of trainable parameters. In both cases, low-rank matrices and vectors can be merged into original weights matrix W, introducing no additional latency.
Collectively, these works create a compelling case for the utilization of frozen random matrices in finetuning methods, providing both theoretical and an empirical foundation for the approach taken in this paper.
3 METHOD
In this section, we introduce Vector-based Random Matrix Adaptation, a novel parameter-efficient finetuning method that builds upon and extends the state-of-the-art method, LoRA. The central in- novation in VeRA lies in the reparameterivation of the low-rank matrices. Specifically, we freeze single pair of randomly initialized matrices, shared across all adapted layers, and introduce train- able scaling vectors that allow for layer-wise adaptation, as shown in Figure [ Similarly to LoRA, trained scaling vectors along with low-rank matrices can be merged into original weights, eliminat- ing additional inference latency.
3.1
METHOD FORMULATION
LoRA Huu et al. 2022) finetunes a matrix product of two low-rank matrices to adapt large-language models for task. Formally, for pretrained weight matrix Wo Rm*x,, the weight update AW constrained to a low-rank decomposition, as expressed in Equation
h = Wo + AW* Wor + BAr,
(I)
where we undeline the parameters updated via gradient descent. This approximation enables the model to keep the original weight Wo frozen while optimizing only the new low-rank matrices A and B. These matrices are much smaller in size than the original matrix due to their rank-reduced and B has shape x n, where min(m.n) serves as the bottleneck
nature. A has shape dimension. In contrast, our VeRA method is expressed as:
h= Wor + AW. Wax + AgBAaAx
In this approach, B and A are frozen, random, and shared across layers, while the scaling vectors b and are trainable, and formally denoted by diagonal matrices A₁ and Ag- This approach can effectively scale and disable rows and columns of both A and B, allowing for layer-wise adaptation with minimal number of trainable parameters. Note that in this setup, B E Ruxr and A E Rrxx
3

---

Published as a conference paper at ICLR 2024
not required to be low-rank. This is because they remain static and do not need to store their values. Instead, varying r leads to a linear increase in the number of trainable parameters via d E RIxT.
3.2 PARAMETER COUNT
Table 1: Theoretical memory required store trained VeRA and LoRA weights for RoBERTause RoBERTaaag and GPT-3 models. We assume that LoRA and VeRA methods are applied on query and key layers of each transformer block.
LoRA
VeRA
Rank
Trainable Parameters Required Bytes # Trainable Parameters Required Bytes
GPT-3 LARGE BASE
36.8K
144KB
18.4K
72KB
16
589.8K
2MB
18.8K
74KB
256
9437. IK
36MB
24.5K
96KB
1
98.3K
384KB
49.2K
192KB
16
1572.8K
6MB
49.5K
195KB
256
25165.8K
96MB
61.4K
240KB
|
4.7M
18MB
2.4M
9.1MB
75.5M
288MB
2.8M
10.5MB
256
1207.9M
4.6GB
8.7M
33MB
We use Lumed denote the number finetuned layers and demoseel to represent the dimension of these layers. The number of trainable parameters in VeRA is then governed by |θ] = Launed X (dmodel +r), contrasting with LoRA's |e| 2 x Launed X dmodel X r. Specifically, for the lowest rank (i.e.. 1). VeRA requires approximately half the trainable parameters of LoRA. Moreover, rank increases, VeRA's parameter count increases by Lumed for each increment, a substantial saving compared to LoRA's mestee This parameter efficiency becomes notably significant in the context extremely deep and wide models, such as GPT-3 (Brown et al., 2020), which has 96 attention layers and hidden size of 12288.
the
Building on this efficiency, the main advantage of VeRA is its minimal memory footprint for storing the trained weight adjustments. Because the random frozen matrices can be regenerated from random number generator (RNG) seed, these do not need to be stored in memory. This substantially reduces the memory requirement, which is now limited the bytes needed for the trained and vectors and single RNG seed. The memory efficiency in comparison to LoRA is shown in Table
3.3 INITIALIZATION STRATEGIES
Shared Matrices: In our method, employ Kaiming initialization (He et al. 2015) for the frozen low-rank matrices A and B. By scaling the values based on matrix dimensions, it ensures that a matrix product of A and B maintains consistent variance for all ranks, eliminating the need to finetune the learning rate for each rank.
The scaling vector b is initialized to zeros, which aligns with the initial- ization of matrix B in LoRA and ensures that the weight matrix is unaffected during the first forward pass. The scaling vector d is initialized with single non-zero value across all its elements, thereby introducing new hyperparameter that may be tuned for better performance.
Scaling Vectors:
Figure iillustrates example initializations for the low-rank matrices and scaling vectors in VeRA. Specifically, the low-rank matrices are initialized using a normal distribution, and the d vector is initialized with ones. Note that alternative initializations, such as uniform distribution for and B, and other non-zero constants for d, are also explored in our experiments.
4 EXPERIMENTS
In this section, we conduct a series experiments to evaluate our finetuning method. We start by comparing our approach LoRA and other baselines the GLUE and E2E benchmarks. Following
4

---

Published as a conference paper at ICLR 2024
this, turn attention to instruction-tuning of Llama models, and image classification with Vision Transformers. Next, we select one task and vary the rank for both methods, LoRA and VeRA, to examine how performance scales with the number of trainable parameters. Lastly, ablation study sheds light on the importance of each component our method, including the influence of different initializations.
Baselines.
We compare VeRA to the following baselines:
Full finetuning the model is initialized with pretrained weights and all parameters are being trained.
Bitfit this baseline involves the sole finetuning of bias vectors, keeping all other parameters fixed. This technique has been investigated in depth by Zaken al. (2022).
Adapter funing- initially introduced by HHoulsy al. (2019). involves the integration of adapter layers between the self-attention and MLP modules, followed by residual con- nection. This setup includes two fully connected layers and nonlinearity and is denoted as Adapter A variation by Lin et al. (2020), Adapter', employs the adapter layer solely LayerNorm. This closely resembles an alter- native design suggested by Pfeiffer et al. (2021). referred to Adapter", Another base- line, termed Arapterrrop by Rückklé et al. (2021) enhances efficiency by omitting certain
after the MLP module and subsequent
adapter layers and represented as Adapter
LoRA (Hu et al. 2022) as introduced in the earlier section.
4.1 GLUE BENCHMARK
We evaluate our approach on the General Language Understanding Evaluation (GLUE) benchmark
(Wang et al. 2019), employing the RoBERTabas and RoBERTalage models (Liu et al. 2019). For RoBERTabase we use a rank of 1024, and for RoBERTatage a rank of 256. The shared matrices are initialized using the uniform version of Kaiming initialization as implemented PyTorch (Paszke) et al. 2019), with an initial value of 0.1 for the d vector.
Our experimental setup generally aligns with that Hu et (2022), applying our method to the query and value projection matrices in each self-attention module and fully training the classification head. Unlike Hu et al. (2022). who used an additional hyperparameter a adjust gradients for the adapted layers, we introduce separate learning rates for the classification head and the adapted layers. We determine the learning rates and the number of training epochs through hyperparameter tuning; for detailed settings, refer to the Table in Appendix A The batch size is set to 64 for RoBERTanae and 32 for RoBERTaaage, with maximum sequence lengths of 5 512 and 128 respectively.
Due to time constraints and budget limitations, omit the time-intensive MNLI and QQP tasks, thus forgoing the use of the MNLI trick for tasks MRPC, RTE, and STS-B. In line with Hu et al. 2022), report the number of trainable parameters attributable to the finetuned layers, explicitly excluding the classification head, which is trained in a standard way. We perform runs with different random seeds, recording the best epoch's 's outcome for each run, and report the median of these results.
Table 2 reveals that VeRA performs competitively with LoRA across both models, yet achieves these results with order of magnitude fewer parameters.
Results.
4.2 E2E BENCHMARK
For the E2E benchmark (Novikova al. 2017). we follow the experimental setup from Hu et al. (2022) and finetune the GPT-2 (Radford et 20120 Medium and Large models. For LoRA we use the implementation and set of hyperparameters provided in Hu et al. (2022). while for VeRA change the rank and learning rate, both which are tuned. Table with all hyperparameter used can be found in Appendix
For the RoBERTabu Tauue model and MRPC, RTE and STS-B tasks, Hu et al. 2022 initialized the model with the best weights finetuned the MNLI task.
5

---

Published as a conference paper at ICLR 2024
Table 2: Results for different adaptation methods on the GLUE benchmark. We report Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for the remaining tasks. In all cases, higher values indicate better performance. Results of all methods except VeRA are sourced from prior work (Hu et al., 2022 Zhang et al. 2023a). VeRA performs par with LoRA with an order magnitude fewer parameters.
Trainable
Method
SST-2
MRPC
CoLA
QNLI
RTE
STS-B B
Avg.
Parameters
FT
125M
94.8
90.2
63.6
92.8
78.7
91.2
85.2
BitFit
0.1M
93.7
92.7
62.0
91.8
81.5
90.8
85.4
BASE
Adpt
0.3M
94.2+t..1
88.5
60.8
93.1
71.5
89.7.a.3
83.0
Adpt
0.9M
94.7+4.3
88.4
62.6+0.9
93.0+0.2
75.942.2
90.340.1
84.2
LoRA
0.3M
95.1
89.7
63,4
93.3
86.640.7
91.5
86.6
VeRA
0.043M
94.6.20.1
89.5
65.610.8
91.820.2
78.7
90.7
85.2
Adpt'
3M
96. +a.3
90.2
68.3
=4.8-u.2
83.8
42.7
87.6
Adpt"
0.8M
96.640.2
89.7
67.842.5
94.8
80.1
91.94..4
86.8
LARGE
Adp
6M
96.2 96.2+
88.7+2.9
66.5
94.7=0.2
83.4
91.0.11.7
86.8
Adpe"
0.8M
96.34..5
87.7
66.342.0
94.7+0.2
72.942.9
91.540.5
84.9
LoRA-FA
3.7M
96.0
90.0
68.0
94.4
86.1
92.0
87.7
LoRA
0.8M
96.2.2..5
90.2
68.221.9
94.8
85.2
92.3
87.8
VeRA
0.061M
96.1
90.9
68.0+0.8
94.420.2
85.9
91.7
87.8
Table 3: Results for different adaptation methods on the E2E benchmark and GPT2 Medium and
Large models. Results with (1.2.3) are taken from prior work: "HHu al. 2022). *(Valipour al. 2022). 3 'Zi et al. 2023). VeRA outperforms LoRA with 3 and 4 times less trainable parameters, for GPT2 Medium and Large respectively.
Trainable Parameters
Method
BLEU
NIST
METEOR
ROUGE-L
CIDEr
FT!
354.92M
68.2
8.62
46.2
71.0
2.47
MEDIUM
Adpt--1
0.37M
66.3
8.41
45.0
69.8
2.40
Adpi
11.09M
68.9
8.71
46.1
71.3
2.47
Adpt
11.09M
67.3
8.50
46.0
70.7
2.44
DyLoRA²
0.39M
69.2
8.75
46.3
70.8
2.46
AdaLoRA3
0.38M
68.2
8.58
44.1
70.7
2.35
LoRA
0.35M
68.9
8.69
46.4
71.3
2.51
VeRA
0.098M
70.1
8.81
46.6
71.5
2.50
FT*
774.03M
68.5
8.78
46.0
69.9
2,45
LARGE
AdptL1
0.88M
69.1
8.68
46.3
71.4
2.49
AdptL1
23.00M
68.9
8.70
46.1
71.3
2.45
LoRA
0.77M
70.1
8.80
46.7
71.9
2.52
VeRA
0.17M
70.3
8.85
46.9
71.6
2.54
Results. We report results from the last epoch. Table Table that VeRA outperforms LoRA with 3 and 4 times less trainable parameters, for GPT2 Medium and Large respectively.
4.3 INSTRUCTION TUNING
Instruction tuning is process by which language models are finetuned to follow specific instructions more effectively (Ouyang et al., (0022). We demonstrate efficacy of VeRA enabling Ilaaaa Touvron et al. 2023a) and Llama2 (Touvron et al., 2023b) models to follow instructions using only 1.6M and 2.4M trainable parameters, for 7B and 13B variants respectively, in contrast to 159.9M 9M and 250.3M trainable parameters when employing LoRA with rank of 64 as proposed by Dettmers! et al. (2023).
We perform finetuning using both LoRA and VeRA, by applying both methods all linear layers except the top one, similarly to Dettmers et al. (2023). Additionally, we leverage the quantization techniques from Dettmers et al. (202)) train the model single GPU.

---

Published as a conference paper at ICLR 2024
For our experiment, employ the Alpaca dataset (Taori et al. 2023). siorD This dataset comprises 51K instructions and demonstrations and is suitable for instruction- tuning. The cleaned version corrects multiple issues such as hallucinations,, merged instructions, and empty outputs. We train for one epoch, preceded by a learning rate sweep.
specifically its cleaned ver-
We evaluate finetuned models MT MTT-Bench (Zheng al. 2023 by generating model responses to pre-defined set of 80 multi-tur questions and subsequently evaluating these using GPT-4 (OpenAl) 2023). GPT-4 reviews the answers and assigns quantitative score scale of 10 each response. We present the average scores alongside the number of trainable parameters in Table
Table 4: Average scores on MT-Bench assigned by GPT-4 the answers generated by models fine-tuned with VeRA and LoRA methods, and the base Llama 13B model. VeRA closely matches performance of LoRA on the instruction-following task, with 100x reduction in trainable parameters.
Model
Method
Parameters
Score
Llama 13B
2.61
LoRA VeRA
159.9M
5.03
LLAMA 7B
1.6M
4.77
LoRA
250.3M
5.31
LLAMA 13B
VeRA
2.4M
5.22
LoRA
159.9M
5.19
LLAMA2 7B
VeRA
1.6M
5.08
LoRA
250.3M
5.77
LLAMA2 13B
VeRA
2.4M
5.93
We find that despite the 100x reduction in the number of trainable parameters, method closely matches the performance of LoRA-basee finetuning.
4.4 IMAGE CLASSIFICATION
To evaluate the method on the image classification task, we adapt Vision Transformer (ViT) (Doso- vitskiy et al., 2021). Base and Large variants, on datasets CIFARIOO 100 (Krizhevsky) 2009), Food101 (Bossard et al. 2014), Flowers 102 (Nilsback & Zisserman, 2008). and RESISC45 (Cheng et al.) 2017). For each dataset we train on a subset of 10 samples per class, and evaluate on the full test set (CIFAR100, F0od101, Flowers 102) or all the remaining samples (RESISC45). We use weights of ViT models pretrained on the ImageNet-211 (Deng et al. 2009) dataset.
We evaluated LoRA and VeRA methods applied the query and value layers of ViT, along with two baselines- fully-finetuned model (referred to as Full), and training the classification head only (referred to as Head). Similarly to the GLUE benchmark, VeRA. We tuned learning rates for all methods and reported results after 10 epochs in Table The reported parameter count excludes the classification head, which has be trained all methods.
rank 8 for LoRA, and rank 256 for
We find that VeRA approaches performance of LoRA on the Base model for three datasets and outperforms it for Flowers102. despite using over 10x fewer trainable parameters. For ViT-Large, outperforms LoRA for three datasets: CIFAR100, Flowers 102 and RESISC45.
4.5 SCALING THE NUMBER OF TRAINABLE PARAMETERS
Finally, investigate the trade-offs involved in parameter scalability for both LoRA and our method using the RoBERTa model on the RTE task from the GLUE benchmark. We use a set of ranks (1, 4, 8, 16, 32, 64) for LoRA, and observe the trade-off between trainable parameters and the accuracy. We replicate each configuration five times for different random seeds, and report the median of results. For LoRA, employ the HuggingFace PEFT (Mangrulkar et al., 2022) implementation, adhering to the hyperparameters specified in Hu al. (2022). Our own method uses the same hyperparameters as employed in the
{1, 4, 16, 64, 256, 1024) for VeRA and

---

Published as a conference paper at ICLR 2024
Table 5: Vision models finetuned with VeRA and LoRA on different image classification datasets. VeRA approaches performance of LoRA for the smaller model, and outperforms it in the case of the large model, with over 10x fewer trainable parameters.
Trainable Parameters
Method
CIFAR100
Food101
Flowers102
RESISC45
Head Full
77.7
86.1
98.4
67.2
viny VIT.
85.8M
86,5
90.8
98.9
78.9
LoRA
294.9K
85.9
89.9
98.8
77.7
VeRA
24.6K
84.8
89.0
99.0
77.0
Head
79.4
76.5
98.9
67.8
Full
303.3M
86.8
78.7
98.8
79.0
LoRA
786.4K
87.0
79.5
99.1
78.3
VeRA
61.4K
87.5
79.2
99.2
78.6
VeRA
86
LoRA
12
valuue
Accuracy [%6]
10
85
||d||
10
15
20
Layer
10
10
# Trainable Parameters
Magnitude of the adapted d vec- tor for query and value matrices across lay- ers for RoBERTa-L on the RTE task.
Figure 3:
Performance of LoRA and VeRA methods for varying ranks on RTE task.
Figure 2:
RTE experiments from the previous subsection. The results, depicted in Figure 2 reveal that our method is significantly more parameter-efficient Notably, when the higher-rank VeRA has the same number of parameters as standard LoRA, it outperforms LoRA by 4 accuracy percentage points.
4.6 ABLATION STUDY
In this section, we conduct an ablation study to examine the impact of individual components of our method. All subsequent experiments focus on the MRPC and RTE tasks and utilize the RoBERTanne model. We adhere to the hyperparameters used in previous experiments, modifying only the com- ponent under investigation for each test. Each experiment is run with 5 random seeds, and we report the mean and standard deviation of the results.
Single Scaling Vector We first investigate the necessity of both the d and b scaling vectors in our method. We create two ablation setups: one that excludes d (termed only b) and another that omits b (termed as only d). In the only d setup, d is initialized with zeros. As shown in Table 6 omitting either scaling vector compromises performance. The only d configuration performs slightly better than its only b counterpart. This disparity in performance underscores the higher expressiveness of
Table 6: Ablation study results for the impact of the d and b scaling vectors and different initializa- tion strategies. Our default settings are highlighted with blue color.
(a) Scaling Vector Ablations
(b) Matrix Initialization
(c) Vector Initialization
Method MRPC
RTE
Matrix Init.
MRPC
RTE
d Init.
MRPC
RTE
VeRA 90.5
85,8
Kaiming Unif.
90.5
85.8
10-1
90.540.7
85.8
only 89.7=0.0
67.0413.9
Kaiming Norm.
90.0=1.
82.645.2
10-7
90.8
84.7
only
81.6 64.3
Uniform 0...0.)
68.9
53.140.8
1.0
70.3+1.2
60.3+12.4
8

---

Published as a conference paper at ICLR 2024
Table 7: Results for selected GLUE tasks using shared and unique random matrices.
Random Matrices
MRPC
RTE
CoLA
STS-B
Shared Unique
90.020.a 90.7.0.3
84.621.5
67.720.8
91.5-0.6
84.6
68.3
91.5-0.2
the d scaling vector over the vector. Specifically, d modulates the rows of both low-rank matrices, thereby influencing a broader aspect of the final constructed matrix. In contrast, only scales the rows of the final matrix resulting from the product of the low-rank matrices.
Initialization of Shared Matrices We examine three different initialization schemes for the shared matrices: Kaiming normal, Kaiming uniform, and uniform initialization within the range [0, 0.1]. As per the results in Table both Kaiming initializations outperform the uniform range initialization, with uniform variant having slightly better results than the normal one.
Initialization of Scaling Vector We further explore the impact of the initialization values for the 1.0, 10-1, and 10-7 The results in Table
d vector. Experiments conducted with dinit 6 show that the choice of dinit significantly influences the method's performance; in the settings we examined, values 10-1 and 10-7 outperformed 1.0, potentially offering more flexibility in the optimization process through early sign changes in selected rows of the frozen matrices.
Magnitude of Adaptation In Figure we provide a visualisation of the magnitude of the changes of the vectors after finetuning on RTE task. Because the low-rank frozen matrices remain the same for each layer, we can directly compare the length of the d vector across layers to account for its relative adaptation. Overall, we find that the largest adaptation happens for query matrices compared to the value ones, indicating a larger need or ease for finetuning model there. Furthermore, similar previous efficient adaptation methods' findings (Zhang al.) 2023b Liu al., 202I) we also observe a higher adaptation for the later layers compared to earlier ones.
Sharing Random Matrices We conduct experiments on RTE, MRPC, CoLA, and STS-B tasks assess the impact sharing random matrices on the performance. We evaluate two setups one with random matrices shared across all adapted layers, and another with uniquely generated ones. Results in Table that the mean performance identical case of tasks RTE and STS-B, and there is slight improvement for MRPC and CoLA when using unique matrices.
CONCLUSION
In this work, we introduce a finetuning method that significantly reduces the number of trainable parameters compared to LoRA, yielding similar better results downstream tasks. Specifically, achieved ten-fold reduction in parameters yielding the same performance on the GLUE benchmark for RoBERTatage- ten-fold reduction image classification tasks, and three-fold reduction the E2E benchmark. This method is particularly well-suited for scenarios that require frequent swapping of numerous finetuned models, such as cloud-based Al services personalized for individual users. Due to the minimal size of the scaling vectors, many versions can reside in the limited memory of a single GPU, thus substantially improving serving efficiency and removing the bottleneck of loading specific models into memory.
While the current study focuses on language and vision models with Transformer architecture, the applicability of the method across different architectures and domains remains an area for future research. Moreover, the performance of the method may benefit from additional refinements, such as dynamic parameter budget allocation, or different initialization and regularization techniques.
ACKNOWLEDGEMEENTS
This work is financially supported by Qualcomm Technologies Inc., the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Min- istry of Economic Affairs and Climate Policy. We also acknowledge financial support from ELLIS Amsterdam. and the use of National Supercomputer Snellius and Distributed ASCI Supercomputer 6 (Bal et al. 2016) for essential computational tasks.

---

Published as a conference paper at ICLR 2024
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Nat- ural Language Processing (Volume 1: Long Pupers), pp. 7319-7328, Online, August 2021. As- sociation for Computational Linguistics. doi: 10.18653/ aacloo000....-1oo5556666 URL https:
568
H. Bal, D. Epema, C. de Laat, R. van Nieuwpoort, J. Romein, F. Seinstra, Snoek, and H. Wijshoff. A medium-scale distributed system for computer science research: Infrastructure for the long term. Computer, 49(05):54-63, may 2016. ISSN 1558-0814. doi: 10. 10.1109/MC.201.11 127.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. In European Conference on Computer Vision, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL lattps://proceeddnggg hew.sedddd......rrse..o.o.o. http:/ppprrfleeeappeeeeeee fitess/apeer//22////// File/ 1e///1177cco0
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 105(10))1865-1883,22 2017.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
--33-304 33-30-vicuna
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009, doi: 10. 200.........2066666
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: 7/open eww n d= cbFdNTTy
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. Training batchnorm and only batchnorm: On the expressive power of random features in [enn)s. In International Conference on Learning Representations, 2021. URL https: dttps::/openrrewee wet.eeewwweeeeeeeeee.eeeeeeeeeee http://rrenee---e1000222222
Kaiming He,, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015. doi: 10. 109/CCV.2015..23..
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruma Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficias transfer learning for nlp, 2019.
10

---

Published as a conference paper at ICLR 2024
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference Learning Representations, 2022. URL
www.eewww...eeeeeeeeeeeeeeeeeeee forum?
iddnZeVKeeeyfF
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018. URL
htttps::/ppenreviee net. Forum?id=ryuppp
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficien transfer learning. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pp. 441-459, Online, November 2020. Association for Computational Linguistics. doi: 18.18653/v 1/2020 orq/ f findingssnnnnnnnneennnnn http://wwo000..nnngnggoom//lll// findings-emnlp.41. URL
41
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P- tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprini arXiv:21 2021.
Yinhan Liu, Myle Out, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers as universal computation engines. Proceedings of the AAAI Conference on Artificial Intelligence, 36 (7):762--7636, Jun. 2022. doi: 10. 10.1609/aaai, v3677.20729. URI. / aaa .org/
index. TatexxppppAAAA//rrrcceeeeee/
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. https:://ithub. com/ huegingface/peef! 2022.
M-E. Nilshack and A. Zisserman. Automated flower classification over large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008
Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Di- alogue, pp. 201-206, Saarbrücken, Germany, August 2017. Association for Computational Lin- guistics, doi: 10. 10.11655/ URL U www.hhhoooooooor/11--1555555
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- ral Information Processing Systems, 2022. URL https: http///pppeeeeeeeeeeeeeeeeeeeeep net forum?id
httpps:://peeeeeeee...t
TG8KACxEON
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL htRpttsss:////t htt.://nwononuooar////ooooooonnn
files/
9/ f 2aap://00777111loooo222229997699

---

Published as a conference paper at ICLR 2024
Hao Peng. Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations, 2021. URL
lttps://epenneviees
http://wwwwwwweeeeeeeeeeeeeeeeee net/ Fo
Jonas PfeitTer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- Fusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487-503, Online, April 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 2021 macl-main..9, URL https: //ac anthologyyyoo.... wrg.looooo.oooooooooooooooooeero
org /ttrg/201111 2021 leacllmaa n..aiin.3 39
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
V. Ramanujan, M. Wortsman, A. Kembhavi, A. Farhadi, and M. Rastegari. What's hidden in a randomly weighted neural network? In 2020 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR). pp. 11890-1 1199, Los Alamitos, CA. USA, jun 2020. IEEE Computer Society. doi: 10.1 100////V222000022000000001
URL https://doi:
seeecoputerscciety..oo orw. 1tt.:/1..1111111111 11t.1111099/099000 2020 2020. 01191
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AdapterDrop: On the efficiency of adapters in transformers. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 7930-7946, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 11186633/112221.mmmlppp cellp-mmin.66. main.626. URL httph:tp://// http://aaaaaanttaaaaeaaa 2021. www.emnnnmmmmmm mai... 626
htt.a///.Laaaaaaaaaaaaaaoooooooa org/
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetorrs for fast personaliza- tion of text-to-image models, 2023.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture of lan- guage: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):e210564611 2021. doi: 10. 10.10733pnas.21056460 18. URL
https: www. pnas. org/ /tt.////bbb10.113//aa...... 2aiaaas61056461111
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follviiii llama model.
https://gt tttt::////tttt....///// httpmm//aattslllaattttttorrrrala
2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull. David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter effi- cient tuming of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558. 2022.
12

---

Published as a conference paper at ICLR 2024
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2019. BtLLtttts///peeeee///
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efffcient fine-tuning for transformer-basee masked language- models, 2022.
Longteng Zhang. Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficien low-rank adaptation for large language models fine-tuning. arXiv preprint Stl::300.003003, 2023a.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh In- lernational Conference on Learning Representations, 2023b. URL httttts://ppeeeeiiiiii
net/ netf/forummic--1166WWWWWW11
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta- lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint arXiv:2309.02411, 2023.
13

---

Published as a conference paper at ICLR 2024
A HYPERPARAMETERS
Table 8: Hyperparameter configurations for different model sizes on GLUE benchmark. Optimizer,
Warmup Ratio, and LR Schedule are taken from Hu al. (2022)
Model
Hyperparameter
SST-2
MRPC CoLA
QNLI
RTE STS-B
Optimizer
AdamW 0.06 Lincar
Warmup Ratio
LR Schedule
Init. Shared Matrices
Kaiming Uniform
Initial Value of
0.1
# GPUs
1 1024
VeRA Rank
BASE
Epochs
60
30
80
25
160
80
Learning Rate (Head)
4E-3
4D-3
IE-2
4E-3
IE-2
IE-2
Learning Rate (VeRA)
4E-3
IB-2
IE-2
IE-2
4E-3
Max Seq. Len.
512
Batch Size Per GPU
64
# GPUs
4
VeRA Rank
256
LARGE
Epochs
10
40
40
20
40
20
Learning Rate (Head)
6E-3
3B-3
6E-3
2E-4
2E-3
2E-3
Learning Rate (VeRA)
1E-2
3B-2
IE-2
1E-2
2E-2
2E-2
Max Seq. Len.
128
Batch Size Per GPU
32
In T we provide the hyperparameters used for the GLUE benchmark in the main paper. Note that due to our academic compute we were able to run full grid searches on any hyperparame- ters. We only evaluated different learning rates and number of epochs and even relied on existing configurations of LoRA (Optimizer, Warmup ratio, LR schedule).
Table 9: Hyperparameter configurations for instruction-tuning
Hyperparameter
LoRA
VeRA
GPUs
Optimizer
AdamW
Warmup Ratio
0.1
Batch Size
Accumulation Steps
Epochs
I.R. Schedule
Cosine
Rank
64
1024
Learning Rate
4F-4
4F-3
14

---

Published as a conference paper at ICLR 2024
Table 10: Hyperparameter configurations for VeRA on the E2E benchmark, for GPT2 Medium and Large models.
Hyperparameter
Medium
Large
GPUs
Optimizer
AdamW
Learning Rate Schedule
Linear
Weight Decay
0.01
Batch Size
Epochs
Warmup Steps
500
Label Smooth
0.1
Rank
1024
Learning Rate
1E-1
2E-2
Table 11: Hyperparameter configurations for VeRA and LoRA for finetuning VIT on the image classification datasets. Full, LoRA and VeRA methods have learning rates for the classi- fication head, and the other for the rest.
Model
Hyperparameter
CIFARIO0
Food101
Flowers102
RESISC45
GPUs
Optimizer
AdamW Linear
I.R Schedule
Weight Decay
0.01
VeRA Rank
256
LoRA Rank
LR-Head (Head)
4E-3
4E-3
4E-3
4E-2
LR (Full)
4E-5
4E-5
4E-5
8E-5
BASE
LR-Head (Full)
4E-3
4E-2
4E-3
4E-3
LR (VeRA)
2D-2
4E-2
4E-2
7E-2
LR-Head (VeRA)
4E-3
4E-2
4E-3
5E-3
LR (LoRA)
4E-3
4E-3
4E-3
4E-3
LR- HR-Heed (LoRA)
4E-3
4E-3
4E-3
4E-3
LR- Head (Head)
4E-4
4E-3
4E-3
LR (Full)
4E-5
4E-5
4E-5
8E-5
LARGE
LR RRad (Full)
4E-3
4E-3
SE-3
4E-3
LR (VeRA)
4E-2
4F-2
4F-2
7F-2
LR-Head (VeRA)
2F-3
2F-3
2F-3
3E-3
LR (LoRA)
4E-3
4E-3
4F-3
4F-3
LR-Head (LoRA)
4E-3
4E-3
4E-3
4E-4
15

---

Published as a conference paper at ICLR 2024
B RELATIVE PERFORMANCE GAIN.
3.0
2.64
Performance Gain (%)
2.0
1.0
0.17
0.15
0.18
0.0
A3))
AdpiP(0.aM) AdptH(eM) AdptH(0.8M)
LoRA
VeRA
Method
Figure 4: Performance gains 1K trainable parameters on the RTE task for RoBERTatage model relative to the baseline. Formula: Aeceuracymetood aacaaaccccccuullaiaaaaa parametsrs 100
Figure quantifies the efficiency of each method in terms of performance gains per IK trainable parameters. For focused comparison, we select the RTE task and RoBERTajage model.
To establish a baseline, we conduct auxiliary experiments where only the classification head is trained while the remainder of the model is frozen. This baseline is constructed using the same hyperparameters as in our VeRA method. We then evaluate the performance gain attributable to each method, normalized by the additional trainable parameters introduced, relative to the baseline. The results clearly show that VeRA yields the highest performance gain per 1K trainable parameters.
C IMPACT ON TRAINING TIME AND MEMORY USAGE
To evaluate the training time and GPU memory benefits of our method, we conducted a comparison between LoRA and VeRA while fine-tuning LLaMA 7B with the same rank (64) instruction tuning dataset, introduced earlier in this work. The results are summarized in Table 12
Table 12: Impact GPU memory usage and training time.
Method
Training Time
GPU Memory
LoRA
568 min
23.42GB 21.69GB
VeRA
578 min
While VeRA includes more operations than LoRA because of the additional vector multiplies in the forward pass, we find that it only results in modest 1 1.8% increase in training time. For the GPU memory, we observe a 7.4% reduction memory usage with VeRA, as does not require storing optimizer states and gradients for shared random matrices.
D SIMILARITIES OF TRAINED WEIGHTS
We compared the weights trained with LoRA and VeRA at single rank of 64 across all query layers. For each method and adapted layer, we constructed a weight difference. In LoRA's case, this involved the multiplication of two low-rank matrices, while for VeRA, it also included multiplication by scaling vectors. We then calculated the cosine similarity these flattened weights. Additionally, compared the similarity between trained LoRA weights and randomly initialized matrices baseline: We find that similarities of VeRA to LoRA are on average 2 while LoRA to random matrices is -8-5..
16

---

Published as a conference paper at ICLR 2024
LoRA-VeRA LoRA-Random
Cosine Similarity
0.004
0.002
0.000
0
10
15
20
Layer
Figure 5: Cosine similarity of LoRA, VeRA, and random weights across layers.
In Figure we can a notable increase in similarity between the trained weights, particularly in the latter layers. This observation aligns with our earlier findings (Figure 3 that the highest adaptation occurs in these layers. These results support the notion that VeRA approximate the weights trained with LoRA.
E EXPRESSIVITY OF VERA
conducted an experiment on the expressivity of LoRA and VeRA on the task of fitting random square 10x10 matrices, with results seen in Figure 6 For given number of trainable parameters, both methods perform equally well, with VeRA providing more flexibility, egg. by allowing for much lower parametrization below LoRA's rank 1.
Mean Loss Comparison
Mean XdYb
0.25
Mean Loss
0.15
0.10
0.05
100
num_ uum_parameters
Figure 6: MSE loss on the task of fitting 10x10 random matrix. LoRA (PQ) and VeRA (XdYb) perform equally well for given number of trainable parameters.
17

---

Published as a conference paper at ICLR 2024
F INSTRUCTION-TUNING WITH VICUNA EVAL
Results and samples from evaluation of instruction tuned Llama 7B model with Vicuna Eval (Chiang al. 2023), predecessor of MT-Bench. The model has been finetuned 10K subset of cleaned Alpaca dataset.
Average scores assigned by GPT-4 to the answers generated by models fine-tuned with VeRA and LoRA methods. Number of trainable parameters and theoretical memory required to store them in single-precision also shown.
Table 13:
Method
Parameters Required Bytes Avg. Score
LoRA VeRA
159.9M 1.4M
610MB
7.36
5.4MBB
7.48
LoRA
VeRA
Write symphony concert review, discussing the orchestra's 's performance and overall audience experience.
ANSWER
The symphony concert was a truly mesmerizing experience, with the orchestra's performance leaving the audience awe. with expertise and passion, led the musi- cians through a series of beautifully composed pieces, each one more captivating than the last. The orchestra's sound was rich and full, with each instrument perfectly balanced and har- mony. The audience was enraptured by the mu- sic, and the applause at the end of cach piece thunderous. The concert was true testa- ment to power of music and the talent of the musicians.
The symphony concert held at the Lincoln Cen- for the Performing Arts Saturday, March 12th, 2022, was truly mesmerizing experi- ence. The New York Philharmonic Orches- tra, under the direction of Music Director Jaap van Zweden, performed a program of works by Beethoven, Strauss, and Mahler.
conductor,
The concert began with Beethoven's Symphony No. 1, which was a perfect introduction to the evening's 's program. The orchestra played with precision and energy, capturing the spirit of the piece and transporting the audience differ- ent time and place.
The audience was immediately drawn in by the orchestra's 's performance, and the energy in the room palpable. The second piece the program was Richard Strauss's Don Quixote, which was real treat for the audience. The orchestra played with great passion and enthu- siasm, bringing the music to life and capturing the spirit of the piece. The audience was en- thralled by performance. and the applause at the end of the piece was thunderous.
REVIEW
Assistant provided a general review a sym- phony concert, which was relevant and accu- rate. However, it lacked specific details such the location of the concert, the date, the chestra's name, the conductor's 's name, pieces performed. This lack of detail made the review less helpful and engaging.
Assistant 2 provided a detailed and engaging re- view of a specific symphony concert. It men- tioned the location, date, orchestra's 's name, con- ductor's name, and the pieces performed. It also described the audience's reaction to each piece, which added depth to the review. This level of detail made the review more helpful, relevant, accurate, and engaging.
SCORE
7
18

---

Published as a conference paper at ICLR 2024
LoRA
VeRA
What if Isaac Newton had focused biology instead of physics?
ANSWER
Isaac Newton brilliant scientist who made significant contributions to both physics and biology. While he is best known for his work in physics, he also made important discoveries in biology, including the discovery of the laws of motion and the theory of gravity.
If Isaac Newton had focused on biology instead physics, he may have made significant contributions to the field of biology. Newton was a brilliant scientist and his work in physics revolutionized the field. However, he also had keen interest in biology and made several observations world.
insights about the natural
If Newton had focused on biology instead of physics, he may have made even greater contributions the field. He server and experimenter, and his keen intellect would have allowed him to make important discoveries in biology. Hc may have studied the anatomy of plants and animals, or the bchavior of living organisms, the interactions between different species.
skilled ob-
For example, Newton the first de- scribe the process photosynthesis, the process by which plants convert sunlight into energy. He also made observations about the behavior of animals, including the way they move and the way they interact with their environment.
Newton's 's work in physics laid the founda- tion for many of the discoveries made in biology, including the understanding of role of gravity in the movement of objects. His work in biology may have built upon this foundation, leading to even greater advances in the field.
Newton's work in biology was not as ex- tensive as his work in physies, but it shows that he had deep interest in natural world and desire to understand it. If he had focused on biology instead of physics, he may have made even greater contributions to the field of biology.
summary, if Newton had focused on bi- ology instead of physics, he may have made even greater contributions to the field, and his work in biology would have been just as significant as his work in physics.
REVIEE
Assistant I's response is somewhat helpful and relevant, but it contains a major inaccuracy. The assistant claims that Isaac Newton made signifi- cant contributions to biology, including the dis- covery of the laws of motion and the theory of gravity. However, these are actually contribu- tions to physics, not biology. The assistant also speculates on what Newton might have stud- ied if he had focused biology, but these purely hypothetical and based on any his- torical evidence. The assistant's 's response is de- tailed and well-structured, but the inaccuracy undermines its overall quality.
Assistant 2's response is less helpful and rele- vant, and it contains a major inaccuracy. The assistant claims that Isaac Newton was the first to describe the process of photosynthesis. This is incorrect; the process of photosynthesis not understood until the 19th century, long after Newton's 's time. The assistant also makes some general observations about Newton's 's interest in the natural world, but these not specific biology and do not provide clear answer the user's 's question. The assistant's response is less detailed and less well-structured than Assistant I'ss response, and the inaccuracy further reduces its quality.
SCORE
4
19

---

Published as a conference paper at ICLR 2024
LoRA
VeRA
How many times has the Earth orbited the Sun since the heginning of life? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step.
ANSWER
The Earth has orbited the Sun approximately 5.5 billion times since the beginning of life. This is based on the assumption that life on Earth began approximately 4.5 billion years ago, and that the Earth has been orbiting the Sun for approximately 4.5 billion years.
It is difficult to determine exactly how many times the Earth has orbited the Sun since the be- ginning of life, as the exact age of the Earth is still matter of debate. However, can make an estimate based on the current scientific con- sensus.
The age of the Earth is estimated to be between 4.54 and 4.58 billion years old, with the most recent estimates placing at 4.54 billion years. This means that the Earth has been orbiting the Sun for approximately 4.54 hillion years. To calculate the number of times the Earth has orbited the Sun since the beginning of life, can use the formula:
To calculate the number of times the Earth has orbited the Sun, we can use the formula for the circumference of a cirele, which pi * r, where pi is approximately 3.14 and is the radius of the circle. The radius of the Earth is approximately 6,371 kilometers, the circumference of the Earth is approximately pi 6,371 = 12,742,200 200 kilometers.
[N = T x 365.25
where Ñ is the number of orbits, T is the age of the Earth, and 365.25 is the number of days in year.
The Earth orbits the Sun at an average distance of approximately 149,600,000 kilometers, so
the Earth travels approximately 149,600,000 Substituting the values into the formula, we get: kilometers per orbit. Therefore, the Earth has orbited the Sun approximately 149,600,000 / 12,742.200 = 1.17 times.
[N 4.54x10 X 4.54x109
366.25
which simplifics to:
[N 1.2 x 10 0]
This means that the Earth has orbited the Sun
This means that the Earth has orbited the Sun approximately 5.5 billion times since the beginning of life.
approximately 1.2 trillion times since the begin- ning of life.
It is important to note that this estimate is based on the current scientific consensus, and that the actual number of orbits may vary slightly duc to uncertainties in the age of the Earth. Addi- tionally, the age of the Earth is constantly being revised new evidence is discovered, this number may change in the future.
REVIEW
Assistant I's response confusing and incor- rect. The assistant tries to calculate the num- ber of Earth's orbits around the Sun by using the formula for the circumference of a circle and the Earth's 's radius, which irrelevant to the question. The assistant then incorrectly states that the Earth has orbited the Sun approximately 5.5 billion times, which contradicts the initial assumption that life on Earth began approxi- mately 4.5 billion years ago.
Assistant 2's response, more accurate and detailed, The assistant cor- rectly states that the exact number of Earth's 's or- bits around the Sun since the beginning of life is difficult to determine due to uncertainties in the age
the other hand, is
Earth. However, the assistant pro- vides a reasonable estimate based on the cur- rent scientific consensus on the age of the Earth. The assistant also correctly uses the formula for calculating the number of orbits and provides clear, step-by-step explanation of the calcu- lation. However, the assistant made a mistake in the final calculation, stating that the Earth has orbited the Sun approximately 1.2 trillion times, which significantly higher than the ac- tual estimate. Despite this error, Assistant 2's response is more helpful, relevant, and accurate than Assistant 1 response.
SCORE
4
7
20

---

Published as a conference paper at ICLR 2024
LoRA
VeRA
Implement program find common elements two arrays without using any extra data structures.
ANSWER
Here Python program can find common elements between two arrays without using any extra data structures: python
Here is a Python program that can find the common elements in two arrays without using any extra data structures:
def common common.clementts(rr. elements(arr), arr2):
python
# Sort both arrays .srt.)ort) arr2.sort()
def common_elennee eelementsemmrrr, arr2): # Initialize set to store the common elements common_set sct()
# Find length of the shorter array length = min(len(arrl), len(arr2))
Iterate over the array for element in arrl: # If the element is already in the set, skip it if element common continue
# Create set store the common ele- ments Jommon__eemente set() # Loop through the shorter array
# Otherwise, add it to the set common set.add(clement)
for in range(length):
# Check if the current element is present in the shorter array if arrl [i] in arr2:
Iterate over the second array for element in amr2:
# Add the element to the common elements set www.oonnceeeetssaaece
# If the element is already in the set, skip it if element in common_sen continue
# Return the common elements
return common_ede cemmonneleements
# Otherwise, add it to the set common sel.add(clement)
This program takes two arrays as input and returns a set containing the common elements between the two arrays. The program first sorts both arrays to ensure that the elements are in the same order. Then, finds the length of the shorter array and creates a set to store the common elements. It then loops through the shorter array and checks if the current element is present the shorter array. If it is, the element added the common elements set. Finally, the common elements are returned as set.
Return set of common elements return common_set
This program takes arrays input and returns a set containing the common elements between the two arrays.
REVIEW
The assistant has provided a solu- tion, but it does not adhere to the problem state- ment. The problem specifically states not any extra data structures, but the assistant's lution uses a set to store the common elements. The assistant's is comments are clear and explain the logic of the code well. However, the code itself is not efficient it the 'in' opera- tor check an element is
Assistant 1:
Assistant The assistant's solution also does not adhere the problem statement as it uses store the common elements. The assis- tant's comments of the code well. However,
clear and explain the logic code itself is not correct. The assistant's solution adds all el- ements from both arrays the set, which means it will find the common elements but rather all unique elements from both arrays. This is fundamental misunderstanding of the problem.
array, which has time complexity of O(n). This makes the overall time complexity of the solution O(n 2). which is not efficient.
ScORE
4
2
21


Reviews:
[{"review_id": "xhSadcuUkF", "reviewer": "Reviewer_Bg6K", "summary": "This work proposes ELoRA, a new method for parameter-efficient finetuning that trains only the scaling vectors for low-rank parametrized versions of linear layers. The proposed method reaches the same quality as LoRA on a variety of benchmarks while having more than 10 times fewer trainable parameters.\n\n---\nPost-rebuttal update: thank you for the revised version! It addresses most of my concerns, and I am happy to increase the score to 8.", "questions": "* Technically, it should be possible to have different random matrices in different layers (the cost of storing one random seed per layer should be negligible), which might increase the capacity of the model even further. I wonder if you have explored this idea in preliminary experiments? It is quite surprising to me that a single random structure is sufficient for PEFT across all layers in a Transformer model — perhaps there is something we can infer from that.\n* I think that the work would benefit from a bit more analysis behind the reasons of why random projections work that well in the context of parameter-efficient finetuning. At the very least, it might be interesting to compare the structure of the learned weights for LoRA and ELoRA trained with a single rank: intuitively, the latter should approximate the former, but is that true in practice?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The work proposes a simple idea that works quite well in practice, according to the experiments.\n* Authors have a comprehensive evaluation setup, testing ELoRA across several model sizes and datasets and comparing it with baselines.\n* The influence of each component of ELoRA is investigated within a detailed ablation study.\n* Overall, the paper is well-written and all the contributions are easy to understand.", "weaknesses": "* The idea of using random matrices for parameter-efficient finetuning has been proposed previously in the context of text-to-image generation [1]. Although the ideas of the submission and that paper are quite different (also, the paper is quite recent), in my opinion, it would be great to mention LiDB (Lightweight DreamBooth) in the related work section to give the reader additional context.\n* My primary concern with respect to the experiments is that the instruction tuning evaluation is performed mostly through GPT-4 queries on a very small dataset. In my opinion, this approach suffers both from the lack of reproducibility (as we know, the model behind the GPT-4 API might change in the future) and a narrow coverage of possible contexts that might result in high variance of the metric. I think that using more established benchmarks (such as MMLU or BIG-Bench) would give a better picture of how instruction-finetuned models perform after PEFT.\n* Similarly, I think that broader evaluation of instruction tuning should include a larger set of models than just LLaMA-2. Personally, I would move Table 4 and Table 5 to the appendix (they take a lot of space and are not directly related to the topic of the paper) and replace that with additional experiments, ideally for models with higher parameter counts.\n\n[1] HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models. Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, Kfir Aberman. 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7ZNJSSnLbz", "reviewer": "Reviewer_JAXk", "summary": "This paper proposes a novel parameter-efficient LLM finetuning algorithm which is considerably more parameter-efficient than LoRA. The idea is to use 2 random matrices shared across all of the layers in the LLM, but to use two sets of diagonal scaling matrices to modulate the shared random matrices differently for each layer. The results indicate that the proposed method requires roughly an order of magnitude less parameters while maintaining or exceeding the performance of LoRA.", "questions": "1) Have you tried extending this method to finetuning large image models? What about large multimodal models?\n2) What are the training time benefits of your method? Is there a GPU memory benefit? Is there a latency benefit? Does your method train faster than LoRA because it has less parameters?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) The paper is very easy to read\n2) The method is simple and easy to understand\n3) The experimental results are extensive. All claims are backed up by experimental evidence. Ablation experiments give further insights", "weaknesses": "I don't really see any weaknesses in this paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tOoSgZEDQz", "reviewer": "Reviewer_iJDt", "summary": "This paper propose a way of reparametrization of the low-rank matrices in a LoRA method, by freezing a single pair of randomly initialized matrices shared across all adapted layers, and introducing trainable scaling vectors that allow for layer-wise adaptation.", "questions": "- Are there any theoretical proof that random matrices with scaling vectors would work better than LoRA?\n\n- Consider evaluating the method on Generative tasks?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Introduced a simple but elegant way to reduce trainable parameters in a LoRA. This effectively reduces storage cost due to introduced parameters for different downstream tasks.", "weaknesses": "- Performance can be hurted using this method, even the evaluation is only on a limited set of GLUE benchmarks. \n\n- The method should be evaluated on generative tasks as well, including translation and question answering. Classification task and summarization task are known to be more resilient to the lose of model capacity or trainable parameters. \n\n- The paper is not well written. Figure 3 and Figure 4 are each on a single but different task. Why not reporting a mean score on GLUE?\n\n- The paper lacks explanation on why Table 3, with fewer tunable parameters, ELoRA has better instruction finetuning scores than LoRA. It is a bit of counterintuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CFmZ5JdSeS", "reviewer": "Reviewer_rHtj", "summary": "> **TL;DR:** The proposed ELoRA algorithm achieves a 10x reduction in trainable parameters compared to LoRA, while maintaining performance levels. This paper should be accepted in its current form. Addressing my concerns and questions would improve my score.\n\nThe paper introduces an efficient finetuning method, ELoRA, for pretrained language models, addressing the storage challenge when deploying numerous adapted models. ELoRA achieves a 10x reduction in trainable parameters compared to LoRA, while maintaining performance levels, as demonstrated on the GLUE benchmark. It leverages shared random low-rank matrices and small layer-wise trainable scaling vectors to reduce memory usage. This approach is ideal for scenarios requiring multiple finetuned models, such as cloud-based personalized AI services, by improving serving efficiency. The method's potential application in various domains and further optimizations like dynamic parameter allocation are areas for future research.", "questions": "* **Q.1.** In Table 2 for the RTE dataset the LoRA algorithm achieves significantly higher results on the RoBERTa-base model compared to the the other algorithms. Is this a mistake?\n* **Q.2.** Why are the PEFT algorithms achieving higher accuracy compared FT on the RTE dataset? Is this just overfitting?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* **S.1.** The paper is well written and is easy to follow. The illustrations and results are informative and clear.\n* **S.2.** The proposed ELoRA algorithm seems novel and tackles an important problem.\n* **S.3.** The experiments are conducted on multiple datasets and architectures with 5 different random seeds. ELORA achieves high accuracy while requiring significantly less parameters compared to existing algorithms.\n* **S.4.** The experiments are thorough and an ablation study is conducted.", "weaknesses": "* **W.1.** The experiments are conducted solely on NLP tasks with LLMs. Adding experiments on different domains such as diffusions on CV would help.\n* **W.2.** The paper does not provide a theoretical analysis on the ELoRA algorithm or its limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)