VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections
Authors: Roy Miles
Pradyumna Reddy
Ismail Elezi
Jiankang Deng
Venue: NeurIPS 2024 poster (2024)
Keywords: parameter efficient fine-tuning
memory efficient training
Primary area: optimization_for_deep_networks
TL;DR: Memory efficient fine-tuning by compressing the saved intermediate activations. Complimentary to peft and generalises to pre-training.

Abstract: Large language models (LLMs) have recently emerged as powerful tools for tackling many language-processing tasks. Despite their success, training and fine-tuning these models is still far too computationally and memory intensive. In this paper, we identify and characterise the important components needed for effective model convergence using gradient descent. In doing so we find that the intermediate activations used to implement backpropagation can be excessively compressed without incurring any degradation in performance. This result leads us to a cheap and memory-efficient algorithm for both fine-tuning and pre-training LLMs. The proposed algorithm simply divides the tokens up into smaller sub-tokens before projecting them onto a fixed 1-dimensional subspace during the forward pass. These features are then coarsely reconstructed during the backward pass to implement the update rules. We confirm the effectiveness of our algorithm as being complimentary to many state-of-the-art PEFT methods on the VTAB-1k fine-tuning benchmark. Furthermore, we outperform QLoRA for fine-tuning LLaMA and show competitive performance against other memory-efficient pre-training methods on the large-scale C4 dataset.

VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections
Roy Miles*
Pradyumna Reddy
Ismail Elezi'
Jiankang Deng
Huawei Noah's 's Ark Lab
Abstract
Large language models (LLMs) have recently emerged as powerful tools for tackling many language-processing tasks. Despite their success, training and fine- tuning these models is still far too computationally and memory intensive. In this paper, identify and characterise the important components needed for effective model convergence using gradient descent. In doing so we find that the intermediate activations used implement backpropagation can be excessively compressed without incurring any degradation in performance. This result leads us to a cheap and memory- algorithm for both fine-tuning and pre-trac pre-training LLMs. The proposed algorithm simply divides the tokens up into smaller sub-tokens before projecting them onto fixed I-dimensional subspace during the forward pass. These features are then coarsely reconstructed during the backward pass to implement the update rules. We confirm the effectiveness of our algorithm as being complimentary to many state-of-thh-an PEFT methods on the VTAB- k fine-tuning benchmark. Furthermore, we outperform QLoRA for fine-tuning LLaMA and show competitive performance against other memory-efficiees pre-training methods the large-scale C4 dataset. Code: httpp://gthhbb.oom/mmmmm///////
Introduction
Large language models (LLMs) have achieved remarkable success wide range of natural language processing tasks [31 47 2). However, training these massive deep learning models remains computationally expensive, requiring vast amounts of data, compute, and memory resources. A key bottleneck for training or adapting these models is the large memory needed to store all the intermediate features required compute the gradients for optimization. This makes it challenging fully leverage the scalability and performance gains promised by larger models on currently available hardware.
Several techniques have been proposed to reduce the memory requirements, such as GaLore [54] gradient checkpointing [8), reversible backpropagation [14, parameter-efficient finetuning [8] quantization [ and activation offloading (30]. GaLore uses low-rank projection of the gradients during training to reduce the memory footprint. Gradient checkpointing reduces the activation memory demands by recomputing the activations in the backward pass instead of storing them. While these methods are promising and lower the memory cost. they also might introduce substantial computational overhead, are limited in their memory savings, or require specialized hardware [ Knowing that compute is the primary mover for the advancements in machine learning [46), it is to be expected that the LLM sizes will continue growing exponentially. Thus, is imperative to develop more efficient and scalable methods that allow better utilization of compute power and training data.
In this work, we present a novel approach for efficient training and finetuning, which we name Vector projected LoRA (VeLoRA). Our approach is based on a key observation that the intermediate
Corresponding authors: roy.miles huawei.com, ismail.elzziee hhuawei.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

y
(a) Coomneter-Effiiiens Finetuning
y
(b) VeLoRA
update rule
update rule
forward pess
06 by
Uw
Dy
LoRA, Hydra, etc.
LoRA, Hydra,
W
W
Adapter
Adapter
grad input
PC
badlward
grad
pass
Z
Z
Figure 1: The memory overhead for backpropagation on single layer consists of storing the intermediate activations and the weights. (a) demonstrates that PEFT methods can reduce the memory by using cheap low-rank adapters. (b) VeLoRA additionally compresses the saved intermediate activations further reduce the memory usage.
activations produced during the forward propagation of deep neural networks, and kept in memory for computing the gradients during backpropagation, can be effectively represented and reconstructed from single and fixed one-dimensional vector without losing any accuracy. This compressed representation can be made very memory efficient, with a controllable hyperparameter that provides trade-off between the compression ratio and the model's performance. By compressing and then reconstructing the activations on the fly, our method reduces the peak activation memory footprint to tiny fraction of what is required to store the original activations. This enables fitting much larger models into limited GPU memory compared to approaches like GaLore or gradient checkpointing.
More concretely, during the forward pass, we propose to divide each input token into set of much smaller sub-tokens. Using a single projection vector, we then project these individual sub-tokens onto minddimnnioonl subspace. Importantly, vector cheaply using first-order batch statistics and then keep it fixed throughout training. We then reconstruct the original tokens using the same vector during the backward pass. Although this reconstruction loses the original gradient properties such as the direction or magnitude, we find that jointly encourages sparsity and locally preserves the gradient similarity, which we attribute the overall effectiveness of the algorithm. By storing these compact representations, we can substantially reduce the memory footprint of the network during training, enabling the accommodation of larger models on hardware with limited memory capacity.
notice that can initialize this projection
Our contributions are the following:
We propose novel compression method that reduces the memory requirement for gradient computation during training and fine-tuning of large neural network models like L.L.s.
We show that, unlike other methods, our compression method does not need expensive operations such as SVD and gradient checkpointing.
We achieve state-of the-art results in various benchmarks while requiring lower GPU mem- ory compared to the baseline methods.
2
Related Work
Memory-Efficien Training. The increase in model size has necessitated the development of smart methods that make training more memory efficient. Gradient checkpointing [8] significantly lowers the memory requirements during model training by recomputing activations for the backward pass instead of storing them during the forward pass. However, doing so increases the training time from the need to re-compute gradients. Adafactor [1] and its followup [I] lowers the memory by working with the row-column outer-produc factorized moments of adaptive optimizers. LOMO [30] was developed for large models and works by fusing the gradient computation and the parameter update in one step to reduce memory usage, effectively only saving the current layer gradients memory. Recently, Gore [54] proposed projecting the gradients onto a lower-dimensional space [9 6). and can reduce the memory during both pre-training and finetuning. However, they store all the full intermediate activations to compute gradient updates. Their memory advantage is derived from

---

computing the first and recond-order statistics of the gradients in a lower- siwer-dimensional space, thus limiting it to second-order optimizers only. Furthermore, GaLore needs an expensive SVD operation that introduces some significant overheads in terms of both memory and computation costs. Unlike these methods, VeL.oR does not introduce any large computation overhead while the same time comes with significant memory reduction. Furthermore, VeLoRA is in principle independent of the underlying optimizer.
Parameter-EAiie Efficient Fine- Fine-Tuning (PEFT) is an emerging field that focuses on fine-tuning a large model with minimal number of trainable parameters. This typically involves freezing and then augmenting the original model with adapter modules. LoRA (Low-Rank Adaptation) (18) is a technique that optimizes few rank-decomposed weight matrices during fine-tuning, rather than updating the entire set of pre-trained weights for each attention layer. This approach substantially reduces the number of trainable parameters, thereby accelerating the fine-tuning process and making it more memory-efficien in Transformers 21 13 Several other methods built upon these works improving the capacity or performance of the model [21] [27 7 42 335 522 49 26 45 [7]. These works can be well- complemented with quantization, further reducing the memory while keeping the performance
The method was later extended to also work with multi-layer perceptrons
[10 24). Our memory-efficient algorithm is complementary to PEFT and can be used to provide
additional memory efficiency in the fine-tuning regime.
Subspace training In 22 15], the authors show that most of the learning process occurs within a significantly low-rank parameter space and that model weights can be effectively optimized within this lower-dimensional subspace. These subspace learning techniques have been widely adopted in various machine learning domains, including meta-learning (23) and continual learning (5). However, unlike VeLoRA, resource-efficient often resulting in an overhead to meet other requirements.
soaninn/fiinetunnng is not the focus of these methods, therefore,
Gradient Sparsification. Recently, there has been a surge in interest for memory- training methods. In [50] only sparse subset of the gradient vector components stored zeroing out the remaining components. Different criteria have been proposed for selecting which gradient components retain, such as Top-K K SGD (43 which keeps only the top-k largest components, Sparsified-SGD 44 and various other sparsification methods (37
28 [6]. More recently, techniques combining quantization and sparsification have been proposed for resource-efficiemt training. Examples include TernGrad [51]. Qsparse-local-SCo
38 40
25
[3]. sparse ternary compression [39] (32 method which combine sparsity with quantizing gradients to just the
and the sparse-signSGD sign. A key difference is how VeLoRA compresses the intermediate activations that are used to compute gradients. Our compression algorithm is fully dense-to-dense without any pruning or sparsification of the activations. This prevents accuracy degradation issues associated with sparse updates and facilitates memory-efficient training.
18
3.0
Z;
16
equation 3
14
600
preserve similarity
stable rank
1/4
800
0.6
n2
sparsity gradient
1k
lnner vor onte features
0.4
proje (Z.)
0.0
0.0
0.2
0.6
0.8
1.0
groups
(a) sub-token stable rank
(b) gradient similarity
(c) visualising compress(: v)
Figure 2: (a) Stable rank for the input activations using different number of groups, with 1 indicating no sub-division of the tokens into smaller sub-tokens. (b) Approximate probability of the feature similarity diverging by least lt (c) visualisation the rank-1 projection of sub-tokens.
3 Method
The task. this work, we propose memory-efficintt modification to the back paok-propagation algorithm. Our primary motivation is to reduce the GPU memory footprint during training without
3

---

resorting to any hardware-specinee quantization schemes as is done with gradient checkpointing [8]
[1 and without trading compute for memory
To formalize the problem statement, let us take a step back and look at the components needed to implement back-propagation. Firstly, during the forward pass, each trainable layer in the neural network needs to store two key tensors in memory the input activations received by that layer, and the layer's trainable weight parameters. Retaining these tensors required for performing the parameter update rules and computing the input gradients. More specifically, during the backward pass, the previously stored input activations and model weights are used to calculate the gradients with respect to the weights and the input via the chain rule of differentiation (see Fig. [T. Storing both these sets of tensors comes with a significant memory overhead which scales with the model size. We focus on optimizing the cost of storing the imput activations, We do this by compressing intermediate activation vectors and then reconstructing them when the original activations are needed for gradient calculations. This is orthogonal to PEFT [18] methods which address the overhead of saving the full-precision weights in memory by introducing cheaper trainable adapters. Further, in Section 4 we show how to combine our method with PEFT methods to achieve state-of-the-ar results.
3.1 VoLoRA
Overview. Here we address the challenge of compressing intermediate activations tensors while preserving the necessary training dynamics for model convergence. Our memory-efficient algorithm consists of two components: (i) The grouping strategy divide the original high-dimensional tokens into much smaller sub-tokens; and (ii) Fixed rank-1 projections of these sub-tokens using cheap heuristically initialized principal components. Given a large pre-trained model, we apply these steps to compress the intermediate activations saved during training while preserving most of the original model's 's training dynamics. We illustrate the general overview of this pipeline in Fig T and show Py lyTorch-like pseudo-code in Algorithm |
Consider a set of input activations that need to be saved in GPU memory during the forward pass. We denote element in this set Z, [x; w) E Nx where N is the number of tokens, and D is the feature depth. We propose to introduce a simple grouping (reshape) operation that partitions the tokens across the depth dimension: group(-): Z E IR BRxNxD x - IR Bx D/M XM with M being the new size of each token, now coined sub-token. This operation can be described as partitioning batch of N tokens into a collection of these much smaller non-overlapping N ND/M M sub-tokens. Then we project each of the sub-tokens onto a rank-1 subspace. The idea is that this grouping operation enables much lower-dimensional fixed subspace to be used throughout training without any degradation in model convergence or performance. We describe the compression steps concisely as follows:
Z groupl.) E IR BxND//MX compress(: :v)
Zp E BBBDDD/MX1
(1)
where z is used to denote the sub-tokens of Z and 2p are the compressed sub-tokens. This compression achieved using the function compress(z. ; v) z v, which projects each sub-token onto a one- dimensional sub-space before saving them in memory. Since M << D it is more memory efficient to store up instead of Z. The initialization strategy for v is important for the performance of the model, however we later show that simple and cheap average over the first batch of sub-tokens can be very effective. Finally, the compressed sub-tokens Zp are reconstructed for the gradient calculation during backward pass follows:
Zp reconstruet v) E IR BxND/MXX ungroup(.)
Ż IR BXNND
Here å and Ż refer to the reconstructed sub-tokens and tokens of z and Z respectively. The reconstruct function projects the sub-tokens z, back onto the vector as very coarse recon- struction of and it is defined as reconstruct(z,. v) 2p VT. The overall compression and reconstruction operation is given as (z)jv(z) =(z v) v"", where v IR Mx1 is a fixed vector of unit length.
To summarize, during the forward pass, compress the intermediate activation tensor Z into compact representation Zp using v. Then, in the backward pass when the original activation Z is needed for gradient computation, we reconstruct an approximation Ž by projecting zp back onto

---

These steps are fundamentally different and complementary to recent works that leverage the low-rank property of gradients like GaLore [54] in two ways: Firstly, they store the uncompressed intermediate activations in memory for the gradient computation. In contrast, we compress these activations explicitly during the forward pass. Secondly, Gorere relies on periodically computing the principal components with SVD to optimally capture the underlying gradient subspace. Our compression method avoids such costly operations, making much more efficient and scalable.
3.2 Insights and properties of VeLoRA
On the importance of compressing sub- tokens. Computing the optimal low-rank sub- space of the gradients using SVD is very com- putationally and memory intensive and often needs offloading the operation to a CPU [54].
Algorithm I VeLoRA, Pytorch-like
der Cinpurd weight, v)
# v: M.M.M M
# forward conpute is preserved out Imput seight
Moreover, periodically updating the projection may be necessary to track any shift in the gra- dient distribution [54]. This is why dividing the tokens up into smaller sub-tokens is necessary. By doing so, it allows us to use cheap sur- rogate rank-1 1 projective map that is initialised and frozen throughout training. Finally, one surprising observation of this grouping opera- tion that the sub-tokee aub-tokens will naturally lie on higher-dimensional subspace than the origi- nal tokens (see figure 2 Thus, our method cannot be faithfully explained through better reconstruction of the gradients, but instead by a suppression of the inherently larger eigenvalues that can in turn help reduce overfitting.
conpate vactor similarity compress (input),
love_for_backwee veight, v)
return aut
def backvard (ctx, grtd_out 2, seight, savad_
recenstruct the input input ingroap(reconstrggg
v))
conpute gradients
grad_input grad_outpus xeight
grad_ - ortdutt a input
return grad_input. grad_veight
Why does a vector projection make sense? Using fixed vector projection throughout training fails to capture any shift in the gradients' distribution. Under the assumption that preserving the original gradient reconstruction is important, this may seem like an undesirable trait. However, quite surprisingly, we find that this shift does not hinder the model's convergence or performance at all. An explanation behind this phenomenon they shift away from the initial distribution and this helps prevent the model from overfitting to the fine-tuned dataset; (ii) Although the vector projection destroys the original gradients' magnitudes and directions, still locally preserves the gradient similarity and this similarity will govern the model's training dynamics 19]
be twofold: (i) The gradients become more sparse as
Consider a rank-1 I decomposition of two sub-tokens: 2; and 2j. We will use the dot-product as the similarity measure simm(-) -) for which we wish to locally preserve. Let assume that both z; and 2 are distributed such that the angles between them and the vector v are normally distributed with mean of 0 and a standard deviation a. With first-order approximation, the probability of the projection and reconstruction scaling the gradient similarity by at least is given as follows (see the Appendix for the full derivation):
vk
Pr (Isim(projj(())), projv sim(zi, 2;)1
Φ
(3)
With A: >> 0 and > 0, this probability is bounded by [0, 1]. Here we can see that similarity is trivially preserved in the limit as a - 0. This indicates that the sub-tokens already lie on a I-dimensional subspace spanned by To further see how these gradient similarities diverge for various values of and a, we plot equation in in Fig. D.. We empirically observe that although the gradient similarity is very dependent
the distribution of features, this non-linear relationship does not hinder the model's ability to converge and generalise. Finally, we also provide an illustrative visualisation in Fig. (right) that shows the locality sensitivity for preserving gradient similarity and the sparsification of gradients when they are orthogonal to the vector v. Both of these components are important properties that attribute to the effectiveness of VeLoRA.
Connection parameter efficient fine-tuning. Although VeLoRA is complimentrry to LoRA, can indeed be viewed under the same umbrella. First let consider LoRA, which will freeze the
5

---

original weights and only update a low-rank adapter:
Wx + ABx (W + AB)a
(4)
Following the same analysis from FLoRA [77, we will freeze A and initialise B with all zeroes. i.e. = Ag and Bo = O. The weight update rule can then be given as follows:
dL dB
LoRA W' W + Ao Bo
W
Ag
(5)
with learning rate and 9 = dL dy small learning rate assumption. In contrast, Vol.ORA (with M = D i.e. no sub-tokens) will update the original weights directly but with compressed gradients:
dy dW see FLoRA 17 for the original full derivation under the
dL dL dW
(dy dW (aV
(dVV)))))
dy dW
vo
(6)
dy
dy
which leads the following similar weight update rule equation
VeLoRA W' = W 1) dW
ngov"
This result highlights that VeL.RA is special case LoRA with data-driven initialisation for Ag- Furthermore, due its construction, VoR is implemented using custom forward and backward operation rather than by modifying the architecture and fusing weights after training. Finally, VeLoRA also provides additional compression through having smaller shared projection for each sub-token. This would resemble reshaping the input tensor before and after the LoRA adapter to enable smaller projection matrices.
Cheap initialisation strategies. GaLore proposes to use the periodically updated SVD principle components to track any shifts in the underlying sub-space of the gradients. Unfortunately. this SVD operation can be very expensive both in terms of memory and compute. Another limitation is that SVD may fail to converge, and it requires casting the features back to 32-bit for numerical stability For this reason, we propose a cheap initialisation strategy. We relax the constraint on tracking the feature distribution. For all of our experiments, we use rank-1 decomposition of sub-tokens and propose to initialize the vector v using the average of sub-tokens from the first batch.
Comparison with the state-of-the-art
In this section, thoroughly evaluate VeLoRA and its individual components. In section 4.2 demonstrate the complementary nature VeLoRA in conjunction with many other existing PEFT methods. Section 4.4 then scales up these results the LLaMA models, whereby significant memory reduction when coupled with 4-bit scalar quantisation. Finally, section (.5)eeend: extends VeLoRA the pre-training setting where we competitive performance alongside real reduction for the on-device GPU memory usage.
achieve
For the VTAB-IK Ik experiments, adapters, while for SSF we applied it to the input scale and shift layers only. For all the other experiments simply applied VeLoRA to the value projection layer and the down projection layer of the MLP blocks.
applied VeLoRA to all the down projections in the trainable
4.1 Implementation details
We performed all the vision experiments subset of the VTAB-Ik 53 benchmark for combina- tion of different vision datasets. We finetuned a ViT-B [12] model pre-trained on ImageNet-21K using the AdamW optimizer with a learning rate of 5e-4 and weight decay of le-4, We performed

---

Table Results on subset of the VTAB-Ik benchmark. All methods use TTT-Base-2244 model
pre-trained on ImageNet-2Ik. The batch sizes and ranks are the same across all tasks.
Natural
Specialized
Structured
Menoay (MB)
Caleshini
Ci
DTD
Plower
Sunt0
EMOGAT
Reinopuby
Clou-Ceart
DMLLib
KITT-DOA
SNORB Azim
SNORB Be
Averge
Merhod
Full uninn
4.25
89.4
53.3
66.
97.3
$7.3
90.7
39.2
33.2
95.3
86.
15.4
62.8
67.2
77.5
31.2
32,8
69.7
VeLaRA
4.02
50.9
55.9
67.8
97.3
58.4
00.4
35.9
65.8
95.8
86.7
75.7
74.7
90.2
77.9
313
31.6
71.2,7
Linear prohing
1.84
SSF
4.13
59.4
74.0
72.9
99.2
91.1
50.7
56.0
53.3
94.5
85.3
75.6
78.5
45.0
76.9
23.0
36.9
72.7
VeLoRA
3:46
74.8
78.6
75.5
24.6
Hydm
3.10
91.3
72.6
70.9
99.2
91.3
95.2
81.9
51.7
78.9
34.5
40.5
74.7
VeLoRA
2,88
91.0
72,8
70.6
99.2
91.4
59.2
$6.0
83.2
94.9
84.3
75.9
62.7
51.6
79.9
34.2
41.4
74.5
(.I)
LoRA
2.86
77.9
VeLoRA
2,74
$8.9
67.3
69.6
99.1
90.7
63.5
53.5
81.9
95.2
83.4
74.3
79.8
47.1
78.9
29.7
40.3
72.7 (U6)
Table 2: Comparison of our method with full fine-tuning, GaLore and LORA GLUE benchmark using pre-trained RoBERTa-Base. Our method reaches the best overall results while showing significant memory improvements, especially compared GaLore. We bold the best results from the considered PEFT methods. The GPU memory is measured on-device.
Memory (GB)
CoLA
STS-B
MRPC
RTE
SST2
QNLI
QQP
Avg
Full Fine-Tuning
4.64
62.24
90.92
91.30
79.42
94.57
87.18
92.33
92.28
86.28
GaLove
4.04
60.35
90.73
92.25
79.42
94.04
87.00
92.24
91.06
85.89
2.71
91.07
78.70
92.89
92.18
85.61
VeLoRA
2,23
64.56
90.81
91.26
77.98
94.38
86.29
92.09
89.91
85,91
the language models experiments on the GLUE 48 benchmark using the encoder-decoder RoBERTa transformer (29). We then scaled our model to large language models causal transformers using the LLama 2 family of models, finetuned in Alpaca dataset (36] and reported results the MMLU benchmark 33. We finally applied our method to the pre-training stage using some smaller LLama (2) models on the C4 dataset (34). All our models were trained using the AdamW optimizer with a learning rate of le-3 and a weight decay of O. We give all the other major hyper-parameters to replicate these experiments in the Appendix and also in the code.
4.2 Vision experiments
We conduct experiments evaluating the performance of VeLoRA for full-tuning and how it comple- ments other PEFT methods. In Tab. we reproduce large set of results for LoRA [18] SSF 227 and Hydra 21 on a subset of the VTAB-1K benchmark, where the sub-token size for each experiment is given in the Appendix. Unlike what is more common in the PEFT literature 220 21 do not perform any task-specific hyperparameter tuning that would change the memory, such as batch size and rank, and to also avoid any potential overfitting to the specific task. For all experiments we used the authors provided implementations for the adapters and integrated them into the same training framework for a fair comparison. We observe that VeLoRA improves the performance compared to full-tuning by 1.5 percentage points (pp), while lowering the memory requirements. We also observe that when combined with PEFT methods, VeLoRA can come with improvement in memory and performance. VeLoRA lowers the memory requirement of SSF 27 by 16% with only a minor degradation pp.) in accuracy. It lowers the memory requirement of Hydra [21 by 7% while improving the accuracy by 0. 0.1pp. Finally, it lowers the memory requirements of LoRA [8] by 4% while improving the accuracy by 0.6pp.
4.3 Roberta experiments
We now evaluate our method with M = 16 on various language tasks, using RoBERTa- Base 29 in the GLUE benchmark, and compare with full fine-tuning, LoRA [8] and GaLore [54,, presenting the results in Tab. 2 We observe that both GaL and LoRA lower the memory requirements compared to fine-tuning from 4.64GB to 4.04GB, respectively to 2.71GB, at a cost of accuracy degradation with GaLore performance lowered by 0.39 pp. while LoRA accuracy drops by 0.67 pp.
7

---

Table Mean 5-shot MMLU test accuracy for LLaMA models finetuned with adapters on Alpaca. The GPU memory estimate consists of the frozen weights, trainable adapters, and input activations.
LLaMA Size
7B
13B
Mean
Method
Alpaca
Memory
Alpaca
Memory
LoRA w/ BFloat16
38.4
8.79
47.2
15.82
42.8
LoRA w/ Float4
37.2
5.77
47.3
9.91
42.3
QLoRA
39.0
5.77
47.5
9.91
43.3
VeLoRA
39.5
4.88
48.0
8.48
43.8
Our method further reduces the memory needed for training to 2.23GB, an improvement of 18% compared to LoRA, and 45% compared to GaLore, while still reaching higher results than either of them. More impressively. VeLoRA reduces the memory by half compared to full fine-tuning with an accuracy degradation of only 0.37 pp. reaching the best tradcoff between memory and accuracy.
4.4 Scaling up to LLaMA
We now scale our method to large language mod- els, demonstrating the effectiveness of VcLoRA in finetuning them. We do comparisons with LoRA on both BFloat16 and Float4, in addition to the recent method of QLoRA [1] which is widely used for fine-tuning LLMs with very low memory budget. We aim to further lower this budget, showing in the process that VeLoRA is also complementary
Table 4: Comparison with low-rank algorithms pre-training various sizes of LLaMA models on C4 dataset. Validation perplexity is reported, along with the on-device GPU memory usage.
60M
130M
Full-Rank
33.52 (1,30G)
25.08 (2.32G)
QLoRA, resulting in much lower memory consumption. We present the results in Tab. 4.4 using M 32 for 7B and M = 128 for 13B. We can see that our method outperforms QLoRA by 0.5pp the Llama model, while reaching a massive perfor- mance increase compared to LoRA models. Fur- thermore, we reach this performance improvement, while at the same time further reducing the memory. particular, we reduce the memory for 0.89GB, a relative improvement 15.4% from QLoRA. We observe that this performance improvement is maintained on the larger model of 13B parameters, where again our method outperforms QLoRA by 0.5pp and lowers the memory requirements by 1.43GB, relative improvement of 14.4%.
GaLore
34.88 (1.27G)
25.36 (2.02G)
LoRA FLoRA VeLoRA
34.99 (0.86G)
33.92 (1.24G)
34.35 (1.27G)
25.88 (2.01G)
33.76 (1.18G)
25.29 (1.83))
r/dmoded
128 / 256
256 768
Training Tokens
1.1B
2.2B
4.5 Pre-training on C4
We now perform an experiment where we use VeLoRA to train language models from scratch in the large C4-English dataset, presenting the results in Tab. 4 We use M 128 for both the 60M and I 130M. while following the same training pipeline and evaluation as GaLore and comparison with LoRA. However, unlike in the GaLore paper [54], which estimates the memory usage using the optimizer weights and memory alone, we choose to compute the real on-device memory. This quantity would take into account the cost of additionally storing the intermediate activations and also highlight the benefits of LoRA in terms of memory since the base weights will be frozen. In contrast to other experiments, our use of VeLoRA here is not with any additional adapters and is simply compressing the input activations for the original trainable base layers. We observe that our method significantly outperforms the other methods, reaching 1.08 pp lower perplexity than GaLore. We observe that our method outperforms GaLore in Llama- 130M too.
Ablations Studies
5.1 Convergence properties
We observed that the rank-1 projections would encourage much higher levels of gradient sparsity (see Fig. 2b). A natural question to ask from this observation is if the gradient sparsity will come at the cost

---

Table 5: All three ablations are done using LLama-7B model. (a) VeLoRA has no loss performance when trained for fewer or more training epochs than QLoRA despite both reducing the memory footprint. (b) Importance in choosing the correct number of sub-token size to find an optimal memory v.8. accuracy trade-off. Using GPU memory estimate for the input activations only. (c) Ablating various initialisation strategies for rank-1 projection and with M
/32.
Epochs
QLoRA
VeLoRA
M
Memory (MB)
Acc
Method
Acc
36.4
36.7
D/6e 64
865
37.9
Random
36.8
37.3
37.5
D/
808
39.5
SVD
37.1
38.4
38.1
D/
779
39.3
Fixed average
39.5
39.1
39.5
D/8
764
37.2
Running average
38,9
training convergence
(b) sub-token size
initialisation strategy
of the model's convergence. In other words, we want to verify if our model needs to be trained longer compensate for the gradient compression. To do so, we evaluate the performance of our model at the end of each epoch and compare it with the performance of a competing model, the QLoRA. As shown Tab. bb. VeLoRA and QLoRA improve at the same rate. Our model outperforms QLoRA by 0.3pp by the end of the first epoch, and keeps this improvement rate, outperforming QLoRA by 0.4pp the end of the training. In this way, we verify that the additional compression of input activations does not affect the model's convergence.
5.2 Sub-token size
We provide an ablation on the impact on the size of each sub-token and the model's performance, showing the results in Tab. 5p. We can see that there is a sweet spot for which the rank-1 projections of sub-tokens using an average initialisation is very effective both in terms of memory and accuracy. However, the size of the sub-tokens is too large (i.e. when M D/8), the gradients will become too sparse, which will hinder performance (see also figure _p). In contrast, if the size of each sub-token is too small, for example with D/64, there a more significant memory compression but at the cost of model performance.
5.3 Initialisation strategy
In Tab. 5p. we show the performance after training with various ways of initialising the vectors for each group. To avoid exceeding memory requirements we sub-sampled the tokens for SVD and also consider the case of instance- wise initialisation. Although we would have expected better performance since the vector will always align with each incoming batch, we found that did not lead to any performance improvement. In contrast, doing SVD initialization comes with drop in performance. This result further confirms that the performance improvement from VeLoRA is not specifically correlated with lower reconstruction error of the input activations.
5.4 Choice of layers
Table 6: Memory v.s. accuracy trade-off for VeLoRA on different layers. We use LLaMA-7B trained alpaca and evaluated on MMLU. We report the GPU memory estimate from the input activations only.
A key design decision for many Parameter-Efficient Fine-Tuning (PEFT) methods, including LoRA, where to in- sert the adapters within the model layers. Since Vel ooRA shares this core architec- tural choice, we aim provide stronger intuition on VeLoRA's suitability by per- forming thorough ablation study ana- lyzing the trade-offs between memory consumption and accuracy when con- sidering different layer placements. We present the results Tab. [6] We observe that we achieve memory improvement in all cases where we adopt Vel.RA. How- ever, to improve the accuracy, VeLoRA
Query
Key
Value
Down
Memory (GB)
Acc
none
1.67
38.1
1.42
36.2
1 1.42
36.2
1.42
36.7
1.01
38.9
1.18
37.4
0.76
39.5
0.51
38.4
0.24
37.0

---

must be used in the MLP down projection. A possible explanation is that this layer might suffer from overfitting on the training data forgetting 4 from the pre-trained data. Overall, we conclude that applying VeL.RA A to the value and down projection appears to be the best choice. We strengthen this claim by using this setting for all other experiments.
5.5
Comparison with gradient checkpointing
We compare VeLoRA to gradient checkpointing, another widely used technique reduce the memory consumption during training. While both methods aim to minimize memory overhead, gradient checkpointing achieves this by recomputing the original activations during the backward pass, leading to reduced memory consumption at the cost of additional compute. In contrast, VeLoRA uses lossy compression of the activations during the forward pass and then reconstructs them in a coarser manner during the backwards, thus avoiding the need for any expensive recomputation. Our results in table 7 show that VeLoRA not only offers comparable reduction memory usage but also leads to faster training times compared to gradient checkpointing, as it reduces the recomputation burden and overhead.
Table 7: On-device training time and memory costs for pre-training LLaMA. Unlike VeLoRA. gradient checkpointing incurs much more significant training time overhead. Batch size of 1. Our method is 17%, 30%, 30%, 47% faster than gradient checkpointing in LLama 60M, 130M, 7B and 13B. We see that the larger the model, the larger the time performance gain.
(a) C4 pre-truining
(b) Alpacu fine-tuning
60M
130M
13B
Method
it/s
memory (GB)
it/s
memory (GB)
it/s
memory (GB)
it/s
memory (GB)
Full
3.12
1.40
2.32
1.64
13.64
1.24
21.35
Gradient Checkpoint VeLoRA
2.47
1.19
1.03
1.85
1.14
7.31
0.78
11.72
2.90
1.18
1.34
1.83
1.50
8.35
1.15
13.28
6 Conclusion
In this work, proposed VeLoRA, novel framework that enables the training of networks, including large language models in a highly memory-efficiee manner. Our approach compresses intermediate activations during the forward pass and coarsely reconstructs them during backpropagation. VeLoRA complements PEFT methods and able to significantly reduce memory requirements while improving the performance. VeLoRA is effective when tested in both moderately-sizer vision transformers as well as large language models. We performed experiments to demonstrate the method's effectiveness on VTAB-IK, MMLU, GLUE, and C4 benchmarks outperforming state-of-the-art methods such as LoRA, QLoRA or GaLore.
Limitations and Broader Impact
Limitations. We performed all experiments Transformer models. Although Transformers have become dominant in machine learning and computer vision, there are other important deep learning networks such as CNNs, RNNs and SSMs. It remains unclear whether our methods can be extended to non- conn--ransffrmr--baaa although our method is computationally lies in the substantial reduction of GPU memory. However, the issue of training time still persists.
models and how such an extension could be accomplished. Moreover,
efficient than competing methods, its primary advantage
Broader Impact. As compute power grows exponentially, model sizes grow even faster, making challenging for smaller institutions, especially academic ones, to conduct high-quality research. This work aims democratize AI research, particularly in large language models, by significantly reducing the memory needed for training, enabling researchers with limited compute resources to train networks and contribute to their research. However, the democratization of Al controversial, with leading institutions like OpenAl, Anthropic, and Google DeepMind becoming more closed due the potential risks of L.MSs in the wrong hands. We acknowledge this concern and do not endorse misuse of our research.
10

---

References
[1] R. Anil, V. Gupta, I. Koren, and Y. Singer. Memory efficient adaptive optimization. In NeurIPS, 2019.
[2] R. Anil, S. Borgeaud, Y. Wu, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth,
K. Millican, D. Silver, S. Petrov, M. Johnson, I. Antonoglou, J. Schrittwieser, A. Glaese, J. Chen,
R. Pitler, T. P. Lillicrap, A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. R. Barham, T. Hennigan, B. Lee, F. Viola, M. Reynolds, Y. Xu, R. Doherty, E. Collins, C. Meyer, E. Rutherford, E. Moreira, K. Ayoub, M. Goel, G. Tucker, E. Piqueras, M. Krikun, I. Barr, N. Savinov, I. Danihelka, B. Roelofs, A. White, A. Andreassen, T. von Glehn, L. Yagati, M. Kazemi, L. Gonzalez, M. Khalman, J. Sygnowski, and al. Gemini: A family of highly capable multimodal models. CoRR, 2023.
[3] D. Basu, D. Data, C. Karakus, and S. Diggavi. Qsparse-local-sgd: Distributed sgd with quantization, sparsification and local computations. In NeurlPS, 2019.
[4] D. Biderman, J. G. Ortiz, Portes, M. Paul, Greengard, C. Jennings, D. King, S. Havens, V. Chiley, J. Frankle, C. Blakeney, and J. P. Cunningham. Lora learns less and forgets less. TMLR, 2024.
[5] A. Chaudhry, Khan, Dokania, and Torr. Continual learning in low-rank orthogonal subspaces. In NeurlPS, 2020.
[6] H. Chen, G. Raskutti, and M. Yuan. Non-convex projected gradient descent for generalized low-rank tensor regression. Mach. Learn. Res., 2019.
[7] S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurlPS, 2022.
[8] T. Chen, B. Xu, C. Zhang, and C. Guestrin. Training deep nets with sublinear memory cost. CoRR, abs/1 0ess/1604.66177 2016.
[9] Y. Chen and M. J. Wainwright. Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees. CoRR, 2015.
[10] T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer. 8-bit optimizers via bise quantization. In ICLR, 2022.
[11] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized lims. NeurlPS, 2023.
[12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[13] V. Fomenko, H. Yu, Lee, S. Hsich, and W. Chen. A note on lora. CoRR, 2024.
[14] A. N. Gomez, M. Ren, R. Urtasun, and R. B. Grosse. The reversible residual network: Back- propagation without storing activations. In NewrIPS, 2017.
[15] G. Gur-Ari, D. A. Roberts, and E. Dyer. Gradient descent happens in tiny subspace. arXiv preprint, 2018.
[16] P. Han, S. Wang, and K. K. Leung. Adaptive gradient sparsification for efficient federated learning: An online learning approach. In Conference on distributed computing systems (ICDCS), 2020.
[17] Y. Hao, Y. Cao, and L. Mou. Flora: Low-rank adapters are secretly gradient compressors, 2024.
[18] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation large language models. In ICLR, 2022.
[19] A. Jacot, Gabriel, and Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In NewrIPS, 2020.
11

---

[20] S. Jie and Z.-H. Deng. Fact: Factor- tunoirttunng for lightweight adaptation on vision transformer. AAAI, 2023.
[21] Kim, H. Yang, Y. Kim, Y. Hong, and Park. Hydra: Multi-head low-rank adaptation for parameter efficient fine-tuning, arXiv preprini, 2023.
[22] B. W. Larsen, S. Fort, N. Becker, and S. Ganguli. How many degrees of freedom do need to train deep networks: a loss landscape perspective. In ICLR, 2022.
[23] Y. Lee and S. Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In ICMI., 2018.
[24] B. Li, Chen, and Zhu. Memory efficient optimizers with 4-bit states. In ICLR, 2023.
[25] Li and T. Hoefler. Near-optimal sparse allreduce for distributed deep learning. In Symposium on Principles and Practice of Parallel Programming. 2022.
[26] V. Lialin, N. Shivagunde, S. Muckatira, and A. Rumshisky, Relora: High-ram training through low-rank updates. ICLR, 2024.
[27] D. Lian, D. Zhou, J. Feng, and X. Wang. Scaling & shifting your features: A new baseline for efficient model tuning. In NeurIPS, 2022.
[28] Y. Lin, S. Han, H. Mao, Y. Wang, and B. Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In ICLR, 2018.
[29] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer and Stoyanov. Roberta: robustly optimized BERT pretraining approach. arXiv preprint, 2019.
[30] K. Lv, Yang, T. Liu, Q. Gao, Q. Guo, and Qiu. Full parameter fine-tuning for large language models with limited resources. CoRR, abs/2306.09982, 2023.
[31] OpenAL. GPT-4 technical report. CoRR, 2023.
[32] C. Park and N. Lee. 53 gdgd-mv: Sparse-signsgd with majority vote for communication-effiiien distributed learning. In IEEE International Symposium on Information Theory, ISIT, 2023.
[33] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. Vito,o Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in Py PoTchh.. 2017.
[34] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 2020.
[35] A. Renduchintala, T. Konuk, and O. Kuchaiev. Tied-lora: Enhacing parameter efficiency of lora with weight tying. ACL, 2024.
[36] T. Rohan, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-follovingg llama model. Technical report, 2023.
[37] D. Rothchild, A. Panda, E. Ullah, N. Ivkin, Stoica, V. Braverman, J. Gonzalez, and Arora. Fetchsgd: Communication-efficini federated learning with sketching. In ICLR, 2020.
[38] A. Sahu, A. Dutta, A. M Abdelmoniem, T. Banerjee, M. Canini, and P. Kalnis. Rethinking gradient sparsification as total error minimization. In NeurlPS, 2021.
[39] F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek. Robust and communication-efficccnn federated learning from non-iid data. IEEE transactions on neural networks and learning systems, 2019.
[40] A. Shanbhag, H. Pirk, and S. Madden. Efficient top-k query processing massively parallel hardware. International Conference on Management of Data, 2018.
[41] N. Shazeer and M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML, 2018
12

---

[42] Y. Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng. K. Keutzer, J. E. Gonzalez, and I. Stoica. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint, 2023.
[43] S. Shi, X. Chu, K. C. Cheung, and S. See. Understanding top-k sparsification in distributed deep learning. arXiv preprint, 2019.
[44] S. U. Stich, J.-B. Cordonnier, and M. Jaggi. Sparsified sgd with memory. In NeurlPS, 2018.
[45] Y. Sung, J. Cho, and M. Bansal. VL.-ADAPTER: parameter-efficient transfer learning for vision-and-languae tasks. In CVPR, 2022.
[46] R. Sutton. The bitter lesson. http://hhgghhooommmaa/2/9111//12 13-th-1 2019.
[47] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, 2023.
[48] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
[49] Y. Wang, Y. Lin, X. Zeng, and G. Zhang. Multilora: Democratizing lora for better multi-task learning. CoRR, 2023.
[50] Wangni, J. Wang. J. Liu, and Zhang. Gradient sparsification for communication-efficienn distributed optimization. In NeurIPS, 2018.
[51] W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In NeurlPS, 2017.
[52] W. Xia, C. Qin, and E. Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning. CoRR, 2024.
[53] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic. Djolonga, A. Pinto, M. Neumann, A. Dosovitskiy, L. Beyer, O. Bachem, M. Tschannen, M. Michalski, O. Bousquet, S. Gelly, and N. Houlsby. The visual task adaptation benchmark. CoRR, 2019.
[54] Zhao, 7. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Tian. Galore: Memory-efficient Ilm training by gradient low-rank projection. In ICML, 2024.
13

---

7 Supplementary Material
7.1 Memory Estimates
this section, we provide a detailed explanation of the memory estimates presented in the paper. For the ablation experiments in Tables Sp and we only estimate the memory consumption allocated for storing the intermediate activations. We do this because all the other memory allocations (weights, optimizer states, etc.) would be the same for each experiment. The reported memory estimates will highlight the impact of each design choice or parameter on the memory of the intermediate activations, which one of the focuses of this paper.
Additionally, for the QLoAA Llama experiments in Tab. we also use the estimated memory, but we compute this from three components: (i) Saving the frozen base weights in memory, these weights are what are quantized; (ii) The trainable adapter weights, which are stored in fp16 format for all methods; and (iii) The memory cost of storing the input features for each trainable layer. We acknowledge that an efficient implementation our method with 4-bit quantization left for future work. However, for all other experiments using fpl6, we want to highlight that we report the real on-device GPU memory usage.
7.2 Roberta Experiments
We fine-tune the pre-trained RoBERTa- Base model on the GLUE benchmark. For the training parameters, we follow the same settings as GaLore [54] which given in table
Table 8: Hyperparameters of fine-tuning RoBERTa base.
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Batch Size
16
16
16
32
16
16
16
16
# Epochs
30
30
30
30
30
30
30
30
Learning Rate
IE-05
IE-05
3E-05
3E-05
1E-05
IE-05
IE-05
IE-05
Max Seq. Len.
512
512
512
512
512
512
512
512
7.3 VTAB-1K Experiments
We reproduce all the reported PEFT methods under fixed rank and size setting across tasks. This is to ensure that the memory usage is fixed for all the tasks. However, for Hydra (21) we do use the optimal scale and dropout parameters provided the authors (see table The rank v.s. memory for each reported training run is provided in table T0 Here we were able to use a higher rank for LoRA to get better performance, while also reducing the total memory. We found that Hydra did not noticeably improve performance when increasing the rank and so for these we maintained the same rank for with and without the addition VeLoRA.
Table 9: The optimal task-specific hyper-parameters proposed for the Hydra method
KITTI-Dist SNORB Azim
Retinopathy
Clevr-Count
-NORB-Ele
Caltech
Cifar100
Flower102
SVHN
Sun397
Camelyon
EuroSAT
Resisc45
DMLab
DTD
Pets
Parameter
Scale
4.0 0.1 0.1
0.1
0.1
3.0
0.1
0.1
0.1
0.1
4.5
1.5
4.5
4.5
3.0 1.0
Dropout
0.1 0.2
0.2
0.0
0.0
0.0
0.2
0.2
0.2
0.2
0.2
0.2
0.2
0.2
0.2
0.2
7.4 Implementation Details
All the experiments in sections and 4.5 were performed using NVIDIA V100 GPUs with the fp16 data type. For the LLaMA experiments in section 4.4
trained on 4 NVIDIA A100 GPUs
14

---

Table 10: The hyper-parameten in VTAB-IK experiments. These parameters are the same for all tasks to avoid any potential overfitting to specific task and to maintain a constant memory usage.
Method
Rank
Memory (GB)
Full
n/a
n/a
4.25
+ VeLoRA
n/a 32
4.02
SSF
n/a
n/a
4.13
+ VeL.RA
n/a
32
3.46
Hydra
4
n/a
2.88
+ VeL.RR
LoRA
4
n/a
2.86
+ VeLoRA
16
16
2.74
using the QLoRA nf4 data type. The LLaMA experiments were based the alpaca-lora repository while the RoBERTa and C4 experiments was based on the GaLore repository
7.5 Gradient Similarity
Using (i; 2,) |2:| |zyl Oig and projv (z.) = |z,| cos 0iv wish how much the gradient similarity is being preserved under various assumptions about the distributions of both z; and Zj. We can introduce measure for this divergence follows:
[Zz, |sim(projv projv simm(z; Zi, Zz))
(8)
|sim ((v z:)v, (v zj)v) - |z:| |2,| cos 0zyl
(9)
Since 0vv and |v] 1, we can simplify this expression as follows:
||,||cos 0zo Paul|zy||cos |cos 0zel |z,] |z,| cos 0zyl
(10)
= ||z||zz|((oos l:v cos Ajel cos 0iz]1
(11)
Without loss generality, we can assume 10jul > 10..l and use 0ig 0jv Ow.-
= =||z [|z.||z]|(/cos Ow cos 0,0l - cos (0fu 0zw))l
(12)
For simplicity, consider the case where both the input activations z, and zj also normalised to unit length. practise, these magnitudes will simply be scaling the divergence linearly.
Let both 0 and 0, be normally distributed with standard deviation and have the vector be appropriately initialised such that their means are 0. To understand how our projection degrades the gradient similarity, we will consider the probability of d(-)) exceeding some scalar k; i.e:
Pr( Pr([sim(proji.en (z.), proj.(z,)) sim 2,)1
(13)
Small a approximation For normally distributed 6, and 0j. cos(O,) and cos(0,) will not be uniformly distributed and instead follow distribution that is more concentrated around their mean values. Thus, we can use a first-order approximation of cos(-):
cos(O,) cos(O, 0,) (0; 0,)
cos(O.) 1,
(14)
2
Therefore, we can express d(-) and Pr(d(-) k) as follows:
thtpps//githuu.cooo// cowps///...oooo///a/acccooooo hhnpss//gghuo.commjjjweiiaaaaaa
15

---

d(z ky: (0, -8,)
(15)
Pr(d(z,, 2j: k) v Pr( (0: 0,) > k)
(16)
= Pr((0: 0,) > 2k)
(17)
Pr(10: - 0 V2k)
(18)
Since 0, -0 0 normally distributed with variance 20 have:
2Pr(0, -0 22k
(19)
Finally, using the cumulative distribution function (CDF) of the normal distribution:
V2k
k
Pr(0; - 0, > v2k) $
+
(20)
/20
a
Vk
Pr(diz, zj:v) k)
2
2
(21)
σ
16

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract reflects the paper's 's contributions.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We have a section names Limitations where we discuss the limitations of our paper.
Justification:
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide the full derivation showing the divergence of gradient similarities between the original activations and our proposed representation. In this derivation we provide all the assumptions and show the full steps in the supplementary. For the connection LoRA we use one of the main results given in FLoRA, while showing all the other steps in the main paper.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the code and dataset information to make the paper reproducible. In addition, we explain our approach and give the hyperparameters.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code, and explain the datasets we used. All datasets are public.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we give all the training details of the method.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Most of our experiments are on large-scale datasets and tasks. Each model is trained until convergence and we have observed very little variation in performance across multiple runs.
8. Experiments Compute Resources
17

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We discuss the compute resources used in section 4.1 and provide the code and checkpoints to reproduce all of the main experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwwwpp.cccc///////ttieea
Answer: [Yes]
Justification: Yes, we obey NeurIPS' code of ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both the potential positive and negative societal impacts of our method in section called Broader Impact.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification: Unfortunately, safeguarding against negative usage of LLMs is an open- problem. While we do not endorse any inappropriate usage of our method, we acknowledge that malicious people might benefit from training large networks with limited computational resources.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all the datasets and code (e.g., LLama) we use in the paper
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: We have released the code under an MIT license.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
18

---

Answer: [NA]
Justification: We do not use any experiments that require crowdsourcing or human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not use any experiments that require crowdsourcing or human subjects.
19


Reviews:
[{"review_id": "kmLwdgG84Y", "reviewer": "Reviewer_sd52", "summary": "In this paper, the authors propose to compress the activations by down-projecting the input tensors with a vector to save memory for large-scale training. The empirical results show that their proposed method, VeLoRa, achieves better performance compared with previous work.\n\nStrengths:\n\n1. The topic of efficient training is important, given that the models are scaled up at a rapid rate.\n2. The paper is well written and easy to follow.\n3. The empirical results confirm the effectiveness of the proposed method.\n\nWeaknesses\n\n1. Missing several closely related work.\n2. In addition to activation low-rank projection, the authors also propose a token grouping strategy. However, the benefit of this technique is unclear to me.\n\n\nGiven the concerns above, I recommend a borderline rejection of this paper.\n\n[1] [Accelerating deep learning with lossy compression](https://open.library.ubc.ca/soa/cIRcle/collections/ubctheses/24/items/1.0412625) \\\n[2] [Flora: Low-Rank Adapters Are Secretly Gradient Compressors](https://arxiv.org/abs/2402.03293)", "questions": "See the weaknesses.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The topic of efficient training is important, given that the models are scaled up at a rapid rate.\n2. The paper is well written and easy to follow.\n3. The empirical results confirm the effectiveness of the proposed method.", "weaknesses": "1. Missing several closely related work. The most related topic is activation compression (e.g. see [1] as an example of such a line of research). In addition, the paper uses GaLore as the baseline. However, there is existing work showing superior performance without conducting expensive SVD operations (e.g. [2]). The paper lacks proper comparisons with these baselines.\n2. In addition to activation low-rank projection, the authors also propose a token grouping strategy. However, the benefit of this technique is unclear to me. In Table 5 (b), the authors show that different numbers of groups will behave inconsistently: some work better while some do not without a clear pattern. It might be the variance that is playing the important role here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for praising the $\\textbf{effectiveness}$ of our method, calling it $\\textbf{important}$, $\\textbf{well-written}$, and $\\textbf{easy to follow}$. We also thank the reviewer for suggesting us to discuss and compare with two related works, and asking us for a clarification with regards to the token grouping strategy.\n\n####  **1. Missing several closely related work.**\n\nWe thank the reviewer for these references, which were new to us. We examined Evans' thesis and the two papers that it is based on (JPEG-ACT and AC-GC). JPEG-ACT appears less relevant as it's a CNN-specific method using JPEG compression for network features. AC-GC is more relevant, addressing activation compression, but differs significantly in its approach. It uses constrained optimization to maximize compression subject to bounded gradient error, unlike our method. \nAC-GC's experiments focus majorly on CNN and MLP networks and give no hints about their effectiveness for Transformers-based architectures which are the primary focus of our work. \nIn our revised manuscript, we will incorporate citations and discussions of these works to provide appropriate context and clearly delineate the distinctions between these approaches and ours.\n\nWe appreciate the author's reference to \"Flora: Low-Rank Adapters Are Secretly Gradient Compressors\" (ICML24), a paper we were not aware of. While published after the NeurIPS deadline, an earlier version was available on ArXiv in February 2024. Flora shares similarities with our work and GaLore in reducing GPU memory usage. However, it doesn't operate directly in activation/gradient space and, unlike our approach, doesn't save memory during backpropagation. Flora can be viewed as an alternative to adaptive optimizers (such as Adam), while VeLoRA is more of an alternative to backpropagation, making them complementary. We will acknowledge and discuss these distinctions in our revised manuscript. \n\nFurthermore, considering the relevance of the paper, we compared Flora to our method showing the results in Figure 1 (attached pdf). We ran the official code of Flora in Galore setting (having the same number of iterations as us, Galore and full-finetuning). We overperform Flora by 0.69 percentage points (pp) in Llama 60M and by 0.41 pp in Llama130M, noting that Flora reaches overall better results than Galore. Furthermore, we show in the table that we report memory improvements compared to Flora, 1.18GB vs 1.27GB in LLama 60M and 1.83GB vs 2.02GB in LLama 130M.\n\nWe will update the manuscript with citation for FLoRA, the comparison and the above discussion.\n\n####  **2. Understanding the token grouping strategy.**\n\nWe appreciate the request for clarification. Projecting the input activations to a lower-dim space can be done in many ways. For example, projecting a 512-dim token to 4 dims can be achieved by:\n\n1. Direct multiplication: Using a [512, 4] matrix, requiring 2048 parameters.\n\n2. Grouped projection: Dividing the 512-size token into 4 sub-tokens of size 128, then projecting each sub-token to a single number. This method uses only 128 parameters, 16 times fewer than the first approach. Note that our 'parameters' are not actually trainable parameters, instead they are just initialized with batch statistics and kept fixed. Finally, we also wish to highlight that using more sub-tokens leads to an even more significant parameter (and memory) saving over a standard linear projection.\n\nWe also wish to highlight that reducing the number of parameters here reduces the cost of the projection both in terms of FLOPs and memory. We have included a figure in the attached PDF to illustrate this concept visually.\n\nNot only is decomposing the token into sub-tokens important for better parameter/memory-efficiency, but it also improves the estimate of our batch statistics for initialising the projection vector $v$. Having more sub-tokens results in more samples for computing the average. Although we demonstrate strong performance for a varying number of sub-tokens (table 5b main paper), we find that careful tuning is optimal. However, we do demonstrate the robustness of VeLoRA to this hyperparameter by using the same value for all of the larger scale tasks provided.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VNCdoTW86J", "reviewer": "Reviewer_KYfS", "summary": "This paper proposes VeLoRA, an activation-compression method to reduce memory consumption. VeLoRA compresses the activations by multiplying them with a vector, and the activations are then decompressed before gradient back-propagation. VeLoRA has proven to be effective on both vision and language models.", "questions": "See weaknesses above.\n\nOverall, VeLoRA is a very interesting method, but the paper should provide a more clear explanation of the intuition or insights behind the method. For this reason, I cannot definitively say that the paper is solid, but my view is generally positive.", "limitations": "Limitations and Broader Impact are discussed in the paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper presents an interesting method to reduce memory consumption when training large models.\n- Experimental results show that VeLoRA can not only reduce GPU memory, but also achieve better performance.", "weaknesses": "- VeLoRA reduces memory usage by compressing activations. However, the widely used gradient checkpointing method can almost entirely eliminate the need for storing intermediate activations by recomputing them. How much speed advantage does VeLoRA offer compared to gradient checkpointing?\n\n- The rank-1 mapping method used in VeLoRA is not intuitive to me. Can you explain why the rank-1 mapping is effective from a theoretical or intuitive perspective? In my opinion, the rank-1 mapping is equivalent to a weighted average, so does this mean that setting $v$ as a constant value $ (1/\\sqrt m, 1/\\sqrt m, \\ldots, 1/\\sqrt m) \\in \\mathbb{R}^m $ would also be feasible?\n\n- Although VeLoRA's reduction in memory usage is not significant, its improvement in model performance is impressive. Can you explain the reasons behind this effect?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing some constructive questions and suggestions for a more thorough and complete set of comparisons. We also appreciate the reviewer highlighting our proposed method as $\\textbf{very interesting}$ and praising its $\\textbf{experimental results}$.\n\n#### **1. Comparison with gradient checkpointing**\n\nWhile other works such as Galore or LoRA do not provide such a comparison, we completely agree with the reviewer that comparing with checkpointing should be mandatory. Thus, we provide a comparison on both the small and large-scale Llama models for pre-training and fine-tuning. See the attached PDF for the table of results, where we can see that although gradient checkpointing can significantly reduce the on-device memory usage, it comes at a relatively significant training time overhead in comparison to VeLoRA.\n\n#### **2. Uniform and normalised initialisation for v.**\n\nProjecting each sub-token onto the vector $v = (1/\\sqrt{m},1/\\sqrt{m},…,1/\\sqrt{m})$ can work, but is much less effective than a more data-driven approach (i.e. using the first-order batch statistics). The reason for this is because we want to preserve as much information as possible when projecting the input activations. However, interestingly, we find that the best reconstruction, i.e. using SVD, does not nescasarily improve the models performance while also incurring the additional compute overheads. We have conducted an additional experiment with the $v = (1/\\sqrt{m},1/\\sqrt{m},…,1/\\sqrt{m})$  and have updated table 5c in the main paper.\n\nSee the attached PDF for the updated table with results for this initialisation strategy. Here we can see that this initialisation is around on-par with a random initialisation, but leads to a significant degradation over more data-driven initialisation strategies.\n\n#### **3. Memory usage and performance improvement.**\n\nWe respectfully disagree with the reviewer that the memory reduction is not significant. The memory reduction is small only when using small networks (VTAB) and compared to LoRA (but still massive to full finetuning). However, in other cases, we get a massive memory improvement. For example, in RoBERTa experiments, our method reduces the memory compared to LoRA by 18% and 52% compared to full-finetuning. In LLaMA 7B, we reduce the memory compared to QLoRA by a further 15.5%, and in LLaMA 13B by 14.5%.\n\nWith regards to the performance improvement, many PEFT methods (including LoRA) have be shown to be more effective on smaller scale tasks and datasets than full fine-tuning (such as VTAB and some sub-tasks of GLUE). With regards to the larger datasets, VeLoRA is performing an explicit regularisation on the gradient flow for the weights. More specifically, this regularisation restricts the gradient flow to follow the average (if using a batch average initialisation) from the pre-trained model. This is quite a strong regularisation that can be what is improving the models generalisation. \n\n#### **4. A more clear explanation of the intuition or insights behind the method is needed.**\n\nWe give a theoretical explanation on the need for projections in section 3.2. Why does a vector projection make sense?. However, we agree with the reviewer that a more detailed explanation on the insights of the method is helpful. Thus, we make in the rebuttal, under the response of reviewer UGFV, a direct theoretical connection behind our method and LoRA, and clearly explain the gain from using the projections. In the interest of space, please see the part **Theoretical connection to parameter efficient fine-tuning** in the rebuttal addressed to reviewer UGFV.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PV7kJjGbI7", "reviewer": "Reviewer_UGFV", "summary": "This paper introduces VeLoRA, a novel method for memory-efficient training and fine-tuning of large language models. The key idea is to compress intermediate activations during the forward pass by projecting sub-tokens onto a fixed 1-dimensional subspace, then coarsely reconstructing them during backpropagation. This approach complements existing parameter-efficient fine-tuning (PEFT) methods and allows for significant memory savings without sacrificing model performance. The authors evaluate VeLoRA on several benchmarks including VTAB-1k, GLUE, MMLU, and C4, demonstrating competitive or improved results compared to state-of-the-art methods like LoRA, QLoRA, and GaLore while reducing memory requirements.", "questions": "How does VeLoRA affect training and inference times compared to baseline methods? Is there a significant computational overhead?\nAre there any particular types of tasks or model architectures where VeLoRA might be less effective or even detrimental to performance?", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  VeLoRA presents an innovative solution to the critical problem of memory efficiency in training large language models. The method is conceptually simple yet highly effective, offering substantial memory savings without compromising performance.\n2. The authors provide extensive experiments across multiple benchmarks and model sizes, from vision transformers to large language models. This thorough evaluation demonstrates the broad applicability and scalability of VeLoRA.\n3. Complementary to existing methods: VeLoRA is shown to be compatible with and complementary to other PEFT methods, enhancing their memory efficiency. This makes the approach highly practical and easy to adopt in existing workflows.", "weaknesses": "Limited theoretical analysis: While the paper provides some intuition for why VeLoRA works, a more rigorous theoretical analysis could strengthen the understanding of its effectiveness and potential limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting the $\\textbf{extensive experiments, broad applicability, and scalability}$. We also appreciate your description of the approach as $\\textbf{highly practical and easy to adopt}$.\n\n#### **1. Limited theoretical analysis: While the paper provides some intuition for why VeLoRA works, a more rigorous theoretical analysis could strengthen the understanding of its effectiveness and potential limitations.**\n\nWe note that under “Why does a vector projection make sense?” we give a mathematical explanation, including an equality which we prove in the appendix. However, we agree with the reviewer that the paper would be even stronger with more theory. We will add further analysis to the final paper and we hope that VeLoRA will inspire other researchers to further study it theoretically, considering its very strong empirical performance, initial theoretical analysis and, potential impact in training large models, and its ease of use.\n\nWe note that we showed experiments in a wide range of settings like: image classification, small language models (RoBERTa), finetuning large language models (LLaMA), and pre-training LLaMA, in all cases our method shows improvement in both accuracy and memory. VeLoRA can be adapted to any Transformer architectures using the analysis done in Sec 5.4. We leave adapting the finding of the paper to other architectures (MLP, CNN, SSM) asfuture work. Given that most SoTA models in modern AI are Transformer-based, we believe VeLoRA has significant potential for widespread impact in the field of resource efficient training.\n\nBelow, we provide a theoretical justification of VeLoRA, and a better connection of it with PEFT methods, such as LoRA, which we will incorporate in the final version of the paper.\n\n#### **Theoretical connection to parameter efficient fine-tuning**\n\nWe provide a theoretical analysis of VeLoRA and its similarities with LoRA. We will provide a more thorough analysis in the updated manuscript.\n\n####  **Part 1. VeLoRA uses a low-rank data-dependant projection of the gradients**.\n\nFor VeLoRA, we replace $\\frac{d y}{d W}$ with a projection onto the vector $v$. For simplicity consider the case where there are **no sub-tokens**:\n\n\\begin{align}\n    \\frac{d y}{d W} \\approx Proj_{v}\\left(\\frac{d y}{d W}\\right) = \\left(\\frac{d y}{d W} \\cdot v\\right)v^T\n\\end{align}\n\n* **Forwards.** $\\frac{d y}{d W} \\cdot v \\in \\mathbb{R}^{ND/M}$ is saved during the forward pass. These projected gradients are **much smaller** than the original gradients.\n* **Backwards.** $\\left(\\frac{d y}{d W} \\cdot v\\right) v^T \\in \\mathbb{R}^{ND/M \\times M}$ is a reconstruction during the backward pass.\n\nThis replacement modifies the gradient of the loss with respect to $W$ to:\n\n\\begin{align}\n    \\frac{d L}{d W} \\approx \\frac{d L}{d y} \\cdot \\left(\\left(\\frac{d y}{d W} \\cdot v\\right)v^T\\right) = \\left( \\frac{d L}{d y} \\cdot \\frac{d y}{d W} \\right) vv^T\n\\end{align}\n\nIf we denote $\\frac{d L}{d y} \\cdot \\frac{d y}{d W}$ as $\\tilde{g}$, we can express the weight update as follows:\n\n\\begin{align}\n    W' = W - \\eta \\frac{d L}{d W} = W - \\eta \\tilde{g} v v^T\n\\end{align}\n\nThis result highlights that we are performing a data-driven rank-1 update on the matrix W.\n\n####  **Part 2. LoRA uses a random low-rank projection of gradients**.\n\nFor LoRA, we simply freeze the original weights and only train a linear low-rank adapter module that is added in parallel:\n\n\\begin{align}\n    y = Wx + ABx = (W + AB)x\n\\end{align}\n\n* **Frozen.** Base weights $W$ are frozen.\n* **Trainable.** Low-rank weights $A$ and $B$ are trainable.\n\nConsidering freezing $A$ and initialising $B$ with all zeroes. i.e. $A = A_0$ and $B_0 = 0$. The weight update rule is given as:\n\n\\begin{align}\n    W' &= W + A_0\\left(B_0 - \\eta \\frac{d L}{d B}\\right) = W - \\eta \\tilde{g} A_0 A_0^T,\n\\end{align}\n\nwith $\\tilde{g} = \\frac{d L}{d y} \\cdot \\frac{d y}{d B}$. Here we can see that VeLoRA is special case of LoRA under the lense of gradient compression. However, there are a few **important distinctions**:\n\n* **Updating base weights directly.** Since we operate in the gradient space we can update the base weights directly. Memory efficiency is achieved by only saving the projected activations in memory and using a coarse reconstruction during the backwards pass.\n* **No additional trainable parameters.** $v$ is fixed throughout training.\n* **Data-driven projections.** The vectors $v$ are initialised using the first-batch statistics, which is empirically much more effective than a random projection.\n\n####  **Part 3.  Extended analysis to use sub-tokens.**\nApplying sub-tokening in the context of LoRA can be done as follows:\n\n\\begin{align}\n    y = Wx + G_r^{-1}((A_0B)G_r(x)),\n\\end{align}\n\nwith $A = A_0$, $B_0 = 0$, and $G_r(\\cdot)$, $G_r^{-1}(\\cdot)$ being the token grouping and ungrouping operations described in the main paper. Here we can see that the proposed sub-tokening strategy motivated in the gradient space can instead be seen as another novel and more parameter efficiency PEFT method. This duality between gradient compression and parameter efficient fine-tuning is an important and a very recent emerging perspective for memory-efficient training. We will provide the more thorough analysis of these results in the updated manuscript.\n\n#### **2 .How does VeLoRA affect training and inference times compared to baseline methods? Is there a significant computational overhead?**\n\nWe provide the training time (iterations/second) in Tab 1, attached pdf. Our method has a small increase in the training time compared to full-finetuning, but is still significantly faster than gradient checkpointing or other competing methods such as Galore (because of their expensive and periodic SVD). We expect that the training time can be further reduced by writing fused CUDA kernels. There is no disadvantage of our method in terms of speed during inference since there are no gradients and thus no need for any low-rank projections.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6mjJR6xK45", "reviewer": "Reviewer_bRtV", "summary": "This paper proposes a novel algorithm named VeLoRA, which achieves memory-efficient training by compressing the intermediate activations of large-scale language models into a fixed one-dimensional subspace. VeLoRA divides tokens into smaller sub-tokens and projects them during the forward pass, then reconstructs the activations during the backward pass, thereby reducing memory usage. Experimental results demonstrate that VeLoRA performs excellently across various benchmarks, complementing other parameter-efficient fine-tuning methods, and significantly reducing memory requirements while maintaining high performance.", "questions": "see Weaknesses.", "limitations": "The authors mention training time issues in the Limitations section, but the relevant results are not presented.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1 The method is easy to understand and succinctly compresses the intermediate activations.\n\n2 The method has a relatively complete derivation and explanatory analysis.", "weaknesses": "1 The number of parameters used by VeLoRA, especially in Tables 1 and 2, is not specified.\n\n2 In Table 1, although there is an improvement in the average results compared to LoRA, the performance degrades on datasets such as Caltech101, Resisc45, and Clevr-Count. Why is this the case?\n\n3 For PEFT methods, the experiments on visual tasks are insufficient. According to the category in \"Yu B X B, Chang J, Wang H, et al. Visual tuning[J]. ACM Computing Surveys, 2023.\", VeLoRA experiments are based only on the Parameter Tuning category, neglecting Prompt Tuning and Adapter Tuning. Therefore, the current results do not fully demonstrate the universality of the proposed method.\n\n4 Although the authors mention time issues in the Limitations section, the training duration before and after introducing VeLoRA should be presented.\n\n5 For Tables 1 and 2, the improvements brought by VeLoRA are not significant. Although the authors declare this in Q7 of the Checklist, a significance analysis is still necessary, especially in cases where the improvement is not evident.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for praising the method's $\\textbf{simplicity}$, $\\textbf{comprehensive derivation}$ and $\\textbf{analysis}$. We are excited to see that the reviewer positively rates the contribution and presentation of the paper. We hope that our rebuttal addresses the reviewer's issues about the paper, mostly related to the training efficiency, number of parameters and method's robustness.\n\n#### **1. The number of parameters used by VeLoRA, especially in Tables 1 and 2, is not specified.**\n\nIn Tables 1 and 2, we utilize the widely recognized ViT-Base-224/16 and RoBERTa-Base models. VeLoRA introduces no additional trainable parameters, **so the number of trainable parameters is exactly the same as the PEFT methods we compare with** (e.g., where we compare with LoRA, the number of trainable parameters is the same as in LoRA, same for Hydra or SSF). This is because our projections are not trainable, instead they are initialized by the batch statistics and then kept fixed.\n\n\n#### **2. In Table 1, performance degrades on specific datasets such as Caltech101, Resisc45, and Clevr-Count. Why is this the case?**\n\nFor a fair evaluation, we perform a full sweep across all datasets, rather than cherry picking individual dataset results. In the cases where VeLoRA underperforms, we would like to highlight that the difference is relatively small. To emphasise this point, we have since performed two additional sweeps to report the standard deviation over all tasks.\n\n#### **3. For PEFT methods, the experiments on visual tasks are insufficient. According to the category in \"Yu B X B, Chang J, Wang H, et al. Visual tuning[J]. ACM Computing Surveys, 2023.\", VeLoRA experiments are based only on the Parameter Tuning category, neglecting Prompt Tuning and Adapter Tuning. Therefore, the current results do not fully demonstrate the universality of the proposed method.**\n\nWe appreciate the reference to the survey paper. It's important to note that the scope of a survey paper is, by design, much broader than our submission. To demonstrate the versatility of our method, we conducted experiments across diverse settings, including image classification, small language models (RoBERTa), fine-tuning large language models (LLaMA), and pre-training LLaMA. In each scenario, our approach shows improvements in both accuracy and memory efficiency.\n\nOur work differs significantly from Prompt Tuning, which is more suited to multi-modal learning or cross-domain tasks. Our experiments focus on single modalities without employing adapters to bridge different modalities or tasks. Moreover, as noted in the LoRA paper (Section 3, \"Directly Optimizing the Prompt is Hard\"), optimizing for prompt tuning presents substantial challenges.\n\nRegarding adapters, we argue that the field has largely shifted towards Parameter-Efficient Fine-Tuning (PEFT) methods since the introduction of LoRA. These methods significantly outperform traditional adapter approaches without introducing additional inference overheads (as discussed in Section 3 of the LoRA paper, \"Adapter Layers Introduce Inference Latency\"). Therefore, in our work, we've conducted comprehensive comparisons with leading PEFT methods, including LoRA, Hydra, QLoRA, and GaLore, to ensure a thorough evaluation of our approach. We will incorporate this discussion in the final draft.\n\nNevertheless, if the reviewer insists that we should do comparison with some particular works in the survey, we would be glad to further communicate this during the discussion period, and we will try our best to compare with them as long as the given methods provide code.\n\n#### **4. Although the authors mention time issues in the Limitations section, the training duration before and after introducing VeLoRA should be presented.**\n\nWe believe there has been a misinterpretation regarding our method's computational efficiency. It's important to highlight that our approach has a similar speed as LoRA, or other comparable methods. In the limitations section, we identified two primary challenges for large models, particularly LLMs: GPU memory constraints and extended training duration (exemplified by LLaMA 3.1's training on over 15 trillion tokens).\nOur method specifically targets the memory issue, enabling researchers with limited GPU resources to train and fine-tune models. However, the inherently large datasets still require large training time. While we've made significant progress in memory efficiency, we acknowledge that reducing training time remains an area for future research, possibly through the development of more compact, information-dense datasets.\nTo reiterate, **our method is faster than existing approaches such as gradient checkpointing**. We provide direct comparisons in the attached pdf to substantiate this claim. Note that our method does not increase the inference time at all, considering there are no gradients in inference.\n\n#### **5. For Tables 1 and 2, the improvements brought by VeLoRA are not significant. Although the authors declare this in Q7 of the Checklist, a significance analysis is still necessary, especially in cases where the improvement is not evident.**\n\nThank you for highlighting this point. We've conducted additional experiments, rerunning each experiment two more times. The average accuracy of our method remains the same with a very small standard deviation of 0.1, with our method continuing to achieve the best overall performance with a consistent margin. Due to space limits we cannot show the results in the attached pdf but will put them directly in the paper.\n\nAt the same time, we'd like to emphasise that the primary advantage of our approach lies in its memory efficiency, particularly for large models (as demonstrated in Tables 2-5). While the performance improvement is certainly beneficial, the method's core value is its significant reduction (sometime up to 45% compared to baseline methods and an average of 17%) in memory usage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' positive feedback on our method's **effectiveness** (sd52, KYfS, UGFV), **simplicity** (bRtV, UGFV), **comprehensive derivation and analysis** (bRtV), **writing quality** (sd52), and **thorough evaluation with comparisons to existing works** (UGFV). We're pleased that our method/topic was described as **important** (sd52), **interesting** (KYfS), and **innovative** (UGFV).
To address the reviewers' concerns, we offer the following responses:
  
$\\;\\;\\;$
  
1. **sd52**: comment on **missed related works**
    * We clarify the differences between our work and the mentioned Ph.D. thesis and ICML24 paper.
    * We provide a comparison with FLoRA (ICML24) - see attached PDF.

2. **sd52**: request for **clarification on the grouping strategy's advantages**:
    * We provide the requested explanation, and a visualization of the procedure.

$\\;\\;\\;$

3. **KYfS**: request for **comparison with gradient checkpointing**:
    * We agree this comparison is crucial and so we now provide additional results for the Llama models.
4. **KYfS**: suggestion for an additional **projection initialisation experiment**:
    * We present the requested experiment results.
5. **KYfS:** concerns about **memory improvements**:
    * We provide a detailed explanation supporting our claims.

$\\;\\;\\;$

6. **UGFV**: recommendation for **more theory** behind our method
    * We provide a theoretical connection between our method and PEFT methods such as LoRA.

7. **UGFV**: questions on **performance and potential limitations**:
    * We offer metrics on the algorithm's speed.
    * We explain that our method is applicable across all Transformer-based models, which are state-of-the-art in almost all AI subdomains.

$\\;\\;\\;$

8. **bRtV's requests**:
    * We rerun the VTAB experiments for another two times and show that our results have a very low standard deviation.
    * We emphasize that VeLoRA does **not** introduce additional training parameters compared to other PEFT methods.
    * We agree to cite the suggested survey paper.
    * Regarding experiments in other settings:
        + We explain that prompt tuning is outside our paper's scope.
        + We note that comparable works (LoRA, Hydra, Q-LoRA, Galore, Flora, etc.) don't include such experiments.
        + We argue that many other adapter methods typically underperform PEFT methods, making comparisons unnecessary.
        + We invite the reviewer to specify any particular method they'd like us to compare with, and we'll attempt to run experiments if code is available during the discussion period.
  
$\\;\\;\\;$
  
We will integrate all of these changes in the camera-ready version of the paper. We provide a figure to better illustrate the grouping strategy and tables for the new experiments in the attached pdf.

Meta-review: The authors propose a “rank-1 on subtokens” approach to compressing activations in the forward pass and reconstructing in the backward pass. They show their method is general, performing better in pretraining and finetuning on both vision and language foundation models. I and the reviewers found the sufficiency (with justification) of a fixed projection novel and interesting. Reviewers approved of the experimental results though wanted more intuitions/comparisons/ablations/significances due to the breadth of existing memory-efficient methods, all of which the authors provided. Ultimately, all four reviewers recommend some form of acceptance with one solid Accept. **I recommend Acceptance.**
Decision: Accept (poster)