Introduction
Large language models (LLMs) have achieved remarkable success on a wide range of natural language processing tasks [ 30 , 46 , 2 ] .
However, training these massive deep learning models remains computationally expensive, requiring vast amounts of data, compute, and memory resources.
A key bottleneck for training or adapting these models is the large memory needed to store all the intermediate features required to compute the gradients for optimization.
This makes it challenging to fully leverage the scalability and performance gains promised by larger models on currently available hardware.

Several techniques have been proposed to reduce the memory requirements, such as GaLore [ 53 ] , gradient checkpointing [ 8 ] , reversible backpropagation [ 14 ] , parameter-efficient finetuning [ 17 , 7 ] , quantization [ 11 ] and activation offloading [ 29 ] .
GaLore uses a low-rank projection of the gradients during training to reduce the memory footprint.
Gradient checkpointing reduces the activation memory demands by recomputing the activations in the backward pass instead of storing them.
While these methods are promising and lower the memory cost, they also might introduce a substantial computational overhead, are limited in their memory savings, or require specialized hardware [ 11 ] .
Knowing that compute is the primary mover for the advancements in machine learning [ 45 ] , it is to be expected that the LLM sizes will continue growing exponentially.
Thus, it is imperative to develop more efficient and scalable methods that allow better utilization of compute power and training data.

In this work, we present a novel approach for efficient training and finetuning, which we name Ve ctor projected LoRA (VeLoRA).
Our approach is based on a key observation that the intermediate activations produced during the forward propagation of deep neural networks, and kept in memory for computing the gradients during backpropagation, can be effectively represented and reconstructed from a single and fixed one-dimensional vector without losing any accuracy.
This compressed representation can be made very memory efficient, with a controllable hyperparameter that provides a trade-off between the compression ratio and the model’s performance.
By compressing and then reconstructing the activations on the fly, our method reduces the peak activation memory footprint to a tiny fraction of what is required to store the original activations.
This enables fitting much larger models into limited GPU memory compared to approaches like GaLore or gradient checkpointing.

More concretely, during the forward pass, we propose to divide each input token into a set of much smaller sub-tokens.
Using a single projection vector, we then project these individual sub-tokens onto a one-dimensional subspace.
Importantly, we notice that we can initialize this projection vector cheaply using first-order batch statistics and then keep it fixed throughout training.
We then reconstruct the original tokens using the same vector during the backward pass.
Although this reconstruction loses the original gradient properties such as the direction or magnitude, we find that it jointly encourages sparsity and locally preserves the gradient similarity, which we attribute to the overall effectiveness of the algorithm.
By storing these compact representations, we can substantially reduce the memory footprint of the network during training, enabling the accommodation of larger models on hardware with limited memory capacity.

Our contributions are the following:

We propose a novel compression method that reduces the memory requirement for gradient computation during training and fine-tuning of large neural network models like LLMs.

We show that, unlike other methods, our compression method does not need expensive operations such as SVD and gradient checkpointing.

We achieve state-of-the-art results in various benchmarks while requiring lower GPU memory compared to the baseline methods.