Insights and properties of VeLoRA
On the importance of compressing sub-tokens. Computing the optimal low-rank subspace of the gradients using SVD is very computationally and memory intensive and often needs offloading the operation to a CPU [ 53 ] .
Moreover, periodically updating the projection may be necessary to track any shift in the gradient distribution [ 53 ] . This is why dividing the tokens up into smaller sub-tokens is necessary.
By doing so, it allows us to use a cheap surrogate rank-1 projective map that is initialised and frozen throughout training.
Finally, one surprising observation of this grouping operation is that the sub-tokens will naturally lie on a higher-dimensional subspace than the original tokens (see figure 2 a).
Thus, our method cannot be faithfully explained through a better reconstruction of the gradients, but instead by a suppression of the inherently larger eigenvalues that can in turn help reduce overfitting.

Why does a vector projection make sense? Using a fixed vector projection throughout training fails to capture any shift in the gradients’ distribution.
Under the assumption that preserving the original gradient reconstruction is important, this may seem like an undesirable trait.
However, quite surprisingly, we find that this shift does not hinder the model’s convergence or performance at all.
An explanation behind this phenomenon can be twofold:
(i) The gradients become more sparse as they shift away from the initial distribution and this helps prevent the model from overfitting to the fine-tuned dataset;
(ii) Although the vector projection destroys the original gradients’ magnitudes and directions, it still locally preserves the gradient similarity and this similarity will govern the model’s training dynamics [ 18 ] .

Consider a rank-1 decomposition of two sub-tokens: z_{i} and z_{j} . We will use the dot-product as the similarity measure sim(\cdot) for which we wish to locally preserve. Let us assume that both \mathbf{z}_{i} and \mathbf{z}_{j} are distributed such that the angles between them and the vector \mathbf{v} are normally distributed with a mean of 0 and a standard deviation \sigma . With a first-order approximation, the probability of the projection and reconstruction scaling the gradient similarity by at least k is given as follows (see the Appendix for the full derivation):

 Pr\left(\left|sim(proj_{\mathbf{v}}(\mathbf{z}_{i}),proj_{\mathbf{v}}(\mathbf{z}_{j}))-sim(\mathbf{z}_{i},\mathbf{z}_{j})\right|>k\right) \approx 2\left(1-\Phi\left(\frac{\sqrt{k}}{\sigma}\right)\right) (3)

With k>0 and \sigma>0 , this probability is bounded by [0,1] . Here we can see that similarity is trivially preserved in the limit as \sigma\rightarrow 0 . This indicates that the sub-tokens already lie on a 1-dimensional subspace spanned by \mathbf{v} . To further see how these gradient similarities diverge for various values of k and \sigma , we plot equation 7 in Fig. 2 c. We empirically observe that although the gradient similarity is very dependent on the distribution of features, this non-linear relationship does not hinder the model’s ability to converge and generalise. Finally, we also provide an illustrative visualisation in Fig. 1 (right) that shows the locality sensitivity for preserving gradient similarity and the sparsification of gradients when they are orthogonal to the vector \mathbf{v} . Both of these components are important properties that we attribute to the effectiveness of VeLoRA.

Cheap initialisation strategies. GaLore [ 53 ] proposes to use the periodically updated SVD principle components to track any shifts in the underlying sub-space of the gradients.
Unfortunately, this SVD operation can be very expensive both in terms of memory and compute. Another limitation is that SVD may fail to converge, and it requires casting the features back to 32-bit for numerical stability [ 32 ] .
For this reason, we propose a cheap initialisation strategy. We relax the constraint on tracking the feature distribution.
For all of our experiments, we use a rank-1 decomposition of sub-tokens and propose to initialize the vector \mathbf{v} using the average of sub-tokens from the first batch.