On the Duality between Gradient Transformations and Adapters
Authors: Lucas Torroba Hennigen
Hunter Lang
Han Guo
Yoon Kim
Venue: ICML 2025 poster (2025)
Keywords: memory
efficient
training
gradient descent
pretraining
linear algebra
LLMs
GaLore
LoRA
Primary area: deep_learning->large_language_models
TL;DR: We explore a connection between linear gradient transformation and one-sided LoRA adapters, unifying existing approaches and deriving improved techniques for memory-efficient training

Abstract: We study memory-efficient optimization of neural networks (in particular language models) with *linear gradient transformations*, where the gradients are linearly mapped to a lower dimensional space than the full parameter space, thus saving memory required for gradient accumulation and optimizer state persistence. The model parameters are updated by first performing an optimization step in the lower dimensional space and then going back into the original parameter space via the linear map's transpose. We show that optimizing the model in this transformed space is equivalent to reparameterizing the original model through a *linear adapter* that additively modifies the model parameters, and then only optimizing the adapter's parameters. When the transformation is Kronecker-factored, this establishes an equivalence between GaLore and one-sided LoRA. We show that this duality between gradient transformations and adapter-based reparameterizations unifies existing approaches to memory-efficient training and suggests new techniques for improving training efficiency and memory use.

On the Duality between Gradient Transformations and Adapters
Lucas Morroba-Hennigee
Hunter Lang' Han Guo
Yoon Kim
Abstract
In response, a wealth of literature has developed around memory-efficient training methods. Most of these fall into one of two families. The first involves modifications to the model parameterization, in particular by introduc- ing "'adapter"' to the model architecture that have a small number of additional parameters, and only tuning those adapters (Houlsby et al., 2019; Li & Liang, 2021; Hu et al., 2022). Adapters such as LoRA increase the total number of parameters but reduce the number of trainable param- eters, resulting in overall memory savings. While LoRA was originally introduced for memory- finetuning, recent works such as ReL.oRA (Lialin et al., 2024), LoRA- the-Explorer (LTE; Huh et al., 2024), and Flora (Hao et al., 2024) find that LoRA can even enable memory-efficient pretraining if the adapters are periodically merged back into the full model (and then reinitialized).
We study memory-efficient optimization of neu- ral networks (in particular language models) with linear gradient transformations, where the gradi- ents are linearly mapped to a lower dimensional space than the full parameter space, thus saving memory required for gradient accumulation and optimizer state persistence. The model parame- ters are updated by first performing an optimiza- tion step in the lower dimensional space and then going back into the original parameter space via the linear map's transpose. We show that op- timizing the model in this transformed space is equivalent to reparameterizing the original model through a linear adapter that additively modi- fies the model parameters, and then only opti- mizing the adapter's ' parameters. When the trans- formation Cronecker-factorede this establishes an equivalence between GaLore (Zhao et al., 2024) and one-sided LoRA (Hu al., 2022). We show that this duality between gradient transfor- mations and adapter-based reparameterizations unifies existing approaches to memory-efficiae training and suggests new techniques for improv- ing training efficiency and memory use.
The second family of methods involves more direct changes to the optimization strategy, either by designing optimizers that store fewer extra bits of information per parameter (Anil et al., 2019; Shazeer & Stern, 2018), or (broadly) compressing the gradients, e.g., via quantiza- tion (Bernstein et al., 2018; Dettmers et al., 2022: Li al., 2024a) or low-rank approximations (Gooneratne et al., 2020; Huang et al., 2023). For LLMs, Gal (Zhao et al., 2024) has recently emerged as promising gradient com- pression approach for memory-efficient pretraining. Ga- Lore transforms the gradient matrices of linear layers via projections derived from an SVD of the gradient matrix, and then performs optimization in this projected space.
1. Introduction
Training neural networks, in particular large language mod- els (LLLMs), can be extremely memory-intensive Stan- dard approaches for LLM training use gradient accumu- lation across multiple batches and optimizers such Adam (Kingma& & Ba, 2015), which maintains estimates of the first and second moments of the (stochastic) gradient. Hence, the amount of GPU memory needed for standard training can be as much as four times the amount of mem- ory needed to store the model (assuming the gradients/opti- mization states are kept in the same precision as the model parameters).
Is there a relationship between methods that directly trans- form/compress gradients, and adapter-based methods that reparameterize the underlying model into frozen and train- able components? In the case of Gal.rre and LoRA, recent works find that the answer is yes, in particular showing that training LoRA adapter with one side frozen can be seen as form of gradient compression where the gradient matri- ces are sketched to lower dimensional space with random matrices (Hao et al., 2024) or through SVD-based projec- tions (Loeschcke et al., 2024).
"Msssacuusetts Institute of Technology. Correspondence to: Lucas Norroba-Hennige <lucastor@ mit.edu>.
In this work, we show that the connection between GaLore and LoRA more general by proving that training a neu- ral network by applying an arbitrary linear transformation to the gradient vector is equivalent (in the sense that the re-
Proceedings of the 1nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

On the Duality between Gradient Transformations and Adapters
sulting models are the same and have the same optimization trajectory) to reparameterizing the neural network through linear adapter that additively modifies the original pa- rameters, and then only training the adapter. When applied to (vectorized) matrices with a particular Kronecker factor- ization of the linear map, our results recover the equiva- lence between GaLore and one-sided LoRA.
mulation across multiple mini-batches of data."
Optimizer states. In addition to the gradient itself, most optimizers used in LLM training persist other state across steps. Adam (Kingma & Ba, 2015) and AdamW (Loshchilov & Hutter, 2019) maintain running av- erages of the gradient and the gradient squared (i.e., an estimate of first- and second-order moments), which re- quire two floats per trainable parameter. Examples of techniques that reduce optimizer memory include 8-bit Adam (Dettmers et al., 2022), which stores Adam states in lower precision, and AdaFactor (Shazeer & Stern, 2018), which modifies Adam to use fewer floats per parameter.
Our empirical experiments study this connection between linear gradient transformations and chapter-based repa- rameterizations in the context of memory-efficient LLM training. First, we perform a comparison across gra- dient projection-basee and LoRA-based approaches for memory-efficient training and find that randomly sketch- ing gradients works particularly well ($4.1). We also ex- ploit the adapter view of jrjjecttedggaaieen training' by de- veloping QLoRA-style (Dettmers et al., 2023) approach to Gal-oresstyle training. Second, we show that the gra- dient projection view of LoRA adapters can improve dis- tributed training of LLMs with parallel LoRA adapters (Huh et al., 2024) by suggesting an initialization scheme of worker-specifie LoRA adapters tailored for distributed training ($4.2). These results collectively demonstrate that this duality between linear gradient transformations and adapter- based reparameterizations is productive lens with which to view neural network optimization, since it unifies several existing approaches and suggests new techniques for improving training efficiency and performance.
Activations. LLM gradients are almost always obtained using reverse-mode automatic differentiation (Griewank & Walther, 2008). This consists of building a description of the LLM during forward pass, in terms of computation graph of its operations, and storing all (possibly intermedi- ate) results required to subsequently compute the gradients of the neural network. The simplest way to reduce acti- vation memory is by breaking batches into smaller micro- batches and performing more gradient accumulation steps. Other techniques include gradient checkpointing (Chen et al., 2016; Jain et al., 2020), which trades off compute for activation memory by recomputing quantities during the backward pass, and random projections (Bershatsky et al., 2022; Liu et al., 2023), which produce stochastic estimators of gradients based on sketched activations.
2. Background
This work is mostly concerned with training LLMs in memory-constrained regimes where the model, optimizer, and gradient memory dominate, since activation storage can be made small by, e.g., reducing the microbatch size, As such, we will only focus on those categories in our cal- culations.
2.1. Memory Characteristics of LLM Training
LLM training makes use of accelerators like GPUs, which requires storing important data in rapidly accessible, on- device memory.2 2 The bulk of this memory consumption can be broken down into four main categories.
2.2. LoRA and GaLore
Model parameters. We must keep the model's parame- ters in memory, since these are used in various stages of the training process (e.g., to compute gradients). Here, it is useful to distinguish trainable parameters (which get up- dated regularly during training) from non-trainable param- eters (which are not updated during training but may still be used in gradient computation).
This paper centers mainly around two memory--fficient training techniques: low-rank adapters (LoRA; Hu et al., 2022) and gradient low-rank projections (GaLore; Zhao et al., 2024). LoRA reparameterizes the model's linear layers as Y (W + AB)X, where W is the model's original weight matrix and A, B are matrices such that rank(AB) < rank(W). W remains frozen and only A, B are optimized; thus, while the total number of model parameters is increased, the number of trainable parameters is decreased, which can lead to memory savings. Recent
Gradients. LLMs are trained using (variants of) stochastic gradient descent, which requires an estimate of the gradient of the loss function with respect to each trainable parame- ter. Standard LLM training uses a large number of samples to estimate the gradient, which necessitates gradient accu-
'While there are methods that perform an optimizer step as soon gradient is estimated (thus eliminating the need to al- locate memory for gradient accumulation; e.g., I.MOO Ly et al., 2024), this is not standard LLM training since it can place re- strictions on sequence length: for example GaLore with LOMO only trains on 256-1ength sequences. We thus train with gradient accumulation in the present work.
'Note that this notion of performing gradient descent with pro- jections the gradient distinct from projected gradient descent (PGD) from the optimization literature.
'Wile offloading to CPU is theoretically possible, bandwidth limitations often make this infeasible in practice.
2

---

On the Duality between Gradient Transformations and Adapters
works obtain even further memory savings by working with compressed version of W (Dettmers al., 2023; Guo et al., 2024; Li et al., 2024b). While LoRA was originally proposed in the context of memory-efficient finetuning, ReLoRA (Lialin et al., 2024), LoRRAtthe-Exploee (Huh et al., 2024), and Flora (Hao et al., 2024) show that by pe- riodically merging the low-rank components with the full weights and reinitializing them. LoRA can enable reason- ably performant memory-effiiien pretraining from scratch.
gradient of some differentiable loss function L of the net- work f with respect to O, computed on random data minibatch. We use the superscript ())() to specify a par- ticular quantity's 's value after t optimizer steps, e.g., 6(t) are the network's parameters after t optimizer steps. We are also interested in the optimization trajectory of a model (e(1), @()), (())). Our results show that two opti- mization trajectories-one from training with linear gra- dient transformations, and one from training with linear adapters are equivalent.
GaLore provides an alternative approach to memory- efficient pretraining. Instead of reparameterizing the weights to be a combination of full-rank and low- rank matrices-whit increases the number of model parameters-GGal.oo ooe performs a low-rank compression of the gradient matrix W instead. The optimizer update is computed in this lower dimensional space, transformed back to the original space, and only then applied to the parameters. Specifically, given gradient matrix W E Rmxx, GaLore uses a matrix P E R (with k < m) to transform the gradient via PW E Rtx", feeds this compressed gradient into a regular optimizer to obtain a pseudo-parameter update A RKxx, and then updates the original parameters via PTA. A. In practice, P is given by the top singular vectors of W, where in order to amortize the cost of SVD, P is updated only every so often. As with LoRA, GaL reduces the memory needed to store the op- timizer states, since optimization happens in the lower di- mensional space.
Typical approaches to neural network optimization use op- timizers that maintain a state 5o and obtain ((++1) via,
(A Se
(1) e(++1) Optimizer(@"
(t) ,Se
( = ( + +
(2)
For example, Adam (Kingma & Ba, 2015) maintains first- and second-moment estimates of the gradient entries in its state <(0) (p&
(t) , (l)). and the optimizer update is:
(++1) = (1 - 81)0) + + (e) Me ((+1) = (1 (11-)))(0 22 Ve,
(G A
P8.1
(+++) +e
where 7 E R$ is the learning rate, BI, .2 [0, 1) control the exponential moving averages of the gradient moments, and is present for numerical stability. In this case, the dimensionality of the optimizer states is proportional to the dimensionality of our gradient estimate dim(e") = d, 2d
i..., A (++1) R2 ((t)
3. Duality between Linear Gradient Transformations and Adapters
,e
Now consider optimizing θ with linearly transformed gra- dient dynamics, where the gradient θ is mapped to an r- dimensional space by a matrix S E grxd In this case, we can use the transpose of the linear map to go back into the original parameter space resulting in the following update:
In this section, we prove that training a neural network us- ing linear transformations of the gradient is equivalent to reparameterizing the neural network using specific linear adapters. We begin with the general case, where all pa- rameters are treated as arbitrary vectors (Thm. 1). We then show how applying a fronecker-factored linear transfor- mation to the gradients of linear layers of the network is equivalent to training the model with a version of LoRA which inserts trainable matrix between the LoRA matri- ces (Prop. 2). From this, we further show that specializing to specific choice of Kronecker-factoree transformation establishes an equivalence between GaLore (Zhao et al., 2024) and "one-sided" LoRA (Hu et al., 2022) where one of the LoRA matrices is initialized in a particular way and kept frozen, while only the other one is trained (Cor. 3); this recovers the equivalence established recent work (Hao al., 2024; Loescheke et al., 2024).
((+1)) (A Sse
Optimizer(SE) (1) (t) -sse)
((++1) (()) + T +
Ase
where we have used the subscript SO to emphasize the fact that the optimizer is now operating on a different space, i.e., as if we were optimizing on IR", instead of the original pa- rameter space, Rd. For example, if we were using Adam as our optimizer, then this change would cause the dimension- ality of the optimizer update and states to be proportional to r instead of d, viz., A ((+1) E RR',s()) E R
c(t)
se
Let us further consider a reparameterization of the neu- ral network parameters as f(X: + SSTSSSTT ST A) with A E RT. Specifically, suppose that we keep θ and fixed, and only
*We omit bias correction for simplicity: one can easily han- dle it by adding the current timestep to our optimizer state. This change would also allow to add learning rate schedules.
3.1. General Case
Let /(X; θ) be neural network over input X with train- able parameters θ e Rdd and further let θ Rd be the
3

---

On the Duality between Gradient Transformations and Adapters
optimize A, resulting in the following update:
layer of moderate size, and becomes rapidly intractable if we consider applying gradient transformations to the en- tirety of a model's parameters. As such, practical applica- tions need to consider matrices S that are efficient to store in memory (and also efficient to apply to +).
(A 1 = Optimizer(A' (A (1) 1 = - +
(3a)
(3b)
Because the above is adapting neural network indirectly via another vector A that is linearly mapped to the original parameter space, we refer to this as using a linear adapter, akin to the usage of "adapter" in the parameter-efficien finetuning literature (Houlsby et al., 2019; Hu et al., 2022). Since S E Rdxr, we have dim(AX)) dim(sX") dim(s$e). i.e., the output and states of our optimizers have the exact same dimension in both cases. This not a coincidence: we now show that optimizing this linear adapter when A is initialized to equivalent to optimizing O in the original neural network with lin- early transformed gradient dynamics. (For this and all sub- sequent proofs, refer to App. A).
To this end, we consider Kronecker-fuctored linear maps of the form S R @ L where L E R xm R Raxda, d r. This particular parameterization of S reduces the memory requirement to O(dym + ndz) and FLOPs to (O(r((Odd.nndd (since SO = vec (LWR)), which can be memory- efficient if d,, dg are small enough. We now show apply- ing Thm. 1 to such an S establishes an equivalence between training with gradients transformed by L LWR, and repa- rameterizing the linear layer as W LLARR ARR" and only training A R
(() dim( ( and
((
proonecker-fcttreee parameterizatton of the linear map). Let W E |m be the parameter matrix of linear layer with corresponding gradient matrix W Rmxx, Further let θ = vec (W) and O = vec (W). Consider training O as above with S = R. s0 L, i.e., by transforming the gradient matrix via LWR. Then the opti- mizer trajectory of W is equivalent to reparameterizing the model as W = W + T.T.R. aand then just training A (after initializing A (0) 0).
Proposition
Theorem 1 (Equivalence of gradient transformations and linear adapters). Suppose we are given initial parameters ())) and state Sge- Let @(t) be the parameters after up- date steps with the linearly transformed gradient dynamics with S. Now consider a linear adapter which reparameter- izes the model as (0) + STA A, where A!)) is initialized to and the optimizer state [ is initialized to E and only
A is optimized. Then we have ( (0) + S A for
all i.e., the optimization trajectories are equivalent.
Remark. Prop. 2 shows that MoRA (Jiang et al., 2024), LoRA-XS (Balazy et al., 2024), and PMSS (Wang et al., 2025), which are recent approaches to parameter-efficient finetuning which reparameterize a linear layer as W + BAC and only train A, can be interpreted as training the model with traffrred gradients where the linear transformation has Kronecker factorization.
Remark. The above only requires that the reparameterized model is equivalent to the original model at initialization, and can therefore be straightforwardly extended to cases where the adapter is not initialized to 0, as long as we have (0)) Ô + ST (0) for some Ô and (0))
Remark. The above theorem holds for any optimizer of the form in eq. (2), e.g, Adam (Kingma Ba, 2015). Notably, AdamW (Loshchilov & Hutter, 2019) does not fit this def- inition due to the way that weight decay is applied. See App. B for discussion about weight decay, and what ad- justments are required to preserve the equivalence.
Finally, as simple corollary we now show that one can set S in a way that recovers GaLore, which in reparameterized form corresponds to one-sided LoRA, i... fixing one of the adapter matrices and only the training the other.
Corollary 3 (GaLore is one-sided LoRA). Let W
3.2. Kronecker-factored Gradient Transformations
R= be the parameter matrix of linear layer with cor- responding gradient matrix W E Rmxx. Without loss of generality, assume m < n. Now consider training W with Optimizer using GaLore, i.e., where we linearly transform the gradient matrix with a matrix P. apply the optimizer. and transform the update via PT. viz.,
The formulation in Thm. 1 assumes very little about the neural network being trained and the gradient transforma- tion (or, equivalently, linear adapter) being applied, which makes it difficult to enable practical memory savings. Con- cretely, consider applying an arbitrary linear transforma- tion to just a single linear layer of neural network with pa- rameters W R mmx, i.e., ((x = WX. In this case we have θ = vec (W)= E Rmn 5 and thus arbitrary linear maps of the form S E Rm require (mmrr) memory to store. This cost is already toi-tiivial for single linear
(AW.sh clt+-1)) = Optimizer(vee (PW(1)) (1))
:5W
w(++1 = ( +P vec (AW)
(4)
where P is an arbitrary matrix of size Rdxxm and d < m controls the dimensionality of the transformation. Then the optimizer trajectory of this network is equivalent to a net- work trained with the reparameterization W- = w(0) +
Intuitivvlly, vee (.) sends matrix to its vectorized form (i.e., stacks its columns into vector), and ve (-) is its inverse (i.e., unstacks vector back into matrix form).
PT A, where only A is learned.
4

---

On the Duality between Gradient Transformations and Adapters
Method
Adapter Parameterization
Trained
Frozen
Persisted
Baseline
W
W
W
ReLoRA (Lialin et al., 2024)
W + BA
A,
B
W
W, A, B
Gradient SVD (GaLore: Zhao al., 2024)
W P A.
P WVD(W)
A
W.P
W. P.P.A
Gaussian (Flora; Hao et al., 2024)
W + P A.
P ~ IO
A
W.P
W, A
Rademacher
W W+P A, P À (111)
A
W.P
W. A
Random Semi-orthogonal
W W+PAA A,
P =
A
W.P P
W. P,
Two-sided Gaussian
W L AR L, L.R~AV(O.)
A
W. R
W A
Two-sided Gradient SVD
W + L AR L R SVD(W)
A
W,L,R R
W R,R.R
A summary of methods tested for our pretraining experiments, where we list the gradient transformation method (which is not relevant for Baselinc/ReLoRA) and the corresponding adapter parameterization. We also break down the reparamcterized model into trained and frozen components, alongside the the set of components that need to be persisted in memory; for methods that make use of moaaretaaatriilieeeeeeeeee random sketching matrices (e.g., Gaussian) one only needs to persist the random number generator seeds for the gradient transformation, saving memory. Random semi-orthogonal matrices-a tall/wide matrix whose columns/rows vectors are also random but are not straightforwardly materializable from a seed, and hence may need to persisted across optimiza- tion steps. In the Gaussian and Rademacher cases, we use &s as shorthand for the constant that ensures that E PP I.
Table 1:
orthonormal
Remark. The original GaLore work advocates for swap- ping the gradient transformation every 200 optimizer steps. This does not break the equivalence in Cor. 3. In the adapter formulation, recomputing the gradient transfor- mation corresponds to merging the learned adapter into the frozen weights, updating the frozen part of the adapter, and resetting the learned part to zero. This effectively amounts to ReLoRA (Lialin al., 2024), where one side of the adapter is kept frozen throughout training.
formance, and how by viewing gradient transformations as adapters, further improve memory efficiency by com- bining the technique with QLoRA-style (Dettmers et al., 2023) training ($4.1). We then show how the converse is also useful: by viewing LoRA adapters through the lens of gradient transformations, we can improve distributed train- ing of LoRA adapters by coordinating the LoRA adapter initialization across different workers ($4.2).
Experimental setup. We consider two moderate-scale lan- guage modeling settings: a 200M setting (training on 5B tokens) and a 1.3B setting (training on 10B tokens)." We use the Ilama Transformer architecture (Touvron et al., 2023a) and train on the SlimPajama (Soboleva et al., 2023) dataset, tokenized using the Llama-2 (Touvron et al., 2023b) tokenizer, using sequences of length 2048, All numbers we report are perplexity on disjoint (validation) set of SlimPajama. We use AdamW (Loshchilov & Hutter, 2019) with weight decay 0.1, 811 = 0.99 and B3 = 0.95.
While Prop. 2 and Cor. 3 focus on the case of a single linear layer, it is straightforward to generalize them to multiple linear layers. For example, one could treat the parameters of all layers as a single vector living in the product space of the individual layers' parameter spaces, and define the gra- dient transformation map on that space as applying the correct projection to each of the layers' parameters individ- ually. This can be implemented by modifying the optimizer step to apply separate linear transformation to each layer.
We warm up the learning rate to 4 x 10-4, before decaying it via a cosine decay schedule to 1 x 10 10-4. We conduct all training in bfloat 6 precision. See App. D for more details on our experimental setup.
Finally, we note that Hao et al. (2024) and Loeschcke et al. (2024) also show that training LoRA adapters with one side frozen with ordinary SGD equivalent to applying a linear transformation to the gradient matrix, and Muhamed et al. (2024) further shows this for more general optimizers in generalization
4.1. Study 1: Memory-Efficient Pretraining
Cor. 3. Our Thm. can be thus be seen of these recent results, where we show that this equivalence generalizes to arbitrary parameters of the neural network.
The discussion in $3 establishes a direct link between Ga- Lore and one-sided LoRA. But how should we set S in practice? From the perspective of accurate gradient esti- mation, it would be ideal to have S = I, since in the vanilla SGD case this would be equivalent to performing SGD with sketched gradients, where S s R O (Murray et al., 2023). For the GaLore case with S = I & P. this amounts to setting P such that PPT v I, which could be achieved by, e.g., using random sketching matrices with the
4. Empirical Study
The equivalences $3 are agnostic to the choice of left and right transformations in S R L. However, one might expect that the choice of L and R should matter for down- stream performance. Hence, in the following sections, we first explore how the choice of S affects pretraining" per-
'While this is not large by modern standards, due to our lim- ited compute resources this is the largest setting at which we can feasibly perform experiments.
'e target the pretraining setting as the gap between ordinary training and memory-efficient training methods is typically larger pretraining than is finetuning.
5

---

On the Duality between Gradient Transformations and Adapters
200M
1.3B
SVD
Rademacher
Two-sided
Model
Gzusian
Random Semi orthogonal
Two sided Gaussian
PPL
Mem.
PPL
Mem.
Projection ertor
Full pretraining
18.58
1.32
12.44
8.04
10
ReLoRA
20.40
1.03
13.94
5.77
QGaLore (INT8)
23.86
0.94
15.23
5.15
10
Gradient SVD (Gal.ore)
21.34
0.96
13.62
5.27
+ INT8
21.38
0.81
13.65
4.06
+ NF4 (LoQT)
26.52
0.73
16.10
3.46
2000
4000
6000
8000
10000
Gaussian (Flora)
20.57
0.93
13.88
5.02
Training steps
+ TNT8
20.55
0.78
13.87
3.81
+ NF4
23.61
0.70
15.64
3.21
Figure 1: Average of gradient reconstruction error ||0 S STSl
Rademacher
20.24
0.93
13.86
5.02
of the various transformations across training steps at 200M scale.
+ INI8
20.26
0.78
13.78
3.81
+ NF4
23.37
0.70
15.64
3.21
Random Semi-orthogonal
20.13
0.96
13.71
5.27
+ INT8
20.32
0.81
13.75
4.06
further merge the adapters into the full weights and reini- tialize them every 200 steps. We see that one-sided trans- formations, regardless of their nature, perform somewhat similarly at both 200M and 1.3B scale, suggesting that us- ing a random gradient transformation matrix that can be cheaply rematerialized on-the-fly may be more economical than using the top singular vectors derived from the gradi- ent as in GaLore. We also find that ReLoRA performs com- parably to one-sided gradient transformations, suggesting that the additional flexibility of ReLoRA (i.e., optimizing two sides of LoRA adapter instead of only one side) is not necessary. Using two-sided Gaussian gradient transforma- tions degrades performance when memory consumption is matched to one-sided methods; two-sided SVD-based pro- jections fare slightly better but still trail behind one-sided methods and incur a much larger memory cost, since two projection matrices must be persisted. While Zhao et al. (2024) report no gap between GaLore and full pretraining, we did not find this to be true on our setup, 10 and instead observe a non-trivial gap between regular (full) training and these memory-efficien pretraining methods.
+ NFF4
23.41
0.73
15.44
3.46
Two-sided Gaussian
23.98
0.93
15.28
5.02
+ INT8
23.94
0.78
15.20
3.81
+ 4F4
27.93
0.70
16.95
3.20
Twa-sided Gradient SVD
22.26
1.13
14.27
6.55
+ INI8
22.08
0.97
14.16
5.35
+ NFA
26.81
0.90
17.14
4.74
Table 2: Pretraining results at 200M and 1.3B scales. We re- port validation perplexity and estimated memory requirements (excluding activations) GBs. GaLore NF4 quantization is equivalent LoQT (Loescheke et al., 2024). QGaLore (Zhang et al., 2024) quantizes both the base weights and the SVD projec- tion to INT8, but does not adopt the LoRA parameterization.
property E|PPT|=1. II As noted by Hao et al. (2024), us- ing random sketching matrix can enable further savings as only the random number generator (RNG) seed needs to be persisted across optimization steps. We thus experi- ment with a variety of sketching matrices for LoRA-based pretraining as shown in Tab. 1.
Another benefit of the adapter parameterization of gra- dient projections is that it allows us to be more mem- ory efficient by quantizing the base weights as done in QLORA (Dettmers et al., 2023). Specifically, given the adapter parameterization e + ST A we can quantize θ and only train A, thus enabling further memory savings. Fi- nally, the adapter parameterization has the additional ben- efit of reducing the number of trainable parameters being registered for automatic differentiation, which allows for gradient accumulation to happen in a lower dimensional space. (See App. D for more discussion.)
When adding quantization to the hase weights (where we use groups of size 256), we find that, across the board, 8-bit integer quantization can be performed without major per- formance degradation, whereas 4-bit NormalFloat quan- tization begins to incur a penalty (4-bit integer did even worse). This suggests that adapter-based training, with a rematerializable gradient transformation and 8-bit integer quantization of the base weights, is promising recipe for memory-efficien pretraining. Finally, we find that QGa- Lore (Zhang et al., 2024), which quantizes the weights to INT8 precision and trains these INT8 weights directly using an SVD gradient transformation, underperforms a QooRA-ttyle approach to quantized GaLore training.
Results. The results are shown in Tab. 2, where we fol- low the original GaLore paper and use a rank of 256 for the 200M model and a rank of 512 for the 1.3B model, and
trainable parameters in the one-sided Gaussian approach. For the two-sided SVD, we use the same rank as in two-sided Gaussian, which incurs more memory since the projection matrices must he persisted across optimization steps.
*This sketching view of LoRA provides possible perspective on why one-sided LoRA finetuning works well in practice (Zhang et al., 2023; Zhu et al., 2024; Hayou et al., 2024).
'The only exceptions for the double-sided methods. For the two-sided Gaussian, we set the rank as to match the number of
Whiich is different from theirs in many ways, e.g., we train on longer sequences with gradient accumulation using b oat 1.6.
6

---

On the Duality between Gradient Transformations and Adapters
Method
Projection Init.
200M
However, DiLoCo still assumes that each worker has enough memory perform a full forward/backward pass the model, i.e. it does not target memory efficiency. A memory-effiiiet distributed training approach that is of particular interest in light of the equivalence in $3 is -hee-lor (LTE; Huh et al., 2024), which can be seen as an extension of ReLoRA to the distributed set- ting. LTE has K independent workers train separate LoRA adapters for small number of local steps, and then per- forms a global step by averaging the adapters across work- ers. The globally-averaged adapter is then merged into the base weights, and optimization continues by resetting and traing the worker-specific LoRA adapters.
Dist. Training (DiLoCo) Dist. ReLoRA (LTE)
18.00
12.77
20.97
13.72
Identical Random
P. = P
21.51
14.28
Independent Random
E[P.P]] = 0 P.P. =
20.11
13.66
Distributed Random
19.81
13.51
Table 3: Results of the distributed training experiments, where four workers are trained independently and synchronized ev- ery 500 steps, following DiLoCo (Douillard et al., 2024). We use random semi semi-orthogonal matrices for the distributed (one- sided) LoRA experiments, For the (re)initializations of worker- specific projections (Px identical shares the projection ma- trix across workers, independent initializes each worker's projec- tion independently, and distributed initializes the worker projec- tions such that they are all orthogonal to one another. The top two rows baselines, viz.. DiLoCo and distributed variant of ReLoRA, which is similar to LTE (Huh et al., 2024).
We will now describe how the equivalence in $3 can be used to derive an improved version of LTE, which trains only one side of the LoRA adapter in each worker, but ini- tializes the frozen side in a worker-aware manner. Consider one-sided analogue of LTE, where the weight wr(.)) for the kth worker after g global and local updates is
Analysis. We have motivated our experiments with sketch- ing matrices from the perspective of accurate gradient com- pression, i.e. we use S to compress the gradient, and then ST to decompress it. From this compression viewpoint, one may then wonder whether different gradient transfor- mations exhibit different reconstruction capabilities, and whether this ultimately dietates the performance of the re- sulting model. As shown in Fig. 1, we find that the gradient reconstruction error does not correlate with performance. As expected, methods that perform SVD on the gradients have low reconstruction error (since SVD explicitly mini- mizes a reconstruction objective), but as shown in Tab. 2, SVD performs similarly to sketching matrices, which have higher reconstruction error. We believe that the relation- ship between the nature of the gradient transformation and downstream performance is fairly complex, and merits fur- ther investigation. Fig. 2 of App- C shows similar results for cosine similarity instead of squared error.
ll..1) wy(..0) + +P(
A (g.9
and only A is trained. The global step is given by,
K
W (g+100) = w
(g.L)
K
1=1
where we have assumed that the global step is performed after L local steps. After global step, we would also reset A by setting A (1+1.00) another (e.g., random) matrix for all
and similarly swap out P3 for
By Cor. 3, local steps must correspond to training the worker weights using a gradient transformation,
w(aa) w( +P(() APW
where we use A (g.1) was performed in the lower dimensional transformed space. Further, a global step in this view can be equivalently seen as defining a global sreudo-gradient A(9) as the average of
PW to denote the optimizer update that
4.2. Study 2: Distributed Pretraining
for the local -ouudo-gradiets {4(0) (g) Ag)
Our second experiment targets distributed pretraining of LLMs across poorly-connected and resource-constrainee workers, which is important for many applications of inter- est, from federated training of LLMs to scaling up LLMs across data centers that are not co-located, i.e., where tech- niques like FSDP are not possible. DiLoCo (Douillard et al., 2024) is recent and effective approach that has workers train independently for some number of iterations using an inner optimizer, and then uses the average change in parameters from each worker as a "pseudo-gradient" on an outer optimizer that updates global copy of the param- eters (i.e., in federated learning: McMahan et al., 2017: Reddi et al., 2021). This approach has since been scaled up to train 10B LLMs across distributed workers. !!
K
(a)
1
Σ
(9)
(9) = (( - w(000)
K
The global weight update is then given by a step using the global -gaadiee,,
((0+ = W(..0) + +(0))
This can be straightforwardly generalized to the use of dif- ferent learning rates and more advanced optimizers.
One approach to initialize/reset the frozen side of worker- specific LoRA adapters (i.e., the gradient projections P,)) is to sample a projection and broadcast it to all workers. However, the MaLore-LoRA, duality suggests different scheme. Thm. shows that training with linear gradi-
"See INTELLECT: and (Jaghouar et al., 2024).
7

---

On the Duality between Gradient Transformations and Adapters
(Rank, Workers)
As in $4.1, distributed GaLore leads degradations both 200M and 1.3B scales compared to the full distributed training baseline (i.e., DiLoCo). However, our distributed random initialization scheme, where workers are "aware" of each, performs well, thus demonstrating the utility of the gradient meansformation-aaappee duality from $3.
Method
(128, 8)
(256, 4)
(512, 2)
Dist. Training (DiLoCo)
17.81
18.00
18.56
Dist. ReLoRA (LTE)
23.76
20.97
19.54
Identical Random Independent Random
23.96
21.51
20.32
20.64
20.11
19.97
Distributed Random
20.32
19.81
19.66
Analysis. We perform study at the 200M scale over how the number of workers and rank affect performance. Intu- itively, larger ranks lead to a larger subspace being trained by each worker (and, in the limit, we should recover some- thing akin to DiLoCo when there is no rank reduction), so we would expect performance to improve as we increase the rank. Indeed, the results for this ablation (shown in Tab. 4) confirm this intuition, likely because Dil.CCo ben- efits from more workers get better estimate of the bpeudo-gradient for the outer optimizer step. More sur- prisingly, we find that this gap is largely bridged by ensur- ing that different gradient transformations are assigned to each worker, with the distributed initialization once again performing the best. It would be interesting to further study how the effectiveness of the distributed initialization scheme changes as we go more extreme settings (e.g.. hundreds of extremely low-rank workers).
Table 4: Results of the distributed pretraining experiments as we vary the rank of the gradient projections and number of work- ers. For the DiLoCo baseline, we only vary the number of work- ers, which means that we increase the number of workers, the DiLoCo baseline can only benefit since we are training on more data without any downside (i.e., a rank restriction). Note that the for the distrihuted ReLoRA baseline (which is similar to LTE, Huh et al., 2024), we have double the number of trainable parameters as in the one-sided methods.
ent transformations only optimizes a subspace of the full model, namely range(ST) where S = 6 P the Ga- Lore case. This suggests different approach to distributed LoRA training, wherein the frozen part of each LoRA adapter, S1. , Sk. is initialized differently, so that the sum of their ranges allows larger subspace to be trained. For example, we could sample random semi-orthogonal matrices P PI....,PP PK uniformly at random, assign each worker S, = I 6 PT, and they will likely each cover different portions of the space. An even stronger strategy would be to demand that range(S, ... range(S) (Sk) must be mutually orthogonal, which can be realized by keeping the P.'s as seti-orthogonal, but enforcing that P.P. = =0 (e.g.. by generating a random m X m orthogonal matrix and having each worker take a different d xx m submatrix.) Intu- itively, this ensures that no worker is duplicating the work of another, since their projections are pairwise orthogonal. We experiment with such identical random, independent random, distributed random initialization schemes.
5. Discussion and Limitations
The preceding studies focus on two situations in which the duality between linear adapters and gradient transforma- tions offers practical insights. We believe there are many other avenues that merit further exploration. For instance, Thm. 1 makes no assumptions about the structure of S; while we only considered Kronecker-factorizee matrices, other linear maps that admit efficient storage and compu- tation would be interesting to explore. The transformation also just needs to be applied to a gradient vector, suggest- ing that one could compress the gradient of multiple layers in a network jointly. This amounts to sharing LoRA pa- rameters across layers the GaLore-LoRA case, which has been found to be successful for parameter-efficient fine- tuning (Renduchintala et al., 2024; Song et al., 2024). ee- gardless of the structure of S, as discussed in $4.1, what characterizes a good S is not clear but has a large impact. It may be possible to learn a good S with meta-learning- style approaches, which can be seen as learning an op- limizer (Andrychowicz et al., 2016; Li & Malik, 2016; Wichrowska et al., 2017: Bello et al., 2017, i.a.). 14 Fi- nally, while we focused on linear gradient transformations, where we proved exact equivalence with a linear adapter parameterization, it may be possible to establish approxi-
Results. The results for the main set of experiments are shown in Tab. 3. We consider two baselines: (i) DiLoCo (Douillard et al., 2024), which has each worker training independently for 500 steps before computing a weeudo-gadden that is used to update the global parame- ters using SGD with Nesterov momentum (Sutskever et al., 2013), and (ii) distributed ReLoRA, which is analog of ReLoRA but adapted to train like DiLoCo, i.e., one trains the LoRA adapter for 500 steps and defines the adapter weight as the -raddint for the Nesterov step; this is very close to LTE, 12 Our distributed GaLore experiments make use of random semi-orthogonal projections since the distributed random initialization for it is easy to compute, and does not add significant communication overhead." 13
LTTE can be seen as using SGD as the optimizer on the pseudo-gradients, but we found this led to worse results in pre- liminary experiments.
"In the GaLore/LoRA case, learning P in this meta-lcarning sense is different from learning P in the ordinary LoRA sense, i.e., when both P and A are trained with gradient descent against the same loss function.
Fach worker just needs the seed used to sample the orthogo- nal matrix, and the indices of the rows will keep.
8

---

On the Duality between Gradient Transformations and Adapters
mate equivalence between non-linear gradient transforma- tions and other types of adapters.
7. Conclusion
We proved a general equivalence between training an LLM with linear transformations of gradients and training with additive linear adapters, and showed the LaLA equivalence is special case of this result. We then used this equivalence to derive more memory-effiiint and per- formant methods for LLM pretraining, including combina- tions of quantization and gradient- methods and improved initialization for distributed adapter pretraining.
Our work has several limitations. Due to compute con- straints, we were only able to scale our experiments to 1.3B, which is small by industry standards. While our duality results are more general, our experiments primar- ily focus on the special case of the GaLore-LoRA, dual- ity. We chose focus primarily on a wide array of gra- dient transformations, but forgo study of the interaction between such transformations and the choice of optimizer, projection reinitialization schedule, etc. Ultimately, we be- lieve that our results signal that these techniques could be applied at larger scales, especially when performing dis- tributed training in memory-constrained regimes.
Impact Statement
The last few years have seen widespread interest in LLMs. Perhaps the most salient finding from the race to build the best LLMs is that increasing parameter counts in tandem with data is of paramount importance. This makes it very hard to train competitive LLMs unless one has the best and latest hardware, which offers the most memory ca- pacity and thus the ability actually train these models in practice. Our research targets exactly this setting, of- fering mathematical connection between two methods at the cornerstone of memory-efficien training, and showing how this connection can lead to further improvements in memory-efficiency and distributed training.
6. Related Work
Memory-efficient training. There is a growing body of research focused on memory-efficient LLM training. This work explores the connections among GaLore (Zhao et al., 2024), LoRA (Hu et al., 2022), QLoRA (Dettmers et al., 2023), and ReLoRA (Lialin et al., 2024). Various ap- proaches in low-rank adaptations have been proposed to en- hance these techniques (Renduchintala et al., 2024; Sheng et al., 2024; Zhang et al., 2023; Xia et al., 2024; Wang et al., 2023b; Hao et al., 2024; Wang et al., 2025), includ- ing efforts to train models from scratch (Kamalakara et al., 2022; Wang et al., 2023a; Zhao et al., 2023). Broadly, memory-efficient training also encompasses methods such as adapters (Houlsby et al., 2019; Karimi Mahabadi et al., 2021), which insert trainable layers and prompt tuning (Li & Liang, 2021; Lester et al., 2021), which optimizes con- tinuous prompts. Additionally, its combination with quan- tization techniques (Kwon et al., 2022) and other methods that update subparts of the parameter vector (Guo et al., 2021; Ben Zaken et al., 2022; Sung et al., 2021) are also relevant.
Acknowledgements
We thank Shannon Zejiang Shen, Li Du, Aniruddha Nrusimha, Jeremy Bernstein, Jyothish Pari, Sami Jaghouar, Johannes Hagemann, and the anonymous reviewers for helpful discussions and feedback. This work was partially supported by the National Science Foundation under CA- REER Award No. 2441872 and the MIT-IBM Watson AI Lab.
References
Adelman, M., Levy, K. Y., Hakimi, I., and Silberstein, M. Faster neural network training with approximate tensor operations. In Neural Information Processing Systems, 2021. URL https::/// //t:://ddL acm. org/ doi/ 10.5555/354 02..54542225....0
Memory-reduction via randomization. Randomization has been used in other contexts to reduce memory con- sumption in automatic differentiation. Adelman et al. (2021) and Liu et al. (2023) perform row/column subsam- pling to reduce the amount of computation and memory required to compute gradients. Bershatsky et al. (2022) also explores Gaussian projections, but in the context of re- ducing activation memory by sketching them. Oktay et al. (2021) construct gradient estimators by computing the gra- dient on a subsample of the paths in the computation graph. More tangentially, MeZO (Malladi et al., 2023) amounts to sketching the gradient of a neural network by performing forward-mon automatic differentiation on random vectors.
Andrychowicz. M., Denil, M., Colmenarejo, S. G., Hoff- man, M. W. Pfau, D., Schaul, T., Shillingford, B., and de Freitas, N. Learning to learn by gradient descent by gradient descent. In Neural Information Processing Systems, 2016. URL https: //dl//ll///// acm. wwg//ool/../ooooooooooooo..//ooo 10. 00555555315555 3157543.
Anil, R., Gupta, V., Koren, T., and Singer, Y. Memory- efficient adaptive optimization. In Neural Information Processing Systems, 2019. URL https: : //dl. acm. htt////i111...0 org/ /10 1.500.555555 55555355553355122 34551611.
9

---

On the Duality between Gradient Transformations and Adapters
Batazy, K., Banaei, M., Aberer, K., and Tabor, J. LoRA- XS: Low-rank adaptation with extremely small number of parameters. arXiv preprint, 2024, URL https:// arxiv. owg// abs/ 00os/2405.1200 17604.
Griewank, A. and Walther, A. Evaluating Derivatives. So- ciety for Industrial and Applied Mathematics, second edition, 2008. URL http::////bb..rr1111133333
1.97808987177L1
Bello, I., Zoph, B., Vasudevan, V., and Le. Q. V. Neural optimizer search with reinforcement leamn ing. In International Conference on Machine Learn- ing, 2017. URL https::/p///oc /ddd.aa. acm. hts//do//10
Guo, D., Rush, A., and Kim, Parameter-efficient transfer learning with diff pruning. In Association for Computational Linguistics, 2021. URL https:// aclanthology orgg 2021 aacl -oong-on...... 378/.
Guo, H., Greengard, P., Xing, E., and Kim, Y. LQ- LoRA: Low-rank plus quantized matrix decomposi- tion for efficient language model finetuning. In In- ternational Conference on Learning Representations, net. forum??
Ben Zaken, E., Goldberg, Y., and Ravfogel, S. BitFit: Simple parameter-efficient fine-tuning for transformer- based masked segguagge-mdells
Association for Computational Linguistics, 2022. URL https:// aclanthology http://222..llshht.t..
2024. URL
id=xw2 9VvŌMmU.
Bernstein, J., Wang. Y.-X., Azizzadenesheli, K., and Anandkumar, A. signSGD: Compressed optimi- sation for non-convex problems. In International Conference on Machine Learning, 2018. URL https: httpss///cccceeings.... wtt.oceeiings.m...presss.. bernstein18a, html.
Hao, Y., Cao, Y., and Mou, L. FLORA: Low-rank adapters are secretly gradient compressors. In- ternational Conference on Machine Learning, 2024. URL https: /dl accm aacm.org/// org/ worg/doiioo /10.5555/ 3 3692070.3692777
Hayou, S., Ghosh, N., and Yu, B. The impact initial- ization on LoRA finetuning dynamics. In Neural In- formation Processing Systems, 2024. URL https: //tp/////peeeeeee //openreview. net taree/forruee forum? d=nnndrrRTck..
Bershatsky, D., Mikhalev, A., Katrutsa, A., Gusak, J., Merkulov, D., and Oseledets, I. Memory-efficient back- propagation through large linear layers. arXiv preprint, 2022. URL https:/ //tp://aaavvv.///a///aae/eeelaaa org/ 2201. 13195.
Houlsby, N., Giurgiu. A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learn- ing, 2019. URL https: https::/ppocceeeinnnt press/v99 hwtp////1h/lee htthhhollsllllyyy hhtml.......................
Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep nets with sublinear memory cost. arXiv preprint, 2016. URL https: /ttp:///axxivv.. org/ /tbb/1604. 06174.
Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8-bit optimizers via block-wise quantization. In International Conference on Learning Representations, 2022. URL bttps://openreview. wttp:viwwwwweeeo.nnntt/ id=shpkpVXzo3h,
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:// openreview net wwwrrmiiddnneeeeeee.....
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettle- moyer, L. QLoRA: Efficient finetuning of quantized LLMs. In Neural Information Processing Systems, 2023. URL http::///pperrvieee net /formm? id-OUIFPHEqJU.
Huang, S., Hoskins, B. D., Daniels, M. W., Stiles, M. D., and Adam, G. C. Low-rank gradient descent for memory-effiiint training of deep in in-memory arrays. ACM Journal on Emerging Technologies in Comput- ing Systems, 2023. URL https:t/p///c htt//oi.ooooooooooooo wowpo.oo......... 1145/3577211
Douillard, A., Feng, Q., Rusu, A. A., Chhaparia, R., Donchev, Y.. Kuncoro, A., Ranzato, M.. Szlam, A., and Shen, J. DiLoCo: Distributed low-communication train- ing of language models. arXiv preprint, 2024. URL https: /ttp//axvvvvo /ttp//aaww...aaaaaa/// /2w./////111 231 05.
Huh, M., Cheung, B., Bernstein, J., Isola, P., and Agrawal, P. Training neural networks from scratch with parallel low-rank adapters. arXiv preprint, 2024. URL https: //arxiv. abs 2402.222 16828.
Gooneratne, M., Sim, K. C., Zadrazil, P., Kabel, A., Beau- fays, F., and Motta, G. Low-rank gradient approximation for memory-efficient on-device training of deep neu- ral network. In International Conference on Acoustics, Speech and Signal Processing, 2020. URL https:// eeexpllee ore. Leee.
Jaghouar, S., Ong. J. M., and Hagemann, J. OpenDiLoCo: An open-source framework for globally distributed low- communication training. arXiv preprint, 2024. URL
10

---

On the Duality between Gradient Transformations and Adapters
Jain, P., Jain, A., Nrusimha, A., Gholami, A., Abbeel, P., Gonzalez, J., Keutzer, K., and Stoica, I. Checkmate: Breaking the memory wall with optimal tensor remateri- alization. In Machine Learning and Systems, 2020. URL https:/ / /aaaiiivvv.. .rgg abs//////////// 1 91 0 02653.
Li, Y., Yu, Y., Liang, C., Karampatziakis, N., He, P., Chen, w., and Zhao, T. LoftQ: LoRA-fine- tuning-aware quantization for large language models. In International Conference on Learning Representa- tions, 2024b. URL https:/ /ttp///ppeereveeeeee nett/ forum? 11=.
Jiang T., Huang, S., Luo, S., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., Zhang. Q.. Wang, D., and Zhuang, F. MoRA: High-rank updating for parameter-efficient fine- tuning. arXiv preprint, 2024. URL hetps:tpss/. /arxiv. org/abs/ 2405 12130.
Lialin, V., Muckatira, S., Shivagunde, N., and Rumshisky, ReLoRA: High-rank training through low-rank up- dates. In International Conference on Learning Rep- resentations, 2024. URL https:/ pestps://opeereeeee openreview. net/ forum? diddD.ddSd......
Kamalakara, S. R., Locatelli, A., Venkitesh, B., Ba, J., Gal, Y., and Gomez, A. N. Exploring low rank train- ing of deep neural networks. arXiv preprint, 2022. URL https: // /eta://arxiie arxiv.i .wr.viioorr//oooooe.oo. abs///////// 2209..22222222222 13569.
Liu, 7.,, Wang, G., Zhong, S., Xu, 7., Zha, D., Tang, R., Jiang, Z. Zhou, K., Chaudhary. V., Xu, S., and Hu, X. Winner-take-al column row sampling for memory efficient adaptation of language model. In Neural In- formation Processing Systems, 2023. URL https: //openreview net www.fformmiooo SquMMNykkk..
Karimi Mahabadi, R., Ruder, S., Dehghani, M., and Hen- derson, J. Parameter-efficient multi-task fine-tuning for transformers via shared hypemetworks. In Association for Computational Linguistics, 2021. URL https: / //aalanthologyy wowpooyy..ooo.ooooooooo..o orr. 2tt///2211..ccccc laooo.ong.44
Loeschcke, S. B., Toftrup, M., Kastoryano, M., Belongie, S., and Snæbjarnatson, V. LoQT: Low-rank adapters for quantized pretraining. In Neural Information Processing Systems, 2024. URL http:::///ppeeevieeeee n
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. URL 0tss/1412.6988
Loshchilov, and Hutter, F. Decoupled weight decay regu- larization. In International Conference on Learning Rep- resentations, 2019. URL https: openreview. net net/ for//formm wid-Bkggkiicaa
Kwon, S. J., Kim, J., Bae, J., Yoo, K. M., Kim, J.-H., Park, B., Kim, B., Ha, J. Sung, N., and Lee, D. AlphaTun- ing: Quantization-wware parameter-efficient adaptation of large-scale pre-trained language models. Empir- ical Methods in Natural Language Processing (Find- ings), 2022. URL https:://// ptapp:://acaaaee http///caantttt wwwllnnholooyyooooo org/ 2022. findings ammlp.2400
Lv, K., Yang, Y., Liu, T., Guo, Q., and Qiu, X. Full param- eter fine-tuning for large language models with limited resources. In Association for Computational Linguis- tics, 2024. URL https:/ acc antho ogy. . .org/ 2024 acl 10l1-10m long. 445.
Lester, B., AI-Rfou, R., and Constant, N. The power scale for parameter-efficient prompt tuning. In Empirical Methods in Natural Language Process- ing, 2021. URL https: //laaaaannoooyy..oooooooo org/ 2021. emn 1p-main-mnnn.. 243.
Malladi, S., Gao, T., Nichani, E., Damian, A., Lee, D., Chen, D., and Arora, S. Fine-tuning language models with just forward passes. In Neural Information Process- ing Systems, 2023. URL https: /opnreviene.... net / forum? weriiiitoootrhhhhte
Li, B., Chen, J., and Zhu, J. Memory efficient optimizers with 4-bit states. Neural Information Processing Sys- tems, 2024a. URL https: https://openreet https////peeeeee.eeee...
McMahan, B., Moore, E., Ramage, D., Hampson, S., and Agüera Arcas, B. Communication- efficient learning deep networks from decentral- ized data. In Artificial Intelligence and Statis- tics, 2017. URL https:/ https://proceedngss press / v54 4/ /t/m//mmaaaaaaa htmml.
Li, K. and Malik, J. Learning to optimize. arXiv preprint, 2016. URL 1606. 01885.
Muhamed, A., Li, O., Woodruff, D., Diab, M. T. and Smith, V. GRASS: Compute efficient low- memory LLM training with structured sparse gra- dients. In Association for Computational Linguis- tics, 2024. URL https: ptttps://aclanthhlooe. www.aaaanlooooo.ooooooo .org/ 2024. emn 1p-maamaa maim.nin in 835/.
Li, X. L. and Liang, P. Prefix-Tuning: Optimizing con- tinuous prompts for generation. Association for Computational Linguistics, 2021. URL https:/ ///
acl wellanhoology.org/2200 agl- aalllloo..... 3533.

---

On the Duality between Gradient Transformations and Adapters
Murray, R., Demmel, J., Mahoney, M. W., Erichson, N. B., Melnichenko, M., Malik, O. A., Grigori, I.., Luszezek, P., Dereziński, M., Lopes, M. E., Liang, T., Luo, H., and Dongarra, J. Randomized numerical linear algebra: A perspective on the field with an eye to software. arXiv preprint, 2023. URL https:/ www.ii.rggaass.////oooooo......a 2302. 11474.
Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In International Conference Machine Learning, 2013. URL https: http:s://rroee /ttpp//ecccee..... mir.press, v28/ wtssssssseerrrr 3. html.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage models. arXiv preprint, 2023a. URL https: //arxiv. warxxvv.rg/aaee www.orgg///............... 2302 13971.
Oktay, D., McGreivy, N., Aduol, J., Beatson, A., and Adams, R. P. Randomized automatic differentiation. In International Conference on Learning Representations, www.eeet/fffooo net./frumm 2021. URL https:t/tss///t http///pppeevvveeeeeeeeeeee httpps:/oppeeeeeieieeeeee id id=xpx92j7CU1Y.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W. Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kar- das, M., Kerkez, V., Khabsa, M., Kloumann, I,, Ko- renev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y.. Mao, Y., Martinet, X., Mi- haylov, T., Mishra, P., Molybog, I,, Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B. Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama Open foundation and fine-tur tne-ttned chat models. arXiv preprint, 2023b. URL https:/ /at//aaxvvv..//// .rgg/ abs/ /20//22322222 1.33000000 09288.
Reddi, J., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., and McMahan, H. B. Adap- tive federated optimization. In International Confer- ence on Learning Representations, 2021. URL https: //openreview, net. / forum? -=nkkii===. 1tA.kkGG331111001 313
Renduchintala, A., Konuk, T., and Kuchaiev, O. Tied- LoRA: Enhancing parameter efficiency of LoRA with weight tying. In North American Chap- of the Association for Computational Linguis- tics, 2024. URL https:// http:///aaaattoooyyo.oo org/ n024-n .81.
Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference Machine Learning, 2018. URL mlr. httpinsss.....res//eee. shazeer188 8a html.
Wang. H., Agarwal, S., Tanaka, Y., Xing, E., Papailiopou- los, D., et al. Cuttlefish: Low-rank model training without all the tuning. Machine Learning and Systems, 2023a. URL https: www.i.owggga....aao.oooo.oooooot abs/ absgg//b///3 02538.
Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., Gonzalez, J., and Stoica, I. S-LoRA: Serving thousands of concurrent lora adapters. In Machine Learning and Systems, 2024. URL httpp:://///.t...tt.ttt //t:////rxvvvv www.iiiw.g/............... abs 2311 03285.
Wang, Q. Hu, X., Xu, W., Liu, W., Luan, J., and Wang. B. PMSS: Pretrained matrices skele- ton selection for LLM fine-tuning. In Inter- national Conference on Computational Linguisties, 2025. URL 2025 mooling-main.... 592.
Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A 627B token cleaned and deduplicated version of RedPa- jama, 2023. URL https: hhttps://ugginggfacee co/ cettasetscceeebrae http//limPaaaa a jama- ama- 627B.
org/
Wang, Y., Lin, Y., Zeng, X., and Zhang, G. Multi- LoRA: Democratizing LoRA for better multi-task learn- ing. arXiv preprint, 2023b. URL https:: http:///rr org/ abs 20la/2330 231..11501.
Song. Y., Zhao, J., Harris, I. G., and Jyothi, S. A. ShareLoRA: Parameter efficient and robust large lan- guage model fine-tuning via shared low-rank adaptation. arXiv preprint, 2024. URL https://arxiv.. http://xxxvvor////// abs/ abs/2406.15 10785.
Wichrowska, O., Maheswaranathan, N., Hoffman, M. W., Colmenarejo, S. G., Denil, M., Freitas, N., and Sohl- Dickstein, J. Learned optimizers that scale and gener- alize. In International Conference on Machine Learn- ing. 2017. URL https://///..... //l///l//// wal.aac...o otg/ dos/10. 3555/330588 3306069.
Sung, Y.-L,, Nair, V., and Raffel, C. A. Training neural networks with fixed sparse masks. In Neural Information Processing Systems, 2021. URL https: //11. acm. org/dol/10. 55551/1..5555// 05111/31552211111113.. 3542113.
12

---

On the Duality between Gradient Transformations and Adapters
Xia, W. Qin, C., and Hazan, E. Chain of LoRA: Effi cient fine-tuning of language models via residual lear- ing. arXiv preprint, 2024. URL pitps:://axit. org/ hts//bb/220 2401. 04151.
Zhang, L., Zhang, L., Shi, S., Chu, X., and Li, B. LoRA- FA: Memory-effiieet low-rank adaptation for large lan- guage models fine-tuning, arXiv preprint, 2023. URL https: //rpiii///xiii/////////a/ org/ abs/ abs//bsss0000 2308 03303.
Zhang, Z., Jaiswal, A., Yin, L., Liu, S., Zhao, Tian, Y., and Wang, Z. Q-GaLore: Quantized GaLore with INT4 projection and layer-adaptive low-rank gradients. arXiv preprint, 2024. URL https: http://///xxvv.og/aaaaaao 2407. 08296.
Zhao, J., Zhang, Y., Chen, Schäfer, F., and Anandku- mar, A. InRank: Incremental low-rank learning. arXiv preprint, 2023. URL https:t//////. /ttp///axvvvv/v//////////////avl wwt.aaaww..../aaa...//o//esreaee 2306. 11250.
Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. GaLore: -fffci-e LLM train- ing by gradient low-rank projection. In International Conference on Machine Learning, 2024. URL https: / /poperevveew
id=hHssrKDix7.
Zhu, J., Greenewald, K., Nadjahi, K., De Ocáriz Borde, H. S., Gabrielsson, R. B., Choshen, L., Ghassemi, M., Yurochkin, M., and Solomon, J. Asymmetry in low-rank adapters of foundation models. In International Confer- ence on Machine Learning, 2024. URL https: /dl. acm. www.or///ooooo /10 htt.://..555555362000 369449997777009.6611
13

---

On the Duality between Gradient Transformations and Adapters
A. Proofs
A.I. Proof of Thm.
Theorem (Equivalence gradient transformations and linear adapters). Suppose we are given initial parameters O and state <0)) Let ( be the purameters after i update sleps with the linearly transformed gradient dynamics with S. Now consider a linear adapter which reparameterizes the model ( + S' A, where A is initialized to 0 and the (0)) optimizer state 2 is initialized Ege. and only A is optimized. Then have ( O A()) for all t, i.e., the optimization trajectories are equivalent.
Proof. To show that the two optimization trajectories equivalent, will use induction to show that after every opti- mizer step 2 0 we have that optimizer states are equivalent, i.e., e(() Sse. which in turn allows us to show that the networks are identical, i... ((t) O + T (t))
Note that initialization, since A (0) 0, we have that
(0) + S = ( + 5 =
which implies that our reparameterized network identical to our original network. By assumption we also have that the optimizer states are equal cx
<(0) Now assume that this is true for t s k, i.e., for all s k
θ T A (1)
(Neural networks equivalent)
(4)
< = E
(Optimizer states equivalent)
(5)
Now note that for & + 1,
S (A + (0) ST A ( +1
(() + A
(6)
where we used eq. (3) in the first equality, and eq. (4) for the second equality. Also by eq. (4) and by the chain rule, we have that the gradients of the loss function at timestep to the (effective) parameters of the networks are the same, i.c.,
(0) ST ATA) (R)), In particular, this means that
A SS A S S ()))
where the first two equalities follow from the chain rule. Specifically, the first equality follows from writing the derivative of the loss w.r.t. the adapter parameter (a) as a function of the derivative of the loss w.r.t. the additive component of the adapter, S A (k); the second equality follows from recognizing that the latter is equal to the derivative of the loss w.r.t. the effective value ( ST (A) of the (adapted) parameter. In turn, expanding A (k) A
(k) (k+1)) Optimizer(A _<(())) (St) (A A S
(R) ck++1)
SS = ses Sse
(8)
where the first equality is the optimizer we use to train the reparameterized model in eq. (3), in the second equality we use eq. (7) and eq. (5). But by eq. (6),
(0) + ST A = (k) + + (()) + (k) (++1))
Eq. (8) proves optimizer states are equivalent for optimizer step &o + 1, whereas eq. (9) establishes the networks are equivalent for optimizer step & + 1, thus completing the proof.
A.2. Proof of Prop. 2
Proposition 2 crronecker-factoree parameterization of the linear map). Let W E RMX be the parameter matrix of a
linear layer with corresponding gradient matrix W T m Further let vec (W) and vec (W). Consider
14

---

On the Duality between Gradient Transformations and Adapters
training O above with S = R L, i.e., by transforming the gradient matrix via LWR. Then the optimizer trajectory of WW equivalent reparameterizing the model as W W +LAR annd then just training A (after initializing A (0) 0).
Proof. From Thm. 1, we know that training linear layers using the gradient transformation R using the reparameterization:
corresponds to
θ O (R' A
and training A instead, using the same optimizer. Letting vee (A) = A, we then have
vec (W) = vec (W()) + (R & L T) vec (A)
(W(O) + vec (LJART)
= vec
where in the first equation we used the fact that (M&N) = M & and in the second equation we used vec (MNO)
(O M) vec (N). Taking vec (-) in both sides completes the proof.
A.3. Proof of Cor. 3
We now state a more general version of Cor. 3, and then prove it.
Corollary 4 (Galore is one-sided LoRA (General)). Let W E mx be the parameter matrix of a linear layer with corresponding gradient matrix W e mxx, Consider training W with Optimizer using Galore, ie., where we linearly transform the gradient matrix with matrix P,
PW
m. S n (i.e., apply from the left)
W
WP > (i.e., apply from the right)
and then apply our optimizer on it, before transforming our update back to parameter space via PT, viz.
(A (t) W.
((+11) Optimizer(vee
(1)
W
:S
((+11)
W(O + P voc-!
(8)
m
S
W(O + vec-1
A
P
m
where P is arbitrary matrix of size Raxm (if m S n) Raxd (otherwise) and d min(m, n) controls the dimen- sionality of the transformation. Then the optimizer trajectory of this neiwork is equivalent to network trained with the reparameterization:
(
A
W
W
+
APT
m >
i.e., adding LoRA adapters where one side is frozen to P and only the other side, A. is learned.
Proof. Define
I,
②
P
S
P
& Im
m
>
15

---

On the Duality between Gradient Transformations and Adapters
where m, is the m identity matrix, and similarly for I, Then note that
vec
(PW)
m S 7
vec
W
vec (WP)
m n
S vec (W)
using vec (MNO) = (OT @ M) vec (N) as before. Similarly, we have that vec (W(+1) = vec (W + S A Hence, by Thm. I, we have that training a network with GaLore is equivalent to introducing parameter A and optimizing using the reparameterizattoon vec (W) vec (W(O)
vec (A). Observe that this choice of S is a special case of Prop. where L P and R I (if m S n) or L I and R P (if m > n). Thus, the reparameterization corresponds LoRA with one of the two adapter matrices frozen P.
B. Weight Decay
Cor. 3 establishes an equivalence between GaLore and LoRA when stateful optimizers are in play (i.e., those satisfying eq. (2)). While Adam (Kingma Ba, 2015) can be straightforwardly recast as a stateful optimizer, turns out that weight decay, as is traditionally implemented in, e.g., AdamW (Loshchilov Hutter, 2019), breaks this symmetry it does not fit our definition of stateful optimizer. From the perspective of our definition, the problem is that optimizer steps with AdamW are not solely a function of the observed gradients up until this point, but also the actual values of the parameters. This is important, since automatic differentiation libraries traditionally distinguish trainable and non-trainable parameters, with weight decay being applied to the former. Since the duality in Cor. 3 changes what the trainable parameters are, this means that the weight decay is applied differently in the gradient transformation view and in the linear adapter view.
For example, consider taking linear layer with weight + and training it with an optimizer that applies weight decay. When training this layer with linear gradient transformation S, after single optimizer step, our new weight is given by
(A (0) (()) QptimierWitouuuWigghhDDooyy//// (0) )(())
.56))
(10)
(1) (0) + + (0) - (0
(11)
where E IR+ is our weight decay penalty. Similarly, following Thm. 1, our linear layer's effective weight after a single optimizer step, in the adapter view, is given by
(1)
= (0) + S (A (0) + 1 AA 10)
(12)
offective
-0 + S (A (A +1(1) (0)) -AS A
(13)
(0) S A A (0)
(0)
(14)
where the final equality follows from Thm. 1. Note that general, we do not have that ( ( which means the optimizer trajectories may diverge.
An alternative interpretation for the above is that weight decay can be seen (roughly) as placing a Gaussian prior on the trainable parameters, but since the set of trainable parameters is different under each view, the equivalence does not immediately hold. We note that it is not outright clear if one implementation of weight decay is superior, so further research is required in this regard. In our experiments, for simplicity, we leave the application of weight decay untouched, i.c., we (implicitly) use the implementation of weight decay that naturally arises from the adapter or gradient transformation views.
Maintaining the equivalence. If one truly cares about preserving the optimizer trajectory even when training with weight decay, practically all one has to do is adjust the application of the weight decay so that it reflects the behavior of weight de- cay in the gradient transformation view or in the linear adapter view. Adjusting the linear adapter weight decay application to match the gradient transformation weight decay application is fairly simple: one just has to compute the effective weight at every timestep, as done above, and decay the frozen base weights directly. The converse is possible but slightly trickier, since the application of weight decay in the linear adapter view requires one to know what (e) is, which may require the
16

---

On the Duality between Gradient Transformations and Adapters
introduction of additional optimizer state. For example, one could store O and solve ST (4)
ellective = O - on
every optimizer state, or one could store and continually update Aetfectvv as part of the optimizer state.
C. Additional Results
SVD
Geaiknn
Radrmih
Resker Stmn-ottlggree
Twosited SVD
Two Caomine
M
hoodla
Conbecttoiiiss
86-
10
82
Figure 2: Full results for gradient reconstruction error (i.c.. - S se (left) and cosine similarity (i.., @os S Se)) of the various transformations training steps with the 200M model. The projections with lowest reconstruction emoor (measured either by L.2 error or cosine similarity with the unprojected stochastic gradient) do not give the best downstream performance (see Tab. 2).
D. Experimental Setup
The details the two architectures we consider are shown in Tab. 5. We also include a discussion on gradient accumulation.
Gradient accumulation. The experimental setup in the original GaLore paper did not perform gradient accumulation, which meant that the maximum sequence length had be short enough (e.g.. 256) such that a single batch could con- tain a large-enough number of sequences for accurate gradient estimation. Our experiments are instead conducted in the standard setting where we assume gradients are accumulated across multiple microbatches. In this case, the reparameteri- zation of GaLore as LoRA has the additional benefit of straightforwardly allowing for gradient accumulation in the lower- dimensional space. Concretely, the most straightforward implementation of GaLore!" will lead to gradient accumulation in the original parameter space, which would consume substantially more memory. In contrast, in the LoRA formulation gradients are accumulated after applying the gradient transformation, providing substantial memory savings without any additional code.'
200M
1.3B
Layers
12
24
Heads
16
16
Embed. dim.
1024
2048
Intermediate dim.
2816
5472
Head dim.
64
128
Query groups
16
16
Batch size
0.5M
1M
Warmup tokens
0.5B
IB
Total tokens
5B
10B
Table 5: Description of the two architectural settings consider for experiments: 200M setting which conduct most of analyses and ablations on, and 1.3B setting which use evaluate our techniques in realistic, large-scale setting.
15 Note that since the right hand side lies in range(S" )) this linear system will have solution. "In the distributed case, this might not incur any additional cost, since (() would already need stored. But this would require solving linear system on every iteration.
E..g.. the official implementation in https: /etp://gghtth/// com/ jiwe www.zahhaaaaaaaoo
15One can implement this optimization in the GaLore form, but this requires additional code. The issue is that most deep learning frameworks will compute the gradients of a parameterized function, and the user then separately passes these as input to an optimizer. If gradients only transformed in the optimizer, then the automatic differentiation module cannot figure out that only the smaller transformed gradient needed, and not the full gradient. We suspect that sufficiently gond compiler should in principle recover this optimization one ensures that entire training steps (viz., all the gradient accumulation steps and optimizer step) are compiled jointly.
17


Reviews:
[{"review_id": "uXFseCei51", "reviewer": "Reviewer_KsUQ", "summary": "This paper investigates the duality between linear gradient transformations and adapter-based reparameterizations in memory-efficient LLM training. In essence, it shows that applying linear transformations to gradients (as in GaLore) is equivalent to reparameterizing the model via adapters (like one-sided LoRA). This connection is interesting, as it unifies several existing methods and could potentially inspire new techniques. However, while the comprehensive summary and explicit unification of existing methods are very helpful, its conceptual insight is somewhat already quite clear from the literature, the contributions beyond what is already known appear incremental.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We identify three key criticisms pointed out in your review, and address them below\n\n### 1: Conceptual insights in the duality derivation are simple\n\nIndeed, while the essence of our derivation was observed previously in the literature, our derivation is more general; it considers a linear adapter applied to any parameter vector and it takes into account optimizer choice (previous works were only showing the equivalence for parameter matrices of linear layers and SGD, a non-stateful optimizer). This view is productive: it allows us to establish a connection between Kronecker-factored gradient transformations and recent two-sided variants of LoRA adapters which have been found to perform well for parameter-efficient finetuning, in particular MoRA, PMSS, and LoRA-XS. This is to the best of our knowledge novel in the literature. We think that this more general result is a worthwhile contribution to the literature.\n\n### 2: The empirical contributions borne out by the insights of the derivation are incremental.\n\nA key criticism seems to be that despite proposing other gradient transformations (e.g., Rademacher, randomized semi-orthogonal), the SVD transformation (i.e., GaLore) is still the highest performing transformation in Table 2 in terms of PPL in the 1B case. However, in Table 2 we see that the SVD transformation underforms the proposed approaches in the more memory-efficient NF4 quantization (i.e., QLoRA) setting: the Rademacher and Random-Orthogonal projections in this setting perform nontrivially better. Furthermore, in the 1B setting without quantization (or with INT8 quantization), the gap between the SVD and some of the other approaches is fairly modest (e.g., 0.1-0.3 PPL), and it is important to take into account that some of these are more memory efficient than the SVD transformation (see memory column in Table 2 for, e.g., Rademacher transformations).\n\nWe also kindly note that a large portion of our paper was dedicated to the application of the gradient transformation–adapter duality to distributed training, though it seems this wasn’t mentioned in the review. In particular, we show that choosing the transformations so they are worker-aware leads to better results than making them randomly different across workers or making them the same across workers.\n\nFinally, we want to emphasize that the goal of the paper wasn’t necessarily to achieve state-of-the-art memory-efficient pretraining results, but more to give a flavor for how our duality can unify existing approaches and lead to new techniques.\n\n\n### 3: Discrepancy between our results and those in the GaLore paper\n\nFirst, we’d like to stress that our setup is different from the GaLore paper in many respects, so it is hard to do a direct comparison. For example, we use a different dataset (SlimPajama instead of C4), we use gradient accumulation, our maximum sequence lengths are substantially higher (2K instead of 256), and our optimizer hyperparameters are not the default Adam hyperparameters, but instead are more consistent with values used in more recent, large LLM training runs (we use the ones for Llama-2 and OLMO).\n\nMore importantly, our decision to use a different setup was deliberate: Since we focus on pretraining, we want to be in the setting that most closely resembles recent LLM training runs. Therefore, we do not see the discrepancies between our results and those in the GaLore paper as a reflection that our setup is suboptimal; rather, we see our results, in part, as an attempt to apply GaLore to a different setting. We also note that such nontrivial differences in performances due to different training setups is common: for example, the Flora paper’s replication of GaLore (in Table 6 of the appendix) also shows substantially different results.\n\n## Answers to individual questions\n\n1. See point 3.\n\n2. This is a great question—we were also puzzled by this. Here, we actually think it is worthwhile taking a step back and looking at the results and models more broadly. Mathematically (from our duality derivation), it is clear that GaLore (and thus one-sided LoRA training) partitions the parameter space and trains only along a subspace of it. This reduces the effective number of parameters in our model, which would usually lead to a performance hit. Further, from our derivation, ReLoRA can be seen as GaLore where the projection is learnt by the optimizer, so unless this small number of new parameters introduce a lot more instability, we arguably shouldn’t expect the results to change much. Both of these observations manifest in our results. From this lens, we think our results are arguably what we’d a priori expect. While we cannot be sure why previous work doesn’t also show this same trend, but we suspect that this may have to do with the differences in the experimental setup (see point 3 above).\n\n3. See point 2.", "claims_and_evidence": "The paper claims that its generalized duality result not only recovers the known equivalence between GaLore and one-sided LoRA but also extends to more general settings. Although the authors provide a comprehensive summary of various optimizers and their relationships (as seen in Table 1), the result itself is rather straightforward. In the empirical evaluation, the best-performing method on the 1B model is still the gradient SVD adapter version of GaLore, and when compared to LoQT, the only difference is a slight change in quantization precision. These findings, while interesting, do not convincingly demonstrate fundamentally new methods that offer a significant improvement over existing ones.", "methods_and_evaluation_criteria": "There are a few minor limitations. See \"Claims And Evidence\" and \"Experimental Designs Or Analyses\"", "theoretical_claims": "The paper’s theoretical result—the equivalence between gradient transformations and adapter reparameterizations—is elegant but not surprising. Generalizing this duality appears to be a natural extension of previous work, and it does not suggest fundamentally new techniques. In particular, the derivation does not yield any novel optimizer that outperforms existing ones, but rather shows that the methods are different views of the same underlying mechanism.", "experimental_designs_or_analyses": "It seems that in the pretraining experiment, the authors mostly follows the setups (same number of tokens, etc) of the Galore paper (perhaps with different context length but this is not specified in the paper). However, according to the 1B experiment all baselines uniformly performs much better than the original setup. I would appreciate a comparison under the same setup to give readers a sense of how well these methods performs. Moreover, while ReLoRA is presented as a strong baseline in the paper, its performance in prior benchmarks has generally been weak. This discrepancy raises questions about whether the reported improvements are robust and representative.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "A notable strength of this work is its focus on an important direction—understanding the equivalence between optimizers and weight-space transformations in memory-efficient training. This is a valuable perspective that could potentially lead to new insights. On the downside, the technical contributions are incremental. The generalization of the duality between Galore and LoRA is quite straightforward, and more importantly, did not lead to new research insights (in terms of predicting new methods beyond what is known). The empirical results, particularly on the largest benchmark, do not convincingly demonstrate the intended claims.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- How do you justify the experimental setup, particularly for the 1B model, where all baselines perform uniformly better than reported in previous work? Would a side-by-side comparison under the exact same setup offer different insights?\n\n- Given that ReLoRA is shown as a strong baseline in your paper, how do you reconcile this with prior benchmarks where ReLoRA has performed poorly?\n\n- Can you comment on whether the duality result leads to any fundamentally new methods or improvements beyond providing a unifying view of existing techniques?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bQTCkTXLIS", "reviewer": "Reviewer_oyUd", "summary": "This paper explores a memory-efficient approach to neural network optimization by mapping gradients into a lower-dimensional space, reducing the memory overhead of both gradient accumulation and optimizer states. After performing updates in this reduced space, parameters are returned to the original space via the transpose of the linear transformation. The authors demonstrate that optimizing in this transformed gradient space is equivalent to reparameterizing the model through a linear adapter—which additively modifies the parameters—and training only the adapter. By employing a Kronecker-factorization of this linear map, the paper further clarifies the connection between GaLore and LoRA, showing they coincide (for one-sided LoRA) under this unified perspective.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\nOur main objective in this paper was to explore pretraining, and thus we selected a pretraining dataset that matches the distribution of pretraining data (SlimPajama, which is based on RedPajama, that tried to match the pretraining data used by Llama, one of the most successful open-source LLMs).\n\nIn terms of model size, because of compute constraints in an academic setting, we scaled as large as we could go (1B). As much as we’d love to scale further, we cannot do a pretraining run of a larger scale on our resources.\n\n> I wrote more comments above but how the convergence or the collapse of the method will be with this method?\n\nBesides the standard convergence results that apply to ML models of this scale, we note that, due to the nature of our equivalence, the theorems in the GaLore paper should still apply (subject to those theorems’ conditions, e.g., any restrictions on the choice of projection). Empirically, we find that all approaches we consider have similar convergence behavior.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The proposed method is interesting but more datasets are needed to be explored.", "theoretical_claims": "Yes, and they seemed correct to me.", "experimental_designs_or_analyses": "Yes, and in comparison with the related works different model sizes need to be investigated to show if the method works.", "relation_to_broader_scientific_literature": "The idea is interesting and could be practical in terms of scale.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "The paper has comprehensive related works. \nIt is also well written and well organized", "other_comments_or_suggestions": "-", "questions_for_authors": "I wrote more comments above but  how the convergence or the collapse of the method will be with this method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kEXqrIELG1", "reviewer": "Reviewer_kUHz", "summary": "This paper studied the connection between weight-efficient optimization and gradient-efficient optimization of transformers, and found that optimizing a model in a compressed gradient space is equivalent to optimizing just an additive low-rank adapter. Through theoretical analysis and empirical study, the authors showed the equivalence between those two lines of efficient optimizations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your response! We will break down your question into parts:\n\n> Could you further explain your findings from experiments in Sec 4.1?\n\nThe key goal of Sec 4.1 is to understand the interplay between the choice of gradient transformation S and overall performance. One important aspect of this interplay is that some transformations are expensive to update (e.g., the SVD transformation works well according to the GaLore paper, but requires running an SVD which is expensive), and others take more memory to store (e.g., the SVD transformation requires you to persist that matrix in memory, but if you are using random Gaussian matrices you just need to store a seed that you will use to regenerate this matrix—this means the whole matrix can be compressed into a handful of bytes).\n\nTable 1 tries to summarize these trade-offs conceptually. For each gradient transformation method, we describe what the parameters (of linear layers) in the model would look like using this approach, which of these parameters would be trained/frozen, and which of these need to be persisted in memory.\n\nTable 2 shows the empirical results (in terms of PPL on a held-out subset of SlimPajama) of training 200M and 1B using each of these methods; since the LoRA formulation is amenable to quantizing the frozen weight, we also include for each method the results using INT8 quantization and NF4 quantization. Beyond the test PPL, we also report the estimated memory usage of each approach.\n\nThe most important point of Table 2 is that, at a minor loss of PPL (i.e., +0.1-0.3 PPL), one can use other transformations that offer memory efficiency (e.g., Rademacher gradient transformations). The other takeaways from the table are that (i) while two-sided gradient transformations may be intuitively nice, they don’t seem to work as well, and (ii) INT8 quantization gives you memory savings without any meaningful PPL degradation, but NF4 starts to incur a non-trivial PPL penalty. \n\nBut we agree that section 4.1 could be explained better and will make sure to expand on the above points in the next iteration of the paper.\n\n\n> The empirical study is not clear enough to me, regarding supporting the previous claim about the equivalence between weight-efficient optimization and gradient-efficient optimization. Table 2 is more like a performance comparison instead of showing any connections between different methods\n> [...] how do these results support the equivalence between weight-efficient optimization and gradient-efficient optimization\n\nSince we prove that there is a mathematical equivalence between training with gradient transformations and training with a weight transformation (aka., a linear, one-sided adapter), the goal of our experiments is *not* to empirically verify this equivalence (since this would be redundant), but instead to leverage it by exploring, e.g., the interplay between choice of gradient transformation and empirical language modeling performance. That said, we will add a small scale experiment in the appendix where this equivalence is validated empirically, by showing that the loss curves of two toy models remain the same during training.\n\n> [...] and how is this equivalence used to build more efficient training methods?\n\nBeyond the new gradient transformations we propose in Table 2, our biggest contribution in terms of more efficient training methods is in combining gradient transformations with distributed training. In particular, in Table 3 we show that making different workers have different gradient transformations *and* making these transformations pairwise orthogonal (i.e., the distributed random method) is better than having them be orthogonal (but not pairwise orthogonal) or having them be the same across workers. In Table 4, we also study the effect of varying the number of workers, and find that the benefits of the distributed random method become more pronounced as the model is partitioned across more and more workers (each training a smaller slice of the model).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. The theoretical derivations are solid to me.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper is a revisit to previous memory-efficient training methods [1, 2, 3] for transformers. It aims to discover the connections and build equivalence between them, both theoretically and empirically. \n\n[1] Zhao, Jiawei, et al. \"GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection.\" International Conference on Machine Learning. PMLR, 2024.\n\n[2] Hu, Edward J., et al. \"LoRA: Low-Rank Adaptation of Large Language Models.\" International Conference on Learning Representations. 2022.\n\n[3] Lialin, Vladislav, et al. \"ReLoRA: High-Rank Training Through Low-Rank Updates.\" International Conference on Learning Representations. 2024.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "### Strengths:\n- The writing is clear and easy to follow. \n- The derivations make sense to me.\n\n### Weaknesses:\n- The empirical study is not clear enough to me, regarding supporting the previous claim about the equivalence between weight-efficient optimization and gradient-efficient optimization. Table 2 is more like a performance comparison instead of showing any connections between different methods. Besides, the analysis in this table is also ambiguous.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. Could you further explain your findings from experiments in Sec 4.1? Like how do these results support the equivalence between weight-efficient optimization and gradient-efficient optimization, and how is this equivalence used to build more efficient training methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k3MU2oP8R1", "reviewer": "Reviewer_u2rQ", "summary": "The authors derive an equivalence between LoRA / adapter training methods, which optimise a low rank addition to weight matrices to reduce optimiser memory usage, and methods that project gradients themselves down to low rank (such as Galore), also to reduce optimiser memory usage. Their result generalises more specific cases that have been shown in recent works. They then investigate the practical insights that might be gained from this observation, performing various LLM pretraining experiments with a range of adapter / gradient projection techniques, and propose a simple modification in the distributed training scenario motivated by the correspondence they prove, which improves performance.\n\n### Update after rebuttals\nI have maintained my score of \"Strong accept\" - please see rebuttal comment", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We address your questions below:\n\n### Key points\n\n> Is it possible to prove a similar equivalence where you use the moore-penrose pseudo-inverse to project back up?\n\nGreat question! It is possible that an equivalence exists if one is not using a LoRA adapter. For example, one could imagine defining a PyTorch primitive function whose forward pass behaves like a regular (one-sided) LoRA adapter $S^\\top A$, but whose backward pass computes the gradient of $A$ using the Moore-Penrose inverse instead of $S^\\top$. In principle this should be equivalent to training with GaLore (with a Moore-Penrose up-projection) in the optimization trajectory sense, though we are not quite sure what the mathematical interpretation of this would be.\n\n> The comparison between various adapter and gradient transformation methods in Table 2 was interesting, though I wasn't sure how this was directly relevant to the equivalence they proved.\n\nThe main aim of Table 2 is to compare (empirically) different choices of $S$, specifically to understand the trade-off between test PPL and estimated memory usage. The results suggest that Rademacher matrices offer better memory efficiency at a slight PPL penalty, and randomized semi-orthogonal matrices offer an even smaller penalty, though perhaps not a memory improvement (since it is not as clear how one could rematerialize them efficiently).\n\n> The proposed modification to training in the distributed setting definitely sounds clever [...] however, this seems to be specifically in the one-sided LoRA setting [...]\n\nYes, you are correct---this result is specific to the one-sided LoRA setting. We focused on this setting because this is what the theorems we prove focus on. That said, we also tried running a full two-sided distributed training setting (which would be equivalent to distributed ReLoRA/LTE; 20.97 PPL [200M] and 13.72 PPL [1.3B]) and found it to underperform the one-sided LoRA with worker-aware initialization. We will include these results in the next version of the paper.\n\n> Do you have an explanation for why the SVD curve in Figure 1 plateaus over time?\n\nGreat question! We suspect that towards the end of training, since the learning rate has annealed, the model is actually shifting very little in parameter space. This means that the gradient is also shifting fairly little. This means that our SVD-based gradient estimator, which is optimal (in the Frobenius error sense) when applied to the gradient the SVD was computed on, remains fairly optimal.\n\n> I was slightly confused by what the authors meant by the \"Transformer++\" architecture\n\nThanks for pointing this out. This part is unclear, and we will fix this in the next version of the paper. By Transformer++ all we mean is a Transformer with the changes made in the Llama architecture, e.g., RMSNorm instead of LayerNorm, SwiGLU activations, rotary embeddings (some papers use this term).\n\n### Other comments/suggestions\n\nThank you for your comments here—we will address them all in the next version of the paper. We also include a brief explanation/clarification where appropriate:\n\n> Please define the notation vec^-1  first used on line 195.\n\nIf $\\text{vec}(\\cdot)$ sends a matrix to its vectorized form (i.e., we stacked up the columns), then $\\text{vec}^{-1}(\\cdot)$ takes a big stacked vector and unstacks it into a matrix of the right size.\n\n> I didn't fully understand the points about why different schemes need different persisted matrices in Table 1\n\nThis point is related to your earlier question about Table 2. Table 1 is trying to be explicit about the cases where it should be possible to just persist the seed that generates a matrix, and regenerate it on-the-fly as needed. This means additional memory is saved since we can get away with just storing the seed for that matrix (~4 bytes), as opposed to storing the matrix itself.\n\n> I also found the discussion around Table 4 slightly confusing. It seems to show that in the baseline, having more workers is beneficial, but in the lora experiments, having more workers is detrimental?\n\nGreat question---In Table 4 we simultaneously vary the number of workers and the rank of the gradient transformation/LoRA adapter, except in the case of the baseline, where we only vary the number of workers. Since we allow all workers to train for the same number of tokens, this means that for the baseline, increasing the number of workers increases the effective number of total tokens we train on *without* any penalty (i.e., reducing the rank). So the baseline will always benefit from more workers without any real negative effects, whereas the other approaches will suffer from the rank being decreased as the number of workers is increased. We mention this briefly in the caption, but we will make sure to make this clearer.\n\n> It would be useful to clarify what the metrics are in all table captions (I think they are all test perplexity?).\n\nThat’s right, they are all test perplexity.", "claims_and_evidence": "- The primary claim of the paper is that they prove that adapter methods are equivalent to gradient low-rank projection methods, which is true. Their theorem covers a general case that applies to most of the recent works such as lora and galore.\n- Their secondary claim of utilising this theory to develop a better algorithm in the distributed training case is also backed up by good experiments, with relatively large models (1B).", "methods_and_evaluation_criteria": "- The proposed theorem and corresponding proof are relevant to the well-known lora and galore algorithms. In particular, they assume that the gradients are projected down to a low-rank subspace using a matrix $\\mathbf S$, and the optimiser steps are then projected back up to the full space using its transpose $\\mathbf S^T$. This is the appropriate setup when $\\mathbf S$ has orthonormal rows / columns, or in the case where $\\mathbb E[\\mathbf S \\mathbf S^T]=\\mathbf I$. I would also be interested to know whether a similar equivalence can be proved in the case where you project back up using the Moore-Penrose pseudo-inverse, which is equivalent to $\\mathbf S^T$ when $\\mathbf S$ is orthogonal but not in general. However, this case might not be of practical relevance anyway, as the Moore-Penrose inverse would be an expensive step to compute as part of an optimisation loop.\n- The comparison between various adapter and gradient transformation methods in Table 2 was interesting, though I wasn't sure how this was directly relevant to the equivalence they proved.\n- The proposed modification to training in the distributed setting definitely sounds clever, since they use different projections for the different workers to ensure that the overall parameter updates are roughly full rank. However, this seems to be specifically in the one-sided LoRA setting, and I'm not sure whether the problems they fix are relevant in the more usual two-sided LoRA setting.", "theoretical_claims": "I checked all the proofs in Appendix A. Theorem 1's proof is a neat proof by induction that takes less than a page. The other two proofs, for proposition 2 and corollary 3, are straightforward applications of Theorem 1 using Kronecker factored matrices, but are still useful to have written in full.", "experimental_designs_or_analyses": "The experimental setup is sound, pretraining LLama models on large text datasets with LoRA adapters from scratch. I was slightly confused by what the authors meant by the \"Transformer++\" architecture, as I could not find this in the cited LLama paper.", "relation_to_broader_scientific_literature": "The discussed schemes of lora, galore, relora etc. are all very relevant and discussed methods in the current PEFT literature. As mentioned in this paper, previous papers have proved equivalences between some of these schemes, such as one-sided lora and galore, and this paper serves to generalise these proofs to a much wider setting that includes many of these schemes.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "### Other strengths\n- I think the proposed equivalence is very useful, as many PEFT schemes like this have been proposed, and it turns out that in many cases they are equivalent.\n### Other weaknesses\n- The experiments in the paper, other than the distributed training setting, do not seem entirely relevant to the main point of the paper. However, since the main point of the paper is the proved equivalence, the experiments are of less importance.", "other_comments_or_suggestions": "- Please define the notation $\\text{vec}^{-1}$ first used on line 195.\n- I didn't fully understand the points about why different schemes need different persisted matrices in Table 1. Slightly more explanation would be helpful.\n- I also found the discussion around Table 4 slightly confusing. It seems to show that in the baseline, having more workers is beneficial, but in the lora experiments, having more workers is detrimental?\n- It would be useful to clarify what the metrics are in all table captions (I think they are all test perplexity?).\n- In proof of theorem 1, I found equation 5 hard to follow, but it was just the chain rule. It might be worth slightly rewriting this small segment to make it clearer.\n- Typo: in line 759, \"imilarly\"", "questions_for_authors": "- What is the Transformer++ architecture? Perhaps I have missed something, but I cannot find it in Touvron et al 2023.\n- Is it possible to prove a similar equivalence where you use the moore-penrose pseudo-inverse to project back up? (I.e. your low rank projection isn't necessarily orthogonal).\n- (not very important but I was curious): Do you have an explanation for why the SVD curve in Figure 1 plateaus over time?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper identifies and explores a duality between linear gradient transformations like GaLore and certain adapter-based reparameterizations like one-sided LoRA. The reviewers found this connection quite interesting and useful as it unifies several existing techniques and could open new ways of thinking about memory-efficient LLM training.

While the reviewers were generally positive about the main contributions, some reviewers found the experimentation to be unclear or unconvincing, and others found the connection itself to be elegant but unsurprising. Given that the author response clarified many aspects of the experiments, and that in any case the experimentation is not the main contribution of the paper, I find these concerns have been adequately addressed. In addition, even if the connection is not surprising, establishing it concretely as this paper does is an important contribution and I believe the ICML community will find the results quite interesting; therefore, I recommend acceptance.
Decision: Accept (poster)