LoQT: Low-Rank Adapters for Quantized Pretraining
Authors: Sebastian Bugge Loeschcke
Mads Toftrup
Michael Kastoryano
Serge Belongie
Vésteinn Snæbjarnarson
Venue: NeurIPS 2024 poster (2024)
Keywords: Quantization
Low-Rank Adaptation
Memory Efficient Training
Large Language Models
Primary area: optimization_for_deep_networks
TL;DR: LoQT enables efficient quantized pretraining of LLMs with results close to full-rank non-quantized models. It enables pretraining of a 13B LLM on a 24GB GPU without model parallel, checkpointing, or offloading strategies during training.

Abstract: Despite advances using low-rank adapters and quantization, pretraining of large models on consumer hardware has not been possible without model sharding, offloading during training, or per-layer gradient updates. To address these limitations, we propose Low-Rank Adapters for Quantized Training (LoQT), a method for efficiently training quantized models. LoQT uses gradient-based tensor factorization to initialize low-rank trainable weight matrices that are periodically merged into quantized full-rank weight matrices. Our approach is suitable for both pretraining and fine-tuning models. We demonstrate this for language modeling and downstream task adaptation, finding that LoQT enables efficient training of models up to 7B parameters on a 24GB GPU. We also demonstrate the feasibility of training a 13B model using per-layer gradient updates on the same hardware.

LoQT: Low-Rank Adapters for Quantized Pretraining
Sebastian Loeschcke* University of Copenhagen sbl @di Bod..ku... dk
Mads Toftrup' Aarhus University
toftrupècs au.dk
Michael J. Kastoryano University of Copenhagen mika@di ku. dk
Serge Belongie University of Copenhagen b.belongie@di ku. dk
Vésteinn Snæbjarnarson University Copenhagen vesnêdi ku. ku.dk
Abstract
Despite advances using low-rank adapters and quantization, pretraining of large models consumer hardware has not been possible without model sharding, of- floading during training, or per-layer gradient updates. To address these limitations, we propose Low-Rank Adapters for Quantized Training (LoQT), method for effi- ciently training quantized models. LoQT uses gradient-based tensor factorization initialize low-rank trainable weight matrices that periodically merged into quantized full-rank weight matrices. Our approach is suitable for both pretraining and fine-tuning models. We demonstrate this for language modeling and down- stream task adaptation, finding that LoQT enables efficient training of models up 7B parameters on 24GB GPU. We also demonstrate the feasibility of training 13B model using per-layer gradient updates on the same hardware.
https://gittub..comsbbbbuu
1 Introduction
Llama 13B
Training large neural networks requires substan- tial hardware and energy resources. Reducing these requirements is important for both cost ef- ficiency and environmental reasons, while also lowering the entry barrier for researchers and practitioners in general. In this work, we target the memory component- a key part of the hard- ware requirements. Memory use during training comes primarily from storing model, the optimizer states, and activations. To target the memory footprint of the weights, vari- applications of quantization 2 3 have been used. For targeting the optimizer states, variations on low-rank adaptation (LoRA) 6
LoQT ASbit LW
Memory Type
00M
LoQT Akhit
Optimizer
LoQT
Model
Forwaed
GaLore AKhit L.
Gradient
GaLore
Unknown
GaLore
Adam
weights of the
10
30
50
60
70
80
Memory
Usage
Figure 1: Memory usage of Llama 13B, rank 1024. LW: per-layer gradient updates. A8bit: Adam 8bit.
7 have been suggested to decrease the number of trainable parameters for fine-tuning, in combination with the use low precision representations. Low-rank approaches for projecting gradients to lower rank have also been suggested [8]. In this work, we combine these approaches to address the model size and optimizer states, resulting in highly memory-efficiae configuration that is also suitable for pretraining.
3
Equall contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1. Initialization
2. Training
3. Merging
Pg
Backward
W
W + P
B
VW
P
P
B
W
W
B
I Totialzzati Tio Hon
B
SVD(VW) UEV
Forward
A
Optimizo
P= U1:: r]
Pg = (P) W = ( B=P (Wa- W)
T. = #
W
VW
Figure 2: Overview of LoQT. (1) Low-rank factors P and B are periodically initialized from the
gradient of the dequantized model weights VW, (2) then only B is trained while P. and Wg are kept quantized and frozen, over an exponentially increasing interval until T,, (3) the low-rank factors are merged back into the quantized model. The process is repeated until training halts.
In typical training configurations, the optimizer states often take up more space than the model itself, methods such as Adam [9] keep track of two parameters for each parameter of the model. While LoRA is memory efficient for parameter-efficien fine-tuning of pretrained models, it has not been shown to work as a pretraining method by itself 77. GaLore (8) significantly reduces the memory needed for the optimizer parameters by storing the optimizer state in low-rank projection, which is then projected up when applied the model weights. Combining this method with quantization would further shrink the footprint of the model but this is not straightforward. Updating the weights of a highly quantized model directly in low-precision space has not been shown to work. This is mainly due to the quantized states.
gradient updates having too small of an impact on the lower-precision
To address these shortcomings, we propose Low-Rank Adapters for Quantized Training (LoQT). LoQT initializes two low-rank factors, P and B, for each weight matrix W. P is initialized using projection of W's gradients into a low-rank subspace, and B is initialized to minimize the quantization error. In our method, B is the only matrix being actively optimized. Only optimizing B means that the size of the gradients and optimizer state shrinks significantly compared to full training or L.RA. The product PB is periodically merged into the full rank matrix W with exponentially increasing gaps to account for smaller updates as the model converges, ensuring we accumulate large enough updates. As W and P do not receive gradient updates, they can be kept quantized, optimizing memory usage even further. It is the large accumulated updates that make it possible to update a quantized model-an as the addition of smaller changes would not register in the quantized state. A high-level overview of our approach is given in Fig. 2
We show that LoQT works well both with and without quantization, enabling only lower memory footprint in the optimizer state but also over the model parameters. Our results show that we get competitive performance to prior methods using significantly less memory, in particular when quantizing the model weights in an application such as training a large language model (LLM).. We also demonstrate comparable performance in language adaption, which we demonstrate on curated Icelandic text dataset [10]. Finally, we show that LoQT also works for fine-tuning pretrained models on down-stream tasks, by training and evaluating on the GLUE M benchmark for natural language understanding and the GSM8K [12] dataset for mathematical reasoning. We ablate several properties of the suggested approach, demonstrating the importance of each component of LoQT. For instance, we find that an exponentially increasing projection gap is particularly crucial for the training of quantized models. An overview of memory savings is given in Fig. I We find that LoQT enables efficient training of 7B models on consumer-grade hardware with 24GB of memory, and makes it feasible to train models with up to billion parameters without model parallelization, by making use of per-layer gradient updates [1].
2

---

2 Efficient Pretraining With LoQT
We now briefly introduce how LoQT works by initializing and training low-rank adapters. The adapters are initialized by taking the singular value decomposition (SVD) of a given layer's s gradients. We use W indicate the full weight matrix of given layer and P for the left factor constructed from the SVD decomposition of the gradient matrix, VW = UEV", such that P consists of the first columns U correspoonnd
the singular vectors with the largest singular values of W. where is given target rank. The update rule for a timestep T; is then given by WT; = WT, + PB. For the steps between T. and Ti+1 only the weights B are updated, while and Wr, remain constant. We describe this in more detail below, followed by discussion periodic updating of the factor P. enabling of quantized pretraining, error compensation, and exponential update intervals. Pseudo-code for LoQT is shown in Fig.
2.1 Background: Gal.ore
Zhao al. 8 find that gradients exhibit low-rank structure during training. They exploit this insight by projecting the gradient to low-rank subspace and applying the Adam optimizer before projecting back to the original dimensions. By doing this, the memory-intensive optimizer states required by Adam are shrunk significantly for low enough ranks.
Definition 2.1 (Gradient Low-rank Projection, def. 3.4 in [8). Gradient low-rank projection (GaLore) denotes the following gradient update rules, where v is the learning rate, P the Adam optimizer, W e Rm is the weight matrix being trained, and T represents the total number of training iterations until the recomputation of the projection matrix:
W1 Wo + +1 Σ Go where PIPPPP G.QQT.
(1)
where is given target rank and P. Rmxr and Q: E Raxr are the top-r singular vectors from the SVD decomposition of the gradient matrix at each iteration t. In practice, this can be approximated by only applying a meddided projection, in
W Wo + v Σ (P G.) or W1 Wo + n Σ (G.Q..).
Additionally, Zhao et al. { empirically show that it is sufficient to keep the projection matrix fixed and only update it once every T iteration.
2.2 Low-rank Gradients as Adapters
We now describe how we initialize the parameters we optimize with LoQT. We start with the GaLore formulation from above and adopt the memory-performant projection (eq. 2 we compute pt G if m < and GQ otherwise. Our goal is to separate trainable weights and static weights, which we achieve by rewriting GaLore in terms of low-rank adaptors. We assume that < if m > the same reasoning holds for Q Using the fact that P. is fixed on the interval (0, we get
trade-off of using only a one-sided
T-1
Wy = Wo +
Σ PP((PTG.)
(3)
1=
= Wo +n P
Σ G()
(4)
BERrxa
It is clear from 4 that we can keep track of low-rank updates using rank-r adaptors. We note that in the interval [0. T only B updated, creating the desired separation. If implemented directly, we would need to compute the gradient with respect W and then project it down using P. Gr. We find that this step is unnecessary; it is sufficient to train B using standard gradient descent.
3

---

Equivalence of Gradient Updates We point out that optimizing the low-rank matrix B via gradient descent is equivalent the projected gradient updates on W, described in Definition Let GW aw
8C denote the loss gradients with respect W and B, respectively. Consider the forward pass y = xW + +rPB, where W is the weight matrix, P is the projection matrix, and B the low-rank update matrix. By the chain rule:
OC and G 7B
G (IP) Oc
(5)
Oy
0L
PT x
(6)
dy
= =
(7)
This derivation establishes that computing gradients with respect to equivalent to projecting the gradients with respect to W onto the low-rank subspace defined by P. Therefore, GaLore's 's low-rank gradient updates are identical to those obtained through backpropagation in LoRA.
2.3 Pretraining with LoRA
Previous work [ has shown that training low-rank weight matrices works well for fine-tuning pretrained weights. However, it has been shown that starting with randomly initialized weights, training low-rank factors, and periodically merging them into a frozen weight matrix W. does not work when starting with randomly initialized matrix 77.. We now address this enable full training using low-rank weight matrices.
Inspired by prior work 8 we periodically update a given layer WT+1 = WT + Pr at fixed steps T E T. This approach allows W to evolve as a sum of low-rank matrices aligning with GaLore's strategy of updating the gradient subspace during training:
(8) where 1 = 17] T, and AWr; = Pr. By. represents the product of the learned matrix B over the -i=1
W₂ = Wo + AWT, + AWT, +... AWT.,
interval T; - T modulated by the gradient projection matrix PI,- After each periodic update at iterations 7; E T, we reinitialize the low-rank factors Pf and By. As in [8], we compute the gradient of WT over single batch, focusing only on VW, without storing optimizer states for it, reducing the memory compared to full-rank training.
For each updated W; and reinitialized P: and B,, new gradient subspace is established for exploring the next T++1 - Ti steps. Our method treats W, as the full-rank repository of accumulated updates. Although it is periodically updated, W, is not part of the optimizer state computations, and the gradients during the single forward pass are offloaded to CPU/RAM. Since the SVD calculations are done layerwise, only the current layer needs to be on GPU, or the SVD can be calculated on CPU. Pt defines the general gradient subspace and trajectory for the upcoming Ti+1 - T; steps, and B, is adjusted to navigate within the direction set by Pt. As only B, is trained, the number of parameters requiring optimizer states is drastically reduced.
2.4 Quantized Training
Recall the update rule of our model, Wr. = WT... + PB, given that B is the only matrix accumu- lating gradients and undergoing changes, the other matrices W and can be kept quantized. This approach allows storing the weights in NF4 precision [3] (see $5.1 for detailed account) without requiring high-pre gigh-rrcision gradients and weights to update W and P. To the best of our knowledge, we are the first to enable efficient 4-bit quantized pretraining using gradient descent without storing the weights 16-bit precision.
We quantize the weights (W) = W and UNTA( (P) = Pa as described in $5.1 During the periodic updates at interval time steps x Tm) Py and Wj, y are dequantized using the inverse function, P A (PNss) and WBF16 = 85F4 ( (Wrr).. After this, Wr, Wr,,, + Pr, Br.., is computed and quantized. The quantization and dequantization processes are applied layer by non-quantized state reduce memory usage. Moreover, the quantization state itself is re-quantized for further efficiency following R We implement LoQT using weight-only quantization, this means that the quantized weights are loaded into memory and then dequantized before computing the matrix multiplications.
layer, ensuring that not all layers are simultaneously

---

Algorithm 1 LoQT: Low Rank Adapters for Quantized Training
Algorithm Initialization Procedure
1: Initialize G
Require: W: Weight, T: Update steps, fE LR, r: rank, (-).:: N-bit quantization function.
2: U.S.V? +- SVD(G")
< U]::r] (First singular vectors)
4: Pg < qN(P)
1: Gw + VWL(W)
B
2: Wo, Po.
Wo,Po, B Initialize ( Gw)
6: W < W
3: for each /. in training steps do
for each e compensation steps C do
4: if E T then
Qe - gN(WW
5:
W + We *s Pq- B,
9:
B p+ P+W Qc)
6:
GW + -WW
10: W W -PB
7: Wq. Pe Bg Initialize G
11: return Qe, Pg
8:
else
9:
Bu+1 B₁ p(,
10:
return
0
Figure 3: Pseudo-code for LoQT.
2.5
Compensating for Quantization Errors
As the quantization process inevitably results in rounding errors there is a discrepancy between the non- quantized and quantized versions of W. We wish reduce this effect as much as possible. While com- pensating for quantization errors has been done before [4], we derive a tailored solution for LoQT.
During the merging update phase, we first dequantize to obtain WT-1 and Pr-1- and then compute the update WT Wr-1 + PT-1BB-1 This is immediately followed by re-quantizing to get QT ONEA (WT). Our goal is to minimize the quantization error (l PT Wrll. Recall that Pr is found based on the gradient and not changed to compensate for the quantization error.
Instead, we solve for Br in the merging step, initializing BT BT def P (Qr WT). where P is the Moore-PPnnosee pseudo-inverse. This approach avoids initializing BT as zeros, as is commonly done [5), and instead uses it for minimizing the quantization error ||Q, Wrll We then iteratively refine By over a maximum of five steps, by recomputing Qr = INH4 (W - Br). improving the alignment between the full- pelliprecision W and its quantized state.
As training advances and the learning rate decays, the magnitude the update By-1 decreases. This leads to negligible differences lg (Q, P, Qrl. which results in the loss plateauing early, as depicted in Fig. 4a. To address this, we implement an exponentially increasing scheduler for updating W. Drawing from the observation that the gradient rank decays exponentially (Lemma 3.1 in (8), we start with an update interval and progressively increase the update intervals by a factor of uh. The sequence of updates is then given by (T)) (r + ) Each T; marks training step t when W is updated. This scheduling ensures more frequent updates earlier training and more well-spaced adjustments later, allowing for accumulation of sufficiently large gradients before each progressive update.
3 Experiments
We evaluate LoQT on language model pretraining by training I.LaMA-based [15] language models on the C4 dataset [16). a collection of text in English that was extracted from the Common Crawl web-scrapes [16). We train models of sizes of 60M, 130M, 350M, and IB parameters, adhering single-epoch training cycles determined by the Chinchilla Scaling Laws [17. While LoQT is capable of training models up to 13 billion parameters on consumer GPUs, compute limits prevent us from training convergence for sizes above 1B. We also benchmark LoQT on the GLUE test- for natural language understanding [18), the GSM8K [12] dataset for arithmetic reasoning and an Icelandic text dataset [10 to evaluate language adaptation via continued continued-pretraaningg Runs were conducted on up to 4x 40GB NVIDIA A A100s 2x 80GB NVIDIA H100s, or single 24GB NVIDIA RTX 3090. The longest run was the training of the IB models, taking approximately four days on the four A100s. The RTX 3090 was used for throughput and to empirically verify memory claims.
5

---

Table 1: Comparison of low-rank pre-training methods for LLaMA2-style language models on the C4 dataset. The table shows validation perplexity. memory estimates, and quantization states for LoQT. The rank ratio r/ 7/dmodel is relative the largest weight matrix dimension. Perplexity values are averaged over three seeds showing mean and standard error. (*) Denotes results from GaLore g Only one seed was used for the IB experiment due to compute constraints.
60M
130M
350M
1B
Full
33.32 +: 0.22 (0.36G)
24.51 + 0.03 (0.76G)
18.87 + 0.18 (2.06G)
15.56 (7.80G)
LoQT (Ours)
33,98 + 0.15 (0.23G)
24.57 0.01 (0.49G)
19.12
0.01 (0.98G)
15.55 (3,16G)
LoQT-nq (No quant.)
33.55 = 0.03 (0.28G)
24.37 0.02 (0.63G)
18.85
0.01 (1.47G)
15.20 (5.11G)
Gal.ore
34.15 + 0.24 (0.24G)
24.81
0.04 (0.52G)
19.47
0.01 (1.22G)
15.64* (4.38G)
LoRA
34.99* (0.36G)
33.92 (0.80G)
25.58* (1.76G)
19.21 (6.17G)
ReLoRA
37.04 (0.36G)
29.37 (0.80G)
29.08 (1.76G)
18.33* (6.17G)
r/dmadee
128 / 256
256 768
256/ 1024
512/ 2048
Training Tokens
1.IB
2.2B
6.4B
13.18
We keep hyperparameters consistent across model sizes, with experiments conducted in BF16 format for memory efficiency. All models are trained with a maximum sequence length of 256, a total token batch size of 131K tokens, and a learning rate warmup for the first 10% of the training steps, followed by cosine annealing to 10% of the initial learning rate. Full experimental details, including the specific hyperparameters for each task, are provided in Appendix B
Baselines For pretraining, we compare LoQT against LoRA [5), ReLoRA 77 GaLore [ and a
non-quantized version of LoQT, LoQT-nq. In our experiments, we apply these parameter-efficient training methods to the attention projection matrices and fully connected layers while maintaining full- rank embeddings. For the fine-tuning experiments, LoRA, ApiQ [4], and LoQT-nq, or a subset thereof. All models that make use of update frequencies trained using the same intervals, these GaLore, ReLoRA, LoQT-nq. and LoQT. We start with an update interval of T = 100 and then exponentially increase the update frequency. This means that we do more frequent updates early and fewer as the model stabilizes (see 4 for more details). A scaling parameter a 0.5 is used for LoQT and GaLore across all models, except for the 1B model where it is decreased 0.25. The same rank is used for all low-rank methods. All models are trained using the Adam optimizer, except GaLore which uses their Gdam optimizer for gradient projection. More details on hyperparameters are provided in the Appendix B
compare LoQT against GaLore, LoftQ [14,
3.1 Pretraining of Generative Language Models
Results and details of pretraining causal language models of sizes 60M, 130M 350M, and 1B parameters are shown in Tab. [ Model sizes are calculated based the full models without any low-rank methods. We see that LoQT and LoQT-nq both perform very close to full rank pretraining and GaLore while using significantly less memory by keeping most of the model weights in quantized state. For the 60M model, full training is only slightly better than LoQT, while we see results improve or stay within the standard error for the other sizes. We also notice slight drop in performance from quantizing the original weight matrix, comparing LoQT and LoQT-nq. The key difference between the approaches is the theoretical memory estimates, e.g. where LoQT uses 59% less memory for the 1B model in full precision and 28% less memory than with Gal.ore.
Table 2: Results for LoQT, LoQT-nq. and GaLore using DeBERTa VoBERTaV3-base models the GLUE development set. We report mean and standard error over three seeds. The best mean results on each dataset are shown in bold.
Rank
Method
MNLI
QNLI
KTE
SST Acc
MRPC fl
CoLA Matt
QQP
STSB PCorr
Average
Ace
Acc
Acc
LoQT-ng LoQT
90.0.10.10
94.2.10.06
0.75
95.910.06
94.1100.25
72.510.41
90.0.1..66
91.510.07
89.1
90.0:2009
94.3:0.04
84.1=0.11
95.5=0.10
94.4:0020
70.5:0.35
89.2:10.22
91.5=10.13
88.7
LoRA
89.910.03
94.0:10.99
83.6 0.12
95.7:0.15
93.5::0.26
69.31.0.77
89.8.10.11
90.7.10.22
88.3
LollQ
90.4110.09
93.2:00.2
8368
95.6 20.07
93.2:10.14
71.110.28
89.6.10.12
91.0.10.09
83.4
32
GaLore
90,3=20.07
94.0=2004
83.7=0.79
95.6=0007
93.4:00.8
70.7+0.24
89.8:20.05
90.6:20.01
88.5

---

3.2 Memory-Efficient Finetuning
We fine-tune the pretrained DeBERTa- V3-base
[9] model on the natural language understanding GLUE [II] tasks using LoQT and compare its performance with full fine-tuning baselines, LoRA, 7 for details on hyperparameters. Results are given Tab. P
LoftQ, and Gal.ore. See Appendix
We find that both LoQT-nq and LoQT perform well. And somewhat surprisingly, they sometimes surpass GaLore, LoftQ, and LoRA. This may indicate that initializing the LoRA factors with information about the gradient of W is beneficial starting point compared to standard initialization methods. Further experiments are needed to confirm and investigate these findings which we leave to future work.
Arithmetic Reasoning on GSM8K We fine-tune quantized Llama-2 models (7B and 13B) on the GSM8K dataset [12 for arithmetic reasoning. As shown in Tab. LoQT achieves average test set accuracies of 42.6% and 52.9% with the 7B and 13B models, respectively, performing comparably to other quantized fine-tuning approaches. Detailed hyper-parameters are provided in Appendix Tab. g
Table 3: GSM8K LLaMA-2 7B and 13B test accuracy with std. error. Best mean is in bold.
Table 4:
Llama-7h 7B fine-tuning on Icelandic. We report test set perplexity.
Method
Bit
LLaMA-2-7B
LLaMA-2-13B
Method
Perplexity
LoRA
16
41.7 = 0.3
51.3 + 0.86
No training
4.90
QLoRA
41.9 0.2
51.6 + 0.29
Full
3.79
LoftQ
4
41.9 0,9
51.3 + 0.96
GaLore
3.96
ApiQ
42.1 0.5
52.4 0.46
LoQT-nq
3.61
LoQT
42.6 = 0.4
52.9= 0.12
LoQT
3.63
We also evaluate LoQT on language adaptation large language model. We continue pretraining of the Llama-2-7B model using curated subset of a public Icelandic text dataset extracted from [10] containing 770k documents. We compare LoQT with NF4 quantization, LoQT without quantization (LoQT-nq). regular training, and GaLore, using consistent hyper-parameters across all methods, results are shown in Tab, LoQT achieves test set perplexity close to that of using full training GaLore while reducing perplexity from 4.90 (non-trained model) to 3.63. Additional details are provided in Appendix C.I
Continued Pretraining of Llama 7B
3.3 Memory and Throughput
Memory Usage An overview of memory usage for GaLore, LoRA and LoQT is given in Tab. We see that LoQT has the same number of parameters as LoRA for a given rank while using less memory for the optimizer states and gradients than in both LoRA and GaLore.
We compare LoQT to GaLore, the approach that gets closest in memory performance, for model of size 13B in Fig. [ and for other model-sizes in Fig. 6 We compare three different cases, applying the methods on their own, combining them with an 8-bit Adam optimizer [20]. and using per-layer weight updates with offloading (while still using 8-bit Adam). We see that LoQT significantly reduces the number of trainable parameters, and optimizer states, compared to GaLore.
Per-layer weight updates are essential for GaLore; without it, additional 12 GB of VRAM is needed for the gradients a 7B model, making fall-parameter fine-tuming impossible on a 24GB GPU. Additionally, the per-layer gradient updates do not work with gradient accumulation. Using LoQT results in lower memory use than GaLore, even with per-layer gradient updates. When not using per- layer gradient updates, the difference becomes more pronounced as seen for the 7B model in Fig. 6
LoQT enables training of 7B models without per-layer computations on a 24GB GPU, allowing for gradient accumulation and higher effective batch sizes. Our memory advantage allows for batch size of 1280 tokens compared to GaLore's 256 for the 7B model on the 24GB RTX 3090. Using

---

Table 5: Comparison memory usage GaLore, LoRA, and LoQT. e Rm* (m S vt), rank
Gal.ore
LoRA
LoQT (Ours)
Weights
mn
mn + mr + nr
ma + mr + n
Optimizer States
mr + 2nr
2mr + 2nr
2nr
Gradients
mn
mr +
Pretraining
Yes
No
Yes
Fine- Tuning
Yes
Yes
Yes
Quantizeable
No
Yes
Yes
per-layer gradient updates, LoQT can train 13B model on single GPU. We refer Fig. in the Appendix for a comparison of how Adam, GaLore, and LoQT scale with increasing context length.
Throughput We evaluate the throughput with sample batch size of 16, and a total batch size of 512 using gradient accumulation, which the largest power of two that fits on the GPU. We update the projection matrix P for every 200 iterations. We evaluate the throughput using IB parameter model and rank 5 522 without per-layer gradient updates. We find that LoQT processes 16% fewer tokens per second than training with only AdamW, at 3996 tokens/s compared to 4782 tokens/s on the RTX 3090.
4 Ablations
Method Ablations
Update Interval Ablations
34
38
Wy
W., P. EC
200
1000
Py
P. EC, ET
36
400
100-1.20
Wa, Pa
No
500
34
Perplexity
Perplexity
28
28
26
26
24
4000
8000
12000
16000
20000
4000
8000
12000
16000
20000
Step
Step
(a) EC: Error compensation, EF: Exp. increasing update interval.
(b) Ablation of update intervals: comparing fixed inter- vals an exponentially increasing schedule.
Figure 4: Ablation results for update intervals, error-compensation, quantization using model size 130m, and rank 256. Wg: quantized W: Py: quantized P; No Q: no quantization. The dynamic update interval 100 + 1.2 grows exponentially for each step N.
Quantization Error Compensation and Initialization 130 million parameter models to assess the impact of error quantization compensation. Fig. 4a shows that quantizing W, or both W and P. without error compensation, or exponential interval updates leads to early stagnation of the loss. We also note that quantizing P has a much smaller effect on the loss compared to quantizing W. Error compensation significantly improves the model's performance, resulting in approximately 3.5 points better perplexity. Adding exponentially increasing update intervals improves perplexity further by an additional 1.5 points, achieving performance close to that of models without quantization.
analyze the validation loss curves of
Without the quantization emoor compensation, detailed in 2.5 LoQT's performance stagnates earlier and diverges more from the other models. This demonstrates the effectiveness of our compensation approach in mitigating the quantization errors introduced during the update of W with PB and subsequent quantization steps.

---

Projection Update Intervals Our scheduling approach ensures more frequent updates earlier on in training when the weight adjustments are larger. As training progresses, the update intervals get larger, allowing for accumulating more updates compensate for smaller changes at each step that might otherwise be canceled out by the quantization errors. Fig- 4b presents an ablation study of progressively increasing update intervals starting at 100 and increasing by factor of 1.2 up to 2500. We show the validation loss curves for fixed update frequencies 200, 400, 500, and 1000.
The results show that exponentially increasing the update interval is particularly beneficial for models employing quantization, enabling them achieve the same perplexity as those without quantization. Conversely. the performance gains are more subtle for models that do not use quantization. We hypothesize that even these models might benefit from the larger projection interval intervals. This could be due to the reduction in the accumulation of errors from frequent updates of the projection factor P. as the influence of outdated optimizer statistics becomes less prevalent. Finally, an ablation on the ranks used for P and B is given in Fig. 5 in the Appendix.
5 Related Work
We now provide an overview of related work on quantization, parameter-efficient fine-tuning methods, and memory-efficient approaches.
5.1 Neural Network Quantization and NF4
Quantization compresses neural networks by converting high-precision values into lower- power-precision formats, significantly reducing storage requirements [21 22 23 20 The process involves taking a datatype of high precision, such as 32-bit, requiring bytes of memory, and converting it into representation with increasing rounding errors but lower memory cost. In this work, we use NF4 quantization [ since it is a 4-bit code it only contains 2 different values. NF4 works by first normalizing values onto the interval [-1 1]. these are then discretized onto quantiles of the normal distribution, ( (see (3) for details). The elements of layer are divided into blocks of 64 weights. Each block B has a scaling factor Ma = max 3/332/--
WNF4 (NF4(W, Ma²)
(9)
argming, q. ww/ Ma/ qel.
(10)
w
Ma)
(11)
M3 UNF.
(12)
We provide an overview of different categories of quantization techniques, and how they relate to LoQT, Appendix [ Compared to prior approaches, LoQT retains the benefits of reduced memory usage while minimizing accuracy loss, using This allows for efficient model updates without the overhead of full matrix storage and re-quantization.
updates on low-rank representation.
5.2 Adaptation of Pretrained Networks
Low-Rank Adaptation (LoRA) [ enables fine-tuning of pretrained models using low-rank adaptors, effectively reducing the memory footprint by only training weight adaptors for targeted layers. However, simple low-rank training using LoRA factor matrices has not been shown work for pretraining 77.
LoRA employs trainable low-rank matrices A and B that used to update W following W, + +--++AB, where W,-1 is frozen to enable precise adjustments within low-rank framework. Since LoRA only trains A and B and keeps W fixed, QLoRA [5) explore quantizing W. They fine-tune quantized model gW) = Wg with 4-bit precision using randomly initialized 16-bit precision factors A and B. To address quantization emrors 3 -W, W). low-rank factors of the quantization error E
have been used [4]
LoQT extends LoRA to both pretraining and fine-tuning. Unlike traditional LoRA, LoQT uses A and B to refine W throughout training, with A initialized from W's gradient projection and B trained along this gradient path. LoQT also incorporates quantization and targeted optimization iterations

---

similar in spirit to LoftQ [4], correcting for quantization errors in W.: thus better aligning it with the original con-quantized W.
5.3 Memory Efficient Optimization
Optimizer memory consumption A significant portion of the memory needed to train neural networks is typically consumed by optimizer states. Notably, Adam [2] one of the most widely used optimizers, uses double the amount of memory as the gradient matrix to maintain first and second-order gradient statistics. Efforts to reduce this overhead have led to the development of adaptive optimization algorithms like Adafactor [24), which achieves sub-linear memory costs by factorizing the second-order statistics into row-column outer product. GaLore [8] expands
this concept by using low-rank factorization and projecting low-rank gradients up to full size when updating model weights.
combines low-rank updates with initial full- rank training. They find that doing one-third of the training in full-rank, and the subsequent two-thirds $.2 results in comparable performance to standard training methods.
Periodic updating of weight matrices ReLoRA 7
in low-rank (see
[ focuses on the structure of the gradients, projecting them into low-rank space using factors P and Q, which are derived from a truncated singular value decomposition (SVD) of the weight matrix gradient, Gw Q, This reduces memory costs associated with storing the optimizer states and aligns with findings from recent studies which suggest that learning primarily occurs within a low-dimensional subspace at a given time [25] 26). This can be further combined with applying per-layer gradient updates, reducing the memory needed for storing the gradients for the full model at once 13
Low-rank Gradients GaLore
LoQT builds on GaLore's gradient projection 2.1 to initialize LoRA factors while updating the full matrix following schedule inspired by ReLora, while only training one low-rank matrix per layer. We achieve comparable quality to GaLore and better performance than ReLoRA while reducing tunable parameters and memory usage compared both approaches.
Discussion and Conclusion
We have presented LoQT, method for memory-effiiint pretraining and adaptation of quantized models. Key insights behind the approach are the benefits initializing low-rank factors using the gradient of the weight matrix and using exponentially increasing update gaps that make updating of a quantized model feasible. While our initial goal was to lower memory usage, to facilitate the training of models such LLMs on consumer- graade hardware, we also cautiously excited about the results sometimes exceeding those of the baselines. We hope to see this explored in more detail in future work.
Our method general and has the potential to open up new ways decreasing memory use and improving training throughput through further optimization of our implementation. This could be done by using other quantization methods such NF2 (3) or quantization of activations, making possible to do the matrix multiplications using modern tensor core formats such as FP8 or INT4.
7 Impact and Limitations
The presented work has the potential to benefit those working in hardwwar--oosstrained settings by enabling more efficient training consumer-gradl grade hardware. We are particularly excited see the method being applied in single GPU settings.
We have validated LoQT several model sizes, by training over many steps, by fine-tuning on a standard benchmark for natural language understanding, mathematical reasoning, and language adaptation. While we are confident in our results, further exploration of training duration, data diversity, and hyper-parameter tuning might lead to different results those settings and we encourage users to confirm the benefit of LoQT for their approach.
10

---

8 Acknowledgements
This work is supported by the Danish Data Science Academy, which is funded by the Novo Nordisk Foundation ANNF21SA0099299 and VILLUM FONDEN (40516). Serge Belongie and Vésteinn Snæbjarnarson are supported by the Pioneer Centre for AI, DNRF grant number Pl. MJK acknowl- edges support from the Carlsberg Foundation and the Novo Nordisk Foundation. Mads Toftrup gratefully acknowledges the Data-Intensive Systems research group at Aarhus University for provid- ing GPU access.
References
[1] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. survey of quantization methods for efficient neural network inference. In Low-Power Com- puter Vision, pages 291-326. Chapman and Hall/CRC, 2022.
[2] Shuming Ma, Hongyu Wang. Lingxiao Ma. Lei Wang, Wenhui Wang. Shaohan Huang, Li Dong, Ruiping Wang. Jilong Xue, and Furu Wei. The era of 1-bit Ilms: All large language models are in 1.58 bits, 2024. URL https: /ttp://///axvv.aa/aa arxiv. org/ abs 2202
[3] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora:
Efficient finetun- ing of quantized Ilms. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 10088-10115 15. Curran Associates, Inc., 2023. URL /ttp:stttts/////ooeeiiiiii neurips cc wttp:ppss.cppppppppppeppppeeeee https//eeeopppoopppppppoeeeeeeee /
[apprr/2007111111111111666000262 Conferrnce pdf
[4] Baohao Liao, Christian Herold, Shahram Khadivi, and Christof Monz. Apiq: Finetuning of 2-bit quantized large language model, 2024. URL htLLhhtps://eet abt abs///////// /2tp./40..0000 05147
[5] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
[6] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lorat: Efficient low rank adaptation of large models. In Forty-first International Conference Machine Learning.
[7] Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High- rank training through low-rank updates, 2023. URL
/207 .06955
[8] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Gal.ore: Memory-efficient efficientt LLM training by gradient low-rank projection. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 61 1212616 143, PMLR, 21-27 Jul 2024. URL httph:tt////// httpss//prroceeding.. mlr www..pp..e......... press/ veress/v335// v235 2tte///22222za222 zhao24s. .ttml
[9] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. arXiv:1412.6980.
[10] Starkaōur Barkarson, Steinpór Steingrímsson, and Hildur Hafsteinsdóttir. Evolving large text corpora: Four versions of the Icelandic Gigaword corpus. In Nicoletta Calzolari. Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2371-2381, Marseille, France, June 2022. European Language Resources Association. URL https://acclartholooyy org/ 2022. 1rec-1 1. 254
[11] Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding. Proceedings of the 2020.
AAAI conference artificial intelligence, volume 34, pages
11

---

[12] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jeny Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URI /taetps://axiv..oom org/ abs/2110. 14168
[13] Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources, 2024. URL https
//arxiv.org www.:/rgggbbbbbo http//bbb//.....///// /20623306..02 09782
[14] Yixiao Li, Yifan Yu, Chen Liang. Nikos Karampatziakis, Pengeheng He, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-wwe quantization for large language models. In The Twelfth International Conference on Learning Representations.
[15] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog. Yixin Nie, Andrew Poulton. Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. arXiv:2307 D.XY::2007,0928..
[16] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
[17] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. arXiv:2203.15556.
[18] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URL httttttts///xxiiii wrg. abs 1804. 07461
[19] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations.
[20] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. In International Conference on Learning Representations.
[21] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized Sbit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-Neuis -NuurPP Edition (EMC2-NIPS), pages 36-39. IEEE, 2019.
[22] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815-88211 2020.
[23] Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R Lyu. Towards efficient post-training quantization of pre-trained language models. Advances in neural information processing systems, 35:1405- 15:145514888, 2022.
[24] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory
cost. In International Conference on Machine Learning, pages 4596-4604. PMLR, 2018.
12

---

[25] Brett W. Larsen, Stanislav Fort, Nic Becker, and Surya Ganguli. How many degrees of freedom do we need to train deep networks: a loss landscape perspective, 2022. URL https
7/arxiv. www/wbs.gbbb...... abs/ /217 05802
[26] Guy Gur-Ari, Damiel A. Roberts, and Ethan Dyer: Gradient descent happens tiny subspace, 2018. UtLps:t:: http://axivv..rr///////12122 /tt///0bb/1112.04554
[27] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantiza- tion aware training for large language models, 2023. arXiv:2205.18885
[28] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4350-4359, 2019.
[29] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization, 2024. URL
http://aww...rrr///////000...... 06118
[30] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv e-prints, pages arXiv-2310, 2023.
[31] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. httpp::////// org/ /bs/
2210. 17323
[32] Tim Dettmers, Ruslan A Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse- quantized representation for near-lossless Ilm weight compression. In The Twelfth International Conference on Learning Representations.
[33] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quiptt: Even better IIm quantization with hadamard incoherence and lattice codebooks. In Forty-first International Conference Machine Learning.
[34] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099. PMLR, 2023.
[35] Gunho Park, Minsub Kim, Sungiae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, Dongsoo Lee, et al. Lut-gemm: Quantized matrix multipli- cation based on luts for efficient inference in large-scale generative language models. In The Twelfth International Conference on Learning Representations.
[36] Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Learnable rounding based on element-wise division for post-training quantization. In International Conference Machine Learning, pages 18913-18939, PMLR, 2023.
[37] Jung Hwan Heo, Jeonghoon Kim, Beomseok Kwon, Byeongwook Kim, Se Jung Kwon, and Dongsoo Lee. Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models. In The Twelfth International Conference on Learning Representations.
[38] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quan- tization for large language models. In The Twelfth International Conference Learning Representations.
[39] Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. Advances in neural information processing systems, 31, 2018.
13

---

[40] Brian Chmiel, Ron Banner, Elad Hoffer. Hilla Ben-Yaace Yaacov, and Daniel Soudry. Accurate neural training with 4-bit matrix multiplications standard formats. In The Eleventh International Conference Learning Representations, 2023.
[41] Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. Advances in neural information processing systems, 31, 2018.
[42] Sergio P Perez, Yan Zhang, James Briggs, Charlie Blake, Josh Levy-Kramed Kramer, Paul Balanca, Carlo Luschi, Stephen Barlow, and Andrew William Fitzgibbon. Training and inference of large language models using 8-bit floating point. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023).
[43] Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low- low-precision training for large-scale vision-langue langual models. Advances 2023.
in Neural Information Processing Systems,
[44] Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang. Bolin Ni, Jingcheng Hu, Ruihang Li, Miaosen Zhang, Chen Li, Jia Ning, Ruizhe Wang, Zheng Zhang, Shuguang Liu, Joe Chau, Han Hu, and Peng Cheng. Fp8-Im: Training fp8 large language models, 2023. arXiv:2310.18313.
[45] Haocheng Xi, Yuxiang Chen, Kang Zhao, KAI JUN TEH, Jianfei Chen, and Jun Zhu. Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per-block quantization. In Forty-first International Conference on Machine Learning.
[46] Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. Loftq: bora-line-uning-awal quantization for large language models. In The Twelfth International Conference on Learning Representations.
[47] Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. ICLR 2024, 2023.
[48] Jiawei Zhao, Yifei Zhang. Beidi Chen, Florian Schäfer, and Anima Anandkumar. Inrank: Incremental low-rank learning, 2024. URL // //axxvv.vv.. wrgg/bs////....o.. 11250
14

---

A Quantization Methods
Quantization methods can be broadly categorized into Quantization-Awar Training (QAT), Post- Training Quantization (PTQ), and Fully Quantized Training (FQT).
integrates quantization in the training process by emulating inference time quantization where the model weights are quantized. By maintaining high precision gradients and optimizer states, QAT allows the model adapt to quantized weights while preserving accuracy. These methods predominantly focus on weight-only quantization approaches, which involve converting weight matrices into low-precision formats and then upcasting them just before computation (30 2]. This allows the main computation to occur at high precision, effectively preserving model accuracy while significantly compressing the model [T] However, QAT can require significant computational resources due to the need for full precision gradient calculations and large optimization states 3].
Quantization-Aware Training (QAT) QAT
28
29
30
Post- post-Training Quantization (PTQ) PQT BI 32 B3
34
35
20
36
37
38
involves converting
pretrained high-precision model into a lower precision format. This can be done directly or by using subset of the training data calibrate the quantization process fine-tune the quantized weights to adapt the model to the quantization. However, PTQ often results reduced accuracy compared to QAT because the model does not learn adjust to quantized weights during training [3 34].
Fully Quantized Training (FQT) FQT aims minimize memory and accelerate the training process by quantizing both forward and backward passes [39 40
42 43.. These methods often require specialized hardware 44 but promising for efficient training, and current approaches
cannot maintain accuracy
LoQT is form of QAT that gets close to FQT. As perform variant of LoRA (see $5.2 factor the layers W into two matrices P and B. quantize the W and with NF4, but keep B in 16-bit precision. We periodically update the W matrices using the product of the fixed P and the updated Bs without ever dequantizing all at once, only layerwise when merging in PB. This approach retains the benefits of reduced memory usage while minimizing accuracy loss, focusing high-pu nigh-precision updates on low-rank representation, and allowing efficient model updates without the overhead of full matrix re-quantization.
Choice of Quantization Method We chose NF4-quantization because it has been shown to work well 44, 3. Unlike other methods that adapt low-rank factors to quantization, such as LoftQ [14 and LQLoRA [47, LoQT operates under different constraints. Specifically, we do not have the flexibility to freely choose both A and factors because the matrix A is already fixed as the projection matrix P containing gradient information. Both LoftQ [4] and LQLoRA 47 use the SVD of the quantization error to initialize the low-rank factors A and B, aiming to minimize the difference between the quantized W and Q AB. The SVD gives the factorization UEVT, where the top singular vectors in U and V are used to initialize the low-rank factors and B, respectively.
In contrast, LoQT takes different approach due to the fixed nature of our low-rank adapter P (analogous to in LoftQ and LQLoRA). Instead of applying SVD to the quantization error, we aim to minimize an objective where W, Q, and P are fixed. We derive B using the formula B = (W - Q). where p* is the Moore-Penross pseudo-inverse of rather than the inverse. Incorporating information about the diagonal approximation of the Fisher information matrix into our objective could potentially reduce the error even further, direction we are interested in exploring in future work.
B Hyperparamters
provide the hyperparameter configurations and setups used our experiments to facilitate the reproduction of all results presented in this paper.
15

---

B.1 Prefraining
For the pretraining results shown in Tab. T adopted configurations from Gal.re [ and tested pretraining methods on different LLaMA 2 model sizes using the C4 dataset. Training was conducted with optimizer states in BFI6 precision, and NF4 precision quantization was used for LoQT. The model rank was adapted based on the largest layer with specific parameters.
Tab. ssowws the ratio r/dmodel. which denotes the rank relative to the largest weight matrix dimension. All experiments used a maximum sequence length of 256, learning rate warmup for the first 10% of training steps, and cosine annealing for the learning rate schedule, decaying 10% of the initial rate. Galore, LoQT-nq. and LoQT used exponentially increasing update frequencies starting at 100 and increasing by 100 + v/² where 1.2 and i is the update counter (see Section C.1 for more details).
We tested learning rates of 0.01, 0.005, 0.001, and 0.0005 across different model sizes. For models ranging from 60M to 350M parameters, learning rate of 0.01 yielded the best performance. In contrast, full-rank models required smaller learning rates: 0.001 for 60M to 350M models and 0.0005 for the model. scale the learning rates for LoQT, LoQT-nq. and Galore, we employed scale parameter set to 0.5 and 0.25 for the IB model. This parameter functions similarly to the LoRA A alpha parameter, determining the weight of the learned factors for LoQT and LoQT-nq. For Galore, experiments indicated that 0.5 was more effective than the 0.25 reported in [8]. This scaling approach effectively adjusts the learning rate, resulting in an actual rate of 0.005 for the multi-head
attention and feed-forward layers in LLaMA models, which is relatively large compared to the 0.001 used for full-rank models. Higher learning rates led to spikes in the training loss for both full-rank and LoQT models.
Table 6: Pretraining hyperparameters of LLaMA models for evaluation. (-) Indicates we did not train such a model.
Model Size
Hiddeen/Intermediatee
Attention Heads
Layers
Steps
Data Amount
Rank
60M
512 1376
8
10K
1.3B
128
130M
768 2048
12
12
20K
2.6B
256
350M
1024 2736
16
24
60K
7.8B
256
IB
2048 / 5461
24
32
100K
13.1B
512
7B
4096/1 1008
32
32
1024
13B
5120/13824
40
40
1536
B.2
Fine-tuning
We test learning rates in the range of 10 5 to x 10-4, For LoQT and LoftQ, we employed normal float NF4 quantization and performed five iterations of optimizing the error of quantization. We used batch size of 32 and maximum sequence length of 256. Tab. summarizes the detailed hyperparameters for tasks in GLUE using the DeBERTaV3- V3-base model. We use fixed projection gap of 2400 for all runs. Each of the parameter-efficien training methods applied all linear layers of the network, including attention projection and feed-forward layers, while the embedding layer is not trained.
Table 7: Hyperparameter setup for LoQT-nq. LoQT, 14f1Q[[4), LoRA 14, and Galore across various tasks the GLUE benchmark.
Method
Haper--arameter
MNLI
RTE
QNLI
MRPC
QQP
SST:2
CoL.A
STS-B
LoQT, LeFTQ
Epochs
5
20
10
60
10
10
20
60
Learning
x 10-4
10
5
10
10-5
10-
5 x 10-
Epochs
10
30
30
30
30
30
30
LoRA,, Galore
Leaning Rate
1 x 10
2
10
1
10
2 10
1x
10
2
10
2 10
3 10
16

---

C Rank Ablation
Fig. 5 shows the validation perplexity versus training steps for various ranks using LoQT-nq and LoQT on a 130 million parameter model over 20,000 iterations, All models employ an exponentially increasing update frequency starting at 100, with a factor of 1.2":. The results demonstrate that both the quantized (LoQT) and non-quantized (LoQT-nq) models follow a similar trajectory for ranks ranging from 64 to 512. However, for the smaller rank of 64, there is a slight divergence between LoQT-nq and LoQT, indicating a limit to how low the rank can be while maintaining accuracy with quantization. This plot highlights the tradeoff between rank and perplexity, suggesting that while our method supports low-rank training, there is minimum rank threshold needed to achieve results comparable regular pretraining.
Rank Ablations
40.0
LaQT-ng-
LoQT-nq- 256
LoQT 61
LoQT 256
37.5
LoQT-nq 128
LoQT-nq 512
35.0
LaQT- 128
LoQT 312
Poplexity
32.5
30.0
27.5
25.0
22.5
4000
8000
12000
16000
20000
Step
Figure 5: Rank ablation for LoQT and LoQT-nq showing perplexity as function of steps.
C.1 Memory Measurements
Fig. 6 demonstrates that LoQT requires less memory than GaLore and Adam, even without using per-layer gradients [33 or Adam 8-bit (20). The gap between LoQT and the baselines increases with larger model sizes. The configurations and ranks for each model are shown in Tab. 6 With LoQT and Adam 8-bit, it is possible to pretrain 13B model with rank 1024 on GPU with 24GB of VRAM. This enables training with LoQT on consumer GPUs, such as the NVIDIA 4090, using a small per-GPU token batch size of 256. Fig. in the main text provides detailed breakdown of each memory component for the 13B model. Maximum memory allocated is measured using nvi luttps://github. com/ Xww.aaaaaaaaaanntnaaaaaaa
Finetuning without merging low-rank factors Tab. showss how LoQT and LoQT-nq perform when not using the merging of factors while training. We see that the results are slightly worse than those where merging is performed: however, the results are still better than LoftQ, LoRA, and Galore, showing that in the finetuning case, where already have pretrained model, we omit merge low-rank factors into W, and still get good results. This would allow W to be kept fixed and quantized while training only the adapters, which enables using LoQT like LoRA, where multiple adapters can be trained for the same set of frozen weights
Task adaptation for GSM8K on Llama 7B and 13B The GSM8K dataset 12 "is dataset of 8.5K high-quality linguistically diverse grade school math word problems created by human problem writers. The dataset is segmented into 7.5K training problems and IK test problems". It has been used extensively to benchmark LLMs and is a good candidate for evaluating LoQT. We experimented with 7B and 13B models, performing a hyperparameter search to find the optimal learning rate for each method. Using the best learning rate, we trained each model over three seeds for three epochs with sequence length of 512, applying 4-bit quantization for fine-tuning Llama-2 models on the GSM8K training set. We report the average test set accuracy and standard error in Tab. B LoQT achieves an accuracy of 42.6 for Llama-7B and 52.9 for Llama-2 13B. Both results are average obtained over three seeds without merging and with rank 64. Tab. 8 lists the fine-tuned and quantized model on the validation set and report the best perplexity across learning rates in Tab. For each of the methods, only the attention projection matrices are trained.
We evaluate the
17

---

Memory Usage vs. Model
Adam
GaLore ASbit LW
60
GaLore
LoQT
GaLore A8bil
LoQT ASbit
Memory Usage (GB)
50
LoQT-nq A8bit
LoQT A8bit LW
40
30
RTX 4090
20
10
0
1b
3b
7b
13b
Model Size
Figure 6: Memory usage for LoQT baselines for different model sizes. LW means per-layer gradient updates as per [13, and A8hit means with Adam 8-bit. We evaluate using a token batch size of 256.
Table 8: Hyper-parameters used for the GSM8K task.
Hyper-parameter
Value
Optimizer
AdamW 0.1
Weight decay
LR
(0.1, 0.5, 3, 43 10 cosine
LR scheduler
Warmup ratio
3%
Epochs
3 16 (7B), (13B) 512
Batch size
Max sequence length
pretraining experiments for models up 1B parameters. To demonstrate the feasibility of LoQT on a larger model size in practical application, do continued pretraining for language adaptation. To this end, start with the mata-llama Ll --a--2-bbhf from Hugging face and run 1500 updates (1 epoch) on the model using batch size of 512. We compare LoQT with NF4 quantization, LoQT-nq (no quantization), regular training, and GaLore. We use rank 1024 for all models where applicable, adam8bit optimizer, and 150 warmup steps. For LoQT and GaLore we use a learning rate of 0.0002 with scale factor of 0.25 and for the others a learning rate of 0.00005. The dataset we train on is curated subset of public Icelandic text dataset [10] with 770k documents and a corresponding evaluation set. We released the co/ co/datasets/ htte///eteeeinnnneeeeeeee i icelandic). We chose Icelandic since the model has a limited performance on the language yet it was included to some degree in the pretraining data, enabling clear improvement trajectory. The results comparing Galore, Regular training (Full), and LoQT are shown Tab, 4 LoQT and LoQT-nq perform the best at 3.61 and 3.63 in perplexity, similar to full training at 3.79, while GaLore gets 3.96 and the original model 4.90.
Continued pretraining of Llama 7B We
data splits at (https: htttps:/hhuggnggee
D Memory-Saving Ablations
To evaluate differences between memory savings with layer-wise gradient computations and 8-bit Adam, conduct an ablation experiment using a 130M parameter model. We compare four settings: regular training, 8-bit Adam, layer-wise gradient updates, and a combination of 8-bit Adam with layer-wise updates, tested on Galore, LoQT, and regular FP16 training. Our results, illustrated in
18

---

Table 9: Results with LoQT, LoQT-nq. and GaLore of DeBERTaV3- models the GLUE development set. We report mean and standard error over three seeds. The best results on each dataset shown in bold. "No-Merge" means we do not update the pretrained matrix W during training.
Rank
Method
MNLI
QNLI
RTE
SST
MRPC
CoLA
QQP
STSB PCorr
Average
32
LoQT-nq
90.010.10
96,210.06
84.8
95.9-0.06
94.1110.5
72.5:0.41
90.010.066
91 91.5100.7
89.1
32
LoQT
90.010.09
94.3:0004
8111=0.1
95.5-0.10
94.4:0.20
70.5:00.35
89:2:10.22
91.5120.13
88.7
32
LoQT-na Merge
90.00.100
94.1+0011
84.5=0.01
95.6-0.03
938-=0.01
72.0=0.01
89.8+0.01
91.6-10.01
89.0
32
LoQT: Merge
95.7=0002
94.2:10.0
32
LaRA
89,5 89.92003
94.0+00.99
83:6:0.12
95.7=0.15
93.5:20.26
69:3:20:47
89.81..11
90.7:t0.22
88.3
32
LenQ
904+0009
93.2+0..2
8388-063
95.6 -0.07
93.2+0014
71.1+0.28
89 89.6++011
91 0+000099
88.4
GaLore
83.7.0.79
90.610001
130m Model - Validation Perplexity
34
WorPe
Wa. EC, EF
Wa. Po EC
No Q
32
Lower is better
Perplexity
Simple 4-bit Quantization W and P: Flattens early.
28
Error Compensation (EC): This gap reduced by 72.%.
Exponentially increasing update intervals (EI) and EC: the gap is reduced by 93 %
24 4000
8000
12000
16000
20000
Step
Figure 7: Naive Quantization of W and P vs including Error Compensation(EE) and Exp. increasing intervals(EI).
Fig. show that adding these memory-saving components introduces small decrease in performance. Importantly, LoQT experiences a proportionally smaller decrease in performance compared to Galore and full training when combining both 8-bit and layer-wise updates. These results demonstrate that while memory savings come with some trade-offs, LoQT maintains good performance. In addition, due to the lower memory requirements of LoQT, enable training of larger models without using layer-wise gradient computations and 8-bit Adam.
Memory savings with varying sequence lengths With larger contexts, the overall memory con- sumption is increasingly influenced by activations. Following prior work 48), our experimentation has focused the setting of shorter context lengths (256 tokens). But as demonstrated in Fig. g the benefit of LoQT does transfer to longer context lengths, enabling training of Llama 7B on consumer hardware with context length of 2048 and 4096 40GB A 100 without activation checkpointing.
E
Generalization to other architectures and models
LoQT should work with any type of deep neural network using linear layers, such as vision trans- formers or state space models. To narrow the scope our work and provide more detailed analysis, however, we choose to focus on a well-studied auto-regressive language model and bi-directional masked language model that has been commonly used as a basis in much of the related work. We hope to see LoQT being used for other model architectures.
19

---

Memory Usage vs. Model
GaLore ASBit
70
GaLore
LoQT ASBi
GnLore ASBit
LoQT A8Bir LW
60
LoQT
Memory Usage (GB)
50
4BB
40
30
24 GB
20
10
256
512
1024 Sequence Length
2048
4096
Figure 8: Memory usage for common context lengths (256 4096) for the LLaMA 7B model sured using torch. cuda. maddamaa___.. _aaa_mmm_y___aa semoor_allocated We include lines representing 16GB, 24GB, and 40GB VRAM limits indicate which configurations fit within the VRAM capacities of standard NVIDIA GPUs.
30
Method
Full
Validation Parplexity (ppl)
28
LoQT
Galore
26
24
22
20
Baselines
Adam Sbit
Adam LW
Adam8bit LW
Figure 9: Validation perplexity with various optimization techniques: Adam Sbit, per-layer updates, and their combinations, compared baseline training without these optimizations. LW means per-layer gradient updates as per 133, and Adam&bit means with Adam 8-bit.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See 22 and 3
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See and in particular the section on limitations 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
While theoretical results are limited, we provide pseudo-code and derivations
in $
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide pseudo-code, hyperparameters in the Appendix, and an implemen- tation is provided as supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code and instructions for setting up and running environment. The code will be open-sourced after submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
cublic//uuddes/Codeuubbsss cosssiionooyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppplgguuuaaaeoeae wtt.eeeeeeoSSSeeesSSSeoeee.. www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, see an Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We run three different seeds for most runs, bar the more expensive 1B runs. Our work is mainly claiming memory saving not claiming SOTA on any benchmarks, but runs are provided show competitive results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See list of resources used in 3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :tattps://neurrre http///rripscccpuullieeeeeeeeeee hcc/ppbliic/tthhccccieeeeel..ll
Answer: [Yes]
Justification:
We went over it and could not find any violations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See $6 and in particular 7 the work enables more efficient model training in comorryeo setting, potential net benefit in many settings.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We reference and respect licenses of the code we build on.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The only asset is the provided code, which is released with documentation, support will be provided.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No participants were involved.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "GK4ixfa86a", "reviewer": "Reviewer_tC4j", "summary": "The authors propose LoQT, a novel method to use low-rank LoRA adaptors directly during pretraining. LoRA adaptors have been used for downstream finetuning before, but not for full pretraining.\n\nLoQT works by first quantization a weight matrix $W$ into $W_q$. Then the quantization error is decomposed into low-rank factors. One factor is the projection $P$ from the low-rank subspace spanned by the principle eigenvectors of the current weight gradient (obtained via SVD on the gradient ), and the other is a trainable factor $B$. $P$ is frozen and also quantized. LoQT will repeat the above initialization process several times over training, and essentially represents the final weight as a sum of low-rank weight updates, each initialized via SVD of the weight update on a certain step.\n\nLoQT leverages previous observation that the weight gradient is low-rank, and that this rank decreases over training as learning rate decreases. The initialization frequency is set of increase over training.\n\nExperiments show that LoQT improves greatly over LoRA variants for pre-training, and improves in downstream accuracy over GaLore, a previous method which also tackles using LoRA adaptors for pretraining.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting idea backed by empirical observation of low-rank weight gradients.\n- Well written paper, good explanation of complicated idea.", "weaknesses": "- This type of approach should work with any type of DNN, but the authors only experiment with LLMs, and just one family of LLM (Llama-2). Some data on how generalizable LoQT is to other architectures would be welcome.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Generalization to other architectures and models \n\nWe agree that LoQT should work with any type of DNN using linear layers, such as vision transformers or state space models. To narrow the scope of our work and provide a more detailed analysis, however, we choose to focus on a well-studied auto-regressive language model and a bi-directional masked language model that have been commonly used as a basis in much of the related work. We believe that Llama 2 is representative of a large family of text-based Transformer LLMs, which overlap in most of the underlying architectural decisions. In Sec. 4.2 (line 231) and Tab. 2, we present results for a diverse set of model sizes ranging from 60M to 1B parameters within the Llama 2 architecture. In Sec. 5 (line 276) and Fig. 4, we provide an ablation study of each model component, as well as the effects of varying update interval sizes. In Sec. 4.3 (line 241) and Tab. 2, we show results for fine-tuning a bi-directional language model, the DeBERTaV3-base model on the GLUE tasks. All experiments are averaged over three seeds to ensure robustness.\n\nAdding to this, we include more fine-tuning experiments of a pre-trained model in the **common response**. These include continued pre-training of the Llama2 7B model on a low-resource language and adaptation on a common benchmarking task (GSM8K) that confirms the memory saving and performance preservation using LoQT for adaptation of an existing model.\n\nHaving said that, we do plan to consider LoQT's generalizability in future work and thank the reviewer for pointing out this strength of LoQT. We will make sure to elaborate on this in the updated manuscript, including what needs to be kept in mind when using quantization for, e.g., vision models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IGTagDsdTY", "reviewer": "Reviewer_ps9K", "summary": "LoQT is a method for low-rank training of high-rank quantized weights, using gradient-based tensor factorization to initialize low-rank trainable weight matrices. LoQT is suitable for both pretraining and fine-tuning models, significantly enhancing its applicability. Despite the 24G GPU memory limitation, LoQT can train a 13b model with only a slight speed reduction.", "questions": "1. Given the computational power demonstrated in row 213 and the excellent efficiency of LoQT, it is surprising that the paper does not present results for larger models. What factors prevent fine-tuning on larger models? Are there any part that only use simulated results without achieving actual speedups?\n2. In Figure 1, LoQT A8bit LW is shown as the most memory-efficient method. Is there any section in the paper where the results of this most extreme version are presented?\n3. Table 3 states that LoRA cannot be used for pretraining. This raises confusion about the data in the LoRA row of Table 1, which focuses on pretraining methods. It appears this might be a mistake carried over from GaLore.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "LoQT can significantly reduce GPU memory requirements for training LLMs with only a minor speed drop. It enables training LLMs from scratch. Gradient-based tensor factorization allows LoQT to freeze and quantize low-rank parameters, further enhancing efficiency.", "weaknesses": "The model size involved in the experiment is too small. Considering that LoQT has always used 24G 4090 as its efficiency target, the experiment only covers the model size that can be fully trained in 24G, which cannot highlight the advantages of LoQT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and recognition of LoQT’s memory efficiency and its practical applicability. % for both pretraining and fine-tuning through gradient-based tensor factorization.\n\n## 1. Size of models\nWe agree that training larger models than 1B would strengthen our work. In particular, the suggestion of fine-tuning on larger models. We have added experiments for fine-tuning a 7B parameter model to better showcase the potential of LoQT on larger scales. We show this successfully in the **common response**. See Tables A and B in the common response, and Figure 2 in the extra PDF. \n\nWhile we would have liked to train models >1B from scratch, these require computational resources beyond those that have been available to us. As reported in Sec. 4.1 (Line 214), training a 1B model takes more than four days on four A100 GPUs for Galore, Regular FP16 training, and LoQT. \nOur aim was to validate our method across different model sizes to demonstrate that LoQT can achieve the same performance as regular FP16 training while maintaining most weights in a 4-bit quantized format and performing only low-rank updates. With this validation, we show that LoQT is effective in terms of quality while using much less memory.\n\n\n## 2. Memory Efficiency in Figure 1\nTo clarify the differences between memory savings with layer-wise gradient computations and 8-bit Adam, we conduct an ablation experiment using a 130M parameter model. \nThis experiment compares four settings: regular training, 8-bit Adam, layer-wise gradient updates, and a combination of 8-bit Adam with layer-wise updates, tested on Galore, LoQT, and regular FP16 training.\nOur results, illustrated in Figure 4 in the extra PDF, show that adding these memory-saving components introduces a small decrease in performance. Importantly, LoQT experiences a proportionally smaller decrease in performance compared to Galore and full training when combining both 8-bit and layer-wise updates. These results demonstrate that while memory savings come with some trade-offs, LoQT maintains good performance. In addition, due to the lower memory requirements of LoQT, we enable training of larger models without using layer-wise gradient computations and 8-bit Adam. \n\n## 3. Clarification on LoRA for Pretraining (Tables 1 and 3)\nWe acknowledge the confusion regarding Table 3’s statement that LoRA cannot be used for pretraining. To clarify, while LoRA does not perform well for pretraining, it is not entirely unusable. As demonstrated by both Galore and ReLoRA, LoRA is comparatively less effective at pretraining.\nWe will update Table 3 to indicate “Yes/Sort-of” and reference ReLoRA, which shows that even with a warm start (full training for 1/3 of the iterations), the difference in perplexity between LoRA and regular pre-training is still significant.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hvHVbtFc6V", "reviewer": "Reviewer_CPTS", "summary": "This paper introduces a novel metho, LoQT, to address the computational and memory challenges in training large neural networks. LoQT factorize the weight matrices of the neural network into low-rank components P and B. During training, only \nB is updated while P and weights are freezed. Similar to ReLoRA, the PB is merged back into the full-rank matrix W periodically. The method enables the training of models with up to 7 billion parameters on a consumer-grade 24GB GPU, and has the potential to train 13B models.", "questions": "1. Is there a specific reason why you apply NF4 in your method? Does it gives much better performance compared with FP4 (E2M1 and E3M0) and INT4?\n2. Figure 1 is based on a fine-tune setting with token-number = 256. Can you provide an analysis of the memory consumption for a pretraining task, with a common batch size and sequence length setting?", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The combination of low-rank adaptation and quantization can significantly reduce the memory footprint.\n2. This method shows end-to-end memory reduction result.\n3. The introduction of quantization does not lead to significant degradation, even at 4-bit.\n4. The baseline setting in the experiments is very clear and thorough.", "weaknesses": "1. The techniques are not very new, and can be somehow viewed as a combination of existing methods like GaLore, ReLoRA, LoftQ(compensate for quantization error by reinitialization) and NF4 quantization (from QLoRA)\n2. More experiments is preferred. For example, more fine-tuning experiments on LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback and for highlighting the strengths of our work.\n\n## 1. More fine-tuning experiments\nGood suggestion. We have included fine-tuning experiments of a 7B model in the **common response**. We adopt a Llama model for a new low-resource language and a commonly used benchmarking task, GSM8K, demonstrating that LoQT also performs well in downstream applications.\n\n## 2. Comparison to prior methods\nWhile our approach combines and builds on ideas explored in prior work like GaLore, ReLoRA, LoftQ, and NF4 quantization, we believe our work meaningfully extends these both in terms of how we deviate from the prior individual works and how we combine them.\n\nWe show that for the first time, LoRA can efficiently and successfully be used for pretraining while enabling the quantization of most weights with 4-bit precision, resulting in significant memory savings. The exponentially increasing update intervals, combined with the error compensation and gradient-based initialization, all come together holistically, where the combined benefits are larger than the sum of the parts.\n\n1. **Separation into Trainable and Frozen Weights**\n   - **GaLore:** Derives a low-rank projector using an SVD of the gradient. However, Galore does not separate trainable weights from frozen weights. In Galore, only the projector matrix $P$ is not trained, while the weight matrix $W$ is updated contiguously at every update step.\n   - **LoQT:** We extend this by introducing a separation between frozen and trainable weights. Both $W$ and one of the factors (the projector $P$) are frozen while the new factor $B$ is trained. This further reduces memory for gradients (only low rank, whereas Galore uses \"full rank\" gradients) and enables pre-training within the LoRA framework. This then enables quantizing most of the model weights, as discussed below.\n\n2. **Gradient Based Initialization and Periodic Merging**\n   - **ReLoRA:** The $A$ and $B$ factors are initialized using Gaussian noise and zeros, and both are trained. ReLora then periodically merges the product of the factors into the weights of $W$ to explore a new subspace. However, ReLoRA requires full rank training for the first 1/3 of the time to be effective, limiting its usability on smaller GPUs.\n   - **LoQT:** We build on ReLoRA by using clearly separated frozen and trainable weights. We show that to attain well-performing pretraining using just low-rank adapters, a gradient-based initialization of the LoRA factors is needed. This enables quantization approaches like NF4 from QLoRA while maintaining comparable precision to regular FP16 training. We only train a single low-rank factor per layer, which enables quantizing the weights of both $W$ and $P$, allowing for additional memory savings.\n\n3. **Quantized Pre-training**\n   - **QLoRA:** Works well for fine-tuning but not pre-training. However, implementing this with ReLoRA, without gradient-based initialization, would face similar problems, and quantization could exacerbate these issues.\n   - **LoQT:** We use the specific NF4 quantization approach from QLoRA, as it has been shown to be superior to FP4 and INT4 quantization by a significant margin. We apply their approach to $W$ and $P$ by introducing error compensation and exponentially increasing update intervals:\n       - **Error Compensation Method:** We introduce an error compensation method for initializing the $B$ matrix, incorporating the quantization error between $W$ and $Q$. This significantly reduces the performance gap between non-quantized and quantized models. We show (Figure 1 in the extra PDF) that simple quantization of $W$ and $Q$ creates a significant gap in model quality. The method reduces this gap by 72%. This is impossible with GaLore due to the continuous updates of $W$. While LoftQ also uses error compensation, their formulation operates under different constraints (see response 4 to Ekw9).\n       - **Adaptive Update Intervals:** We vary the length of update intervals by exponentially increasing intervals for selecting a new projection matrix. This, in combination with the error compensations, reduces the error by 93%. This opens up further research opportunities into how the choice of update intervals affects learning and runtime.\n\nIn summary, our contributions nontrivially build on prior ideas of low-rank training and quantization, extending and combining them in ways that were previously not possible, with significant memory improvements and observed performance gains.\n\n\n## 3. What is the reasoning for choosing NF4? \nWe chose NF4 since it performs better than FP4 (E2M1 and E3M0) and INT4 in related work. As shown in Table 2 (Page 7) of QLoRA, NF4 achieves a significantly better mean perplexity (PPL) than the other quantization methods. While our focus was primarily on reducing the memory footprint of pre-training, we acknowledge that INT4 could enable faster computations since it is directly supported on certain GPUs. However, to gain this benefit, one must do low-precision matrix multiplications, which we see as an interesting future direction.\n    \n## 4. Can you provide an analysis of memory savings with varying sequence lengths? \nGood question. With larger contexts, the overall memory consumption is increasingly influenced by activations. Following prior work, our experimentation has focused on the setting of shorter context lengths (256 tokens). But as demonstrated in Figure 3 in the extra PDF, the benefit of LoQT does transfer to longer context lengths, enabling training of Llama 7B on consumer hardware with a context length of 2048 and 4096 on a 40GB A100 without activation checkpointing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kuirYTiFHh", "reviewer": "Reviewer_Ekw9", "summary": "This paper proposes LoQT, a new method for efficiently training quantized models. LoQT is based on the low-rank decomposition of gradient matrices, and is inspired from GaLore [1] and LoRA [2] methods for transformers architectures. The work’s fundamental contributions are: 1) Propose a low-rank factors initialization (for each pre-trained matrix) based on gradient matrix and (weight) quantization error. 2) Show that Low-rank Gradients can be seen as (low-rank) adapters.\nThe numerical experimental results of the proposed method outperform other baselines on the GLUE task.\n\n\n---\n\n[1] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection, 2024.\n\n[2] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.", "questions": "1) Can you explain the difference between LoQT error compensation mechanism, and already published strategies such as LQLoRA [1] ?\n2) In appendix (Table.4 and Figure.6) you have run LoQT on a LLaMA with 7B parameters, can you provide the corresponding perplexity results ? Is it competitive with the corresponding literature (ApiQ [2], Loftq [3], LQLoRA [1])?\n3) Section 3.3 proposes to update weights $W_t$ (by merging low-rank factors). How does LoQT perform, when no update is realized on the pre-trained weights $W$? In particular, for fine-tuning on 10 different tasks LoQT would require 10 different models, while LoRA strategies would require a single full-rank model and 10 (smaller) adapters. \n4) (minor) l.139 ``From (5) '': I think there is a typo regarding the ref.\n\n---\n\n[1] Guo, H., Greengard, P., Xing, E., and Kim, Y. (2023). Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations.\n\n[2] Liao, B. and Monz, C. (2024). Apiq: Finetuning of 2-bit quantized large language model. arXiv preprint arXiv:2402.05147.\n\n[3] Li, Y., Yu, Y., Liang, C., Karampatziakis, N., He, P., Chen, W., and Zhao, T. (2023). Loftq: Lora-fine-tuning-aware quantization for large language models. In The Twelfth International Conference on Learning Representations.", "limitations": "Impact and limitations properly discussed.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) LoQT successfully addresses the optimizer states memory concern for large neural networks training, by employing a low-rank decomposition of the gradient matrices. Different from GaLore [1], LoQT also quantizes (with NF4 from QLoRA [2]) the weights. LoQT successfully takes into account the induced quantization errors, when updating the low-rank factors.\n2) The authors show that GaLore strategy can be formulated into a low-rank adapters decomposition, and that it is sufficient to train (with gradient descent) one of the low-rank factors.\n3) This paper proposes a new periodically update strategy for fine-tuning pre-trained weights $W_{t+1} = W_t + P_t B_t$, with no need of optimizer states of the (full-rank) weights: only the optimizer states of $B_t$ are required.\n\n---\n\n[1] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection, 2024.\n\n[2] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023.", "weaknesses": "1) LoQT is mainly based on GaLore [1] with an additional step of weight quantization. Different from GaLore, no mathematical analysis is proposed, and the projection steps (SVD) are identical to GaLore's.\n2) The main novelty of LoQT is its initialization that compensates the quantization error induced from the weight quantization. But (i) no contribution about weight quantization is proposed (NF4 strategy from QLoRA [2] is reused), and (ii) no discussion with other strategies that adapt low-rank factors to quantization (ApiQ [3], Loftq [4], LQLoRA [5]) is proposed.\n3) The experiments again strictly follow the benchmark of GaLore [1], no experiments on 'standard' LLMs (such as LLaMA-7B; see for e.g. [5]) is given to fairly assess the computational gain of LoQT.\n\n\n---\n\n[1] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection, 2024.\n\n[2] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023.\n\n[3] Liao, B. and Monz, C. (2024). Apiq: Finetuning of 2-bit quantized large language model. arXiv preprint arXiv:2402.05147.\n\n[4] Li, Y., Yu, Y., Liang, C., Karampatziakis, N., He, P., Chen, W., and Zhao, T. (2023). Loftq: Lora-fine-tuning-aware quantization for large language models. In The Twelfth International Conference on Learning Representations.\n\n[5] Guo, H., Greengard, P., Xing, E., and Kim, Y. (2023). Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough feedback. We appreciate the recognition of LoQT’s efficient handling of optimizer state memory, its effective quantization error management and its periodic update strategy. \n\n## 1. Additional Finetuning Experiments\nExcellent suggestion; we agree that more fine-tuning experiments should be included. We refer to the common comment where we describe fine-tuning of Llama 7B for a downstream task and continue pretraining for adaptation of the model to a new language. In both cases, LoQT excels.\n\n## 2. LoQT is mainly based on GaLore with weight quantization\nWhile our approach combines elements from existing methods like GaLore and others (see our detailed response to CPTS), we believe LoQT introduces significant new contributions, drawing on many related works besides GaLore in a non-trivial manner. We are the first to show that LoRA works well for pretraining given the right initialization, opening up further exploration in the LoRA literature. To our knowledge, we are also the first to present a method that can perform memory-efficient pretraining with most weights in 4-bit precision, resulting in a model that is already in 4-bit precision and thus does not require post-training quantization or quantization-aware training afterwards. \n\nWe show, theoretically, that GaLore can be rewritten to a LoRA format, where one of the low-rank factors is initialized as the gradient projection matrix P. We then prove that only updating B and keeping P and W frozen results in the same gradient updates as Galore. This enables a separation that enables quantization. However, we see that naive quantization of W and P leads to a significant decrease in quality (See Figure 1 in extra PDF). To tackle this, we developed a tailored error compensation approach, which is described in our response below. We also saw that during the later part of training, the errors we get from the quantization increasingly affect the loss. We show that less frequent updates over time help because we do fewer quantizations of W and P. This also speeds up training since it requires fewer SVDs during training.\n\nWithout the clear separations of frozen and trainable weights that enable us to use quantization, and without error compensation initialization of $B$ and without exponentially increasing update intervals, our method would not perform comparably to full training. \n\n## 3. Quantization and Related Work\nWe chose NF4 simply because it has been shown to work well. Our goal is not to introduce a new way of quantization nor to thoroughly compare to the post-training quantization literature. However, we appreciate the suggestion of considering LoQT as a post-training quantization method and will consider this in more detail in the final manuscript.\n\nUnlike other methods that adapt low-rank factors to quantization, such as LoftQ and LQLoRA, LoQT operates under different constraints. Specifically, we do not have the flexibility to freely choose both $A$ and $B$ factors because the matrix $A$ is already fixed as the projection matrix $P$ containing gradient information. \n\nWe will include a more thorough discussion and comparison of ApiQ, LoftQ, and LQLoRA in the final manuscript. See also our answer below for a direct comparison.\n\n## 4. Difference between the LoQT error compensation mechanism and already published strategies such as LQLoRA and LoftQ\n\nBoth LoftQ and LQLoRA use the SVD of the quantization error to initialize the low-rank factors $A$ and $B$, aiming to minimize the difference between the quantized $W$ and $Q + AB$. The SVD gives the factorization $U \\Sigma V^T$, where the top $r$ singular vectors in $U$ and $V$ are used to initialize the low-rank factors $A$ and $B$, respectively. \n\nIn contrast, LoQT takes a different approach due to the fixed nature of our low-rank adapter $P$ (analogous to $A$ in LQLoRA). Instead of applying SVD to the quantization error, we aim to minimize an objective where $W$, $Q$, and $P$ are fixed. We derive $B$ using the formula $B = P^+(W - Q)$, where $P^+$ is the Moore-Penrose pseudo-inverse of $P$ rather than the inverse. Incorporating information about the diagonal approximation of the Fisher information matrix (like in LQLoRA) into our objective could potentially reduce the error even further, a direction we are interested in exploring in future work. Additionally, LQLoRA proposes dynamically adjusting the bit-width.\n\n## 5. Clarification on 7B LLaMA Experiments\n\nWe did not pre-train a 7B LLama model, as mentioned in the table caption of Table 4. \nWe understand this is confusing, but we opted to include them for reference regarding the memory experiments presented in Figures 1 and 6.\nHowever, we have provided new results for continued pre-training on a 7B model, as described in the **common response**.\n\n## 6. Finetuning without merging low-rank factors\nGood suggestion. We have done this experiment showing how LoQT and LoQT-nq perform when not using the merging of factors while training. The results are shown in Table 1 in the extra PDF indicated with (No-Merge). We see that the results are slightly worse than those where merging is performed; however, the results are still better than LoftQ, LoRA, and Galore, showing that in the finetuning case, where we already have a pretrained model, we can omit to merge low-rank factors into W, and still get good results. This would allow W to be kept fixed and quantized while training only the adapters as suggested, at least for some tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the constructive and positive reviews of our work. All reviewers point out the significance of the memory reduction of LoQT in pre-training, and that our experiments show that "quantization does not lead to significant degradation" **CPTS** while "significantly reducing GPU memory requirements for training LLMs" **ps9K**. **tC4j** and **CPTS** find the work "novel". **tC4j** and **ps9K** acknowledge that adapters have not been used for full pretraining before. **Ekw9** points out the benefit of the "new periodical update strategy". **tC4j** finds the paper "well written" and having a "good explanation of a complicated idea" while **CPTS** finds the setting of the experiments "very clear and thorough".

A common theme in the reviews is suggesting experimentation with larger model sizes than 1B, such as fine-tuning a publicly available pre-trained LLM. We agree that these are valuable experiments and include two kinds of fine-tuning results for Llama2-7b. The first is a task-adaption on grade school math questions using the GSM8K dataset [2]. The second is continued pre-training for a low-resource language. The results show that LoQT gives excellent results on both tasks while also reaping the benefits of the significantly lower memory footprint established for the pre-training tasks. This is described in detail below.

Reviewers **CPTS** and **Ekw9** comment on how LoQT compares to prior work. To address these, we describe in detail the key differences to prior work in the individual responses, as well as how we non-trivially combine them to make LoQT work. 

We respond to other comments in the individual responses, we encourage all reviewers to have a look at them. Finally, we would like to thank all reviewers for the quality of their reviews.

## Task adaptation for GSM8K on Llama 7B 
The GSM8K dataset [2] "is a dataset of 8.5K high-quality linguistically diverse grade school math word problems created by human problem writers. The dataset is segmented into 7.5K training problems and 1K test problems". It has been used extensively to benchmark LLMs and is a good candidate for evaluating LoQT. We experimented with models of sizes 7B and 13B, as suggested by the reviewer. We adopt the evaluation and training setup of ApiQ with a sequence length of 512 for 4-bit quantization and finetuning of Llama-2 models (7B and 13B) on the GSM8K training set [2, 3] and report the accuracy on the test set. Using the same hyperparameters as reported by ApiQ (see Tables A4 and A5 of their paper), LoQT achieves an accuracy of 41.6 for Llama-7B and 52.2 for Llama-2 13B. In contrast, the previous best were 39.8 and 51.2, thus surpassing the baselines. Both results are without merging and with rank 64. Results are shown in the Table A below. 


| Method | Bit | GSM8K (acc↑) Llama-2-7B | GSM8K (acc↑) Llama-2-13B |
|--------|-----|--------------------------|---------------------------|
| LoRA   | 16  | 36.9                     | 45.3                      |
| QLoRA  | 4   | 35.1                     | 39.9                      |
| LoftQ  | 4   | 35.0                     | 45.0                      |
| ApiQ   | 4   | 39.8                     | 51.2                      |
| LoQT   | 4   | **41.6**                 | **52.2**                  |

*Table A: Finetuning results of GSM8K on Llama-2-7B and Llama-2-13B.*

## Continued pre-training of Llama 7B
We run our pre-training experiments for models up to 1B parameters. To demonstrate the feasibility of LoQT on a larger model size in a practical application, we do continued pre-training for language adaptation. To this end, we start with the `meta-llama\Llama-2-7b-hf` from Hugging face and run 1500 updates (1 epoch) on the model using a batch size of 512. We compare LoQT with NF4 quantization, LoQT-nq (no quantization), regular training and GaLore. We use rank 1024 for all models where applicable, adam8bit optimizer, 150 warmup steps. For LoQT and GaLore we use a learning rate of 0.0002 with a scale factor of 0.25 and for the others a learning rate of 0.00005. The dataset we train on is a curated subset of a public Icelandic text dataset [1] with ~770k documents, and a corresponding evaluation set, we will release the splits when appropriate. We choose Icelandic since the model has a limited performance on the language yet it was included to some degree in the pre-training data, enabling a clear improvement trajectory.
Validation perplexity results are shown in the Table B below and as a step vs perplexity plot in Figure 2 in the extra PDF.
LoQT and LoQT-nq perform the best at 3.61 and 3.63 in perplexity, similar to full training at 3.79, while GaLore gets 3.96 and the original model 4.90. 

| Model          |  Perplexity ↓ |
|----------------|------------------|
| No Training    | 4.90           |
| Full (FP16)    | 3.79           |
| GaLore         | 3.96           |
| LoQT-nq        | **3.61**           |
| LoQT           | 3.63           |

*Table B: Validation perplexity continued pre-training of Llama 7B on Icelandic corpus.*

We also include additional experiments: 1) memory component ablation experiments (Figure 4 in the extra PDF), showing the effect of using Adam 8bit and layer-wise gradients; 2) experiments demonstrating LoQT’s memory scalability for training Llama 7B with longer context lengths up to 2048 on consumer hardware (Figure 3 in the extra PDF); and 3) an experiment showing fine-tuning results for GLUE without merging factors into $W$ during training (Table 1 in the extra PDF).

## New References

[1] Barkarson, S., et. al., (2022, June). Evolving Large Text Corpora: Four Versions of the Icelandic Gigaword Corpus.

[2] Cobbe, K., et. al., (2021). Training Verifiers to Solve Math Word Problems.

[3] Liao, B., et. al., (2024). ApiQ: Finetuning of 2-Bit Quantized Large Language Model

Meta-review: Reviewers agree that the paper represents a novel contribution in efficient pretraining, albeit being a combination of existing ideas.

Some questions about the scale of the experiments are well answered in the rebuttal, and reviewers have concurred that their questions are well answered.
Decision: Accept (poster)