LoQT: Low Rank Adapters for Quantized Training

Authors: sebastian  loeschcke
mads  toftrup
michael j kastoryano
serge  belongie
vésteinn  snaebjarnarson
Venue: ICML_cc_2024_Workshop_WANT
Published: 2024-05-30 13:25:11

Abstract: Training of large neural networks requires significant computational resources. Despite advances using low-rank adapters and quantization, pretraining of models such as LLMs on consumer hardware has not been possible without model sharding, offloading during training, or per-layer gradient updates. To address these limitations, we propose LoQT, a method for efficiently training quantized models. LoQT uses gradient-based tensor factorization to initialize low-rank trainable weight matrices that are periodically merged into quantized full-rank weight matrices. Our approach is suitable for both pretraining and fine-tuning models, achieving similar performance to full training, which we demonstrate experimentally for language modeling and downstream task adaptation. We find that LoQT enables efficient training of models up to 13B parameters on a consumer-grade 24GB GPU.

Main paper: 
 introduction: 
 Training large neural networks requires substantial hardware and energy resources. Reducing these requirements is thus important for cost efficiency and environmental sustainability, while also lowering the entry barrier for researchers and practitioners. The main barriers in training large models are the compute operations required, as well as the memory needed to store those computations, in this paper we focus on the latter. Memory use during training comes primarily from the weights of the model itself as well as the optimizer states used to train the model. To target the weights, variations on low-rank adaptation (LoRA) (Hu et al., 2021; Hayou et al., 2024b; Dettmers et al., 2023a; Lialin et al., 2023; Liao & Monz, 2024) have been suggested to decrease the number of trainable parameters, in combination with the use of low precision representations. To target the optimizer, low-rank approaches for projecting gradients to a lower rank have been employed (Zhao et al., 2024) . Finally, various applications of quantization (Gholami et al., 2021; Ma et al., 2024; Dettmers et al., 2023a) have been used to decrease memory requirements. In this work, we combine and extend these approaches into a highly memory-efficient training configuration.In typical training setups, the optimizer states take up larger space than the model itself, as methods such as Adam (Kingma & Ba, 2017) need to keep track of two parameters for each weight of the model. GaLore (Zhao et al., 2024) significantly reduces the number of parameters needed for storing the optimizer states by only keeping track of the optimizer state in a low-rank projection, which is then projected up to be applied to the model weights. Combining this method with quantization would further shrink the footprint of the model. However, updating the weights of a highly quantized model directly in low-precision space has not been shown to work. This is mainly due to the higherprecision gradient updates being too small to have an impact on the lower-precision quantized states. Lastly, while LoRA is memory efficient for parameter-efficient fine-tuning of pre-trained models, it does not work as a pretraining method by itself (Lialin et al., 2023) .To address these shortcomings, we propose a new method, LoQT. LoQT initializes two low-rank factors for each (1) Low-rank factors P and B are periodically initialized from the gradient of the dequantized model weights ∇W , (2) then only B is trained while P q and W q are kept quantized and frozen, over an exponentially increasing interval until T i , (3) the low-rank factors are merged back into the quantized model. The process is repeated until training halts.weight matrix W : 1) P , initialized using a projection of W 's gradients into a low-rank subspace, and 2) B, which is initialized to minimize the quantization error. B is the only directly trained matrix, which means that the optimizer state can be shrunk significantly. The product P B is then periodically merged into the full rank matrix W with exponentially increasing scheduling. As W and P do not receive gradient updates, they can be quantized, thus optimizing memory usage even further. The large accumulated updates make it possible to update a quantized model, as the addition of smaller changes would not register in the quantized state. A high-level overview is given in Fig. 2 .We show that LoQT works well with and without a quantized model, enabling not only a lower memory footprint in the optimizer state but also over the model parameters.Our results show that we get competitive performance to prior methods using significantly less memory, in particular when using quantization. We also demonstrate superior performance when fine-tuning pre-trained models by training and evaluating on the GLUE (Wang et al., 2018a) benchmark for natural language understanding. Finally, we ablate several properties of the suggested approach, and we find that an exponentially increasing projection gap is beneficial, not only to our work but also for prior work (Zhao et al., 2024) . This is particularly crucial for the training of quantized models. LoQT enables efficient training of 7B models on consumer-grade hardware with 24GB of memory, and makes it feasible to train models with up to 13 billion parameters, without model parallel or by making use of per-layer gradient updates (Lv et al., 2023) on the same hardware as shown in Fig. 1 . 
neural network quantization and nf4: 
 Quantization compresses neural networks by converting high-precision values into lower-precision formats, significantly reducing storage requirements (Zafrir et al., 2019; Shen et al., 2019; Bai et al., 2021; Dettmers et al., 2022a) .The process involves taking a datatype of high precision, such as 32-bit, requiring 4 bytes of memory, and converting it into a representation with increasing rounding errors but lower memory cost. In this work, we use NF4 quantization (Dettmers et al., 2023a) . Since it is a 4-bit code, it can only represent 2 4 different values. NF4 works by first normalizing values onto the interval [-1 : 1]. These are then discretized onto quantiles of the normal distribution, (q i ) 16 i=1 (see (Dettmers et al., 2023a) for details). The elements of a layer are divided into blocks of 64 weights. Each block β has a scaling factor M β = max w∈β |w 32 |.EQUATIONEQUATIONWe provide an overview of different categories of quantization techniques, and how they relate to LoQT, in Appendix A. Compared to prior approaches, LoQT retains the benefits of reduced memory usage while minimizing accuracy loss, using high-precision updates on a low-rank representation. This allows for efficient model updates without the overhead of full matrix storage and re-quantization.Low-Rank Adaptation (LoRA) (Hu et al., 2021) enables fine-tuning of pre-trained models using low-rank adaptors, effectively reducing the memory footprint by only training weight adaptors for targeted layers. However, simple low-rank training using LoRA factor matrices has not been shown to work for pre-training (Lialin et al., 2023) .LoRA employs trainable low-rank matrices A and B that are used to update W following W t = W t-1 + AB, where W t-1 is frozen to enable precise adjustments within a lowrank framework. Extending this, LoRA+ (Hayou et al., 2024a ) applies different learning rates to B compared to A to optimize tuning. LoRA-FA (Zhang et al., 2023) further simplifies the process by training only B, eliminating the need to store full-rank activations for efficient finetuning. DoRA (Liu et al., 2024) divides the LoRA adapter into independent magnitude and direction components, providing granular control over weight adjustments. PISSA initializes A and B using SVD of W for faster convergence (Meng et al., 2024) and does not use gradient information ∇W . Since LoRA only trains A and B and keeps W fixed, QLoRA (Hu et al., 2021) explore quantizing W . They fine-tune a quantized model q(W ) = W q with 4-bit precision using randomly initialized 16-bit precision factors A and B. To address quantization errors E = |W q -W |, low-rank factors of the quantization error E have been used (Li et al., 2023) .LoQT extends LoRA to both pretraining and fine-tuning. Unlike traditional LoRA, LoQT uses A and B to refine W throughout training, with A initialized from W 's gradient projection and B trained along this gradient path. LoQT also incorporates quantization and targeted optimization iterations similar in spirit to LoftQ (Li et al., 2023) , correcting for quantization errors in W q , thus better aligning it with the original non-quantized W .Optimizer memory consumption A significant portion of the memory needed to train neural networks is typically consumed by optimizer states. Notably, Adam (Kingma & Ba, 2017) , one of the most widely used optimizers, uses double the amount of memory as the gradient matrix to maintain first and second-order gradient statistics. Efforts to reduce this overhead have led to the development of adaptive optimization algorithms like Adafactor (Shazeer & Stern, 2018) , which achieves sub-linear memory costs by factorizing the second-order statistics into a row-column outer product. Ga-Lore (Zhao et al., 2024) expands on this concept by using low-rank factorization and projecting low-rank gradients up to a full-rank when updating model weights.Periodic updating of weight matrices ReLoRA (Lialin et al., 2023) combines low-rank updates with initial fullrank training. They find that doing one-third of the training in full-rank, and the subsequent two-thirds in low-rank (see §2.2) results in comparable performance to standard training methods.Low-rank gradients GaLore (Zhao et al., 2024) , focuses on the structure of the gradients, projecting them into a low-rank space using factors P and Q, which are derived from a truncated singular value decomposition (SVD) of the weight matrix gradient, G W ≈ P r Σ r Q r . This reduces memory costs associated with storing the optimizer states and aligns with findings from recent studies which suggest that learning primarily occurs within a low-dimensional subspace at a given time (Larsen et al., 2022; Gur-Ari et al., 2018) . This can be further combined with applying per-layer gradient updates, reducing the memory needed for storing the gradients for the full model at once (Lv et al., 2023) .LoQT builds on GaLore's gradient projection ( §3.1) to initialize LoRA factors while updating the full matrix following a schedule inspired by ReLora, while only training one low-rank matrix per layer. We achieve comparable quality to GaLore and better performance than ReLoRA while reducing tunable parameters and memory usage compared to both approaches. 
efficient pretraining with loqt: 
 LoQT works by initializing and training low-rank adapters obtained by taking the SVD of a given layer's gradients. Let W indicate the full weights matrix of a given layer, P be the left factor constructed from the SVD decomposition of the gradients matrix: ∇W = U ΣV ⊤ ; i.e. P consists of the first r columns of U corresponding to the singular vectors with the r largest singular values of W . The update rule for an interval [T i-1 , T i ] is then given by W Ti = W Ti-1 + P B, where only the weights of B are updated. P and W Ti-1 are kept constant over the time interval. We describe this in more detail below, followed by a discussion on periodic updating of the factor P , the enabling of quantized pre-training, error compensation, and exponential update intervals. Pseudo-code for LoQT is shown in Fig. 3 .Zhao et al. (Zhao et al., 2024) show that gradients exhibit a low-rank structure during training. They exploit this insight by projecting the gradient to a low-rank subspace and applying the Adam optimizer before projecting back to the original dimensions. By doing this, the memory-intensive optimizer states required by Adam are shrunk significantly for low enough ranks. Definition 3.1 (Gradient Low-rank Projection, def. 3.4 in (Zhao et al., 2024) ). Gradient low-rank projection (GaLore) denotes the following gradient update rules, where η is the learning rate, ρ is the Adam optimizer, and W ∈ R m×n is the weight matrix being trained, and T represents the total number of training iterations between recomputing the projection matrices:W T = W 0 +η T -1 t=0 Gt , where Gt = P t ρ t (P ⊤ t G t Q t )Q ⊤ t ,where P t ∈ R m×r and Q t ∈ R n×r are are the top-r singular vectors from the SVD decomposition of the gradient matrix at each iteration t. In practice, this can be approximated by only applying a one-sided projection, as inEQUATIONGaLore demonstrates that it is sufficient to keep the projection matrix fixed and only update it once every T iterations, which we use in the following.We now describe the process by which we initialize the parameters we optimize in LoQT. We adopt the memoryperformance trade-off of using only a one-sided projection.We compute P ⊤ G if m ≤ n and GQ otherwise. We want to achieve a separation between trainable weights and static weights, which we achieve by rewriting GaLore in terms of low-rank adaptors. Assume, without loss of generality, that m ≤ n. Using the fact that P t is fixed in the interval [0, T ] we have thatEQUATIONFrom ( 4) it is clear that we can keep track of low-rank updates using rank-r low-rank adaptors. We note that in the interval [0, T ] only B is updated, creating the desired separation. If implemented directly, we would need to compute the gradient with respect to W and then project it down using P ⊤ G t . We find that this step is unnecessary; it is sufficient to train B using standard gradient descent.We now show that training the B matrix using gradient descent is equivalent to training w.r. B = (xP ) ⊤ ∂L ∂y = P ⊤ x ⊤ ∂L ∂y = P ⊤ t G w. This shows that calculating the gradient w.r.t B gives the same as projecting the gradient w.r.t W . It is thus clear that GaLore's low-rank gradient updates should be the same as those obtained using backpropagation through LoRA.Previous work has shown that training low-rank weight matrices works well for fine-tuning pre-trained weights. However, it has been shown that training low-rank factors, and periodically merging them into frozen W , does not work when starting with a randomly initialized matrix (Lialin et al., 2023) . Here we address this shortcoming to enable full training using low-rank weight matrices.Inspired by prior work (Lialin et al., 2023; Zhao et al., 2024) , we periodically update a given layer W T +1 = W T + P T B T at fixed steps T ∈ T . This approach allows W to evolve as a sum of low-rank matrices aligning with GaLore's strategy of updating the gradient subspace during training:W t = W 0 + ∆W T1 + ∆W T2 + . . . + ∆W Tn , where t = |T |i=1 T i and ∆W Ti = P Ti B Ti represents the product of the learning from B during the interval T i -T i-1 scaled by the learning rate η and modulated by the gradient projection matrix P Ti . After each update at iteration T i ∈ T , we reinitialize the low-rank factors P T and B T . As in (Zhao et al., 2024) , we compute the gradient of W T over a single batch, focusing only on ∇W T without needing full optimizer states. Not requiring optimizer states reduces the memory increase compared to full-rank training.With the updated W t and reinitialized P t and B t , a new gradient subspace is established for exploring the next T i+1 -T i steps. Our method treats W t as the full-rank repository of accumulated updates. Although it is periodically updated, W t is not part of the optimizer state computations, and the gradients during the single forward pass are offloaded to cpu. Since the SVD calculations are done layerwise only the current layer is needed on GPU, or the SVD can be calculated on CPU. P t defines the general gradient subspace and trajectory for the upcoming T i+1 -T i steps, and B t is adjusted to navigate optimally within the direction set by P t . As only B t is trained, the number of parameters needing optimizer states is drastically reduced.Given that B is the only matrix accumulating gradients and undergoing changes, the other matrices W and P can be kept quantized. This approach allows storing the weights in NF4 precision without requiring high-precision gradient and weights to update W and P . This approach reduces the memory required for these matrices by almost a factor of 4. To the best of our knowledge, we are the first to enable efficient 4-bit quantized training using gradient descent without storing the weights in full precision.We quantize weights q NF4 (W ) = W q and q NF4 (P ) = P q as described in §2.1. During periodic updates at interval time steps ( n i=1 T i ) max n=1 , P q and W q are dequantized using the inverse function, P BF16 = q -1 NF4 (P NF4 ) and W BF 16 = q -1 NF4 (W NF4 ). After this, W Ti = W Ti-1 + P Ti-1 B Ti-1 is computed and quantized. The quantization and dequantization processes are applied layer by layer, ensuring that not all layers are simultaneously in a non-quantized state to reduce memory usage. Moreover, the quantization state itself is re-quantized for further efficiency following (Dettmers et al., 2023a) . We implement LoQT using weight-only quantization, meaning the quantized weights are loaded into memory and then dequantized before computing the matrix multiplications.As the quantization process inevitably leads to a discrepancy between the non-quantized and quantized versions of W we wish to reduce this effect as much as possible. While compensating for quantization errors has been done before (Li et al., 2023) , we need a tailored solution for LoQT.During the merging phase, we first dequantize to obtain W T -1 and P T -1 , and then compute the updateW T = W T -1 + P T -1 B T -1. This is immediately followed by re-quantizing to get Q T = q NF4 (W T ). Our goal is thus to minimize the quantization error ∥(Q T + P T B T ) -W T ∥ F . The projection matrix P t is calculated as described in section 3.1, whereas we are free to choose B t to minimize the quantization error. To achieve this, we solve for B T , givingB T := P + T (Q T -W T ), where P + T is the Moore-Penrose pseudo-inverse. Inspired by (Li et al., 2023) we then iteratively refine B T , by recomputing Q T = q NF4 (W T -P T B T ) and recomputing B t , improving the alignment between the full-precision W and its quantized state.As training advances and the learning rate decays, the magnitude of the update B T -1 to form W T decreases. This leads to negligible differences between |q (Q t + P t B t ) -Q t |, which results in the weights plateauing early, as depicted in Fig. 4a . To address this, we implement an exponentially increasing scheduler for updating W . Drawing from GaLore's observation on the exponential decay of gradient rank (Lemma 3.1 (Zhao et al., 2024 )), we start with a frequency gap τ and progressively increase the update intervals by a factor of ψ. The sequence of updates is then given by (T i ) ∞ i=0 = (τ + ψ i ) ∞ i=0 Each T i marks aG W ← ∇ W L(W ) 2: W Q , P Q , B ← Initialize(W, G W ) 3: for each t in training steps do 4: if t ∈ T then 5: W ← W Q + s • P Q • B t 6: G W ← ∇ W L(W ) 7: W Q , P Q , B t ← Initialize(W, G W ) 8: else 9: B t+1 ← B t -ρ(G B t ) 10:end if 11: end for 12: return θ Algorithm 2 Initialization Procedure 1: Initialize(W, G W ): 2: U, S, V T ← SVD(G W ) 3: P ← U [:, : r] {First r singular vectors} 4: P q ← q N (P ) 5: B ← 0 6: Ŵ ← W 7: for each c in compensation steps C do 8: step t when W is updated. This scheduling ensures more frequent updates earlier in training and more well-spaced adjustments later, allowing more accumulated gradients before each update.Q c ← q N ( Ŵ ) 
experimental setup: 
 We evaluate LoQT by training LLaMA-based (Touvron et al., 2023) language models on the C4 dataset (Raffel et al., 2019) , a collection of processed text in English that was scraped from the internet (Raffel et al., 2019) . We train models of sizes of 60M, 130M, 350M, and 1B parameters, adhering to single-epoch training cycles determined by Chinchilla Scaling Laws (Hoffmann et al., 2022) . While LoQT is capable of training models up to 13 billion parameters on consumer GPUs, compute limits prevent us from training to convergence for sizes above 1B. We also benchmark LoQT on the GLUE test-suite for natural language Baselines For pre-training, we compare LoQT against LoRA (Hu et al., 2021) , ReLoRA (Lialin et al., 2023) , GaLore (Zhao et al., 2024) , and non-quantized version of LoQT, LoQT-nq. In our experiments, LoQT, LoRA, and ReLoRA modify attention and fully connected layers while maintaining full-rank embeddings and normalization layers. This contrasts with GaLore, which keeps weights full-rank but projects gradients to low-rank, and standard full-rank training. For fine-tuning, we benchmark LoQT against Ga-Lore, LoftQ (Li et al., 2023) , LoRA and LoQT-nq. All models use identical update frequencies for GaLore, ReLoRA, LoQT-nq, and LoQT, starting with an update frequency of T = 100 and then with exponentially increasing update frequencies. This means more frequent updates early and fewer as the model stabilizes (see Section 4b for details). All models are trained using the Adam optimizer, except GaLore which uses GaLoreAdam for gradient projection.Results and details for pretraining of language models of sizes 60M, 130M, 350M and 1B parameters are shown in Table 1 . Model sizes are calculated based on the full models without any low-rank methods. We see that LoQT and LoQT-nq both perform very close to full rank pretraining and GaLore, while using significantly less memory by keeping most of the model weights in a quantized state. For the 60M model, full training is only slightly better than LoQT, while we see results improve or be within the standard error for the other sizes. We also notice a slight drop in performance from quantizing the original weight matrix, comparing LoQT and LoQT-nq. The key difference between the approaches are the theoretical memory estimates, e.g. where LoQT requires 59% less memory for the 1B model in full precision and 28% less memory than GaLore.We fine-tune the pre-trained DeBERTa-V3-base1 (He et al., 2023) model on GLUE tasks using LoQT and compare its performance with a full fine-tuning baseline, LoRA, LoftQ, and GaLore. See Appendix 5 for details on hyperparameters.Results are given in Table 2 .We find that both LoQT-nq and LoQT perform well. Somewhat surprisingly, it sometimes surpasses GaLore, LoftQ, and LoRA.This may indicate that initializing the LoRA factors with information about the gradient of W could be a beneficial starting point compared to standard initialization methods. As the goal of this work is to limit memory consumption, we leave out further comparisons that could verify these findings to future work. Memory usage An overview of memory usage for Ga-Lore, LoRA and LoQT is given in Table 3 . We see that LoQT makes use of the same number of trainable parameters as LoRA for a given rank while using less memory for the optimizer states and gradients than in both LoRA and GaLore.We compare the LoQT to the closest in memory performance approach, GaLore, for 13B in Figure 1 , and for other model-sizes in Figure 6 . We compare three different use cases, using the approaches directly, combining them with an 8-bit Adam optimizer (Dettmers et al., 2022b) , and using per-layer weight updates with offloading (while still using 8-bit Adam). We see from the figures that LoQT significantly shrinks both the number of trainable parameters and optimizer states compared to GaLore.Per-layer weight update is essential for GaLore; without it, an additional ∼12 GB of VRAM is needed for gradients in a 7B model, making full-parameter fine-tuning impossible on a 24GB GPU. Additionally, the per-layer gradient updates may not work well with DDP (Distributed Data Parallel) and gradient accumulation. With our method, we can get a lower memory than GaLore even when they use per-layer gradient updates. When not using per-layer gradient updates, this difference becomes even bigger as seen for the 7B model in Figure 6 . This allows us to increase the batch size during training on multi-GPU setups, leading to speed improvements when training larger models. We note that the memory required for storing gradients can be reduced in GaLore by doing gradient accumulation in the low-dimensional space and only projecting back once right before adding the gradient update to the weight matrix. The memory consumption of this approach would fall somewhere between Galore with and without per-layer updates but does still not allow for quantized training.Moreover, our method supports training 7B models without per-layer computations on 24GB GPU. This makes it possible to use multi-GPU training with gradient accumulation, a capability not possible with the current GaLore approach.Our memory advantage allows for a batch size of 1280 tokens compared to GaLore's 256 for the 7B model on the 24GB RTX3090. With per-layer gradient updates, LoQT can train a 13B model on a single GPU, pushing the limits of hardware efficiency.Throughput We evaluate the throughput with a sample batch size of 16 with a total batch size of 512 using gradient accumulation, which is the largest power of 2 that fits on the GPU. We update the projection matrix P every 200 iterations. The per-layer gradient update algorithms apply a weight update for every mini-batch as they do not support gradient accumulation. For evaluation, we use a 1B parameter model with rank 512. We find that LoQT can process 16% fewer tokens per second than only using AdamW, at 3996 tokens/s compared to 4782 tokens/s on the RTX3090. 
ablations quantization error compensation and initialization: 
 To assess the impact of quantization error compensation, we analyze the validation loss curves for a 130 million parameter model. Figure 4a shows that quantizing W or both W and P without error compensation, or exponential frequency updates, causes the loss to stagnate early. We also note that quantizing P has a much smaller effect on the loss compared to quantizing W . Error compensation significantly improves the model's performance, resulting in approximately 3.5 points better perplexity. Adding exponentially increasing update frequency improves perplexity by an additional 1.5 points, achieving performance close to that of models without quantization. Figure 4 : Ablation results for update frequency, error-compensation, quantization, model size 130m, and rank 256. §3.5, LoQT's performance stagnates earlier and diverges more from the other models. This demonstrates the effectiveness of our compensation approach in mitigating the quantization errors introduced during the W update with AB and subsequent quantization steps.Projection update frequency Our scheduling approach ensures frequent updates early in training for substantial weight adjustments, with decreasing frequency as training progresses. This allows for larger updates to compensate for smaller ones canceled out by quantization errors. Figure 4b presents an ablation study on our method of progressively increasing update frequency starting at 100 and increasing by a factor of 1.2 T up to 2500. We show validation loss curves for fixed update frequencies 200, 400, 500, and 1000.The results show that exponentially increasing the update gap is particularly beneficial for models employing quantization, enabling them to achieve the same perplexity as those without quantization while making use of GaLore. Conversely, the performance gains are more subtle for models that do not use quantization and rely solely on GaLore. This could be due to the reduction in the accumulation of errors from frequent updates of the projection factor P , as the influence of outdated optimizer statistics becomes less prevalent. Finally, an ablation on the ranks used for P and B is given in Figure 5 in the Appendix. 
discussion and conclusion: 
 We present LoQT, a method for memory-efficient pretraining and adaptation of quantized models. The key insights behind the approach are the benefits of initializing two lowrank factors using the gradient of the weight matrix and us-ing exponentially increasing update gaps. By training only one low-rank factor in higher precision, we keep the other factor and the main weight matrix frozen and quantized. We then periodically merge in updates from the trainable factor into the quantized original matrix, allowing the frozen matrix to store updates while the trainable factor guides optimization based on gradient information.While our initial goal was to lower memory usage to facilitate the training of models such as LLMs on consumer-grade hardware, we are cautiously excited about the results sometimes being better than the baselines. These evaluations will be explored in more detail in future work.Our method is general and opens up new ways of decreasing memory use as well as improving the training throughput. This could be done by implementing kernel fusion and using other quantization methods such as NF2 (Dettmers et al., 2023a) or quantization of activations, making it possible to do the matrix multiplications using modern tensor core formats such as FP8 or INT4. 
impact and limitations: 
 Our work has the potential to have a significant impact on those working in hardware-constrained settings by enabling more efficient training on consumer hardware. We are particularly excited to see the method being applied in single GPU settings. We validate LoQT on several model sizes, by training over many steps and by fine-tuning on a standard benchmark for natural language understanding. While we are confident in our results, further exploration of training duration, data diversity, and hyper-parameter tuning might lead to different results in those settings.where ψ is 1.2 and i is the update counter (see Section D.1 for more details). We tested learning rates of 0.01, 0.005, 0.001, and 0.0005 across different model sizes. For models ranging from 60M to 350M parameters, a learning rate of 0.01 yielded the best performance. In contrast, full-rank models required smaller learning rates: 0.001 for 60M to 350M models and 0.0005 for the 1B model. To scale the learning rates for LoQT, LoQTnq, and Galore, we employed a scale parameter s set to 0.5 and 0.25 for the 1B model. This parameter functions similarly to the LoRA alpha parameter, determining the weight on the learned factors for LoQT and LoQT-nq. For Galore, our experiments indicated that s = 0.5 was more effective than the 0.25 reported in (Zhao et al., 2024) . This scaling approach effectively adjusts the learning rate, resulting in an actual rate of 0.005 for the multi-head attention and feed-forward layers in LLaMA models, which is relatively large compared to the 0.001 used for full-rank models. Higher learning rates led to spikes in the training loss for both full-rank and LoQT models.  
 
 Appendix: We test learning rates in the range of 1 × 10 -5 to 5 × 10 -4 . For LoQT LoftQ, we employed normal float NF4 quantization and performed 5 iterations of optimizing the error of quantization. We used a batch size of 32 and a maximum sequence length of 256. Table 5 summarizes the detailed hyperparameters for tasks in GLUE using the DeBERTaV3-base model. We use a fixed projection gap of 2400 for all runs.Table 5 : Hyperparameter setup for LoQT-nq, LoQT, LoftQ (Li et al., 2023) , LoRA (Li et al., 2023) , and Galore across various tasks on the GLUE benchmark. 1 × 10 -4 5 × 10 -5 5 × 10 -5 1 × 10 -4 5 × 10 -5 5 × 10 -5 1 × 10 -4 5 × 10 -5LoRA, Galore # Epochs 10 30 30 30 30 30 30 30 Learning Rate 1 × 10 -5 2 × 10 -5 1 × 10 -5 2 × 10 -5 1 × 10 -5 2 × 10 -5 2 × 10 -5 3 × 10 -5Figure 5 shows the validation perplexity versus training steps for various ranks using LoQT-nq and LoQT on a 130 million parameter model over 20,000 iterations. All models employ an exponentially increasing update frequency starting at 100, with a factor of 1.2 Ti . The results demonstrate that both the quantized (LoQT) and non-quantized (LoQT-nq) models follow a similar trajectory for ranks ranging from 64 to 512. However, for the smaller rank of 64, there is a slight divergence between LoQT-nq and LoQT, indicating a limit to how low the rank can be while maintaining accuracy with quantization. This plot highlights the tradeoff between rank and perplexity, suggesting that while our method supports low-rank training, there is a minimum rank threshold needed to achieve results comparable to regular pre-training.Figure 6 demonstrates that LoQT requires less memory than GaLore and Adam, even without using per-layer gradients (Lv et al., 2023) or Adam 8-bit (Dettmers et al., 2022b) . The gap between LoQT and the baselines increases with larger model sizes. The configurations and ranks for each model are shown in Table 4 . With LoQT and Adam 8-bit, it is possible toFrom https://huggingface.co/microsoft/ deberta-v3-base. 


Decision: ## Strengths
* This paper is well written and the methods are clearly explained and motivated
* The paper contains both extensive experiments and a theoretical justification
* This approach enables training a 13B parameter on a single 24GB GPU, which has a high potential
  for a strong impact in the community

## Weaknesses
* Some specific points could be explained more clearly
* This approach adds a new hyper-parameter for projection update frequency, and it is unclear what the best
  value is. Maybe a dynamic approach would be more generalizable to other contexts/neural networks

The general sentiment is very largerly positive, I recommend acceptance as an oral presentation.