Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients

## Abstract

Abstract Training Large Language Models (LLMs) is memory-intensive due to the large number of parameters and associated optimization states. GaLore zhao2024galore , a recent method, reduces memory usage by projecting weight gradients into a low-rank subspace without compromising performance. However, GaLore relies on time-consuming Singular Value Decomposition (SVD) operations to identify the subspace, and the frequent subspace updates lead to significant training time overhead. Moreover, GaLore offers minimal improvements in accuracy and efficiency compared to LoRA in more accessible fine-tuning scenarios. To address these limitations, we introduce Q-GaLore , a novel approach that substantially reduces memory usage by combining quantization and low-rank projection, surpassing the benefits of GaLore. Our method is based on two key observations: (i) the gradient subspace exhibits diverse properties, with some layers converging early in training while others are subject to frequent changes; (ii) the projection matrices are highly resilient to low-bit quantization. Leveraging these insights, Q-GaLore adaptively updates the gradient subspace based on its convergence statistics, achieving comparable performance while significantly reducing the number of SVD operations. We maintain the projection matrices in INT4 format for aggressive memory conservation and preserve weights in INT8 format, incorporating stochastic rounding to capture accumulated gradient information. This approach enables a high-precision training trajectory using only low-precision weights. We demonstrate that Q-GaLore achieves highly competitive pre-training and fine-tuning performance with exceptional memory efficiency. At pre-training , Q-GaLore facilitates training a LLaMA-7B model from scratch on a single NVIDIA RTX 4060 Ti with only 16 GB memory , showcasing its exceptional memory efficiency and practicality. At fine-tuning , it reduces memory consumption by up to 50% compared to LoRA and GaLore, while consistently outperforming QLoRA (by up to 5.19 on MMLU) at the same memory cost. Codes are available at https://github.com/VITA-Group/Q-GaLore .

## 1 Introduction

Since the 2020s, Large Language Models (LLMs) have demonstrated remarkable performance in various disciplines ; ; ; ; ; . However, the immense scale of LLMs, often comprising billions of parameters, presents a formidable challenge for most research groups in terms of training and full fine-tuning. For example, Meta’s LLaMA models were developed with 2048 A100-80GB GPUs for approximately a period of 5 months . Even without factoring in any considerations for product efficiency, pre-training a LLaMA 7B model from scratch with a single batch size necessitates a minimum of 58 GB memory. This breakdown comprises 14 GB for trainable parameters, 42 GB for Adam optimizer states and weight gradients, and 2 GB for activation .

Numerous research efforts have been dedicated to alleviating the substantial costs associated with training LLMs. These endeavors encompass a range of techniques, including small-scale LLM designing ; , efficient scaling optima , training methodologies incorporating sparsity ; ; , sparse model training approaches ; , and low-rank training strategies ; . Among these, GaLore  has emerged as a notable contender, enabling the full-parameter training of LLMs through low-rank gradient updates achieved via Singular Value Decomposition (SVD). Leveraging its low-rank characteristics, GaLore offers a significant reduction—up to 63.3%—in total training memory requirements, facilitating the training of a 7B model with a mere 24GB of memory.

Although GaLore offers substantial memory savings, its 24GB memory requirement still surpasses the available resources in many customer devices. For instance, popular laptop GPUs like the RTX 4060 Ti are equipped with up to 16GB of memory. This limitation raises the question of how we can further reduce the memory footprint of low-rank LLM training to make it accessible to a wider range of hardware configurations. Also, GaLore requires regular updates to the gradient subspace through computationally expensive SVD operations (e.g., every $200$ iterations) to approximate the training trajectory of full-rank training. The computational complexity of SVD operations is roughly on the magnitude of $O(mn^{2})$, where $m$ and $n$ are the dimensions of the matrix. As a result, it takes $\sim 10$ minutes for the LLaMA-7B model to update the subspace, leading to significant training latency.

Figure: Figure 1: Comparison of data types and training flows of different methods. We by default use 8-bits Adam as the inner optimizer. Note that the gradient in GaLore and Q-GaLore is not persistent during training, following the same strategy in ; .
Refer to caption: x1.png

To address these challenges, we delved into the training dynamics of the gradient subspace of GaLore and discovered two intriguing phenomena: (i) The gradient subspace of GaLore demonstrates different behaviors across different layers, in which some layers demonstrates "early bird" properties and converge within the initial training stage while some layers have a stable subspace within a specific window during training and some other layers consistently keeps changing. (ii) The projection matrices of GaLore exhibit excellent quantization-friendliness property, which can be seamlessly quantized to 4-bits without sacrificing training quality.

Inspired by these observations, we propose Q-GaLore, a novel approach that enables the training of large language models with low-precision weights and low-rank gradients. Q-GaLore introduces two modules to reduce memory overhead and training latency:

(i) Low precision training with low-rank gradients: We manage to quantize the entire model (not only the optimizer state as in GaLore ) to 8-bits and the projection matrix to 4-bits, as shown in Figure 1. By utilizing low-precision weights and projection matrices, our approach achieves a reduction of approximately 28.57% in memory requirements for gradient low-rank training where the weight represent the primary component of memory usage post low-rank projection. Additionally, to maintain training stability and approximate the trajectory of high-precision training, we implement Stochastic Rounding (SR)  that provides an unbiased estimation of the gradient trajectory and mitigates gradient information loss, thus enhance the training stability and overall performance.

(ii) Lazy layer-wise subspace exploration: We monitor the convergence levels of the gradient subspace in different layers and adaptively decrease the frequency of SVD operations for the layers whose low-rank subspace does not change significantly over time. This approach reduces the training time associated with SVD, saving over 32 hours for training a 7B model.

We demonstrate the efficacy of Q-GaLore in both pre-training and fine-tuning scenarios. For pre-training, Q-GaLore’s efficiency allows us to reduce the memory requirements of full-rank training and GaLore by 61% and 30%, respectively, across various model sizes from 60M to 7B. Notably, Q-GaLore demonstrates the feasibility of training LLaMA-7B on a single NVIDIA RTX 4060 Ti with only 16GB of memory, achieving performance comparable to its full-rank counterpart. In the context of fine-tuning, Q-GaLore matches the performance of SOTA low-rank approaches including LoRA , QLoRA and GaLore . It reduces memory consumption by up to 50% than LoRA/GaLoRe, while consistently outperforming QLoRA  at the same memory cost.

## 2 Related Work

### 2.1 Low-Rank Adaptation and Training

Optimizing Large Language Models (LLMs) requires a substantial memory footprint to accommodate weights, activations, gradients, and optimization states. Low-Rank Adaptation (LoRA) is a notable technique that introduces low-rank weight adapters for each layer, reducing the memory footprint by only optimizing the adapters, which can later be merged back into the original model. Subsequent enhancements to LoRA, such as quantization , multi-task learning support , and various architectural improvements ; ; ; ; ; ; ; ; , have all focused on fine-tuning scenarios.
Despite the efficiency of low-rank adaptation, its suboptimal performance compared to full parameter optimization has motivated the development of other memory-efficient optimization methods. For instance, ; reduce memory overhead through fused backward operations, eliminating the need to store all weight gradients. Sparse optimization techniques, such as BAdam and LISA , partition parameters into blocks or sample layers based on importance to minimize memory costs while maintaining performance comparable to full parameter fine-tuning.

Early efforts to adapt LoRA for pre-training, such as ReLoRA , still require full-rank learning in the initial stages, resulting in high memory overhead. Recently, GaLore leverages the low-rank properties of gradients to enable full-parameter learning while significantly reducing memory usage during optimization. This approach allows GaLore to achieve better performance than common low-rank adaptation methods such as LoRA, while still being memory-efficient.

### 2.2 Low Precision Training

Low-precision training aims to improve training efficiency by storing data in low-precision formats and leveraging low-precision General Matrix Multiplication (GEMM) operations. This is distinct from post-training quantization, which primarily enhances the inference efficiency of pre-trained models. A significant challenge in low-precision training is potential instability during the training process. SWALP addresses this issue using stochastic weight averaging , but it requires maintaining averaged weights, leading to high memory overhead in large foundational models. Other methods handle instability by scaling gradients or second-order optimizer statistics .

While various low-precision training methods have been explored for smaller-scale convolutional networks ; ; ; ; ; , they are generally not applicable to training large-scale transformers, as large tensors are less suitable for quantization . Some approaches to low-precision training at a larger scale still require maintaining high-precision latent weights during training, significantly increasing memory consumption for large language models ; . This study aims to improve the end-to-end memory efficiency of training large-scale foundational model at scale.

## 3 Methodology

We first introduce the data type and quantization basics in Section 3.1. Section 2 demonstrates the adaptive convergence properties of the gradient subspace, which facilitates efficient training. In Section 3.3, we demonstrate the high tolerance of the projection matrix to quantization. Section 3.4 then discusses stochastic rounding for approximating high-precision training trajectories. The overall pipeline of Q-GaLore is depicted in Figure 4.

### 3.1 Preliminaries on Quantization

Generally, quantization methods are categorized into Post-Training Quantization (PTQ), where quantization is applied to pretrained models without further training; and Quantization-Aware Training (QAT), which incorporates quantization throughout the training process. QAT aims to either generate more quantizable models for faster inference or expedite the training process through low-precision operations. To preserve performance, these methods retain high-precision parameters throughout the training process and apply quantization to transfer the parameters into low-precision data formats during each forward and backward pass. Maintaining high precision parameters occupis massive memory and results in even larger memory requirements than vanilla high precision training. In this work, we focus on improving the memory efficiency of training large language models and do not maintain the high-precision parameters.

In Q-GaLore, the model weights are retrained in INT8 while activations and gradients are computed in BFloat16. Although FP8  offers greater expressiveness than INT8, it is supported only on limited hardware devices,
e.g., the NVIDIA Hopper series GPUs, which are costly and not widely available. Thus, we employ the more general INT8 formats. The pseudocode is presented in the appendix A. To convert data precisions, we utilize block-wise uniform quantization :

$$ $W_{q}=\mathrm{Quant}_{n}(W,s,z)=\mathrm{clamp}(\lfloor\frac{W}{s}\rceil+z,-2^{ n-1},2^{n-1}-1)$ $$

where $W$ and $W_{q}$ represents the original and quantized tensors, respectively. $s$ is the scaling factor and $z$ is the zero point. Both $s$ and $z$ are calculated within each block of the tensors. $n$ is the quantization bits. We default to use block size of $256$ in all implementations.

### 3.2 Layerwise Convergence Behaviors of Gradient Subspace

Figure: Figure 2: Cosine similarity between the adjacent projection matrices captured every 250 training iterations.
Refer to caption: x2.png

GaLore relies on a fixed interval to recompute the gradient space and projection matrices blindly, assuming that the training dynamics of all the layers in LLMs remain the same. One direct implication remains the frequent computation of computationally expensive SVD. To this end, we ask: How does the gradient subspace dynamics varies during the pre-training of LLMs? We investigated the cosine similarity across the projection matrices obtained at regular interval during the pre-training of LLaMa-130M as shown in Figure 2. Our observations are as follows: (i) certain layers exhibit an “early bird” phenomenon, whereby their gradient subspace saturates early during pre-training and remains stable throughout (Top Right, with cosine similarity close to 1); (ii) in some layers, the gradient subspace saturates within a specific window during pre-training (Top Middle); (iii) in other layers, the gradient subspace consistently keeps changing towards the end of training (Top Left).

Figure: Figure 3: Pre-training performance on the LLaMA-130M models. The projection matrices are quantized with different bits.
Refer to caption: x3.png

This observation provides a unique opportunity to monitor the gradient subspace behavior during pre-training and dynamically update the frequency of SVD for each layer if we observe saturation. More specifically, starting with an SVD interval of $t$ for a layer $l$, we monitor the cosine similarity of projection matrices in the previous $k$ intervals. If the cosine similarity across the k intervals remains greater than a threshold (*e.g.,* $\geq$ 40%), we update the interval from ($t\rightarrow 2\times t$) to reduce the compute. This adaptive lazy update can closely mimic the performance of the original GaLore with over 60% reduction in computationally expensive SVD calls. Further ablation studies about the trade-off between SVD calls and performance are presented in Section 4.4.

### 3.3 High Quantization Tolerance of Projection Matrix

The adaptive convergence properties suggest that the projection matrix has a degree of redundancy, indicating that high accuracy is not essential. This observation inspired us to further investigate the functionality of the projection matrix under quantization conditions. We implemented block-wise quantization for the projection matrices, maintaining a uniform block size of 256 across all layers. During these experiments, we ensured that the update steps for the projection matrices remained constant, allowing us to focus exclusively on their quantization characteristics. Figure 3 illustrates the results for the LLaMA-130M models, demonstrating that the projection matrices are highly resilient to quantization, with minimal impact on pre-training quality even when reduced to 4 bits. Based on these findings, we applied quantization to the projection matrices, restricting them to 4 bits. This approach further reduces the memory cost of the optimizer states in low-rank training by 25%.

### 3.4 Approximating High-Precision Training Trajectories Using Stochastic Rounding

When using low-rank training methods such as GaLore, the allocation of memory to maintain model parameters constitutes the majority of the memory overhead. Consequently, we opt to maintain the weights in low precision to enhance memory efficiency during training. The primary challenge of training with low-precision parameters is the significant reduction of gradient information. During each optimization step, the full precision gradient must be quantized to a low precision weight update. However, if the gradient magnitude is not large enough, it will be mitigated via the round-to-nearest scheme. Conventional Quantization-Aware Training (QAT) retains full precision parameters to accumulate small gradient contributions, albeit at the cost of increased memory overhead. To address this issue, we employ Stochastic Rounding (SR) ; ; , that is formulated as the following:

$$ $W_{q}=\mathcal{F}_{SR}(W)=\left\{\begin{array}[]{rcl}\lfloor W\rfloor&&\text{ with probability }p=\lceil W\rceil-W\\ \lceil W\rceil&&\text{with probability }p=W-\lfloor W\rfloor\\ \end{array}\right.$ $$

Under this formulation, the expected value of $W_{q}$ is $E[W_{q}]=\lfloor W\rfloor(\lceil W\rceil-W)+\lceil W\rceil(W-\lfloor W\rfloor)=W$, allowing the low-precision parameters to implicitly accumulate small gradient information. This method achieves comparable performance without the substantial memory requirements associated with maintaining high-precision parameters.

### 3.5 The Q-GaLore  Algorithm

Figure: Figure 4: Illustration of the training flows for Q-GaLore, where the dotted icon denotes intermediate tensors that do not consistently occupy memory.
Refer to caption: x4.png

The pipeline of Q-GaLore is illustrated in Figure 4. The left section of the figure depicts the computation flows, where only the gradients are maintained in high precision to preserve essential training dynamics information. We employ an 8-bit version of the Adam optimizer  as the internal optimizer. During each training iteration, the full-rank gradient is projected into a low-rank format and then incorporated into the optimizer states. To project the gradient into the subspace, we obtain the projection matrix using Singular Value Decomposition (SVD), as described in . The update frequency of the projection matrix is managed through our adaptive update strategy, and the matrix is quantized to 4-bits formats to reduce memory overhead.

Furthermore, after updating the optimizer states, we project the low-rank optimizer states back to full rank and update the parameters. As the weights are consistently maintained at low precision, an additional quantization step is necessary to update the weights. Here, we utilize stochastic rounding to capture the minor-gradient nuances and provide an unbiased estimation of the high-precision weights. Additionally, we employ a fused backward operation as described in ; ; . Upon calculating the gradients for a single layer, we promptly update the corresponding optimizer state and weights, subsequently releasing the memory allocated to the gradients.

## 4 Experiments

In this section, we evaluate the effectiveness of Q-GaLore on both pre-training and fine-tuning tasks. In Section 4.1, we detail the implementation of models, tasks, hyperparameters, and baseline approaches. We then demonstrate that Q-GaLore achieves comparable performance on both pre-training and fine-tuning tasks (Section 4.2). Additionally, Sections 4.3 and  4.4 provide end-to-end memory analysis and extensive ablation studies, respectively.

### 4.1 Implementation Details

##### Network Architecture.

For the pretraining task, we adopt the LLaMA-based architecture with sizes ranging from 60 million to 7 billion, following the setups from ; . During downstream experiments, we select various pre-trained models to evaluate the general effectiveness of Q-GaLore, including RoBERTa  base, LLaMA-3-8B , Gemma-7B , and Mistral-7B .

##### Pre-Training.

We pre-train the LLaMA models on C4 dataset . The C4 dataset is a massive collection of Common Crawl’s web crawl corpus, meticulously filtered and cleaned to ensure high-quality language modeling and training. It is widely used for pre-training large language models due to its diverse and extensive textual content. We train the models on this sufficiently large dataset without data repetition and scales the model size up to 7 billion parameters, a crucial experiment for demonstrating the effectiveness of the proposed methods for practical pre-training.

##### Fine-Tuning.

The downstream tasks cover two categories: (i) GLUE benchmarks , a series of widely used tasks for evaluating the downstream performance of natural language understanding; (ii) MMLU  that evaluates the natural language understanding ability of LLMs, this task covers various domains, including STEM, social sciences, humanities and others.

##### Baselines.

We consider five baseline methods for comparison: (i) Full: Models are trained with the original Adam  optimizer. Both weights, gradients, and optimization states are maintained with full rank and full precision (BF16 format). (ii) Low-Rank: The original weights are factorized into low-rank components: $W=UV$, and $U$ and $V$ are optimized via Adam . (iii) LoRA: LoRA  introduces low-rank adaptors for training the models, $W=W_{0}+UV$, where $W_{0}$ is the pretrained weights, which are frozen during training. We use the initialized weight as $W_{0}$ during pretraining and only optimize $U$ and $V$. And we default to $32$ for LoRA alpha and $0.05$ for LoRA dropout. (iv) ReLoRA: ReLoRA  enhances the original LoRA methods for better pre-training. ReLoRA is a stage-wise LoRA that periodically merges $UV$ into the original $W$ and initializes a new $UV$ for continued training. (v) QLoRA : we use the same hyperparameters: $32$ for QLoRA alpha and $0.05$ for QLoRA dropout. We keep the base models in 8bits for fair comparison. (vi) GaLore : We project the gradient into low-rank format and update the optimizer states. When updating the weight, we project back the low-rank weight update to full-rank. We follow the original hyperparameters, setting the subspace frequency in GaLore to 200 and the scale factor $\alpha=0.25$. The low-rank dimension is chosen as a quarter of the original dimension. Note that no baseline involves quantization, and all data are maintained in BF16 format.

### 4.2 End-to-End Results

**Table 1: Comparison results of various memory-efficient algorithms on pre-training tasks. Experiments are conducted on C4 dataset with LLaMA models. For each experiment, we report both the perplexity and estimated memory. The estimated memory only count for the weights and optimizer states which cost the majority memory overhead. We follow the same settings and collect the results of all baseline methods from , where the training tokens are {1.1B, 2.2B, 6.4B, 13.1B} for {60M, 130M, 350M, 1B} models, respectively.**
| Methods | 60M | 130M | 350M | 1B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Perplexity | Memory | Perplexity | Memory | Perplexity | Memory | Perplexity | Memory |  |
| Full | 34.06 | 0.36G | 25.08 | 0.76G | 18.80 | 2.06G | 15.56 | 7.80G |
| Low-Rank | 78.18 | 0.26G | 45.51 | 0.54G | 37.41 | 1.08G | 142.53 | 3.57G |
| LoRA | 34.99 | 0.36G | 33.92 | 0.80G | 25.58 | 1.76G | 19.21 | 6.17G |
| ReLoRA | 37.04 | 0.36G | 29.37 | 0.80G | 29.08 | 1.76G | 18.33 | 6.17G |
| GaLore | 34.88 | 0.24G | 25.36 | 0.52G | 18.95 | 1.22G | 15.64 | 4.38G |
| Q-GaLore | 34.88 | 0.18G | 25.53 | 0.39G | 19.79 | 0.88G | 16.25 | 3.08G |

#### 4.2.1 Memory-Efficient Pre-training with Q-GaLore

We pre-trained the LLaMA-based models from scratch on the C4 dataset using various memory-efficient methods. The experiments encompassed different model sizes ranging from 60 million to 1 billion parameters, with results reported in Table 1. In each experiment, we report the perplexity values obtained on the validation set. As the primary memory savings are derived from compressing the weight and optimizer states, we provide estimates of the memory overhead associated with storing these components. Detailed discussions on end-to-end memory measurements and throughput comparisons are provided in Section 4.3. For fair comparison, we used the same low-rank dimensions for all the memory-efficient approaches, specifically {128, 256, 256, and 512} for {60M, 130M, 350M, and 1B} models, respectively. And we use 16-bits Adam as the inner optimizer inside GaLore while Q-GaLore implements 8-bit Adam optimizer.

Incorporating adaptive subspace updating, projection and weight quantization, and stochastic rounding, our Q-GaLore method maintains comparable pre-training performance (with less than a 0.84 perplexity increase, compared with the original GaLore approach) while significantly reducing memory overhead. For example, in the experiment of 1 billion model size, training with INT8 weights halved the original memory cost for weights and achieved a 29.68% memory saving against the original GaLore method and a 60.51% memory saving compared to the Full baseline. Compared to GaLore, the additional memory savings primarily come from two sources: (i) INT8 weights require only half the memory overhead of BF16 weights, and (ii) INT4 projection matrices reduce approximately 25% of the memory overhead for optimization states.

**Table 2: Results of pre-training LLaMA-7B model on C4 dataset. Baseline results are obtained from .**
| Steps (K) | 40 | Memory |
| --- | --- | --- |
| Tokens (B) | 5.2 |  |
| 8-bit Adam | 18.09 | 26G |
| 8-bit GaLore | 17.94 | 18G |
| Q-GaLore | 18.83 | 15G |

#### 4.2.2 Q-GaLore Facilitates 7B Model Pre-Training within 16GB Memory

Since the scaling ability of LLMs is a key demand, experiments at the size of 7 billion models serve as an essential part for evaluating the scalability of new architectures or training methods. Thus we pre-trained a 7B LLaMA model from scratch on the C4 dataset. Given the tremendous computational cost of pre-training 7B models, we currently only trained the models for 40K steps, resulting in a higher perplexity than 1B. The models are still under training towards 150k steps and we will update the number once available.

We compared the original 8-bit Adam, 8-bit GaLore (GaLore with 8-bit Adam), and our Q-GaLore method. Note that both 8-bit Adam and 8-bit GaLore only restore the optimization states in 8 bits, leaving the weight and projection matrices in 16 bits while our Q-GaLoremaintains the weight in INT8 and projection matrices in INT4 data format. From Table 2, we can observe that our method achieved matching performance, with a perplexity difference of less than 1. To enhance training stability with low-precision weights, we opted for a reduced maximum learning rate, setting it at 0.004 compared to the baseline’s 0.005. This marginally lower learning rate potentially slows the convergence speed during the initial stages, although the difference in perplexity relative to the baseline is negligible. Notably, our approach not only achieves comparable performance, but requires only around 15GB of memory overhead. This efficiency enabled the pre-training experiments to be conducted on a single Nvidia RTX 4060 Ti, which has a 16GB memory budget.

#### 4.2.3 Memory-Efficient Fine-Tuning with Q-GaLore

Pre-training LLMs is a resource-intensive task that is typically only feasible for large companies or computing centers. In most practical scenarios, memory-efficient fine-tuning of LLMs on specific downstream tasks is more common. To evaluate the effectiveness of Q-GaLore, we selected a diverse set of downstream tasks, including eight tasks from the GLUE benchmark and four subtasks from MMLU, which assess the ability of LLMs to understand natural language. We compared the performance of Q-GaLore with the baseline Full method and three state-of-the-art low-rank optimization approaches: LoRA, GaLore and QLoRA. It is important to note that while GaLore utilizes a 16-bit Adam optimizer, Q-GaLore employs an 8-bit Adam optimizer, further reducing memory requirements without compromising performance.

**Table 3: Comparison results of various memory-efficient fine-tuning algorithms on MMLU tasks. Note that the reported memory stands for the estimated memory overhead for weights and optimizer states. End-to-end memory measurements are discussed at Section 4.3.**
| Model | Methods | Memory | STEM | Social Sciences | Humanities | Other | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-3-8B | Full | 48 GB | 54.27 | 75.66 | 59.08 | 72.80 | 64.85 |
| LoRA | 16 GB | 53.00 | 74.85 | 58.97 | 72.34 | 64.25 |  |
| GaLore | 16 GB | 54.40 | 75.56 | 58.35 | 71.19 | 64.24 |  |
| QLoRA | 8 GB | 53.63 | 73.44 | 58.59 | 71.62 | 63.79 |  |
| Q-GaLore | 8 GB | 53.27 | 75.37 | 58.57 | 71.96 | 64.20 |  |
| Gemma-7B | Full | 51 GB | 30.03 | 37.16 | 34.08 | 35.47 | 34.21 |
| LoRA | 17 GB | 26.23 | 34.94 | 30.88 | 36.96 | 32.18 |  |
| GaLore | 17 GB | 27.33 | 36.74 | 30.82 | 37.90 | 33.20 |  |
| QLoRA | 9 GB | 24.83 | 27.54 | 28.09 | 33.40 | 28.49 |  |
| Q-GaLore | 9 GB | 27.73 | 36.80 | 32.54 | 37.89 | 33.68 |  |
| Mistral-7B | Full | 43 GB | 52.40 | 72.95 | 55.16 | 69.05 | 61.67 |
| LoRA | 14 GB | 52.13 | 72.46 | 55.05 | 68.77 | 61.41 |  |
| GaLore | 14 GB | 51.50 | 73.02 | 55.03 | 69.49 | 61.55 |  |
| QLoRA | 7 GB | 50.00 | 71.29 | 55.84 | 67.66 | 60.70 |  |
| Q-GaLore | 7 GB | 52.23 | 72.82 | 55.01 | 69.30 | 61.62 |  |

Tables  3 and  4 lead to consistent observations: (i) Q-GaLore achieves performance comparable to the full fine-tuning baseline across different models (LLaMA-3-8B, Gemma-7B, Mistral-7B, and RoBERTa-base), with a minimal performance gap of less than 0.65 compared to Full; (ii) Q-GaLore demonstrates
comparable or even superior performance compared to LoRA, with a improvement of 1.02 performance gain on the MMLU benchmark of Gemma-7B while also requiring less memory; (iii) Compared with QLoRA, Q-GaLore demonstrates consistent (up to 5.19) gains of performance across architectures and tasks, at the same memory costs.

**Table 4: Comparison results of various memory-efficient fine-tuning algorithms on GLUE tasks, with the pretrained RoBERTa model (baseline results are obtained from ). We report the Matthew’s correlation for the CoLA task, Pearson correlation for STS-B, average (matched and mismatched) accuracy for MNLI, F1 score for MRPC, and accuracy for all other tasks. Note that the reported memory stands for the estimated memory overhead for weights and optimizer states. End-to-end memory measurements are discussed at Section 4.3.**
| Methods | CoLA | STS-B | MRPC | RTE | SST2 | MNLI | QNLI | QQP | Average | Memory |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Full | 62.24 | 90.92 | 91.30 | 79.42 | 94.57 | 87.18 | 92.33 | 92.28 | 86.28 | 747 MB |
| LoRA | 60.06 | 90.82 | 92.01 | 79.78 | 94.38 | 87.17 | 92.20 | 91.11 | 85.94 | 264 MB |
| GaLore | 61.83 | 90.80 | 91.90 | 79.06 | 93.46 | 86.94 | 92.25 | 91.22 | 85.93 | 257 MB |
| QLoRA | 60.16 | 89.93 | 91.87 | 71.84 | 93.92 | 86.57 | 92.29 | 91.17 | 84.72 | 183 MB |
| Q-GaLore | 61.60 | 90.23 | 91.96 | 79.06 | 94.38 | 86.73 | 92.44 | 90.91 | 85.91 | 176 MB |

### 4.3 End-to-End Memory Measurement

Figure: Figure 5: Results of the memory allocation of training a LLaMA-7B model with a single batch size of 256.
Refer to caption: x5.png

We present an end-to-end memory measurement for training a LLaMA-7B model in Figure 5. Starting from the baseline full parameter training with BF16 Adam optimizer, 8-bits Adam optimizer halves the memory overhead of the optimizer states by quantizing them to a lower precision format. Then, 8-bits GaLore further compresses the memory cost by converting the optimizer states into a low-rank format. Moreover, 8-bits GaLore employs a fused backward operation that sequentially releases the gradient memory, rendering the gradient memory cost negligible. Building on this, Q-GaLore incorporates INT8 weights, which halve the memory requirement for weights. Projection quantization then further reduces the memory allocated to optimizer states. Notably, only Q-GaLore can train a LLaMA-7B model within the 16 GB memory constraint, demonstrating the potential for optimizing models on edge devices. Additionally, due to the varying data formats of gradients and weights, the requisite quantization and dequantization operations incur a throughput overhead of $14.64\%$, as compared to the original GaLore. We will improve the implementation for further work.

### 4.4 Further Investigation and Ablation Study

In this section, we focus on the ablation studies of Q-GaLore, centering on two key questions: Q1: How does Stochastic Rounding (SR) benefit the training process? Q2: What is the trade-off between training performance and SVD counts in Q-GaLore?

##### A1 : Enhanced low-precision training with stochastic rounding.

Stochastic rounding provides an unbiased estimation of accumulated gradient information, which is crucial for low-precision training. We conducted controlled experiments to pre-train LLMs with and without stochastic rounding. To ensure a fair comparison, we maintained consistency in other hyperparameters across the experiments: weights were stored in the INT8 data format, projection matrices were subjected to 4-bit quantization, and the adaptive convergence ratio for the gradient subspace was set at 0.4.

Figure: Figure 6: Ablation study of pre-training with Q-GaLore w/ or w/o Stochastic Rounding (SR). Full curve stands for the perplexity of the final checkpoint that optimized by original Adam optimizer. Each subfigure includes a smaller inset that represents the zoomed-in results.
Refer to caption: x6.png

Figure: Figure 7: Trade-off between performance and SVD counts for updating gradient subspace. Results are normalized by SVD counts of original GaLore.
Refer to caption: x7.png

Figure 6 illustrates the perplexity on the validation set throughout the training process. At each training step, gradient information is quantized back to the low-precision format (INT8), resulting in considerable information loss and suboptimal performance. The perplexity increased by 7.86, 1.98, and 2.27 for models with sizes of 60, 160, and 350 million parameters, respectively. Additionally, we implemented an initial warm-up stage for pre-training for training stability, where the weight updates are generally smaller. During this stage, significant loss of gradient information occurs due to the vanilla round-to-nearest scheme, resulting in a perplexity gap ranging from 18.67 to 47.02, compared with models using stochastic rounding. Meanwhile, Q-GaLore can effectively capture the gradient information without additional memory costs, achieving performance comparable to the Full baseline, with a perplexity gap of less than 1.

##### A2 : Over 60% SVD operations costs can be saved for free.

We explore the trade-off between the number of SVD operations used for updating the gradient subspace and pre-training performance on the LLaMA-130M model. Figure 7 (Right) demonstrates that there is an efficient reduction in SVD counts; with only 36.20% of SVD operations, Q-GaLore can achieve comparable performance to the GaLore baseline, resulting in significant time savings. Specifically, to update the gradient subspace of a LLaMA-7B model, the SVD operation requires approximately 10 minutes when measured on a single NVIDIA RTX A6000 GPU; and this gradient subspace is updated 300 times across 150,000 training iterations. By achieving more than 60% savings in SVD operations, our method significantly reduces the time cost by over 32 hours.

## 5 Conclusion

To overcome these challenges and further enhance memory-efficient training, we propose Q-GaLore, a method that reduces memory usage through quantization and low-rank projection. Our approach is motivated by two key observations during gradient low-rank training: (1) the gradient subspace exhibits diverse properties, with some layers converging at the very early training stages while others are subject to frequent changes; (2) the projection matrices demonstrate high quantization-friendliness and function effectively under 4-bit quantization. Building on these, Q-GaLore enables low-precision training (INT8 for the entire model and INT4 for the projection matrix)
with low-rank gradients and significantly fewer SVD operations. Our experiment results demonstrate that Q-GaLore achieves competitive pre-training and fine-tuning performance, e.g., for the first time facilitating training LLaMA-7B on a single NVIDIA RTX 4060 Ti with only 16GB memory.

## Appendix A More Implementation Details

The pseudo-code of the forward and backward process in PyTorch style are illustrated in the following:

Figure: Listing 1: int8_training

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: See above.
soundness: 3
strengths: This paper is tackling the important problem of democratizing the large-scale ML training given the limited compute resources. To achieve this goal, authors have performed insightful analyses, upon which they carefully develop quantization and layerwise adaptive SVD computations. Finally, authors have demonstrated the effectiveness of Q-GaLore in extensive empirical experiments, encompassing language model pertaining and fine-tuning.
summary: This paper improves the memory and compute efficiency of GaLore with a combination of aggressive quantization and lazy layerwise subspace exploration techniques. Authors motivate these techniques by thoughtfully designed empirical analyses (e.g., layerwise gradient subspace analysis), and validate its effectiveness through diverse experiments. Notably, Q-GaLore reduces memory consumption by up to 50% and saves over 32 hours for training a 7B model compared to the baselines like LoRA and GaLore.
weaknesses: In Table 1, it seems that the performance gap between GaLore and Q-GaLore tends to increase as the model size increases. While it still outperforms LoRA, I personally believe the 0.5 perplexity difference is not negligible. In addition, some hyperparameters like 0.4 cosine similarity threshold looks somewhat arbitrary, and it's unclear how important these hyperparameters are in the final performance.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See Weakness Part.
soundness: 2
strengths: 1. Q-GaLore achieves substantial memory reduction compared to the original Galore paper by applying 8-bit quantization to model weights and 4-bit quantization to the projection matrix. 
2. Q-GaLore reduces training time associated with Singular Value Decomposition (SVD) by decreasing the frequency of SVD updates for layers where the low-rank subspace remains relatively stable over time, which is well-motivated.
summary: The paper propose Q-GaLore to reduce the memory footprint when training LLMs. Q-GaLore is build upon GaLore and further quantize weights and projection matrix into lower precision to improve the performance. Q-GaLore adaptively update the subspace to reduce the overhead correspond to SVD decomposition. This approach achieve better training performance compared with other low-rank training methods and significantly reduce the memory requirement on pretraining of Llama-1B and fine-tuning on 7B scale models.
weaknesses: 1. "pre-training a LLaMA 7B with single batch size" is very strange in Section 1. Discussing the motivation in the fine-tuning scenario is a better choice.
2. The modifications on GaLore framework is relatively incremental (for the quantization section).
3.  How the quantization of weight and projection matrices contribute to the degradation of the models? Which factor contributes more to the performance degradation? For example, the comparison of (FP16 weight, FP16 projection) - (FP16 weight, INT8 projection) - (FP16 weight, INT4 Projection) - (INT8 weight, FP16 projection) - (INT8 weight, INT8 Projection) - (INT8 weight, INT4 Projection) will help to understand how each component and the choice of quantization bit-width affect the final performance.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1.  In line 201, how is the "*cosine similarity across matrices*" defined? Why not use other ways of measuring distances between matrices like Frobenius norm? Usually the cosine similarity measures the similarity between **two vectors** of an inner product space. Are the matrices flattened? If so, along which dimension? 
2. In line 315, the statement "*no baseline involves quantization and all data are maintained in BF16 format*" is confusing, since in the QLoRA baseline weights are quantized. Please clarify this statement or correct it if it's inaccurate.
3. In Tab 1, for the 60M model, why the LoRA (ReLoRA) uses the the same amount of memory (0.36GB) as full fine-tuning? Similarly, for the 130M model, LoRA takes up more memory (0.80GB) than full fine-tuning (0.76G). 
4. In Tab 1, if QLoRA is added as a baseline (QLoRA was included in Tab 2 as a baseline), how much memory is consumed and what is the perplexity? As Q-GoLore adopts quantization, comparing to QLoRA can be reasonable to evaluate memory efficiency.
5. In Tab 1, 2, and 3, how hyper-parameters are determined? Since different methods may have different optimal hyper-params like learning rate.
6. As stated in the weakness. Does the trained model performance heavily depend on the threshold for determining update intervals of projection matrices? If so, how do users determine the value for given a model and a task? If not, what is the recommended value?
7. In line 472, the time cost is reduced by over 32 hours. What is the total time of training in this case?
soundness: 2
strengths: Q-GaLore enhances GaLore with quantization.
- Q-GaLore havles the memory footprint of weights by quantizing weights to 8-bit.
- Q-GaLore adaptively reduces the frequency of the computation-intensive SVD operation on gradients (projection matrices), based on the enlightening observations that there are layers with diminishing gradients. 
- The memory footprint breakdown clarifies the improvement of Q-GaLore.
summary: This work proposes Q-GaLore, a memory-efficient training method based on GaLore. Q-GaLore further reduces the memory footprint by quantizing the model weights to 8-bit and the low-rank gradient terms to 4-bit. Q-GaLore also adaptively reduces the number of SVD operations on gradients throughout the training process. This work also performs experiments to decompose the memory footprint which verifies the efficacy of Q-GaLore.
weaknesses: - Limited novelty with increased complexity. This work mainly applies quantization to GaLore with adaptively reduced SVD operations on projection matrices. However, it introduces more hyper-parameters to tune, such as the threshold for determining update intervals. 
- A lack of end-to-end latency/time evaluation. Q-GaLore introduces extra operations like the dequantization and the calculation of cosine similarity of projection matrices. The extra cost of these operations, in terms of latency and computation, remains unclear compared to the baselines.  If possible, please consider a breakdown of time spent on different operations including the new ones introduced by Q-GaLore.
- Insufficient and vague evaluation on fine-tuning/training experiments.
  - No description of hyper-parameters like learning rates, training epochs, and batche sizes.
  - If the trained model performance heavily depends on the threshold of cosine similarity, a discussion on the threshold selection is necessary. If not, relevant experiments and a recommend value can be provided.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 4
questions: 1. I do not fully understand how the memory requirement is reduced. The memory bottleneck of GaLore seems to be the 32-bit decomposition of large matrices. Could you provide further clarification? If clarified well, I will improve my score.
2. Could you provide a more in-depth explanation of the adaptive lazy update, such as the reasons behind it or its behavior across different layers and matrices? Does this phenomenon occur with different training datasets? Is the lazy matrix the same for the same model under different training datasets?
3. Could the authors provide a more intuitive comparison of the time required to demonstrate the effectiveness of the acceleration?
soundness: 3
strengths: 1. The paper proposes an acceleration method to address the time and memory requirements of the GaLore process. It cleverly observes an early stopping phenomenon in some layers during GaLore training, which reduces the number of SVD decompositions needed. Additionally, the paper employs quantization to lower memory usage.
2. The structure of the paper is clear, making it easy to understand.
3. The experiments are comprehensive.
summary: The paper proposes a method to accelerate the GaLore, which is time-consuming and memory-intensive. It cleverly utilizes observed behaviors to reduce the time and memory required for GaLore. The structure of the paper is clear, and the experiments are thorough.
weaknesses: 1. The paper emphasizes reducing memory usage during the GaLore process. However, the main source of memory consumption in GaLore seems to be the need for SVD on large matrices, which currently only supports 32-bit precision. Without reducing the dimensionality of the decomposition matrices, the proposed method does not seem to genuinely reduce memory requirements.
2. The proposed adaptive lazy update is interesting, but it lacks further explanation. For instance, why does this phenomenon occur, and what is its relationship with different depths and types of layers?