GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning
Authors: Sifan Zhou, Shuo Wang, Zhihang Yuan, Mingjia Shi, Yuzhang Shang, Dawei Yang
Accepted at: acl25

GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning

Sifan Zhou1,2†‡, Shuo Wang1†, Zhihang Yuan1†, Mingjia Shi1, Yuzhang Shang4‡§, Dawei Yang1‡

1Houmo AI 2Southeast University 3Illinois Institute of Technology

Abstract

Large Language Models (LLMs) fine-tuning technologies have achieved remarkable results. However, traditional LLM fine-tuning approaches face significant challenges: they require large Floating Point (FP) computation, raising privacy concerns when handling sensitive data, and are impractical for resource-constrained edge devices. While Parameter-Efficient Fine-Tuning (PEFT) techniques reduce trainable parameters, their reliance on floating-point arithmetic creates fundamental incompatibilities with edge hardware. In this work, we introduce a novel framework for on-device LLM fine-tuning that eliminates the need for floating-point operations in both inference and training, named GSQ-Tuning. At its core is the Group-Shared Exponents Integer format, which efficiently represents model parameters in integer format using shared exponents among parameter groups. When combined with LoRA-like adapters, this enables fully integer-based fine-tuning that is both memory and compute efficient. We demonstrate that our approach achieves accuracy comparable to FP16-based fine-tuning while significantly reducing memory usage (~ 50%). Moreover, compared to FP8, our method can reduce ~ 5 × power consumption and ~ 11 × chip area with same performance, making large-scale model adaptation feasible on edge devices.

1 Introduction

Recent advances in Large Language Models (LLMs) have delivered impressive results in a variety of natural language tasks (Touvron et al., 2023a,b; Liu et al., 2023). LLMs are typically trained in several stages, including large-scale pretraining followed by one or more fine-tuning phases (Dubey et al., 2024; Liu et al., 2024a). LLM fine-tuning approaches like supervised fine-tuning (SFT) (Zhang et al., 2023), usually employ curated, high-quality corpora for refining the model with a standard language modeling (Chiang et al., 2023).

Despite their effectiveness, most LLM fine-tuning approaches require powerful cloud servers or GPUs equipped with large memory capacities. This poses two significant challenges in real-world settings: (1) uploading sensitive data to remote servers poses a fundamental privacy risk, and (2) in many practical scenarios, models must be deployed on resource-constrained edge devices—such as mobile processors or embedded AI accelerators—where memory and power budgets are tightly limited. Such constraints become critical in LLM’s personalized applications, where data cannot be shared with the cloud and model updates must remain local to ensure privacy. Meeting these challenges thus necessitates on-device adaptation methods capable of preserving data privacy and functioning within the limited memory and compute budgets of edge hardware.

Parameter-Efficient Fine-Tuning (PEFT) (Han et al., 2024) techniques such as LoRA (Hu et al., 2021) and QLoRA (Dettmers et al., 2023) alleviate part of this burden by reducing trainable parameters to around 1% of the original model. Unfortunately, they remain reliant on floating-point operations for both forward and backward passes, which clashes with edge-device constraints in three ways. First, during fine-tuning, weights, activation and gradients must be stored and updated in high-precision floating-point. It introduces additional overhead or even makes the LLM fine-tuning impractical on edge devices. Second, floating-point representations incur high memory overhead (e.g., FP16 doubles the memory cost compared to INT8; for a 7B-parameter model, this can surpass 20GB memory during fine-tuning process, presenting substantial challenges for mobile processors). Last but not least, commercial edge AI accelerators (e.g., Qualcomm Hexagon (QUALCOMM, 2024)) typically
get peak throughput only on integers, leaving up to 84% of compute units idle under FP16 training.

Therefore, eliminating floating-point arithmetic for fine-tuning would have a substantial impact on software, hardware, and application design for efficient on-device LLM adaptation (ARM, 2020; Kim et al., 2021). While previous studies on integer quantization (Jacob et al., 2018a; Kim et al., 2021; Xiao et al., 2022; Yuan et al., 2023) verify the feasibility of inference, they do not extend to gradient quantization, which is required for effective fine-tuning of LLMs at the edge.

In this paper, we propose a new framework for resource-efficient on-device LLM fine-tuning, termed GSQ-Tuning. Central to our method is the Group-Shared Exponents Integer format, a novel quantization strategy that replaces floating-point with a specialized integer-based representation. We integrate this with parameter-efficient LoRA-like modules to enable fully on-device fine-tuning without incurring large memory and computation costs. We further examine this design through a Pareto frontier analysis, which demonstrates how various bits-rank settings impact the trade-off between fine-tuning memory costs and accuracy. Extensive experiments across models of varying scales, different fine-tuning datasets, and diverse tasks have demonstrated the effectiveness and generalizability. We highlight our main contributions as follows:

• Group-Shared Exponents Integer Quantization: We introduce a quantization strategy that shares exponents among groups, thereby reducing the storage and computation overhead while still representing model parameters in integer format. Combined with LoRA-like adapters, our method supports fine-tuning under tight memory constraints.

• Integer Forward and Backward Computations: By extending integer quantization pipelines beyond inference to include gradients, both forward and backward passes remain hardware-friendly and efficiently utilize integer-focused edge accelerators.

• Pareto Frontier for Quantization Bits and Low-rank: We demonstrates how various bits-rank settings impact the trade-off between fine-tuning memory costs and accuracy through a Pareto frontier analysis. We empirically show that our approach achieves accuracy on par with FP16-based fine-tuning while dramatically lowering both ~ 50% memory usage. Furthermore, compared with FP8, at comparable performance levels, our method (GSE-INT5) reduces the power consumption of MAC unit by ~ 5 × and decreases chip area by ~ 11 × comparing to the origin.

2 Method

In this section, we present GSQ-Tuning, a fully quantized training method for on-device LLM fine-tuning. We begin by reviewing the fundamentals of LLM PEFT, highlighting the bottlenecks of implementing existing PEFT methods on device, and then review relevant neural network quantization literature (Sec.2.1). Building on these insights, we propose a new LLM fine-tuning framework—Group-Shared Exponents Integer in Fully Quantized Training—for on-device scenarios. To enable this framework, we design two key components: (1) A Group-Shared Exponents Integer data format to replace floating-point representations (Sec.2.2). (2) A Fully Quantized Fine-tuning Framework that leverages our new data format (Sec.2.3). Finally, we explore the performance–efficiency trade-off in GSQ-Tuning via Pareto frontier analysis (Sec.2.4), providing practical guidance for its use.

2.1 Preliminaries

Low-rank Adaptation. LoRA (Hu et al., 2021) is a milestone method that injects trainable low-rank adapters into linear layers, allowing efficient fine-tuning while keeping the original parameters unchanged. Specifically, a LoRA linear layer is parameterized by a non-trainable weight matrix \( \mathbf{W} \in \mathbb{R}^{oc \times ic} \), along with trainable components \( \mathbf{A} \in \mathbb{R}^{rxic} \) and \( \mathbf{B} \in \mathbb{R}^{oc \times r} \), where \( r \) is a small integer. The input \( \mathbf{X} \in \mathbb{R}^{b \times ic} \) and output \( \mathbf{Y} \in \mathbb{R}^{b \times oc} \) correspond to a linear layer with \( oc \times ic \) processing a batch of size \( b \). Building on LoRA, QLoRA integrates it with 4-bit NormalFloat (NF4) quantization and Double Quantization (DQ) techniques, enabling the fine-tuning of a 65B parameter model on a single 48GB GPU with minimal performance loss. In this paper, due to the memory constraint of on-device PEFT, we adopt QLoRA to quantize the weights of LLMs. The formulation is:

\[
\mathbf{Y} = \mathbf{X} D Q (W^{NF4})^T + X A^T B^T
\]

where we omit the transpose for similarity. \( NF4 \) means the 4 bit NormalFloat (NF) data type and
and \( DQ \) is *Double Quantization* operation to map weights from NF4 to BF16 in QLoRA. The low-rank components \( \mathbf{A} \) and \( \mathbf{B} \) and input \( \mathbf{X} \) remain in BF16 during fine-tuning process. However, QLoRA still does not suit on-device PEFT scenarios because the low-rank term \( X A^T B^T \) remains in BF16, whereas most on-device hardware only supports integer operations. This limitation motivates our development of a new LoRA method that relies exclusively on integer operations.

**Quantization.** Quantization (Jacob et al., 2018b) is a crucial technique that maps a floating-point number to a discrete interval using integer values. Given a floating-point (FP) tensor \( \mathbf{x} \) (such as weights, activations or gradients), the \( b \)-bits formulation is:

\[
\mathbf{x}_{\text{int}} = Q(\mathbf{x}) = \operatorname{clamp}\left( \left\lfloor \frac{\mathbf{x}}{s} \right\rfloor + z, 0, 2^b - 1 \right)
\]

where \( s = \frac{\max(|\mathbf{x}|)}{2^{b-1}-1} \) is the scaling factor, \( z \) denotes zero points, \( \lfloor \cdot \rfloor \) refers to the round-to-nearest, and the function clamp(\(\cdot\)) clips values outside the integer range \([q_{\min}, q_{\max}]\) respectively. \([q_{\min}, q_{\max}]\) is the quantization range determined by the bit width \( b \), where \( q_{\min} = -sz \) and \( q_{\max} = -s(2^b - 1 - z) \).

**Fully Quantized Training (FQT).** FQT involves quantizing all tensors—weights, activations, and gradients—needed for computation-intensive operations (like matrix multiplication) during both forward and backward propagation (Wang et al., 2018; Yang et al., 2020; Zhu et al., 2020). When the network’s weights, activations, and gradients are each quantized to 8 bits, this is referred to as W8A8G8 quantization. Notably, FQT is different from Quantization-Aware Training (QAT), we also discuss the difference in Sec. A.1.

2.2 **Group-Shared Exponents Integer Low-bitwidth Floating Point (FP).** Floating-point numbers are a commonly used data representation in deep learning. For instance, FP16 represents each number using 16 bits. Recently, lower-bit floating-point representations, such as FP8, have been introduced into the training processes of deep learning models (Micikevicius et al., 2022; Baalen et al., 2023). FP8 operates in two modes: the E4M3 and E5M2 formats. In these formats, E represents the number of exponent bits and M denotes that of mantissa bits.

Similar to quantization methods, low-bitwidth FP formats can effectively reduce memory storage requirements and decrease the hardware area and energy consumption of computational units.

![Line plot showing standard deviations of weights across layers](page_1012_120_377_246.png)

Figure 1: **In each layer, the weights’ magnitudes are similar.** The standard deviations of weights across layers are less than \( 2^{-2} \) by 3-\(\sigma\) (about probability 99.7\%). The weights are from Vicuna-7B-v1.5.

<table>
  <tr>
    <th></th>
    <th colspan="3">FP8</th>
    <th colspan="4">GSE-INT8</th>
  </tr>
  <tr>
    <th></th>
    <th>S=1</th>
    <th>E=5</th>
    <th>M=2</th>
    <th>E=5</th>
    <th>S=1, M=7</th>
    <th>S=1, M=7</th>
    <th>S=1, M=7</th>
  </tr>
  <tr>
    <td>FP8</td>
    <td>1 ×</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GSE-INT8</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Figure 2: **The GSE format is memory efficient through group-shared exponent bits.** Comparison between FP8 and GSE-Int8.

However, we observe that low-bitwidth FP may not be the optimal solution for LoRA fine-tuning in large-scale models, primarily due to the following 3 reasons: (1) Neural network tensors exhibit spatial locality, meaning that adjacent elements within a tensor tend to have similar magnitudes, leading to redundancy in the exponent bits of FP representations. As illustrated in Fig. 1, the standard deviation of the values in the weight tensor is considerably lower than the magnitude of the values, indicating a small local variation; (2) The limited number of mantissa bits in low-bitwidth FP formats constrains precision, potentially impairing model performance. For instance, the E5M2 format, which has only two mantissa bits, is incapable of representing certain integers below ten, such as 5, 7, and 9; (3) FP computation demonstrates less efficiency in memory, chip area, and power consumption compared to GSE-INT computation, making it less suitable for resource-constrained environments. As shown in Table 7, FP formats incur considerably higher costs in power and chip area compared to integer-based computation, making them less suitable for edge environments.

Due to the inherent characteristics of FP representations and the requirement for relatively high precision in training, it is crucial to explore other data format to reduce both hardware area and energy consumption in resource-constrained cases.
Group-Shared Exponents Integer (GSE-INT).
Inspired with block FP (Zhang et al., 2022), we propose the Group-Shared Exponents Integer (GSE) format as an alternative to FP formats for matrix multiplication in both forward-propagation and back-propagation. This format is also used for storing activations required by back-propagation to reduce memory consumption. As illustrated in Fig. 2, GSE introduces the following key modifications compared to traditional floating-point formats: (1) To leverage the locality of tensor values, we share the exponent across a group of N numbers. That is, all N numbers within the group use the same exponent. (2) The number of bits used for the shared exponent is fixed at 5. (3) The implicit leading 1 in floating-point representations is removed and replaced with a standard integer representation. The numerical representation in GSE is:
\[
x = (-1)^s \cdot 2^e \cdot m
\]
where s is sign, e is the exponent value (For simplicity, we omit the exponent bias), m is the mantissa value. The GSE format is memory efficient through sharing exponent bits. Memory for FP is \( N(E + M + 1) \) and memory for GSE is \( N(M + 1) + E \). As the group size N increases, the memory savings grow proportionally, while the overhead of the shared exponent is negligible.

Matrix Multiplication using GSE. Consider two vectors, A and B, both represented using the GSE format and having a length of N. The dot product of the two vectors can be computed as:
\[
y = 2^{e_A + e_B} \sum_{i=1}^N (-1)^{s_A \oplus s_B} m_{A,i} m_{B,i},
\]
where \( m_{A,i} \) and \( m_{B,i} \) are the integer mantissas of the i-th elements of the vectors. The computation involves a standard integer multiply-accumulate (MAC) operation, followed by scaling with the combined exponent \( 2^{e_A + e_B} \).

The dot product operation can be extended to large-scale matrix multiplication. For two matrices X and Y, we partition the data into groups of size N. Specifically, rows of X are grouped along their elements, with each group sharing a single exponent, and columns of Y are grouped similarly. This grouping strategy simplifies hardware implementation and makes the GSE format a practical and efficient choice for large-scale matrix operations.

Transform from FP to GSE. The transformation from FP representation to GSE format is efficient due to the design of GSE. First, within a group of N FP numbers, identify the largest exponent \( e_{\max} \) among them. Then, using \( e_{\max} \) as the shared exponent for the group. For each FP value in the group, its mantissa is adjusted by adding the implicit leading bit (if applicable) and then right-shifting the value based on the difference between its original exponent and \( e_{\max} \). This process ensures that all values are aligned to the shared exponent, leading the storage and computation efficiency while preserving precision.

Figure 3: Dataflow of GSQ-Tuning. The weight is NF4 in full-rank branch and is FP32 in low-rank branch.

2.3 Fully Quantized Fine-tuning

As illustrated in Fig. 3, our GSQ-Tuning framework introduces a hardware-efficient quantization pipeline. Compared to QLoRA, we fully quantize weights, activations, and gradients to low-bit integers. While QLoRA primarily focuses on 4-bit quantization of frozen base model weights (NF4) while keeping adapters in high precision (BF16), our approach achieves superior computational and memory efficiency. Building on the quantize-compute-dequantize (QCD) paradigm for low-precision matrix multiplication (MM) (Xi et al., 2024), the QCD approach operates in three stages: (1) Quantization: Convert high-precision inputs matrices (e.g., BF16) to low-precision (e.g., GSE-INT6) using a quantizer \( Q(\cdot) \); (2) Computation in low-precision MM: Perform low-precision MM to produce an intermediate output (e.g., GSE-INT6); and (3) Dequantization: Convert output back to high-precision using a dequantizer \( Q^{-1}(\cdot) \).

Forward Propagation. The forward propagation for a linear layer is calculated as follows:

\[
Y^{BF16} = Q^{-1}\left(Q(X^{BF16})Q\left(DQ(W^{NF4})^T\right)\right)
\]
\[
+ Q^{-1}\left(Q(X^{BF16})Q(A^{BF16})^TQ(B^{BF16,T})\right)
\]
Backward Propagation. Gradients are computed directly on quantized tensors using back propagation and chain rule:

\[
\frac{\partial \mathcal{L}}{\partial \mathbf{A}} = Q^{-1}(Q(\mathbf{B})^T Q \left( \frac{\partial \mathcal{L}}{\partial \mathbf{Y}} \right)^T Q(\mathbf{X}))
\]

\[
\frac{\partial \mathcal{L}}{\partial \mathbf{B}} = Q^{-1}(Q \left( \frac{\partial \mathcal{L}}{\partial \mathbf{Y}} \right)^T Q(\mathbf{X})Q(\mathbf{A})^T)
\]

\[
\frac{\partial \mathcal{L}}{\partial \mathbf{X}} = Q^{-1}(Q \left( \frac{\partial \mathcal{L}}{\partial \mathbf{Y}} \right) (Q(\mathbf{W}) + Q(\mathbf{B})Q(\mathbf{A})))
\]

2.4 Pareto Frontier for Quantization Bits and Low-rank.

Co-optimization Principle for Model Bits and Rank. The memory footprint and FLOPs during fine-tuning exhibit strong dependence on both quantization bit-width and LoRA rank \( O(b \cdot r) \) scaling. Excessive values in either dimension impose prohibitive computational burdens: (1) Memory:Mem \( \propto b \cdot r \) (adapter parameter storage); (2) Compute: Flops \( \propto r \cdot d^2 \) (for hidden dimension \( d \)). This necessitates joint optimization of (\( b, r \)) to guide the accuracy-efficiency trade-off space effectively. Pure bit-width reduction sacrifices model capacity, while unrestrained rank scaling inflates computation costs disproportionately.

The effectiveness of GSQ-Tuning hinges on how quantization bit-width interacts with the dimensions of low-rank adapters. To inform real-world deployments, we systematically analyze this interplay by constructing a Pareto frontier that illustrates the balance between model memory consumption during fine-tuning and accuracy across various bits-rank settings. We hope our findings not only highlight optimal configurations, but also offer practical guidelines for practitioners to tailor solutions to specific hardware constraints.

Pareto Frontier Analysis. Based on our GSQ-Tuning, we construct a Pareto frontier by plotting model memory during fine-tuning against validation accuracy across different *bits-rank* configuration. As shown in Fig. 4, the frontier reveals three distinct optimization regimes (1) *High-Bit Low-Rank Regime* (*8-bit, r=64*): Reaches 65.60 Acc with suboptimal efficiency. 0.50 Acc gain from \( r = 16 \) to 64 indicates high-bit quantization inherently limits error magnitude, requiring less rank compensation. (2) *Mid-Bit Balanced Regime* (*6-bit, r=128*): Delivers 65.58 Acc with moderate resources. 0.71 Acc gain from \( r = 16 \) to 128 shows diminishing returns beyond this point (only 0.32 Acc gain from \( r = 128 \) to 512). (3) *Low-Bit High-Rank Regime* (*5-bit, r=512*): Achieves 64.88 Acc with minimal memory footprint. 0.91 Acc gain from \( r = 16 \) to 512 demonstrates that aggressive rank scaling can compensate for severe quantization errors. Besides, Compared to FP8, GSQ-Tuning offers higher finetuning performance at similar or lower memory (Table 2) and reduces chip area (~11×) and power (~5×, Table 7). We also provide the Pareto frontier and detailed results for other models in appendix. Extra extensive results yield similar guidance.

![Pareto curve of accuracy-memory trade-offs. Compared to FP16, our GSQ-Tuning can reduce ~50% memory usage while having the comparable accuracy. Detailed results are in Tab.10](page_1012_120_484_384.png)

Figure 4: Pareto curve of accuracy-memory trade-offs. Compared to FP16, our GSQ-Tuning can reduce ~50% memory usage while having the comparable accuracy. Detailed results are in Tab.10

3 Experiments

Foundation Models and Evaluation Metrics. We apply our method to the entire LLaMA family, including LLaMA-2 (7B/13B/70B)(Touvron et al., 2023b), and LLaMA-3 (3B-8B). We evaluate the fine-tuning models on up to 9 zero-shot commonsense question-answering (CSQA) tasks using the lm-evaluation-harness (version 0.4.7)(Gao et al., 2024), including BoolQ(Clark et al., 2019), HellaSwag (Zellers et al., 2019), LAMBADA (OpenAI)(Radford et al., 2019), OpenBookQA(Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), Winogrande (Sakaguchi et al., 2019), ARC-Easy, and ARC-Challenge (Clark et al., 2018). The fine-tuning dataset follows Alpaca (Taori et al., 2023), with 52K instruction data from text-davinci-003. Besides, we also report the memory cost (Mem.(G)) during the model fine-tuning process.

Training Details. We employ the whole fine-tuning process based on LLaMA-Factory (Zheng et al., 2024). We implement GSQ-Tuning in PyTorch using models from Hugging Face. We freeze the parameters of the linear modules and
Table 1: 0-shot CSQA accuracy comparison with respect to different quantization bits in 64 rank setting. '4-8-8' means quantize weights, activation and gradients to 4-bit, 8-bit and 8-bit. GSQ-Tuning is built on QLoRA, where all weights are quantized as NF4 firstly.Notably, the memory usage for LLaMA series model is for the model weights alone. Since these models are not fine-tuned, no fine-tuning phase (no gradient computation or updates) is involved. Denoted as 16-16-None for weights-activation-gradients.

<table>
  <tr>
    <th>Method</th>
    <th>LLMs branch</th>
    <th>low-rank branch</th>
    <th>Avg</th>
    <th>ARC-c</th>
    <th>ARC-e</th>
    <th>BoolQ</th>
    <th>HellaS.</th>
    <th>OBQA</th>
    <th>PIQA</th>
    <th>SCIQ.</th>
    <th>WinOg.</th>
    <th>Mem.(G)</th>
  </tr>
  <tr>
    <td rowspan="2">LLaMA2-7B<br>w/ <i>QLoRA</i></td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>64.13</td>
    <td>46.25</td>
    <td>74.62</td>
    <td>77.68</td>
    <td>76.01</td>
    <td>44.20</td>
    <td>79.11</td>
    <td>46.11</td>
    <td>69.06</td>
    <td>13.20</td>
  </tr>
  <tr>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>65.69</td>
    <td>47.14</td>
    <td>74.75</td>
    <td>79.50</td>
    <td>76.46</td>
    <td>45.50</td>
    <td>79.63</td>
    <td>50.26</td>
    <td>71.32</td>
    <td>10.73</td>
  </tr>
  <tr>
    <td rowspan="3">w/ GSQ-Tuning</td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td><b>65.60</b></td>
    <td>48.12</td>
    <td>74.24</td>
    <td>79.72</td>
    <td>76.00</td>
    <td>45.80</td>
    <td>79.60</td>
    <td>49.69</td>
    <td>71.67</td>
    <td>7.28</td>
  </tr>
  <tr>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>65.39</td>
    <td>47.70</td>
    <td>74.58</td>
    <td>79.24</td>
    <td>76.05</td>
    <td>44.60</td>
    <td>79.60</td>
    <td>50.41</td>
    <td>70.96</td>
    <td>5.97</td>
  </tr>
  <tr>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td><b>64.18</b></td>
    <td>45.14</td>
    <td>72.69</td>
    <td>75.20</td>
    <td>75.27</td>
    <td>46.40</td>
    <td>79.65</td>
    <td>48.62</td>
    <td>70.48</td>
    <td>5.81</td>
  </tr>
  <tr>
    <td rowspan="2">LLaMA2-13B<br>w/ <i>QLoRA</i></td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>66.65</td>
    <td>48.81</td>
    <td>76.47</td>
    <td>82.45</td>
    <td>79.67</td>
    <td>44.80</td>
    <td>80.36</td>
    <td>48.31</td>
    <td>72.38</td>
    <td>25.7</td>
  </tr>
  <tr>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>67.61</td>
    <td>49.66</td>
    <td>77.23</td>
    <td>83.30</td>
    <td>78.95</td>
    <td>45.40</td>
    <td>80.74</td>
    <td>51.59</td>
    <td>73.24</td>
    <td>17.42</td>
  </tr>
  <tr>
    <td rowspan="3">w/ GSQ-Tuning</td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td><b>67.48</b></td>
    <td>49.57</td>
    <td>77.40</td>
    <td>82.87</td>
    <td>78.88</td>
    <td>46.20</td>
    <td>80.90</td>
    <td>50.72</td>
    <td>73.32</td>
    <td>11.99</td>
  </tr>
  <tr>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>67.35</td>
    <td>49.66</td>
    <td>77.27</td>
    <td>82.75</td>
    <td>78.66</td>
    <td>47.05</td>
    <td>80.90</td>
    <td>50.97</td>
    <td>73.16</td>
    <td>10.89</td>
  </tr>
  <tr>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td><b>66.97</b></td>
    <td>49.91</td>
    <td>76.60</td>
    <td>81.87</td>
    <td>78.15</td>
    <td>46.20</td>
    <td>80.41</td>
    <td>49.54</td>
    <td>73.09</td>
    <td>10.33</td>
  </tr>
  <tr>
    <td rowspan="2">LLaMA2-70B<br>w/ <i>QLoRA</i></td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>70.68</td>
    <td>56.91</td>
    <td>80.05</td>
    <td>85.78</td>
    <td>83.59</td>
    <td>48.60</td>
    <td>82.48</td>
    <td>48.67</td>
    <td>79.40</td>
    <td>137.42</td>
  </tr>
  <tr>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>72.22</td>
    <td>59.81</td>
    <td>82.20</td>
    <td>86.51</td>
    <td>83.89</td>
    <td>50.40</td>
    <td>83.13</td>
    <td>51.48</td>
    <td>80.35</td>
    <td>66.82</td>
  </tr>
  <tr>
    <td rowspan="3">w/ GSQ-Tuning</td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td><b>72.20</b></td>
    <td>59.90</td>
    <td>82.32</td>
    <td>86.51</td>
    <td>83.90</td>
    <td>50.20</td>
    <td>83.08</td>
    <td>51.59</td>
    <td>80.11</td>
    <td>52.17</td>
  </tr>
  <tr>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>72.10</td>
    <td>59.39</td>
    <td>82.15</td>
    <td>86.51</td>
    <td>83.94</td>
    <td>50.00</td>
    <td>83.30</td>
    <td>50.92</td>
    <td>80.58</td>
    <td>48.71</td>
  </tr>
  <tr>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td><b>71.70</b></td>
    <td>58.87</td>
    <td>81.48</td>
    <td>85.90</td>
    <td>83.91</td>
    <td>49.60</td>
    <td>82.81</td>
    <td>50.67</td>
    <td>80.43</td>
    <td>46.98</td>
  </tr>
  <tr>
    <td rowspan="2">LLaMA3-3B<br>w/ <i>QLoRA</i></td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>62.64</td>
    <td>45.90</td>
    <td>71.68</td>
    <td>73.00</td>
    <td>73.64</td>
    <td>43.20</td>
    <td>77.42</td>
    <td>47.08</td>
    <td>69.22</td>
    <td>6.42</td>
  </tr>
  <tr>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>64.11</td>
    <td>48.63</td>
    <td>74.07</td>
    <td>77.22</td>
    <td>73.12</td>
    <td>41.60</td>
    <td>78.51</td>
    <td>49.33</td>
    <td>70.40</td>
    <td>6.78</td>
  </tr>
  <tr>
    <td rowspan="3">w/ GSQ-Tuning</td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td><b>64.02</b></td>
    <td>47.30</td>
    <td>74.44</td>
    <td>77.26</td>
    <td>72.51</td>
    <td>42.40</td>
    <td>78.60</td>
    <td>49.80</td>
    <td>69.86</td>
    <td>3.93</td>
  </tr>
  <tr>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td><b>63.71</b></td>
    <td>47.07</td>
    <td>73.59</td>
    <td>76.64</td>
    <td>72.23</td>
    <td>41.80</td>
    <td>78.20</td>
    <td>48.76</td>
    <td>71.36</td>
    <td>3.47</td>
  </tr>
  <tr>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td><b>62.74</b></td>
    <td>48.04</td>
    <td>73.44</td>
    <td>73.36</td>
    <td>71.96</td>
    <td>40.40</td>
    <td>78.13</td>
    <td>47.65</td>
    <td>68.98</td>
    <td>3.24</td>
  </tr>
  <tr>
    <td rowspan="2">LLaMA3-8B<br>w/ <i>QLoRA</i></td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>67.18</td>
    <td>53.50</td>
    <td>77.74</td>
    <td>81.13</td>
    <td>79.20</td>
    <td>45.00</td>
    <td>80.63</td>
    <td>47.03</td>
    <td>73.24</td>
    <td>15.01</td>
  </tr>
  <tr>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>68.45</td>
    <td>55.63</td>
    <td>80.13</td>
    <td>83.67</td>
    <td>78.78</td>
    <td>44.80</td>
    <td>81.28</td>
    <td>50.41</td>
    <td>72.93</td>
    <td>11.64</td>
  </tr>
  <tr>
    <td rowspan="3">w/ GSQ-Tuning</td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td><b>68.61</b></td>
    <td>55.97</td>
    <td>80.22</td>
    <td>83.61</td>
    <td>78.68</td>
    <td>45.20</td>
    <td>81.50</td>
    <td>50.41</td>
    <td>73.32</td>
    <td>7.63</td>
  </tr>
  <tr>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td><b>68.22</b></td>
    <td>55.55</td>
    <td>79.29</td>
    <td>83.67</td>
    <td>78.47</td>
    <td>44.80</td>
    <td>80.90</td>
    <td>50.05</td>
    <td>73.09</td>
    <td>6.86</td>
  </tr>
  <tr>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td><b>66.69</b></td>
    <td>54.10</td>
    <td>77.99</td>
    <td>81.65</td>
    <td>77.12</td>
    <td>43.80</td>
    <td>79.54</td>
    <td>47.90</td>
    <td>71.43</td>
    <td>6.47</td>
  </tr>
</table>

update a smaller (low-rank) set of parameters during the fine-tuning. The group size of our GSQ-Tuning is 32. We fine-tune models using the 8-bits AdamW optimizer (Dettmers et al.) in bfloat16 precision. We choose the constant learning rate schedule and set the learning rate to be \( 1 \times 10^{-5} \) for all models. In all cases, we tune the hyper-parameters on the base BF16 tasks, and re-use the same values for low-precision training. We always perform single-epoch experiments using a linear learning rate warm-up of 100 steps. The batch size and sequence length are fixed at 16 and 2048. The number of fine-tuning steps is 3.24K for Alpaca.

Hardware Synthesis. We implemented the hardware in Verilog RTL and synthesized it using Synopsys Design Compiler with a 7nm technology library to estimate the process engine’s area, latency, and power consumption (Clark et al., 2016). The hardware operates at 1 GHz and has a capability of 50 TOPS. The memory subsystem is not considered in our settings and analyses.

3.1 Overall Results

GSQ-Tuning Results on LLaMA Family. Here, we compare the fine-tuning performance across LLaMA family (3B 70B) against QLoRA. As shown in Tab. 1, GSQ-Tuning achieves comparable or better zero-shot accuracy across different LLaMA model scales (7B-70B) under a fully low-bit quantization fine-tuing setting. With 8-bit quantization precision (W8A8G8), GSQ-Tuning matches or exceeds QLoRA’s performance on 83% of tasks, despite using 50% fewer bits for activations and gradients. Even at aggressive 5-bit quantization (W5A5G5), the GSQ-Tuning maintains 98.6% of QLoRA’s average accuracy while reducing memory footprint by 2.67×. These results confirm GSQ-Tuning’s effectiveness for resource-constrained edge deployment.

Comparison with FP8. Here, we compare the designed GSE data format with FP8 in fully quantized fine-tuning framework. As shown in Tab. 2, the results demonstrate that the designed GSE implemented in our GSQ-Tuning method achieves superior fine-tuning performance compared to FP8 while significantly reducing computation efficiency. Even under 5-bit settings, GSQ-Tuning maintains fine-tuning performance on par with FP8, further validating its effectiveness. Additionally, to mitigate the impact of rank variations, we report the fine-tuning results using 64-rank setting in Tab. 15
Table 2: 0-shot CSQA accuracy comparison with FPS in different quantization bits in 32 rank setting.

<table>
  <tr>
    <th>Method</th>
    <th>LLMs branch</th>
    <th>low-rank branch</th>
    <th>Avg.</th>
    <th>ARC-c</th>
    <th>ARC-e</th>
    <th>BoolQ</th>
    <th>HellaS.</th>
    <th>OBQA</th>
    <th>PIQA</th>
    <th>SCIQ.</th>
    <th>WinoG.</th>
    <th>Mem.(G)</th>
  </tr>
  <tr>
    <td>LLaMA2-7B</td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>64.13</td>
    <td>46.25</td>
    <td>74.62</td>
    <td>77.68</td>
    <td>76.01</td>
    <td>44.20</td>
    <td>79.11</td>
    <td>46.11</td>
    <td>69.06</td>
    <td>13.2</td>
  </tr>
  <tr>
    <td>w/ QLoRA</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>65.24</td>
    <td>47.27</td>
    <td>75.04</td>
    <td>78.87</td>
    <td>76.11</td>
    <td>44.60</td>
    <td>79.76</td>
    <td>49.95</td>
    <td>70.32</td>
    <td>9.37</td>
  </tr>
  <tr>
    <td>w/ FP8</td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td>64.09</td>
    <td>45.90</td>
    <td>73.27</td>
    <td>77.86</td>
    <td>75.83</td>
    <td>45.40</td>
    <td>78.45</td>
    <td>46.98</td>
    <td>69.06</td>
    <td>6.52</td>
  </tr>
  <tr>
    <td>w/ GSQ-Tuning</td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td><b>65.45</b></td>
    <td>48.12</td>
    <td>74.71</td>
    <td>78.38</td>
    <td>76.14</td>
    <td>46.00</td>
    <td>79.71</td>
    <td>49.64</td>
    <td>70.96</td>
    <td>6.52</td>
  </tr>
  <tr>
    <td>w/ GSQ-Tuning</td>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td><b>64.00</b></td>
    <td>44.97</td>
    <td>73.32</td>
    <td>75.29</td>
    <td>74.95</td>
    <td>44.60</td>
    <td>79.27</td>
    <td>48.93</td>
    <td>70.24</td>
    <td>5.45</td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>67.18</td>
    <td>53.50</td>
    <td>77.74</td>
    <td>81.13</td>
    <td>79.20</td>
    <td>45.00</td>
    <td>80.63</td>
    <td>47.03</td>
    <td>73.24</td>
    <td>15.01</td>
  </tr>
  <tr>
    <td>w/ QLoRA</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>68.31</td>
    <td>55.55</td>
    <td>80.39</td>
    <td>83.36</td>
    <td>78.65</td>
    <td>44.60</td>
    <td>81.28</td>
    <td>50.05</td>
    <td>72.61</td>
    <td>11.02</td>
  </tr>
  <tr>
    <td>w/ FP8</td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td>66.62</td>
    <td>50.77</td>
    <td>76.43</td>
    <td>81.59</td>
    <td>78.17</td>
    <td>43.80</td>
    <td>80.20</td>
    <td>47.44</td>
    <td>74.59</td>
    <td>7.23</td>
  </tr>
  <tr>
    <td>w/ GSQ-Tuning</td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td><b>68.45</b></td>
    <td>55.72</td>
    <td>80.22</td>
    <td>83.43</td>
    <td>78.60</td>
    <td>45.00</td>
    <td>81.18</td>
    <td>50.20</td>
    <td>73.32</td>
    <td>7.23</td>
  </tr>
  <tr>
    <td>w/ GSQ-Tuning</td>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td><b>66.48</b></td>
    <td>51.71</td>
    <td>77.69</td>
    <td>82.11</td>
    <td>76.91</td>
    <td>44.20</td>
    <td>79.43</td>
    <td>48.16</td>
    <td>71.67</td>
    <td>6.07</td>
  </tr>
</table>

Table 3: Cross-modal task evaluation on LLaVA-v1.5-7B: the default setting is QLoRA on 4-bits/64-rank without finetuning. Shared exponents shows robustness to LLM’s finetuning, referring to the comparison with BF16.

<table>
  <tr>
    <th>Settings</th>
    <th>LLMs branch</th>
    <th>low-rank branch</th>
    <th colspan="2">POPE-random</th>
    <th colspan="2">POPE-adversarial</th>
    <th colspan="2">TextVQA MMBench</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th>accuracy precision F1-score</th>
    <th>accuracy precision F1-score</th>
    <th>accuracy precision F1-score</th>
    <th>accuracy precision F1-score</th>
  </tr>
  <tr>
    <td>LLaVA-v1.5-7B</td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>84.19</td>
    <td>84.08</td>
    <td>84.80</td>
    <td>74.40</td>
    <td>69.96</td>
    <td>76.96</td>
    <td>6.51</td>
    <td>55.45</td>
  </tr>
  <tr>
    <td>w/ QLoRA</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>87.47</td>
    <td>92.51</td>
    <td>86.88</td>
    <td>83.87</td>
    <td>85.52</td>
    <td>83.48</td>
    <td>47.68</td>
    <td>67.08</td>
  </tr>
  <tr>
    <td>w/ GSQ-Tuning</td>
    <td>4-8-8</td>
    <td>4-8-8</td>
    <td>87.84</td>
    <td>96.21</td>
    <td>87.08</td>
    <td>84.03</td>
    <td>87.40</td>
    <td>83.28</td>
    <td>45.19</td>
    <td>69.75</td>
  </tr>
  <tr>
    <td>w/ GSQ-Tuning</td>
    <td>4-6-6</td>
    <td>4-6-6</td>
    <td>88.08</td>
    <td>96.08</td>
    <td>87.39</td>
    <td>83.43</td>
    <td>85.80</td>
    <td>82.87</td>
    <td>49.13</td>
    <td>70.19</td>
  </tr>
</table>

Table 4: 0-shot CSQA accuracy on CS170K dataset in 64 rank setting.

<table>
  <tr>
    <th>Method</th>
    <th>LLMs branch</th>
    <th>low-rank branch</th>
    <th>Avg.</th>
    <th>ARC-c</th>
    <th>ARC-e</th>
    <th>BoolQ</th>
    <th>HellaS.</th>
    <th>OBQA</th>
    <th>PIQA</th>
    <th>SCIQ.</th>
    <th>WinoG.</th>
    <th>Mem.(G)</th>
  </tr>
  <tr>
    <td>LLaMA2-7B</td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>64.13</td>
    <td>46.25</td>
    <td>74.62</td>
    <td>77.68</td>
    <td>76.01</td>
    <td>44.20</td>
    <td>79.11</td>
    <td>46.11</td>
    <td>69.06</td>
    <td>13.2</td>
  </tr>
  <tr>
    <td>w/ QLoRA</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>65.78</td>
    <td>51.79</td>
    <td>78.86</td>
    <td>81.28</td>
    <td>75.77</td>
    <td>46.00</td>
    <td>79.98</td>
    <td>53.89</td>
    <td>75.37</td>
    <td></td>
  </tr>
  <tr>
    <td>w/ GSQ-Tuning</td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td><b>67.73</b></td>
    <td>51.79</td>
    <td>78.37</td>
    <td>82.32</td>
    <td>75.74</td>
    <td>46.20</td>
    <td>79.27</td>
    <td>53.12</td>
    <td>75.06</td>
    <td></td>
  </tr>
  <tr>
    <td>w/ GSQ-Tuning</td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td><b>67.56</b></td>
    <td>50.77</td>
    <td>77.74</td>
    <td>82.23</td>
    <td>75.07</td>
    <td>47.40</td>
    <td>79.60</td>
    <td>53.58</td>
    <td>74.11</td>
    <td></td>
  </tr>
</table>

of the appendix. Extensive experiments consistently support the advantages of our approach.

Table 5: Fine-tuning time comparison with full-finetuning and low-rank adapter on alpaca-52K dataset in LLaMA2-7B model.

<table>
  <tr>
    <th>Method</th>
    <th>LLMs branch</th>
    <th>low-rank branch</th>
    <th>Avg.</th>
    <th>Ft-Time.(h)</th>
    <th>Mem.(G)</th>
  </tr>
  <tr>
    <td>LLaMA2-7B</td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>64.13</td>
    <td>0.0</td>
    <td>13.20</td>
  </tr>
  <tr>
    <td>w/ Full-finetuning</td>
    <td>16-16-16</td>
    <td>16-16-16</td>
    <td>65.88</td>
    <td>2.4</td>
    <td>38.56</td>
  </tr>
  <tr>
    <td>w/ QLoRA</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>65.69</td>
    <td>3.2</td>
    <td>10.73</td>
  </tr>
  <tr>
    <td>w/ GSQ-Tuning</td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td><b>65.39</b></td>
    <td>3.2</td>
    <td><b>5.97</b></td>
  </tr>
</table>

Fine-tuning Time Comparison. Here, we compare the fine-tuning time with QLoRA, full-finetune, and our proposed GSQ-Tuning on the LLaMA2-7B model. As shown in Tab. 5, the results demonstrate that full fine-tuning achieves 65.88 accuracy in 2.4 hours but requires up to a typically infeasible 58.56GB memory 10x our GSQ-Tuning’s 5.97GB and 5x QLoRA’s 10.73GB. GSQ-Tuning (GSE-INT6) and QLoRA (BF16+NF4) both converge in 3.2 3.4 hours with matching accuracy (65.39% vs 65.69%), yet GSQ-Tuning uses nearly half QLoRA’s memory. It verifies that GSQ-Tuning matches QLoRA’s training time while offering superior memory efficiency, in resource-constrained edge scenarios where our primary focus. Although PEFT methods like ours inherently require longer convergence times, our primary focus lies in striking a balance between memory efficiency and computational feasibility.

Table 6: Fine-tuning training memory (GB) cost with different fine-tuning settings in the LLaMA family.

<table>
  <tr>
    <th>Setting</th>
    <th>LLaMA2-7B</th>
    <th>LLaMA2-13B</th>
    <th>LLaMA2-70B</th>
    <th>LLaMA3-3B</th>
    <th>LLaMA3-8B</th>
  </tr>
  <tr>
    <td>Full-finetuning</td>
    <td>58.56</td>
    <td>112.80</td>
    <td>99.74</td>
    <td>28.12</td>
    <td>65.86</td>
  </tr>
  <tr>
    <td>w/ QLoRA</td>
    <td>10.73</td>
    <td>17.42</td>
    <td>68.82</td>
    <td>6.78</td>
    <td>11.64</td>
  </tr>
  <tr>
    <td>w/ GSQ-Tuning</td>
    <td><b>5.97</b></td>
    <td><b>10.89</b></td>
    <td><b>48.71</b></td>
    <td><b>3.47</b></td>
    <td><b>6.86</b></td>
  </tr>
</table>

Fine-tuning Memory Comparison. Here, as shown in Table 6, we reported the training memory cost for these baselines when training/full fine-tuning (batch size = 1, sequence length=2048), no activation checkpointing and memory offloading as shown in table R1. However, their huge training memory costs (>10x more than GSQ-Tuning) outside our edge-focused fine-tuning scope, where full finetuing/training is typically infeasible due to resource constraints.

Hardware Efficiency Analysis. Table 7 compares the chip area and power consumption of different formats through hardware synthesis. GSE-INT format demonstrates significant advantages over FP: (1) Area Efficiency: GSE-INT6 process engine requires only 0.47mm^2, 10.7× smaller than
Table 7: Comparison of hardware overhead between FP process engine and GSE-INT process engine (7nm).

<table>
  <tr>
    <th>Format</th>
    <th>Area (mm<sup>2</sup>)↓</th>
    <th>Power (W) ↓</th>
  </tr>
  <tr>
    <td>FP8 (E5M2)</td>
    <td>4.36</td>
    <td>2.53</td>
  </tr>
  <tr>
    <td>FP8 (E4M3)</td>
    <td>5.06</td>
    <td>3.23</td>
  </tr>
  <tr>
    <td>FP7 (E3M3)</td>
    <td>5.05</td>
    <td>2.75</td>
  </tr>
  <tr>
    <td>FP6 (E3M2)</td>
    <td>3.40</td>
    <td>2.09</td>
  </tr>
  <tr>
    <td>GSE-INT8</td>
    <td>0.85</td>
    <td>1.24</td>
  </tr>
  <tr>
    <td>GSE-INT7</td>
    <td>0.61</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>GSE-INT6</td>
    <td>0.47</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td>GSE-INT5</td>
    <td><b>0.39</b></td>
    <td><b>0.53</b></td>
  </tr>
</table>

FP8(E4M3). This stems from simplified integer arithmetic logic and group-wise exponent sharing that eliminates complex alignment. (2) Power Superiority: At comparable bit-widths, GSE-INT6 consumes 0.76W (the 23.52% of FP8’s 3.23W).

3.2 Generalization Experiments

Generalization of GSQ-Tuning for Vision-Language Model (LLaVA). Model used is LLaVA-v1.5-7B (Liu et al., 2024b) with Vicuna-7B-v1.5 (Zheng et al., 2023) as language model and CLIP ViT-L-336px (Radford et al., 2021) as vision tower, connected by a 2-layer MLP. Instruction dataset and other settings for finetuning follow the LLaVA official repository, LLaVA-Instruction (Liu et al., 2024c) and the improved one (Liu et al., 2024b). Tab. 3 shows performance drop of the vanilla quantization of 4-bits/64-rank QLoRA, especially, referring to the TextVQA evaluation. Fine-tuning with GSE shows comparable performance compared to that with BF16. BF16 is of E8M7 while GSE is of E5M7, demonstrating the redundancy of the dynamic range w.r.t. exponents is at least 3-bits much. Moreover, the memory cost of GSE is about a half of BF16.

Generalization of GSQ-Tuning on Other Fine-tune Dataset. Here, we also select Commonsense170K (CS170K) (Hu et al., 2023) to evaluate the generalization ability of GSQ-Tuning across different fine-tuing dataset. CS170K is a dataset constructed from the training sets of BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA with pre-defined templates, comprising 170K commonsense reasoning samples. As shown in Tab. 4, on larger fine-tuning datasets, our GSQ-Tuning also demonstrates comparable or even superior accuracy compared to QLoRA, while being more computationally efficient.

3.3 Ablation Study

Group Size Analysis. As shown in Tab. 8, our GSQ-Tuning with 32 groups achieves optimal accuracy-efficiency balance in 6-bit configurations (W6A6G6). The 32-group setting yields significantly higher average accuracy (65.39) compared to 64-group (64.72) and 128-group (64.27) variants, while maintaining comparable memory efficiency (70.96 vs 69.22/69.53). This sweet spot emerges from the tension between quantization bit width and hardware deployment - smaller groups better capture value distributions but increase computation overhead, while larger groups sacrifice adaptation granularity. We therefore adopt group=32 as the default configuration.

Table 8: The effect of the number of shared group on fine-tuning performance in 64 rank setting.

<table>
  <tr>
    <th>Method</th>
    <th>LLMs branch</th>
    <th>low-rank branch</th>
    <th>Group</th>
    <th>Avg.</th>
    <th>Mem. (G)</th>
  </tr>
  <tr>
    <td>LLaMA2-7B</td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>-</td>
    <td>64.13</td>
    <td>13.2</td>
  </tr>
  <tr>
    <td rowspan="3">w/ GSQ-Tuning</td>
    <td>5-6-6</td>
    <td>6-6-6</td>
    <td>32</td>
    <td><b>65.39</b></td>
    <td>6.17</td>
  </tr>
  <tr>
    <td>5-6-6</td>
    <td>6-6-6</td>
    <td>64</td>
    <td>64.72</td>
    <td>6.32</td>
  </tr>
  <tr>
    <td>5-6-6</td>
    <td>6-6-6</td>
    <td>128</td>
    <td>64.27</td>
    <td>6.56</td>
  </tr>
</table>

Table 9: The ablation of the exponent bit width (E) and the number of shared group (N) on fine-tuning performance in 64 rank setting.

<table>
  <tr>
    <th>Method</th>
    <th>LLMs branch</th>
    <th>low-rank branch</th>
    <th>Group</th>
    <th>Avg.</th>
    <th>Mem. (G)</th>
  </tr>
  <tr>
    <td>LLaMA2-7B</td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>-</td>
    <td>64.13</td>
    <td>13.2</td>
  </tr>
  <tr>
    <td rowspan="5">w/ GSQ-Tuning</td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>32</td>
    <td>62.85</td>
    <td>6.29</td>
  </tr>
  <tr>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>64</td>
    <td>62.66</td>
    <td>6.16</td>
  </tr>
  <tr>
    <td>5-6-6</td>
    <td>6-6-6</td>
    <td>32</td>
    <td><b>65.39</b></td>
    <td>6.32</td>
  </tr>
  <tr>
    <td>5-6-6</td>
    <td>6-6-6</td>
    <td>64</td>
    <td>64.72</td>
    <td>6.17</td>
  </tr>
  <tr>
    <td>6-6-6</td>
    <td>6-6-6</td>
    <td>32</td>
    <td>65.41</td>
    <td>6.35</td>
  </tr>
  <tr>
    <td>6-6-6</td>
    <td>6-6-6</td>
    <td>64</td>
    <td>64.99</td>
    <td>6.18</td>
  </tr>
</table>

Exponenet Bits Analysis. As shown in Tab. 9, experiments demonstrate that fixing exponent bits E=5 achieves the optimal trade-off between accuracy and memory overhead. Adopting a smaller exponent (for example, E = 4) leads to significant performance degradation, while increasing E=6 yields marginal performance gains. This ablation study validates the effectiveness of our fixed shared exponent design E = 5.

4 Related Work

Parameter-Efficient Fine-Tuning (PEFT). PEFT reduces memory and computational costs by introducing a small set of trainable parameters while keeping the pretrained model frozen. Approaches including soft prompt tuning (Wang et al., 2023), partial fine-tuning (Fu et al., 2023), and low-rank adaptation (Hu et al., 2021). Among these, LoRA stands out as a seminal work, injecting trainable low-rank matrices into linear layers to enable efficient fine-tuning without modifying the base model weights. QLoRA
extends this with 4-bit NF4 quantization and Double Quantization, supporting 65B model fine-tuning on a single 48GB GPU with minimal performance degradation. Recent work further improves quantization-aware fine-tuning (Xu et al., 2023; Li et al.) and extends LoRA for better efficiency, stability, and performance (Hu et al., 2023; Liu et al., 2024d; Zhao et al., 2024; Hayou et al., 2024; Meng et al., 2024).

Quantization. Much works (Yuan et al., 2023; Yuan et al.; Hu et al., 2024, 2025b,a) make efforts to accelerate the LLMs. For instance, GPTQ (Frantar et al., 2022) quantizes weights to 3-4 bit with slight accuracy drop based on approximate second-order information. AWQ (Lin et al., 2024) and SmoothQuant (Xiao et al., 2022) explore the scheme of smoothing by detecting the importance of different activation channels. Recent works (e.g., Quarot (Ashkboos et al., 2024), SpinQuant (Liu et al., 2024e)) further suppress outliers by utilizing computation-invariant rotation transformation. However, above methods focus on the inference optimization. Leveraging the key benefits of FQT, several studies (Banner et al., 2018; Wu et al., 2018; Langroudi et al., 2019; Yang et al., 2020; Zhu et al., 2020; Xi et al., 2023) have explored its implementation and optimization. However, challenges remain, especially during the backward pass due to the wide dynamic range of gradients. For example, LM-FP8 (Peng et al., 2023) trains LLMs from scratch using FP8, achieving performance comparable to BF16. Some studies have also explored full integer quantization, utilizing efficient hardware implementations. SwitchBack (Wortsman et al., 2023) quantizes partial matrix multiplications with INT8, but it is limited to vision models with up to 1B parameters. Jetfire (Xi et al., 2024) proposes a 2D block-wise quantization approach that maintains accuracy and achieves significant memory savings (1.4-1.5x) when training in INT8. However, these methods have not yet been explored to fine-tuning tasks, and using FQT to lower-bit (< INT8) poses considerable challenges.

5 Conclusion

In this paper, we propose GSQ-Tuning, a resource-efficient framework that addresses the critical challenges of floating-point dependency, privacy risks, and hardware incompatibility in on-device LLM fine-tuning. By integrating Group-Shared Exponents Integer (GSE) quantization with parameter-efficient adaptation, our method achieves three key advancements:(1) Full Integer Pipeline: Eliminates floating-point operations across both forward and backward passes, reducing memory usage by 50% compared to FP16 while maintaining comparable accuracy. (2) Hardware-Optimized Design: The GSE format reduces metadata overhead via group-wise exponent sharing, enabling 5-8bit integer representations. Combined with LoRA-like adapters, this achieves \( 5 \times \) lower power consumption and \( 11 \times \) smaller chip area compared to FP8 at equivalent accuracy levels. (3) Practical Deployment Guidance: A Pareto frontier analysis guides optimal bit-rank configurations for diverse edge constraints. These innovations establish GSQ-Tuning as a foundational step toward democratizing LLM adaptation for resource-constrained environments. This breakthrough makes private, on-device LLM adaptation practical for sensitive applications. Future work will explore sub-4bit quantization to further push the boundaries of edge AI.

6 Limitations

While our GSQ-Tuning significantly advances on-device LLM adaptation through integer-focused optimization and parameter-efficient quantization, two key limitations warrant discussion:

Non-linear Operator Precision. Our current implementation maintains non-linear operations (e.g., LayerNorm, Softmax) in 16-bit to preserve numerical stability. This introduces partial precision conversion overhead during computation. However, non-linear operations do not contain additional learnable parameters and thus do not consume memory. Moreover, these non-linear operations are generally computation-light, making their computational burden negligible. Future work could explore fully integer implementations for non-linear layers.

Bit-Width Range Constraints. The current framework operates effectively in 5-8bit configurations but didn’t present the performance at extreme low bit (\( \leq 4 \)bit) precision. This stems from gradient direction distortion under extreme quantization—a challenge requiring new error compensation mechanisms. We plan to investigate two directions: (1) 4bit stochastic rounding with gradient-aware scaling, and (2) mixed-precision adapters allocating higher bits to critical gradient dimensions.

Furthermore, future work could explore (1) full integer fine-tuning, (2) low-bit quantized fine-tuning and (3) co-design with emerging integer-optimized AI accelerators.
References

ARM. 2020. Cortex-m. In https://developer.arm.com/processors/cortex-m.

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quartot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456.

Mart van Baalen, Andrey Kuzmin, Suparna S Nair, Yuwei Ren, Eric Mahurin, Chirag Patel, Sundar Subramanian, Sanghyuk Lee, Markus Nagel, Joseph Sorriaga, et al. 2023. Fp8 versus int8 for efficient deep learning inference. arXiv preprint arXiv:2303.17951.

Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. 2020. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701.

Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. 2018. Scalable methods for 8-bit training of neural networks. In Advances in Neural Information Processing Systems, pages 5145–5153.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Liannmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys.org (accessed 14 April 2023), 2(3):6.

Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. 2018. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.

Lawrence T Clark, Vinay Vashishtha, Lucian Shifren, Aditya Gujja, Saurabh Sinha, Brian Cline, Chandrasekaran Ramamurthy, and Greg Yeric. 2016. Asap7: A 7-nm finfet predictive process design kit. Microelectronics Journal, 53:105–115.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. In International Conference on Learning Representations.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115.

Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2019a. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. arXiv preprint arXiv:1911.03852.

Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. 2019b. Hawq: Hessian aware quantization of neural networks with mixed-precision. pages 293–302.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. 2019. Learned step size quantization. In International Conference on Learning Representations.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2020. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.

Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. 2023. On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12799–12807.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noach, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Larla Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazheng Lin, Fengwei Yu, and Junjie Yan. 2019. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4852–4861.

Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Xing Hu, Zhixuan Chen, Dawei Yang, Zukang Xu, Chen Xu, Zhihang Yuan, Sifan Zhou, and Jiangyong Yu. 2025a. Moequant: Enhancing quantization for mixture-of-experts large language models via expert-balanced sampling and affinity guidance. arXiv preprint arXiv:2505.03804.

Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. 2025b. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. arXiv preprint arXiv:2501.13987.

Xing Hu, Yuan Cheng, Dawei Yang, Zhihang Yuan, Jiangyong Yu, Chen Xu, and Sifan Zhou. 2024. I-llm: Efficient integer-only inference for fully-quantized low-bit large language models. arXiv preprint arXiv:2405.17849.

Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Yu, Soujanya Poria, and Roy Ka-Wei Lee. 2023. LLM-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing.

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018a. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2704–2713.

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018b. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2021. I-bert: Integer-only bert quantization. In International conference on machine learning, pages 5506–5518. PMLR.

Hamed F Langroudi, Zachariah Carmichael, and Dhireesha Kudithipudi. 2019. Deep learning training on the edge with low-precision posits. arXiv preprint arXiv:1907.13216.

Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-aware quantization for large language models. In The Twelfth International Conference on Learning Representations.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024b. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024c. Visual instruction tuning. Advances in neural information processing systems, 36.

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024d. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, ZhiLin Yang, and Jie Tang. 2023. Gpt understands, too. AI Open.

Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoorst. 2024e. Spinquant-llm quantization with learned rotations. arXiv preprint arXiv:2405.16406.

Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. PiSSA: Principal singular values and singular vectors adaptation of large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. 2022. Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.

Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, et al. 2023. Fp8-lm: Training fp8 large language models.

QUALCOMM. 2024. Qualcomm hexagon npu. In https://www.qualcomm.com/processors/hexagon.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728.

Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020a. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815–8821.

Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020b. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815–8821.

Hanlin Tang, Xipeng Zhang, Kai Liu, Jianchen Zhu, and Zhanhui Kang. 2022. Mkq-bert: Quantized bert with 4-bits weights and activations. arXiv preprint arXiv:2203.13483.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. 2018. Training deep neural networks with 8-bit floating point numbers. In Advances in Neural Information Processing Systems, pages 7675–7684.

Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogério Feris, Huan Sun, and Yoon Kim. 2023. Multitask prompt tuning enables parameter-efficient transfer learning. arXiv preprint arXiv:2303.02861.

Zheng Wang, Juncheng B Li, Shuhui Qu, Florian Metze, and Emma Strubell. 2022. Squat: Sharpness-and quantization-aware training for bert. arXiv preprint arXiv:2210.07171.

Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. 2023. Stable and low-precision training for large-scale vision-language models. Advances in Neural Information Processing Systems, 36:10271–10298.

Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. 2018. Training and inference with integers in deep neural networks. In International Conference on Learning Representations.

Haocheng Xi, Yuxiang Chen, Kang Zhao, Kaijun Zheng, Jianfei Chen, and Jun Zhu. 2024. Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per-block quantization. arXiv preprint arXiv:2403.12422.

Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. 2023. Training transformers with 4-bit integers. Advances in Neural Information Processing Systems, 36:49146–49168.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. 2022. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438.

Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhenhsu Chen, Xiaopeng Zhang, and Qi Tian. 2023. Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717.

Yukuan Yang, Lei Deng, Shuang Wu, Tianyi Yan, Yuan Xie, and Guoqi Li. 2020. Training high-performance and large-scale deep neural networks with full 8-bit integers. Neural Networks, 125:70–82.

Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. 2023. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089.

Zhihang Yuan, Yuzhang Shang, and Zhen Dong. Pb-lm: Partially binarized large language models. In The Twelfth International Conference on Learning Representations.

Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pages 36–39. IEEE.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830.

Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. 2018. LQ-Nets: Learned quantization for highly accurate and compact deep neural networks. In The European Conference on Computer Vision (ECCV).

Sai Qian Zhang, Bradley McDanel, and HT Kung. 2022. Fast: Dnn training under variable precision block floating point with stochastic rounding. In 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 846–860. IEEE.

Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792.

Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020. Ternarybert: Distillation-aware ultra-low bit bert. arXiv preprint arXiv:2009.12812.

Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyuan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand. Association for Computational Linguistics.

Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. 2017. Incremental network quantization: Towards lossless cnns with low-precision weights. International Conference on Learning Representations.

Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuci Yang, and Junjie Yan. 2020. Towards unified int8 training for convolutional neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1969–1979.
A Appendix

A.1 Differences with Quantization-aware training (QAT):

Quantization-aware training (QAT) (Choi et al., 2018; Zhang et al., 2018; Zhou et al., 2017; Jacob et al., 2018a; Dong et al., 2019b,a; Shen et al., 2020a; Zafirri et al., 2019; Shen et al., 2020b; Tang et al., 2022; Zhang et al., 2020; Bai et al., 2020; Foret et al., 2020; Wang et al., 2022) is an inference acceleration technique which trains networks with quantizers inserted in the forward propagation graph, so the trained network can perform efficiently during inference. QAT can compress activation/weights to extremely low precision (e.g. 1-2 bits). It is tempting to think that directly applying a quantizer for QAT to FQT can lead to similar low activation/weights bit-width. However, even only quantizing the forward propagation for FQT is much more challenging than QAT because: ① QAT requires a converged full-precision model as initialization (Esser et al., 2019) and/or as a teacher model for knowledge distillation (Bai et al., 2020); ② QAT may approximate the discrete quantizer with continuous functions during training (Gong et al., 2019), which cannot be implemented with integer arithmetic. Due to these challenges, it is still an open problem to do FQT with low-bit activations/weights.

A.2 Detailed results on different rank setting:

Here, we also report the results of our GSQ-Tuning on different LlaMA model, including LlaMA2-7B (Tab.10), LlaMA2-13B (Tab.11), LlaMA2-70B(Tab.12), LlaMA3-3B(Tab.13), and LlaMA3-8B(Tab.14). The results consistently demonstrated the effectiveness and efficiency of GSQ-Tuning.

A.3 Comparison with FP8 with 64 rank

Here, we compare the designed GSE data format with FP8 in fully quantized fine-tuning framework with 32 rank setting. As shown in Tab. 15, the results still demonstrate that the designed GSE implemented in our GSQ-Tuning method achieves superior fine-tuning performance compared to FP8 while significantly reducing computation efficiency. Even under 5-bit settings, GSQ-Tuning maintains fine-tuning performance on par with FP8, validating its effectiveness.
Table 10: 0-shot commonsense QA accuracy (%) across different bits and rank on llama2-7B.

<table>
  <tr>
    <th>Method</th>
    <th>rank</th>
    <th>LLMs branch</th>
    <th>low-rank branch</th>
    <th>Avg.</th>
    <th>ARC-c</th>
    <th>ARC-e</th>
    <th>BoolQ</th>
    <th>HellaS.</th>
    <th>OBQA</th>
    <th>PIQA</th>
    <th>SCIQ.</th>
    <th>WinoG.</th>
    <th>Mem. (G)</th>
  </tr>
  <tr>
    <td rowspan="4">LLaMA2-7B</td>
    <td>16-16-None</td>
    <td colspan="2">w/o</td>
    <td>64.13</td>
    <td>46.25</td>
    <td>74.62</td>
    <td>77.68</td>
    <td>76.01</td>
    <td>44.20</td>
    <td>79.11</td>
    <td>46.11</td>
    <td>69.06</td>
    <td>13.2</td>
  </tr>
  <tr>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>w/o</td>
    <td>65.05</td>
    <td>47.53</td>
    <td>75.17</td>
    <td>78.59</td>
    <td>76.09</td>
    <td>44.00</td>
    <td>79.54</td>
    <td>49.44</td>
    <td>70.09</td>
    <td>10.18</td>
  </tr>
  <tr>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td></td>
    <td>65.10</td>
    <td>47.53</td>
    <td>74.71</td>
    <td>78.35</td>
    <td>75.99</td>
    <td>45.00</td>
    <td>79.65</td>
    <td>49.28</td>
    <td>70.32</td>
    <td>6.73</td>
  </tr>
  <tr>
    <td>4-7-7</td>
    <td>7-7-7</td>
    <td></td>
    <td>64.96</td>
    <td>47.18</td>
    <td>75.21</td>
    <td>78.10</td>
    <td>75.98</td>
    <td>44.80</td>
    <td>79.27</td>
    <td>49.95</td>
    <td>69.38</td>
    <td>5.98</td>
  </tr>
  <tr>
    <td rowspan="4">w/ QLoRA</td>
    <td>16</td>
    <td>4-6-6</td>
    <td></td>
    <td>64.87</td>
    <td>46.84</td>
    <td>73.78</td>
    <td>78.07</td>
    <td>75.88</td>
    <td>45.80</td>
    <td>79.22</td>
    <td>49.39</td>
    <td>70.01</td>
    <td>5.32</td>
  </tr>
  <tr>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td></td>
    <td>63.97</td>
    <td>46.76</td>
    <td>72.64</td>
    <td>75.78</td>
    <td>74.95</td>
    <td>45.20</td>
    <td>79.05</td>
    <td>48.62</td>
    <td>68.75</td>
    <td>5.27</td>
  </tr>
  <tr>
    <td>32</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td></td>
    <td>65.44</td>
    <td>47.27</td>
    <td>75.04</td>
    <td>78.87</td>
    <td>76.11</td>
    <td>44.60</td>
    <td>79.76</td>
    <td>49.95</td>
    <td>70.32</td>
    <td>10.37</td>
  </tr>
  <tr>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td></td>
    <td>65.45</td>
    <td>48.12</td>
    <td>74.71</td>
    <td>78.38</td>
    <td>76.14</td>
    <td>46.00</td>
    <td>79.71</td>
    <td>49.64</td>
    <td>70.96</td>
    <td>6.92</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td>16</td>
    <td>4-7-7</td>
    <td></td>
    <td>65.43</td>
    <td>47.35</td>
    <td>74.20</td>
    <td>78.99</td>
    <td>75.84</td>
    <td>46.00</td>
    <td>79.92</td>
    <td>49.59</td>
    <td>71.59</td>
    <td>6.16</td>
  </tr>
  <tr>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td></td>
    <td>65.01</td>
    <td>47.44</td>
    <td>74.62</td>
    <td>78.65</td>
    <td>76.03</td>
    <td>44.00</td>
    <td>79.60</td>
    <td>50.05</td>
    <td>69.69</td>
    <td>5.55</td>
  </tr>
  <tr>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td></td>
    <td>64.00</td>
    <td>44.97</td>
    <td>73.32</td>
    <td>75.29</td>
    <td>74.95</td>
    <td>44.60</td>
    <td>79.27</td>
    <td>48.93</td>
    <td>70.24</td>
    <td>5.45</td>
  </tr>
  <tr>
    <td>32</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td></td>
    <td>65.69</td>
    <td>47.14</td>
    <td>74.75</td>
    <td>79.50</td>
    <td>76.46</td>
    <td>45.50</td>
    <td>79.63</td>
    <td>50.26</td>
    <td>71.32</td>
    <td>10.73</td>
  </tr>
  <tr>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td></td>
    <td>65.60</td>
    <td>48.12</td>
    <td>74.24</td>
    <td>79.72</td>
    <td>76.00</td>
    <td>45.80</td>
    <td>79.60</td>
    <td>49.69</td>
    <td>71.67</td>
    <td>7.28</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td>64</td>
    <td>4-7-7</td>
    <td></td>
    <td>65.47</td>
    <td>47.78</td>
    <td>74.71</td>
    <td>79.51</td>
    <td>76.09</td>
    <td>45.80</td>
    <td>79.60</td>
    <td>49.80</td>
    <td>70.48</td>
    <td>6.52</td>
  </tr>
  <tr>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td></td>
    <td>65.39</td>
    <td>47.70</td>
    <td>74.58</td>
    <td>79.24</td>
    <td>76.05</td>
    <td>44.60</td>
    <td>79.60</td>
    <td>50.41</td>
    <td>70.96</td>
    <td>5.97</td>
  </tr>
  <tr>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td></td>
    <td>64.18</td>
    <td>45.14</td>
    <td>72.69</td>
    <td>73.20</td>
    <td>73.27</td>
    <td>46.40</td>
    <td>79.65</td>
    <td>48.62</td>
    <td>70.48</td>
    <td>5.81</td>
  </tr>
  <tr>
    <td>128</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td></td>
    <td>65.84</td>
    <td>48.24</td>
    <td>74.91</td>
    <td>79.78</td>
    <td>76.27</td>
    <td>45.52</td>
    <td>79.77</td>
    <td>50.48</td>
    <td>71.79</td>
    <td>11.46</td>
  </tr>
  <tr>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td></td>
    <td>65.79</td>
    <td>48.12</td>
    <td>74.83</td>
    <td>80.28</td>
    <td>75.36</td>
    <td>45.80</td>
    <td>79.54</td>
    <td>50.61</td>
    <td>71.19</td>
    <td>8.02</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td>128</td>
    <td>4-7-7</td>
    <td></td>
    <td>65.69</td>
    <td>48.04</td>
    <td>74.87</td>
    <td>79.79</td>
    <td>76.08</td>
    <td>45.00</td>
    <td>79.49</td>
    <td>50.61</td>
    <td>71.67</td>
    <td>7.26</td>
  </tr>
  <tr>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td></td>
    <td>65.58</td>
    <td>47.87</td>
    <td>74.54</td>
    <td>80.09</td>
    <td>76.05</td>
    <td>45.40</td>
    <td>79.38</td>
    <td>50.10</td>
    <td>71.27</td>
    <td>6.10</td>
  </tr>
  <tr>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td></td>
    <td>64.46</td>
    <td>46.50</td>
    <td>72.77</td>
    <td>75.99</td>
    <td>75.31</td>
    <td>46.60</td>
    <td>79.00</td>
    <td>48.98</td>
    <td>70.56</td>
    <td>6.14</td>
  </tr>
  <tr>
    <td>256</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td></td>
    <td>66.12</td>
    <td>48.33</td>
    <td>75.00</td>
    <td>80.94</td>
    <td>76.37</td>
    <td>45.61</td>
    <td>79.97</td>
    <td>51.13</td>
    <td>71.64</td>
    <td>12.93</td>
  </tr>
  <tr>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td></td>
    <td>66.19</td>
    <td>48.55</td>
    <td>75.13</td>
    <td>80.76</td>
    <td>76.14</td>
    <td>47.00</td>
    <td>79.38</td>
    <td>50.72</td>
    <td>71.82</td>
    <td>9.47</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td>256</td>
    <td>4-7-7</td>
    <td></td>
    <td>65.96</td>
    <td>48.46</td>
    <td>75.08</td>
    <td>80.43</td>
    <td>76.04</td>
    <td>45.60</td>
    <td>79.76</td>
    <td>50.72</td>
    <td>71.59</td>
    <td>8.42</td>
  </tr>
  <tr>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td></td>
    <td>65.90</td>
    <td>48.38</td>
    <td>74.16</td>
    <td>79.94</td>
    <td>75.81</td>
    <td>46.80</td>
    <td>79.43</td>
    <td>50.87</td>
    <td>71.82</td>
    <td>7.66</td>
  </tr>
  <tr>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td></td>
    <td>64.59</td>
    <td>46.33</td>
    <td>72.60</td>
    <td>76.51</td>
    <td>75.57</td>
    <td>46.40</td>
    <td>79.60</td>
    <td>49.39</td>
    <td>70.32</td>
    <td>6.75</td>
  </tr>
  <tr>
    <td>512</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td></td>
    <td>66.59</td>
    <td>49.26</td>
    <td>75.20</td>
    <td>81.99</td>
    <td>76.06</td>
    <td>46.74</td>
    <td>79.49</td>
    <td>51.71</td>
    <td>72.27</td>
    <td>15.85</td>
  </tr>
  <tr>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td></td>
    <td>66.52</td>
    <td>49.49</td>
    <td>74.92</td>
    <td>81.28</td>
    <td>73.89</td>
    <td>47.60</td>
    <td>79.49</td>
    <td>51.59</td>
    <td>71.90</td>
    <td>11.40</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td>512</td>
    <td>4-7-7</td>
    <td></td>
    <td>66.33</td>
    <td>48.89</td>
    <td>74.75</td>
    <td>81.41</td>
    <td>76.06</td>
    <td>47.00</td>
    <td>79.54</td>
    <td>51.74</td>
    <td>71.27</td>
    <td>9.95</td>
  </tr>
  <tr>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td></td>
    <td>66.31</td>
    <td>48.55</td>
    <td>75.51</td>
    <td>80.80</td>
    <td>76.42</td>
    <td>46.00</td>
    <td>79.60</td>
    <td>51.64</td>
    <td>71.98</td>
    <td>9.19</td>
  </tr>
  <tr>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td></td>
    <td>64.86</td>
    <td>47.44</td>
    <td>73.15</td>
    <td>76.85</td>
    <td>75.62</td>
    <td>47.00</td>
    <td>79.33</td>
    <td>49.18</td>
    <td>70.32</td>
    <td>8.25</td>
  </tr>
</table>
Table 11: 0-shot commonsense QA accuracy (%) across different bits and rank on llama2-13B.

<table>
  <tr>
    <th>Method</th>
    <th>rank</th>
    <th>LLMs branch</th>
    <th>low-rank branch</th>
    <th>Avg.</th>
    <th>ARC-c</th>
    <th>ARC-e</th>
    <th>BoolQ</th>
    <th>HellaS.</th>
    <th>OBQA</th>
    <th>PIQA</th>
    <th>SCIQ.</th>
    <th>WinoG.</th>
    <th>Mem. (G)</th>
  </tr>
  <tr>
    <td>LLaMA2-13B</td>
    <td>-</td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>66.65</td>
    <td>48.81</td>
    <td>76.47</td>
    <td>82.45</td>
    <td>79.67</td>
    <td>44.80</td>
    <td>80.36</td>
    <td>48.31</td>
    <td>72.38</td>
    <td>25.70</td>
  </tr>
  <tr>
    <td rowspan="4">w/ QLoRA</td>
    <td>16</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>67.32</td>
    <td>49.74</td>
    <td>76.98</td>
    <td>82.94</td>
    <td>78.85</td>
    <td>46.00</td>
    <td>80.52</td>
    <td>50.36</td>
    <td>73.16</td>
    <td>16.56</td>
  </tr>
  <tr>
    <td></td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td>67.35</td>
    <td>49.83</td>
    <td>77.06</td>
    <td>83.09</td>
    <td>78.39</td>
    <td>46.00</td>
    <td>80.47</td>
    <td>50.31</td>
    <td>73.16</td>
    <td>11.13</td>
  </tr>
  <tr>
    <td></td>
    <td>4-7-7</td>
    <td>7-7-7</td>
    <td>67.29</td>
    <td>49.91</td>
    <td>76.94</td>
    <td>83.03</td>
    <td>78.90</td>
    <td>45.40</td>
    <td>80.58</td>
    <td>50.61</td>
    <td>73.01</td>
    <td>10.58</td>
  </tr>
  <tr>
    <td></td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>67.23</td>
    <td>49.66</td>
    <td>76.98</td>
    <td>82.75</td>
    <td>78.79</td>
    <td>46.00</td>
    <td>80.47</td>
    <td>50.05</td>
    <td>73.16</td>
    <td>10.03</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td>16</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>67.47</td>
    <td>49.83</td>
    <td>77.02</td>
    <td>83.24</td>
    <td>78.92</td>
    <td>46.20</td>
    <td>80.58</td>
    <td>50.77</td>
    <td>73.24</td>
    <td>16.85</td>
  </tr>
  <tr>
    <td></td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td>67.49</td>
    <td>49.83</td>
    <td>76.88</td>
    <td>83.15</td>
    <td>78.94</td>
    <td>45.60</td>
    <td>80.79</td>
    <td>51.07</td>
    <td>73.56</td>
    <td>11.42</td>
  </tr>
  <tr>
    <td></td>
    <td>4-7-7</td>
    <td>7-7-7</td>
    <td>67.38</td>
    <td>50.17</td>
    <td>77.06</td>
    <td>82.81</td>
    <td>78.99</td>
    <td>45.40</td>
    <td>80.79</td>
    <td>50.46</td>
    <td>74.30</td>
    <td>10.87</td>
  </tr>
  <tr>
    <td></td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>67.35</td>
    <td>49.83</td>
    <td>77.06</td>
    <td>83.09</td>
    <td>78.89</td>
    <td>46.00</td>
    <td>80.47</td>
    <td>50.31</td>
    <td>73.16</td>
    <td>10.31</td>
  </tr>
  <tr>
    <td></td>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td>66.65</td>
    <td>48.88</td>
    <td>76.18</td>
    <td>82.08</td>
    <td>78.07</td>
    <td>45.60</td>
    <td>80.36</td>
    <td>49.74</td>
    <td>72.77</td>
    <td>9.76</td>
  </tr>
  <tr>
    <td rowspan="4">w/ QLoRA</td>
    <td>32</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>67.61</td>
    <td>49.66</td>
    <td>77.23</td>
    <td>83.30</td>
    <td>78.95</td>
    <td>45.40</td>
    <td>80.74</td>
    <td>51.59</td>
    <td>73.24</td>
    <td>17.42</td>
  </tr>
  <tr>
    <td></td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td>67.48</td>
    <td>47.97</td>
    <td>77.40</td>
    <td>82.87</td>
    <td>78.88</td>
    <td>46.20</td>
    <td>80.90</td>
    <td>50.72</td>
    <td>73.32</td>
    <td>11.99</td>
  </tr>
  <tr>
    <td></td>
    <td>4-7-7</td>
    <td>7-7-7</td>
    <td>67.43</td>
    <td>49.74</td>
    <td>77.27</td>
    <td>82.91</td>
    <td>78.98</td>
    <td>46.00</td>
    <td>80.90</td>
    <td>50.61</td>
    <td>73.09</td>
    <td>11.44</td>
  </tr>
  <tr>
    <td></td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>67.35</td>
    <td>49.66</td>
    <td>77.27</td>
    <td>82.75</td>
    <td>78.66</td>
    <td>79.05</td>
    <td>80.90</td>
    <td>50.97</td>
    <td>73.16</td>
    <td>10.89</td>
  </tr>
  <tr>
    <td></td>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td>66.97</td>
    <td>49.91</td>
    <td>76.60</td>
    <td>81.87</td>
    <td>78.15</td>
    <td>46.20</td>
    <td>80.41</td>
    <td>49.54</td>
    <td>73.09</td>
    <td>10.33</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td>64</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>67.61</td>
    <td>50.34</td>
    <td>77.40</td>
    <td>83.35</td>
    <td>78.89</td>
    <td>46.00</td>
    <td>80.85</td>
    <td>50.92</td>
    <td>72.93</td>
    <td>18.56</td>
  </tr>
  <tr>
    <td></td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td>67.62</td>
    <td>50.34</td>
    <td>77.06</td>
    <td>83.18</td>
    <td>78.96</td>
    <td>46.40</td>
    <td>80.69</td>
    <td>50.92</td>
    <td>73.40</td>
    <td>13.14</td>
  </tr>
  <tr>
    <td></td>
    <td>4-7-7</td>
    <td>7-7-7</td>
    <td>67.57</td>
    <td>50.43</td>
    <td>77.36</td>
    <td>83.06</td>
    <td>79.05</td>
    <td>45.60</td>
    <td>80.85</td>
    <td>51.28</td>
    <td>72.93</td>
    <td>12.58</td>
  </tr>
  <tr>
    <td></td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>67.53</td>
    <td>50.43</td>
    <td>77.31</td>
    <td>83.15</td>
    <td>78.81</td>
    <td>45.80</td>
    <td>80.58</td>
    <td>50.97</td>
    <td>73.16</td>
    <td>12.03</td>
  </tr>
  <tr>
    <td></td>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td>67.10</td>
    <td>49.49</td>
    <td>76.81</td>
    <td>82.88</td>
    <td>78.22</td>
    <td>46.40</td>
    <td>80.03</td>
    <td>50.56</td>
    <td>73.24</td>
    <td>11.48</td>
  </tr>
  <tr>
    <td rowspan="4">w/ QLoRA</td>
    <td>128</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>67.91</td>
    <td>50.77</td>
    <td>77.36</td>
    <td>83.64</td>
    <td>78.88</td>
    <td>46.60</td>
    <td>80.74</td>
    <td>51.69</td>
    <td>73.64</td>
    <td>20.85</td>
  </tr>
  <tr>
    <td></td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td>67.84</td>
    <td>51.11</td>
    <td>77.06</td>
    <td>83.82</td>
    <td>78.80</td>
    <td>46.40</td>
    <td>80.69</td>
    <td>52.00</td>
    <td>72.85</td>
    <td>18.42</td>
  </tr>
  <tr>
    <td></td>
    <td>4-7-7</td>
    <td>7-7-7</td>
    <td>67.74</td>
    <td>50.77</td>
    <td>77.31</td>
    <td>83.79</td>
    <td>78.84</td>
    <td>46.00</td>
    <td>80.63</td>
    <td>51.89</td>
    <td>72.69</td>
    <td>14.87</td>
  </tr>
  <tr>
    <td></td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>67.68</td>
    <td>50.77</td>
    <td>77.19</td>
    <td>83.49</td>
    <td>78.82</td>
    <td>46.00</td>
    <td>80.58</td>
    <td>51.38</td>
    <td>73.24</td>
    <td>14.32</td>
  </tr>
  <tr>
    <td></td>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td>67.22</td>
    <td>50.85</td>
    <td>75.84</td>
    <td>81.72</td>
    <td>78.21</td>
    <td>46.00</td>
    <td>80.36</td>
    <td>50.92</td>
    <td>73.48</td>
    <td>13.76</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td>256</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>67.94</td>
    <td>50.60</td>
    <td>77.48</td>
    <td>83.88</td>
    <td>79.00</td>
    <td>46.40</td>
    <td>80.74</td>
    <td>52.05</td>
    <td>73.40</td>
    <td>25.43</td>
  </tr>
  <tr>
    <td></td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td>67.92</td>
    <td>51.02</td>
    <td>77.27</td>
    <td>83.27</td>
    <td>79.04</td>
    <td>46.40</td>
    <td>81.01</td>
    <td>51.79</td>
    <td>73.56</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td></td>
    <td>4-7-7</td>
    <td>7-7-7</td>
    <td>67.90</td>
    <td>51.19</td>
    <td>77.15</td>
    <td>83.79</td>
    <td>78.82</td>
    <td>46.80</td>
    <td>80.69</td>
    <td>51.79</td>
    <td>73.01</td>
    <td>19.45</td>
  </tr>
  <tr>
    <td></td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>67.82</td>
    <td>51.02</td>
    <td>77.02</td>
    <td>83.85</td>
    <td>78.93</td>
    <td>46.20</td>
    <td>80.90</td>
    <td>51.54</td>
    <td>73.09</td>
    <td>18.89</td>
  </tr>
  <tr>
    <td></td>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td>67.39</td>
    <td>50.94</td>
    <td>76.68</td>
    <td>82.29</td>
    <td>78.39</td>
    <td>46.20</td>
    <td>80.41</td>
    <td>51.69</td>
    <td>72.53</td>
    <td>18.34</td>
  </tr>
</table>

Table 12: 0-shot commonsense QA accuracy (%) across different bits and rank on llama2-70B.

<table>
  <tr>
    <th>Method</th>
    <th>rank</th>
    <th>LLMs branch</th>
    <th>low-rank branch</th>
    <th>Avg.</th>
    <th>ARC-c</th>
    <th>ARC-e</th>
    <th>BoolQ</th>
    <th>HellaS.</th>
    <th>OBQA</th>
    <th>PIQA</th>
    <th>SCIQ.</th>
    <th>WinoG.</th>
    <th>Mem. (G)</th>
  </tr>
  <tr>
    <td>LLaMA2-70B</td>
    <td>-</td>
    <td>16-16-None</td>
    <td>w/o</td>
    <td>70.68</td>
    <td>56.91</td>
    <td>80.05</td>
    <td>85.78</td>
    <td>83.59</td>
    <td>48.60</td>
    <td>82.48</td>
    <td>48.67</td>
    <td>79.40</td>
    <td>137.42</td>
  </tr>
  <tr>
    <td rowspan="4">w/ QLoRA</td>
    <td>16</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>71.72</td>
    <td>58.62</td>
    <td>81.44</td>
    <td>86.39</td>
    <td>83.92</td>
    <td>49.80</td>
    <td>83.03</td>
    <td>50.46</td>
    <td>80.11</td>
    <td>63.90</td>
  </tr>
  <tr>
    <td></td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td>71.65</td>
    <td>58.62</td>
    <td>81.23</td>
    <td>86.36</td>
    <td>83.89</td>
    <td>49.60</td>
    <td>83.19</td>
    <td>50.41</td>
    <td>79.95</td>
    <td>49.17</td>
  </tr>
  <tr>
    <td></td>
    <td>4-7-7</td>
    <td>7-7-7</td>
    <td>71.63</td>
    <td>58.87</td>
    <td>81.57</td>
    <td>86.24</td>
    <td>83.89</td>
    <td>49.20</td>
    <td>83.19</td>
    <td>50.46</td>
    <td>79.64</td>
    <td>47.44</td>
  </tr>
  <tr>
    <td></td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>71.58</td>
    <td>58.62</td>
    <td>81.36</td>
    <td>86.15</td>
    <td>83.84</td>
    <td>49.60</td>
    <td>82.97</td>
    <td>50.41</td>
    <td>79.64</td>
    <td>45.72</td>
  </tr>
  <tr>
    <td></td>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td>71.02</td>
    <td>57.34</td>
    <td>80.56</td>
    <td>83.93</td>
    <td>49.00</td>
    <td>82.59</td>
    <td>49.33</td>
    <td>79.64</td>
    <td>43.99</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td>16</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>72.22</td>
    <td>59.81</td>
    <td>82.20</td>
    <td>86.51</td>
    <td>83.89</td>
    <td>50.40</td>
    <td>83.13</td>
    <td>51.48</td>
    <td>80.35</td>
    <td>66.82</td>
  </tr>
  <tr>
    <td></td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td>72.20</td>
    <td>59.90</td>
    <td>82.32</td>
    <td>86.51</td>
    <td>83.90</td>
    <td>50.20</td>
    <td>83.08</td>
    <td>51.59</td>
    <td>80.11</td>
    <td>32.17</td>
  </tr>
  <tr>
    <td></td>
    <td>4-7-7</td>
    <td>7-7-7</td>
    <td>72.18</td>
    <td>59.81</td>
    <td>82.28</td>
    <td>86.39</td>
    <td>83.88</td>
    <td>50.20</td>
    <td>83.13</td>
    <td>51.54</td>
    <td>80.19</td>
    <td>50.44</td>
  </tr>
  <tr>
    <td></td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>72.10</td>
    <td>59.91</td>
    <td>82.15</td>
    <td>86.51</td>
    <td>83.94</td>
    <td>50.00</td>
    <td>83.30</td>
    <td>50.82</td>
    <td>80.58</td>
    <td>48.71</td>
  </tr>
  <tr>
    <td></td>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td>71.70</td>
    <td>58.77</td>
    <td>81.48</td>
    <td>85.90</td>
    <td>83.91</td>
    <td>49.60</td>
    <td>82.81</td>
    <td>50.67</td>
    <td>80.46</td>
    <td>46.98</td>
  </tr>
  <tr>
    <td rowspan="4">w/ QLoRA</td>
    <td>32</td>
    <td>4-16-16</td>
    <td>16-16-16</td>
    <td>72.39</td>
    <td>60.67</td>
    <td>82.37</td>
    <td>86.84</td>
    <td>84.05</td>
    <td>49.20</td>
    <td>83.19</td>
    <td>52.15</td>
    <td>80.66</td>
    <td>70.96</td>
  </tr>
  <tr>
    <td></td>
    <td>4-8-8</td>
    <td>8-8-8</td>
    <td>72.37</td>
    <td>60.75</td>
    <td>82.49</td>
    <td>87.00</td>
    <td>83.94</td>
    <td>49.40</td>
    <td>83.08</td>
    <td>52.15</td>
    <td>80.19</td>
    <td>56.16</td>
  </tr>
  <tr>
    <td></td>
    <td>4-7-7</td>
    <td>7-7-7</td>
    <td>72.32</td>
    <td>60.41</td>
    <td>82.45</td>
    <td>86.94</td>
    <td>83.94</td>
    <td>49.60</td>
    <td>83.08</td>
    <td>52.15</td>
    <td>80.58</td>
    <td>54.43</td>
  </tr>
  <tr>
    <td></td>
    <td>4-6-6</td>
    <td>6-6-6</td>
    <td>72.28</td>
    <td>59.81</td>
    <td>82.45</td>
    <td>86.91</td>
    <td>83.99</td>
    <td>49.60</td>
    <td>83.35</td>
    <td>51.89</td>
    <td>80.27</td>
    <td>52.70</td>
  </tr>
  <tr>
    <td></td>
    <td>4-5-5</td>
    <td>5-5-5</td>
    <td>71.85</td>
    <td>59.47</td>
    <td>81.90</td>
    <td>86.48</td>
    <td>83.82</td>
    <td>48.20</td>
    <td>83.08</td>
    <td>51.02</td>
    <td>80.82</td>
    <td>50.97</td>
  </tr>
</table>
Table 13: 0-shot commonsense QA accuracy (%) across different bits and rank on Llama3-3B.

<table>
  <tr>
    <th>Method</th>
    <th>rank</th>
    <th>#Bits</th>
    <th>Avg.</th>
    <th>ARC-c</th>
    <th>ARC-e</th>
    <th>BoolQ</th>
    <th>HellaS.</th>
    <th>OBQA</th>
    <th>PIQA</th>
    <th>SCIQ.</th>
    <th>WinoG.</th>
    <th>Mem. (G)</th>
  </tr>
  <tr>
    <td rowspan="2">LLaMA3-3B</td>
    <td>-</td>
    <td>16-16-None</td>
    <td>64.13</td>
    <td>46.25</td>
    <td>74.62</td>
    <td>77.68</td>
    <td>76.01</td>
    <td>44.20</td>
    <td>79.11</td>
    <td>46.11</td>
    <td>69.06</td>
    <td>6.42</td>
  </tr>
  <tr>
    <td>w/ QLoRA</td>
    <td>16</td>
    <td>4-16-16</td>
    <td>65.05</td>
    <td>47.53</td>
    <td>75.17</td>
    <td>78.59</td>
    <td>76.09</td>
    <td>44.00</td>
    <td>79.54</td>
    <td>49.44</td>
    <td>70.09</td>
    <td>6.42</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td></td>
    <td>8-8-8</td>
    <td>65.10</td>
    <td>47.53</td>
    <td>74.71</td>
    <td>78.35</td>
    <td>75.99</td>
    <td>45.00</td>
    <td>79.65</td>
    <td>49.28</td>
    <td>70.32</td>
    <td>3.57</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>64.96</td>
    <td>47.18</td>
    <td>75.21</td>
    <td>78.10</td>
    <td>75.98</td>
    <td>44.80</td>
    <td>79.27</td>
    <td>49.95</td>
    <td>69.38</td>
    <td>3.34</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>64.87</td>
    <td>46.84</td>
    <td>73.78</td>
    <td>78.07</td>
    <td>75.88</td>
    <td>45.80</td>
    <td>79.22</td>
    <td>49.39</td>
    <td>70.01</td>
    <td>3.11</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>63.97</td>
    <td>46.76</td>
    <td>72.64</td>
    <td>75.78</td>
    <td>74.95</td>
    <td>45.20</td>
    <td>79.05</td>
    <td>48.62</td>
    <td>68.75</td>
    <td>2.88</td>
  </tr>
  <tr>
    <td rowspan="2">w/ QLoRA</td>
    <td>32</td>
    <td>4-16-16</td>
    <td>65.24</td>
    <td>47.27</td>
    <td>75.04</td>
    <td>78.87</td>
    <td>76.11</td>
    <td>44.60</td>
    <td>79.76</td>
    <td>49.95</td>
    <td>70.32</td>
    <td>6.54</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td></td>
    <td>8-8-8</td>
    <td>65.45</td>
    <td>48.12</td>
    <td>74.71</td>
    <td>78.38</td>
    <td>76.14</td>
    <td>46.00</td>
    <td>79.71</td>
    <td>49.64</td>
    <td>70.96</td>
    <td>3.69</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>65.43</td>
    <td>47.35</td>
    <td>74.20</td>
    <td>78.99</td>
    <td>75.84</td>
    <td>46.00</td>
    <td>79.92</td>
    <td>49.59</td>
    <td>71.59</td>
    <td>3.46</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>65.01</td>
    <td>47.44</td>
    <td>74.62</td>
    <td>78.65</td>
    <td>76.03</td>
    <td>44.00</td>
    <td>79.60</td>
    <td>50.05</td>
    <td>69.69</td>
    <td>3.23</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>64.00</td>
    <td>44.97</td>
    <td>73.32</td>
    <td>75.29</td>
    <td>74.95</td>
    <td>44.60</td>
    <td>79.27</td>
    <td>48.93</td>
    <td>70.24</td>
    <td>3.00</td>
  </tr>
  <tr>
    <td rowspan="2">w/ QLoRA</td>
    <td>64</td>
    <td>4-16-16</td>
    <td>65.69</td>
    <td>47.14</td>
    <td>74.75</td>
    <td>79.50</td>
    <td>76.46</td>
    <td>45.50</td>
    <td>79.63</td>
    <td>50.26</td>
    <td>71.32</td>
    <td>6.78</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td></td>
    <td>8-8-8</td>
    <td>65.60</td>
    <td>48.12</td>
    <td>74.24</td>
    <td>79.72</td>
    <td>76.00</td>
    <td>45.80</td>
    <td>79.60</td>
    <td>49.69</td>
    <td>71.67</td>
    <td>3.93</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>65.47</td>
    <td>47.78</td>
    <td>74.71</td>
    <td>79.51</td>
    <td>76.09</td>
    <td>45.80</td>
    <td>79.60</td>
    <td>49.80</td>
    <td>70.48</td>
    <td>3.70</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>65.39</td>
    <td>47.70</td>
    <td>74.58</td>
    <td>79.24</td>
    <td>76.05</td>
    <td>44.60</td>
    <td>79.60</td>
    <td>50.41</td>
    <td>70.96</td>
    <td>3.47</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>64.18</td>
    <td>45.14</td>
    <td>72.69</td>
    <td>75.20</td>
    <td>75.27</td>
    <td>46.40</td>
    <td>79.65</td>
    <td>48.62</td>
    <td>70.48</td>
    <td>3.24</td>
  </tr>
  <tr>
    <td rowspan="2">w/ QLoRA</td>
    <td>128</td>
    <td>4-16-16</td>
    <td>65.84</td>
    <td>48.24</td>
    <td>74.91</td>
    <td>79.78</td>
    <td>76.27</td>
    <td>45.52</td>
    <td>79.77</td>
    <td>50.48</td>
    <td>71.79</td>
    <td>6.76</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td></td>
    <td>8-8-8</td>
    <td>65.79</td>
    <td>48.12</td>
    <td>74.83</td>
    <td>80.28</td>
    <td>75.96</td>
    <td>45.80</td>
    <td>79.54</td>
    <td>50.61</td>
    <td>71.19</td>
    <td>4.41</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>65.69</td>
    <td>48.04</td>
    <td>74.87</td>
    <td>79.79</td>
    <td>76.08</td>
    <td>45.00</td>
    <td>79.49</td>
    <td>50.61</td>
    <td>71.67</td>
    <td>4.18</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>65.58</td>
    <td>47.87</td>
    <td>74.54</td>
    <td>80.09</td>
    <td>76.05</td>
    <td>45.40</td>
    <td>79.38</td>
    <td>50.10</td>
    <td>71.27</td>
    <td>3.95</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>64.46</td>
    <td>46.50</td>
    <td>72.77</td>
    <td>75.99</td>
    <td>75.31</td>
    <td>46.60</td>
    <td>79.00</td>
    <td>48.98</td>
    <td>70.56</td>
    <td>3.72</td>
  </tr>
  <tr>
    <td rowspan="2">w/ QLoRA</td>
    <td>256</td>
    <td>4-16-16</td>
    <td>66.12</td>
    <td>48.33</td>
    <td>75.00</td>
    <td>80.94</td>
    <td>76.37</td>
    <td>45.61</td>
    <td>79.97</td>
    <td>51.13</td>
    <td>71.64</td>
    <td>7.61</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td></td>
    <td>8-8-8</td>
    <td>66.19</td>
    <td>48.55</td>
    <td>75.13</td>
    <td>80.76</td>
    <td>76.14</td>
    <td>47.00</td>
    <td>79.38</td>
    <td>50.72</td>
    <td>71.82</td>
    <td>5.37</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>65.96</td>
    <td>48.46</td>
    <td>75.08</td>
    <td>80.43</td>
    <td>76.04</td>
    <td>45.60</td>
    <td>79.76</td>
    <td>50.72</td>
    <td>71.59</td>
    <td>5.13</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>65.90</td>
    <td>48.38</td>
    <td>74.16</td>
    <td>79.94</td>
    <td>75.81</td>
    <td>46.80</td>
    <td>79.43</td>
    <td>50.87</td>
    <td>71.82</td>
    <td>4.90</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>64.59</td>
    <td>46.33</td>
    <td>72.60</td>
    <td>76.51</td>
    <td>75.57</td>
    <td>46.40</td>
    <td>79.60</td>
    <td>49.39</td>
    <td>70.32</td>
    <td>4.67</td>
  </tr>
  <tr>
    <td rowspan="2">w/ QLoRA</td>
    <td>512</td>
    <td>4-16-16</td>
    <td>66.59</td>
    <td>49.26</td>
    <td>75.20</td>
    <td>81.99</td>
    <td>76.06</td>
    <td>46.74</td>
    <td>79.49</td>
    <td>51.71</td>
    <td>72.27</td>
    <td>9.73</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td></td>
    <td>8-8-8</td>
    <td>66.52</td>
    <td>49.49</td>
    <td>74.92</td>
    <td>81.28</td>
    <td>75.89</td>
    <td>47.60</td>
    <td>79.49</td>
    <td>51.59</td>
    <td>71.90</td>
    <td>7.28</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>66.33</td>
    <td>48.89</td>
    <td>74.75</td>
    <td>81.41</td>
    <td>76.06</td>
    <td>47.00</td>
    <td>79.54</td>
    <td>51.74</td>
    <td>71.27</td>
    <td>7.05</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>66.31</td>
    <td>48.55</td>
    <td>75.51</td>
    <td>80.80</td>
    <td>76.42</td>
    <td>46.00</td>
    <td>79.60</td>
    <td>51.64</td>
    <td>71.98</td>
    <td>6.82</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>64.86</td>
    <td>47.44</td>
    <td>73.15</td>
    <td>76.85</td>
    <td>75.62</td>
    <td>47.00</td>
    <td>79.33</td>
    <td>49.18</td>
    <td>70.32</td>
    <td>6.59</td>
  </tr>
</table>
Table 14: 0-shot commonsense QA accuracy (%) across different bits and rank on llama3-8B.

<table>
  <tr>
    <th>Method</th>
    <th>rank</th>
    <th>#Bits</th>
    <th>Avg.</th>
    <th>ARC-c</th>
    <th>ARC-e</th>
    <th>BoolQ</th>
    <th>HellaS.</th>
    <th>OBQA</th>
    <th>PIQA</th>
    <th>SCIQ.</th>
    <th>WinoG.</th>
    <th>Mem. (G)</th>
  </tr>
  <tr>
    <td rowspan="2">LLaMA3-8B</td>
    <td>-</td>
    <td>16-16-None</td>
    <td>67.18</td>
    <td>53.50</td>
    <td>77.74</td>
    <td>81.13</td>
    <td>79.20</td>
    <td>45.00</td>
    <td>80.63</td>
    <td>47.03</td>
    <td>73.24</td>
    <td>15.01</td>
  </tr>
  <tr>
    <td>w/ <i>QLoRA</i></td>
    <td>16</td>
    <td>4-16-16</td>
    <td>68.14</td>
    <td>54.52</td>
    <td>79.50</td>
    <td>83.43</td>
    <td>78.66</td>
    <td>44.80</td>
    <td>80.85</td>
    <td>50.00</td>
    <td>73.32</td>
    <td>10.71</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td></td>
    <td>8-8-8</td>
    <td>68.16</td>
    <td>54.61</td>
    <td>79.84</td>
    <td>83.70</td>
    <td>78.58</td>
    <td>44.80</td>
    <td>80.79</td>
    <td>49.85</td>
    <td>73.16</td>
    <td>7.03</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>68.00</td>
    <td>54.01</td>
    <td>79.29</td>
    <td>83.46</td>
    <td>78.65</td>
    <td>45.00</td>
    <td>80.85</td>
    <td>49.80</td>
    <td>73.01</td>
    <td>6.65</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>67.74</td>
    <td>54.01</td>
    <td>78.70</td>
    <td>83.09</td>
    <td>78.49</td>
    <td>44.00</td>
    <td>80.90</td>
    <td>49.44</td>
    <td>73.32</td>
    <td>6.26</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>66.51</td>
    <td>51.54</td>
    <td>77.27</td>
    <td>81.99</td>
    <td>77.00</td>
    <td>44.40</td>
    <td>78.84</td>
    <td>48.46</td>
    <td>72.61</td>
    <td>5.87</td>
  </tr>
  <tr>
    <td rowspan="4">w/ <i>QLoRA</i></td>
    <td>32</td>
    <td>4-16-16</td>
    <td>68.31</td>
    <td>55.55</td>
    <td>80.39</td>
    <td>83.36</td>
    <td>78.65</td>
    <td>44.60</td>
    <td>81.28</td>
    <td>50.05</td>
    <td>72.61</td>
    <td>11.02</td>
  </tr>
  <tr>
    <td></td>
    <td>8-8-8</td>
    <td>68.45</td>
    <td>55.72</td>
    <td>80.23</td>
    <td>83.43</td>
    <td>78.60</td>
    <td>45.00</td>
    <td>81.18</td>
    <td>50.20</td>
    <td>73.32</td>
    <td>7.23</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>68.29</td>
    <td>54.95</td>
    <td>80.13</td>
    <td>83.36</td>
    <td>78.53</td>
    <td>44.80</td>
    <td>81.01</td>
    <td>50.20</td>
    <td>73.32</td>
    <td>6.84</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>68.08</td>
    <td>55.29</td>
    <td>79.29</td>
    <td>83.55</td>
    <td>78.28</td>
    <td>45.80</td>
    <td>81.07</td>
    <td>49.39</td>
    <td>71.98</td>
    <td>6.46</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>66.48</td>
    <td>51.71</td>
    <td>77.69</td>
    <td>82.11</td>
    <td>76.91</td>
    <td>44.20</td>
    <td>79.43</td>
    <td>48.16</td>
    <td>71.67</td>
    <td>6.07</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td>64</td>
    <td>4-16-16</td>
    <td>68.45</td>
    <td>55.63</td>
    <td>80.13</td>
    <td>83.67</td>
    <td>78.78</td>
    <td>44.80</td>
    <td>81.28</td>
    <td>50.41</td>
    <td>72.93</td>
    <td>11.64</td>
  </tr>
  <tr>
    <td></td>
    <td>8-8-8</td>
    <td>68.61</td>
    <td>55.97</td>
    <td>80.22</td>
    <td>83.61</td>
    <td>78.68</td>
    <td>45.20</td>
    <td>81.50</td>
    <td>50.41</td>
    <td>73.32</td>
    <td>7.63</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>68.57</td>
    <td>55.97</td>
    <td>80.68</td>
    <td>83.73</td>
    <td>78.84</td>
    <td>45.20</td>
    <td>81.01</td>
    <td>50.26</td>
    <td>72.85</td>
    <td>7.24</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>68.22</td>
    <td>55.55</td>
    <td>79.29</td>
    <td>83.67</td>
    <td>78.47</td>
    <td>44.80</td>
    <td>80.90</td>
    <td>50.05</td>
    <td>73.09</td>
    <td>6.86</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>66.69</td>
    <td>54.10</td>
    <td>77.99</td>
    <td>81.65</td>
    <td>77.12</td>
    <td>43.80</td>
    <td>79.54</td>
    <td>47.90</td>
    <td>71.43</td>
    <td>6.47</td>
  </tr>
  <tr>
    <td rowspan="4">w/ <i>QLoRA</i></td>
    <td>128</td>
    <td>4-16-16</td>
    <td>68.77</td>
    <td>56.14</td>
    <td>80.56</td>
    <td>83.98</td>
    <td>79.03</td>
    <td>45.60</td>
    <td>81.34</td>
    <td>50.56</td>
    <td>72.93</td>
    <td>12.13</td>
  </tr>
  <tr>
    <td></td>
    <td>8-8-8</td>
    <td>68.72</td>
    <td>56.57</td>
    <td>80.22</td>
    <td>83.82</td>
    <td>78.80</td>
    <td>45.40</td>
    <td>81.25</td>
    <td>50.41</td>
    <td>73.32</td>
    <td>8.43</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>68.71</td>
    <td>56.48</td>
    <td>80.18</td>
    <td>83.88</td>
    <td>78.78</td>
    <td>45.80</td>
    <td>81.34</td>
    <td>50.36</td>
    <td>72.93</td>
    <td>8.04</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>68.67</td>
    <td>56.91</td>
    <td>79.50</td>
    <td>83.79</td>
    <td>78.71</td>
    <td>46.60</td>
    <td>80.52</td>
    <td>50.36</td>
    <td>73.01</td>
    <td>7.66</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>66.92</td>
    <td>52.47</td>
    <td>78.45</td>
    <td>82.63</td>
    <td>77.22</td>
    <td>44.60</td>
    <td>79.49</td>
    <td>48.52</td>
    <td>71.98</td>
    <td>7.27</td>
  </tr>
  <tr>
    <td rowspan="4">w/ GSQ-Tuning</td>
    <td>256</td>
    <td>4-16-16</td>
    <td>69.09</td>
    <td>56.74</td>
    <td>80.35</td>
    <td>84.56</td>
    <td>79.02</td>
    <td>45.20</td>
    <td>81.83</td>
    <td>50.92</td>
    <td>74.11</td>
    <td>13.81</td>
  </tr>
  <tr>
    <td></td>
    <td>8-8-8</td>
    <td>69.04</td>
    <td>56.57</td>
    <td>80.85</td>
    <td>84.07</td>
    <td>78.97</td>
    <td>45.40</td>
    <td>81.45</td>
    <td>51.28</td>
    <td>73.72</td>
    <td>10.03</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>69.00</td>
    <td>56.83</td>
    <td>80.89</td>
    <td>84.25</td>
    <td>78.96</td>
    <td>45.60</td>
    <td>81.50</td>
    <td>50.46</td>
    <td>73.56</td>
    <td>9.64</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>68.84</td>
    <td>56.74</td>
    <td>79.80</td>
    <td>83.98</td>
    <td>78.84</td>
    <td>46.40</td>
    <td>81.12</td>
    <td>50.77</td>
    <td>73.09</td>
    <td>9.26</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>67.54</td>
    <td>53.33</td>
    <td>78.49</td>
    <td>83.21</td>
    <td>77.38</td>
    <td>44.60</td>
    <td>79.98</td>
    <td>48.93</td>
    <td>73.64</td>
    <td>8.87</td>
  </tr>
  <tr>
    <td rowspan="4">w/ <i>QLoRA</i></td>
    <td>512</td>
    <td>4-16-16</td>
    <td>69.18</td>
    <td>57.17</td>
    <td>80.30</td>
    <td>84.65</td>
    <td>79.28</td>
    <td>46.40</td>
    <td>81.07</td>
    <td>50.36</td>
    <td>74.27</td>
    <td>16.81</td>
  </tr>
  <tr>
    <td></td>
    <td>8-8-8</td>
    <td>69.24</td>
    <td>56.48</td>
    <td>80.47</td>
    <td>85.35</td>
    <td>79.13</td>
    <td>45.40</td>
    <td>81.56</td>
    <td>51.54</td>
    <td>74.03</td>
    <td>13.23</td>
  </tr>
  <tr>
    <td></td>
    <td>7-7-7</td>
    <td>69.16</td>
    <td>56.40</td>
    <td>80.68</td>
    <td>85.26</td>
    <td>79.10</td>
    <td>45.80</td>
    <td>81.28</td>
    <td>51.13</td>
    <td>73.64</td>
    <td>12.84</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>69.01</td>
    <td>56.57</td>
    <td>80.01</td>
    <td>84.56</td>
    <td>78.84</td>
    <td>45.80</td>
    <td>81.23</td>
    <td>51.64</td>
    <td>73.48</td>
    <td>12.45</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>67.90</td>
    <td>54.38</td>
    <td>78.60</td>
    <td>83.97</td>
    <td>78.08</td>
    <td>45.30</td>
    <td>80.24</td>
    <td>50.00</td>
    <td>72.76</td>
    <td>12.07</td>
  </tr>
</table>

Table 15: 0-shot accuracy comparison with FP8 in different quantization bits in 64 rank setting.

<table>
  <tr>
    <th>Method</th>
    <th>#Bits</th>
    <th>Avg.</th>
    <th>ARC-c</th>
    <th>ARC-e</th>
    <th>BoolQ</th>
    <th>HellaS.</th>
    <th>OBQA</th>
    <th>PIQA</th>
    <th>SCIQ.</th>
    <th>WinoG.</th>
    <th>Mem. (G)</th>
  </tr>
  <tr>
    <td rowspan="2">LLaMA2-7B</td>
    <td>16-16-None</td>
    <td>64.13</td>
    <td>46.25</td>
    <td>74.62</td>
    <td>77.68</td>
    <td>76.01</td>
    <td>44.20</td>
    <td>79.11</td>
    <td>46.11</td>
    <td>69.06</td>
    <td>13.20</td>
  </tr>
  <tr>
    <td>w/ <i>QLoRA</i></td>
    <td>4-16-16</td>
    <td>65.69</td>
    <td>47.14</td>
    <td>74.75</td>
    <td>79.50</td>
    <td>76.46</td>
    <td>45.50</td>
    <td>79.63</td>
    <td>50.26</td>
    <td>71.32</td>
    <td>9.73</td>
  </tr>
  <tr>
    <td rowspan="4">w/ FP8</td>
    <td>8-8-8</td>
    <td>64.46</td>
    <td>46.84</td>
    <td>73.61</td>
    <td>77.83</td>
    <td>76.03</td>
    <td>44.60</td>
    <td>79.65</td>
    <td>47.80</td>
    <td>69.38</td>
    <td>6.88</td>
  </tr>
  <tr>
    <td></td>
    <td>8-8-8</td>
    <td>65.60</td>
    <td>48.12</td>
    <td>74.24</td>
    <td>79.72</td>
    <td>76.00</td>
    <td>45.80</td>
    <td>79.60</td>
    <td>49.69</td>
    <td>71.67</td>
    <td>6.88</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>65.39</td>
    <td>47.70</td>
    <td>74.58</td>
    <td>79.24</td>
    <td>76.05</td>
    <td>44.60</td>
    <td>79.60</td>
    <td>50.41</td>
    <td>70.96</td>
    <td>6.17</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>64.18</td>
    <td>45.14</td>
    <td>72.69</td>
    <td>75.20</td>
    <td>75.27</td>
    <td>46.40</td>
    <td>79.65</td>
    <td>48.62</td>
    <td>70.48</td>
    <td>5.81</td>
  </tr>
  <tr>
    <td rowspan="2">LLaMA3-8B</td>
    <td>16-16-None</td>
    <td>67.18</td>
    <td>53.50</td>
    <td>77.74</td>
    <td>81.13</td>
    <td>79.20</td>
    <td>45.00</td>
    <td>80.63</td>
    <td>47.03</td>
    <td>73.24</td>
    <td>15.01</td>
  </tr>
  <tr>
    <td>w/ <i>QLoRA</i></td>
    <td>4-16-16</td>
    <td>68.45</td>
    <td>55.63</td>
    <td>80.13</td>
    <td>83.67</td>
    <td>78.78</td>
    <td>44.80</td>
    <td>81.28</td>
    <td>50.41</td>
    <td>72.93</td>
    <td>11.71</td>
  </tr>
  <tr>
    <td rowspan="4">w/ FP8</td>
    <td>8-8-8</td>
    <td>66.46</td>
    <td>50.77</td>
    <td>76.39</td>
    <td>81.38</td>
    <td>78.19</td>
    <td>43.40</td>
    <td>79.92</td>
    <td>47.29</td>
    <td>74.35</td>
    <td>7.63</td>
  </tr>
  <tr>
    <td></td>
    <td>8-8-8</td>
    <td>68.61</td>
    <td>55.97</td>
    <td>80.22</td>
    <td>83.61</td>
    <td>78.68</td>
    <td>45.20</td>
    <td>81.50</td>
    <td>50.41</td>
    <td>73.32</td>
    <td>7.63</td>
  </tr>
  <tr>
    <td></td>
    <td>6-6-6</td>
    <td>68.22</td>
    <td>55.55</td>
    <td>79.29</td>
    <td>83.67</td>
    <td>78.47</td>
    <td>44.80</td>
    <td>80.90</td>
    <td>50.05</td>
    <td>73.09</td>
    <td>6.86</td>
  </tr>
  <tr>
    <td></td>
    <td>5-5-5</td>
    <td>66.69</td>
    <td>54.10</td>
    <td>77.99</td>
    <td>81.65</td>
    <td>77.12</td>
    <td>43.80</td>
    <td>79.54</td>
    <td>47.90</td>
    <td>71.43</td>
    <td>6.47</td>
  </tr>
</table>

Reviewer paper summaries: This paper presents GSQ-Tuning, a novel method for fully integer-based fine-tuning of Large Language Models (LLMs) on resource-constrained edge devices. Traditional fine-tuning approaches depend heavily on floating-point (FP) computations, making them memory- and power-intensive—thus unsuitable for on-device use and privacy-sensitive applications. To address this, GSQ-Tuning introduces the Group-Shared Exponents (GSE) Integer format, which replaces FP representations with a low-bit integer format that shares exponents within groups of parameters.

This paper presents GSQ-Tuning, a novel parameter-efficient finetuning method built upon QLoRA with the newly designed GSQ (Group-Shared Exponents) Integer format. GSQ format is inspired by the finding that the magnitudes of weights in a single layer of existing pretrained LLMs are similar. GSQ is thus designed to share the exponent section of a group of weights to reduce the memory footprint of LLM training. The experimental results show that GSQ-Tuning can benefit LLM training memory efficiency compared to QLoRA, while maintaining most of the model's end-task performance. Extensive experiments show strong generalizability of GSQ-Tuning on multimodal and finetuning tasks.
Summary of strengths: Unlike prior works limited to inference or partial quantization, GSQ-Tuning achieves fully quantized training (FQT), extending integer computation to gradients and enabling end-to-end integer fine-tuning.. The method is designed with edge hardware constraints in mind, validated through hardware synthesis (7nm chip), showcasing massive savings in chip area (11×) and power (5×) over FP8.

- The GSQ-Tuning method is built based on the finding that the magnitudes of weights in a single layer of pretrained LLMs are similar. This presents a strong motivation and rationale for the method design and makes the paper self-consistent.
- The Pareto frontier experiment in the paper is insightful, pointing out the relationship between quantization and low-rank PEFT in LLM finetuning, which can be useful for future research.
- Extensive experiments show that GSQ-Tuning benefits LLM training's memory efficiency while maintaining most of the LLMs' end-task performance on a variety of tasks.
Summary of weaknesses: - Novelty: For my understanding, the method proposed in this paper has already been studied in a previous paper the author cited [1]. The main contribution of this paper is about the group-shared exponents, this idea is the same as the block floating point in [1]. Also, by using the block floating point method, we can simply do integer only computation. The author cite [1] in their method section. However, they fail to give detailed explanation about the difference between their work and [1], even [1] should be the most related work to this paper.

- I feel the writing of this paper need to be improved. There exist a lot of typos and miss use of math notations in this draft. For example:

l006: "large Floating Point(FP) computation" this makes me confused what it means for large. Also, there is no intention before "("

l177: We can easily see XA^TB^T cannot be multiplied together given the shape defined above of this equation

Generally, for the experiments, both the results and setups are good for me. However, I have strong concerns about the novelty of this paper and the unclear discussion for the related work. I hope the author could further explain this in the rebuttal.



[1] Zhang, Sai Qian, Bradley McDanel, and H. T. Kung. "Fast: Dnn training under variable precision block floating point with stochastic rounding." 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2022.

Besides the memory efficiency, the training time consumption of GSQ-Tuning is not being discussed by the authors in the paper. Since most PEFT methods tend to show a longer convergence time than full finetuning, it is essential for the audience to know whether the same phenomenon applies to GSQ-Tuning.
Comments, suggestions and typos: See the weakness

> Line 245-247:  FP computation demonstrates less efficiency compared to INT computation, making it less suitable for resource-constrained environments

This isn't very certain. For example, for NVIDIA's 5000 series GPUs, the TFLOPs of INT8 and FP8 show the same. Regarding the "efficiency" mentioned here, what specific aspect efficiency does the author wants to demonstrate? (e.g., chip area, power consumption?). The statement should be made more concretely with details.

- The abbreviation "CSQA" only appears in the captions of three tables (Table 1, 2, and 4), without any explanations, which is confusing.

- It is confusing which lines represent GSQ-Tuning data formats and which represent standard formats like FP8/INT8/FP16 in Figure 4. This should be a great opportunity to show the advantage of GSQ-Tuning over standard formats with the same bit width, yet the authors do not discuss it here.
Overall assessment scores: 2

4
Metareview score: 3
Metareview: The paper introduces GSQ-Tuning, a framework for fully quantized training of LLMs specifically designed for on-device fine-tuning under strict memory and power constraints. Unlike previous PEFT methods such as QLoRA, which rely on floating-point operations, GSQ-Tuning replaces all floating-point arithmetic with a Group-Shared Exponents Integer (GSE-INT) format. This format reduces memory and compute costs by sharing exponent bits across parameter groups and enables efficient integer-based matrix operations during both the forward and backward passes. Integrated with LoRA-like adapters, this framework supports end-to-end integer training, achieving ∼50% memory savings over FP16 and significantly reduced power and chip area usage compared to FP8.

Experiments across multiple LLaMA models (3B–70B) and tasks demonstrate that GSQ-Tuning maintains comparable accuracy to FP16 and QLoRA, while offering better efficiency. The paper also explores the Pareto frontier of bit-width and adapter rank, hinting at the possibility of further optimization for accuracy-memory trade-offs. In addition, it presents a hardware synthesis of the GSE-INT engine on a 7nm chip, showing a 5x reduction in power and 11x reduction in area over FP8 designs. These results do support GSQ-Tuning as a practical and scalable solution for on-device model adaptation.
Summary of reasons to publish: The paper is well-motivated and clearly written, presenting a novel method for enabling fully integer-based fine-tuning of LLMs on resource-constrained edge devices. The findings are backed by well-designed experiments and results.