EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
Authors: Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Ping Luo
Accepted at: acl25

EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

Mengzhao Chen1,2, Wenqi Shao†2, Peng Xu1,2, Jiahao Wang1,2, Peng Gao2, Kaipeng Zhang2, Ping Luo†1
1The University of Hong Kong 2Shanghai AI Laboratory

Abstract
Large language models (LLMs) are crucial in modern natural language processing and artificial intelligence. However, they face challenges in managing their significant memory requirements. Although quantization-aware training (QAT) offers a solution by reducing memory consumption through low-bit representations with minimal accuracy loss, it is impractical due to substantial training resources. To address this, we propose Efficient Quantization-Aware Training (EfficientQAT), a more feasible QAT algorithm. EfficientQAT involves two consecutive phases: Block-wise training of all parameters (Block-AP) and end-to-end training of quantization parameters (E2E-QP). To the best of our knowledge, Block-AP is the first method to enable direct training of all parameters in a block-wise manner, reducing accuracy loss in low-bit scenarios by enhancing the solution space during optimization. E2E-QP then trains only the quantization parameters (step sizes) end-to-end, further improving the performance of quantized models by considering interactions among all sub-modules. Extensive experiments demonstrate that EfficientQAT outperforms previous quantization methods across a range of models, including base LLMs, instruction-tuned LLMs, and multimodal LLMs, with scales from 7B to 70B parameters at various quantization bits. For instance, EfficientQAT obtains a 2-bit Llama-2-70B model on a single A100-80GB GPU in 41 hours, with less than 3 points accuracy degradation compared to the full precision (69.48 vs. 72.41). Code is available at https://github.com/OpenGVLab/EfficientQAT.

1 Introduction
Recent advancements in large language models (LLMs) (Touvron et al., 2023; Bubeck et al., 2023; Chiang et al., 2023; Xu et al., 2023a; Ying et al., 2024) have demonstrated impressive capabilities in diverse language tasks such as reasoning (Clark et al., 2018, 2019; Zellers et al., 2019), cognitive processing (Fu et al., 2023; Xu et al., 2023a), and agent-based applications (Qin et al., 2023a,b). However, these models are characterized by their extensive parameters, which pose significant challenges for memory footprint and bandwidth (Kim et al., 2023b; Xu et al., 2024a).

Quantization-aware training (QAT) is a highly effective quantization technique that minimizes quantization errors by incorporating quantization constraints during training. For example, BitNet b1.58 (Ma et al., 2024) can achieve nearly lossless ternary quantization. The precision of QAT is due to two main factors: 1) Fully trainable parameters allow for enough optimized space for gradi-

![Line and bar charts comparing quantization accuracy and PEFT methods](page_1012_312_388_388.png)

Figure 1: (a) EfficientQAT significantly surpasses existing uniform quantization methods, and is either superior to or comparable with vector quantization techniques. (b) EfficientQAT markedly outperforms existing Q-PEFT methods.
ent descent optimization; 2) End-to-end training accounts for interactions among all sub-modules in the models. Despite its performance benefits, QAT demands significant training resources, such as time and GPUs, as well as extensive training data. For instance, BitNet b1.58 requires retraining LLMs from scratch using the entire pre-trained dataset. Therefore, this approach is impractical for extremely large models and has only been verified on 3B models with 100B training tokens.

In optimizing quantization for LLMs, current methods emphasize either fine-grained reconstruction or reducing trainable parameters. While these approaches improve efficiency, they significantly degrade accuracy in low-bit scenarios. Mainstream post-training quantization (PTQ) methods (Lin et al., 2023; Frantar et al., 2022; Shao et al., 2023) focus on block-wise reconstruction (Li et al., 2021). They also restrict the optimization space to alleviate overfitting risk by only training rounding parameters (Nagel et al., 2020; Cheng et al., 2023), clipping thresholds (Shao et al., 2023), or step sizes (Esser et al., 2019; Ding et al., 2023). However, these methods not only limit optimizable parameters but also overlook cross-block interactions, leading to notable accuracy degeneration in low-bit scenarios, as shown in Figure 1a. Conversely, quantized parameter-efficient fine-tuning (Q-PEFT) methods (Dettmers et al., 2023a; Kim et al., 2023a) reduce training costs by freezing quantized parameters and only training a few continuous floats. For example, PEQA (Kim et al., 2023a) and QA-LoRA (Xu et al., 2023b) focus on training continuous quantization parameters. Despite this, their performance remains poor, as depicted in Figure 1b, because the severe performance loss in low-bit scenarios (2-bit and 3-bit) cannot be fully recovered with limited trainable parameters.

To address these challenges, we introduce a novel quantization-aware training framework called EfficientQAT. This framework combines the advantages of fully trainable parameters and end-to-end training, similar to native QAT (Ma et al., 2024), while maintaining the training efficiency of PTQ (Cheng et al., 2023; Shao et al., 2023) and Q-PEFT (Xu et al., 2023b). EfficientQAT introduces block-wise training of all parameters (Block-AP) to enhance the optimizable space and mitigate quantization accuracy loss. Block-AP sequentially trains all parameters, including original weights and quantization parameters (step sizes and zero points), within each transformer block. Several works have been developed based on block-wise reconstruction. However, previous approaches focus on designing additional trainable parameters, such as clipping thresholds for OmniQuant (Shao et al., 2023), weight rounding for AutoRound (Cheng et al., 2023) and BREQC (Li et al., 2021), or LoRA (Hu et al., 2021) parameters for CBQ (Ding et al., 2023). Our Block-AP is the first to directly train all parameters during block-wise reconstruction, achieving superior performance compared to previous methods (see Table 5). Block-AP successfully demonstrates that complex trainable parameter design is unnecessary for effective block-wise reconstruction in LLMs quantization. Furthermore, we introduce end-to-end training of quantization parameters (E2E-QP) to account for inter-block interactions. E2E-QP keeps the quantized weights fixed and trains only the quantization parameters (step sizes) end-to-end.

Thanks to the integration of the proposed Block-AP and E2E-QP, EfficientQAT characterizes itself as a fast-converging, memory-efficient, and high-performing quantization technique. For instance, EfficientQAT can obtain a 2-bit Llama-2-70B model on a single A100-80GB GPU in just 41 hours, with less than 3 points accuracy degradation on 5 zero-shot common-sense tasks compared to its full-precision counterpart (69.48 vs. 72.41). We also evaluate EfficientQAT across scenarios involving model compression and instruction-tuning. In model compression, as illustrated in Figure 1a, EfficientQAT significantly outperforms existing uniform quantization methods by approximately 5 points on accuracy in the challenging 2-bit quantization setting. In terms of instruction tuning, as shown in Figure 1b, EfficientQAT consistently outperforms existing Q-PEFT methods, including QLoRA (Dettmers et al., 2023a), QA-LoRA (Xu et al., 2023b), and PEQA (Kim et al., 2023a). For instance, EfficientQAT surpasses PEQA (Kim et al., 2023a) with 4.5 points MMLU accuracy when fine-tuning with Alpaca dataset.

2 Related Works

Post-Training Quantization of LLMs. PTQ is a pivotal technique for accelerating and deploying LLMs. Quantization approaches generally fall into two categories: weight-only quantization (Frantar et al., 2022; Dettmers et al., 2023b; Lee et al., 2023a; Kim et al., 2023b) and weight-activation quantization (Xiao et al., 2023; Liu et al., 2023c;
Wei et al., 2022, 2023; Yuan et al., 2023; Zhao et al., 2023; Ashkboos et al., 2023; Li et al., 2023a; Ashkboos et al., 2024). Weight-only quantization focuses on compressing weights into low-bit formats, reducing memory demands and enhancing the efficiency of memory-bounded computations in LLMs (Lin et al., 2024; Yuan et al., 2024). Conversely, weight-activation quantization compresses both weights and activations, thus further decreasing the overhead associated with matrix multiplications (Lin et al., 2024). Recent advancements in weight-only quantization include the introduction of vector quantization methods by QUIP#(Tseng et al., 2024) and AQLM(Egiazarian et al., 2024). These methods have shown promising performance but also introduce significant overhead (Gong et al., 2024). Our research continues to explore uniform quantization, which is preferred for its compatibility with hardware implementations.

Quantization-Aware Training of LLMs. QAT can enhance the performance of quantized models beyond what PTQ offers. However, QAT has been less explored in LLMs due to the significant training costs involved. Studies such as LLM-QAT (Liu et al., 2023e) and BitDistiller (Du et al., 2024) investigate the application of knowledge distillation within QAT contexts. Techniques like BitNet b1.58 (Ma et al., 2024) and OneBit (Xu et al., 2024b) employ QAT to achieve extreme binary or ternary quantization levels. Although BitNet b1.58 demonstrates near-lossless performance on models up to 3 billion parameters and 100 billion training tokens with ternary quantization, its applicability to larger models or datasets remains uncertain due to prohibitive training expenses.

Quantized Parameter-Efficient Fine-Tuning of LLMs. Techniques like QLoRA (Dettmers et al., 2023a), INT2.1 (Chai et al., 2023), LQ-LoRA (Guo et al., 2023), and LoftQ (Li et al., 2023b) quantize model parameters to low-bit representations followed by the addition of LoRA (Hu et al., 2021) modules for fine-tuning. However, these methods require merging the LoRA modules into quantized weights, resulting in the model reverting to the FP16 format. Addressing this issue, QA-LoRA (Xu et al., 2023b) redesigns the LoRA module to merge seamlessly into the zero points. The approach most similar to ours is PEQA (Kim et al., 2023a), which uses a round-to-nearest (RTN) method for low-bit quantization and fine-tunes step sizes for task adaptation. However, PEQA experiences significant performance degradation due to limited trainable parameters, which hinders recovery from quantization information loss.

3 EfficientQAT

3.1 Method Overview

In this section, we introduce EfficientQAT, a novel quantization-aware training framework for LLMs that enhances memory efficiency. As illustrated in Figure 2, traditional QAT approaches train the weights \( \mathbf{W} \) and quantization parameters \( s \) (step sizes) and \( z \) (zero points) simultaneously in an end-to-end manner, which significantly increases the memory requirements due to the large number of parameters involved. To address this issue, EfficientQAT adopts a two-stage strategy: block-wise training of all parameters (Block-AP) and end-to-end training of quantization parameters (E2E-QP). In the Block-AP phase, model parameters and quantization parameters are trained block-by-block using reconstruction loss, which not only allows for precise calibration with full training but also reduces memory consumption (Li et al., 2021; Shao et al., 2023) by block-wise training. Following this, the E2E-QP phase fixes the quantized weights and trains the step sizes exclusively on target datasets, thus achieving inter-block interaction in a memory-efficient way. Details on Block-AP and E2E-QP are further described in Sections 3.2 and 3.3, respectively.

3.2 Block-Wise Training of All Parameters

In this section, we introduce the Block-Wise Training of All Parameters (Block-AP) approach, designed to efficiently provide an effective initialization for following end-to-end training.

Quantization and Dequantization. Specifically, Block-AP begins with a standard uniform quantization method:

\[
\mathbf{W}_{int} = \mathrm{clamp}(\lfloor \frac{\mathbf{W}}{s} \rfloor + z, 0, 2^N - 1),
\]

where \( \lfloor \cdot \rfloor \) represents the rounding operation. \( N \) is the target bit number. \( \mathbf{W}_{int} \) and \( \mathbf{W} \) denote the quantized integer and full-precision weights (Float16 or BFloat16 for LLMs), respectively. \( s \) is the scaling factor and \( z \) is the zero point. In the forward propagation, the quantized weights are converted back to full precision as follows:

\[
\widehat{\mathbf{W}} = (\mathbf{W}_{int} - z) \cdot s.
\]
Figure 2: The overall pipeline of naive QAT and proposed EfficientQAT. EfficientQAT introduces two novel processes: Block-wise Training of All Parameters (Block-AP) and End-to-End Training of Quantization Parameters (E2E-QP).

Here, \( \widehat{\mathbf{W}} \) refers to the dequantized weights used in the forward computation. The processes of quantization (Eq.(1)) and dequantization (Eq.(2)) are integrated within the computation graph and can be optimized through gradient descent in a quantization-aware manner.

Block-wise Quantization-aware Training. Traditional QAT methods (Ma et al., 2024; Esser et al., 2019; Liu et al., 2023e) train the entire network using Eq.(1) and Eq.(2) in an end-to-end fashion, which typically requires substantial computational resources and extensive data to prevent overfitting. Here we aim to enhance the training efficiency of QAT. Previous studies, such as BRECQ (Li et al., 2021), have demonstrated that block-wise training achieves faster convergence and requires less training time, data, and memory than end-to-end training given a pre-trained model. Following the methodologies in BRECQ (Li et al., 2021) and OmniQuant (Shao et al., 2023), Block-AP sequentially conducts quantization-aware training within one transformer block before moving on to the next under a block-wise reconstruction framework.

Full Training of Model Weights and Quantization Parameters. Unlike previous methods which optimize several quantization parameters such as rounding parameters (Nagel et al., 2020; Cheng et al., 2023; Lee et al., 2023b), clipping parameters (Shao et al., 2023), and step sizes (Esser et al., 2019; Ding et al., 2023), Block-AP behaves like QAT, training all inherent parameters from Eq.(1) and Eq.(2), including scaling factor s, zero point z, and model weights \( \mathbf{W} \).

In our Block-AP approach, a straightforward full-training regimen outperforms existing partial-training variants (Nagel et al., 2020; Li et al., 2021; Ding et al., 2023) with intricate designs. Traditional training methods involving rounding parameters (Nagel et al., 2020; Li et al., 2021; Ding et al., 2023) serve as regularization techniques, constraining the update range of integral weights to \((-1, +1)\) to mitigate overfitting. However, this approach limits the solution space, potentially hindering the final performance of quantized models. Our empirical findings demonstrate the superiority of full training within our Block-AP over existing partial-training variants (Nagel et al., 2020; Li et al., 2021; Ding et al., 2023), as shown in Table 5.

Following block-wise training, we obtain the quantized model which includes quantized weights \( \mathbf{W}_q \), step sizes s, and zero points z for each quantization group. The weights \( \mathbf{W}_q \) and zero points z are stored in a low-bit format, while step sizes s are stored in FP16. Note that s and z are shared within their respective quantization groups and constitute only a small fraction of the model’s parameters, approximately 1.6% for a group size of 64. Moreover, the model’s memory footprint is substantially reduced by transitioning from full-precision 16-bit weights to 2/3/4-bit quantized weights.

3.3 End-to-End Training of Quantization Parameters

We further introduce the End-to-End Training of Quantization Parameters (E2E-QP), aimed at efficiently training the entire quantized model on target datasets.

End-to-End Training of step sizes. Unlike traditional Quantization-Aware Training (QAT) methods (Liu et al., 2023e; Ma et al., 2024) that train full-precision weights, E2E-QP begins with \( \mathbf{W}_q \) initialized via Block-AP and focuses solely on the training of quantization parameters (s and z). Our findings indicate that training s, z, or both yields similar performance (see Table 6 for details). However, since training z involves converting it from
a low-bits format to full-precision, we typically train only s by default unless specified otherwise to avoid additional memory overhead.

Additionally, within E2E-QP, there is no quantization process as per Equation (1); only the dequantization process occurs as described in Equation (2). Thus, the gradient of the trainable parameter s is computed as \( \frac{\partial \hat{w}}{\partial s} = w_q - z \).

Overall, the memory usage for training in E2E-QP is drastically reduced due to the reduced trainable parameter count. Detailed memory footprints for various model sizes and bits under E2E-QP are listed in Table 7. For instance, the Llama-2-70B model can complete 2-bit QAT through E2E-QP using only 34.2GB of memory. Equipped with E2E-QP, EfficientQAT is adaptable to different scenarios by simply changing the training datasets, which includes applications such as continual pre-training and instruction-tuning (Taori et al., 2023).

4 Experiments

This section presents extensive experiments to verify our proposed EfficientQAT. Section 4.1 and Sec 4.2 present the comparisons with quantization methods and Q-PEFT methods respectively. Section 4.4 details the training cost and inference speed-up of the proposed EfficientQAT. Section 4.3 presents the comprehensive ablation studies of the proposed EfficientQAT.

4.1 EfficientQAT for LLMs Quantization

Training. We conduct experiments on the Llama-2 and Llama-3 models. For Block-AP, we use 4096 samples from RedPajama (Computer, 2023) with a context length of 2048. We train each block with batch size as 2 and epochs as 2, setting the learning rate of quantization parameters as \( 1 \times 10^{-4} \), and the learning rate of weights as \( 2 \times 10^{-5} \) for 2-bit and \( 1 \times 10^{-5} \) for 3/4-bits. For E2E-QP, we also employ 4096 samples from RedPajama (Computer, 2023) but with a context length of 4096. We train the entire model with batch size as 32 and epoch as 1, and set the learning rate of step size as \( 2 \times 10^{-5} \) for 2-bit and \( 1 \times 10^{-5} \) for 3-bits.

PTQ Baseline. We compare our results with PTQ methods from uniform quantization such as GPTQ (Frantar et al., 2022), AWQ (Lin et al., 2023), OmniQ (Shao et al., 2023), ApiQ (Liao and Monz, 2024) and AutoRound (Cheng et al., 2023), and vector quantization including QuIP# (Tseng et al., 2024) and AQLM (Egiazarian et al., 2024). Note that if a result is the best of uniform quantization, we set it to bold.

Accuracy results. We evaluate the zero-shot accuracy on five common-sense reasoning tasks using the v0.4.2 lm-evaluation-harness*. These tasks include WinoGrande (Sakaguchi et al., 2021), PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), Arc-Easy (Clark et al., 2018), and Arc-Challenge (Clark et al., 2018). Table 1 shows that the proposed EfficientQAT significantly outperforms previous methods for uniform quantization across the Llama-2 and Llama-3 model families, as well as in both 2-bit and 3-bit quantization settings. The performance gains are particularly notable in extremely low-bit quantization, such as 2-bit. For instance, EfficientQAT achieves a +3.26% accuracy improvement over AWQ in w3g128 quantization with Llama-3-8B. Moreover, EfficientQAT surpasses DB-LLM by +9.02% accuracy in w2g64 quantization. In comparison to vector quantization, our results show that EfficientQAT outperforms QuIP#(Tseng et al., 2024) in 3-bit quantization, but underperforms in 2-bit scenarios. However, direct comparisons between uniform quantization methods (such as EfficientQAT) and vector quantization methods (such as QuIP#) can be misleading due to fundamental differences in their approaches. Vector quantization often achieves better results at very low bit-widths through complex codebook designs, but this comes at the cost of reduced generalization and deployment flexibility. For instance, EfficientQAT supports both weight and activation quantization, while vector quantization methods are typically limited to weight-only quantization. Furthermore, a recent study, PrefixQuant(Chen et al., 2024b), demonstrates that EfficientQAT improves state-of-the-art weight-activation quantization methods by nearly 0.3 perplexity.

Perplexity results. We also evaluate perplexity on Wikitext2 and C4 using a 2048 context length, following prior studies (Frantar et al., 2022; Shao et al., 2023). The results align with the accuracy comparison, as EfficientQAT consistently achieves lower perplexity across the Llama-2 and Llama-3 model families in both 2-bit and 3-bit quantization. Notably, the benefits are more pronounced in Llama-3 models, which face greater challenges in quantization (Huang et al., 2024). For example, EfficientQAT reduces perplexity by 0.37 and 4.19 points compared to DB-LLM in Llama-2-7B and

*https://github.com/EleutherAI/lm-evaluation-harness
Table 1: Llama 2 & 3 average zero-shot accuracy on 5 common-sense reasoning tasks (†). "-" indicates the result is unreachable in the public papers.

<table>
  <tr>
    <th>Method</th>
    <th>Bits</th>
    <th>Group</th>
    <th>2-7</th>
    <th>2-13</th>
    <th>2-70</th>
    <th>3-8</th>
    <th>3-70</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>16</td>
    <td>-</td>
    <td>64.86</td>
    <td>67.81</td>
    <td>72.41</td>
    <td>68.58</td>
    <td>75.33</td>
  </tr>
  <tr>
    <td>RTN</td>
    <td>3</td>
    <td>128</td>
    <td>62.06</td>
    <td>65.77</td>
    <td>70.83</td>
    <td>58.72</td>
    <td>65.29</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>3</td>
    <td>128</td>
    <td>62.48</td>
    <td>66.18</td>
    <td>71.47</td>
    <td>60.58</td>
    <td>71.28</td>
  </tr>
  <tr>
    <td>AWQ</td>
    <td>3</td>
    <td>128</td>
    <td>62.82</td>
    <td>66.14</td>
    <td>71.41</td>
    <td>64.82</td>
    <td>73.65</td>
  </tr>
  <tr>
    <td>OmniQ</td>
    <td>3</td>
    <td>128</td>
    <td>62.42</td>
    <td>66.18</td>
    <td>71.07</td>
    <td>64.09</td>
    <td>71.90</td>
  </tr>
  <tr>
    <td>AutoRound</td>
    <td>3</td>
    <td>128</td>
    <td>63.72</td>
    <td>66.68</td>
    <td>71.24</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>QuIP#</td>
    <td>3</td>
    <td>-</td>
    <td>63.52</td>
    <td>66.26</td>
    <td>72.13</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <th>EfficientQAT</th>
    <th>3</th>
    <th>128</th>
    <th><b>64.02</b></th>
    <th><b>67.28</b></th>
    <th><b>71.76</b></th>
    <th><b>67.35</b></th>
    <th><b>73.96</b></th>
  </tr>
  <tr>
    <td>OmniQ</td>
    <td>2</td>
    <td>128</td>
    <td>46.98</td>
    <td>53.56</td>
    <td>54.87</td>
    <td>52.66</td>
    <td>60.06</td>
  </tr>
  <tr>
    <td>AutoRound</td>
    <td>2</td>
    <td>128</td>
    <td>54.50</td>
    <td>60.72</td>
    <td>67.70</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <th>EfficientQAT</th>
    <th>2</th>
    <th>128</th>
    <th><b>59.50</b></th>
    <th><b>63.88</b></th>
    <th><b>68.93</b></th>
    <th><b>59.37</b></th>
    <th><b>67.57</b></th>
  </tr>
  <tr>
    <td>AQLM</td>
    <td>2</td>
    <td>2x8</td>
    <td>57.61</td>
    <td>62.22</td>
    <td>69.85</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>QuIP#</td>
    <td>2</td>
    <td>-</td>
    <td>60.61</td>
    <td>64.44</td>
    <td>70.91</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DB-LLM</td>
    <td>2</td>
    <td>64</td>
    <td>56.93</td>
    <td>61.61</td>
    <td>68.01</td>
    <td>51.74</td>
    <td>-</td>
  </tr>
  <tr>
    <th>EfficientQAT</th>
    <th>2</th>
    <th>64</th>
    <th><b>60.14</b></th>
    <th><b>64.48</b></th>
    <th><b>69.48</b></th>
    <th><b>60.76</b></th>
    <th><b>67.89</b></th>
  </tr>
</table>

Llama-3-8B, respectively.

How model size and training tokens affect quantization error. Recent scaling laws for PTQ (Kumar et al., 2024; Ouyang et al., 2024) show that quantization error increases with the number of training tokens and decreases as model size grows. Our results in Table 1 and Table 2 are consistent with these PTQ scaling laws. Additionally, the absolute benefit of our proposed method is more pronounced in smaller models, as they experience greater performance degradation from quantization. For example, DB-LLM loses 7.93 accuracy points with W2G64 on Llama-2-7B, but only 4.40 on Llama-2-70B. As a result, the improvement of EfficientQAT over DB-LLM decreases from 3.21 on Llama-2-7B to 1.47 on Llama-2-70B. However, when we use the relative gain metric \( \frac{\text{EfficientQAT} - \text{DBLLM}}{\text{FP16} - \text{DBLLM}} \), EfficientQAT reduces quantization error by 40% for Llama-2-7B and 33% for Llama-2-70B. The relative gain metric demonstrates the effectiveness of proposed EfficientQAT across different model sizes.

4.2 EfficientQAT for Instruction Tuning

Training and Evaluation. Following existing works (Xu et al., 2023b; Qin et al., 2024), we train Llama-1 models on the Alpaca dataset (Taori et al., 2023) and assess their performance by measuring average 5-shot MMLU (Hendrycks et al., 2020) accuracy works (Xu et al., 2023b; Qin et al., 2024).

The training hyperparameters are identical to those described in Section 4.1, except we replace the Red-Pajama dataset (Computer, 2023) with Alpaca. In line with QLoRA’s methodology (Dettmers et al., 2023a), we adjust the source context length to 384 and the target context length to 128, training for 10,000 steps with a batch size of 16.

Baseline. We benchmark EfficientQAT against several leading methods, including QLoRA (Dettmers et al., 2023a), QA-LoRA (Xu et al., 2023b), PEQA (Kim et al., 2023a), and IR-QLoRA (Qin et al., 2024), across quantization setting of 2, 3, and 4 bits. Consistent with QA-LoRA (Xu et al., 2023b), we also employ GPTQ (Frantar et al., 2022) to quantize the fine-tuned QLoRA models into a low-bit format without FP16 LoRA for equitable comparison.

Results. Both Table 3 and Figure 1b indicate that EfficientQAT significantly outperforms existing Q-PEFT methods. For instance, in channel-wise quantization (group size of -1), EfficientQAT achieves more than 3% higher accuracy than PEQA (Kim et al., 2023a). In the 2-bit quantization scenario, the superiority of EfficientQAT is even more pronounced, surpassing QA-LoRA (Xu et al., 2023b) by 5.1% and 4.0% in 7B and 13B models, respectively, and outperforming PEQA by 4.5% and 8.7% in the same models. Moreover, Table 3 also demonstrates that EfficientQAT outperforms both QA-LoRA and QLoRA with GPTQ in smaller model
Table 2: Llama 2 & 3 Wikitext2 and C4 perplexity (↓), context length 2048. "-" indicates the result is unreachable in the public papers.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Bits Group</th>
    <th colspan="4">Wikitext 2</th>
    <th colspan="4">C4</th>
  </tr>
  <tr>
    <th>2-7</th>
    <th>2-13</th>
    <th>2-70</th>
    <th>3-8</th>
    <th>3-70</th>
    <th>2-7</th>
    <th>2-13</th>
    <th>2-70</th>
    <th>3-8</th>
    <th>3-70</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>16 -</td>
    <td>5.47</td>
    <td>4.88</td>
    <td>3.32</td>
    <td>6.14</td>
    <td>2.85</td>
    <td>6.97</td>
    <td>6.47</td>
    <td>5.52</td>
    <td>8.88</td>
    <td>6.73</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>3 128</td>
    <td>6.29</td>
    <td>5.42</td>
    <td>3.85</td>
    <td>9.58</td>
    <td>5.25</td>
    <td>7.89</td>
    <td>7.00</td>
    <td>5.85</td>
    <td>11.66</td>
    <td>8.64</td>
  </tr>
  <tr>
    <td>AWQ</td>
    <td>3 128</td>
    <td>6.24</td>
    <td>5.32</td>
    <td>3.74</td>
    <td>8.16</td>
    <td>4.69</td>
    <td>7.84</td>
    <td>6.94</td>
    <td>5.81</td>
    <td>11.49</td>
    <td>7.91</td>
  </tr>
  <tr>
    <td>OmniQ</td>
    <td>3 128</td>
    <td>6.03</td>
    <td>5.28</td>
    <td>3.78</td>
    <td>8.27</td>
    <td>4.99</td>
    <td>7.75</td>
    <td>6.98</td>
    <td>5.85</td>
    <td>11.66</td>
    <td>7.97</td>
  </tr>
  <tr>
    <td>BitDistiller</td>
    <td>3 128</td>
    <td>5.97</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>EfficientQAT</b></td>
    <td><b>3 128</b></td>
    <td><b>5.81</b></td>
    <td><b>5.12</b></td>
    <td><b>3.61</b></td>
    <td><b>7.09</b></td>
    <td><b>4.21</b></td>
    <td><b>7.34</b></td>
    <td><b>6.73</b></td>
    <td><b>5.71</b></td>
    <td><b>10.06</b></td>
    <td><b>7.46</b></td>
  </tr>
  <tr>
    <td>OmniQ</td>
    <td>2 128</td>
    <td>11.06</td>
    <td>8.26</td>
    <td>6.55</td>
    <td>18.50</td>
    <td>16.79</td>
    <td>15.02</td>
    <td>11.05</td>
    <td>8.52</td>
    <td>22.46</td>
    <td>15.06</td>
  </tr>
  <tr>
    <td>ApiQ</td>
    <td>2 128</td>
    <td>8.25</td>
    <td>6.71</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>12.04</td>
    <td>9.13</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>BitDistiller</td>
    <td>2 128</td>
    <td>8.08</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>EfficientQAT</b></td>
    <td><b>2 128</b></td>
    <td><b>7.19</b></td>
    <td><b>6.08</b></td>
    <td><b>4.61</b></td>
    <td><b>9.80</b></td>
    <td><b>6.38</b></td>
    <td><b>8.79</b></td>
    <td><b>7.75</b></td>
    <td><b>6.48</b></td>
    <td><b>13.22</b></td>
    <td><b>9.53</b></td>
  </tr>
  <tr>
    <td>AQLM</td>
    <td>2 2x8</td>
    <td>7.24</td>
    <td>6.06</td>
    <td>4.49</td>
    <td>-</td>
    <td>-</td>
    <td>8.96</td>
    <td>7.80</td>
    <td>6.36</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>QuIP#</td>
    <td>2 -</td>
    <td>6.66</td>
    <td>5.74</td>
    <td>4.16</td>
    <td>-</td>
    <td>-</td>
    <td>8.35</td>
    <td>7.45</td>
    <td>6.12</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ApiQ</td>
    <td>2 64</td>
    <td>7.59</td>
    <td>6.44</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>10.56</td>
    <td>8.92</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>CBQ</td>
    <td>2 64</td>
    <td>8.01</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>11.30</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DB-LLM</td>
    <td>2 64</td>
    <td>7.23</td>
    <td>6.19</td>
    <td>4.64</td>
    <td>13.60</td>
    <td>-</td>
    <td>9.62</td>
    <td>8.38</td>
    <td>6.77</td>
    <td>19.20</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>EfficientQAT</b></td>
    <td><b>2 64</b></td>
    <td><b>6.86</b></td>
    <td><b>5.96</b></td>
    <td><b>4.52</b></td>
    <td><b>9.41</b></td>
    <td><b>6.07</b></td>
    <td><b>8.50</b></td>
    <td><b>7.59</b></td>
    <td><b>6.38</b></td>
    <td><b>12.77</b></td>
    <td><b>9.23</b></td>
  </tr>
</table>

memory footprint (larger group size).

Table 3: Llama-1 average MMLU accuracy (5-shot) about instruction-tuning on Alpaca dataset.

<table>
  <tr>
    <th>Method</th>
    <th>Bits</th>
    <th>Group</th>
    <th>7B</th>
    <th>13B</th>
  </tr>
  <tr>
    <td>-</td>
    <td>16</td>
    <td>-</td>
    <td>34.6</td>
    <td>46.3</td>
  </tr>
  <tr>
    <td>PEQA</td>
    <td>4</td>
    <td>-1</td>
    <td>35.8</td>
    <td>45.0</td>
  </tr>
  <tr>
    <td><b>EfficientQAT</b></td>
    <td><b>4</b></td>
    <td><b>-1</b></td>
    <td><b>38.8</b></td>
    <td><b>48.2</b></td>
  </tr>
  <tr>
    <td>QLoRA</td>
    <td>4+16</td>
    <td>-</td>
    <td>38.4</td>
    <td>48.4</td>
  </tr>
  <tr>
    <td>QLoRA w/GPTQ</td>
    <td>4</td>
    <td>32</td>
    <td>36.0</td>
    <td>48.0</td>
  </tr>
  <tr>
    <td>QA-LoRA</td>
    <td>4</td>
    <td>32</td>
    <td>39.4</td>
    <td>49.2</td>
  </tr>
  <tr>
    <td>PEQA</td>
    <td>4</td>
    <td>64</td>
    <td>39.4</td>
    <td>47.4</td>
  </tr>
  <tr>
    <td>IR-QLoRA</td>
    <td>4</td>
    <td>64</td>
    <td>40.8</td>
    <td>49.3</td>
  </tr>
  <tr>
    <td><b>EfficientQAT</b></td>
    <td><b>4</b></td>
    <td><b>64</b></td>
    <td><b>41.2</b></td>
    <td><b>49.5</b></td>
  </tr>
  <tr>
    <td>QLoRA w/ GPTQ</td>
    <td>3</td>
    <td>32</td>
    <td>34.0</td>
    <td>46.1</td>
  </tr>
  <tr>
    <td>QA-LoRA</td>
    <td>3</td>
    <td>32</td>
    <td>37.4</td>
    <td>47.3</td>
  </tr>
  <tr>
    <td>IR-QLoRA</td>
    <td>3</td>
    <td>64</td>
    <td>38.4</td>
    <td>-</td>
  </tr>
  <tr>
    <td>PEQA</td>
    <td>3</td>
    <td>64</td>
    <td>38.5</td>
    <td>46.3</td>
  </tr>
  <tr>
    <td><b>EfficientQAT</b></td>
    <td><b>3</b></td>
    <td><b>64</b></td>
    <td><b>40.0</b></td>
    <td><b>48.2</b></td>
  </tr>
  <tr>
    <td>QLoRA w/ GPTQ</td>
    <td>2</td>
    <td>32</td>
    <td>25.8</td>
    <td>30.9</td>
  </tr>
  <tr>
    <td>QA-LoRA</td>
    <td>2</td>
    <td>32</td>
    <td>27.5</td>
    <td>36.9</td>
  </tr>
  <tr>
    <td>IR-QLoRA</td>
    <td>2</td>
    <td>64</td>
    <td>27.8</td>
    <td>-</td>
  </tr>
  <tr>
    <td>PEQA</td>
    <td>2</td>
    <td>64</td>
    <td>28.1</td>
    <td>32.2</td>
  </tr>
  <tr>
    <td><b>EfficientQAT</b></td>
    <td><b>2</b></td>
    <td><b>64</b></td>
    <td><b>32.6</b></td>
    <td><b>40.9</b></td>
  </tr>
</table>

4.3 Ablation Analysis

The EfficientQAT algorithm is comprised of two main components: Block-AP and E2E-QP. This section evaluates the effectiveness, trainable parameters, and training sample requirements of each component. We present the average perplexity for WikiText2 and C4 datasets, and the average accuracy for five zero-shot reasoning tasks, similar to Table 1.

Effectiveness of each component. As indicated in Table 4, both the Block-AP and E2E-QP components significantly enhance performance, with their combination yielding the best results. Notably, Block-AP outperforms E2E-QP, aligning with findings from BRECQ (Li et al., 2021).

Trainable parameters of Block-AP. Block-AP trains all parameters, including original weights and quantization parameters. Previous methods have introduced various training strategies to mitigate overfitting, such as trained rounding (Nagel et al., 2020; Cheng et al., 2023), clipping thresholds (Shao et al., 2023), and step sizes (Esser et al., 2019; Ding et al., 2023). We compare Block-AP with these methods by modifying only the trainable parameters of Block-AP. As shown in Table 5, Block-AP (training s, z, W) performs best with an acceptable training cost. Additionally, the memory footprint of directly training W is even smaller than that of training the rounding operation, which requires an additional copy of rounding parameters. Additionally, BitNet (Ma et al., 2024) demonstrates that optimizing only the weights, without considering quantization parameters, can still achieve strong performance. However, Table 5 shows that
training only the weights results in a perplexity of 14.32, which is significantly higher than the 8.53 achieved by Block-AP. This difference arises because our quantization approach starts from a pre-trained model and directly optimizes the scaling factors (s) and zero points (z) to minimize quantization errors, making minimal changes to the weights and thus preserving the model’s learned knowledge. In contrast, training only the weights adjusts the scaling factors indirectly, requiring larger weight updates that can disrupt this knowledge. BitNet (Ma et al., 2024), which is trained from scratch, does not face this issue.

Trainable parameters of E2E-QP. We further examine the trainable parameters within E2E-QP. Table 6 shows that training s, z, or both yields similar performance. However, given that converting z from an original low-bit representation to a trainable FP16 format increases the average bit count, we opt to train only s by default.

<table>
  <tr>
    <th>Block-AP</th>
    <th>E2E-QP</th>
    <th>Avg. PPL</th>
    <th>Avg. Acc.</th>
  </tr>
  <tr>
    <td>×</td>
    <td>×</td>
    <td>453.49</td>
    <td>40.69</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>×</td>
    <td>8.53</td>
    <td>58.99</td>
  </tr>
  <tr>
    <td>×</td>
    <td>✓</td>
    <td>9.33</td>
    <td>55.71</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>7.68</td>
    <td>60.14</td>
  </tr>
</table>

Table 4: Effectiveness of each component on Llama-2-7B w2g64 quantization.

<table>
  <tr>
    <th>Param.</th>
    <th># Memory</th>
    <th>Avg. PPL</th>
    <th>Avg. Acc.</th>
  </tr>
  <tr>
    <td>clipping</td>
    <td>6.3M 6.4GB</td>
    <td>11.28</td>
    <td>53.20</td>
  </tr>
  <tr>
    <td>s,z</td>
    <td>6.3M 6.4GB</td>
    <td>10.26</td>
    <td>55.20</td>
  </tr>
  <tr>
    <td>round</td>
    <td>202.4M 8.6GB</td>
    <td>15.50</td>
    <td>45.32</td>
  </tr>
  <tr>
    <td>W</td>
    <td>202.4M 8.5 GB</td>
    <td>14.32</td>
    <td>46.50</td>
  </tr>
  <tr>
    <td>s,z,round</td>
    <td>208.7M 9.3GB</td>
    <td>9.17</td>
    <td>57.14</td>
  </tr>
  <tr>
    <td>s,z,W</td>
    <td>208.7M 8.5GB</td>
    <td>8.53</td>
    <td>58.99</td>
  </tr>
</table>

Table 5: W2g64 Llama-2-7B performance with different trainable parameters in the block-wise training (w/o E2E-QP). “#” indicates trainable parameters count in a block.

<table>
  <tr>
    <th>Param.</th>
    <th>Avg. Bits</th>
    <th>Avg. PPL</th>
    <th>Avg. Accuracy</th>
  </tr>
  <tr>
    <td>s</td>
    <td>2.28</td>
    <td>7.68</td>
    <td>60.14</td>
  </tr>
  <tr>
    <td>z</td>
    <td>2.50</td>
    <td>7.69</td>
    <td>60.08</td>
  </tr>
  <tr>
    <td>s, z</td>
    <td>2.50</td>
    <td>7.68</td>
    <td>60.18</td>
  </tr>
</table>

Table 6: Llama-2-7B w2g64 quantization with different trainable parameters for E2E-QP (w/ Block-AP).

Figure 3: Illustration of training loss, validation loss and average accuracy of w2g64 Llama-2-7b with different training samples size for Block-AP (w/o E2E-QP).

Samples number of Block-AP. We assess the number of training samples for Block-AP, noting that E2E-QP trains all parameters, which may lead to overfitting. To address this, we introduce an additional 64 unseen samples from ReadPajama to evaluate the overfitting issue. We adjust the training epochs to ensure a similar total training time, allowing for fair comparisons across different sample sizes. As illustrated in Figure 3, increasing the number of training samples significantly reduces the gap between training loss and validation loss from 1.07 to 0.06. This reduction corresponds to an increase in the average accuracy for zero-shot tasks from 57.14% to 58.99%. Consequently, we set the default number of training samples for E2E-QP at 4096, as this maintains a minimal gap between training and validation losses.

Samples number of E2E-QP. In the E2E-QP, we train the model for 1 epoch to avoid over-fitting. Our examination of the training sample sizes for E2E-QP, detailed in Table 8, reveals that average perplexity consistently improves as sample sizes increase from 128 to 32,674. However, there is no significant improvement in average accuracy beyond 4096 samples. Therefore, we set the training sample size for E2E-QP at 4096 by default to balance efficiency and performance. Nonetheless, it is possible to further enhance the performance of EfficientQAT by increasing the sample size.

4.4 Efficiency of EfficientQAT

Training Efficiency Table 7 illustrates the required memory and time for training Llalma-2 models using EfficientQAT. The results indicate that the model completes training rapidly, taking 4.8 hours for the 7B model and 40.9 hours for the 70B model. we further compare the training time with other QAT methods, including BitDistiller, and DB-LLM. As shown in Table 9, the training time of Efficien-
Table 7: The detailed training time and training memory of EfficientQAT across different model size and quantization bits on a single A100-80GB GPU.

<table>
  <tr>
    <th rowspan="2">Llama-2</th>
    <th colspan="2">Block-AP</th>
    <th colspan="3">E2E-QP</th>
  </tr>
  <tr>
    <th>Time</th>
    <th>Memory</th>
    <th>Time</th>
    <th>Memory (4-/3-/2-bits)</th>
    <th>Total Time</th>
  </tr>
  <tr>
    <td>7B</td>
    <td>3.3h</td>
    <td>8.5GB</td>
    <td>~1.5h</td>
    <td>7.0/6.4/5.6GB</td>
    <td>4.8h</td>
  </tr>
  <tr>
    <td>13B</td>
    <td>5.6h</td>
    <td>10.3GB</td>
    <td>~2.9h</td>
    <td>11.7/10.6/9.1GB</td>
    <td>8.5h</td>
  </tr>
  <tr>
    <td>70B</td>
    <td>26.6h</td>
    <td>29.9GB</td>
    <td>~14.3h</td>
    <td>48.4/42.0/34.2GB</td>
    <td>40.9h</td>
  </tr>
</table>

Table 8: Llama-2-7B w2g64 quantization performance with different sample numbers for E2E-QP (w/ Block-AP).

<table>
  <tr>
    <th># Samples</th>
    <th>Avg. PPL</th>
    <th>Avg. Accuracy</th>
  </tr>
  <tr>
    <td>128</td>
    <td>8.09</td>
    <td>59.03</td>
  </tr>
  <tr>
    <td>512</td>
    <td>7.88</td>
    <td>59.81</td>
  </tr>
  <tr>
    <td>2048</td>
    <td>7.75</td>
    <td>60.13</td>
  </tr>
  <tr>
    <td>4096</td>
    <td>7.68</td>
    <td>60.14</td>
  </tr>
  <tr>
    <td>8192</td>
    <td>7.63</td>
    <td>60.19</td>
  </tr>
  <tr>
    <td>32764</td>
    <td><b>7.50</b></td>
    <td><b>60.31</b></td>
  </tr>
</table>

Table 9: Comparisons of training time with existing methods in Llama-2-70B.

<table>
  <tr>
    <th>Method</th>
    <th>One A100-80GB? GPU hours (h)</th>
  </tr>
  <tr>
    <td>LLM-QAT</td>
    <td>X</td>
    <td>900</td>
  </tr>
  <tr>
    <td>QuiP#</td>
    <td>X</td>
    <td>300</td>
  </tr>
  <tr>
    <td>AQLM</td>
    <td>✓</td>
    <td>336</td>
  </tr>
  <tr>
    <td>BitDistiller</td>
    <td>X</td>
    <td>64</td>
  </tr>
  <tr>
    <td>PB-LLM</td>
    <td>X</td>
    <td>90</td>
  </tr>
  <tr>
    <td>DB-LLM</td>
    <td>X</td>
    <td>82</td>
  </tr>
  <tr>
    <td><b>EfficientQAT</b></td>
    <td>✓</td>
    <td><b>41</b></td>
  </tr>
</table>

tQAT is significantly lower than that of existing methods. For example, the tuning time of EfficientQAT is only 50% of DB-LLM. Additionally, for quantizing a 70B model, the full process of EfficientQAT can be completed on a single A100-80GB GPU. However, other methods require at least 4 A100-80GB GPUs to quantize a model of this size. Therefore, EfficientQAT is both a time-efficient and memory-efficient QAT method.

Inference Efficiency Due to the leverage of standard uniform quantization, the quantized models of EfficientQAT can also achieve speedup through a lot of toolboxes, such as MLC-LLM (team, 2023), AWQ (Lin et al., 2023), and BitBLAS (Wang et al., 2024), T-MAC (Wei et al., 2024), Marlin (Frantar et al., 2024), etc. For example, Table 10 shows that INT2 quantization of EfficientQAT can enhance the forward-pass speed by approximately 2.9x to 4.4x through BitBLAS (Wang et al., 2024).

5 Conclusion

In this study, we introduce EfficientQAT, a novel method that completes QAT with improved efficiency in both memory usage and training time. Through comprehensive testing, EfficientQAT proves superior to existing PTQ, QAT, and Q-PEFT methods in terms of versatility and performance across various models and quantization levels. Additionally, EfficientQAT leverages a standard uniform quantization, which simplifies deployment using popular toolboxes. We anticipate that EfficientQAT will stimulate further research and improve the compression of Large Language Models (LLMs), making them more efficient and widely accessible.

6 Limitation

EfficientQAT achieves impressive results in low-bit quantization scenarios, but there remains a performance gap compared to full-precision (FP16) models, particularly in 2-bit settings. Reducing this gap without sacrificing efficiency remains a challenge. Additionally, the method depends on the availability of high-quality and diverse datasets, requiring 4096 samples for effective training in both the Block-AP and E2E-QP phases. The performance of the quantized models can vary significantly based on the size and distribution of the training data. This reliance may limit its effectiveness in data-scarce or domain-specific applications.

Acknowledgement

This paper is partially supported by the National Key R&D Program of China No.2022ZD0161000.
References

Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefer, and Dan Alistarh. 2023. Towards end-to-end 4-bit inference on generative large language models. arXiv preprint arXiv:2310.09259.

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefer, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456.

Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. ArXiv, abs/1308.3432.

Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. 2020. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2978–2985.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pages 7432–7439.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Erez Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.

Yuji Chai, John Gkountouras, Glenn G Ko, David Brooks, and Gu-Yeon Wei. 2023. Int2. 1: Towards fine-tunable quantized large language models with error correction through low-rank adaptation. arXiv preprint arXiv:2306.08162.

Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, et al. 2024a. Db-ilm: Accurate dual-binarization for efficient llms. arXiv preprint arXiv:2402.11960.

Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. 2024b. Prefixquant: Eliminating outliers by prefixed tokens for large language models quantization. arXiv preprint arXiv:2410.05265.

Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, and Kaokao Lv. 2023. Optimize weight rounding via signed gradient descent for the quantization of llms. arXiv preprint arXiv:2309.05516.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Together Computer. 2023. Redpajama: an open dataset for training large language models.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023a. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314.

Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefer, and Dan Alistarh. 2023b. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078.

Xin Ding, Xiaoyu Liu, Yun Zhang, Zhijun Tu, Wei Li, Jie Hu, Hanting Chen, Yehui Tang, Zhiwei Xiong, Baoqun Yin, et al. 2023. Cbq: Cross-block quantization for large language models. arXiv preprint arXiv:2312.07950.

Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. 2024. Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation. arXiv preprint arXiv:2402.10631.

Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. 2024. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06118.

Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. 2019. Learned step size quantization. arXiv preprint arXiv:1902.08153.

Elias Frantar, Saleh Ashkboos, Torsten Hoefer, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.

Elias Frantar, Roberto L Castro, Jiale Chen, Torsten Hoefer, and Dan Alistarh. 2024. Marlin: Mixed-precision auto-regressive parallel inference on large language models. arXiv preprint arXiv:2408.11743.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306.13394.
Ruihao Gong, Yang Yong, Shiqiao Gu, Yushi Huang, Yunchen Zhang, Xianglong Liu, and Dacheng Tao. 2024. Llm-qbench: A benchmark towards the best practice for post-training quantization of large language models. arXiv preprint arXiv:2405.06001.

Han Guo, Philip Greengard, Eric P Xing, and Yoon Kim. 2023. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.12023.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.05300.

J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685.

Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. 2024. How good are low-bit quantized llama3 models? an empirical study. arXiv preprint arXiv:2404.14047.

Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joon-suk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. 2023a. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. arXiv preprint arXiv:2305.14152.

Sehoon Kim, Coleman Hooper, Ammar Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2023b. Squeezeelm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629.

Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. 2024. Scaling laws for precision. arXiv preprint arXiv:2411.04330.

Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. 2023a. Owq: Lessons learned from activation outliers for weight quantization in large language models. arXiv preprint arXiv:2306.02272.

Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. 2023b. Flexround: Learnable rounding based on element-wise division for post-training quantization. In International Conference on Machine Learning, pages 18913–18939. PMLR.

Qingyuan Li, Ran Meng, Yiduo Li, Bo Zhang, Liang Li, Yifan Lu, Xiangxiang Chu, Yerui Sun, and Yuchen Xie. 2023a. A speed odyssey for deployable quantization of llms. arXiv preprint arXiv:2311.09550.

Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. 2023b. Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659.

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426.

Baohao Liao and Christof Monz. 2024. Apiq: Finetuning of 2-bit quantized large language model. arXiv preprint arXiv:2402.05147.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. 2023. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978.

Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2024. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. arXiv preprint arXiv:2304.08485.

Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. 2023c. Qllm: Accurate and efficient low-bitwidth quantization for large language models. arXiv preprint arXiv:2310.08041.

Yuanzhan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023d. Mmbench: Is your multi-modal model an all-around player? ArXiv, abs/2307.06281.

Zechun Liu, Barlas Ogun, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2023e. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888.

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and A. Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. ArXiv, abs/2209.09513.

Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. 2024. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197–7206. PMLR.
Xu Ouyang, Tao Ge, Thomas Hartvigsen, Zhisong Zhang, Haitao Mi, and Dong Yu. 2024. Low-bit quantization favors undertrained llms: Scaling laws for quantized llms with 100t training tokens. arXiv preprint arXiv:2411.17691.

Haotong Qin, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xianglong Liu, and Michele Magno. 2024. Accurate lora-finetuning quantization of llms via information retention. arXiv preprint arXiv:2402.05445.

Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shi Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bo Li, Ziwei Tang, Jing Yi, Yu Zhu, Zhenning Dai, Lan Yan, Xin Cong, Ya-Ting Lu, Weilin Zhao, Yuxiang Huang, Jun-Han Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023a. Tool learning with foundation models. ArXiv, abs/2304.08354.

Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023b. Toolllm: Facilitating large language models to master 16000+ real-world apis. ArXiv, abs/2307.16789.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.

Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. 2023. Pb-llm: Partially binarized large language models. arXiv preprint arXiv:2310.00034.

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2023. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

MLC team. 2023. MLC-LLM.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Sounya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396.

Lei Wang, Lingxiao Ma, Shijie Cao, Quanlu Zhang, Jilong Xue, Yining Shi, Ningxin Zheng, Ziming Miao, Fan Yang, Ting Cao, Yuqing Yang, and Mao Yang. 2024. Ladder: Enabling efficient low-precision deep learning computing through hardware-aware tensor transformation. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 307–323, Santa Clara, CA. USENIX Association.

Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, and Mao Yang. 2024. T-mac: Cpu renaissance via table lookup for low-bit llm deployment on edge. Preprint, arXiv:2407.00088.

Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. 2023. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145.

Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. 2022. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402–17414.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR.

Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. 2024a. Besa: Pruning large language models with blockwise parameter-efficient sparsity allocation. arXiv preprint arXiv:2402.16880.

Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023a. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265.

Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengcheng Zhang, Zhensu Chen, Xiaopeng Zhang, and Qi Tian. 2023b. Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717.

Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. 2024b. Onebit: Towards extremely low-bit large language models. arXiv preprint arXiv:2402.11295.

Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi
Lin, Shuo Liu, et al. 2024. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. ArXiv, abs/2308.02490.

Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xing-gang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. 2023. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089.

Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, et al. 2024. Llm inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402.16363.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830.

Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2023. Atom: Low-bit quantization for efficient and accurate llm serving. arXiv preprint arXiv:2310.19102.
Overview of Appendix

This appendix includes the following sections:

• Sec A gives the reproducibility statement to summarize the information related to the reproduction of our method.

• Sec. B describes the gradient calculation in the Block-AP process.

• Sec. C presents the speedup ratio of uniform quantization using BitBLAS (Wang et al., 2024).

• Sec. D details the sources of results for each comparison method to aid reproduction.

• Sec. E presents the sizes of quantized models.

• Sec. F provides additional ablation studies, including those on group size and training datasets.

• Sec. G applies the proposed EfficientQAT to Llava (Liu et al., 2023b) models.

• Sec. H persons the comparisons with some PTQ methods with same number of calibration samples.

• Sec. I presents the detailed accuracy for each zero-shot task.

A  Reproducibility Statement

In this section, we summarize the necessary information to reproduce our results. We provide the training and evaluation details at the beginning of each sub-section in Sec. 4. We also provide the source of detailed results for each compared method in Sec.D.

B  Gradient of Trainable Parameters in Block-AP

Block-AP, aligned with LSQ+(Bhalgat et al., 2020), uses a straight-through estimator (STE)(Bengio et al., 2013) to facilitate gradient computation through the rounding operation. The gradients of scaling factor \( s \) are computed as follows:

\[
\frac{\partial \hat{w}}{\partial s} = \begin{cases}
\lfloor \frac{w}{s} \rfloor - \frac{w}{s}, & 0 \leq \lfloor \frac{w}{s} \rfloor + z \leq 2^{N-1}, \\
-z, & \lfloor \frac{w}{s} \rfloor + z < 0, \\
2^{N-1} - z, & \lfloor \frac{w}{s} \rfloor + z > 2^{N-1}.
\end{cases}
\]

and the gradient with respect to zero point \( z \) is:

\[
\frac{\partial \hat{w}}{\partial z} = \begin{cases}
0, & 0 \leq \lfloor \frac{w}{s} \rfloor + z \leq 2^{N-1}, \\
-1, & otherwise,
\end{cases}
\]

and the full-precision weight \( \mathbf{W} \) can also be updated through its gradient†:

\[
\frac{\partial \hat{w}}{\partial w} = \begin{cases}
1, & 0 \leq \lfloor \frac{w}{s} \rfloor + z \leq 2^{N-1}, \\
0, & otherwise,
\end{cases}
\]

C  Speedup with BitBlas

According to Table 10, INT2 quantization enhances the forward-pass speed by approximately 2.9x to 4.4x.

† \( \hat{w}, w \) is a element from \( \hat{W}, \mathbf{W} \)
Table 10: Speed of the FP16 linear layer matrix-vector multiplication in PyTorch, and relative INT2 speedups in BitBLAS (Wang et al., 2024). Testing on A100-80GB GPU.

<table>
  <tr>
    <th>Llama-2</th>
    <th>7B</th>
    <th>13B</th>
    <th>70B</th>
  </tr>
  <tr>
    <th>size (out_c × in_c)</th>
    <th>4096x4096</th>
    <th>11008x4096</th>
    <th>5120x5120</th>
    <th>13824x5120</th>
    <th>8192x8192</th>
    <th>28672x8192</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>25 us</td>
    <td>61 us</td>
    <td>38 us</td>
    <td>90 us</td>
    <td>91 us</td>
    <td>286 us</td>
  </tr>
  <tr>
    <td>INT2</td>
    <td>9 us</td>
    <td>21 us</td>
    <td>11 us</td>
    <td>26 us</td>
    <td>24 us</td>
    <td>67 us</td>
  </tr>
  <tr>
    <td>Speedup</td>
    <td>3.1x</td>
    <td>2.9x</td>
    <td>3.6x</td>
    <td>3.5x</td>
    <td>3.9x</td>
    <td>4.4x</td>
  </tr>
</table>

D Results Source of Other Method.

In this study, we present a thorough comparison of our method against existing PTQ techniques, including GPTQ (Frantar et al., 2022), AWQ (Lin et al., 2023), OmniQ (Shao et al., 2023), AutoRound (Cheng et al., 2023), QuIP# (Tseng et al., 2024), and AQLM (Egiazarian et al., 2024). We also compare with existing QAT methods, including LLM-QAT (Liu et al., 2023e), BitDistiller (Du et al., 2024), PB-LLM (Shang et al., 2023) and DB-LLM (Chen et al., 2024a). Additionally, we also evaluate quantized parameter-efficient fine-tuning methods such as PEQA (Kim et al., 2023a), QLoRA (Dettmers et al., 2023a), QA-LoRA (Xu et al., 2023b), and IR-QLoRA (Qin et al., 2024). The results we discuss originate from their respective official publications, and other scholarly articles, or are derived from our reproduction. We meticulously document the source of the results for each method as follows:

• GPTQ, AWQ, OmniQ, AutoRound: The zero-shot accuracy results for Llama-2 models using these methods are derived from the AutoRound GitHub repository‡. The perplexity results for the Llama-2 models using GPTQ, AWQ, and OmniQ are taken from the OmniQ paper (Shao et al., 2023). The results for Llama-3 models using AWQ§ and GPTQ¶ were obtained through their open-source implementations.

• QuIP#, AQLM: We replicated the results using the official pre-trained models provided by QuIP#‖ and AQLM**.

• LLM-QAT, BitDistiller: These results are cited from BitDistiller (Du et al., 2024) paper.

• PB-LLM, DB-LLM: These results are cited from recent Llama-3 quantization empirical study (Huang et al., 2024).

• ApiQ: These results are cited from IR-ApiQ (Liao and Monz, 2024) paper.

• PEQA: The per-channel quantization results (g=-1) are cited from their publication (Kim et al., 2023a), and the results for a group size of 64 were produced using our codebase.

• QA-LoRA, QLoRA, QLoRA w/ GPTQ: These results are cited from QA-LoRA (Xu et al., 2023b) paper.

• IR-QLoRA: These results are cited from IR-QLoRA (Qin et al., 2024) paper.

‡AutoRound: https://github.com/intel/auto-round/blob/main/docs/acc.md
§AWQ:https://github.com/mit-han-lab/llm-awq
¶GPTQ:https://github.com/qwopqwop200/GPTQ-for-LLaMa
‖https://github.com/Cornell-RelaxML/quip-sharp
**https://github.com/Vahe1994/AQLM
Table 11: Model size of quantized models. Compression ratio indicates the compression ratio of quantized models compared with FP16 models.

<table>
  <tr>
    <th>Model</th>
    <th># Bit</th>
    <th>Group size</th>
    <th>bits/param</th>
    <th>size (GiB)</th>
    <th>Compression ratio (%)</th>
  </tr>
  <tr>
    <td rowspan="7">LLaMA-2-7B</td>
    <td>16</td>
    <td>-</td>
    <td>16</td>
    <td>12.55</td>
    <td>-</td>
  </tr>
  <tr>
    <td>4</td>
    <td>32</td>
    <td>4.63</td>
    <td>3.98</td>
    <td>68.33</td>
  </tr>
  <tr>
    <td>4</td>
    <td>64</td>
    <td>4.31</td>
    <td>3.74</td>
    <td>70.20</td>
  </tr>
  <tr>
    <td>4</td>
    <td>128</td>
    <td>4.16</td>
    <td>3.62</td>
    <td>71.14</td>
  </tr>
  <tr>
    <td>3</td>
    <td>32</td>
    <td>3.59</td>
    <td>3.35</td>
    <td>73.28</td>
  </tr>
  <tr>
    <td>3</td>
    <td>64</td>
    <td>3.30</td>
    <td>3.13</td>
    <td>75.08</td>
  </tr>
  <tr>
    <td>3</td>
    <td>128</td>
    <td>3.15</td>
    <td>3.01</td>
    <td>75.98</td>
  </tr>
  <tr>
    <td>2</td>
    <td>32</td>
    <td>2.56</td>
    <td>2.42</td>
    <td>80.71</td>
  </tr>
  <tr>
    <td>2</td>
    <td>64</td>
    <td>2.28</td>
    <td>2.21</td>
    <td>82.40</td>
  </tr>
  <tr>
    <td>2</td>
    <td>128</td>
    <td>2.14</td>
    <td>2.10</td>
    <td>83.25</td>
  </tr>
  <tr>
    <td rowspan="7">LLaMA-2-13B</td>
    <td>16</td>
    <td>-</td>
    <td>16</td>
    <td>24.24</td>
    <td>-</td>
  </tr>
  <tr>
    <td>4</td>
    <td>32</td>
    <td>4.63</td>
    <td>7.44</td>
    <td>69.30</td>
  </tr>
  <tr>
    <td>4</td>
    <td>64</td>
    <td>4.31</td>
    <td>6.98</td>
    <td>71.21</td>
  </tr>
  <tr>
    <td>4</td>
    <td>128</td>
    <td>4.16</td>
    <td>6.75</td>
    <td>72.16</td>
  </tr>
  <tr>
    <td>3</td>
    <td>32</td>
    <td>3.59</td>
    <td>6.22</td>
    <td>74.33</td>
  </tr>
  <tr>
    <td>3</td>
    <td>64</td>
    <td>3.30</td>
    <td>5.78</td>
    <td>76.16</td>
  </tr>
  <tr>
    <td>3</td>
    <td>128</td>
    <td>3.15</td>
    <td>5.56</td>
    <td>77.07</td>
  </tr>
  <tr>
    <td>2</td>
    <td>32</td>
    <td>2.56</td>
    <td>4.40</td>
    <td>81.87</td>
  </tr>
  <tr>
    <td>2</td>
    <td>64</td>
    <td>2.28</td>
    <td>3.98</td>
    <td>83.58</td>
  </tr>
  <tr>
    <td>2</td>
    <td>128</td>
    <td>2.14</td>
    <td>3.77</td>
    <td>84.44</td>
  </tr>
  <tr>
    <td rowspan="7">LLaMA-2-70B</td>
    <td>16</td>
    <td>-</td>
    <td>16</td>
    <td>128.48</td>
    <td>-</td>
  </tr>
  <tr>
    <td>4</td>
    <td>32</td>
    <td>4.63</td>
    <td>37.83</td>
    <td>70.55</td>
  </tr>
  <tr>
    <td>4</td>
    <td>64</td>
    <td>4.31</td>
    <td>35.34</td>
    <td>72.49</td>
  </tr>
  <tr>
    <td>4</td>
    <td>128</td>
    <td>4.16</td>
    <td>34.10</td>
    <td>73.46</td>
  </tr>
  <tr>
    <td>3</td>
    <td>32</td>
    <td>3.59</td>
    <td>31.26</td>
    <td>75.67</td>
  </tr>
  <tr>
    <td>3</td>
    <td>64</td>
    <td>3.30</td>
    <td>28.87</td>
    <td>77.53</td>
  </tr>
  <tr>
    <td>3</td>
    <td>128</td>
    <td>3.15</td>
    <td>27.67</td>
    <td>78.46</td>
  </tr>
  <tr>
    <td>2</td>
    <td>32</td>
    <td>2.56</td>
    <td>21.40</td>
    <td>83.34</td>
  </tr>
  <tr>
    <td>2</td>
    <td>64</td>
    <td>2.28</td>
    <td>19.16</td>
    <td>85.09</td>
  </tr>
  <tr>
    <td>2</td>
    <td>128</td>
    <td>2.14</td>
    <td>18.04</td>
    <td>85.96</td>
  </tr>
</table>

E  Size of Quantized Models

This section illustrates model size reduction achieved through quantization. Models quantized to low-bit representations are more compact.

We implement N-bit quantization with a grouping size of \( g \), where each group of \( g \) weights shares the same FP16 step size and an N-bit zero point. Consequently, the average number of bits per parameter is calculated as \( N + \frac{N+16}{g} \). It is important to note that only the linear layers within the transformer blocks are quantized; other layers, such as normalization layers, embeddings, and the classification head, remain in FP16 format. Table 11 provides detailed comparisons of quantized model sizes and their compression ratios.

Table 12: Llma-2-7B 2-bit quantization performance with different group sizes for proposed EfficientQAT.

<table>
  <tr>
    <th>Group</th>
    <th>Avg. Bits</th>
    <th>Avg. PPL</th>
    <th>Avg. Accuracy</th>
  </tr>
  <tr>
    <td>32</td>
    <td>2.56</td>
    <td>7.59</td>
    <td>60.28</td>
  </tr>
  <tr>
    <td>64</td>
    <td>2.28</td>
    <td>7.68</td>
    <td>60.14</td>
  </tr>
  <tr>
    <td>128</td>
    <td>2.10</td>
    <td>7.99</td>
    <td>59.50</td>
  </tr>
  <tr>
    <td>256</td>
    <td>2.07</td>
    <td>8.18</td>
    <td>58.67</td>
  </tr>
</table>

F  Additional Ablation Analysis

Quantization Group Size. The group size is a crucial hyperparameter in weight-only quantization. A smaller group size offers more granular compression and reduces quantization loss but increases the number of quantization parameters required. As indicated in Table 12, a group size of 64 strikes an optimal balance for 2-bit quantization using EfficientQAT. It outperforms a group size of 128 by achieving a 0.31 lower perplexity and a 0.64% higher accuracy, yet it slightly underperforms compared to a group
Table 13: Block-AP (w/o E2E-QP) results of Llama-2-7B in different calibration datasets.

<table>
  <tr>
    <th>Bits</th>
    <th>Dataset</th>
    <th>Wiki PPL</th>
    <th>C4 PPL</th>
    <th>Avg. Accuracy</th>
  </tr>
  <tr>
    <td>w3g128</td>
    <td>WikiText2</td>
    <td>5.72</td>
    <td>7.52</td>
    <td>63.24</td>
  </tr>
  <tr>
    <td>w3g128</td>
    <td>C4</td>
    <td>5.92</td>
    <td>7.38</td>
    <td>63.82</td>
  </tr>
  <tr>
    <td>w3g128</td>
    <td>Redpajama</td>
    <td>5.91</td>
    <td>7.41</td>
    <td>63.50</td>
  </tr>
  <tr>
    <td>w2g64</td>
    <td>WikiText2</td>
    <td>6.73</td>
    <td>9.89</td>
    <td>58.26</td>
  </tr>
  <tr>
    <td>w2g64</td>
    <td>C4</td>
    <td>7.87</td>
    <td>9.30</td>
    <td>59.24</td>
  </tr>
  <tr>
    <td>w2g64</td>
    <td>Redpajama</td>
    <td>7.70</td>
    <td>9.36</td>
    <td>58.99</td>
  </tr>
</table>

size of 32, with a marginal difference of 0.09 in perplexity and 0.14% in accuracy.

**Training Dataset.** More trainable parameters can increase the risk of overfitting. Previous works (Gong et al., 2024) show that a similar distribution between the calibration dataset and the test dataset can improve test accuracy. RedPajama and C4 datasets are diverse, while WikiText2 is simpler and sourced from Wikipedia. The close distribution of training and test datasets for WikiText2 results in significantly lower WikiText2 perplexity when using it as a calibration dataset. However, the average accuracy of zero-shot tasks in Table R7 shows that Block-AP’s generation ability is excellent, with only 0.26% and 1.28% accuracy declines when changing the calibration dataset from RedPajama to WikiText2 for w3g128 and w2g64, respectively. Additionally, using C4 as a calibration dataset can even increase the average accuracy by 0.2-0.3 points. Overall, we recommend using Block-AP with more diverse calibration datasets like C4 or RedPajama.

Table 14: Results about instruction tuning of large vision-language models. We following the overall training pipeling of LLaVA-1.5 (Liu et al., 2023a) and just change the fine-tuning methods. ‘QLoRA + Block-AP’ indicates that we leverage proposed Block-AP to quantized the QLoRA models into low-bits for fair comparisons. † MME’s perception scores are normalized to 100 percent.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="2">#Bit</th>
    <th colspan="2">MMbench</th>
    <th>MME<sup>†</sup></th>
    <th>MM-Vet</th>
    <th>ScienceQA</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th>Training</th>
    <th>Inference</th>
    <th>MMbench</th>
    <th>MME<sup>†</sup></th>
    <th>MM-Vet</th>
    <th>ScienceQA</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="4">LLaVA-1.5-7B</td>
    <td>LoRA</td>
    <td>16</td>
    <td>16</td>
    <td>66.1</td>
    <td>73.8</td>
    <td>30.2</td>
    <td>68.4</td>
    <td>59.6</td>
  </tr>
  <tr>
    <td>QLoRA</td>
    <td>4+16</td>
    <td>16</td>
    <td>64.1</td>
    <td>72.8</td>
    <td>30.3</td>
    <td>68.0</td>
    <td>58.8</td>
  </tr>
  <tr>
    <td>QLoRA + Block-AP</td>
    <td>4+16</td>
    <td>16</td>
    <td>63.6</td>
    <td>72.0</td>
    <td>29.8</td>
    <td>67.7</td>
    <td>58.3</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>4</td>
    <td>4</td>
    <td>64.4</td>
    <td>73.2</td>
    <td>30.3</td>
    <td>68.1</td>
    <td>58.8(+0.5)</td>
  </tr>
  <tr>
    <td rowspan="4">LLaVA-1.5-13B</td>
    <td>LoRA</td>
    <td>16</td>
    <td>16</td>
    <td>68.5</td>
    <td>77.1</td>
    <td>38.3</td>
    <td>71.2</td>
    <td>63.8</td>
  </tr>
  <tr>
    <td>QLoRA</td>
    <td>4+16</td>
    <td>16</td>
    <td>67.6</td>
    <td>76.9</td>
    <td>36.0</td>
    <td>69.9</td>
    <td>62.7</td>
  </tr>
  <tr>
    <td>QLoRA + Block-AP</td>
    <td>4+16</td>
    <td>16</td>
    <td>67.7</td>
    <td>76.7</td>
    <td>35.6</td>
    <td>69.3</td>
    <td>62.4</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>4</td>
    <td>4</td>
    <td>67.5</td>
    <td>74.8</td>
    <td>35.6</td>
    <td>70.2</td>
    <td>62.0(+0.4)</td>
  </tr>
  <tr>
    <td rowspan="4"></td>
    <td>QLoRA</td>
    <td>4+16</td>
    <td>3</td>
    <td>66.8</td>
    <td>75.5</td>
    <td>34.5</td>
    <td>68.4</td>
    <td>61.3</td>
  </tr>
  <tr>
    <td>QLoRA + Block-AP</td>
    <td>4+16</td>
    <td>3</td>
    <td>67.4</td>
    <td>74.8</td>
    <td>35.3</td>
    <td>69.3</td>
    <td>61.7(+0.4)</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>3</td>
    <td>3</td>
    <td>67.4</td>
    <td>74.8</td>
    <td>35.3</td>
    <td>69.3</td>
    <td>61.7(+0.4)</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>2</td>
    <td>2</td>
    <td>62.5</td>
    <td>72.1</td>
    <td>32.5</td>
    <td>65.0</td>
    <td>58.0</td>
  </tr>
  <tr>
    <td></td>
    <td>EfficientQAT</td>
    <td>2</td>
    <td>2</td>
    <td>63.9</td>
    <td>73.1</td>
    <td>33.9</td>
    <td>68.6</td>
    <td>59.9(+1.9)</td>
  </tr>
</table>

G Instruction Tuning for LVLMs.

Traditional Q-PEFT methods only do experiments on the language models. In this section, we further extend proposed EfficientQAT into Large vision-Language models (LVLMs) such as LLaVA (Liu et al., 2023b).

**Training and Evaluation.** For the fine-tuning of large vision-language models (LVLMs), we largely align with LLaVA1.5 (Liu et al., 2023a), which encompass the training model, datasets, and hyperparameters††. Unlike LLaVA1.5, which begins fine-tuning with full-precision Vicuna models using either full

††For comprehensive details, please consult the official repository at https://github.com/haotian-liu/LLaVA.
fine-tuning or LoRA-based methods (Hu et al., 2021). EfficientQAT starts with Vicuna models already quantized using our Block-AP method and continues with our E2E-QP fine-tuning approach. The training process involves two steps: initially freezing the LLM and pre-training a projector to align features with a Vision Transformer (ViT), followed by end-to-end fine-tuning of both the LLM and the projector. For EfficientQAT, we modify the learning rates in the second step to \(2 \times 10^{-5}\) for 4-bit and \(3 \times 10^{-5}\) for 2-bit and 3-bit.

Evaluation. Evaluation of the fine-tuned LVLMs are conducted across four benchmarks: MME (Fu et al., 2023), MM-Vet (Yu et al., 2023), MMBench (Liu et al., 2023d), and ScienceQA (Lu et al., 2022).

Baseline. We compare our results with those of QLoRA (Dettmers et al., 2023a), applying our Block-AP method to quantize the QLoRA fine-tuned models to low bits for fair comparison.

Results. As shown in Table 14, EfficientQAT outperforms QLoRA (Dettmers et al., 2023a) in low-bit settings for both LLaVA-1.5-7B and LLaVA-1.5-13B models, consistent with previous results in LMMs. Remarkably, the 2-bit LLaVA-1.5-13B model trained with EfficientQAT achieves an average score of 59.9, surpassing the 59.6 of the FP16 LLaVA-1.5-7B model trained with LoRA. However, there is a slight performance decrease observed in the 4-bit EfficientQAT and 16-bit QLoRA compared to the 16-bit LoRA, indicating that further research is needed to optimize Q-PEFT within LVLMs.

H Comparisons with the Same Number of Data Samples

The main experiments use 4096 samples for the proposed method. However, some PTQ methods, such as OmniQuant (Shao et al., 2023) and GPTQ (Frantar et al., 2022), use only 128 samples for quantization. To ensure a fair comparison, we also evaluate EfficientQAT against OmniQuant and GPTQ using the same number of data samples. As shown in Table 15, the performance of OmniQuant (Shao et al., 2023) and GPTQ (Frantar et al., 2022) stabilizes at 128 samples and does not improve with additional data, while EfficientQAT continues to benefit from more samples. Even with only 128 samples, EfficientQAT significantly outperforms OmniQuant (8.02 PPL vs. 15.02 PPL). Furthermore, Table 1 shows that EfficientQAT surpasses DB-LLM, which uses 20k samples, despite EfficientQAT using only 4096 samples. These results confirm the consistent superiority of EfficientQAT over other uniform quantization methods, highlighting its effectiveness.

<table>
  <tr>
    <th>Method</th>
    <th>Precision</th>
    <th>64</th>
    <th>128</th>
    <th>256</th>
    <th>512</th>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>W3g128</td>
    <td>7.91</td>
    <td>7.89</td>
    <td>7.90</td>
    <td>7.89</td>
  </tr>
  <tr>
    <td>OmniQuant</td>
    <td>W3g128</td>
    <td>7.70</td>
    <td>7.75</td>
    <td>7.73</td>
    <td>7.74</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>W3g128</td>
    <td>7.40</td>
    <td>7.37</td>
    <td>7.36</td>
    <td>7.35</td>
  </tr>
  <tr>
    <td>OmniQuant</td>
    <td>W2g128</td>
    <td>15.23</td>
    <td>15.02</td>
    <td>14.95</td>
    <td>14.93</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>W2g128</td>
    <td>9.01</td>
    <td>8.95</td>
    <td>8.85</td>
    <td>8.83</td>
  </tr>
</table>

Table 15: C4 perplexity of Llama-2-7B with different training samples.

I Full Results

In Table 1, we present the average accuracy for five zero-shot tasks. This section offers a detailed breakdown of the task-specific accuracy numbers. Specifically, 16 and 17 detail the performance of 3-bit and 2-bit quantization, respectively.
Table 16: 3-bit Llama 2 & 3 zero-shot accuracy by lm_eval v0.4.2 ( acc is reported, not acc_norm )

<table>
  <tr>
    <th>Model</th>
    <th>Method</th>
    <th>Bits</th>
    <th>Group</th>
    <th>WinoGrande</th>
    <th>HellaSwag</th>
    <th>ArcC</th>
    <th>ArcE</th>
    <th>PiQA</th>
    <th>Average accuracy†</th>
  </tr>
  <tr>
    <td rowspan="7">2-7B</td>
    <td>-</td>
    <td>16</td>
    <td></td>
    <td>69.22</td>
    <td>57.16</td>
    <td>43.52</td>
    <td>76.26</td>
    <td>78.07</td>
    <td>64.85</td>
  </tr>
  <tr>
    <td>RTN</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>67.56</td>
    <td>54.90</td>
    <td>38.57</td>
    <td>72.98</td>
    <td>76.28</td>
    <td>62.06</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>68.59</td>
    <td>53.66</td>
    <td>40.19</td>
    <td>73.74</td>
    <td>76.01</td>
    <td>62.44</td>
  </tr>
  <tr>
    <td>AWQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>67.40</td>
    <td>54.98</td>
    <td>41.64</td>
    <td>74.07</td>
    <td>76.01</td>
    <td>62.82</td>
  </tr>
  <tr>
    <td>OmniQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>66.69</td>
    <td>54.42</td>
    <td>39.85</td>
    <td>74.37</td>
    <td>76.77</td>
    <td>62.42</td>
  </tr>
  <tr>
    <td>AutoRound</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>68.27</td>
    <td>55.33</td>
    <td>42.92</td>
    <td>75.25</td>
    <td>76.82</td>
    <td>63.72</td>
  </tr>
  <tr>
    <td>QuIP#</td>
    <td>3</td>
    <td>-</td>
    <td></td>
    <td>68.19</td>
    <td>55.85</td>
    <td>41.89</td>
    <td>74.62</td>
    <td>77.04</td>
    <td>63.52</td>
  </tr>
  <tr>
    <td rowspan="7">2-13B</td>
    <td>EfficientQAT</td>
    <td>3</td>
    <td>128</td>
    <td><b>69.14</b></td>
    <td><b>55.90</b></td>
    <td><b>42.83</b></td>
    <td><b>74.66</b></td>
    <td><b>77.58</b></td>
    <td><b>64.02</b></td>
  </tr>
  <tr>
    <td>-</td>
    <td>16</td>
    <td>-</td>
    <td></td>
    <td>72.22</td>
    <td>60.07</td>
    <td>48.29</td>
    <td>79.42</td>
    <td>79.05</td>
    <td>67.81</td>
  </tr>
  <tr>
    <td>RTN</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>70.72</td>
    <td>57.74</td>
    <td>44.62</td>
    <td>77.69</td>
    <td>78.07</td>
    <td>65.77</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>70.88</td>
    <td>57.83</td>
    <td>45.65</td>
    <td>77.99</td>
    <td>78.56</td>
    <td>66.18</td>
  </tr>
  <tr>
    <td>AWQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>71.82</td>
    <td>58.58</td>
    <td>44.62</td>
    <td>77.95</td>
    <td>77.75</td>
    <td>66.14</td>
  </tr>
  <tr>
    <td>OmniQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>70.01</td>
    <td>58.46</td>
    <td>46.16</td>
    <td>77.86</td>
    <td>78.40</td>
    <td>66.18</td>
  </tr>
  <tr>
    <td>AutoRound</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>71.59</td>
    <td>59.11</td>
    <td>45.82</td>
    <td>78.58</td>
    <td>78.29</td>
    <td>66.68</td>
  </tr>
  <tr>
    <td>QuIP#</td>
    <td>3</td>
    <td>-</td>
    <td></td>
    <td>72.45</td>
    <td>58.26</td>
    <td>44.62</td>
    <td>77.90</td>
    <td>78.07</td>
    <td>66.26</td>
  </tr>
  <tr>
    <td rowspan="7">2-70B</td>
    <td>EfficientQAT</td>
    <td>3</td>
    <td>128</td>
    <td><b>72.06</b></td>
    <td><b>59.01</b></td>
    <td><b>47.95</b></td>
    <td><b>79.00</b></td>
    <td><b>78.40</b></td>
    <td><b>67.28</b></td>
  </tr>
  <tr>
    <td>-</td>
    <td>16</td>
    <td>-</td>
    <td></td>
    <td>77.98</td>
    <td>64.77</td>
    <td>54.44</td>
    <td>82.70</td>
    <td>82.15</td>
    <td>72.41</td>
  </tr>
  <tr>
    <td>RTN</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>77.90</td>
    <td>61.98</td>
    <td>52.39</td>
    <td>81.10</td>
    <td>80.79</td>
    <td>70.83</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>77.66</td>
    <td>62.94</td>
    <td>53.67</td>
    <td>81.65</td>
    <td>81.45</td>
    <td>71.47</td>
  </tr>
  <tr>
    <td>AWQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>76.48</td>
    <td>63.75</td>
    <td>53.67</td>
    <td>81.40</td>
    <td>81.77</td>
    <td>71.41</td>
  </tr>
  <tr>
    <td>OmniQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>76.48</td>
    <td>63.54</td>
    <td>52.82</td>
    <td>81.02</td>
    <td>81.50</td>
    <td>71.07</td>
  </tr>
  <tr>
    <td>AutoRound</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>76.56</td>
    <td>63.83</td>
    <td>52.56</td>
    <td>81.73</td>
    <td>81.50</td>
    <td>71.24</td>
  </tr>
  <tr>
    <td>QuIP#</td>
    <td>3</td>
    <td>-</td>
    <td></td>
    <td>76.24</td>
    <td>64.22</td>
    <td>55.89</td>
    <td>82.11</td>
    <td>82.21</td>
    <td>72.13</td>
  </tr>
  <tr>
    <td rowspan="7">3-8B</td>
    <td>EfficientQAT</td>
    <td>3</td>
    <td>128</td>
    <td><b>77.27</b></td>
    <td><b>64.20</b></td>
    <td><b>53.75</b></td>
    <td><b>81.73</b></td>
    <td><b>81.83</b></td>
    <td><b>71.76</b></td>
  </tr>
  <tr>
    <td>-</td>
    <td>16</td>
    <td>-</td>
    <td></td>
    <td>72.61</td>
    <td>60.17</td>
    <td>50.43</td>
    <td>80.09</td>
    <td>79.60</td>
    <td>68.58</td>
  </tr>
  <tr>
    <td>RTN</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>66.54</td>
    <td>50.87</td>
    <td>36.69</td>
    <td>65.36</td>
    <td>74.16</td>
    <td>58.72</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>70.88</td>
    <td>55.13</td>
    <td>37.80</td>
    <td>65.24</td>
    <td>73.83</td>
    <td>60.58</td>
  </tr>
  <tr>
    <td>AWQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>70.96</td>
    <td>55.43</td>
    <td>44.20</td>
    <td>75.84</td>
    <td>77.69</td>
    <td>64.82</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>3</td>
    <td>128</td>
    <td><b>71.51</b></td>
    <td><b>57.81</b></td>
    <td><b>48.81</b></td>
    <td><b>80.01</b></td>
    <td><b>78.63</b></td>
    <td><b>67.35</b></td>
  </tr>
  <tr>
    <td rowspan="7">3-70B</td>
    <td>-</td>
    <td>16</td>
    <td>-</td>
    <td></td>
    <td>80.51</td>
    <td>66.36</td>
    <td>60.41</td>
    <td>86.99</td>
    <td>82.37</td>
    <td>75.33</td>
  </tr>
  <tr>
    <td>RTN</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>65.90</td>
    <td>54.22</td>
    <td>48.46</td>
    <td>78.83</td>
    <td>79.05</td>
    <td>65.29</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>78.14</td>
    <td>62.58</td>
    <td>52.99</td>
    <td>82.07</td>
    <td>80.63</td>
    <td>71.28</td>
  </tr>
  <tr>
    <td>AWQ</td>
    <td>3</td>
    <td>128</td>
    <td></td>
    <td>78.85</td>
    <td>64.26</td>
    <td>58.36</td>
    <td>84.51</td>
    <td>82.26</td>
    <td>73.65</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>3</td>
    <td>128</td>
    <td><b>78.65</b></td>
    <td><b>65.58</b></td>
    <td><b>58.53</b></td>
    <td><b>84.72</b></td>
    <td><b>82.32</b></td>
    <td><b>73.96</b></td>
  </tr>
</table>
Table 17: 2-bit Llama 2 & 3 zero-shot accuracy by lm_eval v0.4.2 ( acc is reported, not acc_norm )

<table>
  <tr>
    <th>Model</th>
    <th>Method</th>
    <th>Bits</th>
    <th>Group</th>
    <th>WinoGrande</th>
    <th>HellaSwag</th>
    <th>ArcC</th>
    <th>ArcE</th>
    <th>PiQA</th>
    <th>Average accuracy†</th>
  </tr>
  <tr>
    <td rowspan="6">2-7B</td>
    <td>-</td>
    <td>16</td>
    <td></td>
    <td>69.22</td>
    <td>57.16</td>
    <td>43.52</td>
    <td>76.26</td>
    <td>78.07</td>
    <td>64.85</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>55.17</td>
    <td>32.59</td>
    <td>21.25</td>
    <td>40.45</td>
    <td>58.32</td>
    <td>41.56</td>
  </tr>
  <tr>
    <td>OmniQ</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>55.88</td>
    <td>40.28</td>
    <td>23.46</td>
    <td>50.13</td>
    <td>65.13</td>
    <td>46.98</td>
  </tr>
  <tr>
    <td>AutoRound</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>61.01</td>
    <td>40.28</td>
    <td>32.25</td>
    <td>65.99</td>
    <td>72.96</td>
    <td>54.50</td>
  </tr>
  <tr>
    <td>AQLM</td>
    <td>2</td>
    <td>2x8</td>
    <td></td>
    <td>65.27</td>
    <td>49.96</td>
    <td>32.85</td>
    <td>66.92</td>
    <td>73.07</td>
    <td>57.61</td>
  </tr>
  <tr>
    <td>AQLM</td>
    <td>2</td>
    <td>1x16</td>
    <td></td>
    <td>65.19</td>
    <td>53.42</td>
    <td>39.68</td>
    <td>74.07</td>
    <td>76.88</td>
    <td>61.85</td>
  </tr>
  <tr>
    <td>QuIP#</td>
    <td>2</td>
    <td>-</td>
    <td></td>
    <td>65.67</td>
    <td>52.19</td>
    <td>37.88</td>
    <td>71.84</td>
    <td>75.46</td>
    <td>60.61</td>
  </tr>
  <tr>
    <td rowspan="2">EfficientQAT</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>66.22</td>
    <td>50.84</td>
    <td>36.52</td>
    <td>69.78</td>
    <td>74.16</td>
    <td>59.50</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>2</td>
    <td>64</td>
    <td></td>
    <td>65.98</td>
    <td>51.58</td>
    <td>36.86</td>
    <td>70.96</td>
    <td>75.30</td>
    <td>60.14</td>
  </tr>
  <tr>
    <td rowspan="6">2-13B</td>
    <td>-</td>
    <td>16</td>
    <td></td>
    <td>72.22</td>
    <td>60.07</td>
    <td>48.29</td>
    <td>79.42</td>
    <td>79.05</td>
    <td>67.81</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>55.80</td>
    <td>41.06</td>
    <td>21.93</td>
    <td>55.60</td>
    <td>67.08</td>
    <td>48.29</td>
  </tr>
  <tr>
    <td>OmniQ</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>57.93</td>
    <td>46.23</td>
    <td>30.29</td>
    <td>63.22</td>
    <td>70.13</td>
    <td>53.56</td>
  </tr>
  <tr>
    <td>AutoRound</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>64.33</td>
    <td>53.35</td>
    <td>38.57</td>
    <td>71.17</td>
    <td>76.17</td>
    <td>60.72</td>
  </tr>
  <tr>
    <td>AQLM</td>
    <td>2</td>
    <td>2x8</td>
    <td></td>
    <td>66.22</td>
    <td>54.62</td>
    <td>40.10</td>
    <td>73.06</td>
    <td>77.09</td>
    <td>62.22</td>
  </tr>
  <tr>
    <td>AQLM</td>
    <td>2</td>
    <td>1x16</td>
    <td></td>
    <td>70.09</td>
    <td>57.62</td>
    <td>43.52</td>
    <td>75.25</td>
    <td>78.29</td>
    <td>64.95</td>
  </tr>
  <tr>
    <td>QuIP#</td>
    <td>2</td>
    <td>-</td>
    <td></td>
    <td>69.06</td>
    <td>56.53</td>
    <td>42.92</td>
    <td>75.72</td>
    <td>77.97</td>
    <td>64.44</td>
  </tr>
  <tr>
    <td rowspan="2">EfficientQAT</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>68.90</td>
    <td>55.66</td>
    <td>42.83</td>
    <td>75.04</td>
    <td>76.99</td>
    <td>63.88</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>2</td>
    <td>64</td>
    <td></td>
    <td>68.36</td>
    <td>55.27</td>
    <td>41.89</td>
    <td>74.83</td>
    <td>77.04</td>
    <td>63.48</td>
  </tr>
  <tr>
    <td rowspan="6">2-70B</td>
    <td>-</td>
    <td>16</td>
    <td></td>
    <td>77.98</td>
    <td>64.77</td>
    <td>54.44</td>
    <td>82.70</td>
    <td>82.15</td>
    <td>72.41</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>49.57</td>
    <td>25.04</td>
    <td>22.70</td>
    <td>25.08</td>
    <td>49.51</td>
    <td>34.38</td>
  </tr>
  <tr>
    <td>OmniQ</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>64.33</td>
    <td>35.45</td>
    <td>33.28</td>
    <td>67.21</td>
    <td>74.10</td>
    <td>54.87</td>
  </tr>
  <tr>
    <td>AutoRound</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>74.90</td>
    <td>59.65</td>
    <td>46.59</td>
    <td>78.37</td>
    <td>79.00</td>
    <td>67.70</td>
  </tr>
  <tr>
    <td>AQLM</td>
    <td>2</td>
    <td>2x8</td>
    <td></td>
    <td>75.61</td>
    <td>61.94</td>
    <td>51.45</td>
    <td>79.76</td>
    <td>80.47</td>
    <td>69.85</td>
  </tr>
  <tr>
    <td>AQLM</td>
    <td>2</td>
    <td>1x16</td>
    <td></td>
    <td>76.01</td>
    <td>62.78</td>
    <td>52.99</td>
    <td>81.36</td>
    <td>81.07</td>
    <td>70.84</td>
  </tr>
  <tr>
    <td>QuIP#</td>
    <td>2</td>
    <td>-</td>
    <td></td>
    <td>75.77</td>
    <td>62.86</td>
    <td>52.65</td>
    <td>81.90</td>
    <td>81.39</td>
    <td>70.91</td>
  </tr>
  <tr>
    <td rowspan="2">EfficientQAT</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>73.64</td>
    <td>61.58</td>
    <td>49.23</td>
    <td>80.01</td>
    <td>80.20</td>
    <td>68.93</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>2</td>
    <td>64</td>
    <td></td>
    <td>74.59</td>
    <td>61.78</td>
    <td>50.77</td>
    <td>80.13</td>
    <td>80.14</td>
    <td>69.48</td>
  </tr>
  <tr>
    <td rowspan="3">3-8B</td>
    <td>-</td>
    <td>16</td>
    <td></td>
    <td>72.61</td>
    <td>60.17</td>
    <td>50.43</td>
    <td>80.09</td>
    <td>79.60</td>
    <td>68.58</td>
  </tr>
  <tr>
    <td>AQLM</td>
    <td>2</td>
    <td>1x16</td>
    <td></td>
    <td>71.82</td>
    <td>55.44</td>
    <td>41.21</td>
    <td>74.24</td>
    <td>77.80</td>
    <td>64.10</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>65.67</td>
    <td>50.74</td>
    <td>36.01</td>
    <td>69.15</td>
    <td>75.30</td>
    <td>59.37</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>2</td>
    <td>64</td>
    <td></td>
    <td>67.72</td>
    <td>51.86</td>
    <td>37.03</td>
    <td>71.17</td>
    <td>76.03</td>
    <td>60.76</td>
  </tr>
  <tr>
    <td rowspan="2">3-70B</td>
    <td>-</td>
    <td>16</td>
    <td></td>
    <td>80.51</td>
    <td>66.36</td>
    <td>60.41</td>
    <td>86.99</td>
    <td>82.37</td>
    <td>75.33</td>
  </tr>
  <tr>
    <td>AQLM</td>
    <td>2</td>
    <td>1x16</td>
    <td></td>
    <td>78.22</td>
    <td>63.47</td>
    <td>50.34</td>
    <td>78.83</td>
    <td>79.65</td>
    <td>70.10</td>
  </tr>
  <tr>
    <td rowspan="2">EfficientQAT</td>
    <td>2</td>
    <td>128</td>
    <td></td>
    <td>69.46</td>
    <td>60.75</td>
    <td>48.81</td>
    <td>79.25</td>
    <td>79.60</td>
    <td>67.57</td>
  </tr>
  <tr>
    <td>EfficientQAT</td>
    <td>2</td>
    <td>64</td>
    <td></td>
    <td>74.03</td>
    <td>61.60</td>
    <td>49.06</td>
    <td>77.40</td>
    <td>77.37</td>
    <td>67.89</td>
  </tr>
</table>

Reviewer paper summaries: The paper introduces EfficientQAT, a two-stage Quantization-Aware Training (QAT) algorithm for very low bit compression of LLMs. The first stage, Block-AP, involves training all parameters within transformer blocks individually (block-wise), enabling memory-efficient and fast quantization. The second stage, E2E-QP, refines the quantization parameters of the model obtained from Block-AP by training them end-to-end on additional calibration data.

This two-stage approach significantly reduces both computational time and memory requirements. For instance, EfficientQAT can quantize a 70B LLaMA model on a single GPU in less than two days. Specifically, it achieves a WikiText2 perplexity of 4.21 at 3-bit and 6.07 at 2-bit quantization, compared to 2.85 in FP16. Notably, performance degradation on standard benchmarks (Arc, Hellaswag, PiQA, and Winogrande) remains modest, with only about a 6-point drop observed for the largest LLaMA-3 model at 2-bit quantization and 3-point drop for the largest LLaMa 2 model. Additionally, EfficientQAT is shown applicable to instruction-tuned and vision-based models and surpasses quantized-PEFT approaches like QLoRA combined with GPTQ.

This paper proposes Efficient Quantization Aware Training (EfficientQAT), a more feasible QAT algorithm. EfficientQAT involves two  consecutive phases: Block-wise training of all parameters (Block-AP) and end-to-end training of quantization parameters (E2E-QP). Block-AP  enables direct training of all parameters in a block-wise manner, reducing accuracy loss in low-bit scenarios by enhancing
the solution space during optimization. E2E-QP then trains only the quantization parameters (step sizes) end-to-end, further improving
the performance of quantized models by considering interactions among all sub-modules.

This paper presents EfficientQAT, a Quantization-Aware Training(QAT) method for LLMs. By combining block-wise training of all parameters (Block-AP) and end-to-end training of quantization parameters (E2E-QP), the method significantly improves QAT efficiency and accuracy.
Summary of strengths: * Practical 2-bit quantizability: Experiments demonstrate the practical feasibility of EfficientQAT at 2-bit quantization. Moreover, EfficientQAT requires only a single GPU device for model as large as 70B and less than 2 days of compute time.

* Potential for improved returns with compute scaling: Figure 3 and Table 8 demonstrate that increasing the number of calibration samples significantly helps recover performance. This property of QAT contrasts with PTQ methods, which do not have scope for benefiting from scaling. Thus, EfficientQAT has the potential for further performance improvement with additional compute. However, this was only explored as part of the ablation experiments, and relative gains from this scaling relative to other techniques were not covered.

Block-AP successfully demonstrates that complex trainable parameter design is unnecessary for effective block-wise reconstruction in LLMs quantization.   E2E-QP keeps the quantized weights fixed and trains only the quantization parameters (step sizes) end-to-end. 

EfficientQAT characterizes itself as a fast-converging, memory-efficient, and high-performing quantization technique. For instance, EfficientQAT can obtain a 2-bit Llama-2-70B model on a single A100-80GB GPU in just 41 hours, with less than 3 points accuracy degradation on 5 zero-shot common-sense tasks compared to its full-precision counterpart (69.48 vs. 72.41).

1. **Strong Performance**: EfficientQAT achieves superior training speed, obtaining a 2-bit Llama-2-70B model in 41 hours on a single A100-80GB GPU with <3% accuracy drop(69.48 vs. 72.41 FP32 baseline).

2. **Effective for Smaller Models**: The method demonstrates good performance on sub-4-bit smaller models(LLaMA-2-7B and LLaMA-3-8B).
Summary of weaknesses: * Missing key baselines: The Appendix D mentions replicating the 2-bit quantization methods—QUIP# and AQLM, which have shown stronger performance in low-bit settings than OmniQuant and GPTQ. These methods are cited several times in the paper (lines 171, 172, 941, 953, and footnotes 6 & 7). I was unable to find any results of these reported in the paper. These would help analyse the claims about the method’s relative performance to existing methods.

* Fairer comparison of EfficientQAT to PTQ can be done: The PTQ methods (GPTQ, AWQ, OmniQuant) considered in Tables 1, 2, and 3 run at least an order of magnitude faster than EfficientQAT (GPTQ, for example, takes less than 4 hours for a 175B parameter model on a single GPU [4]). Direct comparisons on perplexity & accuracy should also accompany highlighting computational costs and acknowledge this computational difference. Preferably, comparing these with only the Block-AP stage, or applying E2E-QP after these PTQ methods would paint a more accuracy picture of the efficiency of EfficientQAT.

* Several choices for modeling and training could be better motivated: The authors employ the more complex Learned Step Size Quantization (LSQ+) rather than simpler straight-through estimator (STE) [1] used in previous works of ultra-low-bit QAT of LLMs [2,3]. Similarly, the choice of uniform quantization over vector quantization, or initialization from a faster PTQ method like AWQ in the Block-AP stage could be backed by empirical or theoretical arguments.

* Limited novelty of the paper: The authors' claimed novelty regarding intra-block fine-tuning is limited, as similar block-wise approaches have been previously explored, e.g., AQLM and QUIP#. Additionally, these methods also employ different forms of the multi-stage pipeline (where stages differ in granularity at which it is employed) for efficiency. For example, where other methods first quantize a single layer and then tune over transformer blocks. Although EfficientQAT combines intra-block parameter updates with subsequent global quantization-parameter tuning uniquely, the differences and similarities can both be acknowledged to improve the work.

* Relative gains from EfficientQAT diminishes with increase in scale: From Figure 1a, we see that DB-LLM and AutoRound starts to near the performance of EfficientQAT at 70B parameter scale, narrowing the gap from 7B parameter for LLaMa 2. Additionally, degradation is far more several for models trained for more tokens. In Tables 1 and 2, for example, the LLaMa 3 70B model which was pretrained by Meta on 15T tokens, degrades to a performance less than that of LLaMa 2 70B which was trained on only 2T tokens. Similar trend is observed for 7B parameter models. Please consider discussing about this.

[1] [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://arxiv.org/abs/1308.3432)

[2] [BitNet: Scaling 1-bit Transformers for Large Language Models](https://arxiv.org/abs/2310.11453)

[3] [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://arxiv.org/pdf/2402.17764)

[4] [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323)

The novelty of the proposed method may be limited. The block wise training has been widely adopted in many quantization works such as BRECQ (Li et al., 2021) and OmniQuant (Shao et al., 2023). The training follows very basic quantization techniques with equation (1) and (2). The End-to-End Training of Quantization Parameters is very straightforward, which just trains part of the model. The technical contribution may be limited. 

 The comparison may be tricky. Many baselines are post-training quantization methods which use much less data samples. For example, OminiQuant uses 128 randomly selected 2048-token segments from WikiText2. But this paper uses much more data, with 4096 samples from RedPajama with a context length of 2048, and  another 4096 samples from RedPajama but with a context length of 4096.  Typically, using more data in quantization leads to a better performance. It is reasonable that the proposed method can achieve a better performance than baselines with much more data samples. If only using 128 samples, as shown in figure 3, the proposed method does not perform well and it is still a question whether it can outperform baselines. It is better to make a fair comparison with similar or the same number of data samples for quantization. 

For the training hours, apparently, using a larger number of data samples leads to a longer training time. The training hours comparison with different number of training samples do not make much sense. It is better to compare the training time under the same number of training samples. It is better to provide more details.

1. **Limited Technical Novelty**: While the combination of Block-AP and E2E-QP is interesting, the core contributions primarily refine existing QAT workflows rather than introducing substantial technical breakthroughs.


2. **Unfair Comparison #1**: The proposed method only implements weight quantization, while baselines(e.g., OmniQuant, LLM-QAT) employ both activation and weight quantization.

3. **Unfair Comparison #2**: Configuration discrepancies exist. This psper uses 4,096 RedPajama samples(2,048 seqlen for Block-AP, 4,096 for E2E-QP), while baselines like GPTQ(Appendix D) use only 512 calibration samples from Pile-10k. Figure 3 shows sensitivity to sample size(128-4,096), and Table 13 reveals performance variance with Wiki/C4 datasets.


4. **Flawed Comparison with LoRA-based Methods #1**: Full-parameter tuning in this paper (vs. LoRA's parameter efficiency) creates resource asymmetry. 

5. **Flawed Comparison with LoRA-based Methods #2**: Block-wise optimization (vs. LoRA's global optimization) raises questions about Table 3 results. Are improvements come from Block-AP or E2E-QP? What if combining E2E-QP with GPTQ/AWQ/QA-LoRA weights?


6. **Insufficient Ablation of E2E-QP**: The ablation study(Block-AP vs E2E-QP) inadequately isolates their individual contributions like noted in Weakness #5.

7. **Incomplete Baseline Selection**: The exclusion of vector quantization methods (e.g., QuIP#) is unjustified. QuIP#'s Appendix F.7 shows Hessian matrices need only one-time computation per model, contradicting the claimed "significant overhead."
Comments, suggestions and typos: - The paper repeatedly mentions "full-precision weights" (lines 034, 107, 136, 218, 250, 253, etc.) which typically means FP32 but in other places explicitly states "half-precision (FP16)" (lines 203, 309). Is this a typographical error, or is scaling from FP16 to FP32 required for stable implementation in your pipeline?

- For Figures 1a and 1b and their corresponding tables, could the authors add random-guess baselines and FP16 performance as references?

- Could the authors provide individual task performance metrics for EfficientQAT models to able comparison with numbers reported from QUIP# [1] and other techniques? Alternatively, since QUIP# provides competitive 2-bit quantization, would it be possible to directly compare EfficientQAT's models sized at 2.1–2.5 bits (Appendix E) against the corresponding QUIP# models?

[1] QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks: https://arxiv.org/abs/2402.04396

see the weakness.

1. See Weakness.

2. What calibration sample sizes/datasets were used for baseline methods? 

3. What is the full experimental settings for baselines?

4. Table 9's training time comparison appears unreliable due to unspecified data volume configurations across methods.
Overall assessment scores: 4

2

4