TesseraQ: Ultra Low-Bit LLM Post-Training Quantization with Block Reconstruction

## Abstract

Abstract Large language models (LLMs) have revolutionized natural language processing, albeit at the cost of immense memory and computation requirements. Post-training quantization (PTQ) is becoming the de facto method to reduce the memory footprint and improve the inference throughput of LLMs.
In this work, we aim to push the upper limit of LLM PTQ by optimizing the weight rounding parameters with the block reconstruction technique, a predominant method in previous vision models.
We propose TesseraQ, a new state-of-the-art PTQ technique, to quantize the weights of LLMs to ultra-low bits.
To effectively optimize the rounding in LLMs and stabilize the reconstruction process, we introduce progressive adaptive rounding. This approach iteratively transits the soft rounding variables to hard variables during the reconstruction process. Additionally, we optimize the dequantization scale parameters to fully leverage the block reconstruction technique.
We demonstrate that TesseraQ can be seamlessly integrated with existing scaling or clipping-based PTQ algorithms such as AWQ and OmniQuant, significantly enhancing their performance and establishing a new state-of-the-art.
For instance, when compared to AWQ, TesseraQ improves the wikitext2 perplexity from 14.65 to 6.82 and average downstream accuracy from 50.52 to 59.27 with 2-bit weight-only quantization of LLaMA-2-7B.
Across a range of quantization schemes, including W2A16, W3A16, W3A3, and W4A4, TesseraQ consistently exhibits superior performance.

## 1 Introduction

Large Language Models (LLMs) have revolutionized natural language processing with their remarkable capabilities. LLMs such as, GPT-4 (Bubeck et al., ) and LLaMA-3 (Meta, ), contain hundreds of billions of parameters. While this scale enables their impressive performance, it also poses significant deployment challenges (Zhou et al., ). LLMs require substantial memory and computational resources, making them impractical for many real-world applications, especially on consumer devices or in resource-limited environments (Dettmers et al., ).
Quantization addresses this issue by reducing the precision of the model’s parameters and activations, typically from 32-bit floating-point (FP32) to lower bit-width representations such as 8-bit or 4-bit integer (INT8, INT4).
This technique significantly decreases the model’s memory footprint to increase the I/O throughput, often with marginal performance loss.

Post-Training Quantization (PTQ) (Gholami et al., ) has perhaps become the most widespread and the easiest way to compress the LLM by reducing the bitwidth of the pretrained model’s parameters. For example, with a single GPU and a small number of input sequences, GPTQ (Frantar et al., ) can compress an FP16 LLM into INT4 format by deriving the exact solution for quantization error minimization.
Recent works like AWQ (Lin et al., ), QuaRot (Ashkboos et al., ) and OmniQuant (Shao et al., ) have pushed the compression limit further with INT3 weight-only quantization achieving a small performance gap with respect to the FP16 baseline. However, in a more challenging scenario like INT2 weight-only quantization and weight-and-activation quantization, these methods still incur a large performance gap compared to the original FP16 model.

We conjecture that the major reason for the low performance on ultra low-bit PTQ is limited optimization space. Most works only focus on optimizing distribution transformation or weight clipping ranges (Lin et al., ; Wei et al., ; Shao et al., ). While being straightforward, they prove inadequate for extremely low-bit scenarios due to the constrained optimization space. For instance, in per-channel weight quantization, a single clipping range or transformation scale must account for 4k$\sim$20k weight elements in one channel, resulting in suboptimal quantization performance.
We contend that to enhance LLM PTQ performance further, adjustment of the entire weight tensor is necessary. However, it is non-trivial to tune billions of parameters simultaneously.

Figure: Figure 1: The overall workflow of our proposed method. (a) We apply TesseraQ to optimize the weight rounding parameters when the transformation scale and clipping range are determined using prior methods like AWQ/OmniQuant. (b) We propose Progressive Adaptive Rounding (PAR) for block-wise reconstruction, which iteratively hardens some rounding variables and optimizes the rest soft rounding variables till all variables become binary.
Refer to caption: x1.png

To this end, we propose TesseraQ, a block reconstruction method tailored for LLM rounding optimization. We found that rounding optimization on a transformed and clipped LLM (Fig. 1(a)) brings significantly better performance than GPTQ.
To accommodate the billions of parameter spaces in LLMs, our approach removes the dependency of regularization loss in the original rounding optimization processes  (Nagel et al., ; Li et al., ) by introducing Progressive Adaptive Rounding (PAR).
As shown in Fig. 1(b), PAR iteratively hard rounds certain rounding variables to binary values and optimizes the remainder to compensate for the rounding error. Moreover, we propose dequantization scale tuning to further decrease the reconstruction error.
Leveraging block-wise reconstruction, we can efficiently and effectively optimize each LLM block on a single GPU. We have validated TesseraQ across various LLMs and uniform quantization bit-widths, demonstrating superior post-training performance and establishing new state-of-the-art quantized LLMs.
We summarize our contributions as follows

- 1.
We propose TesseraQ, a block reconstruction-based weight rounding optimization method for LLMs. TesseraQ can be combined with existing transformation or clipping methods like AWQ, OmniQuant, and QuaRot to obtain state-of-the-art results.
- 2.
TesseraQ contains Progressive Adaptive Rounding and Dequantization Scale Tuning. Both methods can stabilize the reconstruction process and effectively optimize post-training performance.
- 3.
Our method obtains state-of-the-art performance on both perplexity metric and zero-shot accuracy metric. For example, our method improves OmniQuant perplexity results from 37.4 to 8.0 on LLaMA-2-7B W2A16 quantization. Moreover, TesseraQ+QuaRot improves the average accuracy by 10% on LLaMA-3.1-8B W3A3 quantization as compared to GPTQ+QuaRot.

## 2 Preliminaries

This section briefly introduces the existing research directions in LLM PTQ. We adopt uniform affine quantization, which essentially discretizes the floating-point representation of weights/activations into low-bit fixed-point representation, given by

$$ ${\mathbf{W}}^{q}=\mathrm{clamp}\biggl{(}\Bigl{\lfloor}\frac{{\mathbf{W}}}{s} \Bigr{\rceil}+z,0,2^{N}-1\biggr{)},\ \ \text{where }s=\frac{\gamma\max({ \mathbf{W}})-\beta\min({\mathbf{W}})}{2^{N}-1},\ z=-\Bigl{\lfloor}\frac{\beta \min({\mathbf{W}})}{s}\Bigr{\rceil}.$ (1) $$

where $s$ and $z$ denote the quantization step size and the zero point. The resulting ${\mathbf{W}}^{q}$ is in the INT-$N$ format. To restore it back to its original range, the dequantization step is given by $\hat{{\mathbf{W}}}=s\times({\mathbf{W}}^{q}-z)$.

Optimization Objective.
The plain rounding-to-nearest (RTN) method directly quantifies the model weights to integers without further optimization. However, this method usually results in significantly low task performance. To improve the LLM PTQ performance, parameters related to quantization are optimized with different objectives. For example, GPTQ (Frantar et al., ) and AWQ (Lin et al., ) utilize the layer-wise reconstruction objective, given by

$$ $\min_{\epsilon}({\bm{\mathsfit{L}}}(\theta+\epsilon)-{\bm{\mathsfit{L}}}( \theta))\approx\sum\nolimits_{\ell=1}^{L}\bigl{|}\bigl{|}\hat{{\mathbf{W}}}^{( \ell)}{\mathbf{X}}^{(\ell)}-{\mathbf{W}}^{(\ell)}{\mathbf{X}}^{(\ell)}\bigr{|} \bigr{|}_{F}^{2},$ (2) $$

where ${\bm{\mathsfit{L}}}$ is the loss function parameterized by weights in the whole model $\theta$ and quantization noise $\epsilon=\hat{\theta}-\theta$. $\ell\in\{1,2,\dots,L\}$ is the layer index and ${\mathbf{X}}$ is the input activations.
While this layer-wise objective can provide efficient and exact solutions as in GPTQ, the objective does not consider inter-layer correlation like self-attention and residual connections in LLM. To this end, the block-wise reconstruction objective has been proposed (Li et al., ), as

$$ $\min_{\epsilon}({\bm{\mathsfit{L}}}(\theta+\epsilon)-{\bm{\mathsfit{L}}}( \theta))\approx\sum\nolimits_{b=1}^{B}\bigl{|}\bigl{|}\mathrm{block}(\hat{ \theta}^{(b)},{\mathbf{X}}^{(b)})-\mathrm{block}(\theta^{(b)},{\mathbf{X}}^{(b )})\bigr{|}\bigr{|}_{F}^{2}.$ (3) $$

where, $\mathrm{block}$ refers to one decoder block in LLMs comprising self-attention, projection, feed-forward and normalization layers. In practice, both layer-wise and block-wise objectives enable efficient calibration on a single GPU due to their local computation attributes. However, block-wise objectives exhibit better performance than layer-wise objectives as they better approximate the global loss (i.e., Eq. (missing) 3 left side) by accounting for contributions from multiple layers.

Optimization Space.
Generally, three kinds of optimization spaces are explored in LLM PTQ, (1) the scale transformation, (2) the clipping range (i.e., finding the suitable $\gamma,\beta$), and (3) the weight values. They can be tied with either layer-wise or block-wise objectives. For instance, AWQ (Lin et al., ) and OS+ (Wei et al., ) optimize transformation and clipping range using Eq. (missing) 2, while OmniQuant (Shao et al., ) does similar optimization with Eq. (missing) 3.
Since scale/clipping optimization methods are well-explored, in this paper, we aim to optimize weight values using block-wise objectives to further push the compression limits of LLM PTQ.

## 3 TesseraQ: Ultra Low-Bit Post-Training Quantization

Figure: Figure 2: Perplexity comparison of TesseraQ with other PTQ methods on LLaMA-2-7B model quantized to different weight precision (INT2, INT3). g denotes the group size.
Refer to caption: x2.png

### 3.1 Problem Statement

Element-wise weight adjustments were also studied in GPTQ (Frantar et al., ), in which the weights are computed using closed-form solutions using the inverse Hessian matrix.
However, we found that this technique cannot be used to improve other transformation-based PTQ methods. As shown in Fig. 2 on the right side, applying GPTQ on an AWQ checkpoint barely improves the perplexity metric over AWQ across various bitwidth configurations on the LLaMA-2-7B model (Touvron et al., )
Gong et al. () also report similar observations with GPTQ.
We hypothesize that the reason for the failed improvement of GPTQ+AWQ could be the inaccurate layer-wise reconstruction objective and its sub-optimal solution.
Therefore, we use weight rounding optimization (Nagel et al., ; Li et al., ), which is a different optimization space compared to GPTQ, given by

$$ $\begin{split}\min_{\alpha}\ &\bigl{|}\bigl{|}\mathrm{block}(\hat{\theta},{ \mathbf{X}})-\mathrm{block}(\theta,{\mathbf{X}})\bigr{|}\bigr{|}_{F}^{2},\\ \text{s.t. }&\hat{\theta}={\mathbf{s}}\times(\theta^{q}-{\mathbf{z}}),\ \ \theta^{q}=\mathrm{clamp}\Bigl{(}\lfloor\frac{\theta}{{\mathbf{s}}}\rfloor+ \alpha+{\mathbf{z}},0,2^{N}-1\Bigr{)},\ \ \alpha\in\{0,1\}^{d}.\end{split}$ (4) $$

Here, $\theta$ denotes the total $d$ weight parameters of linear layers in the block, $\alpha$ is the rounding variable. Note that we omit the block index for simplicity. This adaptive rounding optimization space restricts the range of each weight parameter.
The restricted nature of rounding optimization will lead to more stable training and better convergence, especially when dealing with billions of parameters.
Nonetheless, directly optimizing the rounding variable $\alpha$ is non-trivial since it is not continuous.

### 3.2 Progressive Adaptive Rounding

To optimize $\alpha$, we introduce a differentiable rounding optimization framework called progressive adaptive rounding (PAR) that does not rely on regularization loss or the straight-through estimator in contrast to previous works (Nagel et al., ; Hubara et al., ).
To start with, we relax the rounding variable into a continuous variable by using the Sigmoid reparameterization $\alpha=\sigma(\nu)$.
Therefore, $\nu$ can be initialized as $\nu=\sigma^{-1}(\theta/s-\lfloor\theta/s\rfloor)$, resulting in $\hat{\theta}=\theta$.
The PAR algorithm divides all rounding variables into two sets: $\mathcal{S}_{\text{Hard}}$ and $\mathcal{S}_{\text{Soft}}$, standing for the *hard* and *soft* rounding of the variable $\nu$. Formally, we define the rounding function as

$$ $\alpha_{i}=\begin{cases}\sigma(\nu_{i})=\frac{1}{1+\mathrm{exp}(-\nu)}&\text{ if }i\in\mathcal{S}_{\text{Soft}}\\ \sigma^{\prime}(\nu_{i})=\mathbf{1}_{\nu_{i}>0}&\text{if }i\in\mathcal{S}_{ \text{Hard}}\end{cases}.$ (5) $$

The $\sigma^{\prime}(\nu_{i})$ is a hard rounding function that returns 1 if $\nu_{i}$ is larger than 0, otherwise it returns 0. Starting from an empty hard rounding set, we iteratively put variables from $\mathcal{S}_{\text{Soft}}$ into
$\mathcal{S}_{\text{Hard}}$ (called *Harden Phase*), and optimize the remaining soft rounding variables to compensate for the hard rounding loss (called *Soften Phase*).
We elaborate on them in the following two subsections.

Harden Phase.
Intuitively, after setting rounding variables to hard ones, we would expect minimum loss change in the block output error. Therefore, we define a score metric

$$ $HS(\nu)=|\sigma(\nu)-0.5|.$ (6) $$

Essentially, the lower the score, the closer the soft rounding variable ($\sigma(\nu)$) is to 0.5, implying that rounding these variables to binary values will result in a larger increase in reconstruction loss.
As a result, in the Harden Phase, we sort the parameter indices based on their $HS$ and select the lowest $P$% of them to $\mathcal{S}_{\text{Hard}}$.
The hyper-parameter $P$ should increase from 0 to near 100 during block reconstruction. During the early stage of the reconstruction, $P$ can be increased rapidly, however, in the later stage, we slowly increase $P$ since the learnable soft variables are becoming fewer in each iteration. In our experiments, we find that TesseraQ is not sensitive to any specific decay schedule for $P$, as long as we progressively slow down the increasing rate of $P$.
We conduct an ablation study of how to schedule the change of $P$ in Sec. 4.3.

Figure: Algorithm 1 TesseraQ Calibration process

Soften Phase.
For this stage, we employ the gradient-descent optimization to optimize the soft rounding variable

$$ $\min_{\nu_{i,i\in\mathcal{S}_{\text{Soft}}}}\ \bigl{|}\bigl{|}\mathrm{block}( \hat{\theta},{\mathbf{X}})-\mathrm{block}(\theta,{\mathbf{X}})\bigr{|}\bigr{|} _{F}^{2}.$ (7) $$

This objective can be optimized via gradient-based training like Adam (Kingma, ).
During implementation, it would be too expensive to use masking to indicate soft rounding or hard rounding. Instead, for memory-efficient implementation, we can safely set the hard-rounding variables to $\infty$ or $-\infty$, which returns zero gradients in the sigmoid function.
We find that optimizing Eq. (missing) 7 with nearly 200 steps can sufficiently decrease the block reconstruction error across different LLM models.

Post-Processing.
After the entire PAR procedure is finished, we apply hard-rounding $\sigma^{\prime}(\cdot)$ to all variables merge their values into the original weights, and then we can use the standard quantization formula (i.e., Eq. (missing) 1).
The merging can be effectively implemented by

$$ $\theta\leftarrow\theta+{\mathbf{s}}\times(\sigma^{\prime}(\nu)-0.5)$ (8) $$

We provide a pseudocode for the learning process in Algorithm 1.

### 3.3 Dequantization Scale Tuning

During the PAR process, the quantized tensor $\theta^{q}$ undergoes continuous changes. To accommodate these dynamic adjustments, we propose a method that optimizes the dequantization scale concurrently with the rounding variable.
Specifically, for the dequantization step, we introduce an additional parameter ${\mathbf{v}}$ and represent it as

$$ $\hat{\theta}=2\sigma({\mathbf{v}})\times{\mathbf{s}}\times(\theta^{q}-{\mathbf {z}}).$ (9) $$

By initializing ${\mathbf{v}}=\mathbf{0}$, we initialize the dequantization scale factor ($2\sigma({\mathbf{v}})$) to 1 and subsequently adjust it to a value within the range $(0,2)$.
Note that we avoid optimizing the scale $s$ in the quantization step (Eq. (missing) 1) since, (1) any change in $s$ would result in a change of the rounding mechanism (Nagel et al., ), (2) the optimization requires straight-through estimation (Shao et al., ) which leads to biased gradient calculation.
Experiments in Sec. 4.3 demonstrate that dequantization scale tuning can benefit the final quantization performance of TesseraQ by a large margin.

## 4 Experiments

### 4.1 Experiements Setup

Most of our experiment setups are similar to OmniQuant (Shao et al., ), which also adopts block reconstruction loss function. Specifically, we employ asymmetric uniform quantization with 2/3/4-bit integers. We test both per-group and per-channel weight quantization. For example, we use the notation *W2A16g64* to denote the 2-bit per-group (group size is set to 64) weight-only quantization, where activations are FP16.
In weight-activation quantization experiments (all INT precision), defaults are W4A4, W3A3, and W4A8 with per-channel weight and per-token activation quantization  (Dettmers et al., ; Shao et al., ).

Calibration Data and Comparison.
We report two types of evaluation metrics, the perplexity metric for evaluating the upstream datasets like WikiText2 (Merity et al., ), C4 (Raffel et al., ), and the average accuracy of 5 downstream reasoning tasks including PIQA (Bisk et al., ), ARC easy/challenge (Clark et al., ), WinoGrande (Sakaguchi et al., ) and HellaSwag (Zellers et al., ). The perplexity is evaluated with 2048 sequences.
We use 512 2048-token segments from the WikiText2 training dataset as calibration data for perplexity comparison and for downstream task comparison, we sample same amount of calibration data from the C4 training dataset.
We use lm_eval (ver0.4.2) to evaluate the accuracy.

Training.
We set the total PAR number of iterations $K$ to 20 and gradually increase the $P_{k}$ from 0 to 100%.
In each iteration, we optimize the learnable parameters ($\nu$ and ${\mathbf{v}}$) for 250 training steps. We use the Adam optimizer with a fixed learning rate of $1e-3$. We add $1e-4$ weight decay to ${\mathbf{v}}$ during training. The batch size is set to 4.
We use AWQ transformation (Lin et al., ) to initialize our model since we find AWQ initialization is slightly better than OmniQuant across all configurations except W2A16 quantization. For W2A16, AWQ yields very high perplexity. Thus, in the W2A16 case, we directly use the pretrained OmniQuant model for initialization.

Models and Baselines. For the upstream tasks, we follow OmniQuant (Shao et al., ) to test weight-only quantization results on LLaMA-1-7B/13B/30B/65B (Touvron et al., ), LLaMa-2-7B/13B/70B (Touvron et al., ).
In this case, we compare GPTQ (Frantar et al., ), OmniQuant (Shao et al., ), AWQ (Lin et al., ) with asymmetric clipping implementation (Gong et al., ).
For downstream tasks, we test LLaMA-2-7B, LLaMA-3.1-8B/70B across 5 downstream tasks.
We compare GPTQ, AWQ, OmniQuant, and a recent rounding optimization method SignRound (Cheng et al., ).

Table: Table 1: Weight-only quantization results of LLaMA-1 and LLaMA-2 models. We report WikiText2 perplexity (PPL $\downarrow$). *, ${\dagger}$ means initialized from AWQ, and OmniQuant, respectively.

### 4.2 Main Results

Perplexity Evaluation.
We summarized the Wikitext2 perplexity (PPL) results in Table 1. Our method consistently outperforms existing methods like AWQ and OmniQuant, particularly for the low-bit W2A16 configuration. Remarkably, in the W2A16 case, all existing methods except OmniQuant failed to successfully quantize the models (yielding $>1e3$ perplexity). On the LLaMA-2-7B model, OmniQuant only obtains 37.37 PPL while our method largely improves this result to 8.05.
We observe that in general, the lower the bitwidth, the more improvement we can obtain from TesseraQ . This confirms our initial intuition that extremely low-bit weight quantization requires a thorough adjustment of each weight element. Additionally, the C4 (Raffel et al., ) PPL results are provided in Appendix: Table 9. Note that the C4 results for OmniQuant are re-evaluated from the official checkpoint to align the evaluation protocol. Overall, C4 PPL results concur with the Wikitext2 results, demonstrating a similar trend in performance improvement. For example, TesseraQ improves the PPL of LLaMA-2-7B model from 90.64 to 14.82 with W2A16 quantization.

Table: Table 2: Weight-only quantization Results of various LLMs. We report the accuracy of 5 common sense reasoning tasks $(\uparrow)$. * means initialized from AWQ.

Downstream Tasks Evaluation.
We also test the weight-only quantization performance on five reasoning tasks. The results are summarized in Table 2, for LLaMA-2-7B, LLaMA-3.1-8B/70B(^1^11We did not implement OmniQaunt on LLaMA-3.1 models due to its high resource & time demands.). Notably, we found that the LLaMA-3.1-8B model demonstrates low quantization resiliency, as also shown in Huang et al. ().
For example, with W2A16g128 AWQ, this model drops more than 30% average accuracy on downstream tasks, while the gap is 15% for the LLaMA-2-7B model. Fortunately, our TesseraQ can substantially increase the average performance on the downstream tasks, bringing the gap between W2 and FP16 to only 9%.
TesseraQ also outperforms a recent rounding optimization method, SignRound (Cheng et al., ), for W2A16g128, demonstrating the effectiveness of our method.

Weight-Activation Quantization Evaluation.
Finally, we test weight-activation quantization scenarios with per-channel weight quantization and per-token activation quantization.
With quantized activations, the inference speed of LLMs on GPUs/TPUs can be improved especially in the prefill stage (Lin et al., ).
We experiment with W4A4, and W4A8 quantization and compare with three baselines, SmoothQuant, OS+, and AWQ.
The results are provided in Table 3 (W4A8 results are provided in Appendix: Table 10).
Table 3 summarizes the perplexity on WikiTex2, C4 and average accuracy on downstream tasks. (The detailed accuracy of each downstream task is located in Appendix: Table 12.)
We observe a consistent improvement of 12% accuracy with TesseraQ compared to AWQ.
Additionally, we also combine our method with a recent rotation-based quantization method, QuaRot (Ashkboos et al., ), and compare QuaRot+GPTQ and QuaRot+TesseraQ with W4A4 and W3A3 quantization.
Combined with QuaRot, TesseraQ also exceeds GPTQ by 10% accuracy on 8B/70B models with W3A3 quantization, demonstrating the superiority of TesseraQ.

Table: Table 3: W4A4/W3A3 quantization results of LLaMA-3.1. We use per-channel weight quantization and per-token activation quantization *, ^† means initialized from AWQ, QuaRot.

Results on Mistral-7B.
Additionally, we also test the performance of our method on the Mistral-7B model Jiang et al. (), which achieves high pretrained accuracy and demonstrates higher quantization resiliency.
We test its weight-only quantization (W2A16g128, W3A16g128) and weight-activation quantization (W4A4, W4A8) performance in the Appendix (Table 11). Our TesseraQ consistently outperforms other methods like SignRound, AWQ, and GPTQ.

Results on Smaller-Size LLM for Edge Inference.
In addition to LLMs that are deployed on GPUs, we also test the performance of smaller-size LLMs geared for edge devices. We test LLaMA-3.2-1/3B models and compare them with AWQ in Table 4. We observe that our method significantly outperforms AWQ across different bitwidths in WikiText2 perplexity and average downstream task performance.

Table: Table 4: Quantization Results of LLaMA-3.2 for edge inference.

### 4.3 Ablation Studies

Below ablation studies are conducted with the LLaMA-2-7B model with W2A16g128 quantization.

Calibration Data.
In this section, we compare the performance of different calibration datasets and sizes. We sample calibration data from either WikiText2 (Merity et al., ) or C4 (Raffel et al., ) training dataset.
We also experiment with the different sample sizes, ranging from 128 to 512. Meanwhile, we change the batch size during rounding optimization, ranging from 1 to 4.

Table 5 demonstrates the task performance (PPL and average accuracy metric) as well as the calibration costs (algorithm runtime and GPU memory footprint).
First, we find that the source of calibration data will impact the perplexity evaluation. The performance benefits if evaluation data and calibration data are from the same dataset. For example, the C4-calibrated model has 1.2 higher WikiText2 PPL than the WikiText2-calibrated model.
Second, increasing the number of samples and the batch size consistently improves the task performance. However, it may also lead to higher runtime and GPU memory consumption, which may be alleviated via multi-GPU calibration. Nevertheless, it is worthwhile to note that even with 128 samples and a batch size of 1, our TesseraQ can significantly improve the baseline AWQ results.

Table: Table 5: Ablation studies of calibration data source and data sizes. We report the LLaMA-2-7B W2A16g128 quantization results with task performances and calibration costs.

Algorithm choices.
We also test the algorithm choices in TesseraQ . To be more specific, we experiment with block reconstruction with or without progressive adaptive rounding (PAR) and dequantization scale tuning (DST) and compare their final task performance.
As shown in Table 6, both PAR and DST contribute a lot to the final perplexity metric (denoted

Table: Table 6: TesseraQ Algorithm choices.

by WT2 (WikiText2) and C4) and average accuracy (denoted by Avg.). Remarkably, applying one of them solely can also improve the AWQ baseline (first row) results by a large margin.

Figure: Figure 3: Ablation study of PAR schedule. We experiment several rule-based $P$ adjustments and one handcrafted adjustment. *(AWQ baseline results: average PPL: 16.66, average acc.: 50.52).*
Refer to caption: x3.png

PAR Schedule.
We investigate how to adjust the $P$ during progressive adaptive rounding. In our implementation, we use a handcrafted design, which manually decreases the soft rate (i.e., the percentage of soft rounding variable) as shown in Fig. 3. Our handcrafted design gradually decays the soft rate. To demonstrate that our PAR is quite robust to the schedule of soft rate, we also test several rule-based adjustments, which adjust the soft rate as $\frac{1}{\mathrm{exp}(tx)}$, where $x\in(0,1]$ is the scaled iteration number and $t$ is the temperature hyper-parameter. We test $t=\{2,3,4,5\}$ and compare it with our handcrafted implementation with LLaMA-2-7B W2A16g128 quantization. The results in Fig. 3 show that $t=4,5$ and our handcrafted adjustments obtain the best performance. Overall, we find that our algorithm is not sensitive to the scheduling, and has consistently superior performance than the AWQ initialized model.

### 4.4 Visualization

In this section, we provide visualizations of our calibration process to interpret the effectiveness of our method. The experiments are conducted on LLaMA-2-7B with W2A16g128 quantization.
We first compare the loss convergence value in OmniQuant and TesseraQ, both of which calibrate the model with block reconstruction loss.
To ensure a fair comparison, we use the same AWQ initialization to these two methods and align all training hyper-parameters.
As shown in Fig. 4, during the first block reconstruction, TesseraQ reduces more loss than OmniQuant. In the following blocks, the loss gap between our method and Omniquant keeps on increasing.
Consequently, TesseraQ will have a much lower model output error due to the cumulative effect of reconstruction.

Figure: Figure 4: Reconstruction loss convergence. We compare the block reconstruction loss of OmniQuant and TesseraQ during optimization. Our method significantly reduces the loss in each block.
Refer to caption: x4.png

Since rounding variables ($\alpha$) are binary, we also demonstrate the number or percentage of rounding variables that flip after TesseraQ. In Table 7, we show the number and the percentage of flipped variables. Overall we observe around 3%$\sim$8% of variables flip, amounting to over 10M parameters per block. This is a significantly larger value than AWQ/OmniQuant, which justifies why TesseraQ further improves the performance.
We also found that attention layers tend to have less flipped rounding compared to MLP layers. 2/3-bit quantization also flips more than 4-bit quantization.

### 4.5 Hardware Evaluation

To demonstrate the weight compression effect and the inference throughput change, we test LLaMA-3.1-8B/70B/405B under different GPU environments, kernel backend and different bitwidths. Table 8 summarizes the results of inference throughput (generated token per second) with batch size 1 or 16. Remarkably, W2A16g128 reduces the weight memory of the 405B model from 756GB to 114 GB and the 70B model from 132 GB to 21 GB. However, the INT2 dequantization kernel (in Triton () support) is currently less optimized, especially for larger models, expending lower throughput compared to FP16. We find that INT4 with Exllama kernel can increase the throughput when batch size is 1 and achieve similar throughput with FP16 model when batch size is 16. Nonetheless, it is worthwhile to note that our TesseraQ complies with standard uniform quantization formats and can be deployed with various kernels that support uniform quantization on various devices, e.g., GPU, CPU, TPU, edge processor.

Table: Table 7: Number (percentages) of rounding variables that flip after TesseraQ. We average the numbers across all blocks on LLaMA-2-7B.

Table: Table 8: Comparison of weight memory compression and inference throughput. We measure LLaMA-3.1 series model under various bitwidth/backend. WM stands for weight memory, $TP_{n}$ denotes inference throughput with a batch size of $n$ (output token/s).

## 5 Related Work

Quantization has been a primary method to compress and accelerate off-the-shelf large models.
Survey papers by Gholami et al. () and Nagel et al. () have systematically summarized the progress of quantization.
Here, we list several major quantization works, especially for LLMs.

Post-Training Quantization for LLMs.
While Quantization aware Training (QAT) guarantees better task performance in low-bit quantization, PTQ is more suitable for LLM due to its less reliance on computing resources and training data.
PTQ methods like Frantar et al. (); Lin et al. (); Wei et al. (; ); Shao et al. (); Chee et al. (); Liu et al. () improve the uniform quantization performance by optimizing weights, transformation scales, and clipping ranges. Our method continues improving the uniform quantization effect by incorporating rounding optimization.
Other works try to improve PTQ in LLMs in different ways. For example, AQLM and GPTVQ (Egiazarian et al., ; van Baalen et al., ) explore non-uniform quantization schemes for weight-only quantization, which may better match the distribution of weights.
LLM.int8 (Dettmers et al., ), BiLLM (Huang et al., ), SiLLM (Huang et al., ) apply mixed-precision quantization to keep salient weights in high precision and maintain the accuracy. However, these methods cannot be applied to quantize activations and thus cannot support integer MatMul.
QuaRot (Ashkboos et al., ), SpinQuant (Liu et al., ) target activation outliers and eliminate them through the rotation matrix. We have demonstrated that our method can also be combined with them.

QAT for LLM.
Recent works also explore QAT-based quantization for LLMs. To reduce data access, LLM-QAT (Liu et al., ) generates language data for data-free QAT.
To prevent massive weight memory usage, Q-LoRA (Dettmers et al., ) applies quantization-aware low-rank adaptation for finetuning.
Recently, BitNet and BitNet b.158 (Wang et al., ; Ma et al., ) trained a 1-bit and 1.58-bit model from scratch, enabling multiplication-free LLM. However, these methods are hard to scale up due to the massive memory and computation requirements, especially for more than 70B models. As a result, they only focus on 1B$\sim$3B-scale models.

## 6 Conclusion

In this paper, we have proposed TesseraQ, a PTQ method for effectively calibrating large language models. Based on block reconstruction, TesseraQ optimizes weight rounding through a progressive approach that iteratively hardens and softens the rounding variables.
Together with dequantization scale tuning, TesseraQ can be seamlessly combined with other PTQ methods like transformation, clipping, and rotation, to reach new state-of-the-art performance. We demonstarte TesseraQ’s superiority on open source LLaMA models. TesseraQ establishes a new state-of-the-art for quantized LLMs, in terms of perplexity, downstream accuracy and hardware performance.

#### Acknowledgments

This work was supported in part by CoCoSys, a JUMP2.0 center sponsored by DARPA and SRC, the National Science Foundation (CAREER Award, Grant #2312366, Grant #2318152), the DARPA Young Faculty Award and the DoE MMICC center SEA-CROGS (Award #DE-SC0023198). We would also like to thank Amir Yazdanbaksh, Joao Carreira and James Laudon for their helpful feedback and insightful suggestions towards this work.

## Appendix A More Experimental Results

In this section, we include additional experimental results from the main section.

### A.1 Results on C4

We demonstrate the perplexity results on the C4 datasets in Table 9. Note that the OmniQuant results are re-evaluated using the official checkpoint, which is slightly higher than the original paper results (Shao et al., ).
Since the evaluation protocol can be different across different papers,
we ensure use of the same evaluation protocol to compare different methods.
Note, we restrict all models here from using the WikiText2 calibration data as the calibration data will affect the perplexity metric as shown in our ablation study.
The improvements of our method over existing approaches are consistent with the results on the WikiText2 dataset.

Table: Table 9: Weight-only quantization results of LLaMA-1 and LLaMA-2 Models. We report C4 perplexity in this table. *, ${\dagger}$ means initialized from AWQ, and OmniQuant, respectively.

### A.2 W4A8 Quantization

We also provide the W4A8 quantization in Table 10. Overall we find a small difference in W4A8 quantization due to the 8-bit per-token activation quantization.

Table: Table 10: Weight-activation quantization Results of various LLMs. We report the accuracy of 5 reasoning tasks $(\uparrow)$.

Table: Table 11: Weight-activation quantization Results of Mistral-7B. We report the accuracy of 5 reasoning tasks $(\uparrow)$.

### A.3 Detailed Accuracy of W4A4/W3A3 Quantization

Table 12 provides the detailed accuracy of each zero-shot tasks in W4A4/W3A3 quantization.

Table: Table 12: W4A4/W3A3 quantization results of LLaMA-3.1. We use per-channel weight quantization and per-token activation quantization *, ^† means initialized from AWQ, QuaRot.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Why the author didn't compare with Adaround and signRound in Table1 and other experiments?
2. Compared with the traditional Layer-wise Rounding method Adaround, what is the superiority of the proposed TesserQ, is the block-wise reconstruction granulity or the Harden and Soften design? Why? The author should add more related discussion and results to analysis it.
soundness: 3
strengths: 1. The idea of optimizing the weight rounding error in LLM is interesting, while most previous methods focus on the weight clipping or distribution transmation.
2. The author proposes Progressive Adaptive Rounding strategy by deviding all rounding variables into harden and soften.
3. The performance in W2A16 setting is good and TesseraQ is orthogonal to other post-training quantization (PTQ) methods for LLM.
summary: This paper presents a PTQ method，aTesseraQ, for LLM.  The author propose optimize weight rounding through a progressive
approach based on block reconstruction. Besides, the author also propose dequantization scale tuning to mitigate the reconstruction error.
The experiments on LLaMa models shows good performance, and this training-based PTQ method could be implemented with other PTQ methods.
weaknesses: 1. unclear writing: The writing in Section 3.2 of the paper is not sufficiently clear, making it somewhat difficult to read and understand. Moreover, this section constitutes the core content of the methodology and pertains to the detailed description of the method itself.
2. Outdated Comparison: The comparison methods in Table 1 appear to be somewhat outdated. The authors should compare with more advanced weight-only methods to demonstrate its superiority. 
3. Besides, other tables comparison seems inadequate. The authos should also report the comparison results with other Rounding method, such as Adaround and signRound.
4. The author should discuss the method's limitation.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. The choice to restrict rounding to ±1 steps seems limiting. Why not allow multiple step sizes as in AdaQuant? This decision could impact optimization flexibility, especially for ultra-low bit quantization where precision is crucial. Has this design choice been empirically validated against alternatives?

2. The effectiveness with weight transformation techniques is unclear. Specifically, would TesseraQ maintain its benefits when applied after Hadamard matrix transformations (as in QuaRot)?
soundness: 3
strengths: 1. The paper presents a clear motivation for ultra-low bit LLM quantization, with a well-justified technical solution. The authors effectively demonstrate how current methods struggle with low-bit quantization and explain why optimizing the entire weight tensor is necessary. The progressive adaptive rounding approach is sound to handle the problem.

2. The writing quality is exceptionally clear and professional. Complex technical concepts are explained systematically, from basic quantization principles to advanced optimization techniques. The figures and algorithms are informative and well-designed, particularly Figure 1 which clearly illustrates the workflow.

3. The experimental results show substantial improvements over strong baselines. For example, improving WikiText2 perplexity from 37.37 to 8.05 on LLaMA-2-7B with W2A16 quantization is a significant achievement. The ablation studies effectively isolate the contributions of PAR and DST.

4. The practical utility is demonstrated through real-world evaluations. The authors test different hardware configurations, measure actual throughput, and analyze memory consumption.
summary: This paper introduces TesseraQ, a post-training quantization method for LLMs that focuses on optimizing weight rounding parameters using block reconstruction techniques. The key design is a Progressive Adaptive Rounding approach that iteratively converts soft rounding variables to hard variables, along with dequantization scale tuning.
﻿
The method is designed to work with existing quantization approaches like AWQ and OmniQuant, significantly improving their performance particularly for ultra-low bit scenarios (2-4 bits). Results show major improvements in both perplexity metrics and downstream task accuracy across different LLaMA models and quantization schemes.
﻿
The main contribution is enabling better ultra-low bit quantization of LLMs through tuning of quantization rounding while maintaining model performance, demonstrated through comprehensive empirical results.
weaknesses: 1. The paper lacks detailed technical comparisons with similar methods. While mentioning prior approaches like AdaRound and AdaQuant,  it doesn't experimentally demonstrate why the proposed methods are superior to the these methods. The rationale behind several design choices (sigmoid reparameterization; harden score metric in Eq 6; avoiding scale optimization in the quantization step) isn't fully justified through comparative experiments. 

2. The evaluation demonstration has problems. 
- The inconsistency in model selection between Tables 1, 2 and 3 makes it difficult to draw comprehensive conclusions. 
- Runtime and GPU memory analysis is limited to the 7B model.  As the runtime and memory consumption are important metircs, the authour should provide comprehensive efficiency metrics across different model sizes. 
- The paper would be stronger with complete results for newer models like LLaMA-3 across all experiments.

3. The technical novelty is relatively incremental. Both block reconstruction and scale tuning are well-established in PTQ literature. While the paper makes useful improvements, it builds primarily on existing techniques rather than introducing fundamentally new concepts.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: The paper is generally clear and well-written, the algorithm has been fully presented. While according to my perspective in the weakness section, I have the following question:

1. Why does the paper adopt Progressive Adaptive Rounding (PAR) instead of the more common Straight-Through Estimator (STE) for rounding policy estimation? What specific limitations or performance issues with STE justify the need for PAR in the overall design? If STE is applied, whether such progressive-stage training is still required?

2. Could the authors justify the theoretical basis for the improvement achieved by rounding optimization over the baseline?
soundness: 3
strengths: The paper offers an interesting and effective add-on for block-wise reconstruction for LLM quantization with optimized rounding policy. Especially for Ultra-low bit scenarios, such combination advances the such field where prior quantization methods struggle to maintain performance for various tasks.

The methodology is well-designed, with clear theoretical motivation for each component, including PAR and dequantization scale tuning. The experimental results are robust, demonstrating TesseraQ’s superiority across multiple benchmarks such as WikiText2 and downstream reasoning tasks. The paper carefully evaluates TesseraQ’s effectiveness by comparing it to several existing methods, providing quantitative evidence that supports the proposed contributions.

In terms of the overall writing quality, the paper is generally well-organized and explains technical concepts like rounding optimization and block reconstruction in a clear and structured manner.  

The significance of the paper lies in its potential impact on LLM deployment, particularly for memory-constrained environments such as consumer devices and edge processors. TesseraQ pushes the boundaries of ultra-low-bit quantization, a growing need as LLMs become larger and more ubiquitous.
summary: The paper presents a novel method to compress large language models (LLMs) into ultra-low-bit formats while maintaining high performance. It introduces Progressive Adaptive Rounding (PAR), which gradually decides rounding schedules for groups of parameters within LLMs while retaining the training flexibility via Sigmoid-based soft-hard transition, improving quantization stability. By integrating with block-wise reconstruction instead of layer-wise optimization, TesseraQ captures inter-layer dependencies more effectively, and de-quantization scale tuning further reduces reconstruction errors. The method integrates seamlessly with existing quantization approaches like AWQ and OmniQuant, significantly improving perplexity and accuracy across benchmarks. TesseraQ achieves state-of-the-art results, offering a practical solution for deploying LLMs efficiently in memory-constrained environments.
weaknesses: 1. The paper proposes PAR instead of the common STE implementation. There is a lack of justification why direct utilization of STE for the rounding policy estimation would result in worse performance so that PAR is significant in the overall design. 

2. The paper purposes an effective rounding optimization add-on for enhancing the block-wise quantization for LLM, but there is a lack of theoretical analysis of the different between: A) BLOCK-WISE & Normal Rounding (round to the nearest) and  B) the proposed BLOCK-WISE & Rounding Optimization. If A) is well trained and fully converged, would it theoretically have the similar mathematical foundations as B)?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The new rounding error optimization method proposed in the paper mentions that compared to previous methods ( does not rely on regularization loss , but HS and sorting were introduced. Does it have advantages in training convergence speed and video memory usage?
2. The paper combines the proposed rounding error optimization method with other methods for experimentation and comparison with other quantization methods, but does not use other rounding error optimization methods for the same experiment, Can relevant experiments be supplemented ?
3.When comparing quantitative indicators such as W4A4/W3A3 that include activation, the QuaRot initial method was used, but it was not compared with using the QuaRot method,Can relevant experiments be supplemented?
soundness: 2
strengths: The paper proposes a new rounding optimization design called TesseraQ， it can be combined with other PTQ methods to achieve new performance levels, establishing a new technological level for quantifying LLM in terms of confusion, downstream accuracy, and hardware performance.
summary: Large language models (LLMs) has recently gained widespread attention，but face challenges in memory and computation requirements.  Post-training quantization (PTQ) is a common approach to address these issues. This work mainly proposes some optimization methods for PTQ targeting LLM.The proposed TesseraQ, a novel PTQ technique, can quantize LLMs to ultra-low bits. It introduces progressive adaptive rounding to iteratively transition soft rounding variables to hard ones during reconstruction and optimizes the dequantization scale parameters. TesseraQ can be integrated with existing PTQ algorithms like AWQ and OmniQuant. Experiments show that TesseraQ significantly improves performance.
weaknesses: 1.  3.2 Some expressions are not very clear, and some variables have not been explained
2.  The new rounding error optimization method proposed in the paper mentions that compared to previous methods ( https://arxiv.org/pdf/2004.10568 ) does not rely on regularization loss , but HS and sorting were introduced. It doesn't have a significant advantage.For example, computational complexity and convergence speed.
3.  The paper combines the proposed rounding error optimization method with other methods for experimentation and comparison with other quantization methods, but does not use other rounding error optimization methods for the same experiment, and does not demonstrate the superiority of this method.For example, the Adaround mentioned in the article.
4. When comparing quantitative indicators such as W4A4/W3A3 that include activation, the QuaRot initial method was used, but it was not compared with using the QuaRot method alone. The advantage of the proposed new rounding error optimization method in improving accuracy cannot be clearly demonstrated.Suggest to use QuaRot as a baseline separately to include the results.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Why didn't the authors compare their method with OmniQuant for weight-activation quantization?
soundness: 2
strengths: The authors introduce two techniques to optimize more parameters under PTQ settings, enhancing the performance of quantized models through block-wise reconstruction. These approaches, combined with existing transformation-based methods, deliver improved model accuracy without adding inference overhead.
summary: TesseraQ is a new PTQ method for ultra-low bit quantization of LLM weights. It features progressive adaptive rounding and optimized dequantization scale parameters, improving block reconstruction. Integrated with PTQ algorithms like AWQ and OmniQuant, TesseraQ significantly enhances performance.
weaknesses: 1. While the paper claims to establish a new state-of-the-art, SpinQuant, as discussed in the related work, appears to far outperform the proposed method. It would be beneficial if the authors could initialize their model from SpinQuant and then apply their method to demonstrate its effectiveness.
2. Line 141 mentions that GPTQ can cannot be used to improve other transformation-based methods. However, GPTQ can significantly enhance the performance of rotation-based transformation like SpinQuant and QuaRot.
3. The rationale for using progressive freezing instead of a dynamic regularization term (i.e., AdaRound) for optimizing rounding values is not clearly articulated, nor is it convincingly demonstrated why this method surpasses AdaRound. The explanations in Lines 075 to 077 lack substantial evidence.
4. The paper does not compare its progressive rounding approach with existing rounding methods such as AdaRound, AdaQuant, and FlexRound, missing an opportunity to justify the need for progressive rounding.
5. Lines 230 to 231 discuss the potential negative impacts of optimizing the scale parameter $s$ in quantization. However, experimental evidence is needed to substantiate that learning $s$ is detrimental, as the current argument does not conclusively link optimization of $s$ to performance degradation.
6. The paper optimizes significantly more parameters than previous methods using limited data, raising concerns about potential overfitting. A user study or chat test could provide essential validation.
7. Line 473 states a significantly larger value than AWQ/OmniQuant, but it's unclear what this comparison signifies as these methods do not optimize rounding to flip variables.
8. The necessity of the sigmoid function $\sigma(\cdot)$ in Equation (9) should be experimentally validated. Why not use $\mathbf{v} \times \mathbf{s} \times (\theta^{q}-\mathbf{z})$ with $\mathbf{v}$ initialized to $\mathbf{1}$?
9. Further evaluation is needed on the effect of larger $t$ values in the expression $\frac{1}{\exp(tx)}$ and other scheduling functions in the **PAR Schedule** to demonstrate the method's robustness to different schedules.
10. More comparisons with block-wise reconstruction or learning-based methods, e.g, DuQuant (https://arxiv.org/abs/2406.01721) and QLLM (https://arxiv.org/pdf/2310.08041) could help confirm its effectiveness.
11. The author does not provide results for the proposed method without using transformation-based initialization.

Meta-review: The paper introduces TesseraQ, a novel post-training quantization (PTQ) technique for large language models (LLMs), aiming to improve performance by optimizing weight rounding through a block reconstruction approach. The authors also propose a progressive adaptive rounding strategy and the optimization of dequantization scale parameters. Experimental results show that TesseraQ significantly improves perplexity and downstream task accuracy when applied to the LLaMA-2 model, particularly with ultra-low bit quantization. The authors also highlight that TesseraQ can be seamlessly integrated with existing PTQ methods, offering a flexible and effective solution for optimizing LLMs.

The strengths of this paper lie in its innovative approach to weight rounding and the proposed progressive adaptive rounding strategy. The experiments are well-conducted, showing that TesseraQ outperforms previous methods, such as AWQ, on multiple quantization schemes. Additionally, the ability to integrate TesseraQ with existing PTQ methods enhances its practical applicability. The results, particularly in terms of perplexity reduction and downstream task performance, are promising and contribute to advancing the field of LLM optimization.

However, several concerns were raised by reviewers that the authors attempted to address but did not fully resolve. One major issue is the inconsistency in the reported results, especially in comparison to similar methods like SpinQuant and QuaRot. The reported perplexity for LLaMA-2-7B is notably higher than previous benchmarks, which raises doubts about the generalizability of the method. The authors have not sufficiently aligned their results with these previous studies, and the lack of detailed comparison with full versions of baseline methods undermines the claimed improvements. Additionally, the experimental setup is not sufficiently detailed, leaving unclear how some of the discrepancies arose.

Given these unresolved issues, particularly the divergence in results and the absence of a thorough comparison with full baseline versions, I recommend rejecting the paper at this time. While the proposed method is promising and the experimental results are noteworthy, the issues with result alignment, insufficient comparisons, and lack of methodological clarity need to be addressed before this work can be considered for acceptance. A more thorough evaluation across different datasets and models, along with an in-depth experimental setup description, would strengthen the paper significantly.

Decision: Reject