Attention-aware Post-training Quantization without Backpropagation

## Abstract

Abstract Quantization is a promising solution for deploying large-scale language models (LLMs) on resource-constrained devices.
Existing quantization approaches, however, rely on gradient-based optimization, regardless of it being post-training quantization (PTQ) or quantization-aware training (QAT), which becomes problematic for hyper-scale LLMs with billions of parameters.
This overhead can be alleviated via recently proposed backpropagation-free PTQ methods; however, their performance is somewhat limited by their lack of consideration of inter-layer dependencies.
In this paper, we thus propose a novel PTQ algorithm that considers inter-layer dependencies without relying on backpropagation.
The fundamental concept involved is the development of attention-aware Hessian matrices, which facilitates the consideration of inter-layer dependencies within the attention module.
Extensive experiments demonstrate that the proposed algorithm significantly outperforms conventional PTQ methods, particularly for low bit-widths.

## 1 Introduction

The explosive growth in complexity (parameters) of large-scale language models (LLMs) Touvron et al. (); Zhang et al. () has resulted in a proportional increase in computational costs, which has prompted an urgent need for efficient model processing and compression strategies.
Quantization has emerged as a pivotal solution in this context, and it serves as an essential step in the deployment of AI models on resource-constrained devices that primarily support fixed-point arithmetic.
By reducing precision, the memory bandwidth requirements can be alleviated, and the significant parallelism of quantized models can be SIMDified using highly efficient vector processing units, such as neural processing unit (NPU).

Two main categories of quantization approaches have been proposed to preserve the performance of original full-precision models: quantization-aware training (QAT) and post-training quantization (PTQ).
Although QAT can potentially outperform PTQ, its practicality diminishes considerably when handling hyper-scale LLMs featuring billions of parameters.
Consequently, recent quantization efforts have been directed toward PTQ.

Although classic PTQ methods have successfully quantized small-scale models (Nagel et al., ; Li et al., ), they rely on time-consuming gradient-based optimization, so their efficacy decreases when the complexity of LLMs increases.
Accordingly, backpropagation-free PTQ methods have been developed for LLMs (Frantar et al., ; Xiao et al., ; Jeon et al., ); however, their performance is somewhat limited owing to the lack of consideration of inter-layer dependencies.
Recent studies have attempted to consider inter-layer dependencies (Shao et al., ; Ma et al., ), but they still rely on time-consuming gradient-based optimizations.

In this paper, we propose a novel quantization algorithm that considers inter-layer dependencies without relying on backpropagation.
Our primary contributions can be summarized as follows:

- •
We propose a novel PTQ algorithm called BoA.(^1^11BoA: Backpropagation-free optimization for Attention-aware PTQ)
To avoid time-consuming gradient-based optimization, we adopt the Hessian-based strategy introduced by (Frantar and Alistarh, ).
The primary novelty lies in the fact that we exploit the attention reconstruction error, not the layer-wise reconstruction error, when approximating the Hessian.
This facilitates the consideration of the inter-layer dependencies within the attention module (Section 3.1).
- •
While the proposed Hessian facilitates the consideration of inter-layer dependencies, it requires a large amount of memory and high computational cost.
Therefore, we incorporate several techniques to mitigate the computational overhead, including Hessian relaxation, efficient computation of inverse Hessians, and head-wise simultaneous quantization (Section 3.2).
- •
We evaluate BoA via extensive experiments on publicly available LLMs (Section 4).
Our results demonstrate that BoA outperforms conventional LLM PTQ methods by a significant margin, particularly for low-bit precision (*e.g.*, INT2).

## 2 Related Works

### 2.1 PTQ for LLMs

When calibration data are available, PTQ primarily aims to minimize the increase in task loss incurred by quantization.
Consider a neural network parameterized by weights $\mathbf{W}$.
Provided that the network is trained to convergence, the problem of quantizing $\mathbf{W}$ to minimize task loss degradation can be formulated as (LeCun et al., )

$$ $\displaystyle\min_{\Delta\mathbf{w}}~{}~{}\Delta\mathbf{w}^{T}\cdot\mathbf{H}^ {(\mathbf{w})}\cdot\Delta\mathbf{w},$ (1) $$

where $\mathbf{H}^{(\mathbf{w})}$ is the Hessian related to the flattened weight $\mathbf{w}$ and $\Delta\mathbf{w}$ is a weight perturbation caused by the quantization.
Owing to the infeasibility of computing and storing the exact Hessian $\mathbf{H}^{(\mathbf{w})}$, many studies have assumed independence between layers, which relaxes 1 into the following layer-wise reconstruction problem (Nagel et al., ):

$$ $\displaystyle\min_{\Delta\mathbf{W}^{(\ell)}}~{}$ $\displaystyle\left\|\Delta\mathbf{W}^{(\ell)}\mathbf{X}^{(\ell-1)}\right\|_{F} ^{2},$ (2) $$

where $\mathbf{X}^{(\ell-1)}$ is the input to the $\ell$-th layer parameterized by $\mathbf{W}^{(\ell)}$.

To solve 2, early efforts aimed to optimize the weight-rounding mechanisms (Nagel et al., ; Hubara et al., ; Li et al., ; Jeon et al., , ).
Instead of selecting the nearest quantization bin, these studies attempted to assign quantized values that minimize layer-wise reconstruction errors.
In AdaRound (Nagel et al., ), an algorithm for optimizing the weight-rounding mechanism through backpropagation has been proposed.
Further, this algorithm has been extended to Brecq (Li et al., ) to consider the inter-layer dependencies within a certain network block (*e.g.*, Transformer block), thereby enhancing the low-bit quantization performance.

Although these algorithms have successfully quantized small-sized models such as ResNet, they are heavily dependent on time-consuming gradient-based optimizations.
This renders their application to LLMs with billions of parameters challenging.
Consequently, recent efforts have shifted towards the development of cost-effective quantization methods for LLMs.
For example, a backpropagation-free weight-rounding method called OPTQ (or GPTQ) has been proposed (Frantar et al., ).
Furthermore, algorithms leveraging additional “foldable” parameters capable of being integrated into other layers (*e.g.*, LayerNorm) have been proposed (Xiao et al., ; Lin et al., ; Jeon et al., ; Shao et al., ; Ma et al., ).
Rather than optimizing a weight-rounding mechanism, these studies focused on enhancing the performance of the nearest-rounding by suppressing activation outliers or conducting more precise quantization based on foldable parameters.

### 2.2 Review of OPTQ

Among the quantization methods developed for LLMs, OPTQ has been extensively focused upon owing to its efficiency.
OPTQ can quantize hyper-scale LLMs with more than 30B parameters in a few GPU hours with negligible performance degradation for INT3/INT4 Frantar et al. ().
In addition, OPTQ can be used to boost the performance of other quantization methods that leverage foldable parameters Lin et al. (); Jeon et al. (); Lee et al. ().

In OPTQ, weights are quantized one by one according to a pre-defined order.
Once a weight is quantized, OPTQ updates the remaining (not-yet-quantized) weights to compensate for the task loss degradation caused by quantization.
When the $q$-th weight $w_{q}$ is quantized, the weight-update $\bm{\delta}{\bm{w}}$ is mathematically expressed as

$$ $\displaystyle\bm{\delta}{\bm{w}}$ $\displaystyle=-\frac{w_{q}-{\mathcal{Q}}(w_{q})}{[\mathbf{U}]_{q,q}}[\mathbf{U }]_{q,:},$ (3a) $\displaystyle\mathbf{U}$ $\displaystyle=\operatornamewithlimits{Chol}(\mathbf{H}^{-1})^{T},$ (3b) $$

where $\operatornamewithlimits{Chol}(\cdot)$ denotes a Cholesky decomposition (*i.e.*, $\mathbf{U}$ is an upper triangular matrix satisfying $\mathbf{H}^{-1}=\mathbf{U}^{T}\mathbf{U}$).
Owing to the difficulty in computing the exact Hessian $\mathbf{H}$, OPTQ approximates $\mathbf{H}$ by assuming layer-wise independence.
Consequently, the task loss degradation is relaxed to a layer-wise reconstruction error (as in 2) Nagel et al. (), which yields the following Hessian equation:

$$ $\displaystyle\mathbf{H}^{({\bm{w}}^{(\ell)})}$ $\displaystyle\approx 2\mathbf{X}^{(\ell-1)}\mathbf{X}^{(\ell-1)^{T}}\otimes \mathbf{I},$ (4) $$

where $\otimes$ denotes the Kronecker product operation and $\mathbf{I}$ is the identity matrix.(^2^22For any $\mathbf{M}_{1}$ and $\mathbf{M}_{2}$, the second-order derivative of $\|\mathbf{M}_{1}\Delta\mathbf{W}\mathbf{M}_{2}\|_{F}^{2}$ with respect to $\Delta\mathbf{w}$ is $2\mathbf{M}_{2}\mathbf{M}_{2}^{T}\otimes\mathbf{M}_{1}^{T}\mathbf{M}_{1}$ (see Appendix A for the proof).)
In Appendix D, we provide the pseudocode of the OPTQ algorithm.

As evident, the approximated Hessian relies only on the input.
Therefore, OPTQ cannot consider the influence of other layers when compensating for the quantization error based on the Hessian (see 3a).
In other words, OPTQ does not consider the inter-layer dependency within the attention module, which is a key feature of Transformers.
Owing to ignorance of inter-layer dependency, OPTQ exhibits somewhat limited low-bit (*e.g.*, INT2) quantization performance (Jeon et al., ).
Thus, we aim to develop attention-aware Hessians, thereby incorporating the inter-layer dependencies within the attention module.

## 3 Method

**Table 1: Proposed attention-aware Hessians**
| Layer | $\mathbf{H}$ |
| --- | --- |
| $\mathbf{W}_{Q,h}$ | $2\mathbf{X}\mathbf{X}^{T}\otimes\mathbf{K}_{h}^{T}\mathbf{K}_{h}$ |
| $\mathbf{W}_{K,h}$ | $2\mathbf{X}\mathbf{X}^{T}\otimes\mathbf{Q}_{h}^{T}\mathbf{Q}_{h}$ |
| $\mathbf{W}_{V,h}$ | $2\mathbf{X}\mathbf{A}_{h}^{T}\mathbf{A}_{h}\mathbf{X}^{T}\otimes\mathbf{W}_{<br>\text{out},h}^{T}\mathbf{W}_{\text{out},h}$ |
| $\mathbf{W}_{\text{out},h}$ | $2\mathbf{X}_{\text{out},h}\mathbf{X}_{\text{out},h}^{T}\otimes\mathbf{I}$ |
| $\mathbf{W}_{\text{fc1}}$ | $2\mathbf{X}_{\text{fc1}}\mathbf{X}_{\text{fc1}}^{T}\otimes\mathbf{I}$ |
| $\mathbf{W}_{\text{fc2}}$ | $2\mathbf{X}_{\text{fc2}}\mathbf{X}_{\text{fc2}}^{T}\otimes\mathbf{I}$ |

Figure: Algorithm 1 BoA

### 3.1 Proposed attention-aware Hessian

To consider the inter-layer dependencies within the attention module, we exploit the attention reconstruction error rather than the layer-wise reconstruction error when approximating the Hessian.

For an input sequence $\mathbf{X}\in\mathbb{R}^{d\times L}$, the output of the multi-head attention (MHA) is expressed as

$$ $\displaystyle\operatornamewithlimits{MHA}(\mathbf{X})$ $\displaystyle=\sum_{h=1}^{H}\mathbf{W}_{\text{out},h}(\mathbf{A}_{h}\mathbf{V} _{h})^{T},$ (5a) $\displaystyle\mathbf{A}_{h}$ $\displaystyle=\sigma\left(\frac{\mathbf{Q}_{h}\mathbf{K}_{h}^{T}}{\sqrt{d_{h}} }\right),$ (5b) $\displaystyle[\mathbf{Q}_{h}|\mathbf{K}_{h}|\mathbf{V}_{h}]$ $\displaystyle=\mathbf{X}^{T}[\mathbf{W}_{Q,h}^{T}|\mathbf{W}_{K,h}^{T}|\mathbf {W}_{V,h}^{T}],$ (5c) $$

where $\sigma$ is the row-wise softmax function, $d_{h}$ is the embedding dimension of the $h$-th attention head, $\mathbf{W}_{\{Q,K,V\},h}\in\mathbb{R}^{d_{h}\times d}$, and $\mathbf{W}_{\text{out},h}\in\mathbb{R}^{d\times d_{h}}$.

#### Hessian for 𝐖 Q , h subscript 𝐖 𝑄 ℎ \mathbf{W}_{Q,h} bold_W start_POSTSUBSCRIPT italic_Q , italic_h end_POSTSUBSCRIPT

When $\mathbf{W}_{Q,h}$ is quantized, $\mathbf{W}_{\text{out},h}$ and $\mathbf{V}_{h}$ in 5a remain unchanged, but the attention weight $\mathbf{A}_{h}$ changes.
Using the first-order Taylor polynomial, the perturbation in $\mathbf{A}_{h}$ can be approximated as

$$ $\displaystyle\Delta\mathbf{A}_{h}$ $\displaystyle=\sigma\left(\frac{(\mathbf{Q}_{h}+\Delta\mathbf{Q}_{h})\mathbf{K }_{h}^{T}}{\sqrt{d_{h}}}\right)-\sigma\left(\frac{\mathbf{Q}_{h}\mathbf{K}_{h} ^{T}}{\sqrt{d_{h}}}\right)$ $\displaystyle\approx\frac{\Delta\mathbf{Q}_{h}\mathbf{K}_{h}^{T}}{\sqrt{d_{h}} }\mathbf{J}_{\sigma}^{T}=\frac{\mathbf{X}^{T}\Delta\mathbf{W}_{Q,h}^{T}\mathbf {K}_{h}^{T}\mathbf{J}_{\sigma}^{T}}{\sqrt{d_{h}}},$ (6) $$

where $\mathbf{J}_{\sigma}$ is the Jacobian matrix of the softmax function $\sigma$.
Thus, the attention reconstruction error is expressed as

$$ $\displaystyle\|\Delta\operatornamewithlimits{MHA}(\mathbf{X})\|_{F}^{2}$ $\displaystyle=\|\mathbf{W}_{\text{out},h}(\Delta\mathbf{A}_{h}\mathbf{V}_{h})^ {T}\|_{F}^{2}$ $\displaystyle=\left\|\frac{\mathbf{W}_{\text{out},h}\mathbf{V}_{h}^{T}\mathbf{ J}_{\sigma}\mathbf{K}_{h}}{\sqrt{d_{h}}}\Delta\mathbf{W}_{Q,h}\mathbf{X}\right \|_{F}^{2}.$ (7) $$

Combining this with Footnote 2 yields the following Hessian for $\mathbf{W}_{Q,h}$:

$$ $\displaystyle\mathbf{H}^{(\mathbf{w}_{Q,h})}$ $\displaystyle=2\mathbf{X}\mathbf{X}^{T}\otimes\frac{\mathbf{K}_{h}^{T}\mathbf{ J}_{\sigma}^{T}\mathbf{V}_{h}\mathbf{W}_{\text{out},h}^{T}\mathbf{W}_{\text{ out},h}\mathbf{V}_{h}^{T}\mathbf{J}_{\sigma}\mathbf{K}_{h}}{d_{h}}.$ (8) $$

#### Hessian for 𝐖 K , h subscript 𝐖 𝐾 ℎ \mathbf{W}_{K,h} bold_W start_POSTSUBSCRIPT italic_K , italic_h end_POSTSUBSCRIPT

When $\mathbf{W}_{K,h}$ is quantized, the attention weight $\mathbf{A}_{h}$ changes as in the quantization of $\mathbf{W}_{Q,h}$.
By following the steps for 6, $\mathbf{A}_{h}$ can be approximated as

$$ $\displaystyle\Delta\mathbf{A}_{h}$ $\displaystyle\approx\frac{\mathbf{Q}_{h}\Delta\mathbf{K}_{h}^{T}}{\sqrt{d_{h}} }\mathbf{J}_{\sigma}^{T}=\frac{\mathbf{Q}_{h}\Delta\mathbf{W}_{K,h}\mathbf{X} \mathbf{J}_{\sigma}^{T}}{\sqrt{d_{h}}},$ (9) $$

and thus the attention reconstruction error can be expressed as

$$ $\displaystyle\|\Delta\operatornamewithlimits{MHA}(\mathbf{X})\|_{F}^{2}$ $\displaystyle=\left\|\Delta\mathbf{A}_{h}\mathbf{V}_{h}\mathbf{W}_{\text{out}, h}^{T}\right\|_{F}^{2}$ $\displaystyle=\left\|\frac{\mathbf{Q}_{h}}{\sqrt{d_{h}}}\Delta\mathbf{W}_{K,h} \mathbf{X}\mathbf{J}_{\sigma}^{T}\mathbf{V}_{h}\mathbf{W}_{\text{out},h}^{T} \right\|_{F}^{2}.$ (10) $$

Thus, we obtain the following Hessian for $\mathbf{W}_{K,h}$:

$$ $\displaystyle\mathbf{H}^{(\mathbf{w}_{K,h})}$ $\displaystyle=2\mathbf{X}\mathbf{J}_{\sigma}^{T}\mathbf{V}_{h}\mathbf{W}_{ \text{out},h}^{T}\mathbf{W}_{\text{out},h}\mathbf{V}_{h}^{T}\mathbf{J}_{\sigma }\mathbf{X}^{T}\otimes\frac{\mathbf{Q}_{h}^{T}\mathbf{Q}_{h}}{d_{h}}.$ (11) $$

#### Hessian for 𝐖 V , h subscript 𝐖 𝑉 ℎ \mathbf{W}_{V,h} bold_W start_POSTSUBSCRIPT italic_V , italic_h end_POSTSUBSCRIPT

When quantizing $\mathbf{W}_{V,h}$, only $\mathbf{V}_{h}$ changes.
Thus, the attention reconstruction error is expressed as

$$ $\displaystyle\left\|\Delta\operatornamewithlimits{MHA}(\mathbf{X})\right\|_{F} ^{2}$ $\displaystyle=\|\mathbf{W}_{\text{out},h}(\mathbf{A}_{h}\Delta\mathbf{V}_{h})^ {T}\|_{F}^{2}$ $\displaystyle=\|\mathbf{W}_{\text{out},h}\Delta\mathbf{W}_{V,h}\mathbf{X} \mathbf{A}_{h}^{T}\|_{F}^{2},$ $$

which yields the following Hessian for $\mathbf{W}_{V,h}$:

$$ $\displaystyle\mathbf{H}^{(\mathbf{w}_{V,h})}$ $\displaystyle=2\mathbf{X}\mathbf{A}_{h}^{T}\mathbf{A}_{h}\mathbf{X}^{T}\otimes \mathbf{W}_{\text{out},h}^{T}\mathbf{W}_{\text{out},h}.$ (12) $$

#### Hessian for 𝐖 out , h subscript 𝐖 out ℎ \mathbf{W}_{\text{out},h} bold_W start_POSTSUBSCRIPT out , italic_h end_POSTSUBSCRIPT

When $\mathbf{W}_{\text{out},h}$ is quantized, the attention reconstruction error is expressed as

$$ $\displaystyle\left\|\Delta\operatornamewithlimits{MHA}(\mathbf{X})\right\|_{F} ^{2}$ $\displaystyle=\|\Delta\mathbf{W}_{\text{out},h}(\mathbf{A}_{h}\mathbf{V}_{h})^ {T}\|_{F}^{2}.$ $$

Thus, the corresponding Hessian is obtained as

$$ $\displaystyle\mathbf{H}^{(\mathbf{w}_{out,h})}$ $\displaystyle=2\mathbf{V}_{h}^{T}\mathbf{A}_{h}^{T}\mathbf{A}_{h}\mathbf{V}_{h }\otimes\mathbf{I}$ $\displaystyle=2\mathbf{X}_{\text{out}}\mathbf{X}_{\text{out}}^{T}\otimes \mathbf{I},$ (13) $$

where $\mathbf{X}_{\text{out},h}=(\mathbf{A}_{h}\mathbf{V}_{h})^{T}$ is the input to the out-projection layer.

Figure: Figure 1: Illustration of the proposed BoA for the query projection $\mathbf{W}_{Q}$.
Refer to caption: x1.png

**Table 2: INT2 quantization performance (perplexity $\downarrow$) of the proposed BoA and the conventional OPTQ.**
| Dataset | Method | 125M | 350M | 1.3B | 2.7B | 6.7B | 13B | 30B |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| WikiText-2 | RTN | 5.5e3 | 2.8e4 | 1.1e5 | 9.5e3 | 2.8e4 | 1.9e5 | 1.7e5 |
| OPTQ | 232.8 | 98.65 | 66.76 | 37.44 | 24.74 | 18.97 | 13.12 |  |
| BoA | 141.6 | 57.40 | 48.71 | 26.20 | 22.71 | 18.76 | 12.15 |  |
| PTB | RTN | 4.3e3 | 2.8e4 | 1.1e4 | 6.8e3 | 1.8e4 | 1.2e5 | 1.7e5 |
| OPTQ | 384.8 | 135.9 | 112.0 | 64.59 | 42.36 | 26.95 | 20.25 |  |
| BoA | 199.2 | 90.87 | 78.73 | 40.76 | 33.77 | 25.34 | 18.52 |  |
| C4 | RTN | 3.7e3 | 1.6e4 | 7.7e3 | 7.7e3 | 1.4e4 | 9.7e4 | 5.8e4 |
| OPTQ | 178.6 | 71.89 | 64.11 | 33.94 | 24.86 | 20.08 | 14.45 |  |
| BoA | 118.1 | 54.07 | 48.92 | 26.57 | 23.03 | 19.22 | 13.84 |  |

**Table 3: INT2 zero-shot task performance (accuracy $\uparrow$) of the proposed BoA and the conventional OPTQ.**
| Model | Size | Method | ARC-c | ARC-e | HellaSwag | MMLU | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OPT | 1.3B | OPTQ | 22.53 | 35.61 | 34.03 | 22.93 | 28.78 |
| BoA | 22.53 | 38.72 | 36.00 | 23.12 | 30.09 |  |  |
| 2.7B | OPTQ | 24.40 | 38.47 | 37.87 | 23.04 | 30.95 |  |
| BoA | 25.51 | 42.89 | 43.68 | 23.14 | 33.81 |  |  |
| 6.7B | OPTQ | 25.60 | 42.85 | 43.29 | 24.09 | 33.96 |  |
| BoA | 26.62 | 44.91 | 44.52 | 24.33 | 35.10 |  |  |
| 13B | OPTQ | 26.62 | 44.15 | 50.09 | 24.59 | 36.36 |  |
| BoA | 27.47 | 47.39 | 54.42 | 25.21 | 38.62 |  |  |
| 30B | OPTQ | 31.57 | 52.99 | 60.55 | 25.27 | 42.60 |  |
| BoA | 31.48 | 53.24 | 62.58 | 26.41 | 43.43 |  |  |
| LLaMA | 13B | OPTQ | 32.17 | 58.71 | 57.48 | 23.53 | 42.97 |
| BoA | 33.79 | 59.01 | 59.73 | 23.90 | 44.11 |  |  |
| 30B | OPTQ | 37.12 | 62.84 | 65.09 | 31.16 | 49.05 |  |
| BoA | 37.88 | 63.47 | 66.31 | 33.11 | 50.19 |  |  |

**Table 4: INT2 performance (perplexity $\downarrow$) of BoA integrated with methods exploiting foldable parameters.**
| Folding Param.<br>Comput. Method | Dataset | Method | OPT |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 125M | 350M | 1.3B | 2.7B | 6.7B | 13B | 30B |  |  |  |
| SmoothQuant | WikiText-2 | OPTQ | 229.4 | N/A | 39.88 | 27.31 | 20.03 | 15.32 | 13.55 |
| BoA | 151.2 | N/A | 31.62 | 24.45 | 18.55 | 14.29 | 12.49 |  |  |
| PTB | OPTQ | 292.3 | N/A | 64.17 | 44.75 | 32.01 | 22.03 | 19.26 |  |
| BoA | 223.9 | N/A | 58.17 | 38.87 | 27.85 | 19.79 | 17.97 |  |  |
| C4 | OPTQ | 151.4 | N/A | 38.13 | 26.80 | 21.22 | 16.19 | 14.42 |  |
| BoA | 130.4 | N/A | 34.20 | 24.95 | 20.92 | 15.33 | 13.90 |  |  |
| Z-Fold | WikiText-2 | OPTQ | 156.0 | 102.5 | 33.97 | 27.10 | 18.07 | 16.29 | 13.24 |
| BoA | 107.9 | 54.72 | 29.38 | 23.96 | 17.18 | 15.14 | 12.41 |  |  |
| PTB | OPTQ | 206.9 | 130.7 | 53.80 | 46.08 | 26.79 | 23.73 | 19.27 |  |
| BoA | 166.1 | 82.27 | 49.18 | 39.45 | 24.94 | 22.86 | 18.11 |  |  |
| C4 | OPTQ | 108.8 | 71.37 | 31.67 | 25.98 | 19.79 | 17.21 | 14.13 |  |
| BoA | 86.07 | 49.39 | 28.65 | 24.19 | 19.01 | 16.17 | 13.67 |  |  |

**Table 5: INT2 zero-shot performance (accuracy $\uparrow$) of BoA integrated with methods exploiting foldable parameters.**
| Folding Param.<br>Comput. Method | OPT | Method | Tasks | Average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ARC-c | ARC-e | HellaSwag | MMLU |  |  |  |  |
| SmoothQuant | 1.3B | OPTQ | 23.38 | 40.15 | 37.47 | 23.00 | 31.00 |
| BoA | 22.18 | 42.00 | 37.48 | 23.10 | 31.19 |  |  |
| 2.7B | OPTQ | 25.85 | 42.72 | 42.46 | 23.10 | 33.53 |  |
| BoA | 27.73 | 44.70 | 44.24 | 22.95 | 34.91 |  |  |
| 6.7B | OPTQ | 25.68 | 46.25 | 45.24 | 23.40 | 35.14 |  |
| BoA | 27.56 | 48.15 | 46.20 | 23.90 | 36.45 |  |  |
| 13B | OPTQ | 29.52 | 52.53 | 56.63 | 24.90 | 40.90 |  |
| BoA | 31.57 | 53.66 | 58.69 | 25.35 | 42.32 |  |  |
| 30B | OPTQ | 29.18 | 54.46 | 60.04 | 24.54 | 42.06 |  |
| BoA | 31.57 | 55.93 | 62.18 | 25.54 | 43.81 |  |  |
| Z-Fold | 1.3B | OPTQ | 23.89 | 42.05 | 40.45 | 23.07 | 32.37 |
| BoA | 24.74 | 43.98 | 40.31 | 23.45 | 33.12 |  |  |
| 2.7B | OPTQ | 25.00 | 41.46 | 43.13 | 23.16 | 33.19 |  |
| BoA | 26.37 | 43.06 | 45.18 | 23.50 | 34.53 |  |  |
| 6.7B | OPTQ | 30.46 | 48.78 | 52.46 | 25.64 | 39.34 |  |
| BoA | 28.58 | 49.75 | 55.45 | 26.67 | 40.11 |  |  |
| 13B | OPTQ | 28.58 | 48.78 | 55.38 | 24.75 | 39.37 |  |
| BoA | 28.84 | 49.87 | 58.32 | 24.60 | 40.41 |  |  |
| 30B | OPTQ | 31.83 | 53.70 | 61.34 | 24.96 | 42.96 |  |
| BoA | 30.12 | 57.53 | 63.63 | 24.85 | 44.03 |  |  |

### 3.2 Proposed BoA

The proposed BoA algorithm quantizes weights by repeating the quantization and weight-update steps; once BoA quantizes one weight, it updates the remaining (not-yet-quantized) weights by exploiting the Hessian-based weight-update formula in 3a (see Algorithm 1).
The key difference over OPTQ is that we exploit the attention-aware Hessians developed in Section 3.1, which facilitates the consideration of the inter-layer dependencies within the attention module.

Notably, the proposed attention-aware Hessians are significantly more complex than the conventional Hessian in 4, which may incur high computational costs.
For example, computing the proposed Hessians is more expensive than computing the existing Hessian in 4.
In this subsection, we present techniques for mitigating the computational overheads incurred by the proposed attention-aware Hessians.

#### Hessian relaxation

The largest overhead related to the computation of the proposed Hessians is the Jacobian matrix $\mathbf{J}_{\sigma}$ in Section 3.1 and Section 3.1.
For one input sequence, the shape of $\mathbf{J}_{\sigma}$ is $H\times L\times L\times L$, which requires a large amount of memory and high computational cost.

To mitigate such computational overhead, we establish a relaxed Hessian that does not require the computation of $\mathbf{J}_{\sigma}$.
To this end, we build an upper bound for the attention reconstruction error in 7, which will be used as its surrogate:

$$ $\displaystyle\|\Delta\operatornamewithlimits{MHA}(\mathbf{X})\|_{F}^{2}$ $\displaystyle~{}~{}~{}\leq\left\|\frac{\mathbf{W}_{\text{out},h}\mathbf{V}_{h} ^{T}\mathbf{J}_{\sigma}}{\sqrt{d_{h}}}\right\|_{F}^{2}\cdot\left\|\mathbf{K}_{ h}\Delta\mathbf{W}_{Q,h}\mathbf{X}\right\|_{F}^{2}.$ (14) $$

Moreover, we note that the term $\|\mathbf{W}_{\text{out},h}\mathbf{V}_{h}^{T}\mathbf{J}_{\sigma}\|_{F}^{2}$ in 14 is constant and does not affect quantization.(^3^33The weight-update $\bm{\delta}\mathbf{w}$ in 3a is not affected by the constant multiple of $\mathbf{H}$ ($[c\mathbf{U}]_{q,:}/[c\mathbf{U}]_{q,q}=[\mathbf{U}]_{q,:}/[\mathbf{U}]_{q,q}$).)
Thus, we do not need to consider this term when computing the Hessian.
In short, we use the term $\|\mathbf{K}_{h}\Delta\mathbf{W}_{Q,h}\mathbf{X}\|_{F}^{2}$ as a surrogate of the attention reconstruction error when deriving the Hessian for $\mathbf{W}_{Q,h}$, which results in the following relaxed Hessian:

$$ $\displaystyle\mathbf{H}^{(\mathbf{w}_{Q,h})}$ $\displaystyle=2\mathbf{X}\mathbf{X}^{T}\otimes\mathbf{K}_{h}^{T}\mathbf{K}_{h}.$ (15) $$

Similarly, we can establish a relaxed Hessian for $\mathbf{W}_{K,h}$ as follows:

$$ $\displaystyle\mathbf{H}^{(\mathbf{w}_{K,h})}$ $\displaystyle=2\mathbf{X}\mathbf{X}^{T}\otimes\mathbf{Q}_{h}^{T}\mathbf{Q}_{h}.$ (16) $$

Table 1 summarizes the proposed Hessians for each layer inside the Transformer block.

#### Efficient computation of inverse Hessians

Owing to the size of the proposed attention-aware Hessians being $dd_{h}\times dd_{h}$, the complexity of the computation of the inverse Hessian (see 3b) would be ${\mathcal{O}}(d^{3}d_{h}^{3})$ in our approach.
This is considerably more expensive than the complexity ${\mathcal{O}}(d^{3})$ in OPTQ, where the inverse of only the column-wise Hessian $\mathbf{X}\mathbf{X}^{T}\in\mathbb{R}^{d\times d}$ (in 4) is needed Frantar et al. ().

For the efficient computation of the inverse Hessians, we exploit the useful properties of the Kronecker product (17a-17c in Appendix A).
For simplicity, let $\mathbf{H}=\mathbf{H}_{\text{col}}\otimes\mathbf{H}_{\text{row}}$ where $\mathbf{H}_{\text{col}}\in\mathbb{R}^{d\times d}$ and $\mathbf{H}_{\text{row}}\in\mathbb{R}^{d_{h}\times d_{h}}$, then we obtain

$$ $\displaystyle\mathbf{H}^{-1}$ $\displaystyle=(\mathbf{H}_{\text{col}}\otimes\mathbf{H}_{\text{row}})^{-1}= \mathbf{H}_{\text{col}}^{-1}\otimes\mathbf{H}_{\text{row}}^{-1}.$ $$

This implies that the inverse Hessian $\mathbf{H}^{-1}$ can be computed by computing $\mathbf{H}_{\text{col}}^{-1}$ and $\mathbf{H}_{\text{row}}^{-1}$ (line 6 in Algorithm 1) whose complexity is ${\mathcal{O}}(d^{3})+{\mathcal{O}}(d_{h}^{3})$ ($={\mathcal{O}}(d^{3})$) and not ${\mathcal{O}}(d^{3}d_{h}^{3})$.
Similarly, we can efficiently compute the Cholesky decomposition with the same order of complexity as OPTQ.
Specifically, if $\mathbf{L}_{1}=\operatornamewithlimits{Chol}(\mathbf{H}_{\text{col}}^{-1})$ and $\mathbf{L}_{2}=\operatornamewithlimits{Chol}(\mathbf{H}_{\text{row}}^{-1})$, $\mathbf{H}^{-1}$ can be expressed as

$$ $\displaystyle\mathbf{H}^{-1}$ $\displaystyle=\mathbf{L}_{1}\mathbf{L}_{1}^{T}\otimes\mathbf{L}_{2}\mathbf{L}_ {2}^{T}=(\mathbf{L}_{1}\otimes\mathbf{L}_{2})(\mathbf{L}_{1}\otimes\mathbf{L}_ {2})^{T}.$ $$

Subsequently, noting that the Kronecker product of lower triangular matrices is also lower triangular, we obtain

$$ $\displaystyle\operatornamewithlimits{Chol}(\mathbf{H}^{-1})$ $\displaystyle=\mathbf{L}_{1}\otimes\mathbf{L}_{2}$ $\displaystyle=\operatornamewithlimits{Chol}(\mathbf{H}_{\text{col}}^{-1}) \otimes\operatornamewithlimits{Chol}(\mathbf{H}_{\text{row}}^{-1}).$ $$

Thus, we can obtain $\operatornamewithlimits{Chol}(\mathbf{H}^{-1})$ by computing $\operatornamewithlimits{Chol}(\mathbf{H}_{\text{col}}^{-1})$ and $\operatornamewithlimits{Chol}(\mathbf{H}_{\text{row}}^{-1})$ (line 7 in Algorithm 1).
Consequently, the computational complexity of the Choleksy decomposition would be ${\mathcal{O}}(d^{3})$ and not ${\mathcal{O}}(d^{3}d_{h}^{3})$.

#### Simultaneous quantization of different heads

In contrast to OPTQ wherein only column-wise Hessian is considered, the proposed BoA models both column- and row-wise Hessians (see Fig. 1(a)).
Using the row-wise Hessian, we can compensate for the quantization error of a certain row by updating the other rows.
However, in this case, the rows must be quantized sequentially (not simultaneously).
For example, the second row can be quantized after being updated to compensate for the quantization error of the first row.
This is in contrast to OPTQ, wherein the two rows are quantized simultaneously.

To accelerate the quantization process, we assume independence between different attention heads (see Fig. 1(a)).
Under this assumption, the rows related to different heads are independent and can thus be quantized together.
For a better understanding, we consider the query projection $\mathbf{W}_{Q}$ as an example (see Fig. 1(b)).
In the quantization step, we stack the $j$-th rows $[\mathbf{W}_{Q,h}]_{j,:}$, constructing $\mathbf{W}_{Q}^{(j)}\in\mathbb{R}^{H\times d}$ (line 9 in Algorithm 1).
Because the rows of $\mathbf{W}_{Q}^{(j)}$ are mutually independent (*i.e.*, the row-wise Hessian is the identity), $\mathbf{W}_{Q}^{(j)}$ can be quantized quickly, as in OPTQ (line 10 in Algorithm 1).
Following quantization, we compensate for the quantization error by updating the remaining rows.
In this update step, we use the refined weight-update formula (line 11 in Algorithm 1); the detailed derivation for this is provided in Appendix B.

## 4 Experiments

### 4.1 Experimental setup

To evaluate the performance of the proposed BoA, we quantize publicly available language models including OPT Zhang et al. (), BLOOM Scao et al. (), and LLaMA Touvron et al. ()).
As in Frantar et al. (); Jeon et al. (); Chee et al. (), we quantize only weights and retain activations with full precision because activations do not pose a significant bottleneck for the inference of LLMs Frantar et al. (); Kim et al. ().
As a calibration dataset, we use 128 random 2048 token segments from the C4 dataset Raffel et al. ().
Thus, we do not use any task-specific data for quantization.
We evaluate the performance of the quantized models using benchmark datasets (*e.g.*, WikiText-2 Merity et al. (), C4 Raffel et al. (), and PTB Marcus et al. ()) and zero-shot tasks.
All experiments were conducted using a single NVIDIA A100 GPU (80 GB).

When determining a quantization order, the heuristic introduced by (Frantar et al., ) can be employed; the column/row corresponding to the largest $\text{diag}(\mathbf{H}_{\text{col}})/\text{diag}(\mathbf{H}_{\text{row}})$ (*i.e.*, the most sensitive column/row for quantization) is first quantized for better compensation.
Empirically, we observed that this heuristic could occasionally enhance the performance, yet at other times, it may result in inferior performance.
We conduct experiments with and without this heuristic and report the better results.

### 4.2 Comparison with OPTQ

We compare the proposed BoA with the conventional OPTQ Frantar et al. (), which is our primary baseline.
For both algorithms, we set per-channel quantization parameters (*i.e.*, scale and zero-point) to minimize the layer-wise reconstruction error (line 5 in Algorithm 1).
We note that in OPTQ, the Min-Max-based quantization parameters have been used (Frantar et al., ); however, this results in significantly worse quantization performance (Chee et al., ; Jeon et al., ).
To compare the pure performance of the proposed BoA and OPTQ, we do not use foldable parameters in this experiment.
The results of using the foldable parameters are presented in Section 4.3.

First, we compare the perplexity (PPL) performance of BoA and OPTQ (Table 2(b)).
The performance of the rounding-to-nearest (RTN) method (which naively assigns the nearest quantized value) is also included for comparison, as in Frantar et al. ().
While RTN collapses completely for low bit-widths, BoA and OPTQ exhibit reasonable PPL, even for INT2 quantization.
This is because BoA and OPTQ minimize the task loss degradation, not the weight quantization error $\Delta\mathbf{W}$, by exploiting the Hessian.
Moreover, our method outperforms OPTQ for all models.
In particular, the performance gap is significant for low bit-width (*i.e.*, INT2) and small-sized models suited for resource-limited devices (*e.g.*, mobile devices).

To compare the zero-shot performance of the proposed BoA and OPTQ, we measure the accuracy of the quantized models for several tasks and then average the results.
We note that the zero-shot setting is maintained in our experiments because we do not use task-specific data for quantization.
As shown in Table 3, the proposed BoA outperforms OPTQ for all models.
The key factor leading to such an outstanding performance is that we consider inter-layer dependencies within the attention module by targeting attention-wise reconstruction.
This is in contrast to OPTQ, where layers are assumed to be independent.

### 4.3 Integration with Existing Works

As mentioned in Section 2.1, recent studies have used foldable parameters for more precise quantization Xiao et al. (); Jeon et al. (); Lin et al. (); Shao et al. (); Ma et al. ().
Because these approaches are orthogonal to ours, we can further enhance the quantization performance by integrating them with the proposed BoA.
Among the various algorithms, we employ SmoothQuant (Xiao et al., ) and Z-Fold (Jeon et al., ) in our integration because they efficiently compute foldable parameters without time-consuming gradient-based optimization.(^4^44SmoothQuant and Z-Fold determine foldable parameters using a pre-defined heuristic rule and alternating least squares, respectively.)

Table 4 summarizes the PPL performance of the proposed BoA combined with SmoothQuant and Z-Fold.
For comparison, we also summarize the integration results for the conventional OPTQ.
Overall, the performances of both BoA and OPTQ improve when using the folding parameters.
Moreover, the performance gap between the proposed BoA and OPTQ remains significant, even when the folding parameters are used.
In particular, when combined with Z-Fold, BoA outperforms other algorithms that learn attention-aware folding parameters via backpropagation by a significant margin (further details in Section C.3).

A similar behavior is observed in the zero-shot results (Table 5); the performance is boosted by leveraging the folding parameters, and BoA outperforms OPTQ for all models regardless of the folding parameter computation method.

## 5 Conclusion

In this paper, we proposed a novel PTQ algorithm called BoA.
To consider the inter-layer dependencies within the attention module while circumventing time-consuming gradient-based optimization, we approximated the Hessian matrices by exploiting the attention reconstruction error.
Furthermore, to mitigate the computational overhead incurred by the proposed attention-aware Hessians, we incorporated several techniques, such as Hessian relaxation, efficient computation of inverse Hessians, and head-wise simultaneous quantization.
Finally, through extensive experiments, we demonstrated the efficacy of the proposed BoA algorithm.

## 6 Limitations

Recall that we used the attention reconstruction error to approximate the Hessian matrix.
If the reconstruction error for the entire Transformer block is used, then the dependencies between more layers, including fully-connected layers, can be considered, resulting in further enhancement at the expense of higher computational cost.
Furthermore, we did not consider activation quantization in this study because activations do not pose a significant bottleneck, and the inference of LLMs can be sufficiently accelerated by reducing memory movement via weight quantization (Frantar et al., ; Kim et al., ).
For the scenarios wherein the activation needs to be quantized, orthogonal techniques (Xiao et al., ; Yao et al., ) can be combined with the proposed BoA, which will be considered in future studies.

## Appendices

## Appendix A Proof of Footnote 2

In our proof, we use the following useful properties of the Kronecker product:

$$ $\displaystyle\operatornamewithlimits{vec}\left(\mathbf{M}_{1}\mathbf{M}_{2} \mathbf{M}_{3}\right)$ $\displaystyle=\left(\mathbf{M}_{3}^{T}\otimes\mathbf{M}_{1}\right) \operatornamewithlimits{vec}(\mathbf{M}_{2}),$ (17a) $\displaystyle\left(\mathbf{M}_{1}\otimes\mathbf{M}_{2}\right)^{T}$ $\displaystyle=\mathbf{M}_{1}^{T}\otimes\mathbf{M}_{2}^{T},$ (17b) $\displaystyle\left(\mathbf{M}_{1}\otimes\mathbf{M}_{2}\right)\left(\mathbf{M}_ {3}\otimes\mathbf{M}_{4}\right)$ $\displaystyle=\mathbf{M}_{1}\mathbf{M}_{3}\otimes\mathbf{M}_{2}\mathbf{M}_{4},$ (17c) $$

where $\operatornamewithlimits{vec}(\cdot)$ denotes the vectorization operation.

Using 17a, we have

$$ $\displaystyle\left\|\mathbf{M}_{1}\Delta\mathbf{W}\mathbf{M}_{2}\right\|_{F}^{2}$ $\displaystyle=\left\|\left(\mathbf{M}_{2}^{T}\otimes\mathbf{M}_{1}\right) \Delta\mathbf{w}\right\|_{2}^{2}=\Delta\mathbf{w}^{T}\left(\mathbf{M}_{2}^{T} \otimes\mathbf{M}_{1}\right)^{T}\left(\mathbf{M}_{2}^{T}\otimes\mathbf{M}_{1} \right)\Delta\mathbf{w},$ $$

where $\Delta\mathbf{w}=\operatornamewithlimits{vec}(\Delta\mathbf{W})$.
In addition, by 17b and 17c, we have

$$ $\displaystyle\Delta\mathbf{w}^{T}\left(\mathbf{M}_{2}^{T}\otimes\mathbf{M}_{1} \right)^{T}\left(\mathbf{M}_{2}^{T}\otimes\mathbf{M}_{1}\right)\Delta\mathbf{w}$ $\displaystyle=\Delta\mathbf{w}^{T}\left(\mathbf{M}_{2}\otimes\mathbf{M}_{1}^{T }\right)\left(\mathbf{M}_{2}^{T}\otimes\mathbf{M}_{1}\right)\Delta\mathbf{w}$ $\displaystyle=\Delta\mathbf{w}^{T}\left(\mathbf{M}_{2}\mathbf{M}_{2}^{T} \otimes\mathbf{M}_{1}^{T}\mathbf{M}_{1}\right)\Delta\mathbf{w}.$ $$

Finally, by exploiting the fact that $\frac{\partial^{2}\mathbf{x}^{T}\mathbf{A}\mathbf{x}}{\partial\mathbf{x}^{2}}=
\mathbf{A}+\mathbf{A}^{T}$, we obtain

$$ $\displaystyle\frac{\partial^{2}\left\|\mathbf{M}_{1}\Delta\mathbf{W}\mathbf{M} _{2}\right\|_{F}^{2}}{\partial\Delta\mathbf{w}^{2}}$ $\displaystyle=\mathbf{M}_{2}\mathbf{M}_{2}^{T}\otimes\mathbf{M}_{1}^{T}\mathbf {M}_{1}+\left(\mathbf{M}_{2}\mathbf{M}_{2}^{T}\otimes\mathbf{M}_{1}^{T}\mathbf {M}_{1}\right)^{T}$ $\displaystyle\overset{(a)}{=}\mathbf{M}_{2}\mathbf{M}_{2}^{T}\otimes\mathbf{M} _{1}^{T}\mathbf{M}_{1}+\left(\mathbf{M}_{2}\mathbf{M}_{2}^{T}\right)^{T} \otimes\left(\mathbf{M}_{1}^{T}\mathbf{M}_{1}\right)^{T}$ $\displaystyle=2\mathbf{M}_{2}\mathbf{M}_{2}^{T}\otimes\mathbf{M}_{1}^{T} \mathbf{M}_{1},$ $$

where (a) follows from 17b.
This completes the proof.

## Appendix B Refined Weight-update Formula

We recall that the Hessian-based weight-update formula is given by (Frantar and Alistarh, ; Frantar et al., )

$$ $\displaystyle\bm{\delta}{\bm{w}}$ $\displaystyle=-\frac{w_{q}-{\mathcal{Q}}(w_{q})}{[\mathbf{U}]_{q,q}}[\mathbf{U }]_{q,:}\text{ where }\mathbf{U}=\operatornamewithlimits{Chol}(\mathbf{H}^{-1} )^{T}.$ $$

For the proposed attention-aware Hessians in Table 1, we have

$$ $\displaystyle\mathbf{U}_{h}=\mathbf{U}_{\text{col},h}\otimes\mathbf{U}_{\text{ row},h},$ $$

where $\mathbf{U}_{\text{col},h}=\operatornamewithlimits{Chol}(\mathbf{H}_{\text{col}
,h}^{-1})^{T}$ and $\mathbf{U}_{\text{row},h}=\operatornamewithlimits{Chol}(\mathbf{H}_{\text{row}
,h}^{-1})^{T}$ (see Section 3.2).
Therefore, the weight-update formula can be recast as

$$ $\displaystyle\bm{\delta}{\bm{w}}_{h}$ $\displaystyle=-\frac{w_{q}-{\mathcal{Q}}(w_{q})}{[\mathbf{U}_{\text{col},h} \otimes\mathbf{U}_{\text{row},h}]_{q,q}}[\mathbf{U}_{\text{col},h}\otimes \mathbf{U}_{\text{row},h}]_{q,:}.$ $$

Figure: Figure 2: Illustration of the Hessian information when $d_{\text{row}}=2$ and $d_{\text{col}}=3$
Refer to caption: x2.png

For simplicity, suppose we quantize the first (0 0-th) row.
When the weight $[\mathbf{W}_{h}]_{0,j}(=[\mathbf{W}^{(0)}]_{h,j})$ in the $j$-th column is quantized, the weight-update of the $i$-th row is simplified as (see Fig. 2 for the ease of understanding)

$$ $\displaystyle[\bm{\delta}\mathbf{W}_{h}]_{i,:}$ $\displaystyle=-\frac{[\mathbf{W}_{h}]_{0,j}-{\mathcal{Q}}([\mathbf{W}_{h}]_{0, j})}{[\mathbf{U}_{\text{row},h}]_{0,0}[\mathbf{U}_{\text{col},h}]_{j,j}}[ \mathbf{U}_{\text{row},h}]_{0,i}[\mathbf{U}_{\text{col},h}]_{j,:}$ $\displaystyle=-\frac{[\mathbf{W}_{h}]_{0,j}-{\mathcal{Q}}([\mathbf{W}_{h}]_{0, j})}{[\mathbf{U}_{\text{col},h}]_{j,j}}\cdot\frac{[\mathbf{U}_{\text{row},h}]_ {0,i}[\mathbf{U}_{\text{col},h}]_{j,:}}{[\mathbf{U}_{\text{row},h}]_{0,0}}$ $$

Thus, after the quantization of all weights in the first row, the total amount of the weight-update for the $i$-th row can be expressed as

$$ $\displaystyle[\bm{\delta}\mathbf{W}_{h,\text{total}}]_{i,:}$ $\displaystyle=-\sum_{j=0}^{d_{\text{col}}-1}\frac{[\mathbf{W}_{h}]_{0,j}-{ \mathcal{Q}}([\mathbf{W}_{h}]_{0,j})}{[\mathbf{U}_{\text{col},h}]_{j,j}}\cdot \frac{[\mathbf{U}_{\text{row},h}]_{0,i}[\mathbf{U}_{\text{col},h}]_{j,:}}{[ \mathbf{U}_{\text{row},h}]_{0,0}}$ $\displaystyle=-\frac{[\mathbf{U}_{\text{row},h}]_{0,i}}{[\mathbf{U}_{\text{row },h}]_{0,0}}\sum_{j=0}^{d_{\text{col}}-1}\frac{[\mathbf{W}_{h}]_{0,j}-{ \mathcal{Q}}([\mathbf{W}_{h}]_{0,j})}{[\mathbf{U}_{\text{col},h}]_{j,j}}\cdot[ \mathbf{U}_{\text{col},h}]_{j,:}.$ $$

Furthermore, by noting that (see line 8 in Algorithm 2)

$$ $\displaystyle[\mathbf{E}_{\text{OPTQ}}]_{h,j}$ $\displaystyle=\frac{[\mathbf{W}_{h}]_{0,j}-{\mathcal{Q}}([\mathbf{W}_{h}]_{0,j })}{[\mathbf{U}_{\text{col},h}]_{j,j}},$ $$

we obtain

$$ $\displaystyle[\bm{\delta}\mathbf{W}_{h,\text{total}}]_{i,:}$ $\displaystyle=-\frac{[\mathbf{U}_{\text{row},h}]_{0,i}}{[\mathbf{U}_{\text{row },h}]_{0,0}}\sum_{j=0}^{d_{\text{col}}-1}[\mathbf{E}_{\text{OPTQ}}]_{h,j}\cdot [\mathbf{U}_{\text{col},h}]_{j,:}=-\frac{[\mathbf{U}_{\text{row},h}]_{0,i}}{[ \mathbf{U}_{\text{row},h}]_{0,0}}[\mathbf{E}_{\text{OPTQ}}]_{h,:}\mathbf{U}_{ \text{col},h}.$ $$

As a result, the weight-update matrix to compensate for the quantization error of the first row is given by

$$ $\displaystyle[\bm{\delta}\mathbf{W}_{h,\text{total}}]_{0:,:}$ $\displaystyle=-\frac{[\mathbf{U}_{\text{row},h}^{T}]_{0:,0}[\mathbf{E}_{\text{ OPTQ}}]_{h,:}\mathbf{U}_{\text{col},h}}{[\mathbf{U}_{\text{row},h}]_{0,0}}.$ (18) $$

By taking similar steps as above, we can easily generalize 18 for the $j$-th row as follows:

$$ $\displaystyle[\bm{\delta}\mathbf{W}_{h,\text{total}}]_{j:,:}$ $\displaystyle=-\frac{[\mathbf{U}_{\text{row},h}^{T}]_{j:,j}[\mathbf{E}_{\text{ OPTQ}}]_{h,:}\mathbf{U}_{\text{col},h}}{[\mathbf{U}_{\text{row},h}]_{j,j}}.$ (19) $$

## Appendix C Additional Experimental Results

In this appendix, we provide experimental results omitted in the main text due to the page limitation.

### C.1 Comparison with OPTQ

Table 6(c) and Table 7(c) summarize the INT3/INT4 quantization performances (perplexity) of the proposed BoA and the conventional OPTQ on various sizes of OPT, BLOOM, and LLaMA models.
As evident from Table 2(b), Table 6(c), and Table 7(c), BoA uniformly outperforms OPTQ, and the performance gap is significant for low bit-width (*i.e.*, INT2) and small-sized models suited for resource-limited devices (*e.g.*, mobile devices).

**Table 6: Quantization performance (perplexity $\downarrow$) of the proposed BoA and the conventional OPTQ on OPT.**
| Precision | Method | 125M | 350M | 1.3B | 2.7B | 6.7B | 13B | 30B |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FP16 | Baseline | 27.65 | 22.00 | 14.63 | 12.47 | 10.86 | 10.13 | 9.56 |
| INT4 | RTN | 37.28 | 25.94 | 48.20 | 16.92 | 12.10 | 11.32 | 10.98 |
| OPTQ | 30.24 | 23.50 | 14.84 | 12.53 | 11.09 | 10.26 | 9.608 |  |
| BoA | 28.93 | 22.90 | 14.72 | 12.44 | 10.88 | 10.16 | 9.571 |  |
| INT3 | RTN | 1.3e3 | 64.57 | 1.3e4 | 1.6e4 | 5.8e3 | 3.4e3 | 1.6e3 |
| OPTQ | 38.74 | 26.31 | 16.70 | 14.01 | 11.91 | 10.85 | 9.911 |  |
| BoA | 33.68 | 24.69 | 15.93 | 13.43 | 11.53 | 10.58 | 9.826 |  |

**Table 7: Quantization performance (perplexity $\downarrow$) of the proposed BoA and OPTQ on BLOOM and LLaMA.**
| Precision | Method |  | BLOOM |  | LLaMA |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 560M | 1.1B | 1.7B | 3B | 7.1B |  | 13B | 30B |  |  |
| FP16 | Baseline |  | 22.42 | 17.69 | 15.39 | 13.48 | 11.37 |  | 5.091 | 4.101 |
| INT4 | RTN |  | 25.82 | 19.98 | 16.96 | 14.75 | 12.09 |  | 5.525 | 4.536 |
| OPTQ |  | 23.44 | 18.54 | 15.90 | 13.90 | 11.63 |  | 5.262 | 4.285 |  |
| BoA |  | 23.28 | 18.32 | 15.81 | 13.84 | 11.58 |  | 5.243 | 4.262 |  |
| INT3 | RTN |  | 56.74 | 49.85 | 63.37 | 39.07 | 17.35 |  | 11.78 | 14.87 |
| OPTQ |  | 26.63 | 20.80 | 17.71 | 15.39 | 12.42 |  | 5.721 | 4.848 |  |
| BoA |  | 25.90 | 20.28 | 17.12 | 14.91 | 12.19 |  | 5.676 | 4.725 |  |

### C.2 Integration with SmoothQuant and Z-Fold

Table 4 and Table 8 summarize the PPL performances of the proposed BoA combined with SmoothQuant and Z-Fold.
For comparison, we also summarize the integration results for the conventional OPTQ.
Overall, we observe that the performances of both BoA and OPTQ improve when leveraging the folding parameters.
We also observe that even when the folding parameters are used, the performance gap between the proposed BoA and OPTQ is still significant, especially for INT2 quantization and small-sized models suited for resource-limited devices.
In particular, when combined with Z-Fold, BoA outperforms other algorithms that learn attention-aware folding parameters via backpropagation (*e.g.*, OmniQuant (Shao et al., ), AffineQuant (Ma et al., )) by a large margin (see Table 9(c) in Section C.3).

**Table 8: INT3 performance (perplexity $\downarrow$) of BoA integrated with methods exploiting foldable parameters.**
| Folding Param.<br>Comput. Method | Dataset | Method | OPT |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 125M | 350M | 1.3B | 2.7B | 6.7B | 13B | 30B |  |  |  |
| SmoothQuant | WikiText-2 | OPTQ | 39.56 | N/A | 16.32 | 13.55 | 11.90 | 10.68 | 9.857 |
| BoA | 34.58 | N/A | 15.83 | 13.33 | 11.58 | 10.38 | 9.846 |  |  |
| PTB | OPTQ | 58.00 | N/A | 24.00 | 20.36 | 17.18 | 15.45 | 14.46 |  |
| BoA | 51.44 | N/A | 22.78 | 19.83 | 16.74 | 15.18 | 14.40 |  |  |
| C4 | OPTQ | 34.98 | N/A | 17.78 | 15.68 | 13.50 | 12.55 | 11.74 |  |
| BoA | 31.52 | N/A | 17.43 | 15.48 | 13.35 | 12.48 | 11.73 |  |  |
| Z-Fold | WikiText-2 | OPTQ | 39.59 | 25.97 | 16.10 | 13.54 | 11.65 | 10.64 | 9.887 |
| BoA | 33.31 | 24.22 | 15.91 | 13.33 | 11.28 | 10.53 | 9.814 |  |  |
| PTB | OPTQ | 53.08 | 39.23 | 22.73 | 20.18 | 16.64 | 15.22 | 14.57 |  |
| BoA | 46.59 | 36.80 | 22.29 | 19.54 | 16.44 | 15.16 | 14.53 |  |  |
| C4 | OPTQ | 33.67 | 26.45 | 17.33 | 15.50 | 13.28 | 12.46 | 11.73 |  |
| BoA | 30.00 | 25.04 | 17.13 | 15.32 | 13.20 | 12.41 | 11.71 |  |  |

### C.3 Comparison with OmniQuant and AffineQuant

We compare the proposed BoA with OmniQuant (Shao et al., ) and AffineQuant (Ma et al., ), recently proposed algorithms that learn attention-aware folding parameters via backpropagation (see Table 9(c)).
As evident, BoA itself outperforms conventional attention-aware quantization schemes in almost all cases, even though BoA does not rely on time-consuming backpropagation and thus enables fast quantization (see Table 9(c)(c)).
While OmniQuant shows comparable speed by reducing the number of learnable parameters (Shao et al., ), it sometimes diverges or collapses (*i.e.*, PPL is larger than $10^{3}$) for INT2 quantization.(^5^55In OmniQuant and AffineQuant, group-wise quantization parameters have been used to supplement the INT2 quantization performance (Shao et al., ; Ma et al., ), which incurs additional memory and computational costs.)
Furthermore, when combined with SmoothQuant or Z-Fold (that finds folding parameters without backpropagation), the performance gap between BoA and OmniQuant/AffineQuant is significant, which demonstrates the efficacy of the proposed method.

**Table 9: Performance (perplexity $\downarrow$) of BoA and conventional backpropagation-based attention-aware methods.**
| Dataset | Method | 125M | 1.3B | 2.7B | 6.7B | 13B | 30B |
| --- | --- | --- | --- | --- | --- | --- | --- |
| WikiText-2 | OmniQuant Shao et al. () | NaN | NaN | NaN | 2.3e4 | 4.5e5 | 3.8e5 |
| AffineQuant Ma et al. () | 174.5 | NaN | 42.26 | 26.25 | 38.89 | 5.6e5 |  |
| BoA | 141.6 | 48.71 | 26.20 | 22.71 | 18.76 | 12.15 |  |
| BoA + SmoothQuant | 151.2 | 31.62 | 24.45 | 18.55 | 14.29 | 12.49 |  |
| BoA + Z-Fold | 107.9 | 29.38 | 23.96 | 17.18 | 15.14 | 12.41 |  |
| PTB | OmniQuant Shao et al. () | NaN | NaN | NaN | 5.0e4 | 3.7e5 | 2.9e5 |
| AffineQuant Ma et al. () | 254.2 | NaN | 55.58 | 37.36 | 50.10 | 3.1e5 |  |
| BoA | 199.2 | 78.73 | 40.76 | 33.77 | 25.34 | 18.52 |  |
| BoA + SmoothQuant | 223.9 | 58.17 | 38.87 | 27.85 | 19.79 | 17.97 |  |
| BoA + Z-Fold | 166.1 | 49.18 | 39.45 | 24.94 | 22.86 | 18.11 |  |
| C4 | OmniQuant Shao et al. () | NaN | NaN | NaN | 3.0e4 | 2.0e5 | 2.1e5 |
| AffineQuant Ma et al. () | 107.0 | NaN | 34.45 | 25.11 | 31.50 | 3.3e5 |  |
| BoA | 118.1 | 48.92 | 26.57 | 23.03 | 19.22 | 13.84 |  |
| BoA + SmoothQuant | 130.4 | 34.20 | 24.95 | 20.92 | 15.33 | 13.90 |  |
| BoA + Z-Fold | 86.07 | 28.65 | 24.19 | 19.01 | 16.17 | 13.67 |  |

### C.4 Results for Different Calibration Datasets

When constructing a calibration dataset, we randomly sample 128 sequences from the C4 dataset (see Section 4.1).
By changing the seed for the sampling, we can obtain different calibration datasets, which leads to different quantization results.(^6^66Tables 2(b) to 9(c) present the results for seed 0 0.)
In this appendix, we report the corresponding results and overall statistics.
Due to the limited computational resources, we conducted this experiment only for our main comparison (*i.e.*, the performances of the proposed BoA and the conventional OPTQ).

**Table 10: Performance (perplexity $\downarrow$) of the proposed BoA and the conventional OPTQ for different seeds.**
| Dataset | Seed | Method | 125M | 350M | 1.3B | 2.7B | 6.7B | 13B | 30B |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| WikiText-2 | 0 | OPTQ | 232.8 | 98.65 | 66.76 | 37.44 | 24.74 | 18.97 | 13.12 |
| BoA | 141.6 | 57.40 | 48.71 | 26.20 | 22.71 | 18.76 | 12.15 |  |  |
| 10 | OPTQ | 276.2 | 103.4 | 66.30 | 36.74 | 24.64 | 20.05 | 13.34 |  |
| BoA | 147.4 | 57.54 | 47.23 | 25.95 | 23.11 | 18.52 | 12.17 |  |  |
| 20 | OPTQ | 243.9 | 98.70 | 65.09 | 36.33 | 24.94 | 19.78 | 13.17 |  |
| BoA | 139.4 | 61.30 | 45.11 | 27.00 | 24.06 | 17.82 | 12.33 |  |  |
| 50 | OPTQ | 269.9 | 90.11 | 64.88 | 33.84 | 24.54 | 19.47 | 13.41 |  |
| BoA | 160.7 | 59.47 | 44.13 | 26.75 | 23.09 | 18.70 | 12.15 |  |  |
| 100 | OPTQ | 228.3 | 98.94 | 71.51 | 36.72 | 25.55 | 19.39 | 13.18 |  |
| BoA | 147.8 | 60.56 | 47.43 | 26.85 | 23.61 | 18.49 | 12.17 |  |  |
| Mean<br>$\pm$ Stdev | OPTQ | 250.2 $\pm$ 22 | 97.96 $\pm$ 4.8 | 66.91 $\pm$ 2.7 | 36.21 $\pm$ 1.4 | 24.88 $\pm$ 0.40 | 19.53 $\pm$ 0.41 | 13.24 $\pm$ 0.12 |  |
| BoA | 147.4 $\pm$ 8.3 | 59.25 $\pm$ 1.8 | 46.52 $\pm$ 1.9 | 26.55 $\pm$ 0.45 | 23.32 $\pm$ 0.53 | 18.46 $\pm$ 0.38 | 12.19 $\pm$ 0.077 |  |  |
| PTB | 0 | OPTQ | 384.8 | 135.9 | 112.0 | 64.59 | 42.36 | 26.95 | 20.25 |
| BoA | 199.2 | 90.87 | 78.73 | 40.76 | 33.77 | 25.34 | 18.52 |  |  |
| 10 | OPTQ | 324.1 | 161.6 | 112.7 | 62.42 | 38.91 | 27.92 | 19.80 |  |
| BoA | 185.4 | 86.16 | 76.02 | 39.73 | 33.79 | 23.55 | 18.08 |  |  |
| 20 | OPTQ | 350.4 | 127.9 | 111.6 | 62.64 | 39.84 | 27.80 | 20.57 |  |
| BoA | 188.7 | 86.60 | 79.31 | 41.69 | 34.89 | 24.59 | 18.06 |  |  |
| 50 | OPTQ | 433.8 | 134.2 | 122.1 | 59.46 | 43.05 | 27.64 | 20.43 |  |
| BoA | 206.8 | 84.94 | 87.29 | 41.91 | 36.36 | 24.56 | 18.27 |  |  |
| 100 | OPTQ | 479.2 | 142.9 | 125.8 | 59.49 | 38.26 | 27.56 | 20.02 |  |
| BoA | 164.8 | 87.16 | 77.32 | 41.67 | 35.19 | 24.56 | 17.98 |  |  |
| Mean<br>$\pm$ Stdev | OPTQ | 394.5 $\pm$ 63 | 140.5 $\pm$ 13 | 116.8 $\pm$ 6.6 | 61.72 $\pm$ 2.2 | 40.48 $\pm$ 2.1 | 27.57 $\pm$ 0.38 | 20.21 $\pm$ 0.31 |  |
| BoA | 189.0 $\pm$ 16 | 87.15 $\pm$ 2.2 | 79.73 $\pm$ 4.4 | 41.15 $\pm$ 0.91 | 34.80 $\pm$ 1.1 | 24.52 $\pm$ 0.64 | 18.18 $\pm$ 0.22 |  |  |
| C4 | 0 | OPTQ | 178.6 | 71.89 | 64.11 | 33.94 | 24.86 | 20.08 | 14.45 |
| BoA | 118.1 | 54.07 | 48.92 | 26.57 | 23.03 | 19.22 | 13.84 |  |  |
| 10 | OPTQ | 189.7 | 72.94 | 64.19 | 33.27 | 24.40 | 20.40 | 14.44 |  |
| BoA | 115.8 | 51.41 | 49.76 | 27.00 | 24.04 | 18.58 | 13.83 |  |  |
| 20 | OPTQ | 163.1 | 71.75 | 64.19 | 32.83 | 24.66 | 20.37 | 14.41 |  |
| BoA | 111.1 | 50.92 | 48.81 | 26.96 | 23.31 | 19.15 | 13.80 |  |  |
| 50 | OPTQ | 190.8 | 69.02 | 64.99 | 32.69 | 24.55 | 20.13 | 14.48 |  |
| BoA | 119.2 | 51.21 | 46.68 | 27.11 | 23.73 | 18.96 | 13.80 |  |  |
| 100 | OPTQ | 168.5 | 69.56 | 67.54 | 33.65 | 25.25 | 20.23 | 14.51 |  |
| BoA | 116.7 | 50.69 | 48.49 | 27.17 | 24.09 | 19.63 | 13.91 |  |  |
| Mean<br>$\pm$ Stdev | OPTQ | 178.2 $\pm$ 12 | 71.03 $\pm$ 1.7 | 65.00 $\pm$ 1.5 | 33.28 $\pm$ 0.53 | 24.75 $\pm$ 0.33 | 20.24 $\pm$ 0.14 | 14.46 $\pm$ 0.039 |  |
| BoA | 116.2 $\pm$ 3.1 | 51.66 $\pm$ 1.4 | 48.53 $\pm$ 1.1 | 26.96 $\pm$ 0.24 | 23.64 $\pm$ 0.46 | 19.11 $\pm$ 0.38 | 13.83 $\pm$ 0.045 |  |  |

**(a) INT3 Quantization**
| Dataset | Seed | Method | 125M | 350M | 1.3B | 2.7B | 6.7B | 13B | 30B |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| WikiText-2 | 0 | OPTQ | 38.74 | 26.31 | 16.70 | 14.01 | 11.91 | 10.85 | 9.911 |
| BoA | 33.68 | 24.69 | 15.93 | 13.43 | 11.53 | 10.58 | 9.826 |  |  |
| 10 | OPTQ | 40.72 | 26.33 | 16.99 | 13.97 | 11.89 | 10.84 | 9.881 |  |
| BoA | 34.30 | 24.74 | 16.15 | 13.43 | 11.43 | 10.59 | 9.756 |  |  |
| 20 | OPTQ | 38.72 | 26.74 | 17.17 | 13.89 | 11.98 | 10.94 | 9.941 |  |
| BoA | 34.24 | 24.94 | 16.02 | 13.32 | 11.66 | 10.51 | 9.791 |  |  |
| 50 | OPTQ | 37.72 | 26.55 | 16.93 | 14.08 | 11.86 | 10.92 | 9.769 |  |
| BoA | 34.73 | 25.00 | 16.19 | 13.44 | 11.44 | 10.54 | 9.768 |  |  |
| 100 | OPTQ | 38.37 | 26.10 | 16.97 | 14.27 | 11.89 | 10.88 | 9.840 |  |
| BoA | 34.51 | 24.92 | 16.14 | 13.72 | 11.49 | 10.52 | 9.847 |  |  |
| Mean<br>$\pm$ Stdev | OPTQ | 38.85 $\pm$ 1.1 | 26.41 $\pm$ 0.24 | 16.95 $\pm$ 0.17 | 14.04 $\pm$ 0.14 | 11.91 $\pm$ 0.044 | 10.89 $\pm$ 0.045 | 9.868 $\pm$ 0.067 |  |
| BoA | 34.29 $\pm$ 0.39 | 24.86 $\pm$ 0.13 | 16.09 $\pm$ 0.11 | 13.47 $\pm$ 0.15 | 11.51 $\pm$ 0.094 | 10.55 $\pm$ 0.036 | 9.798 $\pm$ 0.038 |  |  |
| PTB | 0 | OPTQ | 57.62 | 39.35 | 24.77 | 21.53 | 17.56 | 15.68 | 14.56 |
| BoA | 48.50 | 36.83 | 23.53 | 20.33 | 16.86 | 15.21 | 14.50 |  |  |
| 10 | OPTQ | 55.56 | 39.13 | 25.19 | 21.60 | 17.19 | 15.66 | 14.66 |  |
| BoA | 48.37 | 36.81 | 23.35 | 20.27 | 16.71 | 15.13 | 14.50 |  |  |
| 20 | OPTQ | 56.22 | 39.97 | 25.47 | 21.42 | 17.24 | 15.67 | 14.60 |  |
| BoA | 47.01 | 37.07 | 23.31 | 20.22 | 16.68 | 15.16 | 14.54 |  |  |
| 50 | OPTQ | 56.68 | 39.69 | 24.94 | 21.44 | 17.36 | 15.74 | 14.52 |  |
| BoA | 51.05 | 37.14 | 23.69 | 20.19 | 16.76 | 15.22 | 14.46 |  |  |
| 100 | OPTQ | 52.15 | 38.75 | 25.13 | 21.32 | 17.44 | 15.79 | 14.53 |  |
| BoA | 48.66 | 36.52 | 23.50 | 19.93 | 16.77 | 15.21 | 14.46 |  |  |
| Mean<br>$\pm$ Stdev | OPTQ | 55.65 $\pm$ 2.1 | 39.38 $\pm$ 0.48 | 25.10 $\pm$ 0.26 | 21.46 $\pm$ 0.11 | 17.36 $\pm$ 0.15 | 15.71 $\pm$ 0.057 | 14.57 $\pm$ 0.058 |  |
| BoA | 48.72 $\pm$ 1.5 | 36.87 $\pm$ 0.24 | 23.48 $\pm$ 0.15 | 20.19 $\pm$ 0.16 | 16.75 $\pm$ 0.068 | 15.19 $\pm$ 0.038 | 14.49 $\pm$ 0.035 |  |  |
| C4 | 0 | OPTQ | 33.90 | 26.68 | 18.18 | 16.10 | 13.60 | 12.62 | 11.76 |
| BoA | 31.12 | 25.39 | 17.74 | 15.83 | 13.34 | 12.52 | 11.75 |  |  |
| 10 | OPTQ | 34.16 | 26.68 | 18.19 | 16.07 | 13.60 | 12.63 | 11.76 |  |
| BoA | 31.72 | 25.34 | 17.72 | 15.70 | 13.34 | 12.52 | 11.74 |  |  |
| 20 | OPTQ | 34.07 | 26.88 | 18.19 | 16.07 | 13.58 | 12.62 | 11.75 |  |
| BoA | 31.29 | 25.46 | 17.72 | 15.73 | 13.33 | 12.53 | 11.74 |  |  |
| 50 | OPTQ | 33.68 | 26.80 | 18.16 | 16.06 | 13.60 | 12.67 | 11.76 |  |
| BoA | 31.11 | 25.40 | 17.73 | 15.73 | 13.37 | 12.54 | 11.74 |  |  |
| 100 | OPTQ | 33.80 | 26.57 | 18.20 | 16.12 | 13.61 | 12.66 | 11.76 |  |
| BoA | 31.38 | 25.41 | 17.75 | 15.70 | 13.36 | 12.55 | 11.75 |  |  |
| Mean<br>$\pm$ Stdev | OPTQ | 33.92 $\pm$ 0.20 | 26.72 $\pm$ 0.12 | 18.18 $\pm$ 0.015 | 16.08 $\pm$ 0.026 | 13.60 $\pm$ 0.014 | 12.64 $\pm$ 0.023 | 11.76 $\pm$ 0.0045 |  |
| BoA | 31.32 $\pm$ 0.25 | 25.40 $\pm$ 0.044 | 17.73 $\pm$ 0.013 | 15.74 $\pm$ 0.053 | 13.35 $\pm$ 0.016 | 12.53 $\pm$ 0.012 | 11.75 $\pm$ 0.0053 |  |  |

## Appendix D Pseudocode for OPTQ

In this appendix, we provide the pseudocode of the conventional OPTQ (Frantar et al., ), which is omitted in the main manuscript due to the page limitation.

Figure: Algorithm 2 OPTQ

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Refer to 2 in weakness. What is the effectiveness of proposed approaches in terms of efficiency?
soundness: 3
strengths: 1. The proposed BOA consider inter-layer dependencies within the attention module when optimize a weight-rounding mechanism. It is beneficial to maintain higher quantization accuracy, especially at low-bit precision.

2. The proposed BOA method demonstrates impressive results, especially in the low-bit regime (e.g., INT2 quantization).

3. The paper includes extensive experiments across multiple model types and sizes, demonstrating  scalability across LLMs of different parameter counts.
summary: This paper presents a novel post-training quantization (PTQ) method, termed BOA (Backpropagation-free Optimization for Attention-aware PTQ), targeting large language models (LLMs) without relying on backpropagation. The approach introduces attention-aware Hessian matrices that capture inter-layer dependencies within the attention module, aiming to improve quantization accuracy, especially at low bit-widths (e.g., INT2). BOA incorporates techniques like Hessian relaxation and efficient computation of inverse Hessians to mitigate the high computational costs. The method is benchmarked against existing PTQ approaches on LLMs, demonstrating improved performance in terms of perplexity and zero-shot task accuracy.
weaknesses: 1. Novelty Limitations: The primary contribution, the attention-aware Hessian matrix, is an incremental improvement over existing Hessian-based PTQ methods. While capturing inter-layer dependencies within the attention module is beneficial, the idea is not a novel quantization paradigm. 

2. The authors introduce optimizations approaches like Hessian relaxation and efficient computation of inverse Hessians, but the results did not show the effect of these optimization methods.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: The advantage over existing methods is not substantial, suggesting the need for further validation, such as, SmoothQuant, LLMC,QuIP etc.
soundness: 2
strengths: The topic of this paper is of significant importance and represents one of the most active and rapidly evolving research areas in the field. As LLMs grow increasingly complex, their deployment on resource-constrained devices requires innovative solutions to reduce computational and memory demands. Quantization, as a compression technique, has gained considerable traction for enabling efficient deployment of LLMs without sacrificing model accuracy.
summary: The paper introduced the BOA post-training quantization algorithm designed for LLMs that overcomes the limitations of traditional quantization methods, which struggle with inter-layer dependencies and backpropagation requirements in LLMs. BOA leveraged attention-aware Hessian matrices to better capture inter-layer interactions within the attention module, enhancing performance, especially at low bit-widths. The algorithm employed Hessian relaxation and head-wise simultaneous quantization, to attempt to reduce computational and memory costs, making it feasible for quantizing LLMs without backpropagation.
weaknesses: The technical approach of this paper is relatively straightforward, lacking intricate or highly novel methodologies. Additionally, certain English terminology within the paper is used imprecisely, which may affect clarity and readability. The comparison methods are somewhat limited, providing a narrow benchmark for evaluating the proposed technique. Moreover, while the experimental results demonstrate some improvements, the advantage over existing methods is not substantial, suggesting the need for further validation, such as, SmoothQuant, LLMC,QuIP etc.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1.How does the performance of BOA compare when tested on more advanced models, such as the LLaMA3 series, instead of the relatively outdated models used in the paper?
2.How does BOA's accuracy compare to more recent quantization methods, such as QuaRot and SpinQuant?
soundness: 2
strengths: 1.	The paper introduces an innovative PTQ method that cleverly captures inter-layer dependencies within attention modules through attention-aware Hessian matrices while avoiding backpropagation overhead. 
2.	BOA is compatible with other techniques, such as SmoothQuant and Z-FOLD, enabling further improvements in quantization accuracy by integrating different quantization strategies.
summary: This paper presents a post-training quantization method called BOA that incorporates inter-layer dependencies without relying on backpropagation. BOA leverages attention-aware Hessian matrices to capture dependencies within the attention module, a relatively rare approach in existing PTQ methods. Additionally, BOA demonstrates compatibility with techniques like SmoothQuant and Z-FOLD, allowing for further enhancements in quantization performance. However, despite these strengths, BOA does not show sufficient memory and processing time benefits compared to existing PTQ methods. The experiments are conducted on outdated models, and the comparison methods lack recent advancements. Adding more experiments with up-to-date models and techniques would strengthen the paper.
weaknesses: 1.	The experiments are primarily conducted on BLOOM, LLaMA1, and OPT models, which are somewhat outdated compared to current state-of-the-art models. The paper lacks validation on more recent models, such as the LLaMA3 series.
2.	Although the paper introduces various techniques to reduce computational overhead and claims to use a Hessian-based strategy to avoid time-consuming gradient-based optimization, as shown in Table 13, BOA’s actual overhead in terms of memory and processing time is greater than GPTQ. Additionally, in Tables 3, 4, and 5, even under 2-bit quantization, BOA's improvement over GPTQ is marginal. For Table 6, it’s worth noting that GPTQ can also integrate certain quantization algorithms, like QuaRot [1] and SpinQuant [2], to achieve better results. Including comparisons with these methods is recommended.	

[1] Ashkboos S, Mohtashami A, Croci M L, et al. Quarot: Outlier-free 4-bit inference in rotated LLMs. arXiv preprint arXiv:2404.00456, 2024.
[2] Liu Z, Zhao C, Fedorov I, et al. SpinQuant—LLM quantization with learned rotations. arXiv preprint arXiv:2405.16406, 2024.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer weaknesses for details.
soundness: 3
strengths: 1. Introducing inter-layer interaction in a training-free manner is innovative.
2. The paper is well-written.
summary: This paper presents a training-free post-training quantization method based on GPTQ. It introduces inter-layer interaction by calculating Hessian matrices using an attention module instead of a simple linear module in LLMs. Additionally, the paper proposes techniques to improve the efficiency of Hessian matrix calculations.
weaknesses: 1. The experimental setup is somewhat outdated. Additional experiments on newer models, such as LLama-2 and LLama-3, are needed.
2. Although the paper introduces a training-free PTQ method, it may be slower than training-based methods. For example, Table 2 shows that BOA takes 1 hour to quantize 2.7B models, while GPTQ quantizes larger 13B models in only 21 minutes. OmniQuant, a training-based method, requires only ~1.1 hours for 7B models. The paper should provide comprehensive comparisons of quantization times to demonstrate the proposed method's effectiveness.
3. The paper focuses on 2-bit per-channel quantization and mentions that "group-wise parameters result in additional memory costs and processing time during inference." However, weight-only quantization aims to alleviate memory constraints during the decoding stage. Group-wise quantization introduces negligible overhead but significantly improves performance and is a common practice in existing inference engines. Therefore, the paper should include results for group-wise quantization.

Meta-review: This paper introduces BOA, a post-training quantization (PTQ) method for large language models (LLMs) that avoids backpropagation by leveraging attention-aware Hessian matrices to capture inter-layer dependencies within the attention module. BOA demonstrates improved quantization accuracy, particularly at low bit-widths (e.g., INT2), and incorporates techniques like Hessian relaxation and head-wise simultaneous quantization to reduce computational overhead. While reviewers commend the paper's focus on resource-efficient quantization for LLMs and its compatibility with other techniques like SmoothQuant and Z-FOLD, they find the contributions incremental and the novelty limited. The experiments are primarily conducted on outdated models (e.g., BLOOM, LLaMA1, OPT) and lack validation on state-of-the-art models or comparisons with recent quantization methods (e.g., QuaRot, SpinQuant). Additionally, BOA’s processing and memory overheads remain higher than some existing methods, while its performance improvements are marginal. Overall, reviewers recognize the importance of the topic but suggest that stronger experimental validation and comparisons are needed to make BOA a more compelling contribution.

Decision: Reject