AffineQuant: Affine Transformation Quantization for Large Language Models
Authors: Yuexiao Ma
Huixia Li
Xiawu Zheng
Feng Ling
Xuefeng Xiao
Rui Wang
Shilei Wen
Fei Chao
Rongrong Ji
Venue: ICLR 2024 poster (2024)
Keywords: post-training quantization
large language model
Affine Transformation
Primary area: general machine learning (i.e., none of the above)

Abstract: The significant resource requirements associated with Large-scale Language Models (LLMs) have generated considerable interest in the development of techniques aimed at compressing and accelerating neural networks. 
Among these techniques, Post-Training Quantization (PTQ) has emerged as a subject of considerable interest due to its noteworthy compression efficiency and cost-effectiveness in the context of training. 
Existing PTQ methods for LLMs limit the optimization scope to scaling transformations between pre- and post-quantization weights. 
This constraint results in significant errors after quantization, particularly in low-bit configurations. 
In this paper, we advocate for the direct optimization using equivalent Affine transformations in PTQ (AffineQuant). 
This approach extends the optimization scope and thus significantly minimizing quantization errors. 
Additionally, by employing the corresponding inverse matrix, we can ensure equivalence between the pre- and post-quantization outputs of PTQ, thereby maintaining its efficiency and generalization capabilities. 
To ensure the invertibility of the transformation during optimization, we further introduce a gradual mask optimization method. 
This method initially focuses on optimizing the diagonal elements and gradually extends to the other elements. 
Such an approach aligns with the Levy-Desplanques theorem, theoretically ensuring invertibility of the transformation. 
As a result, significant performance improvements are evident across different LLMs on diverse datasets. 
Notably, these improvements are most pronounced when using very low-bit quantization, enabling the deployment of large models on edge devices. 
To illustrate, we attain a C4 perplexity of $15.76$ (2.26$\downarrow$ vs $18.02$ in OmniQuant) on the LLaMA2-$7$B model of W$4$A$4$ quantization without overhead. 
On zero-shot tasks, AffineQuant achieves an average of $58.61\%$ accuracy ( $1.98\%\uparrow$ vs $56.63$ in OmniQuant) when using $4$/$4$-bit quantization for LLaMA-$30$B, which setting a new state-of-the-art benchmark for PTQ in LLMs. 
Codes are available at: https://github.com/bytedance/AffineQuant.

Published as a conference paper at ICLR 2024
AFFINEQUANT. AFFINE TRANSFORMATION QUANTI- ZATION FOR LARGE LANGUAGE MODELS
Yuexiao Huixia Li,, Xiawu Zhengl.34, Feng ling?, Xuefeng Xiao*,
Rui Wang", Shilei Wen'; Fei Chao', Rongrong Ji
Key Laboratory of Multimedia Trusted Perception and Efficient Computing. Ministry of Education of China, School of Informaties, Xiamen University, 361005, P.R. China. 2 ByteDance Inc. 3 Peng Cheng Laboratory, Shenzhen, China.
4 Institute of Artificial Intelligence, Xiamen University.
ABSTRACT
The significant resource requirements associated with Large-scale Language Models (LLMs) have generated considerable interest in the development of tech- niques aimed at compressing and accelerating neural networks. Among these techniques, Post- Training Quantization (PTQ) has emerged as subject of consid- erable interest due to its noteworthy compression efficiency and cost-effectiveness in the context of training. Existing PTQ methods for LLMSs limit the optimiza- tion scope to scaling transformations between pre- and post-quantization weights. This constraint results in significant errors after quantization, particularly in low- bit configurations. In this paper, we advocate for the direct optimization using equivalent Affine transformations in PTQ (Affineuuant). This approach extends the optimization scope and thus significantly minimizing quantization errors. Ad- ditionally, by employing the corresponding inverse matrix, we can ensure equiv- alence between the pre- and post-quantization outputs of PTQ, thereby maintain- ing its efficiency and generalization capabilities. To ensure the invertibility of the transformation during optimization. we further introduce gradual mask op- timization method. This method initially focuses on optimizing the diagonal ele- ments and gradually extends to the other elements. Such approach aligns with the Levy-Desplanques formation. As result, significant performance improvements evident across different LLMs on diverse datasets. Notably, these improvements are most pro- nounced when using very low-bit quantization, enabling the deployment of large models on edge devices. To illustrate, we attain a C4 perplexity of 15.76 (2.26., 18.02 in OmniQuant) on the LLaMA2-7B model of W4A4 quantization with- out overhead. On zero-shot tasks, AffineQuant achieves an average of 58.61% % accuracy (1.9%% t vs 56.63 in OmniQuant) when using 4/4-bit quantization for LLaMA-30B, which setting new state-of-tt sate--ff-the-re
theorem, theoretically ensuring invertibility of the trans-
ahr art benchmark for PTQ in LLMs.
INTRODUCTION
Large Language Models (LLMs) (Zhang et al. 2022 Touvron et al. 2023a b) attract increasing attention due their impressive performance. However, emergent logical reasoning abilities (Wei et al. 2022a) are only present in models above certain size threshold. Hence, the training and in- ference efficiency of LLMs necessitates careful consideration. Specifically, the potential utilization of LLMs for inference ing model inference, Quantization is regarded as one of the most promising methods among these compression compression methods. In particular, it maps weights activations lower bit representations, effectively reducing the memory usage of the model. Additionally, optimizing the compilation of operators for low-bit operations MMLC 2023) significantly enhance their efficiency and accelerate model inference.
mobile and edge devices drives motivation to focus on accelerat-
* "This work was done when Yuexiao Ma was intern at ByteDance Inc. Code is available at: https::/ htpp:///gthhii httggght...oobyyteeeeeegeeeo 1 *Coresponding Author: mji@xmu.edu.co

---

Published as a conference paper at ICLR 2024
Quantization Error
Before Transform
After Transform
AW Q: PPL 251.84
OmniQuant: PPL 75.43
Ours: PPL 42.29
Fixed Point
Fixed Point
Fixed Point
Fixed Point
Fixed Point
Point
Scaling Transformatin $ W
Translation W +
Affine Transformatin AW
The effect of scaling, translation and affine transformation on the quantization of the weights. The term "Fixed Point" refers to the quantization levels in n-bit quantization. 8, b, and A are the scaling factor, translation factor, and affine transformation matrix, respectively. We assume that the input channel and output channel is 2. We consider each output channel as a two-dimensional vector.
Figure/ 1:
Meanwhile, due to the substantial computational resources and high-quality data required for train- ing LLMs (Zhang al. 2023a c). implementing quantization through model fine-tuning is chal- lenging. Therefore, the research community increasingly emphasizing training- free algorithms, termed Post- Dest-Trainnng Quantization (PTQ) (Wei et al., 2022c; Xiao et al., 2023 Wei et al. 2023 Lin et al. 2023; Shao et al. [023] Yuan et al. 2023). Post- Post-training quantization allows for efficient optimization with less calibration data. However, this process can lead significant performance degradation, particularly in small-sized models or low-bit scenarios.
widely adopted in most PTQ methods. As illustrated in Figure T AWQ (Lin et al. 2023) enhances scale computation by optimizing statistics and introduces the mean square error loss between the pre- and post-quantization feature maps an optimization metric for the first time in LLMs. Recently, Omniquant (Shao et al. (023) introduces block-wise learnable scale and shift parameters for enhanced optimization. In higher dimensions, the concept of equiva- lent quantization is also gaining attention. RPTQ Yuaan et al. (023) achieves activation quantization per-cluster by sorting the columns of activation values. The reordering can be mathematically rep- resented by converting the scale from vector into matrix form, where each row and column cor- responds to a single scale value. This transformation effectively rearranges the activation columns and weight rows in an equivalent manner.
Equivalent transformations
In summary, the evolution of equivalent quantization progresses from manual design to gradient op- timization, from diweennnnional high-dimensional transformations, and from single-scale merg- ing to a combination of multiple operations, including translation and reordering. Equivalent quanti- zation offers advantages in two main aspects. Firstly, by ensuring consistency between the pre- and post-quantization outputs, the introduced quantization noise can be effectively mitigated through optimization of the equivalence transform parameters. This aligns with the concept of post-training quantization, where the equivalence transform acts as an intermediate agent for noise improvement. Secondly, different types of equivalence transforms orthogonal each other. Intuitively, the introduction of each new type of equivalence transform expands the parameter optimization space, resulting in performance improvements.
Therefore, we propose algorithm for equivalent affine transformation. Specifically, we left- multiply the affine transform matrix to weights in the linear layer and right-multiply the activations with the transform matrix inverse. Guided by the mean square error loss, we optimize the affine transformation matrix, resulting in consistently lower loss compared to other algorithms on a wide range of models during the optimization process. Furthermore, trices during the optimization process. The Levy-Desplanques theorem (Naimark & Zeheb, 1997) demonstrates that the strictly diagonally dominant matrices are invertible. To ensure that the affine transformation matrix is strictly diagonally dominant, we employ diagonal initialization and gradual mask methods. In this way, the optimization of high-dimensional matrices with limited calibra-
explore the invertibility of ma-
2

---

Published as a conference paper at ICLR 2024
tion data is stabilized through gradual optimization process that involves freezing the parameters. In terms of inference efficiency, our method is consistent with other methods after matrix merg- ing. Finally, our method achieves ofte-of-theear performance in LLMs quantization, particularly in scenarios involving small-scale models or lower bit configurations. Overall, our contributions are summarized follows:
We propose novel affine transform in PTQ, which retains the benefits of PTQ. assures efficiency and generalization, significantly minimizes quantization error, especially under low-bit quantization, and enables the deployment of LLMs on edge devices.
We propose novel optimization algorithm that guarantees invertibility throughout the pro- cess, utilizing the eevy-Desplanques theorem, and simultaneously reduces computational costs.
Our method obtains the state-of-the- performance for large language model quantiza- tion, especially on low-bit or small models. Without additional overhead, on the w4a4 configuration of LLaMA2-7B, our perplexity on the C4 dataset is 15.76 (2.26. vs 18.02 in OmniQuant), Similarly, on the w4a4 configuration of LLaMA-30B, our accuracy on 6 zero-shot tasks is 58.61% (1.98 f vs 56.63 OmniQuant).
RELATED WORK
Quantization can be classified into two main categories based on algorithmic efficiency and data requirements: Quantization Aware Training (QAT) and Post-Traine Training Quantization (PTQ). QAT (Bondarenko et al., 2023 Choi et al. 2018 Yao et al. 2021 Gong al., 2019; Wang et al., 2019 Esser et al. 2019] Sun et al. 2020; Lee et al. 2021) requires a substantial amount of data for fine-
tuning the model weights, making it challenging to support LLMs (Zhang et al. 2023b). In contrast,
focus on PTQ (Nagel et al. 2020 Li et al. 2021 Wei al. 2022b Ma et al.) 2023 Liu et al. Cai et al. 2020: Cheng et al. (023) in this study due to its efficient algorithmic approach.
2021
most-Training Quantization (PTQ). In the field of CNN, PTQ primarily focuses optimizing weight rounding strategies. Adaround (Nagel et al. 2020) improves quantization models through optimized weight rounding, considering rounding up or down. BRECQ ct. (02I) introduces block-wise optimization process and incorporates squared gradient information. QDRÖP (Wei et al.) 20226) enhances the performance of quantized models by randomly drop quantized activations.
Large Language Model Quantization. To address computational resource constraints, the com-
munity has focused efficient quantization algorithms for Large Language Models (LLMs). LLMs quantization can be categorized into weight-only quantization (Frantar & Alistarh, 2022 Lin et al.
2023 Shao et al. 2023 Kim al. 2023) and weight-activation quantization (Xiao al. 2023 Wei
et al. 2023 Yao et al. 2022 Dettmers et al. 2022 Shao et al. 2023). depending on whether acti-
vations are quantized. Given the large size LLMs, memory access efficiency becomes primary bottleneck for acceleration. Weight-only quantization addresses this by compressing model weights to lower bit precision, effectively mitigating the memory wall problem (Kim et al. 2023).
METHODOLOGY
In this section, we introduce AffineQuant, an approach that utilizes equivalent affine transformation for quantization. Compared to other methods, AffineQuant consistently maintains optimal mean square error throughout the optimization process. We also explore the reversibility of the affine transform matrix during optimization. To ensure stability, we propose gradual masking approach based on the Levy- Desplanqqus theorem (Naimark & Zeheb] 1997) to maintain the affine transform matrix as strictly diagonally dominant matrix. Lastly, we analyze the inference efficiency of LLMSs following the optimization performed by AffineQuant.
3.1 AFFINEQUANT
When considering the concept of equivalent transformations from a physical perspective, we can draw analogies to certain operations. For instance, in SmoothQuant (Xiao et al. (002), we can analogize scale scaling operations for vectors, while in Outlier Suppression+ (Wei et al. 2023
3

---

Published as a conference paper at ICLR 2024
analogize shift translation operations for vectors. Similarly, rotations of vectors can also be classified as equivalent transformations.
We define the pseudo-quamtization function follows:
((x)) = A (clamp (1 x
+ 2p.00.2 - -2p)
(I)
where A, 2p and n are the quantization step-size, zero point and bits, respectively. 1.] is the round- ing operation. As depicted in Figure [ AffineQuant involves left-multiplying the affine transform matrix A by weight matrix W to better align the weight distribution with the quantization func- tion (-)). Expanding the optimization space enables smaller quantization errors in the transformed weights, leading to a reduction in perplexity. Simultancously, we night-multiply the inverse of the affine transform matrix A by the activation value X to maintain the invariance of the matrix multi- plication output between activations and weights. For single linear layer, AffineQuant formulates the following optimization problem:
arg min XW X(A))
AffineQuant incorporates the essence of AWQ (Lin et al. 2023) and SmoothQuant (Xiao et al. 2023)
when the main diagonal elements of the matrix A are computed from weight and activation statisties.
aligns with OmniQuant (Shao et al. 2023) by exclusively updating the diagonal elements of A. The reordering matrices used in RPTQ (uuan al. 2023) matrix A when each row and column of A contains only one occurrence of the element 1. In summary, AffineQuant encompasses various previous equivalent quantization algorithms, thereby expanding the optimization possibilities for the weight distribution W.
subset of the affine transformation
In Figure I let the weight matrix W 2222 have output channels and input channels. The scaling factor, translation factor, and affine transformation matrix are denoted as b. and A, respectively. We divide the weight matrix into vectors (V1, w2} based on output channels. The scaling trans- form 8; * uniformly scales each element of The translation transform e; + b; shifts e along different axes. The affine transformation Au, allows for arbitrary repositioning of v4. However, the scaling and translation transformations are limited in their ability to map dimensions in to adjacent quantized fixed points. contrast, the affine transformation guarantees convergence of all dimensions in a vector to the quantized fixed point. In other words, the affine transformation aligns the weight distribution with the noise introduced by the quantization function Q(x) in Equation resulting in reduced quantization error. It is worth noting that normalizing the affine transformation matrix by rows (A + s' 'A)) where each row of the matrix A' has norm of 1, transforms A' into a standard rotation matrix. This rotation matrix rotates the output channels of the weights while pre- serving their magnitudes. The scaling factor performs scaling on the rotated vectors. Therefore, the affine transformation matrix A combines both scaling and rotation equivalent transformations and is orthogonal to the translation transformation.
The perplexity (ppl) exhibits exponential relationship with the cross-entropy (CE) loss, which is positively correlated with the mean square error of the output activation before and after quantiza- tion, demonstrated in (Nagel et al. 2020 Li et al., 2021). Hence, optimizing perplexity can be achieved by optimizing the mean square error before and after quantization. Specifically,
PPL x Log & || - -'A''((AW))
(3)
In large language models quantization, the optimization objective of AffineQuant is as follows,
arg min
((x - 5) ((W)) b I
A,5
where fü is the i-th transformer block. (X- A-1 Q (AW), b b 8W are the activation, weight and bias after the equivalent transformation, respectively. We combine the affine and translation transformations and use the mean square error of the transformer block output, both pre- and post- quantization, as the optimization objective.
4

---

Published as a conference paper at ICLR 2024
A
Gradual Mask
train
freeze
0
0
0
0
<
CE
0
0
Q
0
a
0
0
0
0
0
X
0
0
0
0
0
aE
a
1
X
a
0
0
1
Epoch
Figure 2: The gradual mask operates on the affine transformation matrix, gradually incorporating the elements of matrix A near the diagonal into the training process training progresses.
Figure 3 illlustrates the mean square error loss optimization for the last transformer block of LLaMA- 7B and OPT-1.3B. Notably, AffineQuant exhibits a lower initial loss compared to OmniQuant (Shao) ct al. )023) due to the superior performance of the affine transformation matrices in the preced- ing blocks. Additionally, AffineQuant demonstrates faster loss convergence and superior overall optimization performance in the last block compared to OmniQuant. These results reaffirm the significant potential of invertible matrix optimization. Figure 5 and 6 in Appendix [A.4 presents random sampling of multiple stability factors (alpha), which impact on quantization loss conver- gence. The data reveals a notable link between the last transformer block's quantization loss and the quantized model's performance. This implies AffineQuant's effectiveness in reducing quantization loss in Figure thereby enhancing the model's quantization performance during optimization.
3.2 REVERSIBILITY AND GRADUAL MASK
In the optimization process, it is necessary to invert the affine transformation matrix. However, do not include any constraints in the objective function (Equation 4 to ensure the matrix remains full rank or well-conditioned. Therefore, how to keep the matrix invertible during the optimization process? To begin, let's define a strictly diagonally dominant matrix as follows:
Definition (Strictly Diagonally Dominant Matrix) A matrix is considered strictly diagonally dominant if the absolute value of each diagonal element is greater than the sum of the absolute values of the remaining elements in the corresponding row. Specifically,
au
Σ laigh for all i. #f
(5)
The Soyy-Desllanques theorem (Naimark & Zeheb 1997) establishes that all strictly diagonally dominant matrices are invertible. By initializing the affine transformation matrix with diagonal elements, we ensure initially is strictly diagonally dominant. Although utilizing second-order momentum and lower learning rates in the optimizer can assist in satisfying the requirements of the Levy-Desphi eewy-Despanques theorem, the optimization large affine transform matrices still faces instability challenges as the model size increases.
To ensure that the affine transformation matrix remains strictly diagonally dominant during opti- mization, introduce gradual mask approach, illustrated in Figure 2 At the start of each optimization block, we freeze all elements except for those on the main diagonal. As the optimiza- tion progresses, gradually unfreeze the elements the main diagonal. Eventually, all matrix elements become learnable for optimization. This freezing mechanism, referred to as the Gradual Mask (GM), defined as follows:
i=
GM4
0< li - S x hidden size, otherwise,
(6)
0
5

---

Published as a conference paper at ICLR 2024
LLaMA-7b-w2a16
OPT-1.30-.3al1g122
QuniqQuan Quin
QusiQuast
0.50
0.45
Loss
Loss
0.40
0.35
0.30
4
5
20 Epoch
25
30
35
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
Epoch
Figure 3: Mean square error loss of the transformrr block of LLLMM-7bb and OPT-1.3b. "w2a16" means 2-bit weight-only quantization. "w3a16g128 128" means 3-bit grouping 128 weight-only quanti- zation. We optimize 40 and 20 epochs in the last block of LLaMA-7b and OPT-1.3b, respectively.
Where G.yy is the i-th row, j-th column element of the mask matrix. t is the target epochs. e E [1, the current epochs. "hidden size" the dimension of the affine transformation matrix. is the stability factor. Within the attention module, we apply a gradual mask in each attention head. GM a learning rate regulator that achieves its purpose by element-wiss dot-producting with the matrix A. Specifically, the impact of the GM matrix on the optimization process can be divided into two aspects. Here, we present the optimization process for the matrix A after incorporating the GM.
Forward:
A = Ae GMc,
ØL. 0A*
Backward:
Act1 = A& + 1) 0A 0Ae
(8)
aT.
A + G aA:
Where is the Hadamard product. Ag and G.M, are the matrices and Gradual Mask (GM) matrix in epoch e, respectively. 17 is the learning rate of matrix A. L is the optimization loss. The GM matrix effectively reduces the magnitude of non-principal diagonal elements matrix during forward propagation when the stability factor is less than 1. This ensures the existence of stable inverse matrix of A" in the optimization process during epoch e, as per the Levy-Desplanques theorem. In backward propagation, GM affects the learning rate 17, thereby suppressing the update rate of non- primary diagonal elements in matrix A. Consequently, the impact GM on 77 ensures that matrix A in epoch Notably, as a approaches 0, the optimization process converges stably and becomes equivalent to OmniQuant (Shao et 2023). Additionally, Appendix A.2 includes theorem demonstrating that a sufficiently small stability factor a ensures the strictly diagonal dominance of matrix A during optimization.
maintains strictly diagonally dominant, satisfying the Levy-Desplanques theorem.
The concept of gradual adaptation is also present in the post-training quantization of vision models. In Adaround (Nagel et al., (2020), the gradient update of parameters is controlled using gradual pow- ers of 3 in the soft quantization function. When 3 is sufficiently large, only values close to 0 or 1 are updated due to gradient limitations. gradually released. However, it is important to note that the goals and approaches of the two methods are distinct. Adaround (Nagel et al. 2020) employs the gradual power 3 to prevent fast convergence of the objective function, which can lead to suboptimal optimization. On the other hand, the gradual mask AffineQuant ensures the strictly diagonally dominant property of the affine transformation matrix. Appendix A.6 showcases heat maps depicting different block affine transformation matri- ces at various epochs, demonstrating the effectiveness of the gradual mask approach in maintaining strictly diagonally dominant matrices.
optimization progresses, the gradient of all rounded values

---

Published as a conference paper at ICLR 2024
Table I: Weight-only quantization PPL()) results on the OPT model WikiText2 dataset.
Config
Method
125M
1.3B
2.7B
6.7B
13B
30B
FP16
27.65
14.63
12.47
10.86
10.12
9.56
RTN
1.2e3
1.3e4
1.6e4
6.5e3
4.6e3
1.5e3
GPTQ
Frantar et al.
2022
53.05
21.17
16.83
15.09
11.73
10.30
w3al6
AWQ
Lin et al. 2023
69.43
28.01
263.10
15.13
20.09
35.74
AmniQuan (Saao et al.
2023
35.66
16.68
13.80
11.65
10.87
10.00
AffineQuant
30.56
15.94
13.15
11.44
10.76
9.98
RTN
51.22
119.00
297.98
23.54
46.03
18.80
GPTQ
Frantar et
2022
39.24
16.47
13.69
11.65
10.35
9.73
w3al6g128
AWQ
Lin et al. 2023
36.74
16.32
13.58
11.41
10.68
9.85
OmniQuant Quant Shao ct al
2023
32.25
15.72
13.18
11.27
10.47
9.79
AffineQuant
30.21
15.61
12.98
11.18
10.51
9.81
RTN
37.28
48.17
16.92
12.10
11.32
10.97
GPTQ Frantar al. 2022
31.43
15.56
12.82
11.41
10.31
9.63
w4al6
AWQ (Ln 2023
32.28
15.49
12.93
11.30
10.39
9.77
OmniQuant Shao ct al.
2023
29.45
15.04
12.76
11.08
10.30
9.65
AuaineQuant
28.39
14.92
12.64
10.96
10.26
9.65
RTN
30.47
15.29
13.02
11.15
10.30
9.94
GPTQ Frantar et al.
2022
29.81
14.89
12.52
10.93
10.17
9.58
w4a16g128
AWQ Lan al. 2023
29.15
14.94
12.74
10.93
10.21
9.59
OmniQuan (Shao
2023
28.86
14.88
12.65
10.96
10.20
9.62
Affine AfaineQuant
28.33
14.79
12.58
10.92
10.19
9.62
Table 2: AffineQuant and OmniQuant quantization performance of LLaMA-7B, 13B, 30B on six zero-shot datasets using 4/4 bit quantization.
PIQA
ARC-e
WinoGrande
BoolQ
ARC-c
HellaSwag
Avg.
Dataset
(1)
(1)
(f)
(t)
(t)
(1)
(f)
FP16
77.47
52.48
67.07
73.08
41.46
73.00
64.09
LLaMA-7B
OmniQuant Shao et al. 2023
66.15
45.20
53.43
63.51
31.14
56.44
52.65
w/ad
AffineQuant
69.37
42.55
55.33
63.73
31.91
57.65
53.42
FP16
79.10
59.89
70.31
68.01
44.45
76.21
66.33
LLaMA-13B w4ad
OmniQuant Shao
2023
69.69
47.39
55.80
62.84
33.10
58.96
54.37
AffineQuant
66.32
43.90
54.70
64.10
29.61
56.88
52.58
FP16
80.08
58.92
72.53
68.44
45.47
79.21
67.41
LLaMA-30B
OmniQuant Shao
2023
71.21
49.45
59.19
65.33
34.17
64.65
56.63
w4a4
AffineQuant
70.84
49.41
58.64
70.12
37.12
65.53
58.61
3.3 EFFICIENCY
Optimize Efficiency. PyTorch's linear algebra library (Paszke al. 2019) offers matrix inverse computations in both float and double precision. Consequently, we maintain the model's precision as either float double throughout the optimization process. Furthermore, approximate computations of the matrix inverse may contain errors due to the numerical precision limitations of the computer. Therefore, we analyze memory consumption, optimization time, error magnitude, and the impact on model performance for both precision types in Section
Inference Efficiency. In line with similar algorithms, we integrate the affine transformation matrix with other layers. Subsequently, we perform half-precision inference on the network. For all linear layers, we merge the affine transformation matrix with the weight and bias parameters. In addition, only optimize the diagonal elements of the affine matrix after LayerNorm for weight-activation quantization. This allows us to merge the affine matrix with the LayerNorm weights and bias. Consequently, AffineQuant can be achieved without introducing any additional overhead to model inference. Tables and demonstrate AffineQuant's 's superior performance over other methods in zero-shot and PPL. tasks, even without additional overhead, using 4/4-bit quantization.
2

---

Published as a conference paper at ICLR 2024
Table 3: Quantization performance of LLaMA1&2 WikiText2 and C4 datasets using 4/4 bit
weight-activation quantization.
Datasets
LL
Methods
1-7B
1-13B
1-30B
2-7B
2-13B
FP16
5.68
5.09
4.10
5.47
4.88
SmoothQuant Xiao et al. (2023)
25.25
40.05
192.40
83.12
35.88
WikiTex12
W4A4
OmniQuant Shao et al. (2023)
11.26
10.87
10.33
14.26
12.30
AffineQuant
10.28
10.32
9.35
12.69
11.45
FP16
7.08
6.61
5.98
6.97
6.46
SmoothQuant Xiao et al. (0023)
32.32
47.18
122.38
77.27
43.19
W4A4
OmniQuant Shao al. AffineQuant
2023
14.51
13.78
12.49
18.02
14.55
13.64
13.44
11.58
15.76
13.97
Table 4: PPL., memory usage, optimization runtime, and merge error for the OPT model under three precision schemes. The "double" scheme maintains double precision for both the model and the transform matrix. The "float" scheme indicates that both the model and the transform matrix are in float precision. The "float-double" scheme denotes that the model in float precision while the transform matrix is in double precision.
OPT-125M w2a16g61
OPT-6.7B wla16
Merge Error
PPL.
Memory Utilization Runtime
PPL.
Memory Utliizttion Runtime
FP16
24.60
11.74
Double
1.88e 16
42.43
7065.5Mb
1.19h
11.91
41414.3Mb
16. 7h
Float
2.58e
42.91
3586.6Mb
0.78h
11.90
21188 9Mb
8.65h
Float-Double
3.48e 4
42.88
3663.6Mb
0.85h
11.96
23189.5MB
12.72h
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Algorithm Details. In AffineQuant, the stability factor decreases the model size increases, the quantization bits decrease, and the group size increases. For OPT-6.7B and smaller models, we set 1e 2 for configurations with weight quantization
1. As the model size increases, of 3 bits or more. For other configurations, we select a from the set {le - 2, le 3, 1e 4). Then, exclude the affine transformation between the linear layers in the MLP module. This because the optimization the large transformation matrix inflated dimensions is challenging. Additionally, the presence of the activation function renders the equivalent transformation of the matrix invalid.
4.2 EVALUATION EXPERIMENTS
As demonstrated in Tables and 3 we observe consistent performance improvements across all models with various quantization configurations. This indicates that AffineQuant is not reliant on particular quantization configuration. Notably, AffineQuant exhibits significant improvements, par- ticularly in cases of now-bit quantization or smaller model sizes. Specifically, in the w3a16g128 configuration the OPT-1 15M model, achieve perplexity reduction of 5.10, surpassing the performance of OmniQuant (Shao et al. 2023) by a large margin. Furthermore, we achieve per- plexity reductions of 2.26 and 1.57 on LLaMA2-7B models with C4 and WikiText2 datasets, under the w4a4 quantization configuration. The aforemenniooed results underscore the importance of ex- panding the optimization space for the equivalence factor in challenging quantization tasks. For additional dataset results, please refer to the Appendix.
4.3 ABLATION STUDY
Impact of numerical precision. We compare various metrics, including merge error, PPL, memory usage, and optimization runtime, for different precision schemes. Specifically, in the "float-double" scheme, convert the activations weights double precision, multiply them with the trans-
8

---

Published as a conference paper at ICLR 2024
Table 5: Effect of different stability factors a on model performance of OPT-125M and LLaMA-7B.
Dataset
FP16
1c0
1c-1
1c-2
le-3
1c-4
1e-5
1e-6
1c-7
1e-8
WikiText2
27.65
42.08
45.32
65.77
76.03
76.78
76.00
76.48
76.55
76.46
OPT-125M
w2a16g128
PTB
32.54
63.60
68.29
114.41
135.72
125.38
124.12
132.36
132.45
113.75
C4
24.60
45.80
50.84
70.44
79.60
81.48
79.72
79.67
80.70
79.54
LLaMA-7B w2a16
WikiText2
5.68
NaN
NaN
9.53
10.63
10.90
10.99
10.72
11.63
11.67
C4
7.08
NaN
NaN
14.89
14.62
15.31
15.22
14.78
16.66
17.33
Table 6: Contributions of gradual mask on OPT-125M and LLaMA-7B model.
Scheme
WikiTex12
PTB
C4
Scheme
WikiText2
C4
FPT6
27.65
32.54
24.60
FPT6
5.68
7.08
OPT-125M
LLaMA-7B
w3a16
With Gradual
32.10
39.85
29.97
w2a16
With Gradual
9.53
14.89
Without Gradual
53.52
90.47
62.17
Without Gradual
NaN
NaN
formation matrix, and then truncate them float precision. For the merge error, we define two linear layers with input and output channels set to 4, 096. We randomly sample the affine trans- formation matrix A IB4006x44096 and the input activation X E 2204884096 We conduct 1, 000 runs to calculate the mean square error averages of the linear layer outputs before and after merging A for different precision schemes. The results are presented in Table 4 Notably, the case of double precision optimization, the computational error of the matrix inverse is minimized. Despite the higher time and memory usage compared to other schemes, the smaller merge error results in a minor improvement in PPL. However, this improvement is not significant for larger-scale models.
we adjust the stability factor a in Equation The affine transform theoretically converges to the scale transform a approaches O. We observe that as decreases, the model performance of OPT-125M and LLaMA-7B converges to OmniQuant (Shao) al. 2023). However, in the case of LLaMA-7B, larger stability factor does not guarantee the strictly diagonal dominance of the transformation matrix, which can lead to training collapse. Hence, is necessary increase a while ensuring the validity of the Sevy-Desplanques theorem (Naimark & Zcheb 1997).
Effects of stability factor. In Table 5
Contribution of gradual mask. In Equation 6 we gradually release the elements of the mask ma- trix close to the diagonal. In Table 66 when we remove the gradual approach, the OPT-125M and LLaMA-7B models exhibit poor performance or fail complete training. This indicates that updat- ing all parameters of the affine transformation matrix at the outset is not conducive to maintaining its invertible or well-conditioned properties. The gradual mask approach provides a stable means to optimize large matrices.
5 CONCLUSION
Post-training quantization based equivalence shows significant potential. However, previous equivalence methods have limited the optimizable weight space, resulting in a large quantization error in the transformed weight distribution. This limitation becomes more pronounced in the case of small models and low-bit quantization. Affine transformation methods address this issue by sig- nificantly expanding the optimizable weight space. Previous transformation methods can be seen as a special case of affine transformation or orthogonal to it. Moreover, the Levy-Desplanques theorem provides a theoretical foundation for maintaining the stability of matrix optimization in high-dimensional spaces. Following this theorem, our proposed gradual mask ensures that the ma- trix remains strictly diagonally dominant during the optimization process. This guarantees the ma- trix's invertibility well-conditioned property and further reduces the mean square error objective function. Our approach consistently improves performance across a wide range of quantization con- figurations for all models. Notably, the affine transformation method demonstrates great potential for improving performance, particularly for small models and low-bit configurations. In the future, optimizing the affine transformation matrix more effectively deserves careful consideration.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by National Science and Technology Major Project (No. 2022ZD01 (820), the National Science Fund for Distinguished Young Scholars NN.620256033,, the National Natural Science Foundation of China (No. U2IB2037, No. U22B2051, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), and the Natural Science Foundation of Fujian Province of China No.2022J06001..
REFERENCES
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing, arXiv preprint arXiv:2306. 12.3006.12992 /2929, 2023.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1911691177,,, 2020.
Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, and Kaokao Lv. Opti- mize weight rounding via signed gradient descent for the quantization of Ilms. arXiv preprint arXiv:2309.05516, 2023.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 1805.0608, 2018.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers scale. arXiv preprint arXiv:2208.07333 2022.
Steven K Esser, Jeffrey McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned step size quantization. arXiv preprint 192 2019.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances Neural Information Processing Systems, 35:4475- 4488, 2022.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 10.00.73333 17323, 2022.
Ruihao Gong, Xianglong Liu, Shenghu Jiang. Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 4852-4861, 2019.
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li. Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023.
Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Leamable rounding based on element-wiss division for post-training quantization. arXiv preprint arXiv:2306.00317, 2023.
Junghyup Lee. Dohyung Kim, and Bumsub Ham. Network quantization with element- gradient scaling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6448 6457, 2021.
Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:
2021.
Ji Lin. Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang. and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint 009799 2023.
10

---

Published as a conference paper at ICLR 2024
Zhenhua Liu, Yunhe Wang. Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quanti- zation for vision transformer. Advances in Neural Information Processing Systems, 34:28092- 28103, 2021.
Yuexiao Ma, Huixia Li, Xiawu Zheng. Xuefeng Xiao, Rui Wang, Shilei Wen, Xin Pan, Fei Chao, and Rongrong Ji. Solving oscillation problem in post-training quantization through theoreti- perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7950-7959, 2023.
Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of Workshop held Plainsboro, New Jersey, March 8-11, 1994, 1994.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 11Xiv:1609.07843 2016.
MLC. MLC-LLM, 2023. wwllatttte///gttabb.oo////laaaaa
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down? adaptive rounding for post-training quantization, In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
Leonid Naimark and Ezra Zeheb. An extension of the levy-desplanque theorem and some stability conditions for matrices with uncertain entries. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 44(2):167-170, 1997.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learming Research, 21 21(1555485-5511, 2020.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13137, 2023.
Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Kailash Gopalakrish- nan. Ultra-low precision 4-bit training of deep neural networks. Advances in Neural Information Processing Systems, 33: 196-1807 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307. 092888 2023b.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8612-8620, 2019.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint mrXiv:2206.07666 2022a.
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Ran- domly dropping quantization for extremely low-bit post-training quantization. arXiv preprint arXiv:2203.055700 2022b.
11

---

Published as a conference paper at ICLR 2024
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. Advances in Neural Information Processing Systems, 35:17402-17411 2022c.
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xian- glong Liu. Outlier suppressiont: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145, 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qi- jing Huang, Yida Wang, Michael Mahoney, al. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning, pp. 11875-1 1886. PMLR, 2021.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang. Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 18-27168-27188 2022.
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304. 01089, 2023.
patperparrattteerooiiiiiiiioioii Shaokun Zhang, Feiran Jia, Chi Wang, and Qingyun Wu. Targeted hyperparameter optimization with lexicographic preferences over multiple objectives. In The Eleventh International Conference Learning Representations, 2023a. URL. wwt.:::ttp/...cc........t......t pestps://openreveee wwt.:ntttooommmmmm13m3333
4567Ma
Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling-Hao Chen, Jiale Liu, Qingyun Wu, and Tongliang Liu. Ideal: Influence-driven selective annotations empower in-context learners in large language models. arXiv preprint arXiv:2310. 10873 2023b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106 2022.
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse llms. arXiv preprint arXiv:2310.089) 2023c.
A APPENDIX
A.1 PARETO FRONTS BASED ON WEIGHTED MEMORY
In Figure 4 We show the Pareto frontiers of AffineQuant and OmniQuant based on weighted mem- ory and PPL trade-off for LLaMA1&2 models of different sizes in the 4/4 bit quantization con- figuration. The results clearly demonstrate that AffineQuant consistently outperforms the current Oiiec-Of-hh-AAA method, OmniQuant, without any additional overhead.
A.2 STRICTLY DIAGONAL DOMINANCE GUARANTEED
To avoid confusion in the proof between a and the elements the matrix A, we temporarily denote the affine matrix A as N.
Theorem When the stability factor a small enough, if N. is strictly diagonally dominant, then Not1 is strictly diagonally dominant.
12

---

Published as a conference paper at ICLR 2024
C4
WikiText2
18
OmniQuant LLaMA
OmnnQuaatt LLaMA
14
Affineuuntt LLaMA
Aftineuuant LLaMA
17
OmniQuant LLaMA2
OmniQuant LLaMA2
AfticChant
Aftinchant LLaMA2
13
16
PPL
15
12
14
11
13
10
12
4
6
10
12
14
16
4
6
10
12
14
16
Weight Memory (Gb)
Weight Memory (Gb)
Figure 4: PPL vs. meight-memory Pareto-optimal curves for LLaMA1&2 models of different sizes the 4/4 bit quantization configuration on C4 and WikiText2.
Proof 1 Without loss of generality, we take the i-th row of Ne- Since Ne is a strictly diagonally dominant matrix, we have,
|
Σ
mngl-
(10)
j
Where n. niz are the elements of the epoch e in the i-th row, i-th column and i-th row, j-th column the absolute value of the i-th diagonal element
of the matrix N. According to the above Equation 9 of the e+1 epoch of matrix N is,
OT.
|n
n9
(11)
Om
ØL
=|n
(12)
On
Where 9h = 1, ne are the i-th diagonal elements of GM, No at epoch e, respectively. L is the loss epoch e, Further.
OL
e++1 28
1 In
+
(13)
a=0
nhe is the scale when the matrix is initialized. Therefore, the diagonal values of the matrix N are not equal to during the optimization process. Next, we focus on the right-hand side of Equation T0 at epoch e+ I. Similarly.
OL
Σ A
n
etl
In
+
ng.
(14)
On
jfi
aL Fer
+
1
(15)
jti
Where S S e is the epoch at which n4j starts updating. In other words, n = 0, and as gets smaller Tiÿ gets closer to the diagonal. In addition, gi Therefore, we have,
aL
Σ J4
e+1 m
= nja
(16)
On
j
13
13

---

Published as a conference paper at ICLR 2024
To make Extt mij < |n
we ler
07.e mnl
07.e
< [i + 1
(17)
Om
z=h
|1 + 1) Σ 0 Dnee
a
E == Bre
ØL
(18)
Thus, when the stability factor is sufficiently small, if Ne is strictly diagonally dominant matrix, then No+1 is strictly diagonally dominant matrix. The theorem is proved.
A.3
COMPARISON WITH FLEXROUND
Please refer Table
Table 7: AffineQuant vs. FlexRound. We perform accuracy comparisons on 6 zero-shot tasks.
PIQA
ARC-c
WinoGrande
BoolQ
ARC-c
HellaSwag Avg.
Dataset
(t)
(t)
(t)
(t)
(t)
(1)
(t)
FP16
77.37
52.52
66.85
73.12
41.38
72.99
64.04
LLaMA-7B w4a16
FlexRound Lec al. (2023)
77.75
50.80
66.06
70.73
40.27
71.97
62.93
AffineQuant
77.53
51.85
66.93
70.89
38.65
71.49
62.89
FPI6
79.11
59.89
70.01
68.53
44.54
76.23
66.38
LLaMA-13B
FlexRound Lee et al. (2033)
78.78
59.55
70.40
66.39
43.77
75.52
65.73
w4a16
AffineQuant
78.84
59.55
69.38
69.48
43.52
75.18
65.99
A.4 LOSS AND MODEL PERFORMANCE
We maintain consistent matrix initialization while randomly sampling the stability factor a, which influences loss convergence, for LLaMA-7B and OPT-6.7B. Using AffineQuant, we obtain the per- formance of 4/4 bit quantized models based on the sampled solution. In Figure 5 we present scatter plots depicting the output loss of the last transformer block and the corresponding model per- formance on different datasets. These plots demonstrate a significant positive correlation between loss and model performance, with correlation coefficients of 0.95.0.96 7B in WikiText2, respectively. Based on this observation, we conclude that the quantization loss of the last transformer block's 's output exhibits a strong correlation with overall model performance.
OPT-6.7B and LLaMA-
A.5 ADDITIONAL. EXPERIMENT
We list additional experiments including:
1. the OPT model on the PTB dataset.
2. OPT model on C4 dataset.
3. LLaMA1&2 on the WikiText2 dataset.
Each experiment included models different scales and wide range of quantitative configurations.
A.6 AFFINE MATRIX
Figure 7
presents a comprehensive collection of affine transformation matrices, encompassing various transformer block locations, training epochs, layers, and quantization configurations. AeeLLAfeeMMatrix... denotes the affine transformation matrix at fcl, -atttnffna--aiiiis rep- resents the affine transformation matrix proj, and Loky_Affin__Matrin affine transformation matrix at qkv. To ensure consistency, we normalize the matrix values within the range of 0 to using specified normalization method. Notably, all matrices exhibit the property
corresponds to the
14

---

Published as a conference paper at ICLR 2024
LLaMA-7b W4A4
OPT-6.7b W4A4
12.3
11.4
12.2
11.2
wikiText2 PPL
WikiText2 PPL
11.0
12.1
10.8
12.0
10.6
11.9
10.4
11.8
10.2
6.25
6.50
6.75
7.00
7.25
1.2
1.3
1.4
Loss
Figure 5: The relationship between WikiText2 PPL and quantization loss of last transformer block on I.LMA-7B and OPT-6.7B with 4/4 bit quantization.
LLaMA-7b W4A4
OPT-6.7b W4A4
13.6
15.5
13.5
15.0
13.4
C4 PPL
C4 PPL
14.5
13.3
13.2
14.0
13.1
13.5
13.0
6.25
6.50
6.75
7.00
7.25
1.1
1.2
1.3
1.4
Loss
Loss
Figure The relationship between C4 PPL and quantization loss of last transformer block on LLaMA-7B and OPT-6.7B with 4/4 bit quantization.
of being strictly diagonally dominant. Additionally, the low-bit bit affine transformation matrix demon- strates higher capability to learn rotational features, thereby reducing the model's quantization error compared to the high-bit configuration.
Furthermore, the training epochs progress, the affine transformation matrix acquires primary diagonal elements. On the other hand, the persistence of an approximate diagonal matrix at high quantization bits elucidates the modest performance improvement observed in high-bit quanti- zation configurations. This phenomenon may also be attributed to the relatively small performance gap between the quantized model and the full-precision model.
A.7 EXPERIMENTAL DETAILS
To ensure a fair comparison, we align most of our optimization parameters with those of Omni- Quant (Shao et al., 2023). Specifically, apply INT2. INT3, and INT4 wnly-weight quantization to OPT (Zhang et al. 2022) and LLaMA1&2 (Touvron et al. (202a b) models. Additionally, we em- ploy grouping strategies of 64 128 for weight quantization with different bit configurations. The
15

---

Published as a conference paper at ICLR 2024
Table 8: Welght-only quantization PPL(() results on the OPT model PTB dataset.
Config
Method
125M
1.3B
2.7B
6.7B
13B
30B
FP16
32.54
16.96
15.11
13.08
12.33
11.84
4.6e3
7.1e3
2.5e4
5.7e3
3.0e4
6.2e3
GPTQ
Frantaa el al.
2022
635.17
130.88
61.36
25.24
20.46
15.15
w2al6g128
AWQ Lin et al 2023
263.88
71.87
43.15
19.49
17.61
14.92
OmniQuan Shao
2023
126.49
34.33
25.28
18.92
16.74
14.51
AffineQuant
65.23
30.06
27.11
18.22
16.35
14.09
RTN
5.1e3
19.4e3
7.7e4
6,1e3
8.2e3
4.1e3
GPTQ Frantar et al.
2022
245.28
55.61
36.12
19.45
17.02
14.05
w2a16g64
AWQ Lin el al 2023
143.18
41.19
25.08
18.00
15.83
14.92
Omni OmniQuant Qaant Shho et a
2023
112.10
30.36
22.63
17.58
15.70
13.98
AffincQuant
60.90
27.21
21.50
17.07
15.32
13.68
RTN
1.2e3
1.1e4
1.0e4
5.2e3
3.6e3
1.4e3
GPTQ Frantar ct al
2022
34.05
27.39
15.94
13.75
13.71
12.54
w3al6
AWQ Lin al 2023)
80.73
33.20
224.11
18.46
35.45
66.68
OmniQuan Qaant Shao
2023
40.76
19.06
16.29
13.77
12.96
12.19
AffineQuant
38.38
19.14
16.32
14.19
13.54
12.48
RTN
64.67
222.13
337.75
39.90
65.33
34.27
GPTQ FFrnntar et al
2022
45.17
19.90
17.06
14.24
12.84
12.54
w3al6g128
AWQ [Lnn et al. 2023)
44.07
19.59
16.52
13.98
12.87
66.68
OmniQuan (Shao
2023
45.29
20.42
17.08
14.23
13.49
12.54
Affineuuant
36.70
18.64
16.11
13.59
12.97
12.14
RTN
44.98
33.63
22.23
16.05
15.40
14.17
GPTQ
Frantar et al.
2022
37.75
18.23
15.94
13.75
12.58
11.98
w4a16
AWQ Lin el al 2023
38.74
18.35
15.70
13.59
12.72
12.06
Omni Quant (Shao al
2023
34.94
17.80
15.52
13.41
12.62
11.95
AffineQuant
34.29
17.55
15.49
13.30
12.54
11.97
RTN
36.50
33.63
22.23
16.05
15.40
14.17
GPTQ Frantar et al.
2022
35.48
17.41
15.42
13.21
12.42
11.89
w4a16g128
AWQ Lin al. 2023
34.95
17.46
15.33
13.28
12.46
11.90
Omni OmniQuant uaan Shao al
2023
34.28
17.40
15.28
13.25
12.46
11.94
AffineQuant
34.00
17.33
15.25
13.27
12.44
11.94
model's performance evaluated the WikiTex12 (Merity et al. 2016), PTB (Marcus al. 1994),
and C4 (Raffel et al. 2020) datasets, For algorithm optimization, we randomly select 128 segments from the WikiText22 training set, each containing 2048 tokens, as the calibration dataset. We leverage the scale of SmoothQuant (Xiao et al. 2023) to initialize the diagonal of the affine transformation matrix. As the affine transtormation is orthogonal the translation operation, we incorporate the optimization of the learnable parameter shift and initialize it using Outlier Suppression+ (Wei al.) 2023). Our optimizer, learning rate, epoch, and learnable clipping of quantization parameters are Nvidia
consistent with OmniQuant (Shao et al. (2023). The optimization process is performed A100 GPU. We conduct a comparative analysis various weight-only quantization methods, in- cluding GPTQ (Frantar al. 2022), AWQ (Lin et al. 2023), and OmniQuant (Shao et al. 2023).
16

---

Published as a conference paper at ICLR 2024
Table 9: Weight-only guantization PPL(() results on the OPT model dataset.
Config
Method
125M
1.3B
2.7B
6.7B
13B
30B
FP16
24.60
14.72
13.16
11.74
11.19
10.69
RTN
5.0e3
7.7e3
3.8e1
5.2c3
2.8e1
6.5e3
GPTQ
Frantar et al.
2022
597.66
60.88
33.83
18.55
16.34
12.89
w2a16g128
AWQ [Ln
2023
168.35
38.38
26.41
16.48
14.73
12.98
Omni Quant (hao
2023
80.10
27.33
21.11
16.67
14.92
13.12
AffineQuant
46.22
23.28
23.10
15.62
14.60
12.93
3.9e3
7.3e3
1.2c5
6.3e3
7.5e3
4.0e3
GPTQ Frantar et al.
2022
133.51
31.31
23.23
16.24
14.48
12.24
w2a16g64
AWQ Lin el al. 2023)
90.19
27.34
20.01
15.20
13.90
12.43
OmniQuant (Shao
2023
64.01
23.71
19.16
15.44
14.16
12.80
AffineQuant
42.43
21.87
17.72
14.86
13.92
12.49
RTN
722.83
6.1e3
1.2e4
5.8e3
3.3e3
1.4e3
GPTQ
Frantar el al.
2022
37.75
19.45
13.75
12.28
11.34
w3al6
AWQ Lin et al 2023
55.73
24.56
154.49
15.84
23.71
55.01
Omni Quant (Shao et a.
2023
32.17
17.10
14.93
12.78
12.13
11.37
AtfineQuant
28.19
16.42
14:27
12.72
12.04
11.21
RTN
40.13
126.47
372.23
32.56
44.12
25.70
GPTQ Frantar et al
2022
30.08
16.47
14.54
12.48
11.58
10.91
w3a16g128
AWQ Lan al
30.39
16.27
14.19
12.30
11.61
10.96
Omni OmniQuant Shao
2023
29.34
16.11
14.15
12.31
11.63
10.98
AffinQuuant
27.53
16.02
13.92
12.21
11.63
10.99
RTN
31.58
24.68
17.61
13.38
12.35
11.90
GPTQ Frantar ct al
2022
27.12
15.37
13.75
12.15
11.36
10.80
w4a16
AWQ LLn et al. 2023
27.64
15.65
13.71
12.04
11.42
10.83
Omni OmniQuant Shao
2023
26.36
15.28
13.58
11.97
11.41
10.80
AffineQuant
25.47
15.18
13.43
11.90
11.36
10.80
RTN
26.79
15.71
13.79
12.31
11.51
10.94
GPTQ Frantar et al.
2022
25.96
15.05
13.10
11.87
11.26
10.74
w4a16g128
AWQ [Lin et al 2023
25.90
15.04
13.39
11.87
11.28
10.75
OmniQuan Shao
2023
25.63
15.03
13.38
11.85
11.29
10.75
AffineQuant
25.26
14.98
13.32
11.84
11.27
10.75
17

---

Published as a conference paper at ICLR 2024
Table 10: Weight-only quantization PPL(() results the LLaMA1&2 model C4 dataset.
Config
Method
1-7B
1-13B
1-30B
2-7B
2-13B
FP16
7.08
6.61
5.98
6.97
6.46
28.26
13.22
28.66
402.35
12.51
GPTQ Frantar et al. 2022
9.49
8.16
7.29
9.81
8.02
w3a16
AWQ
Lin el al. 2023)
13.26
9.13
12.67
23.85
13.07
OmniQuan (Shao
2023
8.19
7.32
6.57
8.65
7.44
AffineQuant
8.03
7.20
6.55
8.57
7.56
RIN
8.62
7.49
6.58
8.40
7.18
GPTQ (Frantar el al.
2022
7.85
7.10
6.47
7.00
w3al6g128
AWQ (Lin et al. 2023
7.92
7.07
6.37
7.84
6.94
OnniQuant (Shao et al
2023
7.34
6.76
6.11
7.35
6.65
AffineQuant
7.75
7.04
6.40
7.83
6.99
RTN
7.93
6.98
6.34
7.71
6.83
GPTQ FFrnnarr et al.
2022
7.43
6.84
6,20
7.37
6.70
w4a16
AWQ Lan at
7.52
6.86
6.17
7.68
6.74
Omni Quant (Shao
2023
7.34
6.76
6.11
7.35
6.65
AffineQuant
7.30
6.75
6.10
7.29
6.64
RTN
7.37
6.69
6.06
7.24
6.58
GPTQ (Frantar ct al
2022
7.21
6.69
6.06
7.12
6.56
w4a16g128
AWQ Lin et al. 2023)
7.21
6.70
6.05
7.13
6.56
Omni OmniQuant Quant (Shao
2023
7.21
6.69
6.06
7.12
6.56
AffineQuant
7.20
6.69
6.05
7.12
6.56
18

---

Published as a conference paper at ICLR 2024
Table 11: Weight-only quantization PPL(() results on the LLaMA1&2 model WikiText2 dataset.
Config
Method
1-7B
1-13B
1-30B
2-7B
2-13B
FP16
5.68
5.09
4.10
5.47
4.88
RTN
1.1c5
6.8c4
2.4c4
3.8e4
5.6e4
GPTQ (Frantar ct al. 2022
2.1c3
5.5c3
499.75
7.7c3
2.1c3
w2al6
OmniQuant (Saa et al.
2023
15.47
13.21
8.71
37.37
17.21
AffineQuant
9.53
7.54
8.35
35.07
12.42
RTN
1.9c3
781.20
68.04
4.2c3
122.08
GPTQ Frantar al. 2022
44.01
15.60
10.92
36.77
28.14
w2al6g128
AWQ Lin al. 2023
2.6e5
2.8e5
2.4e5
2.2e5
1.2c5
OmniQuant Quant (Sao el a.
2023
10.53
8.37
7.77
12.84
9.15
AfaineQuant
13.51
7.22
6.49
10.87
7.64
RTN
188.32
101.87
19.20
431.97
26.22
GPTQ Frantar al. 2022
22.10
10.06
8.54
20.85
22.44
w2al6g64
AWQ Lin 2023
2.5e5
2.7e5
2.3e5
2.1e5
1.2e5
Omni Quant (hao a..
2023
7.62
7.14
10.56
8.14
AffineQuan
8.35
6.98
6.20
9.05
7.11
RTN
25.73
11.39
14.95
539.48
10.68
GPTQ Frantar et
2022
8.06
6.76
5.84
8.37
6.44
w3a16
AWQ
Lin ct al. 2023
11.88
7.45
10.07
24.00
10.45
O OmniQuant Shao et al.
2023
6.49
5.68
4.74
6.58
5.58
AffineQuant
6.30
5.60
4.68
6.55
5.62
RTN
7.01
5,88
4.87
6.66
5.51
GPTQ Frrntar 2022
6.55
5.62
4.80
6.29
5.42
w3a16g128
AWQ (in al 2023
6.46
5.51
4.63
6.24
5.32
OmniQuant Quant Shao ct a.
2023
6.15
5.44
4.56
6.03
5.28
AffineQuant
6.14
5.45
4.59
6.08
5.28
RTN
6.43
5.55
4.57
6,11
5.20
GPTQ Frantar et al.
2022
6.13
5.40
4.48
5.83
5.13
w4a16
AWQ (Li
2023
6.08
5.34
4.39
6.15
5.12
OmniQuant Shao ct a
2023
5.86
5.21
4.25
5.74
5.02
AffineQuant
5.84
5.20
4.23
5.69
5.01
RTN
5.96
5.25
4.23
5.72
4.98
GPTQ Frantar ct al.
2022
5.85
5.20
4.23
5.61
4.98
62288
AWQ Lin et al. 2023
5.81
5.20
4.21
5.62
4.97
OmniQuant Shão el al
2023
5.77
5.17
4.19
5.58
4.95
AffineQuant
5.77
5.17
4.19
5.58
4.95
19

---

Published as a conference paper at ICLR 2024
500
1.0g
2000
2510
2000
3500
4000
500
100g
150g
200g
2500
2000
2500
2000
Stov
1000
7.0g
2000
2300
300
4000
300
Llama2-7b
w3a16g128
1300
2000
E3yer32-Epoch220
25300
1200
1008
3000
30001
4000
starh
4000
1000
2000
3000
4000.
sadd
40000
7000
2000
0
Llama2-13b
2000
2000
W2a16
3000
3000
Layer10-Epoch40
5000
500
1000
1:00p
2009
2010
7099
2000
500
1000
1.00
2009
2209
5000
500
1000
1:00g
2009
200
Llama-7b
11000
w4a16
30000
Layer20-Epoch20
20070
1000
3000
2006
1004
50
1000
2009
100
1000
200
4000
1000
2000
5000
1000
1020
11000
Llama-13b
2000
w2a16g128
Layer35-Epoch40
10000
1000
300
2008
3000
1008
2008
2000
100
2008
1080
1080
OPT-13b
2000
2000
w3a16
Layer15-Epoch20
3009
3009
5000
200
.000
3000
10d
2000
3000
5000
2007
30t co0
Nay Var fo (ap
2000
2000
OPT-125M
w2a16g128
LayerO-Epoch20
3001
760
760
27g
400)
100
100
200
300
100
cou
160
208
300
Say (op
2000
2000
OPT-125M
w2a16g128
And
Layerô-Epoch40
Sed
ood
600
(a) fel _ffine_MMatrix A
(b) out Affine Lit_AAfi____tttti Matrix_ A
(c) gkv Affine Matrix A
Figure 7: Affine transformation matrix for different quantization configurations, different layers, and different training epochs for OPT and LLaMA1&2.
20


Reviews:
[{"review_id": "ab2Ws1ZLzu", "reviewer": "Reviewer_gGYu", "summary": "The paper introduces a method for post-training quantization of Large Language Models (LLMs), specifically focusing on the linear layers. This quantization is uniform, employing fixed bit widths across the layers. The proposed approach involves learning an affine transformation (which is more general than single scaling) of weights before quantization by minimizing the MSE loss. The authors aim to ensure that this matrix remains non-singular, employing the Levy-Desplanques theorem and their proposed Gradual Mask (GM) method. Experiments are conducted on various model variants, including OPT and LLaMA, where the proposed algorithm is comparable or outperforms previous works, especially in low bit-width configurations.\n\n=================================\n\nUpdate: \nThe authors addressed my concerns, and I have increased the given rating correspondingly.", "questions": "1. The method employs non-differentiable components in its optimization approach. It would be interesting to understand how the authors address this challenge in practice.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper expands on pre-quantization transformations by introducing invertible affine transformations, which are more versatile than previous methods that rely on simple scaling.\n2. The proposed post-training quantization method for LLMs addresses a timely and pressing need due to the growing popularity of LLMs with their large model sizes.\n3. The method's ability to avoid model retraining is a substantial practical advantage since retraining LLMs can be computationally expensive and time-consuming.\n4. The weight transformation before quantization is not limited to LLMs; it can be applied to linear layers in different models as well.\n5. Inference speed is unaffected, as the additional matrices can be fused with the weight matrices.\n6. The paper provides extensive experimental results, including comparisons across different datasets and NLP models.", "weaknesses": "1. A significant weakness of this paper is the lack of clarity in explaining the implementation of the core concept, which involves the use of strictly diagonal matrices and the proposed Gradual Mask (GM). Figure 2 suggests that the GM matrix is element-wise multiplied by the matrix A, but the description implies a different interpretation, where it functions as a learning rate for each element in A. This discrepancy needs further clarification to provide a complete understanding of the method.\n\n2. The hyper-parameters $b$ (bit-width) and $\\alpha$ (stability factor) may introduce significant computational overhead in the pursuit of determining the optimal trade-off between model size and accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V4dcRqItSy", "reviewer": "Reviewer_5dxF", "summary": "Propose affine transformation in PTQ (in line with \"equivalent transformations\" line of work) based on specialized optimization approach involving a “gradual masking” to ensure a viable affine matrix is trained.", "questions": "- In terms of a computation vs quantization performance tradeoff, what do the authors think about just fine-tuning the model after some standard quantization method? I understand how the gradual mask training approach makes this cheaper than fine-tuning the full model, but it still appears expensive enough that it’s difficult to get results on the largest OPT/Llama models. I think this is an interesting method, more of a point of discussion.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- A more general approach than several proceedings works in the space of “equivalent transformations”, and a nice explanation in Section 3.1\n- strong empirical results", "weaknesses": "- I imagine there is an increased computational cost to the proposed method. Is that the reason why results on larger OPT (ie 66B) or Llama (70B) models were not reported? My understanding is that this is the main tradeoff: a more powerful quantization method, but at an increased computational cost?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WHx30xcIpr", "reviewer": "Reviewer_E6YB", "summary": "The paper introduces \"AffineQuant\", a method to optimize Large-scale Language Models (LLMs) using equivalent Affine transformations in Post-Training Quantization (PTQ). Traditional PTQ techniques often resulted in significant errors, especially in low-bit configurations. AffineQuant expands the optimization scope, reducing these errors. Using a unique gradual mask optimization method aligned with the Levy-Desplanques theorem, invertibility of transformations is ensured. The results show AffineQuant outperforms existing methods, especially in low-bit configurations, making it a promising tool for model compression and deployment.", "questions": "included in weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written and successfully exhibit their features.\n- This paper extends current PTQ papers to introduce affine transformation to MRE (minimum reconstruction error) methods. It could be reasonable in the line of quantization papers' history.", "weaknesses": "- The paper appears to overlook contemporary quantization methodologies like FlexRound. Notably, FlexRound seems to be a successor in the lineage of MRE-PTQ techniques, such as BrecQ and QDrop, designed for LLM compression. A comparative analysis with these methods would accentuate the novelty and efficacy of the proposed technique. Reference for consideration: https://arxiv.org/abs/2306.00317.\n\n- Regarding Figure 3, my understanding is that the loss of a partial layer during MRE PTQ might not directly correlate with improved model performance, especially in the context of generative AI models.\n\n- I have a few observations concerning Section 4, which details experimental results:\n   1) A significant concern is the sole reliance on PPL scores as a performance metric. While it corresponds to dataset loss, recent LLM compression research suggests that PPL may not fully capture the generative capabilities post-quantization. It might be worthwhile to consider metrics like common-sense reasoning or evaluations using the MMLU dataset.\n   2) Additionally, an increase in PPL scores by over 10 points indicates a significant degradation in the generation capabilities of the quantized model. Thus, contrasting the performance of such models, like the W2A16 results mentioned in the abstract, might not provide meaningful insights. A closer look at the actual generation outputs from these severely quantized models could reveal issues like non-coherent results or repeated verbiage.\nLastly, larger models could yield different outcomes, as they might exhibit distinct patterns of outliers or weight distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B1guzAPsxD", "reviewer": "Reviewer_MkcQ", "summary": "This work develops a quantization method that uses an affine matrix transformation to quantize the weights of a transformer. By doing so, the objective function has more parameters to be optimized and can achieve lower quantization error when the weight matrix is compressed to a low-bit representation (2-bit). \n\nRecommendation: The presentation and results are currently misleading. I might miss something, but currently, the method has no practical benefit. If the authors can explain to me what I am missing, I am happy to raise my score significantly (2-4 points). I will also raise my score by 2 points if the presentation of 2-bit results is dropped in favor of an analysis of information density (scaling curves that show Pareto fronts in terms of performance per bit of total memory footprint).", "questions": "- How does the process work at inference time? If you optimize with the affine matrix during quantization, you surely need it during inference time.\n- How fast is inference for this method compared to baselines?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The results from affine quantization are strong compared to baselines.", "weaknesses": "- The paper misleadingly highlights 2-bit quantization when the same paper shows higher information density for 4-bit quantization (4-bit + 7B params > 2-bit + 13B params, etc.). This is a mistake that is propagated in the literature. I will not accept work that presents quantization results like this because it is very misleading.\n- It is unclear what the benefit of the method is. To use the quantization in practice, the inverse affine transformation matrix is needed to convert the input tensor. However, by having to store this matrix, the size of the model is doubled, which negates FLOPs and memory benefits. Am I missing something?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)