Adaptive Quantization Error Reconstruction for LLMs with Mixed Precision
Authors: Lin Ou
Jinpeng Xia
Yuewei Zhang
Chuzhan Hao
Hao Henry Wang
Venue: COLM (2024)
Keywords: Quantization, Large Language Models, Mixed Precision, Error Reconstruction, Low-Rank Decomposition
TL;DR: We propose an adaptive mixed precision and low-rank approximation quantization error reconstruction, which allows us to automatically search for optimal parameter settings in a discrete space.

Abstract: Large language models (LLMs) has demonstrated superior performance on various downstream tasks. However, their practical applications are hindered by their immense memory and computation requirements. Although recent post-training quantization methods can effectively reduce memory usage and improve computational efficiency, they often overlook the varying sensitivity of different layer weights to bit precision. Additionally, the previous methods suffer from significant accuracy loss under low-bit quantization (2-3 bits). To address these limitations, we propose Adaptive Mixed Precision and Low-Rank Quantization Error Reconstruction for LLMs (AMLQ), which achieves state-of-the-art performance under the approximate average bit precision overall. Furthermore, we introduce the low-rank decomposition to reconstruct quantization error based on the output features. Experimental results demonstrate that this method can be effectively combined with various quantization techniques and bring considerable performance gains. Our approach comprehensively considers model performance and inference efficiency, offering more than 3$\times$ speedup over the FP16 execution.

Published as a conference paper at COLM 2024
Adaptive Quantization Error Reconstruction for LLMs with Mixed Precision
Lin Ou, Jinpeng Xia; Yuewei Zhang, Chuzhan Hao Hao Wang"
Alibaba Cloud, Alibaba Group {oulin. ol, qih.a...iip. xjp, zyw, cashenry@1 com
Abstract
Large language models (LLMs) has demonstrated superior performance on various downstream tasks. However, their practical applications are hindered by their immense memory and computation requirements. Al- though recent post-training quantization methods can effectively reduce memory usage and improve computational efficiency, they often overlook the varying sensitivity of different layer weights to bit precision. Addi- tionally, the previous methods suffer from significant accuracy loss under low-bit quantization (2-3 bits). To address these limitations, we propose Adaptive Mixed Precision and Low-Rand Quantization Error Reconstruc- tion for LLMs (AMLQ), which achieves sfate-of-the-art performance under the approximate average bit precision overall. Furthermore, we introduce the low-rank decomposition to reconstruct quantization error based on the output features. Experimental results demonstrate that this method can be effectively combined with various quantization techniques and bring considerable performance gains. Our approach comprehensively consid- ers model performance and inference efficiency, offering more than 3x speedup over the FP16 execution.
1 Introduction
Large language models (LLMs) such as GPT-4 (Bubeck al., 2023) and Qwen (Bai al., 2023), have shown excellent performance on various natural language processing (NLP) tasks (Brown al., 2020; Touvron al., 2023a; Zhao al., 2023; Feng et al., 2024). Meanwhile, experimental evidence indicates that emergent capabilities only manifest when the model scale is sufficiently large (typically exceeding 10B parameters) (Wei et al., 2022a). This also results in considerable computational and memory requirements, posing significant challenges for the practical applications in real-world scenarios. Therefore, quantization techniques capable of mitigating computational and memory pressures exhibit excellent prospects in the era of LLMs (Frantar et al., 2022; Dettmers et al., 2023; Xiao et al., 2023).
Quantization is often categorized into two main approaches: post-training quantization (PTQ) and quantization-awal training (QAT). While QAT tends to maintain better experi- mental accuracy than PTQ, its high training cost renders it less practical. Therefore, PTQQ becomes an attractive research task for reducing computational and memory cost (Dettmers et al., 2023; Lee al., 2023a; Shao al., 2023; Lin et al., 2024). PTQ technique that quantizes a pre-trained LLM directly, without additional training. The quantization er- ror propagates and accumulates through the LLMs, leading to substantial task accuracy degradation. To maintain the original model accuracy, Frantar et al. (2022) employs second- order information to iteratively round grouped weights and correct the quantization error. Additionally, due to the presence of magnitude outliers in the model layer weights, the quantization process is severely affected (Wei et al., 2022b; Xiao et al., 2023). Thus, an intuitive approach involves identifying and isolating outliers within the weight distribution.
"The first two authors contributed equally
I *Corresponding author

---

Published as a conference paper at COLM 2024
These outliers can then be subjected to mixed-precision quantization weight scaling. Specifically, certain salient weights can be retained in FP16 (half floating point) at finer granularity while quantizing the rest to low-bit precision (Dettmers et al., 2023; Tang et al., 2024). Alternatively, protecting the salient weights can be implemented by learning fine- grained weight scaling (Lee et al., 2023b; Luo et al., 2023; Lin et al., 2024; Zhang et al., 2024). Chee et al. (2024); Egiazarian et al. (2024) introduce codebook mechanism to achieve low-bit quantization (2 to 3 bits per parameter).
Normalized emror sensibiity
AMLO
AMQ
0.8
0.6
12
Rank
0.6
SVD
L'OER
0.4
AMLO
k_proj
0.2
gate_proj
dawn_proj
0.0
1
16
21 Layer
26
36
255 25 27 29
35 57
39
1
6
11
16
21 Layer
26
31
36
Average procision
Figure 1: Motivation of our proposed AMLQ. (a) The varying quantization error sensitivity of different layer weights in LLMs. We observe that the deep layers are more sensitive than others in quantization bit precision. (b) The introduction of the low-rank approximation quantization error reconstruction can mitigate the significant degradation of model perfor- mance at low-bit precision. (c) The output activation errors decomposed by our approach are low-rank, while the layer weight errors are not low-rank.
However, empirical evidence suggests that different layers in a neural network have the varying importance redundancy (Han et al., 2015; Liu et al., 2019). Similarly, different layer weights in LLMs exhibit varying sensitivity to bit precision, as illustrated in Figure 1. Therefore, global quantization strategies often do not achieve optimal experimental results. We expect to get the optimal bit precision combination for each layer weight. In addition, the previous methods Frantar al. (2022); Dettmers et al. (2023); Lin et al. (2024) suffer from significant accuracy loss under low-bit quantization (Figure 1). Although using the codebook can have good performance (Egiazarian et al., 2024), is relatively slow in inference. Zhang et al. (2024) also leverages low-rank approximation to compensate weight errors low-bit quantization to some extent. However, Yu & Wu (2023) proves that the weights of transformer-based models are surprisingly not low-rank, which makes it less effective to leverage low-rank approximation to correct quantization errors base on layer weights. Simultaneously, this low-rank error reconstruction method is also sensitive to different layer weights.
In this paper, we address these limitations and propose adaptive mixed precision and low-rank quantization error reconstruction for LLMs, dubbed AMLQ. The core idea of our method to adaptively select bit-precision and the rank of the error correction matrix for different layer weights. First, we propose an activation-induced adaptive search method for selecting optimal bit precision for different layer weights. Through this approach, we achieve relatively optimal parameter setting under predefined average bit precision. Second, to further correct quantization errors, we also introduce an adaptive low-rank quantization tintiiiaiinnn error reconstruction based activation. Compared to low-rank approximation applied directly to model weights, this method demonstrates superior performance. Overall, we keep higher-bit precision and higher-rank correction matrices for more sensitive layers, while adopting more aggressive compression strategies for less sensitive ones. This error reconstruction method also has the flexibility to adapt to other advanced quantization techniques, providing plug-and-play capabilities. Our contributions are as follows:
We propose an adaptive mixed precision and low-rank approximation quantization error reconstruction, which allows us to automatically search for optimal parameter settings in a discrete space.
Experimental results demonstrate that our approach achieves state-of-the-art perfor- mance at the approximate average bit precision. In the case of low-bit quantization, our
2

---

Published as a conference paper at COLM 2024
approach also reaches relatively fast inference speed. Simultaneously, proposed low- rank quantization error reconstruction can be plugged into other advanced quantization techniques to optimize their performance.
2
Related work
Post-Training Quantization of LLMs. The PTQ is a very practical technique for the ap- plication of LLMs. It remains challenging due to the presence of magnitude outliers in model weights and activations. Exiting methods can be broadly categorized into two fields: weight-only and weight-activation quantizations. Weight-only quantization focuses on converting weights to low-bit precision. Frantar et al. (2022) implements block-wise itera- tive quantization error correction using grouped weights. Dettmers et al. (2023); Lee et al. (2023a); Lin et al. (2024) emphasize that small fraction of salient weights are much more important for LLMs' performance compared to others. Therefore, Dettmers et al. (2023) and Lee et al. (2023a) employ mixed-precision quantization to safeguard vital weights, while Lin et al. (2024) opts for channel-wise scaling to protect salient weights induced by activations. Weight-activation quantization compresses both weights and activations, which can provide computational improvements, but execute with large amounts of accuracy loss relative to their uncompressed counterparts (Yuan al., 2023; Shao et al., 2023). this paper, we focus on the weight-only quantization because we expect to keep the model performance as much as possible after quantizing.
Quantization. Mixed-precision quantization refers to the process of assigning appropriate quantization bit-widths the weights and activations of each layer in model. The goal is to achieve an optimal balance between accuracy and hardware metrics in the quantized model. Dong et al. (2019) utilizes second-order information to assess the quantization sensitivity of each layer in the model, thus allocating the appropriate bit-width. Wang et al. (2019) employs reinforcement learning techniques to search for the quantization bit-widths of different layers, which incorporates feedback from hardware simulators and achieves a hardware-aware mixxed-preciiioe strategy. Additionally, there are some methods to implement this using training -bsed algorithms (Cai & Vasconcelos, 2020; Yang & Jin, 2021). However, the above methods are not yet easily adapted to quantization for LLMs.
Low-Rank Quantization Error Reconstruction. LoRA (Hu et al., 2022) leverages the concept of low-rank matrices to make the training process of LLMs extremely efficient and fast, which can significantly reduce the trainable parameters and GPU memory. Dettmers al. (2024); Li et al. (2024) introduce quantification techniques into LoRA-based tuning methods, which are not post-training quantization method and fuse the low-rank matrices back to original high-bit weights. Zhang et al. (2024) leverages the separated low-rank matrices to correct the quantization weight errors. However, it ignores the fact that the quantization weight error matrix is high-rank, even after scaling it.
Methodology
Our method requires a small set of calibration data to measure the sensitivity of different layers and the importance of intra-layer channels, and then adaptively selects the bit precision of layer weights and the rank of error reconstruction matrices. Our method works on all linear layers in LLMs except the weights of embedding and LM head. According to the weight of the i-th layer W E "R" the input activation value E RR and the quantized weight W. the final quantitative target can be denoted as:
min |WX - W2
(1)
3.1 Mixed-precision quantization
As mentioned above, the quantization sensitivities vary among layers and intra-layer channels. Therefore, to achieve the predefined average bit precision (also known as BPW, bit per weight) for the entire model, different quantization bit-widths be used for different
3

---

Published as a conference paper at COLM 2024
layers channels during quantization. The sensitivity 5 is measured by calculating the output loss after quantization, which is computed as follows:
ô |WX - W1,x1////wxI1
(2)
where W E R is the weight of the i-th layer, X E R is the activation of the input value, Wg is the quantized weight. || ||2 represents the function of f--norm. For the weight of each layer, we will first define our quantization bit-precision candidate set B E {2, 4, 5, 6,8, 16), and then preset different proportions of mixed-precision combinations from the set, with a total of possibilities. All combinations form a discrete search space Q E {q1.92. , ) For each combination, we can calculate the quantization loss (1(9;) and the corresponding total bit size b, (4;)) for the i-th layer weights after quantization of the j-th mixed-preeeee -rexciipeeiiion combination. Finally, our goal is to minimize the objective function as follows:
min L = Σ 4(g)) 9
(3)
s.1. by (q:) < Piar
(4)
where denotes the number of layers of the LLM. Parget bit is the predefined upper bound on the number of bit parameters for the all model weights. This optimization process can be viewed as an integer programming problem, and it will be very time-consuming to find the exact global optimal solution from the whole search space Q. The sensitivity of each layer in the model is independent of the quantization bit precision of the other layers. Therefore, we use a greedy strategy to approximate the solution. Although our greedy strategy may not always find the optimal configuration, it can greatly shorten the search time. In section 4, the experimental results also demonstrate that the mixed precision combination found through this greedy strategy can achieve ofate-of-the-art performance.
Specifically, we obtain the quantized weight W, in the following process. We comprehen- sively refer to the current superior GPTQ (Frantar et al., 2022) and AWQ (Lin et al., 2024) methods. First, AWQ proposes to use the magnitude of the input activation value to help scale the salient channel weights thereby protecting the salient weights during quantization. We use this method to calculate the average amplitude of activation S = mean (ortun_coo((0 (X), and set a searchable hyperparameter a control the intensity of the amplitude. Hence, the scaled weights are W, 5% W. To ensure that the output activation Y is not affected, the input activation X also needs to be scaled to X, (St)-1 X. The complete output activation represented as follows:
== WX W,X, (W S (($)) X)
5
We then calculate the inverse Hessian matrix H of the scaled input activation value (refer to GPTQ). Due to our scaling of weights and activations, we do not need to compute XX" repeatedly for reducing the amount of computation. The complete computation process is shown in Equation 6.
H (2x,X. +AI)-1 = ( (aang((())))))xddddd((3(3)333113 + A1)-1
(6)
where diag(-) denotes turning the vector into a diagonal array. The H will be used to evaluate the importance of each channel within the weight. Similarly, we also group channels and then assess the importance of weights within each group. For each group of weights, we reorganize the order, placing the more critical groups at the forefront. 1 the more important groups with higher bit precision in the predefined search space. Finally, we can refer to the pseudocode in Algorithm to calculate the quantized weight Wg-
Furthermore, we introduce the idea of mixed precision, quantizing
4

---

Published as a conference paper at COLM 2024
3.2 Quantization error reconstruction based on low-rank decomposition
For the quantized Wg, there a quantization error AW with the original FP16 weight W, which can be obtained by:
AW WW W
The rank of AW is relatively high, as illustrated in Figure 1. There are significant errors to decompose it into low-rank matrix using Singular Value Decomposition (SVD) directly, LQER (Zhang et al., 2024) introduces the scaling factor to reduce the difficulty of decom- position and achieves good results. Moreover, low-rank but the output activation value is low-rank. Therefore, we directly decompose the output activation error AY E RR" using Principal Component Analysis (PCA).
observe that the layer weights are not
=X Wy AWX
(8)
We regard AY as instantiations of the random eigenvector y (each in RR",, and then compute the covariance matrix between them as follows:
Cov(y) = Elyy"] Ely]Ely
(9)
where E[-] denotes the function of calculating expectation. Since Cov(y) is positive semi- definite matrix, its eigenvalue decomposition (i.e., PCA) is
Cov(y) USUT
(10)
We extract the first columns of U Ru and get Ug "R Mui ~ I, hence
Fly) UpU (y Fly))
(11)
y 2 uuu y Blyl + Blyl
(12)
Therefore, the quantization error AY can be represented as
AY A u, - +
(13)
Let Ut and ut AW form two low-rank matrices E R and E Rise respectively, E[y] - Ug of Bly] is superimposed on the bias b of the original model. Therefore, the output feature Y of linear layer consists of the following parts:
W, + BAX+b
(14)
The layer weight W E R is usually a large matrix. For instance, in the case of Llama2- 70B (Touvron et al., 2023a), the minimum dimensions for {m,c} are {8192, 8192). Addi- tionally, the newly added two parameters {A, B} will increase the average bit precision of k))m1 When DFC
64, will only increase the average bit precision by less than 0.2. For the unbiased linear layer, our method adds ı additional parameters, which are only about 0.02% of the W parameters, almost negligible.
As mentioned above, the quantization sensitivity of each layer is different. It can also be observed in Figure that the rank of the quantization error of each layer is also varying. Therefore, adopt the concept of adaptive search to determine the rank in the quantization error reconstruction process. The rank in low-rank decomposition is predefined in the set (0,32, 64, 96, 128). Additionally, combine it with adaptive mixed- mixee-precision quantization. The whole optimization process is
min
L= Σ k(
(15)
s.t. Σ bx(g < Piarget.bil
(16)
where 1((g, (. denote the quantization error and the total number bit parameters at the i-th layer choosing k; as rank and as quantization bit precision. Piarget.bit can be obtained by multiplying the number of model parameters by the predefined average bit precision. We can get sub-optimal solution in short time using the same greedy strategy.

---

Published as a conference paper at COLM 2024
Algorithm 1 AMLQ
1: procedure QUANTONELAYER(WW, Groupsize, Qparams, Rank, Grida)
2:
5x average X, dim 1)
3:
for a,qj,TK in Ap(Crrida.Qarrass.RRa do
4:
Wg = W * St.
5:
X₂ = (5) - x
6:
H = Cholesky (2X,X3 + )
7:
W &idx reorder Wg, H
8:
Wg = reorder(quant W Groupsize, qi) 8itx)
9:
AY = (W - X
10:
B, A, = decompose(AY,,
11:
Quantloss | - (W,X + BAX + b)1 [
12:
Losslist. Losslist.appendd (aappeennuunnnosssss.s (j)
13:
end for
14:
Return Losslist
15:
end procedure
16:
17: procedure QUANTMODELLLAyers,, Gropppi Qparams, Rank, Ptarget.bit. Grida)
18:
for W, X in Layers do
19:
Lossiisteeeet(Quuan@holllayyyyy X, Groupsize, Qparams, Rank, Grdz))
20:
end for
21: BestModel greedy-search(Losslle Ptarget bit)
22:
end procedure
3.3 Inference performance
In the inference stage, we design a special CUDA operator for mixed-precision weights to perform hegh-performance inverse quantization and calculation processes. We also provide pseudocode Algorithm 1. By reading the low-bit weights in one go using the INT4 vector type, the calculation intensity is increased, the memory access bottleneck is overcome, and the generation speed of the model decoding phase is improved. Compared to the FP16 model, the speedup can be more than 3 times. For the introduced low-rank error reconstruction module, we refer to Punica (Chen et al., 2023) to implement the bgmv operator, and the overall generation speed only reduced by 5%.
Experiments
4.1 Experimental Settings
Quantization. We experiment with weight-only quantization. The default settings are INT4/INT3/INTI IINT2 per-channel weight quantization, with an average bit precision (or BPW) of about 2.3 4.3. Group-wise weight quantization represented by 'g', whose sizes are set from {32, 64, 128) in our adaptive mixed precision method. We set the group size to 128 in other methods. Our grid size of is set to 20 following Lin et al. (2024). The search space for the rank of the low-rank approximation matrices is set from {0, 32, 64, 96, 128). When the rank of a layer are not added. Following Shao et al. (2023), our calibration dataset is sampled from the WikiText2 (Merity et al., 2017) dataset, which contains 128 samples of 2048 tokens.
O, this means that matrices of low-rank decomposition
Baselines. We conduct experiments on the currently most influential open-source LLMs of different scales, such as Llama (Touvron al., 2023a), Llama-2 (Touvron et al., 2023b), and Qwen (Bai et al., 2023). We compare our approach with FP16 model, GPTQ (Frantar et al., 2022), AWQ (Lin et al., 2024), OmniQuant (Shao et al., 2023), QUIP# (Tseng et al., 2024), and LQER (Zhang et al., 2024) at the different BPW settings. OmniQuant and QuiP# are further optimized low-bit precision quantization and QuiP# is the most effective quantization method at present. However, their actual inference efficiency is still low. Compared to

---

Published as a conference paper at COLM 2024
AMLQ, proposed AMQ denotes the approach that uses only mixed precision and does not introduce low-rank enror reconstruction.
Evaluation. Following previous methods (Frantar et al., 2022; Xiao et al., 2023; Lin et al., 2024; Zhang et al., 2024), we profiled the quantized models on language modeling tasks and downstream tasks. Specifically, 2017) and the accuracy on ARC (easy) (Clark et al., 2018), ARC (challenge) (Clark et al., 2018), PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2021), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), CEval (Huang et al., 2023), GSM8K (Cobbe et al., 2021), and HumanEval (Chen et al., 2021).
report the perplexity (PPL) on WikiText-2 (Merity al.,
4.2 Main Results
We firstly compare the performance of the quantization models by perplexity on the WikiText-2 dataset. The results of llama-7b, llama-13b, Ilama2-7b, Ilama2-13b, and llama2- 70b quantized by AWQ, OmniQuant, Quip#, L'QER, AMQ, and AMLQ respectively, are shown in Table 1. We also report the final average bit precision w for each method. We can observe that the low-rank quantization error reconstruction module adds about 0.2 BPW at different bit precision. At almost all model sizes, AMLQ achieves better results than AWQ and OmniQuant. AWQ and OmniQuant deal with outliers heuristically, while LQER uses an error reconstruction method. We observe that AMQ shows a significant accuracy degradation at 2-bit precision, but with the introduction of the error reconstruction, AMLQ outperforms OmniQuant. Quip# achieves the best quantization performance by codebook. However, its inference efficiency is slow and it is difficult to be applied in practical applica- tions. In contrast, our proposed AMLQ, empowered by the CUDA operator, can achieve about times the inference speed of Quip#.
LLaMA
LLaMA-2
Avg. PPL (L)
Avg. w bit
Method
Q Config
7b
13b
7b
13b
70b
FP16
5.68
5.09
5.47
4.88
3.32
16
AWQ
INT4, g128
5.81
5.20
5.62
4.97
0.12
4.1
OmniQuant
INT4, g128
5.77
5.17
5.58
4.95
3.40
0.08
4.1
QuiP#
INT4
5.76
5.17
5.56
4.95
3.38
0.07
4.0
LQER
INT4, g128
5,89
5.20
5.58
4.96
0.12
4.3
AMQ
Dynamic
5.79
5.18
5.57
4.96
3.40
0.09
4.1
AMLQ
Dynamic
5.77
5.17
5.56
4.95
3.39
0.08
4.3
AWQ
INT3, g128
6.46
5.51
6.24
5.32
0.60
3.1
OmniQuant
INT3, g128
6.15
5.44
6.03
5.28
3.78
0.44
3.1
QuiP#
INT3
5.98
5.31
5.79
5.10
3.56
0.26
3,0
AMQ
Dynamic
6.13
5.40
6.01
5.20
3.66
0.38
3.1
AMLQ
Dynamic
6.04
5.38
5.82
5.13
3.64
0.33
3.3
AWQ
INT2, g128
2.6e5
2.8e5
2.2e5
1.2e5
2.2e5
2.1
OmniQuant
INT2, g128
9.72
7.93
11.06
8.26
6.55
3.81
2.1
QuiP#
INT2
6.86
5.97
6.66
5.74
4.16
0.99
2.0
AMQ
Dynamic
15.19
8.73
15.03
8.68
8.53
6.34
2.3
AMLQ
Dynamic 7.81
6.49
7.68
6.34
4.64
1.70
2.5
Table A comparison of perplexity ({) on WikiTText-2. The highest value per column group is in bold and the second highest value is underlined. Partial results are from their original papers. AMLQ outperforms AWQ, OmniQuant, LQER, and AMQ. Most methods suffer significant performance degradation at low-bit quantization, but our approach is on par with QUIP# which is specifically optimized for the low-bit setting, while providing substantially higher inference efficiency.
7

---

Published as a conference paper at COLM 2024
Size
Method
WinoGrande
PiQA
HellaSwag
ArcE
ArcC
Avg. accuracy (1)
13B
FP16
72.14
79.16
60.06
79.46
48.38
67.84
13B
Quip#
72.61
79.05
59.47
78.62
46.59
67.10
13B
GPTQ
72.14
78.24
59.50
78.54
47.10
67.10
13B
AWQ
72.69
78.67
60.12
78.91
47.10
67.49
13B
AMLQ
72.30
79.05
59.92
79.25
47.35
67.57
70B
FP16
78.06
82.15
64.77
82.74
54.44
72.43
70B
Quip#
77.74
82.43
64.45
82.28
54.52
72.28
70B
GPTQ
77.51
81.72
64.17
82.70
53.69
71.95
70B
AWQ
78.22
81.83
64.59
82.58
54.12
72.18
70B
AMLQ
78.22
81.99
64.52
83.08
54.39
72.44
Table 2: A comparison of five downstream task % accuracy (f) with INT4 quantization LLaMA-2 models. Bold text indicates the best results. Compared with the strong baselines, AMLQ achieves the best average accuracy.
Size
Method
MMLU
CEval (val)
GSMSK
HumanEval
Avg. accuracy (t)
14B
GPTQ
62.46
69.43
61.41
40.24
58.38
14B
AWQ
64.56
69.09
63.22
52.43
62.32
14B
AMLQ
64.21
70.49
64.21
53.61
63.13
72B
GPTQ
72.82
79.01
75,95
59.87
71.91
72B
AWQ
73.23
78.74
75.36
62.13
72.38
72B
AMLQ
74.00
78.5
77.33
62.19
73.00
Table comparison of downstream task % accuracy (t) with INT4 quantization on Qwen-Chat. AMLQ also achieves the best accuracy among all Qwen models.
Table shows the performance of AMLQ, GPTQ, AWQ and Quip# on the zero-shot down- stream tasks, which are evaluated using the Im-eoul-harness! We can observe that AMLQ achieves the most favorable results across all five tasks. The comparative analysis between AMLQ, GPTQ ans AWQ on the Qwen model is further illustrated in Table 3. The open- source qwen--gt peveen-plq--ntte is employed as a benchmark. AMLQ also shows significant accuracy improvement over GPTQ.
4.3 Ablation Studies
4.3.1
Adaptation of the low-rank quantization error reconstruction.
Our proposed error reconstruction based on low-rank quantization stands as orthogonal to the existing quantization techniques, illustrating its potential as integrable enhancement module for various methods. Table 4 delineates the Aperplexity measurements of the model the WikiText-2 dataset low bits quantization(ITT233 The values of Aperplexity represent the relative reduction in perplexity. The low-rank error reconstruction method marginally enhances all methods at 3-bit quantization. Moreover, significantly amelio- rates the performance of OmniQuant at 2-bit quantization, and even achieves modest improvement on the Quip# method. demonstrates the effectiveness the low-rank error reconstruction approach in acting as a supplementary plug-in module and augmenting the capabilities of the underlying methods.
8

---

Published as a conference paper at COLM 2024
LLaMA
LLaMA-2
Method
Q Config
Avg. bit
7b
13b
7b
13b
70b
FP16
5.68
5.09
5.47
4.88
3.32
16
QuiP#+
INT3
0.01
0.02
0.00
0.02
0.00
3.2
AWQ+
INT3, g128
0.10
0.04
0.23
0.04
3.3
OmniQuant+
INT3, g128
0.01
0.01
0.02
0.02
0.03
3.5
QuiP#+
INT2
0.02
0.04
0.01
0.02
0.02
2.2
OmniQuant+
INT2, g128
0.17
0.27
0.34
0.26
0.14
2.5
Table 4: Our proposed low-rank error reconstruction is orthogonal to other advanced quan- tization techniques. + denotes the introduction of our low-rank error reconstruction based the original methods. It further closes the performance gap under low-bit quantization (INT2/3) when combined with AWQ and OmniQuant. Results are WikiText-2 AaaTText-AApeppexxiitt Aperplexity of LLaMA models.
4.3.2 Impact of the low-rank error quantization reconstruction
We can achieve almost arbitrary bit quantization due to our mixed-precision strategy. Table reports the perplexity of the Ilama2-7B model at different BPW. Our model performance shows a significant degradation below 3-bit precision without the use of low-rank error reconstruction. By introducing the low-rank error reconstruction method, the loss of the model performance can be effectively reduced at low-bit precision. When the BPW is lower, the effect of the error reconstruction method is more significant.
AMQ
AMLQ
APPL
ABPW
15.03
7.68
7.35
2.3
10.46
6.92
3.54
2.5-+2.7
9.48
6.27
3.21
2.7-2.9
6.42
5.98
0.44
2.9-+3.1
6.01
5.82
0.19
3.1-333
5.86
5.73
0.13
3.3-+3.5
5.79
5.66
0.13
3.5+3.77
5.7
5.61
0.09
3.7+3.9
5.66
5.57
0.09
3.9-+411
5.57
5.56
0.01
4.1-43.3
Table 5: Perplexity on WikiText-2 aross a range of average bit precision settings. The APPL represents the gains from our proposed low-rank eтror reconstruction. The ABPW column denotes the changes in BPW after the introduction of low-rank error reconstruction. We can observe that this method achieves significant gains under the low-bit quantization precision setting.
4.4 Quantitative Analysis
Given these overall performance improvements, we further analyze whether performance improvements are reflected in the selection of better bit precision and rank in each layer (as shown in Figure 2). We use the q_proj layer in the llama2-13b model as an example, which demonstrates the final selection of our proposed adaptive mixed precision quantization and low-rank error reconstruction in different blocks. Similar to the model sensitivity analysis, the higher average bit precision is used to quantize the more sensitive layers in the middle part, thus reducing the quantization error. We also observe that the ultra-low ranks are chosen to reconstruct the quantization error within those blocks with the highest precision such 28 to 36, and even some layers choose not to reconstruct the error. The reason for

---

Published as a conference paper at COLM 2024
this phenomenon comes from the presence of low quantization errors after the high-bit precision quantization. However, it will lead us to prefer the extreme rank (0 or 128) due to the effect of our search strategy.
4.00
q_proj
128
3.75
Average bit precision
96
3.50
3.25
Rank
3.00
2.75
32
2.50
2.25
0
11
31
11
26
31
Layer
Layer
Figure Quantitative analysis of AMLQ's selection of average bit precision and the rank of correction matrices in different layers.
Conclusions
In this work, propose Adaptive Mixed Precision and Low-Rank Quantization Error Reconstruction (AMLQ), an efficient quantization method for weight-only LLM compression within low-bit or high-bit precision. AMLQ mainly based on the observation that different layer weights have varying sensitivity to quantization bit precision. Therefore, we design an adaptive search algorithm obtain the optimal combination of bit widths. Furthermore, the performance of most quantization techniques degrades significantly at low-bit precision and thus we introduce adaptive low-rank approximation to reconstruct quantization errors. Combining both the adaptive methods, our approach ultimately achieves the balance between performance and inference efficiency at low or high bit precision. Compared to the FP16 model, the speedup can be more than 3 times with almost no performance loss. This will further promote the practical applications of LLMSs in real-world scenarios.
References
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. Xiv preprint ar 2309 230.1.699,, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
10

---

Published as a conference paper at COLM 2024
Language models are few-shot learners. Adoances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. ar Xiv preprint ar Xrxx:2303.12212, 2023.
Zhaowei Cai and Nuno Vasconcelos. Rethinking differentiable search for muixed-precisie neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2349-2358, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. Punica: Multi-tenant lora serving. ar Xiv preprint ar xrXiv:2310.18547, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo- hammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint or xrXiv:2107.03374 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning 2018
challenge. Xrv preprint
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arv preprint 1rXiv:2110.1416,, 2021.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for near-lossless 1lm weight compression. arv preprint arXiv:2306.03007, 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. Advances in Neural Information Processing Systems, 36, 2024.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 293-302, 2019.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and
Dan Alistarh. Extreme compression of large language models via additive quantization. ar Xiv preprint 7XXi:2440.006116 2024.
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras: An efficient multitask tuning for large language models. arXiv preprint arXi::2403.03432 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. post-training compression for generative pretrained transformers. arviv preprint arXio:2200.17323, 2022.
GPTQ: Accurate
11

---

Published as a conference paper at COLM 2024
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: http://///eeeeeeeeeeeeeeeee net/ forum//ddnieeWeeeFfY..
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems, 2023.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Lessons learned from activation outliers for weight quantization in large language models. arXiv preprint arXiv:2306.0222 2023a.
Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Learnable rounding based on element-wise division for post-training quantization. In International Conference on Machine Learning, pp. 18913-18939. PMLR, 2023b.
Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and quantization for large language models. In The
Tuo Zhao. Loftq: LoRA-fine BeReA--ne-tunin--awe Twelfth International Conference on Learning Representations, 2024. URL net/forrum?id-LLLPWPAAAYY
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangx- uan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. In MLSys, 2024.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019. URL https::/ www.tt//oormm//dd1113333355
Yan Luo, Yangcheng Gao, Zhao Zhang, Jicong Fan, Haijun Zhang, and Mingliang Xu. Long- range zero-shot generative deep network quantization. Neural Networks, 166:683-691, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference Learning Representations, 2017. URL https: //openreview httpe/iwwwwweeeeeeeeeeeeeeeeeeee wwt net/forummid=Byy 2udxx..
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprini arXiv:2308.132 13137, 2023.
Hanlin Tang, Yifu Sun, Decheng Wu, Kai Liu, Jianchen Zhu, and Zhanhui Kang. Easyquant: An efficient data-free quantization algorithm for llms. arXio preprint arXiv:2403.02773 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXio preprint arXio:2302.13971, 2023a.
12

---

Published as a conference paper at COLM 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09928, 2023b.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#:: Even better IIm quantization with hadamard incoherence and lattice codebooks. arviv preprint #rX0:22402.04336, 2024.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8612-8620, 2019.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions Machine Learning Research, 2022a. ISSN 2835-8856. URL https://openrevii net www.nneett/taaee weee//oorumiiyynsSSsssttt
Survey Certification.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35: 17402-17414, 2022b.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-380999 PMLR, 2023.
Linjie Yang and Qing Jin. Fracbits: Mixed precision quantization via fractional bit-widths. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10612-10620, 2021.
Hao Yu and Jianxin Wu. Compressing transformers: features are low-rank, but weights are not! In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11007-11015, 2023.
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXfo:2304.01089, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-11 117219-14.. URL https: http:///alaaaagggy/a...oo www.ey/..gg//111//
Cheng Zhang, Jianyi Cheng, George A Constantinides, and Yiren Zhao. Lqer: Low-rank quantization error reconstruction for llms. arXiv preprint arXio:2402.02444 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. ar Xiv preprint Xxtx:2303.18223 2023.
13


Reviews:
[{"review_id": "QwCmmrrx2U", "reviewer": "Reviewer_SayR", "summary": "This paper present Adaptive Mixed Precision and Low-Rank Quantization Error Reconstruction (AMLQ), a quantization method for LLMs that adapts bit precision based on weight sensitivity. Different from previous works, it might be the first to indicate the sensitivity of different layers. This approach balances performance and efficiency, achieving speedups with minimal performance loss, enhancing the practical application of LLMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__R1__: The primary advantage of our proposed AMLQ lies in its ability to maintain quantization effectiveness and inferencing speed under ultra-low quantization bits while achieving SOTA average scores on downstream tasks. As seen in Table 1, AMLQ significantly outperforms most existing quantization approaches, with only QUIP# ranking higher. However, QUIP# employs layer-wise fine-tuning and an E8 codebook. AMLQ can save 10x quantization time and increase inference speed 3x compared to QUIP#. In Table 2, AMLQ achieves SOTA overall scores across five downstream tasks, demonstrating its superior performance in practical deployment applications. \n\n__Q1__: __(1)__ In the public evaluation datasets presented in Table 1, most existing quantization methods perform well at 4 bits, with AMLQ ranking among the top. Notably, OmniQuant and QUIP#, which show comparable effects to AMLQ, require backpropagation to update parameters during quantization, making AMLQ more advantageous in terms of quantization time and computing resource utilization. Furthermore, our AMLQ exhibits a significant performance advantage under 2-3 low-bit quantization (shown in Table 1). \\\n__(2)__ We wholeheartedly agree with your perspective. As you mentioned, the GPTQ method is an excellent work from an earlier time, yet it underperforms in ultra-low bit quantization scenarios. In contrast, AMLQ excels in ultra-low bit quantization applications. In Table 1, we showcase the quantization performance at 2-3 bits across various datasets. AMLQ achieve substantial improvements over existing strong baseline models and performed comparably to the QUIP# method. It’s worth noting that QUIP# utilizes layer-wise fine-tuning and an E8 codebook, whereas AMLQ can save 10x quantization time and improve 3x inference speed compared to QUIP#. Furthermore, according to experimental results from the QUIP# paper, AMLQ's performance surpasses that of QUIP#'s non-fine-tuned version in most scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper notes that different layers have varying sensitivities to quantization and introduces the concept of quantization error reconstruction along with its theoretical basis.", "reasons_to_reject": "Although AMLQ's quantization error reconstruction shows some effectiveness compared to GPTQ and AWQ, its performance is not significant compared to the current strongest baseline, and in most cases, it does not have a competitive advantage.", "questions_to_authors": "The two innovations proposed in this paper AMLQ are meaningful, and the ablation studies in the paper confirm this; however, the experimental results in Table 1 are not significant overall. The method in this article follows the GPTQ paradigm, yet this classic and relatively old work seems to struggle with quantization scenarios below 4 bits. It is recommended that the authors explore settings on extremely low-bit quantization and consider how to apply your ideas in this setting, which might yield more significant results.", "extra_scores": {}}, {"review_id": "jBBnngfOH6", "reviewer": "Reviewer_SfE9", "summary": "This paper proposed a rank adaptation and mixed-precision method for improved accuracy of quantized adapter-based LLM inference. The proposed method was motivated by the varying quantization sensitivity of the Transformer layers to assign higher bit-precision and/or ranks to the more sensitive layers. The authors proposed a greedy method to optimize the bit-precision and rank allocation, and justify their choice by comparing quantization performance with the other quantization methods.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__R1__: Our paper addresses two key issues with prior quantization methods. First, we tackle varying sensitivity to quantization across different layers and channels by proposing an adaptive mixed-precision approach. Our method enables a theoretically superior combination of precisions, proven to outperform uniform precision techniques. Second, we counter the significant accuracy drop in low-bit conditions witnessed in previous methods with our novel low-rank quantization error reconstruction. Through theoretical analysis and empirical evidence, we show how output activations are low-rank and how they can be leveraged to mitigate quantization errors. This validates the effectiveness and theoretical soundness of our approach.\n\n__R2__: Our method innovates for LLMs by capitalizing on uneven quantization sensitivity across LLM layers and the advantageous activation value reordering strategy, similar to that in GPTQ. The adoption of low-rank matrix error reconstruction improves quantization, particularly at ultra-low bits. Our low-rank assumption leads to a unique matrix initialization that requires minimal data, divergent from backpropagation-reliant methods, and is especially effective post-training, tailored for LLMs.\n\n__R3__: Our AMLQ method offers time and performance superiority to QuiP#, which succumbs to long quantization times and complex design. Despite marginal improvements QuiP# may demonstrate due to its unique codebook design, AMLQ surpasses conventional methods like GPTQ, AWQ, and OmniQuant, showcasing its broad empirical superiority.\n\n__R4__: Facing a large search space, we adopt a relative optimality approach with our iterative, bisecting greedy strategy. This balances optimality and computational efficiency by systematically narrowing down the layer combinations, guiding us towards a practical solution within the target bits-per-weight.\n\n__R5__: We will soon provide a comprehensive speed analysis for the Llama2-13B-chat model, with preliminary tests highlighting our solution's inference speed superiority. At 2.5 bits, our method is significantly faster than QuiP# at 2 bits and a non-quantized model at fp16, achieving 46.08 tokens/s, which further increases to 117.7 tokens/s with our specialized inference framework.\n\n__Q1__: The experiments in Fig. 1 are carried out in llama2-13b. The calibration dataset is sampled from the WikiText2. Due to space limitations, please refer to the detailed explanation of Fig. 1 in Reviewer 4's Q2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors provide a practical method to robustify extremely-low precision quantized LLM inference.\n- The authors provide an extensive performance comparison with the existing state-of-the-art quantization methods.", "reasons_to_reject": "- The proposed methods lack theoretical justification of why it overcomes the challenges of the previous method. In particular, the authors claimed that the output activation values are low-rank, but there is no explanation why they reached to this claim. If it is a simple observation, the readers would expect more extensive empirical study to support this claim.\n\n- Also, the proposed methods seem to be composition of well-known quantization methods. The proposed mixed precision allocation method seems to be a straightforward application of existing mixed precision methods into LLMs. Also, the adapter reconstruction (also called adapter initialization) is widely studied; e.g., \n\n> LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models (https://arxiv.org/abs/2310.08659) \n\n> LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning (https://arxiv.org/abs/2310.08659) \n\n> ApiQ: Finetuning of 2-Bit Quantized Large Language Model (https://arxiv.org/abs/2402.05147)\n\nDue to lack of in-depth analysis of the proposed method in comparison with these existing works, the novelty of the proposed methods is not clear.\n\n- Also, the experimental results show that the proposed method is only marginally better than QuiP#; even in Table 4, the accuracy gain from the proposed rank adaption for QuIP# seems to be very marginal. Therefore, it is not clear if the proposed method is empirically superior.\n\n- The proposed optimization technique seems to be greedy, and there is no analysis of its optimality.\n\n- The authors claimed that the proposed methods outperform the other methods thanks to its efficient CUDA implementation. However, there is no code provided or experimental results comparing the inference speed.", "questions_to_authors": "The explanation about Fig. 1 seems to lack explanation details. For example, how the authors obtain Fig, 1(C)? Which model or calibration dataset are used for this analysis?", "extra_scores": {"ethics_concerns_details": "N/A"}}, {"review_id": "4WSvsHOOMX", "reviewer": "Reviewer_n3yk", "summary": "This paper investigated the post-training quantization method for LLMs. It mainly combines two methods for two observed issues: 1) given the observation that different layers have varying sensitivity to quantization, it uses adaptive mixed precision search to search for optimal precision for different layers. 2) it also uses an existing low-rank approximation LQER to reconstruct quantization errors.  Combining both of the above methods, it seems the proposed approach could achieve a balance between performance and inference efficiency, especially for low-bit settings.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "__R1__: Your perspective is highly valuable. We've strategically designed AMLQ to minimize the total quantization error across linear layers while keeping the overall average bits per weight (bpw) under a predefined target. Through experimental observation, we identify that different layers have varying sensitivities to quantization. Accordingly, we tailor bit allocation to minimize total quantization errors—assigning lower bpw values to shallower, less sensitive layers, and higher bpw values to deeper, more sensitive ones.\n\n__R2__: We apologize for any confusion. Group size, mixed precision strategy, and proportions are critical for determining the layer's bit width for quantization. For example, dividing a (5120,5120) weight layer into 128-sized groups yields 40 groups. Quantizing 40% at 2 bits, 30% at 3 bits, and the rest at 4 bits averages to 3.3 bpw. The smaller groups require more quantization parameters, leading to a larger bit width. We select group sizes as multiples of 32 to optimize CUDA kernel efficiency, typically requiring smaller groups for lower precision but greater proportions. Therefore, we set three group sizes (32,64, and 128) to match the mixed precision quantization. Tables 1 and 2 validate our method.\n\n__Q1__: Both our method and LQER use two low-rank matrices to approximate a decomposed matrix, but while LQER decomposes the quantization error matrix $\\delta_W$, we decompose $\\delta_Y$'s covariance matrix (in Equations 10). Lower matrix rank means less error in approximation, and our experiments show that activation values are low-rank, meaning the rank of $\\delta_Y$ is lower compared to the quantization error matrix $\\delta_W$. Fig.1(c) in our paper visualizes this low rank, supporting our method's superior error reconstruction.\n\n__Q2__: Table 1 compares the perplexities on WikiText-2 of AMLQ with other strong baselines, using results from their original papers. LQER has no results below 4-bit and its closed-source code prevents reproduction for 2-3 bit results. Figure 2 shows lower bit widths for shallower layers and higher for deeper ones in the llama2-13b model, aligning with k_proj’s quantization error trends in Figure 1(a). Our greedy binary search might not find the best quantization combination, but better strategies and layer-wise fine-tuning could improve results. We will optimize the search strategy for a balance between effectiveness and efficiency. Overall, AMLQ performs better in most cases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed approach of combining two adaptive methods seems useful and could balance the performance and inference efficiency by considering both per-layer quantization and low-rank.", "reasons_to_reject": "1. The independent assumption and the sum of quantization error seem problematic as a good approximation. \n2. Equation (6), not sure the group setting is the same as the claimed \"layer-wise\" quantization bit-width. Not sure it is the optimal dynamic setting.", "questions_to_authors": "Q1. Not quite clear the difference between the proposed low-rank reconstruction error and the LQER. What is the benefit? Please clarify more. \nQ2. Could you explain more about Table 1 (Avg bit 2.0+). why is there no LQER for the 3 and 2-bit set? Also, please show your searched optimal mixed precision for each layer and their quantization loss. What can be improved?", "extra_scores": {}}, {"review_id": "44Fev2vYAR", "reviewer": "Reviewer_cTcX", "summary": "Post-training quantization of large language models is a popular and generally required technique to make inference cost affordable, while avoiding the difficulties of quantization aware training. This paper proposes an adaptive mixed precision and low-rank approximation quantization error reconstruction scheme, which allows more fine-grained control over the degree of quantization applied in different layers of the model. The approach achieves state-of-the-art performance at the approximate average bit precision and good inference speed in the case of low-bit quantization, on popular benchmarks (and with suitable adaptation data).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__R1__: For comparison with other advanced methods, we utilize consistent evaluation datasets. Additionally, we apply AMLQ to multiple business datasets for text2sql and docqa tasks. The comparative results against GPTQ and AWQ are as follows:\n| task  | AMLQ | GPTQ | AWQ  | FP16 |\n|-|-|-|-|-|\n| text2sql | __88.7__ | 84.4 | 85.7 | 88.7 |\n| docqa  | __90.3__ | 86.5 | 88.2 | 91.3 |\n\n__Q1__: The quantity and diversity of a calibration dataset indeed affect the quantization results. Our experiments show that when the calibration data size exceeds 128, the performance gap in the quantized models is very small. However, as the volume of calibration data decreases, there's a notable decline in performance, a perspective also discussed in papers such as AWQ. Moreover, since our method is based on GPTQ, the optimal calibration dataset must be sampled from the entire training data, primarily impacting model generalization. From our experiments, this effect on the results is minimal. In this section, we make a fair comparison by referring to the settings from related papers.\n\n__Q2__: As described by Eq. 2, the normalized error sensitivity is defined as $\\delta = \\|WX-W_qX\\|_2^2 / \\|WX\\|_2^2$. Fig. 1(a) aims to illustrate that the error sensitivity differs among weight types and layers. For the same type of weight, such as k_proj, there is a significant discrepancy between the beginning, middle, and end. Additionally, there are substantial variations between different weight types, such as k_proj and down_proj, even within the same layer. Fig. 1(b) illustrates the impact of applying quantization error reconstruction methods across various bits per weight (bpw), demonstrating significant performance improvements at lower bit widths. In Fig. 1(c), SVD represents the direct calculation of the rank of $\\Delta W = W - W_q$. L^2QER scales $\\Delta W$  then calculates its rank. AMLQ determines the rank of the covariance matrix of $\\Delta Y$ from Eq. 7-10.\n\n__Q3__: Fig. 2 shows the search results for bits per weight (bpw) and the required rank of the error reconstruction matrix for each q_proj layer in llama2-13b, which is also related to error sensitivity. Higher sensitivity in intermediate layers necessitates higher average bpw. For example, in layers 28-36, using the highest average bpw results in smaller quantization errors, so only the minimum necessary rank is used for error reconstruction, indicating no need for additional low-rank matrices.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's approach is well motivated and solves a realistic problem: post training quantization is preferable to quantization aware training and the proposed approach offers more fine-grained control to meet fixed compute/ size budgets than competing approaches.\n\nThe paper is well written and the experiments seem comprehensive and well thought out. The approach seems to offer significant advances over other methods.", "reasons_to_reject": "The set of evaluation data sets seems somewhat limited, but I am no expert in the field.", "questions_to_authors": "How big is the influence of the adaptation data set on the final results? I understand that the setting is standardized (i.e. same as in previous literature), but how much do results depend on the choice of fine-tuning data?\n\nFigure 1 could do with some additional explanations (or links to appropriate definitions). What is the \"normalized error sensitivity\"? Does it really vary so much across the layers or is the difference mostly between the last & first vs the middle layers? \n\nFigure 2, right plot -- what am I supposed to see here? The \"rank\" of the correction matrices seems to jump around quite erratically, does it really correspond to the errors? \nFigure 2, left plot -- which curve in Figure 1a is this supposed to correspond to? k_proj? gate_proj?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Quality
- Pro
  - Comprehensive and well-thought-out experiments
  - address important problem in PTQ
- Con
  - Some concerns of empirical quality in particular with respect to low-rank activation experiments

Clarity
- Pro
  - well-written
- Con
  - some figures unclear
  - additional theoretical justification would add to clarity
  
Originality
- Pro
  - novel way of fine-grained quantization by mixed precision + low-rank approximation
- Con
  - techniques are a composition of existing techniques
  - lacks theoretical insights why it is working

Significance
- Pro
  - More fine-grained control of PTQ
  - provides new insights into PTQ
- Con
  - marginal improvements over existing methods

**Decision**

The paper is well-motivated and well-written, and it provides a novel way of fine-grained quantization. The overall experimental quality and rigor is sufficient. The main points are standing against acceptance is the lack of theoretical insight, the presented method being just a combination of existing methods, and the overall marginal improvements compared to previous work. Overall, the reviewers deem the work borderline and falls slightly short of the COLM acceptance curve. As such, I do not recommend acceptance.

EDIT: I chatted with the PCs about this case and was encouraged to include my personal opinion on this matter to reach a decision.

I think this paper is a case of work that does not change the field, but augments existing evidence between ideas and helps researchers to really figure out what other papers might have strong signals for future research. I think these papers are essential for a healthy research environment. I, personally, am all for accepting this paper.

With this, I update my decision to recommend "Maybe acceptance", to balance the judgement of the reviewers with my personal opinion.
Decision: Accept