DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
Authors: Haokun Lin
Haobo Xu
Yichen Wu
Jingzhi Cui
Yingtao Zhang
Linzhan Mou
Linqi Song
Zhenan Sun
Ying Wei
Venue: NeurIPS 2024 oral (2024)
Keywords: Model compression
Post-training Quantization
PTQ of Large Language Models
Primary area: natural_language_processing
TL;DR: We identify massive outliers in the down-projection layer of the FFN module and introduce DuQuant, which uses rotation and permutation transformations to effectively mitigate both massive and normal outliers.

Abstract: Quantization of large language models (LLMs) faces significant challenges, particularly due to the presence of outlier activations that impede efficient low-bit representation. Traditional approaches predominantly address Normal Outliers, which are activations across all tokens with relatively large magnitudes. However, these methods struggle with smoothing Massive Outliers that display significantly larger values, which leads to significant performance degradation in low-bit quantization. In this paper, we introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers. First, DuQuant starts by constructing the rotation matrix, using specific outlier dimensions as prior knowledge, to redistribute outliers to adjacent channels by block-wise rotation. Second, We further employ a zigzag permutation to balance the distribution of outliers across blocks, thereby reducing block-wise variance. A subsequent rotation further smooths the activation landscape, enhancing model performance. DuQuant simplifies the quantization process and excels in managing outliers, outperforming the state-of-the-art baselines across various sizes and types of LLMs on multiple tasks, even with 4-bit weight-activation quantization. Our code is available at https://github.com/Hsu1023/DuQuant.

DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
Haokun Lin Haobo Xu*?, Yichen Wun',Jingzhi Cui², Yingtao Zhang", Linzhan Mou Linqi Song', Zhenan Sun 1,3 Ying Weil 4,5
School of Artificial Intelligence, University of Chinese Academy of Sciences Tsinghua University NLPR & MAIS, Institute Automation, CAS City University of Hong Kong "Zhejiang University
haokun. www.crippccia.............. iaa.iaa......m. xuhb20@maile tsinghua. edu. cn wuyichen. am970gmail com com znsunOnlpr.ies ac. cn werg..eiizjje edu. cn aa..a.aa...........
Abstract
Quantization of large language models (LLMs) faces significant challenges, par- ticularly due to the presence of outlier activations that impede efficient low-bit representation. Traditional approaches predominantly address Normal Ouiliers, which are activations across all tokens with relatively large magnitudes. However, these methods struggle with smoothing Massive Outliers that display significantly larger values, which leads to significant performance degradation in low-bit quanti- zation. In this paper, we introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers. First, DuQuant starts by constructing the rotation matrix, using specific outlier dimensions as prior knowledge, to redistribute outliers to adjacent channels by block-wise rotation. Second, We further employ zigzag permutation to balance the distribution of outliers across blocks, thereby reducing block-wise variance. A subsequent rotation further smooths the activation landscape, enhanc- ing model performance. DuQuant simplifies the quantization process and excels managing outliers, outperforming the state-of-thee and types of LLMs on multiple tasks, even with 4-bit weight- activation quantization. Our code is available at https:: //ggt/ttttt com/HHsull023/Duuuan.
baselines across various sizes
1 Introduction
Large language models (LLMs) 51 50
have demonstrated exceptional performance across wide range of natural language processing tasks. However, their billions of parameters present considerable deployment challenges on resource-constraine, edge devices, particularly terms of memory usage and inference speed [22 15| (6]. In response to these challenges, network quantization methods [20 23 have been extensively explored to minimize memory usage by converting floating- point parameters into low-bit formats [8 32 S and to expedite inference by quantizing both
activations and weights for accelerating the matrix multiplication process 64
34 74].
Among LLM quantization methods, a primary issue is the presence of activation outliers, which enlarge the quantization step sizes and subsequently cause significant accuracy loss (59]. To mitigate this problem, current research has developed various methods address Normal Outliers in activations, which are persistent in several channels across all tokens
13 64 However, besides
Normal Outliers, there exists another type of activation outlier 48
35 termed Massive Outliers.
"Equal contribution. Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

All tokens, Relevantly large volues
Limited tokeess, Extremely large values
Map bette mmaeht
betoe ancor
Hard to climinate
New outilers
[M] LLLLLO02, 70, Jayperr Jitn, Satt___Pr (_Frr)
(b) LLAR222 75, 20__operl_ STN, Down, Proj
(c) Activation Change with SmrothhQ pant
(4) Sie git Change sitt SmoothQuart
Normal Outtlers
Massive Outters
SmoothQuant Eliminate Massive Outtiers
Figure 1: Visualizations of Outliers in LLaMA2-7B. (a) Input activation of Layerl attention key projection shows Normal Outliers with relatively high magnitudes across all token sequences. (b) Input activation of Layer1 FFN down projection reveals Massive Outliers, presenting extremely high magnitudes (around 1400) at very few tokens. (c) Application of SmoothQuant on FFN down projection, illustrating its struggle with massive outliers in the Activation matrix. (d) Corresponding weight changes with Amoothhuann, highlighting the emergence of new outliers.
These outliers are characterized by their exceedingly high values and limited occurrence in a subset of tokens, as depicted in Figure [bb. Unfortunately, existing LLM quantization methods struggle to effectively address these Massive Outliers. For instance, SmoothQuant [64], despite using a smooth factor to shift some of the activation outliers the weight part, still cannot effectively handle Massive Outliers with extremely large values, as shown in Figure [d).). OmniQuant [47] and AffineQuant [39], on the other hand, exhibit training instability issues (34] due to the presence of Massive Outliers. Consequently, there is a pressing need for an LLM quantization approach that effectively addresses both Normal and Massive Outliers.
To tackle this challenge, we propose the Dual transformations Quantization (DuQuant) method. Our motivation is to redistribute the activation outlier values across different channels, facilitating casier quantization. Specifically, we construct the orthogonal rotation matrix and the orthogonal permutation matrix. By multiplying these matrices with the activations, column transformations on the activations, which in turn allows for the redistribution of outliers. For the rotation transformation aspect, we first identify specific dimensions of outliers as the prior knowledge and employ a greedy algorithm to construct the rotation matrix. To enhance the multiplication efficiency, responsible for a small portion of the activations. However, this approach may result in uneven outlier magnitudes across different blocks. Therefore, we propose the zigzag permutation for reordering the activation channels, which promotes a more uniform distribution across different blocks. Concretely, distribute the channels with the highest activations across the blocks in back-and-forth pattern. After establishing blocks with uniformly distributed outlier magnitudes, we employ another rotation transformation to further redistribute the outliers within each block. Note that we multiply the weight matrix with the transpose of the rotation and permutation matrices at the same time, preserving the linear layer equivalence and smoothing weights. Theoretical analysis confirms that the rotation and permutation transformations greatly mitigate quantization challenges induced by outliers.
effectively perform
utilize diagonal plock-wise rotation matrices, with each matrix
As result, DuQuant offers several clear advantages over QuaRot 2 (1) DuQuant's optimal rotation matrix, derived through greedy search guided by prior knowledge, surpasses QuaRot's Hadamard rotation in managing outliers; (2) our unique zigzag permutation significantly reduces activation variance across blocks, providing a distinct advantage for handling massive outliers; and (3) by jointly smoothing weights and activations, DuQuant avoids time-consuming GPTQ algorithm in QuaRot. Extensive evaluations demonstrate that our DuQuant approach significantly outperforms existing 4-bit weight-activation quantization baselines across various benchmarks. Notably, DuQuant achieves a 5% improvement in Commonsense QA tasks across all LLaMA model sizes and 10% increase in zero-shot MMLU benchmarks for the Vicuna-vl. 1.t.aa-l1..11111 5-155.11.... Moreover, in practical applications with the LLaMA2-7B 7B model, DuQuant not only accelerates pre-filling phase by up to 2.08x but also reduces memory usage during decoding phase by 3.50x, with minimal impact on performance: only 0.61 increase in perplexity and 2.71 1 % drop in accuracy compared to the FP16 model. These results highlight the effectiveness of DuQuant in enhancing the efficiency and capacity of quantized LLMSs.
2 Motivation
Normal Outliers and Massive Outliers. Previous works L3 72 32 have highlighted the challenge posed by activation outliers in LLMs for model compression. These outlier features consistently

---

manifest large values across specific feature dimensions and are present in all token sequences [64] which we refer to as Normal Outlies. Recently, a distinct type of outlier 48 351, termed Massive Outliers, has been observed in LLMs.. The primary distinctions between normal and massive outliers are: 1) Normal outliers persist across all token sequences, whereas massive outliers are confined to limited number of tokens. 2) Massive outliers exhibit significantly larger magnitudes, often surpassing 100 and being approximately 1000 times greater than the median of other activations [48]. In our study, we delve deeper into the impact of these two distinct types of outliers on quantization.
Massive Outliers Exist at the Second Linear Layer of FFN Module. In contrast to previous studies 48 33 that observe massive outliers the output of Transformer blocks, we first discover that these extremely large activations exist at the input of the down-projection layer within the FFN module. As depicted in Figure the input of the down- poon-proecction layer the LLaMA2-7B model Layer contains single activation of significant magnitude (approximately 1400). This activation isolated to one token and therefore classified as one of massive activations. This phenomenon is consistently observed across different layers and sizes of models, illustrated in Appendix [
Massive Outliers Enlarge Quantization Difficulty. Although previous studies [64] [47 39 I have proposed various approaches to eliminate outlier features, they still face challenges in effectively managing massive outliers. SmoothQuant [64]. for instance, attempts to shift the quantization difficulty from activations weights by dividing the activation by per-channel smoothing factor and multiplying it to the weight matrix. Nevertheless, we observe that this transfer at the input of the down-pre cwwn-projection layer can cause the weights of the down-projection to display noticeable outliers, as demonstrated in Figure [ This issue arises because massive outliers cause the smoothing factor to become significantly large. Moreover, extremely large outliers can lead optimization-basee based methods to encounter problems with loss explosion. Both OmniQuant 447 and AffineQuant (39) have had to exclude their learnable parameters for the down projection layer due to unstable gradients. Given the poor accuracy observed with 4-bit quantization, QUIK (1] opts to use INT8 quantization for the down projection layer and Atom [74] applies INT8 quantization for 128 outlier channels. Consequently, massive outliers introduce new challenges to the quantization process that existing methods cannot fully address. This observation has motivated us develop rotation and permutation transformations, which effectively handles both massive and normal outliers and achieves state-of of-the--rt performance.
3 Method
In this section, we delve into the distribution of outliers and introduce our proposed DuQuant method. The DuQuant method is built two key components: 1) the block-diagonal rotation matrix, tasked with the local redistribution of feature outliers, and 2) the zigzag permutation, responsible for the global reordering of outliers across different blocks.
3.1 Preliminaries
As the common modules within each transformer block of LLMs, both Multi-head Self-Attention (MSA) and Feed-Forward Network (FFN) fundamentally consist of basic linear layers, which can be represented as, Y = X W E RxCCt Here, X E I T is the activation input and W E
R.C. x &Cuu denotes the weight matrix. In this paper. we focus on integer uniform quantization [25]
of both activation and weight, aiming to achieve better hardware support. Specifically, the b-bit quantization process maps the FP16 tensor X to low-bit integer X.:
X
+ 0, 2 - 1). where A max(x) minn(X)
, ===- mmin(X) (X)
X, = -clamp
(I)
26
A
The notation L- means the nearest rounding operation, A is the quantization step size and z represents the zero point. Following [64 47, 34, 39), we employ per-token quantization for activation and per-channel quantization for weight, which entails assigning different step sizes individual tokens
of activations (Ax E IR. Tx) and different output channels of weights (Aw E R RCCCoo)).
3.2 The proposed DuQuant Method
To address the Normal Outlirs issue stated in Section 2 current quantization methods, such as SmoothQuant (64] and OmniQuant [64), usually adopt the smooth technique. Concretely, it involves the utilization of a per-channel smoothing diagonal matrix, denoted as A, to scale the input activation and weight matrix. The adjustment allows to rewrite the original linear layer Y X W = (X-
3

---

Rotation
Rotation
outlier
ABs Mac
Rocation
Variance
Dutierr
Outlee Sorted Index
Activation Matrix
2
Permutation
Easy
Outlee
20
1.2
2.5
Ziying Index
Rotation
1.9
1.8
1.2
07
Veriance
0.175
3.2
422
3.0
3.
34
0.9
BB
2.5
3.5
09
22
(b) Massive outlier
(c) Example of Rotation and Permutation Transformation
Figure 2: Transformation Steps for Activation Matrices after smooth technique. (a) Sequential trans- formations Normal Outliers: @ initial rotation reduce outliers within blocks, @ permutation to evenly distribute outliers across blocks, and a second rotation for further smoothing. (b) Activation changes for Massive Outliers before and after DuQuant. (c) A sample matrix for highlighting the continual reduction of outliers through rotation and permutation, with outliers marked in dark blue.
A- W). The diagonal element A; within computed as A,- max x1)xx (a 1 where a is a hyper- parameter representing the migration strength. However, despite the ability of this smoothing technique shift the quantization challenge from activations to weights, still faces difficulties in effectively managing Massive Outliers, as depicted in Figure This challenge stems from the extremely large massive outliers inducing large scaling factors A,. which in turn introduce new outliers in the weight matrix and result in significant performance declines in 4-bit quantization.
According these findings, propose the DuQuant method, which includes the Rotation and Permutation transformations based on the smooth technique. By combining rotation transformation and channel permutation, our DuQuant method aims to redistribute these features within the activation space, thereby mitigating the effects of both Normal and Massive Outliers.
The Rotation Transformation. In contrast to the smooth technique, our aim is to apply a rotation matrix for row or column transformations, mitigating the impact of both Normal and Massive outliers. The ideal rotation matrix, denoted as R. should possess the following properties: 1) R is an orthogonal matrix satisfying RR I and |R| 1.. This allows
reformulate the linear layer within the transformer Y X W (R 2) R should be capable of effectively target the positions of outliers and effectively mitigating them through matrix multiplication. However, due the Massive Outliers usually randomly distributed within the activation space, it is challenging to directly identify the optimal rotation matrix R capable of mitigating outliers through a single rotation transformation. To address this problem, we employ greedy search with prior knowledge to compute rotation matrix R. thereby approximating the ideal rotation matrix Specifically, the calculation of R involves the following steps,
Identify the feature dimension (1)) where the outlier are primarily concentrated, i.c., d()) arg max, Xxz]). )). Here, X;j represents the element in the i-th row and j-th column of X.
Based on the searched dimensions d(1), we construct the rotation matrix as follows,
R E RQE gay,
Q
(2)
Q'
Here, Ega is the switching matrix used to swap the first and the 1 columns of the activation, and R represents an orthogonal initialized rotation matrix, in which the first row is specifically uniformly distributed. The motivation behind this to mitigate outliers in the first column after the transformation by Ega, To further increase the randomness of the rotation operation,

---

we retain the first column, where outliers have been mitigated, and randomly rotate the other columns by multiplying them with random orthogonal matrix Q'
Let N denote the greedy search steps, then the approximated rotation matrix R R", where n = arg [1:Ng( (max..) (XR ... R*))))- Each R is constructed according to Eqn. 2 and the identified feature dimension d(i), Appendix G provides detailed pseudo code.
Through this construction manner, we can ensure that the approximated optimal rotation matrix R can effectively mitigate outliers with large magnitudes, as opposed to merely using a randomly selected orthogonal rotation matrix. Nevertheless, directly constructing the entire rotation matrix is time- consuming and results substantial memory overhead. For fast matrix multiplication, following [63] we approximate the rotation matrix R E CR. XC.n in a block-wise manner,
R BlockDiag (Re,:-.. Rog);
where Rs. E R22 x2" denotes square matrix of the i-th block, which is constructed following the three steps mentioned above. And the block numbers K calculated by K == -Cnn/"
The Permutation Transformation. Despite adopting the block-diagone rotation matrix R. for its time and storage efficiency, its focus on local information introduces a potential limitation in further reducing the outliers. This is because the rotation transformation, conducted within each small block, cannot integrate the information across different blocks to further minimize outliers. Consequently, one block may have relatively larger outliers while another block has smaller outliers, resulting in high variance among different blocks, as shown in Figure 2 This limitation explains that merely utilizing the block-diagonal rotation matrix is insufficient to effectively reduce the overall outliers.
To effectively mitigate the overall outliers, it is essential to balance the outliers' magnitudes among various blocks. Specifically, within each small block, we denote the largest outlier in dimension dj as O,. Meanwhile, Mr represents the mean value of all O, in the i-th block, where 1, 2, ..., K. Then the variance in activation magnitudes across various blocks can be expressed as,
Var([Mb,; Mo,,.., Mbx)).-
To minimize this variance and further reduce the overall outliers, we introduce the zigzag permuta- tion. Concretely, we generate a zigzag sequence that starts by assigning channels with the highest activations to the first block. The process continues by assigning channels with the next highest activations to the subsequent blocks in descending order until the end of block K. Upon reaching the final block, the order reverses, starting from the channel with the next highest activation and proceeding in ascending order. This aak--nd-forthh patterning continues throughout all the blocks, ensuring that no single block consistently receives either the highest lowest activation channels. is worth noting that the constructed permutation is an orthogonal matrix, which we denote as P. achieve
satisfying the conditions PP I and |P| =1. By employing the zigzag permutation, a balanced distribution of outliers across different blocks. This allows us to use an additional rotation transformation to further smooth the outliers. Figure proviides an illustration of outlier mitigation.
The Overall DuQuant Method. To effectively mitigate both Normal and Massive Outliers, we first employ the smooth technique to shift the quantization challenge from activations to weights. Next, we introduce the block-diagonal rotation matrix R to locally redistribute feature outliers within the activation space. We then propose the zigzag permutation matrix for globally balancing the outliers across different blocks, followed by another application of the block-diagona rotation transformation. To sum up, the linear layers within the transformer can be rewrite as,
Y = x W [X
Ray ay) P R
P (A-(AA-))))
(5)
where the notation denotes the orthogonal permutation matrix learned via the zigzag manner, the R11) and R22) represent the first and second block-diagone rotation matrix, respectively.
Remark 1. It is worth noting that the proposed DuQuant method can simultancously smooth the weight matrix. While the commonly adopted smooth technique effective, it can cause the weight matrix of the droj-prooection layer to exhibit pronounced outliers, leading to performance degradation. However, in the proposed DuQuant method, the rotation transformation we designed is applied to not only the activation input but also the weight matrix. As result, the outliers induced by the smooth technique can be mitigated through our approximated rotation matrix R. yielding
5

---

smoother, more quantization-fiin friendly weight matrix. Moreover, this approach eliminates the reliance on complex weight quantization techniques, such as GPTQ T8 used Atom 74] and QuaRot [2.
Remark 2. To further decrease the computation and memory costs, we initially construct the k-th block rotation matrix Reg, with the k-th block containing the largest outlier. We then assign Rs. = RR, for all 1 s S K. This strategy not only effectively mitigates the impact of outliers, but also reduces the number of block rotation matrices from K to 1, significantly reducing computation and memory requirements. Importantly, incorporating the invertible matrix G from Eqn. (5) significantly cases the quantization challenges for X and W. Consequently, the quantization process acts as Y = (G - WW) = =x W = x Xx WW, -w)).
3.3 Theoretical Analysis
To further demonstrate the effectiveness of the proposed DuQuant method, we conduct theoretical analysis of the rotation and permutation transformations. Theorem [ shows that within each block, the constructed rotation matrix effectively mitigates the maximum outlier, thereby reducing the outlier magnitude through a greedy search. Theorem reveals that the employed zigzag permutation ensures balanced upper bound shared among different blocks. This suggests that the zigzag permutation effectively reduces the variance shown in Eqn. (4) and thus assists the rotation matrix in further decreasing the outliers. Please refer to Appendix B for detailed proofs.
Theorem 1 (Rotation). For the activation input X E RTTxCi, R E =2" is a diagonal block
matrix constructed as per Eqn. (3). For a specific block b,, let (,)+) represent the maximum outlier of the j-th dimension d within the input. Then, we can deduce that,
max ISSS2"
O, S
max 15552
Oz(Xo.).
(6)
Theorem 2 (Zigzag Permutation). For the activation input X E RT'X..., it can be divided into K blocks, where K = Cnn Let O3 denote the max outlier of the dimension dj in X, the reordered outliers from large to small is expressed as (1), 0(2) (o0)),,.,,(GG OCr,), O(C,)) Moreover, the Mr. represents the mean value of all ; in the i-th block, = 1,2, K. Let = max max(1(((+1) - 10 =
1, 2, ..., 1.n-1. Then, following the zigzag permulation described in Section 3.2 the mean value Mo, within each i-th block consistently satisfies,
Mo. S 5000 (2" K
1)(22-1 1)68 i= 1,2, K.
2
4 Experiment
Models and Evaluations. We apply our DuQuant on pre-trained LLMs: LLaMA (7B-65B) [11,
LLaMA2 (7B-70B) (52), LLaMA3 (8B, 70B), Mistral, Phi2 and instruction-tumed LLMs: Vicuna- v1.5 (7B-13B) 13B) [1]]. We evaluate quantized pre- pre-trained LLMs on language generation tasks and commonsense QA tasks. Specifically, we assess the perplexity on Wiki'Tex12 (40) and C4 [44] datasets, as well as the zero-shot accuracy on PIQA (.. ARC [12,, BoolQ [II,, HellaSwag 770 and
WinoGrande 45 datasets. Moreover, we evaluate quantized Vieuna models MMLU 21 and
MT-Bench 76 benchmarks, as well as their long-form generative capabilities on LongBench 4
39 we apply per-token activation quantization and per-channel weight quantization. Given that W8A8 quantization has been estab- lished as lossless in precision by SmoothQuant (64, our primary evaluation in this paper focuses on 4-bit and 6-bit quantization for weights and activations. As for details, we quantize all intermediate activations, excluding the SoftMax output. Moreover, we have developed two types of quantized models, denoted DuQuant and DuQuant-uwwc. For DuQuant, we employ round-to-nearest quanti- zation, using a clipping ratio of 0.9 for activations and 0.8 for weights. To improve weight matrix quantization, DuQuant-uwc integrates the learnable weight clipping (LWC) technique from Omni- Quant. Concretely, LWC adjusts weights by training parameters 7,3 E [O, 1] to compute step size
Implementation Details. In line with prior studies 34
A 7 - min in Eqn. (I) Notably, the smoothing diagonal matrix and the learned weight clipping factor can be integrated into the quantized weights, introducing no additional computational or memory costs. More details and hyperparameters are left in Appendix C
Baselines. We compare with state-of-the-and (SOTA) weight-activation PTQ methods, including SmoothQuant [44, Outlier Supression+ [59], OmniQuant (47, QLLM 34 AffineQuant (39], and Atom (7). For Atom, we reproduce the results with no group-wise asymmetric quantization.

---

Table 1: Perplexity (L) results under 4-bit weight-activation quantization. The results for W6A6 can be found in Table D8 Atom and OmniQuant unprocessed group-query attention for LLaMA2-70B.
Dataset
#Bit
Method
1-7B
1-13B
1-30B
1-65B
2-7B
2-13B
2-70B
FP16
5.68
5.09
4.10
3.53
5.47
4.88
3.31
SmoothQuant
25.25
40.05
192.40
275.53
83.12
35.88
26.01
OmniQuant
11.26
10.87
10.33
9.17
14.26
12.30
NaN
WikiTex12
QuaineQuant
10.28
10.32
9.35
12.69
11.45
W4A4
QL.LM
9.65
8.41
8.37
6.87
11.75
9.09
7.00
Atom
8.15
7.43
6.52
5.14
8.40
6.96
NaN
DuQuant
6.40
5.65
4.72
4.13
6.28
5.42
3.79
DuQuant-awe
6.18
5,47
4.55
3,93
5.33
FP16
7.08
6.61
5.98
5.62
6.97
6,46
5.52
SmoothQuant
32.32
47.18
122.38
244.35
77.27
43.19
34.61
OmniQuant
14.51
13.78
12.49
11.28
18.02
14.55
NaN
C4
AffineQuant
13.64
13.44
11.58
15.76
13.97
W4A4
QLLM
12.29
10.58
11.51
8.98
Atom
10.34
9.57
8.56
8.17
10.96
9.12
NaN
DuQuant
7.84
7.16
6.03
7.05
DuQuant-Ewe
7.73
7.07
5.93
7.79
7.02
Table 2: Zero-shot QA (1) results of LLaMA1 models under 4-bit weight- quantization.
The results for LLaMA2 models and W6A6 quantization can be found in Table [J
D9
and DT0
Model
Method
PIQA
ARC-E
ARC-C
BoolQ
HellaSwag
WinoGrande
Avg.
FP16
77.47
52.48
41.46
73.08
73.00
67.07
64.09
SmoothQuant
49.80
30.40
25.80
49.10
27.40
48.00
38.41
OS+
62.73
39.98
30.29
60.21
44.39
52.96
48.43
LLaMA1-7B W4A4
OmniQuant
66.15
45.20
31.14
63.51
56.44
53.43
$2.65
AffineQuant
69.37
42.55
31.91
53.42
QLLM
68.77
45.20
31.14
57.43
56.67
51.84
Atom
71.44
47.74
35.49
67.71
63.89
55.01
$6.88
DuQuant
76.44
50.04
38.99
70.98
69.39
64.72
61.76
DuQuant-awe
76.22
50.04
38.31
61.18
FP16
79.10
59.89
44.45
68.01
76.21
70.31
66.33
SmoothQuant
61.04
39.18
30.80
61.80
52.29
51.06
49.36
OS+
63.00
40.32
30.38
60.34
53.61
51.54
49.86
LLaMA1-13B W444
OmniQuunt
69.69
47.39
33.10
62.84
$8.96
55.80
$4.37
AffineQuant
66.32
43.90
29.61
64.10
56.88
54.70
52.58
QLLM
71.38
47.60
34.30
63.70
59.43
55.28
Atom
71.38
49.07
36.69
64.53
68.00
58.56
58.04
DuQuant
77.26
58.04
41.55
67.55
73.62
66.69
64.12
DuQuant-ave
77.64
57.32
41.21
66.79
74.12
65.98
63.84
FP16
80.08
58.92
45.47
68.44
79.21
72.53
67.44
SmoothQuant
58.65
35.53
27.73
60.42
35.56
48.06
44.83
OS+
67.63
46.17
34.40
60.70
54.32
52.64
$2.62
LLaMA1-30B
OmniQuant
71.21
49.45
34.47
65.33
64.65
59.19
56.63
W444
AffineQuant
70.84
49.41
37.12
70.12
65.53
58.64
58.61
QLLM
73.83
50.67
38.40
67.91
58.56
$7.87
Atom
71.98
49.07
40.02
66.85
70.45
58.64
59.50
DuQuant
78.56
56.99
42.32
66.73
76.70
65.15
DuQuant-awe
78.73
56.52
43.17
68.84
77.53
70.96
65.96
FP16
80.79
58.71
46.24
82.29
80.72
77.50
71.04
SmoothQuant
64.47
40.44
29.82
59.38
39,90
52.24
47.71
LLaMA1-65B
OS+
68.06
43.98
35.32
62.75
50.73
54,30
$2.52
W4A4
OmniQuant
71.81
48.02
35.92
73.27
59.51
$9.22
QLLM
73.56
52.06
39.68
Atom
74.48
51.60
40.61
73.76
73.78
62.12
62.73
Du DuQuant
79.71
57.95
45.05
79.82
78.66
72.29
68.91
Du DuQuant-uwe
79.98
58.29
44.80
77.89
79.22
72.21
68.73
4.1 Main Results
Quantization of LLaMA1 and LLaMA2 Models. We conduct comprehensive comparison of our DuQuant with several SOTA baselines on I.LMAAI and ILaMA2 models. Results for W4A4 quantization presented in this Section, while results for W6A6 quantization are provided in

---

Table 3: Zero-shot and five-shot results the MMLU benchmark for Vicuna-v1.5- under 4-bit weight-activation quantization. The results for Vicuna-vl v1.--b can be found in Table
Model
Method
MMLJU shot)
MMLJU shot) 1
STEM
Hums
Social
Others
Avg.
STEM
Hums
Social
Others
Avg.
FP16
43.70
62.72
62.74
51.97
65.26
62.40
55.78
Vicuma-v v1.5 1.3 W4A4
SmoothQuant
OnnnQuaant
21.70
24.29
26.57
30.35
22.13
23.16
28.75
22.82
28.79
25.31
24.97
31.13
26.00
28.99
27.08
29.05
25.84
Atom
32.54
39.60
46.02
46.11
41.07
35.35
39.21
59.72
45.77
45.01
DuQuant
40.82
46.61
58.73
57.59
50.94
40.92
48.78
60.42
57.71
51.96
DuQuant.owe
40.13
47.48
$8.86
$7.83
$1.08
41..42
48.52
58.73
57.74
51.61
Table 4: Long- contect generation results for 4-bit Vicuna models the LongBench benchmark.
Viesma
Setting
Queper
QMNum
MultiNews
TREC
Trivin(QA
SAMSum
DuReader
Repoliench-PP
Avg
23.27
26.91
Vicuma-v1.5.72 W4444
SmecthQuast
4.11
2.00
6.05
15.00
1.62
1.55
4.24
25.92
7.56
OrnisQuant
1.62
3.93
2.64
1.00
0.81
0.61
1.87
14.97
3.43
Atom
17.97
20.24
24,60
58,00
67.20
37.94
19.41
29.34
34.
DuQuant
19.98
21.15
25.85
64,00
78,91
42.24
23.15
47.66
40.37
24.41
26.53
Vicuma-v1.5-122 1.5-13B W444
Ouniiuaant
Smooth:Quar
0.68
2.18
2.95
1.78
2.83
3.54
9.00
1.50
1.13
1.83
0.45
13.83
11.57
8.46
4.77
Atom
17.67
20.23
23.39
59.00
80.75
38.72
21.79
37.31
37.36
DuQuant
18,93
20.72
26.59
66.50
83,04
42.67
26.02
38,09
40.32
Table 5: Perplexity and QA results of LLLMM3-8B under 4-bit/6- bit weight- quantization.
dBis
Method
WikFTest2
C4
PTB
PIQA
ARC-E
ARC-C
BoolQ
HellaSwag
WinoGrande
Avg.
9.91
50.85
77.78
53.41
81.28
79.16
72.84
74,22
SmootthQunt
7.07
9.57
11.69
73.94
75.88
49.19
77.58
77.39
70.8
71.68
LLaMA3-B 8B W6A6
Aftneeuant
OmniQQuant
7.35
7.24
9.99
9.82
11.90
12:30
75:73
78.90
73.95
73.32
47.35
45.08
74.95
74.59
72.08
76.77
70.88
70.56
70.41
70.11
DuQuant
6.27
8.38
10.77
80.20
77.27
52.05
80,12
79.14
72.77
73.59
DaQuant-owe
6.27
SmoothQuant
278.02
54.57
31.9
31.26
DoniQuant
3,6483
28003
3,09c3
26.94
34.57
37.98
36.08
LLAMA3-88
Afhineuuane
21,21e3
34.60e3
16.7263
50.71
25.93
26.02
40.55
26.07
48.46
36.29
W444
Alom
22. 14
31.83
40.04
62. 05
49.4 45
12
60.31
53.75
$6.04
52.
DuQuant
8.56
11.98
13.66
75.08 6M
68. 48
41.81
71.99
73.07
66.22
66.21
DuQuant-oww
8,06
11.29
13.19
76.22
70.41
43.69
74.34
73,87
67,50
67.72
Appendix D Table I indicates that our DuQuant quantized models notably outperform other baselines on both the WikiTeex2 and C4 datasets. Notably, LWC technique further enhances model capacity, with our DuQuant-Lwe achieving comparable performance with FP16 models. Table 2 and Table DJ showcase the zero-shot accuracy of W4A4 quantization on Commonsense QA tasks, where DuQuant significantly improves the average accuracy. Our method surpasses QLLM by +9%, and Atom by +5% for all model sizes. These results demonstrate the superiority of our rotation and permutation transformation, which establishes new SOTA performance by effectively eliminating outlier features.
Quantization of instruction-tuned Models. We quantize Vicuna-v1., 1.5 10 models to assess the generalizability of our DuQuant. Table 3 illustrates that our quantized models surpass the baselines across all task categories on MMLU benchmark. For Vicuna-13B, our DuQuant-twe surpasses Atom by 10.01 % under zero-shot settings and 6.95% under five-shot settings. Moreover, we compare our DuQuant with Atom and OmniQuani using MT-Bench and utilize GPT-4 to evaluate the answers from quantized models. As shown in Figure 3 DuQuant quantized models significantly outperform both Atom and OmniQuant in win rates. Specifically, for Vicuna- (icuna-7B. DuQuant only lost 16 and I times Atom and OmniQuant, respectively, while achieving 68 and 155 wins against them.
Evaluation of Long-context Generation. To further evaluate the long-text generative capabilities, we follow (37 33 (55] and conduct a comprehensive comparison of DuQuant against state-of-the baselines on the LongBench [4. which includes variety of generative tasks to provide broader evaluation. We set the maximum sequence length to 3500 for Vicuna models, with results presented in Table DuQuant achieves performance comparable FP16 models, demonstrating the effectiveness of our dual transformations. More detailed results on different subtasks are listed in Table [3] [4]
Quantization of LLaMA3 Models. LLaMA3, known for its superior performance in various tasks, faces significant degradation in low-bit quantization 24). To address this, we apply our DuQuant to quantize -L-MA3-8B. Table displays the perplexity and zero-shot accuracy results. Notably, under W6A6 setting, our DuQuant achieves performance comparable to FP16 model. Furthermore, unlike other methods that show weaker results under W4A4 setting, our DuQuant maintains competitive performance, indicating its robustness with I.L.MA3. We attribute this success to the advanced handling of outliers achieved through dual transformations, which is not restricted to specific models.
8

---

4.2 Ablation Study
Module-wise Impact. We ablate four distinct operations within DuQuant: only the smoothing technology like SmoothQuant; 2) one rotation following the smoothing operation; 3) sequence of rotation, permutation, and another rotation without smoothing; and k4) full DuQuant approach. Ssows that the smoothing operation plays basic role in our DuQuant by shifting activation outliers to weight. The initial rotation significantly enhances model performance, yielding competitive PPL results. Finally, permutation combined with second rotation further enhances the quantized model.
Table 6: Influence of different components in DuQuant under 4-bit weight-activation quantization.
Modules
LLaMA2-7B
LLaMA2-13B
Smooth
Rotation
Permutation
Rotation
WikiTest2.
C4
Wikilext2
C4
NaN
1379.46
160.30
203.87
8.45
10.63
14.32
21.73
7.92
10.64
5.96
7.44
6.79
8.51
6.06
8.03
6.28
7.90
5.42
7,05
Influence of Normal/Massive Outliers. In this section, we comprehensively explore the influence of massive and normal outliers quantization. Notably, we observe that massive outliers primarily occur at the down-projection of the FFN module. To isolate their effect, we remove the rotation and permutation transformations, applying only the smoothing technique to all down-prope nown-projection inputs. The resulting perplexity for LLaMA2-7B and LLaMA-13B showed significant degradation, presented in 7 Conversely, when eliminate the rotation and permutation transformations for normal outliers, the performance decrease was noticeable but less severe compared to massive outliers. These findings indicate that: 1) massive outliers exert more substantial impact quantization, corroborating our claims in Section 2 2) the smoothing technique alone struggles to fully mitigate the influence of outliers, particularly massive ones; and 3) rotation and permutation methods prove highly effective against both types of outliers, leading to superior performance.
Table 7: Outliers impact on quantization. We only apply the smooth technique on Normal and Massive outliers for W4A4 quantization.
Figure 3: GPT-4 evaluation on the MT-
Vicunaw1.5-78
Alm
Outlier Type
LLaMA2-7B
LLaMA2-13B
OreeBuant
155
Normal
Massive
Wiki
C4
WikIText2 I C4
Alor
18.16
26.42
10:51
16.01
6.28
7.90
5.42
7.05
320
Comparison with QuaRot [2] In light of the recent introduction of Hadamard rotations by QuaRot to eliminate outlier features, we have undertaken a detailed analysis to highlight the key differences between our DuQuant and QuaRot. To ensure a balanced evaluation, we have re- implemented QuaRot in accordance with our quantization settings. The results demonstrate that 1) the rotation matrix constructed by DuQuant outperforms QuaRot's 's approach of simply selecting ran- domly initialized Hadamard matrix. As depicted in Figure our DuQuant more effectively smooths activations than QuaRot. This attributed to the prior knowledge utilized by DuQuant to accurately target the outliers; 2) As demonstrated by the perplexity in Table QuaRot employs GPTQ for their weight quantization method, whereas our DuQuant, with its sophisticated outlier management, attains competitive results using RTN quantization. The superiority proves the effectiveness of our zigzag permutation to enhance capacity. For a more comprehensive comparison, please refer to Appendix
Table 8: PPL (L) comparison under W4A4 setting. Figure LLaMA2-7B Attention key _ey_proj.
Method
1-7B
1-13B
1-30B
2-7B
2-13B
FP16
5.68
5.09
4.10
5.47
4.88
QuaRot-RTN
7.08
6.57
5.44
9.66
6.73
QuaRot-GPTO
6.44
5.63
4.73
6.39
5.75
DuQuant
6.40
5.65
4.72
6.28
5.42
DuQuanterne
6.18
5.47
4.55
6.08
5.33
(4) Hadamard
(b) Ours
Permutation Frequency. We conduct ablations on the rotation and permutation frequencies in
DuQuant. As shown in Figure 5 "Perm I" (two rotations with one permutation) achieves stronger

---

performance compared with "Perm 0" (no permutation), while incurring an additional 8.9% computa- tional cost on LLaMA2-7B and 9.3% on LLaMA2-13B compared to the W4A4 setup. Considering the approximately 2x speedup and the impressive performance, these additional costs are deemed acceptable. Further permutations, like "Perm 2." do not improve performance and reduce inference efficiency. Consequently, "Perm 1" strikes the best balance between perplexity and inference speed, making it the optimal configuration for DuQuant.
Inference Speedup. To assess the inference speedup delivered by our DuQuant, we adopt the measurement strategy and W4A4 kernel from [2]. We evaluate the layer-wise speedup of LLaMA2 models one NVIDIA 3090 GPU, with results detailed in Table 9 and TO We set the pre-filling sequence length at 2048 and decode for 128 steps. In the pre-filling stage, DuQuant achieves a 2.08x speedup over FP16 for L.a.MA-7HH and x speedup for A2-MA-1-3B with slight variations across different batch sizes. In the decoding stage, batching the token generation phase yields high throughput without any downside (3).. Consequently, we enlarge the batch size to 64 and the results for LLaMA2-7B in Table [0 prove DuQuant achieves speedup comparable to QuaRot. More detailed analyses and end-to-end speedup available in Appendix E.
Table 9: Layer-wise wise speedup during pre- rre-filiin
Figure 5: Computational overhead analysis.
stage for 4-bit weight-activation quantization.
Model
Batch Size
Speedup
2020 Spundia111
2:30 MLLI
25.
1.95
2.122
LLaMA2-7B
2.03x
16
2.08x
2.15x
16
LLaMA2-138
2.30x
35-
16
2.34x
0.5 W1hh1
Aam pam
Far
Memory Consumption. We measure the peak memory usage of DuQuant with the W4A4 kernel on LLaMA2-7B using a single NVIDIA 3090 GPU. We process 2048 tokens for pre-filing and run 128 decoding steps, with the results listed in [T] In the pre-filling stage, DuQuant, SmoothQuant, and QuaRot achieve up to 3.2x memory reduction, while QLLM performs worse. In the decoding stage, DuQuant maintains strong memory efficiency, with superior performance.
Table 10: Decoding stage.
Table I1: Peak memory usage with a batch size of 1.
INT4, BS=64
Speedup
LLaMA2-7B
pre-filling (GB)
Saving
Decoding (GB)
Saving
FP16
FP16
15.282
13.638
SmoothQuant
1.508x x
SmoothQuant
4.782
3.196x
3.890
3.506x
QLLM
0OM
QLLM
5.349
2.857
3.894
3.502
QuaRot
1.442x
QuaRot
4.784
3.194x
3.891
3.505
DuQuant
1.321x
DuQuant
4.786
3.193x
3.893
3.503
Runtime, Our DuQuant stands out for its efficiency, surpassing other
Table 12: Quantization runtime on one NVIDIA A100.
baselines (47 34] 39 74. The block-
Model
Omni.
Affine.
QLLM
Atom
DuQuant
wise rotation ensures fast multiplica- tion between the rotation and activa- LLaMA2-13B
LLaMA2-7B
2.0h
9.1h
1.1h
20min
50s
3.2h
16.0h
1.7h
36min
71s
IJAMA2-70B
14.6h
18.6h
9.3h
3.5h
270s
tion matrices. Zigzag permutation, involving simple channel swaps, is much faster than complex algorithms like Simulated Annealing, as
discussed in Appendix [.3 Moreover, the advanced management of outliers makes DuQuant not rely on GPTQ or gradient- training. Hence, DuQuant enables a rapid quantization process shown in
Table F24
e.g., we successfully quantize LLaMA2-13B in just 71s with superior performance.
5 Conclusion
In conclusion, this paper presents DuQuant, an innovative quantization strategy for large language models (LLMs) that effectively addresses the challenge of outlier activations. By integrating rotation and permutation transformations. DuQuant effectively mitigates the impacts of both massive and normal outliers. This strategic redistribution of outliers not only simplifies the quantization process but also leads to substantial improvements in model performance. Consequently, DuQuant establishes new ofate-of-the-art results in 4-bit aeight-activation quantization scenarios. This advancement enhances the deployment of efficient LLMs in resource-constrained environments.
10

---

Acknowledgements
This work is supported in part by the National Natural Science Foundation of China (Grant No. U23B2054, 62276263, and 62371411), the Research Grants Council of the Hong Kong SAR un- der Grant GRF 11217823 and Collaborative Research Fund C1042-23GF, InnoHK initiative, the Government of the HKSAR, Laboratory for Al-Powered Financial Technologies.
References
[1] Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Towards end-to-end 4-bit inference generative large language models. arXiv preprint arXiv:2310.09259, 2023.
[2] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated Ilms. arXiv preprint arXhv:2404.00455 2024.
[3] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701 2020.
[4] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
[5] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.3303333 3432, 2013.
[6] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33: 13:187771901, 2020.
[8] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
[9] Mengzhao Chen, Wenqi Shao. Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Efficientqat: Efficient guantization-awall training for large language models. arXiv preprint 2rXiv:2407.11066 2024.
[10] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et Vicuna: open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna, lmsys. org (accessed 14 April 2023), 2023.
[11] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2924-2936, 2019.
[12] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 05400.0..55..... 2018
[13] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8):: 8-bit matrix multiplication for transformers at scale. In Conference on Neural Information Processing Systems, 2022.
11

---

[14] Tim Dettmers, Ruslan A. Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: A sparse- saanse-quantzed representation for near-lossless LIM weight compression. In The Twelfth International Confer- ence on Learning Representations, 2024.
[15] Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In Proceedings of the 4/st International Conference on Machine Learning. PMLR, 2024. URL [arXiv: 2406.02924).
https://arxin org/ abs 2406. 02924
[16] Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. Bitdistiller: Unleashing the potential of sub-4-bit Ilms via self-distillation. arXiv preprint Ttt.::4022.00631 2024.
[17] Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. Skvq: Sliding- window key and value cache quantization for large language models. arXiv preprint arXiv:2405.0622 2024.
[18] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
[19] Yuxin Guo, Siyang Sun, Shuailei Ma. Kecheng Zheng, Xiaoyi Bao, Shijie Ma, Wei Zou, and Yun Zheng. Crossmae: Cross-modality masked autoencoders for region-aware audio-viss pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26721-26731, 2024.
[20] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
[21] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. International Conference Learning Representations, 2020.
[22] Jung Hwan Heo, Jeonghoon Kim, Beomseok Kwon, Byeongwook Kim, Se Jung Kwon, and Dongsoo Lee. Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models. arXiv preprint arXiv:2309.15535 2023.
[23] Lu Hou, Quamming Yao, and James Kwok. Loss-aware binarization of deep networks. arXiv preprint arXiv: 161 1.01600, 2016.
[24] Wei Huang, Xudong Ma. Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. How good are low-bit quantized llama3 models? an empirical study. arXiv preprint MXXx:22444.14447, 2024.
[25] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient aategerarittmmetic-on inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704 2713, 2018.
[26] Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023.
[27] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[28] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07622 2023.
12

---

[29] Liang Li. Qingyuan Li, Bo Zhang, and Xiangxiang Chu. Norm tweaking: High- performmmre low-bit quantization of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18536-18544, 2024.
[30] Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai. Huazhong Yang, and Yu Wang. Evaluating quantized large language models. arXiv preprint arXiv:2402.181555 18158, 2024.
[31] Haokun Lin. Haoli Bai, Zhili Liu, Lu Hou, Muyi Sun, Linqi Song, Ying Wei, and Zhenan Sun. Mope-clip: Structured pruning for efficient vision-language models with module-wise pruning error metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3770-27380, 2024.
[32] Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00978 2023.
[33] Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Mini- cache: Kv cache compression in depth dimension for large language models. arXiv preprint 2.M.::2405.14366 2024.
[34] Jing Liu, Ruihao Gong. Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. QLLM: Accurate and efficient low-bitwidth quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL,https: htttps://openrevierr
[35] Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. IntactKV: Improving large language model quantization by keeping pivot tokens intact. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024, pages 7716 7716-7741, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 10, org /wwg///22..... findings acl. 460
v1/2024. findins--acl4666
URL
[36] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantiza- tion aware training for large language models. arXiv preprint arXiv:2305. 17888. 2023.
[37] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.0277
2024.
[38] Shijie Ma, Fei Zhu, Zhun Zhong, Wenzhuo Liu, Xu-Yao Zhang, and Cheng-Lin Liu. Happy: debiased learning framework for continual generalized category discovery. arXiv preprint arXiv:2410.06533 2024.
[39] Yuexiao Ma. Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. arXiv preprint arXiv:2403 2024.
[40] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2016.
[41] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMLR, 2020.
[42] Davide Paglieri, Saurabh Dash, Tim Rocktäschel, and Jack Parker-Holder. Outliers and calibration sets have diminishing effect on quantization of modern llms. arXiv preprint arXiv:2405.20883, 2024.
[43] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, fñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative Ilm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecure (ISCA), pages 118-132 132. IEEE, 2024.
13

---

[44] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21()):5485-551, 2020.
[45] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9))99-106, 2021.
[46] Yuzhang Shang, Zhihang Yuan, and Zhen Dong. PB-LLM: Partially binarized large language models. In The Twelfth International Conference on Learning Representations, 2024.
[47] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quan- tization for large language models. In The Twelfth International Conference Learning Representations, 2023.
[48] Mingjie Sun, Xinlei Chen, J 7icco Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402. 17762, 2024.
[49] Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization. arXiv preprint arXiv:2203. 10705, 2022.
[50] Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint arXiv:2407. 13633, 2024.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13977, 2023.
[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09256 2023.
[53] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipit: Even better Iim quantization with hadamard incoherence and lattice codebooks. arXiv preprint SiKi::2402204396 2024.
[54] Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, et al. Efficient large language models: A survey, arXiv preprint arXiv:2312.03866 1, 2023.
[55] Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, and Mi Zhang. D2o: Dynamic discriminative operations for efficient generative inference of large language models. arXiv preprint arXiv:2406. 13035,0035, 2024.
[56] Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference. arXiv preprini arXiv:2406. 18139, 2024.
[57] Haoxuan Wang, Yuzhang Shang, Zhihang Yuan, Junyi Wu, and Yan Yan. Quest: Low-bit diffusion model quantization via efficient selective finetuning, arXiv preprint arXiv:2402.03666. 2024.
[58] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprini arXiv:2403.07337, 2024.
[59] Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1648-1665, 2023.
14

---

[60] Junyi Wu, Haoxuan Wang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Ptq4dit: Post- Post-training quantization for diffusion transformers. arXiv preprint arXiv:2405.16000 2024.
[61] Yichen Wu, Long-Ka Kai Huang, and Ying Wei. Adversarial task up-sampling for meta-learning. Advances in Neural Information Processing Systems, 35:31 102-31115, 2022.
[62] Yichen Wu, Long-Kai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth International Conference on Leurning Representations, 2024.
[63] Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. Training transformers with 4-bit integers. Advances in Neural Information Processing Systems, 36:49146-49168 2023.
[64] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
[65] Haobo Xu, Yuchen Yan, Dingsu Wang, Zhe Xu, Zhichen Zeng. Tarek F Abdelzaher, Jiawei Han, and Hanghang Tong. Slog: An inductive spectral graph neural network beyond polynomial filter. In Forty-first International Conference Machine Learning, 2024.
[66] Jaewoo Yang, Hayun Kim, and Younghoon Kim. Mitigating quantization errors due to activation spikes in glu-based llms. arXiv preprint arXiv:2405. 2445,4480 2024.
[67] Lianwei Yang, Zhikai Li, Junrui Xiao, Haisong Gong, and Qingyi Gu. Mgrq: Post-training quantization for vision transformer with mixed granularity reconstruction. arXiv preprint arXiv:2406.0922 092.9, 2024.
[68] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Conference on Neural Information Processing Systems, 2022.
[69] Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01083 2023.
[70] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[71] Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternaryber: Distillation-aware ultra-low bit bert. arXiv preprint arXiv:2009. 128/2, 2020.
[72] Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL httpps//
openreview hwtp:/iwwwnneeeeeeeeee wwt. wtwts//ffoommiididim T
[73] Tianchen Zhao, Xuefei Ning, Tongcheng Fang, Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang. Mixdq: Memory- few-step eext-to-image diffusion models with necric-decouple mixed precision quantization. arXiv preprint a7X.v.2405.17873, 2024.
[74] Yilong Zhao, Chien- Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving. arXiv preprint arXiv:2310.
2023.
[75] Zhenghao Zhao Yuzhang Shang, Junyi Wu, and Yan Yan. Dataset quantization with active 2024.
learning based adaptive sampling. arXiv preprint arXiv:
[76] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.0566
2023.
[77] Yinan Zhou, Yuxin Chen, Haokun Lin, Shuyu Yang, Li Zhu, Zhongang Qi, Chen Ma, and Ying Shan. Doge: Towards versatile visual document grounding and referring. arXiv preprint arXiv:2411.17122 2024.
15

---

Appendix Overview
Section A
Related work.
Section
Theory proofs.
Section
Additional implementation details.
Section
D
More empirical results.
Section
More detailed ablation studies.
Section
Detailed comparison with QuaRot.
Section
Algorithm for rotation matrix.
Section
Limitations and broader impacts.
Section
!
More visualization examples.
A Related Work
A.1 Network Quantization
Neural networks achieve great success in recent years 665 77 38
61
31
19
62 quantization [I
63
67
[7] 57 75 is widely utilized technique aimed at reducing model size and memory usage. training (QAT) [7]
Research in this area generally falls into two main categories: quantization-aa
3 49] and post-training quantization (PTQ) 41 60 73). QAT involves training quantized model weights using additional data, often with the assistance a straight-through estimator (STE) [5]. However, the computational cost associated with QAT poses challenges, particularly for large language models (LL with millions of parameters, which necessitate significant amounts of data for retraining [36] 16 9. In contrast, PTQ has gained popularity for LLMS (54 22 69
37 due
to its efficient approach, involving the training of quantized models using a small amount of data, known calibration data [18]. However, PTQ often leads significant performance degradation, especially when employing low-bit settings [18, 53 46 30). Consequently, our work focuses on enhancing the performance of low-bit PTQ quantized models.
A.2 Post Training Quantization of LLM
Post-training quantization for LLMs can be categorized into weight-only quantization 332 28 29
and weight-activation quantization [68] 64 74]. We focus on 4-bit weight-activation quantization due to the actual speedup it provides with low-bit quantization kernels [I]. Quantizing LLMs faces challenges due to activation outlier features persisting across different tokens and layers [13] 64). Some approaches [3 74
retain small portion of crucial outlier channels at high precision (c.g., INT8), which poses challenges to hardware compatibility and leads to additional memory footprint. Other methods [64] 59 47) attempt shift quantization difficulty from activation to weight channels. However, the learnable equivalent transformation in OmniQuant (47 and the affine transform matrix in AffineQuant (39 exhibit instability as discussed in Section The channel disassembly and assembly in QLLM (34,, coupled with LoRA-tuning, incur significant time costs. Notably, these methods demonstrate poor performance under W4A4 quantization. We attribute this degradation to the ineffective handling of outlier features, especially massive outliers. Hence, we propose DuQuant effectively eliminate outlier features through rotation matrices and channel permutation, achieving state-of- of-the-art performance. In contrast with QuaRot (2) also utilizing Hadamard matrices to enhance weight-activation quantization, our approach uniquely incorporates knowledge about the actual outlier channels. Furthermore, unlike QuaRot, which relies on GPTQ [18] for weight quantization. our permutation transformation has been proven helpful and efficient, facilitating faster quantization process. The more detailed analysis and comparison with QuaRot are left in Appendix [F] In addition, unlike RPTQ (69 and SK VK (17), which use channel reordering to cluster similar activations, our method employs Permutation transformations with fundamentally different goal: to evenly distribute outliers across blocks. This balanced distribution is crucial for enabling effective secondary rotations, ultimately leading to smoother activations that facilitate easier quantization.
16

---

B Proofs
Theorem 1 (Rotation). For the activation input X E RT R E I22""x²" is a diagonal block matrix constructed as per Eqn. (3) For a specific block by let O₂(+) represent the maximum outlier of the j-th dimension d, within the input. Then, we can deduce that,
(o,,,, 1SJS2"
O2(Xb,)
15J52"
Proof. In the case of specific block b; the potential maximum value O2 (X,, Rs.). where Xs; RTx22, can be achieved by ensuring that the different j-th column outliers O2 (Xo, are located in the same t-th row of the activation input Xb,- This can be formally defined as follows,
1
max (R,, (Xo, E 15552"
(2(X,,Eaa)) 61 + + Om (Xa, Bgn))M
VM
where = and || + 8 + + 83 = 1, (Ra, is orthogonal)
Without loss of generality, let's assume that 2 and define m == max; 67. Then we can derive that 6m /2n Consequently, can obtain the following inequality.
max Ox Rd,) SS$2"
V2n O2 Edv) )- -81 +... + Oae (Xa, Eat
$ 01 Egm)
+ (O,,X,, +M. E, ))m
(1) (X,, Ega))
+ +0 (Xa, V2" - (X,,
/2"
max 151$2"
OX())
The inequality (1) holds because the switch matrix Ego has swap the largest outliers O1(X,)) in the first column, i.c., (X,,, Ex))) max Om PXa, E2()).
25MSM
Theorem 2 (Zigzag Permutation). For the activation input X E RTxCi it can be divided into K blocks, where K = Cnn/2".... Let Oj denote the max outlier of the dimension dj in X, the reordered outliers from large to small is expressed as O(1), 0(2) ,.... (CCaa) Moreover, the Mr, represents the mean value of all O₂ in the i-th block, = 1,2. K. Let == max maxf1O(i+1) - 0 1, 1.2. ..., C.n1. Then, following the zigzag permutatioon described Section 3.2 the mean value Mo, within each i-th block consistently satisfies,
Ms; 00)) (2 K 1)(2"11-1) 1)
1,2,3 ..., K.
(10)
Proof. According to the zigzag permutation described in Section 3.2 and considering the reordered
outliers (1), ((2) ...., (C.,), we redistribute the channels they occupy (i.e., 0! Jacross different
blocks. Specifically, for the i-th block, it contains the following channels,
b, = {02mk+) (0(m+1)K-+++ m 0, 0,,,..., 2 1}
{((()) 012K-1+1 0 2 K+ O****++1)
Since 5 = maxf 001 1,2, Cin 1, then we can get
1 (00)) + +(26) + +.. -2 +
2k+1
O²"K)
2"
1
< (011) + (4K - 1)6 + (8K - 1)8 + [22+1 -4)K - 110 (2" - 1)83
(11)
2n
(2" K - 12(22-1-11
< (1)) 1)
2n
Similarly, we can deduce that all My, (i 1, 1,2, ..., K) share the same upper bound after applying our zigzag permutation.
17

---

C Additional Implementation Details
In this work, all experiments are done on NVIDIA RTX 3090 GPUs for small-scale models and NVIDIA A100 GPUs for large-scale models. We sequence length 2048 for all evaluation tasks.
we randomly select 128 sampled sequences from WikiTexr2 dataset, with the sequence length of 2048. For rotation and permutation transformations, the rotation block size 2" is set to 128, and maximum greedy search steps N equals 256. We adopt once permutation times for efficciecyy. We conduct detailed ablation studies in Appendix E.2
For calibration data, following 47
E.3 E.4
Regarding quantization details, for multiplications between activations in MSA, such as Query and Key, attention outputs and Value, we apply Hadamard rotation matrix for rapid and straightforward processing. A Hadamard matrix is an orthogonal and symmetric matrix filled with elements +1/ /22". For smooth parameter a, set 0.6 for DuQuant and 0.5 DuQuant-Lwe.. We clip the maximum activation values in all projection blocks, and the clipping ratio is set to 0.9. For DuQuant we also clip the maximum values in weight matrices, with clipping ratio of 0.8. For LWC, we keep the same default epoch numbers of 20, batch size as 1, learning rate as 5e-3, and zero weight decay, as [4]
D
More Empirical Results
Table DI: Zero-shot common-sense QA (1) results of LLaMA2 models under 4-bit WA quantization.
Model
Method
PIQA
ARC-E
ARC-C
BoolQ
HellaSwag
WinoGrande
Avg.
FP16
76.88
53.54
40.53
71.13
72.96
67.25
63.72
SmoothQuant
60.17
35.23
27.13
57.92
37.08
49.57
44.52
OS+
63.11
39.10
28.84
51.30
45.93
45.66
LLMMA2-7B W4A4
OmniQuant
65.61
44.28
30.38
62.66
53.51
51.85
51.38
AffineQuant
67.36
44.23
31.91
62.75
$4.38
55.18
52.64
QLLM
67.68
44.40
30.89
58.45
56.59
51.60
Atom
69.75
47.35
34.22
62.42
63.21
56.51
55.58
DuQuant
75.24
51.89
36.77
67.86
69.54
62.12
60.57
DuQuant-iwe
75.68
50.00
37.46
69.24
69.74
63.93
61.01
FP16
79.05
57.91
44.20
69.02
76.60
69.69
66.08
SmoothQuant
62.30
40.28
30.72
60.49
42.24
49.96
47.67
OS+
64.47
41.46
32.17
59.30
51.38
49.76
LLaMA2-13B
OmniQuant
69.80
47.22
33.79
65.47
59.34
55.49
55.19
W4A4
AffineQuant
68.55
47.64
32.34
66.97
59.97
55.07
55.09
QLLM
70.46
48.48
34.39
62,80
55.41
54.31
Atom
71.16
50.89
37.88
63.91
67.51
58.40
58.29
DuQuant
77.31
55.60
41.55
66.61
73.68
66.06
63.47
DuQuant-Iwe
77.26
56.23
42.15
65.78
73.68
65.43
63.42
FP16
81.01
59.68
47.95
75.87
80.87
76.95
70.39
SmoothQuant
64.09
41.84
32.00
58.56
54.21
51.07
50.30
LLaMA2-70B
OS+
66.16
42.72
34,90
56.93
52,96
50.73
W4A4
QLLM
74.27
50.59
37.20
71.62
59.43
58.62
Du Quant
79.27
58.16
46.07
70.46
79.21
74.19
67.89
Du DuQuant-iww
79.82
59.76
46.76
73.12
79.38
74.11
68.83
Zero-shot QA Results for 4-bit LLaMA2 Models. Table showcases the zero-shot common- sense QA results for INT4 quantized LLaMA2 models. Our DuQuant method excels across various model sizes and datasets, demonstrating state-of-the-art performance in commonsense reasoning tasks. For example, DuQuant outperforms Atom by 5.43% for the LLaMA2-78 model and by 5. 188 for the LLaMA2-13B model. In contrast Atom (74), which relies on GPTQ for weight quantization and maintains 128 channels at INT8, thereby increasing memory usage, our method offers a rapid and more efficient weight-activation quantization solution through Rotation and Permutation.
MMLU Results for 4-bit Vicuna-v1.5-7B. Vicuna-v1.5 models 10 fine-tuned from LLaMA- 2 models using high-quality user-shared conversations, are considered state-of-the-art chatbots. Table D22 displays the INT4 quantization results for Vieuna-v1.5 11.-7B on the MMLU benchmarks. In comparison to SmoothQuant, OmniQuant, and Atom, DuQuant method exhibits the smallest
18

---

performance decline and maintains competitive capacities in both zero-shot and five-shot settings. These results demonstrate the effectiveness of DuQuant in generalizing to instruction-tume models.
Table D2: Zero-shot and five-shot results on the MMLU benchmark for quantized Vicuna-v1.5-78.
Model
Method
MMLU shol)
shot)
STEM
Hums
Social
Others
Avg.
STEM
Hums
Social
Others
Avg.
FP16
38.70
45,42
56.13
56.01
49.07
39.56
45.76
58.14
57.43
50.22
Vicuna-v1.5-77 W4A4
SmoothQuant
27.10
25.16
27.40
26.71
26.59
25.22
25.06
24.99
26.68
25.49
OmniQuant
27.20
24.00
27.14
25.08
25.86
29.39
24.95
27.30
26.39
Atom
30.28
34.73
38,97
40.56
36.14
31.97
35.37
40.46
40.81
37.15
Quaut
35.85
42.66
52.03
51.23
45.44
38,90
42.57
51,80
51.23
46.13
DuQuant-tve
35.18
41.91
51.28
50.52
44.72
37.34
42.21
53.07
51.76
46.10
Long-context Evaluation Results. LongBench 4 is proposed to access the long-context gen- eration ability of LLMs, which covers several key long-text application scenarios. We evaluate the 4-bit quantized Vicuna models on five different tasks. Specifically, Qasper, MultiFieldQA, and NarrativeQA (Fl score) Single-Document QA tasks; DuReader (Rouge-L score) and 2WikiMulti- QA tasks; QMSum, GovReport (F1 score) and MultiNews
hopQA (F1 score) are (Rouge-L score) are Summarization tasks; TREC (Accuracy CLS). TriviaQA (FI score), and SAM- Sum (Rouge-L score) are Few-shot Learning tasks; and RepoBench-P (similarity score) is Code Completion task. Table DD3 and Table D44 show that our DuQuant outperforms other baselines by clear margin, maintaining the ability for long context generation tasks compared with FP16 models.
Table D3: Long-context generation results the LongBench benchmark for 4-bit Vicuna-V v1.5-BB
Vicuma-v1.5-78
RepoBenchhPP
ieltiFieldQA-een
GovReport
MultiNews
DuReader
2WikiMQA
TriviaQA
FP16
48.23
38,30
27.93
26.91
25.53
18.02
82.59
SmoothQuant
6.05
OmniQuant
14.97
2.30
2,51
2.64
1.87
0.48
0.81
Atom
29.34
31.15
23.60
24.60
19.41
17.10
67.20
DuQuant
47.66
35.62
25.66
25.85
23.15
15.09
78.91
Vicuna-vi.5-7B
QMSum
MultFicldQAAzh
NarrativeQA
Qasper
SAMSum
TREC
Avg
FP16
21.07
32,56
14.96
23.27
41.06
66.00
35.88
SmoothQuant
2.00
1.75
4.11
15.00
OmniQuant
3,93
1.40
1.10
1.62
0.61
1.00
2.71
Atom
20.24
21.55
11.57
17.97
37.94
58.00
29.21
DuQuant
21.15
29.56
11.31
19.98
42.24
64.00
Table D4: Long-context generation results on the LongBench benchmark for 4-bit Vicuna-v1.5-135
Vicuna-v1.5-13B
RepoBench-P
MultFieldQQAeen
GovReport
MultiNews
DuReader
2WikIMQA
TriviaQA
FP16
43.08
42.69
28.43
26.53
27.57
29.40
86.81
SmoothQuant
11.57
1.64
2.81
3.54
6.71
1.39
1.83
OmniQuant
8.46
4.32
0.74
2.83
13.83
0.75
1.13
Atom
37.31
37.31
19.34
23.39
21.79
15.16
80.75
DuQuant
38.09
26.02
22.07
Vicuna-y1.5-13B
QMSum
MultiFieldQA-zzh
NarrativeQAA
Qasper
SAMSum
TREC
FP16
21.24
40.44
15.41
24.41
41.97
68.00
40.64
SmoothQunn
2.95
0.82
0.97
2.18
0.35
1.50
4.21
OmniQuant
1.78
1.06
0.62
0.68
0.45
9.00
4.58
Atom
20.23
28.02
8.81
17.67
38.72
59.00
33.58
DuQuant
20.72
Comparison with FP16 models on MT-Bench. We conducted additional comparisons using the MT-Bench between our INT4 quantized models and the FP16 models. As shown Table for both 7B and 13B models, our DuQuant performs comparably FP16, which further underscores the effectiveness of dual transformations in maintaining high accuracy even with reduced precision.
Table DS: More results on MT-Bench.
DuQuant FP16
Former Win
Tie
Former Loss
Vicuna-v1.5-70
36
Vicuna-v1.5-13B
43
64
Results for 4-bit Mistral-7B and Phi2-2.8B. We have extended the application of DuQuant to
include Mistral 227 and Phi2 (26) under 4-bit WA quantization. From Table D6 we can observe that
19

---

DuQuant largely surpasses other baselines, particularly with Mistral-7B. Regarding the Phi2-2.8B model, it often experiences instability in matrix multiplication between queries and values, leading to overflow issues and posing great challenges to quantization. However, while DuQuant may not perform as well as FP models, it still significantly outperforms other baselines. addition, we have visualized the massive outliers in the down projection layer of the Mistral-7B model and the feature space after our dual transformations. These visualizations are shown Figure 19 It can be observed that our DuQuant perfectly eliminates these outliers. These results underscore the effectiveness of our dual transformation approach addressing massive outliers across various types LLMs.
Table D6: Perplexity results of Mistral-7B and Phi2-2.8B under 4-bit weight-activation quantization.
Model
Method
WikiText2
C4
Model
Method
WikiText2
C4
FP16 RTN
5.25
7.75
FPI6
9.71
12.76
306.26
300.07
RTN
230.59
253.79
Mistral-7B W4A4
SmoothQuant
100.59
158.02
Phi2-2.8B
SmoothQuant
63.84
83.24
OmniQuant
5490.31
6094.82
W4A4
OmniQuant
NaN
NaN
Atom
8.65
12,43
Atom
35.72
41.26
DuQuant
5.86
8.48
DuQuant
20.65
22.49
Results for 4-bit LLaMA3-70B. As LLaMA3 models have proven to be sensitive to quantization, we apply our DuQuant to the 3LaMA3-70BB and present the results in Table D7 Due to time constraints, do add learnable weight clipping. The results demonstrate that our DuQuant- quantized models outperform SmoothQuant by 12.9% Commonsense QA tasks and significantly reduce perplexity across the WikiText2, C4, and PTB datasets. These improvements underscore the robustness of our DuQuant method when applied to the LLMMA3-70B model.
Table D7: Perplexity and QA results of LLaMA3-70B under 4-bit weight-activation quantization.
WilFlext2
C4
PTB
PIQA
ARC-E
ARC-C
BoolQ
HellaSwag
WinoGrande
Avg. 1
FP16
2.9
6.9
8.2
82.4
86.9
60.3
85.2
84.9
80.6
80.1
LLAMA3-708 W4A4
SmoothQuant
9.6
1699
75.8
64.4
63.7
DuQuant
49
8.3
80.8
57.3
81.3
82.1
76.6
To thoroughly evaluate the effectiveness of our DuQuant models, we conduct comprehensive assessments under the W6A6 quantization setting. The perplexity results for language generation tasks are displayed in TableeD8 while the zero-shot accuracy for Commonsense QA tasks is detailed in Tables D9 and D10 Our findings reveal that DuQuant not only surpasses other baselines but also achieves nearly lossless performance with FP16 models in these tasks. Interestingly, in several instances, DuQuant slightly outperforms DuQuant-uwe. This suggests that the Rotation and Permutation transformations alone are sufficient to create highly competitive quantized models under W6A6 settings, without the need for additional enhancements such as the learnable weight clipping (LWC) technique. These outcomes highlight the exceptional versatility and robustness of DuQuant across various quantization scenarios, confirming its potential leading solution in post-training quantization for large language models.
W6A6 Quantization Results.
Table D8: Preplexity (L) results on the WikiText2 and C4 datasets under 6-bit WA quantization.
Dataset
#Bit
Method
1-7B
1-13B
1-30B
1-65B
2-7B
2-13B
2-70B
FP16
5.68
5.09
4.10
3.53
5.47
4.88
3.31
SmoothQuant
6.03
4.55
3.88
6.20
5.18
3.69
OmniQuant
5.96
4.38
3.75
5.87
5.14
3.71
WikiText2
QLLM
5.89
5.28
4.30
3.73
5.91
5.08
3.55
W6A6
DuQuant
5.73
5.13
4.14
3.57
5.53
4.92
3.35
DuQuant-uwe
5.74
5.13
4.15
3.60
5,53
4.92
3,35
FP16
7.08
6.61
5.98
5.62
6.97
6.46
5,52
SmoothQuant
7.47
6.97
6.34
5.99
7.76
6.76
5.88
OmniQuant
7.43
6.84
6.22
5.82
7.48
6.74
5.91
C4
QLLM
7.34
6.82
6.17
5.80
7.31
6.71
5.76
W6A6
DuQuant
7.12
6.64
6.00
5.64
7.03
6.50
5.54
DuQuant.rwe
7.13
6.64
6.01
5.64
7.03
6.50
5.54
20

---

Table D9: Zero-shot common-sense QA (1) results of LLaMAI models under 6-bit WA quantization.
Model
Method
PIQA
ARC-E
ARC-C
BoolQ
HellaSwag
WinoGrande
Avg
FP16
77.47
52.48
41.46
73.08
73.00
67.07
64.09
SmoothQuant
76.75
$1.64
39.88
71.75
71.67
65.03
62.81
OS+
76.82
51.35
41.13
72.08
71.42
65.98
61.13
LLaMA1-BB W6A6
OmniQuant
77.09
$1.89
40.87
72.53
71.61
65.03
63.17
AffineQuant
76.60
$2.29
40.63
72.65
71.29
63.85
62.89
QLLM
77.26
$2.02
41.04
71.40
65.19
61.38
DuQuant
77.53
51.47
41.13
72.78
72.76
66.69
63.73
DuQuant-twe
77.42
52.65
40.53
71.53
72.64
67.72
63.75
FP16
79.10
59.89
44.45
68.01
76.21
70.31
66.33
SmoothQuant
77.91
56.60
42.40
64.95
75.36
69.36
64.43
LLaMAI-13B W6A6
OS+
78.29
56.90
43.09
66.98
75.09
69.22
64.92
OuniQuant
78.40
57.28
42.91
67.00
75.82
68.27
64.95
QLLM
77.91
57.70
42.92
75.02
69.14
64.54
Du DuQuant
78.62
59.51
44.03
68.44
75,98
70.08
66.11
DuQuant-uwe
79.16
59.39
43.69
75.81
69.06
65.87
FP16
80.08
58.92
45.47
68.44
79.21
72.53
67.44
SmoothQuant
77.14
57.61
42.91
65.56
78.07
69.92
65.20
LLaMAI-300 W6A6
OS+
80.14
58.92
45.05
68.02
77.96
71.98
67.01
OmniQuant
79.81
58.79
45.22
68.38
78.95
72.21
67.23
QLLM
79.65
58.08
44.11
78.38
73.24
66.69
DuQuant
79.43
59.34
44.54
70.15
78.89
72.77
67.52
DuQuant-iwe
80.09
57.95
45.05
68.72
79.17
73.09
67.35
FP16
80.79
58.71
46.24
82.29
80.72
77.50
71.04
SmoothQuant
80.25
$7.92
45.50
80.22
80.18
74.76
69.80
LLaMAI-65B W6A6
OS+
79.67
55.68
45.22
80.02
78.03
73.95
68.76
OmniQuant
81.01
58.12
46.33
80.64
79.91
75.69
70.28
QLLM
80.14
57.79
45.05
79.74
74.59
67.46
Du
80.96
59.09
46.76
82.20
80.68
77.27
71.16
Du DuQuant-iwe
80.63
58.00
46.50
82.08
80.49
76.87
70.76
Table D10: Zero-shot common-sense QA (1) results of LLaMA2 models under 6-bit WA quantization.
Model
Method
PIQA
ARC-E
ARC-C
BoolQ
HellaSwag
WinoCrande
Avg
FP16
76.88
53.54
40.53
72.96
67.25
63.72
SmoothQuant
75.57
53.62
39.93
69.54
71.76
66.14
62.76
OS+
76.22
52.74
40.70
71.89
65.19
61.35
LLMMA2-7B W6A6
OmniQuant
76.55
53.83
40.96
55.89
65.59
60.26
QLLM
77.48
$2.99
39.33
71.38
65.98
61.43
DuQuant
76.99
$2.99
40.87
70.40
72.49
67.32
63.51
DuQuant-iww
76.88
52.31
40.44
69.72
72.60
66.93
63.15
FP16
79.05
$7.91
44.20
69.02
76.60
69.69
66.08
SmoothQuant
78.29
57.41
43.86
69.50
75.02
66.93
65.17
OS+
78.29
59.13
43.34
75.37
67.56
64.74
LLaMA2-13B W6A6
OmniQuant
78.24
57.58
43.86
71.10
75.52
68.35
65.78
AffineQuant
78.35
57.58
43.34
66.73
74.71
68.59
64.88
QLLM
78.78
58.29
43.77
75.10
68.43
64.87
DuQuant
78.62
56.94
43.43
68.35
76.19
69.22
65.46
DuQuant-Iwe
78.94
$7.95
44.11
68.81
76.17
68.98
65.83
FP16
81.01
59.68
47.95
75.87
80.87
76.95
70.39
SmoothQuant
79.87
$7.32
45.65
77.13
79.01
74.03
68.84
LLaMA2-70B W6A6
OS+
79.33
$9.09
47.18
79.46
75.06
68.02
OuniQuant
80.20
60.27
46.84
80.55
76.01
68.77
QLLM
80.63
59.01
45.99
79.64
75.37
68.13
DuQuant
80.96
59.39
47.27
77.34
80.70
76.40
70.34
DuQuant-Iwe
81.18
59.26
47.78
77.86
80.68
76.95
70.62
21

---

E More Ablation Studies
E.1
Time Speedup and Memory Saving
The current generation of LLMs usually splits into pre-filling and decoding phases and deploys on two separate machines [43] Here, present more speedup and memory-saving results for these two phases achieved with the LLaMA2-7B model on a single NVIDIA RTX 3090 GPU. We set the input sequence length to 2048 and the decoding steps 256. End-to-end results for time speedup and memory savings during the pre-filling stage are shown in Table ETT and E12 We can observe that DuQuant achieves maximum speedup of 2.01 during the pre-filling phase, with speedup increasing as the batch size grows. From Table E12 DuQuant demonstrates significant memory savings, effectively reducing memory usage by up to 3.20x through quantization. For the decoding phase, we enlarge the batch size to 64 and measure speedup along with memory usage for one LLaMA2-7B layer, constrained by the 24 GB memory of the GPU. As shown Table E13 DuQuant maintains speedup and memory usage comparable to QuaRot. These results underscore the efficiency of DuQuant optimizing resource utilization, highlighting its potential enhance performance and reduce costs in deploying large language models, particularly in resource-constraing environments.
Table E11: End-to-end pre-filling speedup on LLaMA2-7B model.
Batch Size
FP16 Time
DuQuant Time
Speedup
1
568ms
294ms
1.93
2
1003ms
509ms
1.97
3
1449ms
720ms
2.01 x
Table E12: Peak memory usage during pre-filling ree-iilingg phase of LLaMA2-7B model.
Batch Size
FP16 Mem.
DuQuant Mem.
Saving Factor
1
15.28GB
4.79GB
3.20x
2
17.94GB
5.94GB
3.02x
3
20.56GB
7.10GB
2.90x
Table E13: Decoding phase results of one LLaMA2-7B layer with batch size of 64.
Method
Time (ms)
Saving Factor
Memory (GB)
Saving Factor
FP16
659
3.550
SmoothQuant
437
1.508x
1.669
2.127x
QLLM
0OM
00M
QuaRot
457
1.442x
1.678
2.116x
DuQuant
499
1.321x
1.677
2.117x
E.2 Effects of Rotation Matrix
Ablation of Rotation Block Size. To further explore the impact of rotation block size, we apply varying block sizes in the rotation matrices to both LLaMA2-7B and LLaMA2- 13B models and evaluate the perplexity of the quantized models. The results, presented in Table B14 indicate that increasing block sizes generally improves model performance. This improvement occurs because larger block sizes allow outliers to be distributed across more channels, evening out values throughout the aedivation/weigh matrix thereby enhancing quantization accuracy and performance. Additionally, quantization runtime decreases with larger block sizes, likely due to more efficient transformations during the reshaping of original activation/weight matrices. Consequently, we adopt 128 as our rotation block size for all experiments for efficiency and effectiveness.
Ablation of Rotation Times. Identifying the optimal rotation matrix R is complex challenge, so we employ a greedy search algorithm to approximate the matrix as R. We conduct an ablation study on the number of greedy steps N and summarize the results in Table E15 Initially, as N increases, the model performance improves, reflecting our ability to determine R more effectively. However, when N reaches 1024, the model begins to overfit. Consequently, we have chosen N 256 for all experiments, as it offers the optimal balance between model performance and time usage.
22

---

Table E14: Impact of rotation block size.
LLaMA2-7B
LLaMA2-13B
Block Size
WikiText2 4
C4 +
Time/s
WikiText2
C4 4
Time/s
18.69
26.48
64.4
8.81
13.03
97.7
10.77
15.04
53.8
7.02
9.68
80.8
16
8.69
11.46
48.2
6.12
8.12
75,2
32
6.96
8.85
48.3
5.61
7.35
76.2
64
6.38
8.07
50.1
5.45
7.13
74.0
128
6.28
7.90
48.6
5.42
7.05
74.0
Table E15: Impact of rotation times.
LLaMA2-7B
LLMMA2-1BB
Rotation Times
WikiText2 +
C4 4
Time/s
WikiText2
C4 +
Time/s
6.60
8.41
22.9
5.48
7.12
37.7
6.34
8.04
22.6
5.41
7.06
38.7
16
6.32
7.98
28.8
5.43
7.05
41.8
64
6.34
7.98
29.0
5.43
7.06
47.0
256
6.28
7.90
48.6
5.42
7.05
74.0
1024
6.31
8.01
129.7
5.46
7.12
179.8
E.3
Effects of Permutation Algorithm.
As discussed in Section 3.2 rotation transformations within each block are limited and unable re- distribute outliers across different blocks. To address this, we introduce permutation transformation aimed balancing outliers more comprehensively. Our primary goal to minimize the variance among different blocks, as outlined in Eqn. 4 We explore several optimization algorithms, with results detailed in Table E16 Note that the variance values are measured on activation values of the query project in the first layer of each model, and the time in the table represents the runtime of calibration. The Zigzag permutation notably reduces the variance to 3.0e-4, achieving this with minimal time expenditure and yielding competitive perplexity results. While Simulated Annealing slightly outperforms Zigzag in terms perplexity for the LLaMA2-7B model, it was significantly more time-consuming, and the marginal gains did not justify the additional complexity. Therefore, we select Zigzag permutation as our preferred method, leading to smoother outlier distribution and more effective quantized models.
Table E16: Impact of channel permutation algorithm.
I.aMM2-78
LLaMA2-13B
Permutation Method
WikiText2
C4
Variance
Time/s
WikiText2 |
C4
Variance
Timels
Permutation
7.92
10.64
3.9c-2
27.5
5.96
7.94
3.le-2
44.7
Random
6.40
8.08
4.9c-3
89.5
5.43
7.07
3.9e-3
148.6
Simulated Annealing
6.26
7.89
1.7c-4
769.6
5.42
7.06
1.5e-4
1257.8
Zigzag
6.28
7.90
3.0e-4
48.6
5.42
7.05
2.5e-4
74.0
E.4 Effects of Calibration Datasets
Ablation Different Calibration Datasets. We apply our DuQuant to quantize the LLaMA2-78 model using different calibration datasets, with results presented in Table E17 It can be observed that the selection calibration datasets has a relatively minor impact on quantization performance. This is because our method uses the calibration data solely to identify outlier channels, rather than for gradient-based parameter learning as seen methods like OmniQuant [47] and AffineQuant [39]. This ablation study underscores the robustness of our DuQuant method.
23

---

Table E17: Ablation of calibration datasets.
Eval. WikiTex12 +
LLMMA2-7B
C4 +
WikiTex12
6.28
7.90
Calib.
C4
6.25
7.87
Calibration-free Quantization. To further explore the robustness of DuQuant under varying calibration conditions, generate random calibration data within the vocabulary range of the model, setting the sample count to 256. The results, shown in Table Table]ET8] indicate that even in calibration- free settings, our method continues perform well, achieving results that are competitively close to those obtained with actual calibration data. This demonstrates that DuQuant could provide viable solution in real-world scenarios where obtaining specific calibration data challenging or impossible. In addition, our findings suggest that outliers are inherent to certain model layers, reflecting characteristics of the model weights or modules, especially in recent L.Mss. This aligns with concurrent research: [66] identified consistent massive outliers at the FFN down projection layer GLU-based LLMs, such LLaMA, Mistral, Mixtral, SOLAR, and Gemma, while [42] reported that, although OPT models are sensitive to different calibration sets, newer models demonstrate robustness to outliers and maintain stable activations. These insights reinforce the idea that outliers more linked to the internal structure of model weights and modules than to calibration data. DuQuant's ability to deliver high performance without relying on traditional calibration data creates opportunities for deploying quantized models in environments with stringent privacy requirements or limited data availability. This highlights a promising direction for future research, focusing on improving model adaptability and deployment flexibility.
Table E18: Calibration-free quantization, where we generate random data within vocabulary range.
Eval. WikiText2 $
LLaMA2-7B
C4 1
Randomly Generated
6.25 6.25
7.86
Calib.
WikiTexx2
7.87
Eval. WikiTex12 &
LLaMA2-13B
C4 J
Randomly Generated
5.45
7.05
Calib.
WikiTeex2
5.44
7.05
Ablation of Different Numbers of Calibration Samples. We utilize DuQuant to quantize the LLaMA2-7B model using varying numbers of calibration samples from the WikiText2 dataset, with results detailed in Table E19 Interestingly, the quantization performance shows low correlation with the number of samples, demonstrating the robustness of DuQuantThis stability arises because we utilize the mean activation values from these samples to construct our rotation matrices. Since we average the activations, the influence of any single, potentially non-representative sample is minimized, ensuring consistent performance. Notably. as we use mean values, the time cost of our quantization process remains constant regardless of the number of samples, enhancing the efficiency.
Table E19: Ablation of different numbers in the calibration dataset.
# of Samples
WikiText2 +
C4 |
6.29
7.88
32
6.31
7.99
6.29
7.88
128
6.28
7.90
256
6.23
7.88
24

---

Detailed Comparison with QuaRot
Table F20: Comparison of quantization settings between QuaRot and DuQuant.
Sctting
Weight
Activation
Query
QuaRot
per-channel symmetric
per-token symmetric
FPl6
DuQuant
per-channel asymmetric
per-token asymmetric
per-token asymmetric
Table F21: Evaluation results between QuaRot and DuQuant under DuQuant settings.
Model
Method
Wikiffed2
e4
PIQA
ARC-E
ARC-C
BoolQ
HellaSwag
WinoGrande
Avg T
73.08
73.00
67.07
LLaMAL-BB W444
QuaRot-RIN
7.08
8.73
74.59
48.57
36.01
68.99
65.69
58.56
46.03
QuaRot GPTQ
6.44
7,87
76.17
49.96
38.23
70.80
69.29
63.06
61.25
DuQuant
6.40
7.84
76.44
50.04
38.99
70.98
69.39
64.72
61.76
DuQuant-owe
50.04
70.09
69.82
62.59
61.18
FP16
5.47
6.97
76.88
53.54
40.53
71.13
72.96
67.25
63.72
LLaMA2-7B W4A4
QuaRor-RTN
9.66
11.98
69.48
46.25
32.76
64.80
60.75
56.67
44.04
QuaRet-GPTQ
67.89
68.87
61.33
59.8 85
DuQuant
6.28
7.90
75.24
$1.89
36.77
67.
54
62.12
60.57
DuQuant-twc
6,08
7.79
75.68
50.00
37.46
69.24
69.74
63,93
61.01
FP16
6.14
8.88
80.85
77.78
53.41
81.28
79.16
72.84
74.22
LLaMA3-8B W444
QualRtt
63.46
62.75
QuaRot-GPQQ
8.69
12.40
74.54
67.38
40.61
70.43
70.47
65.11
64.76
DuQuant
DuQuant-uwe
8.06
11.29
76.22
70.41
43.69
74.34
73.87
67.80
67.72
Table F22: Evaluation results between QuaRot and DuQuant under QuaRot settings.
Model
Method
WikiTex12
C
PIQA
HellaSwag
LAMBADA
Avg f
FP16
5.47
6.97
79.11
69.06
75.99
74.58
46.25
73.90
69.82
LLaMA2-7B
QuaRot-RTN
57.27
58.26
W444
QuaRot-GPTQ
6.1
76.77
63.77
72.16
69.87
40.87
70.39
65.64
QuaRot Setting
DuQuant
6.23
7.91
76.28
66.93
72.96
69.99
40.53
69.61
66,05
DuQuantawe
6.01
7.67
77.64
72.97
70.37
41.81
69.53
66.69
FP16
4.88
6.46
80.47
72.22
79.39
77.48
49.23
76.75
72.59
LLaMA2-13B
QuaRat- RTN
6.09
77.37
67.32
73.11
70.83
43.69
70.66
67.16
W444
QuaRot-GPTO
5.4
78.89
70.24
76.37
72.98
46.59
73.67
69.79
QuaRot Setting
DuQuant
DuQuantsawe
5.27
6.93
78.73
70.88
77.20
74.07
47.27
73.96
70.35
Table F23: Matrices comparison between DuQuant and QuaRot under W4A4 quantization.
Model
LLaMA2-7B
LLaMA2-13B
Dataset
WikiText2
C44
WikiText2
C4
QuaRot
9.66
11.98
6.73
8.69
DuQuant 7.92
10.64
5.96
7.94
Table F24: Quantization runtime comparison on single NVIDIA A100 80G GPU,
Model
LLaMA2-7B
LLaMA2-13B
LLaMA2-70B
QuaRot
20min
36min
5.1h
DuQuant
50s
71s
270s
In this section, we present detailed comparison between our DuQuant and QuaRot 2 QuaRot employs Hadamard matrices mitigate outliers in activations and utilizes the GPTQ algorithm for weight quantization to achieve competitive performance. However, our DuQuant method demonstrates several distinct advantages:
Effective Use of Prior Knowledge: DuQuant leverages prior knowledge to accurately target and eliminate outliers through multiple rotations, achieving smoother activation distribution compared to the Hadamard transformation, demonstrated in Figure
25

---

Efficient Channel Permutation: Our channel permutation not only further smooths outlier features but also benefits from rapid implementation, enhancing overall performance.
Simultaneous Weight Matrix Smoothing: Unlike QuaRot, DuQuant directly and efficiently
smooths the weight matrix, avoiding the time-consuming GPTQ algorithm and accelerating the
quantization process, as demonstrated high quantization efficiency in Table F24
Experimental results underscore the superiority of DuQuant over QuaRot. We first summarize the experimental setting differences between the original paper of QuaRot with ours in Table [ For fair comparison, we reproduce the QuaRot under 4-bit per-channel weight and per-token activation asymmetric quantization. Table F2T displays the perplexity (PPL) and zero-shot accuracy for mod- els LLaMAI-7B, LLaMA2-1 7B, and LLaMA3-8B. Our DuQuant method consistently outperforms QuaRot-RTN across all benchmarks, showcasing our advanced weight matrix management. Further- more, compared to QuaRot-GPTO. DuQuant and DuQuant-Lwe achieve better average accuracy across six QA tasks and demonstrate superior performance the WikiText and C4 datasets, particularly for LLaMA3-8B. Moreover, we further provide the results DuQuant under the setting utilized the original paper of QuaRot in Table F22
DuQuant still surpasses QuaRot by large margin.
Additionally, we assess the effectiveness of the rotation matrices utilized DuQuant which incorpo- rate prior knowledge against the Hadamard matrices used in QuaRot. We omit the permutation step DuQuant and directly contrast with QuaRot-RTN. Results F22 show that our DuQuant without permutation outperforms QuaRot by clear margin, which confirms that rotation trans- formation is more effective than Hadamard by leveraging prior knowledge. It is worth noting that because Hadamard matrix is orthogonal and symmetric, multiplies by itself to yield the identity matrix. other words, the Hadamard matrix is not suitable for greedy searches aimed at finding smaller outliers. These findings differentiate DuQuant from QuaRot and highlight the effectiveness of our approach in managing outliers for post-training quantization of large language models.
G Algorithm for Rotation Matrix
Algorithm 1 Construction of the Rotation Matrix
Input: Pre-initialized rotation matrix R. greedy search steps activation matrix X with shape of [T, Cinl
Output: Rotation matrix R
function ret_rotation__ matrix (X, R, N)
I: T. Ci = shappe
2: R = eeye(Cinn)
3: max., X
for in 1, ...,
channel_me maanel_maxx-e = X.aass)..mx((tfff = valuue
6:
arg channhanneleaceeeeeeeaaaaaaaneea max)
7:
Obtain randomly initialized orthogonal matrix Q' with shape of [Cmn 1, Ca - 1]
8:
Q' concat([zeros((.r Ci 1). Q' dim=l)
9:
Q concaa((zenos(1,, C, Q'], dim=0)
10:
00 1
matmul(R. Q)
12:
R'[: outlier outlier_channel] R 0] R([: 0J. R'[: outlier_channe
channel. :]. R' R'10, :] R' R'[0, :]. R'[outlier_cha channel_ R' ['loutterr_caanel
14:
R = matmul(R. R')
15:
X matmul(X.. R')
if max.,; [X,y] then
17: R R
a= maxi, [X,]]
19: end if
20:
end for
21: return R
we describe operations on Query, Key, and Value states as per-tensor quantization for consistency. To be precise, these operations effectively applied hea-head basis.
'Following prior works 47
34 39
26

---

H Limitations and Broader Impacts
The primary limitation of our method is the lack of specialized strategy for calibra- tion data selection. We adhere to established practices [47 39 34 74 [ by randomly selecting 128 samples from the WikiText2 dataset to compute the mean embeddings that inform our rotation matrix and zigzag permutation order. We also explore the possibility of calibration-fre quantization and show some promising results. However, further investigating more tailored choices for calibration data can potentially enhance the performance of our quantized models. We leave this for future study.
Limitations.
Broader Impacts. Our work identifies the presence massive outliers in down- cwwn-projection layer of FFN modules, which significantly complicates low-bit weight-activation quantization. To address this challenge, we implement combination of rotation and permutation matrices effectively smooth both massive and uniform outliers, proving both fast and effective. Consequently, we establish new state-of-the-arr for INT4 weight-activation post-training quantization methods. Our approach aims to accelerate large language models and reduce memory usage during deployment, offering substantial benefits to the field LLM research. These advancements could lead to more efficient and accessible LLM applications, facilitating broader usage and enabling more sustainable AI implementations.
I More Visualizations
We provide additional visualizations normal and massive outliers in various models (LLaMAI, LLaMA2, Vicuna-v1.5 Mistral) from Figure [o Figure In each figure except Mistral, the left side illustrates changes in normal outliers before and after applying our rotation and permutation transformations, while the right side shows the changes in massive outliers before and after trans- formations. It is evident that massive outliers consistently occur in the pooneprooection layer of the FFN module across all models, supporting our findings discussed in Section Conversely, normal outliers appear in different modules within the transformation block. For instance, Figure [3 shows normal outliers the up-projection layer of the FFN module in LLaMAI-131 Significantly, both massive and normal outliers are reduced markedly after our rotation and permutation transformations, leading to easier quantization of activations. This underscores the effectiveness of our DuQuant in managing outlier features across diverse LLM models.
0225
0.551
0.100
1103
UL Layerts_Atmn O
Permutztion
(c) LLaMA3, Laper2, Down, Proj
lunation Permocation
Massive Outliers
Figure I1: Activation change with the use of our DuQuant for LLaMA1-1 1-7B.
3.17
(1) LLAAALLTRRR L3R Layerl_At MRn.K K.P.K.P
[b) After Rotetion and Permutation
(6 LLaMAL_13 Lavers, WWN.FF.OOOo Proj
After Rotation and Pormutation
Normal Outtiers
Massive Outliers
Figure 12: Activation change with the use our DuQuant for LLaMAI-
27

---

0.35
6363
E00
Permutztion
138
Rocation
Normal Outliers
Massive Outliers
Figure 13: More examples of activation change with the use of our DuQuant for LLaMA1-13B.
1.
1.
3000
(4] LaaAAL 558, Layor11 Attnn Kin,K_Pe Iroj
[6] After Rotutioon on and that nunutttion
(4) LLAMMA L00. Laver FNN, Down. Proy
(40 After Rotation and Permutation
Normal Outtlers
Massive
Figure 14: Activation change with the use our DuQuant for LLaMA1-65B.
335
uat
215
125
2.8
Layori_AArs J. P.Pro
After Pormutziion
(C LLAMAZ
Layer3, Down. Proj
Permctation
Outtiers
Massive Outliers
Figure 15: Activation change with the use of our DuQuant for LLaMA2- 13B.
28

---

2339
6179
130
601
0m0
708, Layort,
[B)
(GLLLMMO_700) Laver2
Massive Outliers
Figure 16: Activation change with the use of DuQuant for LLaMA2-70B.
220
4.75
2.00
1.20
011
139
1.34
110
100
030
v1.5 Layort, Proj
(b) After Rotation and For nunttttion
[r] Milunnn vi 5, Eapert FFN Dowe _an]
(4) After Ronation and Permutetion
Normal Outtlers
Massive
Figure Activation change with the use our DuQuant for Vicuna-v1.5-1
2.4
109
313
0.5
(a) Vicuna, vt. Laver Altin Proj
[b] After Rotation and Por normtaaioon
Vinuna,vv
FFN Dowwn
After Rotation and Permubeiion
Normal Outtlers
Massive Outliers
Figure 18: Activation change with the use of our DuQuant for Vicuna-vl.5-1333
126
Misral FFN
Massive Outilers
Massive Outtiers
Figure 19: Massive activation change with the use of our DuQuant for Mistral7B.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We state the contributions in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the main limitations in Appendix
H
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification:
Please refer to Appendix B for related assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Implementation details are provided in Appendix code at https:///tttiiii http://ggithuub...//iii///// Haamm/Hsu1223 2//u////uuuant
C We have released our
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer:
[Yes]
Justification:
We have released our code at htttpp///gicuub..co//bblllllll
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee //nips.com htte://alllllgiiiee///oeeeuaaass ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please refer to Section and Appendix C for experimental settings. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our study used fixed seed for all quantization operations, following standards in post-training quantization, and thus did not report statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
32

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еπтог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments are done on NVIDIA RTX 3090 GPUs and A100 GPUs.
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
inne?
Answer: [Yes]
Justification: We confirm the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impacts of work in Appendix H
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
33

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all datasets and models utilized in our experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppsiipteeeee www.ree.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
34

---

Answer: [NA]
Justification: This work does not release new assets.
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects. The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
35


Reviews:
[{"review_id": "gSreW0tN2w", "reviewer": "Reviewer_x516", "summary": "The paper proposes a new LLM quantization method named DuQuant (for Dual transformations Quantization). This method is able to quantize the weights and activations of an LLM to 4 or 6 bits without losing significant precision.\n\nThe paper identifies the issue of Massive outliers in the activations of a LLM. These are outliers within the outliers, having magnitudes in the order of 100-1000. Traditional quantization methods, such as SmoothQuant, cannot deal with these outliers, because they attempt to address the errors locally. DuQuant, on the other hand, uses a sequence of rotation, permutation, and another rotation to spread these outliers out across many weights, resulting in a more accurate capture.\n\nThe results are supported with theoretical results as well as experiments. DuQuant outperforms the baselines on perplexity, QA and MT-bench.", "questions": "No outstanding questions.", "limitations": "No limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to understand (apart from a few minor typos). I especially like figure 2 that give a strong intuition to how the method works and also why it eliminates Massive outliers. The algorithm, the justification and the results are easy to follow.\n\nThe experimental results are very convincing. They demonstrate an improvement in prediction quality in W4A4 across the board. The paper shows extensive experiments looking at various Llama models as well as Vicuna. Compared to the baselines (SmoothQuant, AffineQuant, OmniQuant, Atom, QLLM), DuQuant always performs the closest to the uncompressed model. This result is shown across various tasks: perplexity, QA and MT bench.\n\nThe paper also includes ablations and runtime costs of the required transforms: the transforms incur an additional 8.9-9.3% computational cost at inference time compared to standard W4A4.", "weaknesses": "A shortcoming of the paper is the lack of close comparison to QuaRot. QuaRot is a very similar technique and it deserves to be in the related works section as well as be a baseline in the experiments. While there is a detailed comparison in the appendix, the work should also be mentioned in the main body.\n\nThe paper could be improved at places, but I don't consider any of these shortcomings a major issue:\n* The paper has numerous typos (eg.: Line 11: Typo, Line 188: typo, Line 313: Figure reference missing). Please proofread.\n* The paper is extremely dense, its sometimes difficult to distinguish the figure captions from the main body. Please use proper paddings for tables and figures where possible.\n* The description of how the rotation matrix is constructed could be more detailed (Lines 141 - 147). Given that this is a core algorithmic contribution of the paper, it should be clear to the reader how it works. Perhaps a figure or pseudocode would be helpful here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time in dealing with our work. We will answer the question and discuss point by point as follows. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.\n\n**W1**: The comparison with the QuaRot part should be highlighted in the main text.\n\n> - Thanks for the suggestion. We acknowledge that QuaRot is an important baseline. As suggested, we will move some of the comparison experiments from the appendix to the main text and highlight them in the related work section.\n>\n> - To further compare with QuaRot, we implemented DuQuant under **QuaRot original quantization settings**, which is different from the 4-bit per-token activation quantization and per-channel weight quantization setting used in our paper. We compared DuQuant with **the original results reported in QuaRot paper**. It can be observed from the following table that DuQuant still surpasses QuaRot on both PPL and QA evaluations.\n>\n>   | LLaMA2 W4A4        | Method      | WiKi↓    | C4↓      | PQ↑   | WG↑   | HS↑   | A-e↑  | A-c↑  | LA↑   | Avg↑      |\n>   | ------------------ | ----------- | -------- | -------- | ----- | ----- | ----- | ----- | ----- | ----- | --------- |\n>   | 7B QuaRot Setting  | FP16        | 5.47     | 6.97     | 79.11 | 69.06 | 75.99 | 74.58 | 46.25 | 73.90 | 69.82     |\n>   |                    | QuaRot-RTN  | 8.37     | -        | 72.09 | 60.69 | 65.40 | 58.88 | 35.24 | 57.27 | 58.26     |\n>   |                    | QuaRot-GPTQ | 6.10     | -        | 76.77 | 63.77 | 72.16 | 69.87 | 40.87 | 70.39 | 65.64     |\n>   |                    | DuQuant     | 6.23     | 7.91     | 76.28 | 66.93 | 72.96 | 69.99 | 40.53 | 69.61 | 66.05     |\n>   |                    | DuQuant-LWC | **6.01** | **7.67** | 77.64 | 67.80 | 72.97 | 70.37 | 41.81 | 69.53 | **66.69** |\n>   | 13B QuaRot Setting | FP16        | 4.88     | 6.46     | 80.47 | 72.22 | 79.39 | 77.48 | 49.23 | 76.75 | 72.59     |\n>   |                    | QuaRot-RTN  | 6.09     | -        | 77.37 | 67.32 | 73.11 | 70.83 | 43.69 | 70.66 | 67.16     |\n>   |                    | QuaRot-GPTQ | 5.40     | -        | 78.89 | 70.24 | 76.37 | 72.98 | 46.59 | 73.67 | 69.79     |\n>   |                    | DuQuant     | 5.39     | 7.05     | 78.51 | 70.88 | 76.80 | 74.62 | 48.21 | 73.92 | **70.49** |\n>   |                    | DuQuant-LWC | **5.27** | **6.93** | 78.73 | 70.88 | 77.20 | 74.07 | 47.27 | 73.96 | 70.35     |\n\n**W2**: Typos and article typesetting. Detailed description of how to construct rotation matrix.\n\n> - Thank you for your detailed feedback. We will diligently correct the typos and enhance the layout in our revised manuscript to improve its readability.\n> - Regarding the construction of the rotation matrix, we have provided a detailed pseudo code below and will involve this part in the revised manuscript to elucidate the process more clearly for the readers.\n>\n> ```\n>   INPUT: pre-initialized rotation matrix R0, greedy search steps n, activation matrix X with shape of [N, C]\n>   OUTPUT: rotation matrix R\n>   FUNCTION get_rotation_matrix(X, R0, n)\n>   \tR = eye(C) # size: [C, C]\n>   \tfor i in 1...n: # greedy search loop\n>   \t\tchannel_max = X.abs().max(dim=0).values # size: [C]\n>   \t\toutlier_channel = argmax(channel_max)\n> \n>   \t\tObtain randomly initialized orthogonal matrix Q' with the shape of [C-1, C-1]\n>   \t\tQ' = concat([zeros(n-1, 1), Q'], dim=1)\n>   \t\tQ = concat([zeros(1, n), Q'], dim=0)\n>   \t\tQ[0, 0] = 1\n>   \t\tR' = matmul(R0, Q)\n> \n>   \t\tR'[:, outlier_channel], R'[:, 0] = R'[:, 0], R'[:, outlier_channel] # swap columns\n>   \t\tR'[outlier_channel, :], R'[0, :] = R'[0, :], R'[outlier_channel] # swap rows\n>   \t\tR = matmul(R, R')\n>   \t\tX = matmul(X, R')\n>   \treturn R\n> ```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8PlgVEJJ0a", "reviewer": "Reviewer_nwuk", "summary": "The paper explores new approaches in LLM quantization. The work tries to address the performance degradation due to massive outliers in the weights. The work show competitive performance across different settings, up to 4-bit weight activation quantization. The work also provides solid experiments and visualization on the effectiveness of the proposed methods.", "questions": "From the benchmarks, it is shown that the proposed method almost matches the fp16 results (unquantized). It would be interesting to see the mt-bench of the quantized vs fp16 model to see if such an observation still holds.", "limitations": "NA", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I think the paper is well-written. The experiments, plots, and tables are well clearly tied to the story on large/abnormal outlier in the activation. \n2. The work is easy to follow, especially in the methodology. \n3. The results in the tables show that work offer very competitive performance comparing to previous methods.", "weaknesses": "1. I am a bit puzzled by the results in Table.1 vs Table.2. It is shown that lower perplexity does not necessarily reflect in higher benchmark accuracy. This conflict exists within DuQuant (DuQuant vs DuQuant_LWC) and other methods like AffineQuant,OmniQuant vs QLLM. The authors might want to consider adding some justification to this conflict to further improve the soundness of the tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: Confusion Regarding the Discrepancies Between Table 1 and Table 2 Results.\n\n> - We would like to provide individual clarifications for the results in Table 1 and Table 2 below and explain the reasons behind their discrepancies.\n> - **Table 1** presents perplexity (PPL) results on the WikiText-2 and C4 datasets, which are typically used to evaluate the language generation capabilities of LLMs. **Table 2** shows the results of the Common Sense QA task, which focuses on the model's comprehension abilities, different from the generation abilities PPL focused on. As Table 1 and Table 2 **emphasize different aspects of model performance**, this explains the conflicting results between the two tables.\n> - In addition, PPL usually reflects how well a model predicts a sequence of words, and it **cannot** measure the model's effectiveness in handling sequence-level tasks in practical applications [1, 2]. To further evaluate our model's generative capabilities, we have added a comprehensive comparison of DuQuant against other state-of-the-art baselines on the **LongBench** [1], which includes a variety of **generative tasks** to provide a broader evaluation. The W4A4 results are presented as follows:\n>  \n>   |Vicuna|Setting|RepoBench-P|MultiFieldQA-en|GovReport|MultiNews|MultiFieldQA-zh|2WikiMultihopQA|\n>   |-|-|:-:|:-:|:-:|:-:|:-:|:-:|\n>   |7B|FP16|48.23|38.30|27.93|26.91|32.56|18.02|\n>   ||SmoothQuant|25.92|4.66|2.62|6.05|0.88 |2.02|\n>   ||OmniQuant|14.97|2.30|2.51|2.64|1.40|0.48|\n>   ||Atom|29.34|31.15|23.60|24.60|21.55|17.10|\n>   ||DuQuant|47.66|35.62|25.66|25.85|29.56|15.09|\n>   |13B|FP16|43.08|42.69|28.43|26.53|40.44|29.40|\n>   ||SmoothQuant|11.57|1.64|2.81|3.54|0.82|1.39|\n>   ||OmniQuant|8.46|4.32|0.74|2.83|1.06|0.75|\n>   ||Atom|37.31|37.31|19.34|23.39|28.02|15.16|\n>   ||DuQuant|38.09|44.12|26.97|26.59|30.85|22.07|\n> \n>   |Vicuna|Setting|TriviaQA|QMSum|LSHT|DuReader|NarrativeQA|Qasper|SAMSum|TREC|Avg|\n>   |-|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n>   |7B|FP16|82.59|21.07|22.25|25.53|14.96|23.27|41.06|66.00|39.21|\n>   ||SmoothQuant|1.62|2.00|0.00|4.24|1.75|4.11|1.55|15.00|4.62|\n>   ||OmniQuant|0.81|3.93|0.00|1.87|1.10|1.62|0.61|1.00|1.56|\n>   ||Atom|67.20|20.24|17.25|19.41|11.57|17.97|37.94|58.00|33.19|\n>   ||DuQuant|78.91|21.15|19.00|23.15|11.31|19.98|42.24|64.00|**37.25**|\n>   |13B|FP16|86.81|21.24|24.00|27.57|15.41|24.41|41.97|68.00|40.77|\n>   ||SmoothQuant|1.83|2.95|0.00|6.71|0.97|2.18|0.35|1.50|2.73|\n>   ||OmniQuant|1.13|1.78|0.00|13.83|0.62|0.68|0.45|9.00|3.93|\n>   ||Atom|80.75|20.23|21.00|21.79|8.81|17.67|38.72|59.00|30.61|\n>   ||DuQuant|83.04|20.72|23.75|26.02|13.36|18.93|42.67|66.50|**38.75**|\n>  - From the table, our DuQuant outperforms other baselines by a clear margin, representing the superior ability for long context generation tasks. \n> \n> [1] Longbench: A bilingual, multitask benchmark for long context understanding, ACL 2024.\n>\n> [2] Do long-range language models actually use long-range context? EMNLP 2021.\n\n**Q1**: MT-Bench evaluation between DuQuant and FP16 models.\n> - As suggested, we conducted additional comparisons using the MT-Bench between our INT4 quantized models and the FP16 models. The results are presented in the table below.  \n>\n> | DuQuant vs FP16 | Former Win | Tie  | Former Loss |\n> | --------------- | ---------- | ---- | ----------- |\n> | Vicuna-7B       | 36         | 56   | 68          |\n> | Vicuna-13B      | 43         | 53   | 64          |\n>\n> - The results indicate that our **quantized models perform comparably to FP16**, underscoring the effectiveness of our dual transformation approach in maintaining high accuracy even with reduced precision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mAchsqZv8y", "reviewer": "Reviewer_XTv7", "summary": "The paper presents a new post-quantization method (DuQuant) that targets low-precision (4-bit / 6-bit) weight and activation quantization. The authors show how the presence of massive outliers affects quantization when using existing methods (smoothing is not sufficient with SmoothQuant / OmniQuant training is not stable for layers exhibiting massive outliers). They propose their method for better handling both normal and massive outliers - by utilizing orthogonal rotation and permutation matrices, with a simple zigzag permutation scheme for a better / more even distribution of outliers. They provide theoretical proofs for both their rotation and zigzag-permutation operations grounding their proposed algorithm. They showcase the strength of the propsed methods as it can be enabled with simple quantization, and not rely on expensive quantization methods like GPTQ to achieve new state of art quantized models across a range of different models.", "questions": "1. Many times the perplexity numbers and downstream performance are not 1:1 correlated. This most likely has to do with the standard error of the downstream tasks? If so, it will be good to clarify this for the reader.\n2. Figure 5 is not clear in explaining the different settings - what do Perm 0, 1, and 2 correspond to? Also in the paragraph above the figure, in lines 312-313, it is not clear which Figure is being referred to.\n3. For Table 6, it will be better to explain how to read the table. It took multiple passes to understand the full setup of the ablations and the associated results in the table.\n4. It is surprising that the method is largely calibration free - indicating that the outliers are more a property of the model weights, and activations are suffering as a by-product of this? can the authors clarify their intuition around this.", "limitations": "The authors address the limitations and broader impacts of their work. One thing they do mention in the checklist is reporting of statistical signficance - which I do not see anywhere in the paper. Can authors point to where these results are? Or equivalently change the checklist to reflect this.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a theoretically grounded approach to low-precision (4-bit / 6-bit) quantization of LLMs.\n2. The paper showcases how massive outliers are challenging for existing methods to adapt to (for e.g., SmoothQuant just fails and OmniQuant sees unstable gradients in training layers with massive outliers). The authors also particularly show that the down-projection layers in FFNs particularly face massive outliers - which inhibit effectively their quantization.\n3. The authors propose a new RAP (rotation and permutation) based method for enabling more even distribution of outliers from the activations to weights.\n    - They first start with the smoothing operation proposed in SmoothQuant, through ablations they show this helps get better post-quant model quality.\n    - They follow this up with a rotation operation, with the constraints that it should be able to as evenly distribute the outliers through the matrix multiplications. They show that a single rotation process cannot handle this effectively and design an iterative but greedy process that solves this using block-diagonal rotation matrices.\n    - Followed by this, they propose a new zigzag permutation operation to evenly distribute large outliers across different blocks. Finally, they apply another rotation to ensure that the outliers are maximally reduced and spread across the weights.\n4. They follow the proposal of the method through theoretical analysis, showing how their methods will results in either optimal results / have bounds in each phase of the smoothening to ensure maximal quality.\n5. They propose two variants of the algorithm - one standard with specific activation / weight clipping coeffs and another with LWC enabled from the OmniQuant algorithm. They follow these with 4-bit and 6-bit results for the LLaMA-1,2,3 and Vicuna-1.5 models, showcasing that their method outperforms other existing methods in these settings.\n6. They provide a comparison with the recently proposed QuaRot method and how their proposed algorithm performs better than the algorithm, while being competitive for implementation performance.", "weaknesses": "1. Most of the evals (except MT-Bench) are logit-based evals (and not generative). This has a side effect of hiding some of the inherent limitations of low-precision quantization algorithms (i.e., error accumulation across generated tokens).\n2. One limitation of the proposed benchmarks (for implementation performance) is that they measure the prefill performance, but do not show any generation performance. This typically dominates over prefill performance.\n3. One thing is the reporting of results is not consistent across the models - for eg. some tables use Atom for reporting model quality at 4-bits, but some models do not report this performance. While the benefits of the method are clear given the higher accuracies on downstream tasks, it is difficult to judge how the differences translate on models of higher quality (e.g. Llama-2 vs Llama=3 70B).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point. Please kindly let us know whether you have any further concerns.\n\n**W1**: More evaluations on generative tasks.\n> - To better access the generative ability of quantized models, we evaluate DuQuant on LongBench and report INT4 results below:\n>  \n>   |Vicuna|Setting|RepoBench-P|MultiFieldQA-en|GovReport|MultiNews|MultiFieldQA-zh|2WikiMultihopQA|\n>   |-|-|:-:|:-:|:-:|:-:|:-:|:-:|\n>   |7B|FP16|48.23|38.30|27.93|26.91|32.56|18.02|\n>   ||SmoothQuant|25.92|4.66|2.62|6.05|0.88 |2.02|\n>   ||OmniQuant|14.97|2.30|2.51|2.64|1.40|0.48|\n>   ||Atom|29.34|31.15|23.60|24.60|21.55|17.10|\n>   ||DuQuant|47.66|35.62|25.66|25.85|29.56|15.09|\n>   |13B|FP16|43.08|42.69|28.43|26.53|40.44|29.40|\n>   ||SmoothQuant|11.57|1.64|2.81|3.54|0.82|1.39|\n>   ||OmniQuant|8.46|4.32|0.74|2.83|1.06|0.75|\n>   ||Atom|37.31|37.31|19.34|23.39|28.02|15.16|\n>   ||DuQuant|38.09|44.12|26.97|26.59|30.85|22.07|\n> \n>   |Vicuna|Setting|TriviaQA|QMSum|LSHT|DuReader|NarrativeQA|Qasper|SAMSum|TREC|Avg|\n>   |-|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n>   |7B|FP16|82.59|21.07|22.25|25.53|14.96|23.27|41.06|66.00|39.21|\n>   ||SmoothQuant|1.62|2.00|0.00|4.24|1.75|4.11|1.55|15.00|4.62|\n>   ||OmniQuant|0.81|3.93|0.00|1.87|1.10|1.62|0.61|1.00|1.56|\n>   ||Atom|67.20|20.24|17.25|19.41|11.57|17.97|37.94|58.00|33.19|\n>   ||DuQuant|78.91|21.15|19.00|23.15|11.31|19.98|42.24|64.00|**37.25**|\n>   |13B|FP16|86.81|21.24|24.00|27.57|15.41|24.41|41.97|68.00|40.77|\n>   ||SmoothQuant|1.83|2.95|0.00|6.71|0.97|2.18|0.35|1.50|2.73|\n>   ||OmniQuant|1.13|1.78|0.00|13.83|0.62|0.68|0.45|9.00|3.93|\n>   ||Atom|80.75|20.23|21.00|21.79|8.81|17.67|38.72|59.00|30.61|\n>   ||DuQuant|83.04|20.72|23.75|26.02|13.36|18.93|42.67|66.50|**38.75**|\n\n**W2**: The generation stage performance.\n\n> - As the prefill phase is usually compute-bound while the decoding phase is known to be memory-bound [1], we compare the **memory consumption reduction** of DuQuant with other baselines during the generation stage. Evaluations were conducted on a 3090 with batch size 1.\n>\n>   |LLaMA2-7B, INT4|Memory (GB)|Saving Factor|\n>   |-|-|-|\n>   |FP16|13.638|-|\n>   |SmoothQuant|3.890| 3.506x|\n>   |QLLM|3.894|3.502x|\n>   |QuaRot|3.891|3.505x|\n>   |DuQuant|3.893|3.503x|\n> \n> [1] Quarot: Outlier-free 4-bit inference in rotated llms, arXiv 2024.\n\n**W3**: Baselines across all models.\n> We acknowledge the omission of some baseline results for the LLaMA2-70B and LLaMA3-70 models. This is because,\n>   - We encountered NaN perplexity results on the LLaMA2-70B and LLaMA3-70B models for some baselines, like Atom, leading us to exclude these results from QA task evaluations.\n>   - Possibly due to inadequate management of massive outliers, AffineQuant and OmniQuant experienced **instability** when learning on the 70B models, often resulting in gradient explosions.\n\n**Q1**: The discrepancies between PPL and other downstream tasks.\n> - PPL is utilized to assess the generation abilities of LLMs, while downstream tasks like QA in our paper mainly evaluate the comprehension abilities of LLMs. They focus on different aspects of model capacities, which may result in discrepancies between tasks. In addition, PPL might not be a reliable evaluation to reflect the model’s effectiveness in real-world tasks [2]. Thus, to better evaluate DuQuant under practical applications, we experiment on LongBench as your suggestion in W1.\n>\n> [2] Longbench: A bilingual, multitask benchmark for long context understanding, ACL 2024.\n\n**Q2 & 3**: Detailed illustrations of Figure 5 and Table 6.\n> - We apologize for any confusion caused by the unclear descriptions and will clarify these illustrations in the revised paper.\n> - **Figure 6**: This figure shows ablations of rotation and permutation frequencies in DuQuant. \"Perm 0\" indicates a single rotation, \"Perm 1\" signifies two rotations with one channel permutation, and \"Perm 2\" includes three rotations with two permutations. Results show that \"Perm1\" offers the best balance between PPL and inference speed, which we adopted as the final configuration in DuQuant.\n> - **Table 5**: The table presents ablations on four distinct operations within DuQuant. A check mark indicates the inclusion of an operation. The configurations tested are 1) only the smoothing technology like SmoothQuant; 2) one rotation following the smoothing operation; 3) a sequence of rotation, permutation, and another rotation without smoothing; and 4) the full DuQuant approach. These results underscore the contribution of each component to the overall effectiveness of DuQuant.\n\n**Q4**: Discussion about calibration-free experiments.\n> - Our findings in Appendix E.4 demonstrate that DuQuant does not depend on specific calibration data, suggesting that outliers are inherent to certain model layers and are **characteristic of the model weights or modules**. This is supported by two recent works:\n>   -  [3] identifies consistent massive outliers specifically at the FFN down projection layer in GLU-based LLMs, such as LLaMA, Mistral, Mixtral, SOLAR, and Gemma.\n>   -  [4] investigates the impact of calibration sets on quantization, finding that while OPT models are sensitive to varying calibration sets, newer models like Llama, Command-R, and Mistral show robustness to outliers and stable activations.\n> - These insights confirm that outliers exhibit **consistent distributions** for recent LLMs, which is a property of the weights and modules.\n>  [3] Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs, arXiv 2024.\n>\n>  [4] Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs, arXiv 2024.\n\n**L1**: Statistical significance.\n> - We apologize for the oversight. Our study used a fixed seed for all quantization operations, following standards in post-training quantization, and thus did not report statistical significance. We will correct this in the checklist of our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rwcUJbOfRY", "reviewer": "Reviewer_8e4p", "summary": "This work proposed a transformation (composition of orthogonal and permutation transformation) that makes LLMs more quantization-friendly (accounting for the presence of outlier features). The approach is validated on several modern LLMs from Llama-1,2,3 families.", "questions": "* Do the learned rotation possess incoherence property [1]? \n\n* Can one boost GPTQ performance further (as it is strictly stronger method compared to RTN) with DuQuant rotation matrices? \n\n* What is the context length for Llama-3-8B evaluation? Typically, one uses the training context length for comparison (8k for Llama-3 model family). \n\n[1] Chee, Jerry, et al. \"Quip: 2-bit quantization of large language models with guarantees.\" Advances in Neural Information Processing Systems 36 (2024).", "limitations": "See Weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The introduced method makes sense and targets the specific case of Massive outliers, that is not accounted in previous weight+activation approaches. The obtained results are pretty strong and achieve state-of-the-art at W4A4 quantization. \n\nThis approach is pretty fast and takes a couple of minutes even on large LLMs.\n\nThis work conducts a study on location and impact of specific types of outliers.\n\nSpeedups are quite significant.", "weaknesses": "The idea of applying rotation transformation for simplifying weight/activation quantization is not novel and (to my knowledge) was first introduced in QuaRot. The introduced method proposes a specific form of orthogonal/permutation matrices. QuaRot results for Llama-2-7b quantization reported in QuaRot paper are significantly better than the one presented in Table 7 and only marginally inferior to DuQuant numbers. I would suggest comparing with the numbers from original paper for fairness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's constructive comments on our paper. We will respond to the reviewer's feedback with detailed explanations for each point. \n\n**W1**: Highlight of Ours and detailed comparison with QuaRot. \n\n>   - We appreciate the reviewer's comments. We have dedicated **Appendix F** to highlighting our novel contributions and demonstrating our superiority over QuaRot. We summarize our key contributions below.\n>   - Instead of adopting the Hadamard rotation used in QuaRot, we use a greedy search algorithm that leverages prior knowledge to compute an **approximately ideal rotation matrix** that (1) is orthogonal and (2) specifically targets the positions of outliers, redistributing them across adjacent channels. **Figure 4** advocates the superiority of DuQuant in mitigating outliers compared to the Hadamard rotation.\n>   - We introduce the **zigzag permutation** that reduces activation magnitude variance between blocks, further reducing the overall outliers. The ablation study in **Table 5** highlights the significance of this permutation.\n>   - Our rotation and permutation matrices **simultaneously smooth weights** and activations. Consequently, DuQuant avoids the time-consuming GPTQ used by QuaRot. **Table F17** illustrates the significantly higher efficiency of DuQuant.\n>   - Due to these contributions, DuQuant consistently outperforms QuaRot. Although the results reported in QuaRot are higher than our reproduced ones, **the experimental settings in the original paper of QuaRot differ from ours**.\n>     - We summarize these setting differences in the following table.\n>\n>       |Setting|Weight|Activation|Key/Value|Query|\n>       |-|-|-|-|-|\n>       |QuaRot|symmetric|symmetric|group-wise asymmetric|FP16|\n>       |DuQuant|asymmetric|asymmetric|asymmetric|asymmetric|\n>\n>   - The comparison reported in our paper is **fair**, as we reproduced QuaRot under our settings. The results in Table 7 and Table F15-F17 consistently demonstrate the superiority of DuQuant over QuaRot.\n>\n>   - **Under the setting utilized in the original paper of QuaRot**, as suggested by the reviewer, we also provide the results of DuQuant in the table below. **DuQuant still outperforms QuaRot** by a large margin in PPL and QA tasks. Note that all the results of QuaRot are directly brought from its original paper.\n>\n>     |LLaMA2 W4A4|Method|WiKi↓|C4↓|PQ↑|WG↑|HS↑|A-e↑|A-c↑|LA↑|Avg↑|\n>     |-|-|-|-|-|-|-|-|-|-|-|\n>     |7B QuaRot Setting|FP16|5.47|6.97|79.11|69.06|75.99|74.58|46.25|73.90|69.82|\n>     ||QuaRot-RTN|8.37|-|72.09|60.69|65.40|58.88|35.24|57.27|58.26|\n>     ||QuaRot-GPTQ|6.10|-|76.77|63.77|72.16|69.87|40.87|70.39|65.64|\n>     ||DuQuant|6.23|7.91|76.28|66.93|72.96|69.99|40.53|69.61|66.05|\n>     ||DuQuant-LWC|**6.01**|**7.67**|77.64|67.80|72.97|70.37|41.81|69.53|**66.69**|\n>     |13B QuaRot Setting|FP16|4.88|6.46|80.47|72.22|79.39 |77.48|49.23|76.75|72.59|\n>     ||QuaRot-RTN|6.09|-|77.37|67.32|73.11|70.83|43.69|70.66|67.16|\n>     ||QuaRot-GPTQ|5.40|-|78.89|70.24|76.37| 72.98|46.59|73.67|69.79|\n>     ||DuQuant| 5.39 |7.05|78.51|70.88|76.80| 74.62|48.21|73.92|**70.49**|\n>     ||DuQuant-LWC|**5.27**|**6.93**|78.73|70.88| 77.20|74.07|47.27| 73.96 |70.35|\n\n**Q1**: Does the rotation transformation possess an incoherence property [1]?\n> Yes, our learned rotation transformation indeed possesses the incoherence property [1].\n>    - As described in [1], the incoherence of weight and Hessian matrices is ensured by multiplying them with a Kronecker product of **random orthogonal matrices**. \n>    - While our approach includes a greedy search step to learn the matrix, the final matrix $\\hat{\\mathbf{R}}$ obtained remains **orthogonal**. This is because the product of orthogonal matrices $\\hat{\\mathbf{R}} = \\mathbf{R}^1\\mathbf{R}^2\\cdots \\mathbf{R}^n$ remains orthogonal.\n>\n> [1] Quip: 2-bit quantization of large language models with guarantees, NeurIPS 2024.\n\n**Q2**: Can one boost GPTQ performance further with DuQuant rotation matrices?\n> We appreciate this valuable question. We demonstrate below that DuQuant is **compatible with and contributory to stronger methods, including GPTQ.**\n>    - We implement DuQuant+GPTQ by applying GPTQ exclusively on the four key layers after a dual transformation so that the computational overhead introduced by GPTQ is minimized. These four key layers are selected according to the compression difficulty, as suggested in ShortGPT [2].\n>    - The table below shows that this combination leads to an additional performance boost, further validating the effectiveness of DuQuant.\n>\n>       |LLaMA2 W4A4|WiKi|C4|\n>       |-|-|-|\n>       |7B-DuQuant|6.28|7.90|\n>       |7B-DuQuant+GPTQ|6.15|7.73|\n>       |13B-DuQuant|5.42|7.05|\n>       |13B-DuQuant+GPTQ|5.39|6.96|\n>\n> [2] Shortgpt: Layers in large language models are more redundant than you expect, arXiv 2024.\n\n**Q3**: Context length for LLaMA3-8B evaluation.\n> - We follow OmniQuant [3] and [4] to set the context length to 2048 in Table 4 of our paper. We will include this detail in Appendix C.\n> - We also follow the reviewer's suggestion to conduct a PPL evaluation for all baselines under an 8k context length and 4-bit quantization, as shown below:\n> \n>   |LLaMA3-8B (8k)|WiKi|C4|PTB|\n>   |-|-|-|-|\n>   |FP16|6.14|8.62|9.91|\n>   |SmoothQuant|225.65|242.70|277.38|\n>   |Atom|18.07|26.76|34.97|\n>   |DuQuant|7.57|12.24|12.44|\n>\n>  - Despite the increase in context length, all methods perform better. This improvement is attributed to 1) the 8k context length ensures the model is evaluated under the same conditions as it was trained, 2) the longer context length provides more historical context for the model to predict the next word, which helps the model generate text more accurately. \n>  - It can be observed that DuQuant continues to outperform the baselines.\n>\n> [3] Omniquant: Omnidirectionally calibrated quantization for large language models, ICLR 2024.\n>\n> [4] How good are low-bit quantized llama3 models? an empirical study, arXiv 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "03iCetjPVL", "reviewer": "Reviewer_Rbrg", "summary": "The paper introduces a method called DuQuant, a new quantization method specialized for LLMs. The paper notes that \"massive outliers\" cause previous quantization approaches to be less effective or powerful, and then proposes a new quantization method which ameliorates the effect of such massive outliers. Theoretical analysis shows that the method has good properties, such as successfully spreading out the outlier mass directly before quantization, which facilitates better outcomes. Experiments show state-of-the-art performance across standard language modelling benchmarks on Llama and Llama-2 class models, preserving most of the base model's performance.", "questions": "- Previous work on Massive Activations have shown their ubiquity in many types of LLMs, not just Llama, and even on certain types of vision models. Is it possible that this work can be done on other types of LLMs/transformers? Does the backbone model you pick significantly change the effectiveness of the method (e.g., Phi vs Llama vs Mistral)?", "limitations": "The authors are generally precise about the method's strengths and weaknesses, and include a limitations section.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The exposition/writing is very clear. \n- All major steps of the method are motivated, and the theoretical calculations straightforwardly apply to the method's real-world implementation. The design choices are either made straightforwardly or chosen via experiments/ablations.\n- Experimental results show that the overall model performance is better than other (previously state-of-the-art) quantization methods. The time and memory costs are also reduced, owing to the simple construction of the matrices in the construction (block diagonal, orthogonal, or permutation matrices).", "weaknesses": "Several potentially desirable properties of the model may not be completely covered by the analysis. For example:\n- The memory consumption reduction seems good, but the presentation of this result is postponed to the appendix E. Could you find a way to put this in the main body (maybe moving back the presentation of one of the several pure performance metrics such as PPL, or one of the ablation studies), and compare to other state-of-the-art models? In my opinion, since people usually only do quantization for better memory consumption or speeding up inference, measuring the performance on both of these axes is crucial.\n- The speedup is only measured for pre-filling. Is it possible to compute end-to-end speedup for open-ended generation? It would be great to see the impact of the method on some more realistic workloads (again, across the axes of performance, memory consumption, and speedup), to supplement the thorough analysis of each component in the paper.\nTo address both these issues, along with an empirical study, an analysis of the asymptotic runtime/memory complexity of the quantization procedure would be great if possible.\n\nAlso a slight nitpick: at the bottom of page 2, massive activations are not the same phenomenon as attention sinks, though they can be related.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for your thoughtful and positive feedback on our work. We are particularly grateful for your recognition of the various aspects of our research. Below, we have provided a detailed explanation for your remaining concern as follows. Please do not hesitate to let us know if you have any further questions. \n\n**W1**: Comparison of memory reduction.\n\n> - Thanks for the suggestion. We will move the memory consumption reduction results to the main body for better visibility. Additionally, to provide a more comprehensive comparison, we have conducted evaluations of our DuQuant alongside SmoothQuant [1], QLLM [2], and QuaRot [3] using one RTX 3090 GPU during the prefilling stage. These comparisons will be detailed in the revised manuscript to highlight the relative efficiencies of each method.\n>\n>   |LLaMA2-7B, INT4, BS=1|Prefilling Memory (GB)|Saving Factor|\n>   |-| -| -|\n>   |FP16|15.282|-|\n>   |SmoothQuant |4.782| 3.196x|\n>   |QLLM|5.349|2.857x|\n>   |QuaRot|4.784|3.194x|\n>   |DuQuant|4.786|3.193x|\n>\n> - From the table, we can observe that 4-bit quantization methods can effectively reduce memory usage during the pre-filling stage.  DuQuant, SmoothQuant, and QuaRot achieve **significant reductions up to 3.2x**, while the QLLM performs much worse.\n> \n>  [1] Smoothquant: Accurate and efficient post-training quantization for large language models, ICML, 2023.\n>\n>  [2] QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models, ICLR, 2024.\n>\n>  [3] Quarot: Outlier-free 4-bit inference in rotated llms, arXiv 2024.\n\n**W2**: Analysis of model performance, memory reduction, and inference speedup.\n\n> - Thank you for your valuable comment. LLM generation process includes a pre-filling stage, which is compute-bound, along with a decoding stage, which is memory-bound [1]. To comprehensively analyze these procedures, we will list a table in the revised manuscript comparing model performance with key features such as decoding memory usage and pre-filling time. For pre-filling, we measure time usage by sending one sentence with 2048 tokens and we decode 128 steps to compute peak memory usage. All the experiments are conducted on a single RTX 3090 GPU.\n>\n>   |INT4, BS=1|Time (ms)|Saving Factor|Memory (GB)|Saving Factor|WiKi↓|QA  avg.↑|\n>   |-|-|-|-|-|-|-|\n>   |FP16|568| -| 13.638|-| 5.47| 63.72 |\n>   |SmoothQuant | 248 | 2.290x| 3.890| 3.506x| 83.12|44.52|\n>   |QLLM|435|1.306x | 3.894| 3.502x| 9.09| 51.60|\n>   |QuaRot|284|2.000x | 3.891| 3.505x| 6.39| 61.25|\n>   |DuQuant|288|1.972x| 3.893| 3.503x| 6.28|61.76|\n>\n> - From the table, we can observe that our DuQuant effectively speeds up the pre-filling stage and largely reduces memory usage during decoding stage while providing **the most competitive results**.\n> - Due to time constraints and the significant workload involved, we couldn't test all methods on real-world generative tasks that require complex CUDA kernel optimizations. However, we are committed to continually optimizing DuQuant to improve its speed and efficiency in future updates.\n> - In addition, we present a simple **time complexity analysis** of our quantization process. We denote activation as $X\\in\\mathbf{R}^{N\\times C}$, block size as $B$, and greedy search step size as $n$. The complexity of obtaining the rotation matrix of a linear projection is caused by (1) QR decomposition $O(nB^3)$, (2) Rotation matrices multiplication $O(nB^3)$, and (3) Multiplication between $X$ and rotation matrics $O(n\\times NC/B \\times B \\times B)=O(nNCB)$. Thus, the total complexity is $O(nB^3+nNCB)$. For example, in LLaMa2-7B k_proj, we take $N=2048, C=4096$. We set $B=128,n=256$, then we can get the approximate computation complexity $nB^3+nNCB\\approx 2.7\\times 10^{11}$, which is less than the necessary WA multiplication complexity (approximately equal to $NC^2\\approx 3.4\\times 10^{11})$. This simple analysis demonstrates the efficiency of our quantization process.\n> \n> [1] Quarot: Outlier-free 4-bit inference in rotated llms, arXiv 2024.\n\n\n**W3**： Massive outliers and attention sinks\n> - Thank you for the clarification. We will correct the distinction between massive activations and attention sinks in the revised manuscript.\n\n\n**Q1**: DuQuant on Mistral and Phi models.\n\n> - We appreciate the inquiry and have extended the application of DuQuant to include Mistral and Phi models under 4-bit WA quantization. The PPL results are shown in the table below:\n>\n>   |Mistral-7B|WiKi| C4|\n>   |-|-|-|\n>   |FP16| 5.25| 7.75|\n>   |RTN| 306.26| 300.07|\n>   |SmoothQuant |100.59|158.02|\n>   |OmniQuant|5490.31|6094.82|\n>   |Atom| 8.65|12.43|\n>   |DuQuant| **5.86**| **8.48**|\n>\n>   |Phi2-2.8B|WiKi|C4|\n>   |-|-|-|\n>   |FP16 |9.71|12.76|\n>   |RTN|230.59|253.79|\n>   |SmoothQuant| 63.84|83.24|\n>   |OmniQuant|NaN|NaN|\n>   |Atom| 35.72|41.26|\n>   |DuQuant|**20.65**|**22.49**|\n>\n> - From the table, we can observe that DuQuant **largely surpasses other baselines**, particularly with **Mistral-7B**. Regarding the Phi2-2.8B model, it often experiences **instability** in matrix multiplication **between queries and values**, leading to **overflow** issues and posing great challenges to quantization. However, while DuQuant may not perform as well as FP models, it still significantly outperforms other baselines.\n> - In addition, we have visualized the massive outliers in the **down_proj** layer of the **Mistral-7B** model and the feature space after our dual transformations. These visualizations are available in the PDF file included in the general response section. It can be observed that our DuQuant perfectly eliminates these outliers.\n> - These results underscore the effectiveness of our dual transformation approach in addressing massive outliers **across various types of LLMs**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **General Response for All Reviewers**

>**Summary**:
> 
> We sincerely thank all reviewers for their valuable time and insightful feedback, which is very helpful in further improving the quality of our paper. We are grateful that the reviewers appreciate (1) "the technical contributions of dual transformations are notable" (XTv7, Rbrg); (2) "the method is well-motivated for managing massive/normal outliers" (all reviewers) ; (3) "the paper is well-written/well-organized" (x516, nwuk); and "the theoretical analysis guarantees the performance" (x516, XTv7, Rbrg). We are also encouraged that the experiments are "comprehensive" (YKYC) and the experimental results are "competitive" (x516) and "convincing" (nwuk, Rbrg). In addition, our speedup and runtime test are acknowledged by reviewers as "significant" (8e4p).

> 
> 
>In order to provide greater clarity on the revisions made to our paper and the experiments we conducted to address the reviewers' questions, we have summarized the modifications and experiments made during the rebuttal period as follows:


>**Additional Experiments**:
> 
>- We provide additional experiments on **LongBench** to evaluate the generative ability of quantized models. (Reviewer XTv7 W1)
>- We apply DuQuant on **Mistral-8B** and **Phi2-2.8B** to demonstrate the effectiveness on other model types. (Reviewer Rbrg Q1)
>- We compare the DuQuant quantized Vicuna with the **FP16 models** on **MT-Bench**. (Reviewer nwuk Q1)
>- We implement DuQuant under **QuaRot quantization settings** and compare with results from their **original paper**. (Reviewer x516 Q1, Reviewer 8e4p W1)
>- We combine DuQuant with **GPTQ** to further boost the quantized model. (Reviewer 8e4p Q2) 
>- The analysis of the **context length** for LLaMA3-8B evaluation. (Reviewer 8e4p Q3)
>- The analysis of the **memory usage** during the **decoding** stage. (Reviewer XTv7 W2)
>- The **comparison with baselines** for memory consumption reduction. (Reviewer Rbrg W1) 
>- The analysis of **memory consumption, time usage, and model performance**. (Reviewer Rbrg W2)

>**Clarifications**:
>
>- The detailed comparison with QuaRot includes analysis from various perspectives (Reviewer 8e4p W1) and additional experiments. (Reviewer x516 Q1, Reviewer 8e4p W1)
>- The illustration of the discrepancies between PPL and other downstream tasks. (Reviewer nwuk W1, Reviewer XTv7 Q4)
>- The pseudo-code to clarify the construction of the rotation matrix. (Reviewer x516 W2)
>- The reason why the reporting of results is not consistent across the models. (Reviewer XTv7 W3)
>- The detailed illustrations of Figure 5 and Table 6. (Reviewer XTv7 Q5)
>- The discussion about calibration-free experiments. (Reviewer XTv7 Q6)

>**Attachment**:
>- We visualize the massive activations change with our DuQuant on Mistral-7B in the attached PDF.

Meta-review: The authors propose a new method for weight and activation quantization in LLMs that targets the existing issue of large-magnitude activations and their effect on quantization. Their method applies a carefully-designed linear transformation to better distribute these outliers, with theoretical and practical demonstrations of success. Initial reviews of the paper were positive, and increased further after clarifications and additional results during the rebuttal and discussion phase. I concur with the reviewers and recommend acceptance. The authors should carefully incorporate feedback from the reviewers into the final revision.
Decision: Accept (oral)