DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation
Authors: Jingyang Xiang
Sai Qian Zhang
Venue: COLM 2025 (2025)
Keywords: W4A4 quantization
randomized hadamard transforms
randomized orthogonal transforms
outlier
masssive activation
TL;DR: We explained why random Hadamard is superior to randomized orthogonal transforms in the W4A4 quantization process and proposed an optimization method for the rotation matrix.

Abstract: Rotating the activation and weight matrices to reduce the influence of outliers in large language models (LLMs) has recently attracted significant attention, particularly in the context of model quantization. Prior studies have shown that in low-precision quantization scenarios, such as 4-bit weights and 4-bit activations~(W4A4), randomized Hadamard transforms can achieve significantly higher accuracy than randomized orthogonal transforms. Notably, the reason behind this phenomenon remains unknown. In this paper, we find that these transformations show substantial improvement in eliminating outliers for common tokens and achieve similar quantization error. The primary reason for the accuracy difference lies in the fact that randomized Hadamard transforms can slightly reduce the quantization error for tokens with massive activations while randomized orthogonal transforms increase the quantization error. Due to the extreme rarity of these tokens and their critical impact on model accuracy, we consider this a long-tail optimization problem, and therefore construct a simple yet effective method: a weighted loss function. Additionally, we propose an optimization strategy for the rotation matrix that involves alternating optimization of quantization parameters while employing orthogonal Procrustes transforms to refine the rotation matrix. This makes the distribution of the rotated activation values more conducive to quantization, especially for tokens with massive activations. Our method enhances the Rotated LLMs by achieving dual free, **Outlier-Free** and **Massive Activation-Free**, dubbed as **DFRot**. Extensive experiments demonstrate the effectiveness and efficiency of DFRot. By tuning the rotation matrix using just a single sample, DFRot achieves a perplexity improvement of 0.98 and 0.95 on W4A4KV4 and W4A4KV16, respectively, for LLaMA3-70B, a model known for its quantization challenges. Code is available at https://github.com/JingyangXiang/DFRot.

Published as a conference paper at COLM 2025
DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation
Jingyang Xiang
Sai Qian Zhang
New York University
New York University sai. s zhangenyu.. edu
xlangxiangginngggg@@emaa.......
Abstract
Rotating the activation and weight matrices to reduce the influence of outliers in large language models (LLMs)) has recently attracted signifi- cant attention, particularly in the context of model quantization. Prior studies have shown that in low-precision quantization scenarios, such as 4-bit weights and 4-bit activations (W4A4), randomized Hadamard trans- forms can achieve significantly higher accuracy than randomized orthog- onal transforms. Notably, the reason behind this phenomenon remains unknown. In this paper, we find that these transformations show substan- tial improvement in eliminating outliers for common tokens and achieve similar quantization error. The primary reason for the accuracy differ- ence lies in the fact that randomized Hadamard transforms can slightly reduce the quantization error for tokens with massive activations while randomized orthogonal transforms increase the quantization error. Due to the extreme rarity of these tokens and their critical impact on model accuracy, we consider this long-tail optimization problem, and therefore construct a simple yet effective method: a weighted loss function. Addi- tionally, we propose an optimization strategy for the rotation matrix that involves alternating optimization of quantization parameters while em- ploying orthogonal Procrustes transforms to refine the rotation matrix. This makes the distribution of the rotated activation values more conducive to quantization, especially for tokens with massive activations. Our method enhances the Rotated LLMs by achieving dual free, Outlier-Free and Mas- sive Activation-Free, dubbed as DoRtt. Extensive experiments demonstrate the effectiveness and efficiency of DFRot. By tuning the rotation matrix using just a single sample, DFRot achieves a perplexity improvement of 0.98 and 0.95 on W4A4KV4 and W4A4KV16, respectively, for LLaMA3- 70B, a model known for its quantization challenges. Code is available at https://gii wwwb. cwm /tt.tubb..omnnnnngggiiiinnnnoo
Introduction
Large Language Models (LLMs) have shown exceptional abilities across numerous domains. Cutting-edge open-source models like LLaMA (Touvron et al., 2023) and Mistral (Jiang et al., 2023), along with proprietary LLMs such GPT (Achiam et al., 2023) and Gem- ini (Team et al., 2023), are now being applied in wide range of applications, including natural language understanding (Zellers et al., 2019; Hendrycks al., 2020), machine transla- tion (Zhang et al., 2023), content generation (Mo et al., 2024), recommendation systems (Wu et al., 2023; Wang et al., 2024; 2025) and agent (Li al., 2025).
However, the remarkable success of LLMs is largely reliant significant computational resources. LLMs often consist of billions of parameters, making them not only resource- intensive to train but also challenging to deploy on devices with limited computational capacity, such as mobile phones and edge devices. Additionally, the high memory and processing demands not only drive up hardware costs but also significantly increase energy consumption, leading to serious deployment concerns. To address these challenges, re- searchers and engineers actively exploring various model compression techniques (Fran-
1

---

Published as a conference paper at COLM 2025
tar et al., 2022; Xiao et al., 2023; Lin et al., 2024a; Yao et al., 2022; Frantar & Alistarh, 2023; Ashkboos et al., 2024a; Wei et al., 2024; Zhao et al., 2025). These techniques aim to reduce the size of LLMs while maintaining their performance effectively as possible, achieving balance between efficiency and accuracy.
Unfortunately, the presence of outliers in the activations (Dettmers et al., 2022; Zeng al., 2022) often leads to a significant reduction in model accuracy when PTQ is applied directly. To address this problem, earlier approaches have either scaled weights and activations (Xiao et al., 2023; Wei et al., 2023; Shao et al., 2023), shifting the quantization challenges from activations to weights, or employed mixed-precision techniques to isolate outliers (Dettmers et al., 2022), thereby minimizing the LLM's quantization enror.
Recent research (Ashkboos et al., 2024b) has demonstrated that rotating activations in LLMs can effectively eliminate most outliers while preserving computational invariance, ensuring that the LLM's output remains identical to its original results. Moreover, the rotation matrices can be merged into the weights, imposing no additional burden on network inference. This innovative computational invariance (Ashkboos et al., 2024a) has garnered significant attention from researchers.
Although rotation widely recognized as an important method for the quantization of LLMs, there remain many unresolved issues. For example, shown in Table 1, when activations are reduced to 4-bit, the reasons why randomized Hadamard trans- forms (RH) often achieve significant improvement compared to randomized orthogonal transforms (RO) (Ashkboos et al., 2024b; Liu et al., 2024) have not yet been fully understood. However, while directly training rotation matrices can yield good results (Liu et al., 2024), the training process will cause substantial computational resources and adds complexity to the quantization process.
In this paper, we first investigate the underlying reasons why RH outperforms RO. We find that for ordinary tokens consisting primarily of outliers (Achiam et al., 2023), both RO and RH transformations can equally reduce quantization error when applied to these tokens. As shown in Figure 3, in terms of quantization error, there is no substantial difference between the two transformations. In contrast, for special tokens with massive activations (Sun et al., 2024), using RO on these activations surprisingly leads to an increase in quantization error. Our experiments show that this inability to efficiently manage massive activations greatly restricts the accuracy of quantized LLMs. On the other hand, while RH performs better than RO, it only manages to maintain or slightly reduce the quantization error for these large activations. This observation indicates that both transformation methods struggle to effectively manage massive activations in LLM quantization.
Building on these insights, we propose a novel optimization method to enhance the per- formance of quantized LLMs, achieving both Outlier-Free and Massive Activation-Free, e.g. dual free (DFRot). By treating scarce tokens with massive activations as long-tail distributed data, we develop simple yet effective weighted loss function. Additionally, we intro- duce an alternating optimization approach to refine the rotation matrices and quantization parameters, further minimizing quantization error. Extensive experiments demonstrate the effectiveness of our proposed method. Specifically, by tuning the rotation matrix with just single sample, DFRot achieves a PPL improvement of 0.95 and 0.98 on W4A4KV4 and W4A4KV16 for LLaMA3-70B with WikiTText-2, model recognized for its quantization challenges (Huang et al., 2024).
2 Related Work
2.1 Eliminating outliers via Scale Invariance
The initial idea behind suppressing outliers through scale invariance stems from the obser- vation that weights are easier to quantize than activations, and outliers in activations often appear in a few fixed channels Dettmers et al., 2022. Based this, SmoothQuant (Xiao et al., 2023) first proposes that we can offline migrate the quantization difficulty from activa- tions to weights via scale invariance. SmoothQuant enables an INT8 quantization of both
2

---

Published as a conference paper at COLM 2025
X,R
X2RL
Westestrog
Masti-llad
Feet-teraad
Winuheel
X.R. R.
X:It: E.R. (X) Kij.
Rotate indide block
Robate indide blook
W
Awit
kwn
Swit
An illustration of rotational invariance in the LLaMA architecture. The rotation matrix R can be integrated into the residual connection, ensuring the network retains rotational invariance. The rotation inner the block can further reducing outliers within block. Both of them make LLM fewer outliers and be easier to quantize. The rotation matrix R R R2, Ri and R. can be integrated into weights. R3 and R4 need to compute online.
Figure 1:
weights and activations for all the matrix multiplications in LLMs. Furthermore, Outlier Suppression+ (Wei et al., 2023) proposes a fast and stable scheme effectively calculate scaling values, achieving a better balance in quantization burden. To reduce manual design and further enhance quantization performance in extremely low-bit quantization, Omni- Quant (Shao et al., 2023) introduces Learnable Weight Clipping and Learnable Equivalent Transformation, efficiently optimizing the quantization process for both weight-only and weight-activation quantization. In the clipping W4A8 quantization, QQQ (Zhang et al., 2024) proposes to dynamically handle outliers through adaptive smoothing. QServe (Lin et al., 2024b) proposes SmoothAttention to effectively mitigate the accuracy degradation caused by 4-bit KV quantization. Both QQQ and QServe have effectively enhanced the performance of LLMs in W4A8 quantization.
2.2 Eliminating outliers via Rotational Invariance
Although scale invariance can reduce outliers and improve quantization performance, merely transfers the outliers from activations to weights and has not eliminated them fun- damentally. When the magnitude of the outliers is large, scaling struggles to achieve an effective balance between weights and activations. Recently, researchers have found that applying rotation matrices to networks can effectively reduce outliers without increasing the complexity of LLMs. QulP Chee et al. (2024) is the first to suggest that quantization can benefit from the incoherence between weight and Hessian matrices. employed randomized orthogonal matrices generated by Kronecker product to enhance their inco- herence. QulP# (Tseng et al., 2024) replaces the randomized orthogonal matrices with randomized Hadamard matrices, which are faster and possess better theoretical properties. QuaRot (Ashkboos et al., 2024b) is the first work to apply rotational invariance (Ashk- boos et al., 2024a) for model quantization. QuaRot finds that randomized Hadamard transformations yield better results compared to randomized orthogonal transformations. SpinQuant (Liu al., 2024) and OSTQuant (Hu et al., 2025) further extends the rotation matrices to trainable space and applied Cayley optimization (Li et al., 2020) to refine them, achieving significant improvements across diverse datasets.
3
Rotational Invariance, Quantization and Massive Activation
3.1 Rotational Invariance
First, we briefly introduce rotational invariance in LLMs, using the structure of LLaMA as an example. We assume that the in the RMSNorm has been fused into the follow linear layers' weights, including W₁, W₁, Wo, Wup and Wgate and RMSNorm applies to each row of R, is an rotation matrix, we have the commutation RMSNorm (X) R, (Ashkboos et al., 2024a). This property implies that multiplying the input of RMSNorm by R₁ is equivalent to multiplying the RMSNorm output by R₁-
the activations X as X + / |X, property RMSNorm(XRR
3

---

Published as a conference paper at COLM 2025
=ive > EIRG) EAM E1Ve a) EIRII EIRO
EIRO) ENantea EIRO
E3R0 Elventia> EIRO1
Mondy 2003
16
L8
03 LLaMAZ-78 78
1d LLaMAZ
(dl LLMMAB 88
Figure 2: Comparison of 4-bit activation quantization error E(.) for each token with NR, RO and RH for (a) LLaMA-7B, (b) LLaMA2-7B, (c) LLaMA2-13B and (d) LLaMA3-SB. The tokens are from model.layers.6.pos attention layernorm. Best viewed in color.
2824
2187
59
350
203
1200
Figure 3: Comparison of 2D 4-bit quantization errors for tokens with NR, RO, RH and DFRot for LLaMA3-SB from Figure 2.
(4) Vanile Quent 20.83
Quant
Id Quami Error 29.33
Drfttt Quant Emon
Figure 4: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 2.
As shown in Figure 1, remove outliers in the input activations, rotation matrix R is applied to the embedding layer W Wembeddingr resulting in a new input activation X₁ According to the above, we can know once we transform Wg. Wy, W, and W, in the Multi- Head Attention (MHA) to R,W, R W₁, R,W,, and W,R,, the hidden feature within the MHA will remain unchanged, and the original output feature Y1 will become Y1 R The following Ferd-Forward Network's input X2 from the residual connection will be modified
further transform Wup, Wgate and Wdown R W.p, R Wgate and Waown Rú, the hidden feature within the FFN will also remain unchanged, and the output feature X3 will be modified (X2- Y2 R1 X, Based on mathematical induction, we can get that X, + Y, (X, Y, R1 X R1 for the n-th module. To this end, by transforming Wimdhead into R W hondd the network output will remain unchanged.
to (X, + Y1 R1 R₁
There is also rotational invariance within the block. For MHA, we can insert head-wise rotation matrices R₂ and R for W. and W. and R3 for Query and Key after RoPE. For
FEN, we can insert R, and R between Swish and Wdomn These approaches can further eliminate outliers and reduce quantization error while keeping the block output unchanged. In this paper, we only discuss Rj. For R2, R3, and R4, we follow the QuaRot (Ashkboos et al., 2024b) settings and use Hadamard matrices.
3.2 Why the Randomized Hadamard is better than Randomized Orthogonal?
Based on the computational invariance described in Section 3.1, it evident that the choice of rotation matrices is critical for ensuring the accuracy performance of the quantized model. Therefore, a natural question arises: What type of rotation matrix offers the most advantageous properties?

---

Published as a conference paper at COLM 2025
LLaMA-7B
LLaMA2-BB
LLaMA2-13B
LLaMA3-8B
Method
4-4-4
4-4-16
4-8-16
4-4-4
4-4-16
4-8-16
444
4-4-16
488-6
444
4-4-16
4-8-16
GPTQ
NaN
NaN
NaN
NaN
NaN
NaN
Inf
Inf
6.01
Inf
Inf
7.29
(RO) QuaRot
6.68
6.62
5.80
7.96
7.71
5.61
6.00
5.92
4.99
10.54
10.15
6.52
(RO) QuaRot.FF160
6.30
6.27
6.17
6.10
5.38
5.34
7.83
7.68
(RH) QuaRot
6.37
6.33
5.81
6.27
6.20
5.61
5.51
5.46
5.01
8.20
8.02
6.52
(RH) QuaRot.FP160
6.30
6.28
6.17
6.10
5.40
5.37
7.82
7.67
Table 1: WikiText-2 perplexity (L) results for RO and RH for LLaMA models. The 4-4-4, 4-4-16, 4-8-16 represent W4A4KV4, W4A4KV16, W4A8KV16 respectively. We show the failed GPTQ using NaN and the perplexity results> 100 by Inf. QuaRot.FP160) denotes retaining tokens with massive activations as FP16.
We begin by focusing on RO and RH, as both QuaRot (Ashkboos et al., 2024b) and Spin- Quant (Liu et al., 2024) have demonstrate that RH delivers substantial improvements over RO in LLMs. We conducted experiments by applying RO and RH to the LLaMA models respectively, followed by weight quantization using GPTQ under various quantization settings. The results are shown in Table 1. Benefiting from the outlier elimination through rotational invariance, we find that for dynamical token-wise 8-bit activation quantization, both RO and RH lead to significant performance improvements compared to standard quantization. Additionally, no substantial performance difference is observed between the two transformations. However, under 4-bit dynamical token-wise activation quantization, RH significantly outperforms RO.
To investigate the performance differences between RH and RO under 4-bit activation setting, we plot the corresponding quantization error after applying 4-bit quantization to the multiple tokens. We also display the quantization error for the baseline setting where quantization applied without rotating the activation to better understand the impact of using the rotation matrix. As shown in Figure 2, compared to the no rotation (NR), both RO and RH effectively reduce the quantization error for most tokens across different models. While RH slightly lowers the quantization error, the difference between the two methods is minimal for the majority of tokens. This leads to the question: What explains the significant difference in PPL during quantization when their quantization errors are similar?
To answer this question, we turn our attention to massive activation (Sun et al., 2024), a rare but significant feature in LLMs. As shown in Figure 2, the red points represent quantization error for the tokens with massive activation. While most tokens show large quantization errors under NR, these special tokens display significantly smaller errors, which can be observed from Figure 3. It is normal since each token has a fixed 12 norm after RMSNorm processing, shown in Figure 4(a), tokens with massive activation naturally exhibit smaller quantization errors when quantized to 4-bit. Figure 4 presents the quantization result for the token with massive activation after applying NR, RO, and RH. Surprisingly, the rotation operations do not significantly reduce quantization errors for these tokens. In fact, compared to NR, RO greatly increases their quantization error, while RH only marginally reduces it. This leads us to question whether tokens with massive activation are the primary cause of the significant accuracy discrepancies between RH and RO.
To investigate this further, we build upon QuaRot by retaining tokens with massive activa- tions in FP16 format for both RO and RH, while applying 4-bit quantization to the remaining input tokens, denoted as (RO) QuaRot.FP16. and (RH) QuaRot.FP166 As shown in Table 1, for all LLaMA models, the performance gap between RH QuaRot and RO QuaRot is totally surprising that by simply retaining these extremely few tokens (often
disappeared. It is surprising less than one- mnetthousandth) as FP16, we can completely eliminate the performance difference between RO and RH. Therefore, we can make the following conclusion:
Why the Randomized Hadamard is better than Randomized Orthogonal?
RH RO- Tokens with Massive Activations: RH is better than RO because it per- forms more effectively when reducing the quantization error for tokens with massive activations in 4-bit activation quantization.
5

---

Published as a conference paper at COLM 2025
3.3 Optimization Objectives and Calibration Data Selection
As mentioned above, although retaining tokens with massive activations as floating-point numbers can significantly enhance model accuracy, this approach is akin token-level version of ALM.int8(). It still requires fine-grained mixed-precision computations during the process, which will introduce additional system level optimization. Therefore, in this paper, we focus on W4A4 quantization to maintain simplicity and efficiency in the computation process. We consider a loss function of the following form:
g() Ex ||XX| (xxR),
(1)
where x E IRIC the token vector from a calibration dataset X E RLxC, C the hidden size and L is the number of tokens. R₁ E RCx. satisfies R; R I, 8x is the quantization parameters and ((x,gx) E R is the quantization of the x. The expectation IE [-] is taken over the token distribution. For the ease of analysis, we use the mean squared error ll2-
Meanwhile, we introduce our data selection principle. We denote the calibration dataset as I X, the tokens with massive activations as X and the remaining tokens as X \ X":
L( = E xxxaix X" ||*R] - 2(xR1 8x 8x) |13]
+ Exxx"
| - 2(xR1 )0] (2)
During calibration, we apply a weighted loss prioritize the quantization error on tokens with massive activations, with representing the weight.
The motivation behind this principle stems from the observations in Table 1. Since X is the key factor contributing to the performance gap between RO and RH, simply optimizing R via Eq. fails to specifically target X"", On the other hand, compared to the NR Table 1, RO also significantly improves performance, indicating that reducing the outliers on X X can enhance quantization performance, optimizing only for X" has the risk of increasing the quantization error for X X"", ultimately degrading the model's performance. Hence, is crucial to optimize both X and X X Naturally, we can regard this a long-tail optimization problem, where X represents the long-tail but important data. Using weighted approach to optimize the quantization loss is a simple yet highly effective method. Ablation studies in Section 4.2 further demonstrate the advantages of this strategy.
3.4 Solution Methods
Optimizing R₃ is challenging task. Since R₁ influences every MHA and FFN in the net- work, adjusting the activation distribution in one layer impacts the quantization results across all layers. This makes it difficult to optimize layer by layer or block by block (Shao et al., 2023; Wei et al., 2023). A straightforward approach is to use training methods for quantization-wware fine-tuning of the rotation matrix across the entire network (Liu et al., 2024). Although it does not require retaining the gradients of the weights or the corre- sponding optimizer states, it still demands substantial computational resources during the quantization process.
In this paper, we focus on improving the effectiveness of rotation matrices in mitigating outliers and massive activation. Intuitively, we hypothesize that rotation matrix that minimizes quantization error will lead to better performance. Drawing inspiration from Simsiam (Chen & He, 2021), we propose to regard quantization representation Q (xRig)) as cluster centroids f/x In the context, optimizing R, and g is equivalent to optimizing R₁ and (EM)-like
1/x, which can be viewed as an implementation of an Expectation-Maxiiiza algorithm, as shown in the following equation:
ming C(R,, 1/)) EE Exexumi
||XR||
1/-12] 2
[|IXR] 1x113]
E =E
|| - 1xx11
where 7/- = ((xRn,g) and caa fx|rEXE E x \ x U (xx/xx E X") This formulation is analogous to k-means clustering (Macqueen, 1967), and Rj and f/x act like the kernel
6

---

Published as a conference paper at COLM 2025
function and cluster centroids, respectively. Similar to k-means clustering, the problem described in Eq can be approached using an alternating algorithm, where one set of variables is fixed while solving for the other. Formally, can alternate between solving these two subproblems:
.nx); T - arg min L R (
arg
Ri + arg min mig & (R1,
(4)
f
where represents the iteration index of the alternating rounds, and s/x and Ri denote the values of and R at round t.
Solving for the cluster centroids 1/x- The set of quantization parameters 8x further contains the quantization scale 5x and zero point Zx- In this paper, adopt dynamic asymmetric per-token quantization for activations. Therefore, we can independently determine the optimal quantization scheme for solving 6x and 2x for each xR1:
(x 2+x) = clamp (L
xR1
+ Zx, 0,2N -1).
8x
(5)
max (xR, min(Xx))
min(xR,)
where 5x
2
Sx
where [-] indicates round operation, N is the bitwidth.
Solving for R₁- The right side od Eq 4 is well-known as Procrustes problem (Mulaik, 2009). which involves finding the optimal rotation matrix R, that best aligns two sets of points, minimizing the Frobenius norm of their difference. The solution to this problem can be obtained through Singular Value Decomposition (SVD). Specifically, given input matrices X and its quantized version Q(X,g) the optimal R₁ can be found:
R uvT, where U, Σ, VT SVD(X" ((X,x)))))))))
(6)
where we treat the quantization parameters g' as constant.
One-step optimization. To find an improved rotation matrix R₁ and quantization parame- ters 8x, we perform the iterative process shown in Eq 4. Specifically, a calibration set x is randomly sampled from X, the iterative process can be specified as:
24 arg min Exexoal
|
(x
,/xx + (x
then the resulting quantization parameters will be used produce the rotation matrix:
Ri
< arg min Exexoal
[||XR] v/112]]
(8)
Rj
The detailed algorithm is provided in Algorithm
4 Experiments
Experiment settings. We implemented DFRot based on QuaRot. In this paper, to sim- plify the problem, we apply dynamic asymmetric per-token quantization for activation values. The KV-cache is quantized using asymmetric quantization with group size of 128. GPTQ (Frantar et al., 2022) are used for weight with per-channel symmetric quantization, where a linear search for the clipping ratio is applied to minimize squared error. We use a sample with sequence length of 2048 from WikiText-2 (Merity al., 2016) training set to genrate calibration dataset x initialize the rotation matrix R, with RH, and optimize it for 100 iterations. After obtainíng the optimized rotation matrix R1, we apply it to the corresponding model and achieve rotational invariance. We use 128 samples each with sequence length of 2048, the calibration dataset for GPTQ quantization.
7

---

Published as a conference paper at COLM 2025
LLaMA3-BB
LLLMA3-70B
LLaMAZ-7B LLaMA2-135 LLaMA---0
LLaMA-7B
LLaMA-13B LLaMA-30B
BBits
Method
0-shoff
O-shot Wiki
D-shot"
Wiki
O-shot"
Wiki O-shot"
Wiki
O-shoof
Wiki
O-shot"
Wiki
O-shot Wiki
W-A-KV
Avg(1)
(1) LAng (?)
(L)
Avg((1) (1)
Avg.(1)
(4)
LAg (1)
(L)
Avg
(4)
(Avg(1))
(.)
AAvg(")
(1)
16766-66
Foatingoonn
66.09
6.4,
7338
2.86
68.21
3:27
67.61
4.8
71.99
"332
64.48
75.68
16.67
7509
70.00
-410"
KIN
33.42
6e22
31.21
Bes
32,44
30.86
865
30.90
764
32.51
7e3
31.65
Se4
31.57
Ze3
SuanthQuant
34.67
2x2
nan
35.86
3e2
34.64
1e3
GPTO
32,98
5c2
31.47
4c4
32.72
nan
30:11
4e5
30.86
man
32.12
1c3
31.51
3e3
30.88
2c3
4-4-16
QuaRot
61.86
8.11
68.25
5.92
61.63
6.17
64.66
5.45
69.96
3.89
61.65
6.33
61.85
5.57
67.79
4.74
DFRot
63.01
7.78
69.82
4.97
62.42
6.13
65.34
5.39
3.99
62.25
6.30
64.47
5.58
68.06
4.78
SpinQuant"
64.1T
7:228
66.
6.10
37.37
6.78
63:23
1524
70.58
3..68
61.82
6:00
64.34
5:36
68.08
4.53
QST
68.14
7.24
72.21
3.97
63,90
5.60
66-24
5.14
70,92
3.57
62.72
6.04
68.80
5.40
68.52
4.43
RIN
35.18
762
30.82
bes
32.67
nan
30198
765
31.73
764
32,87
1e4
31.35
Sel
31.64
263
SmoothQuant
32.96
1c3
33.76
3e2
32.12
nan
33.36
1e5
35.54
3e22
33.32
3e2
33.28
Se2
34,65
1e3
CPTQ
33.71
6c2 31.20
4e4
33.52
27.85
5c3
31.80
2c3
30.65
3e3
31,07
2c3
4-44
Omniquant
14.26
48.46
11,261
45.63
QuaRot
61.38
8.28
68.29
6.02
61.81
6.25
64.44
5.49
69.96
3.92
61.21
6.37
64
5.99
67.92
4.77
DFRat
62.94
7.91
: 69.62
5.03
61.80
6.25
64.95
5.43
68.78
4.02
61.84
6.36
64.26
5.62
67.93
4.81
SpinQuant"
64.10
7.55
66.31
6.24
62.01
5,96
64.13
5.7
70.57
3.61
61.32
6.12
61.95
5.39
68.14
4.55
O6T OSTQuant
65.37
7.29
71.69
4.01
63.18
5.91
65:41
5.25
70.84
3.59
62.55
6.07
65:45
5:40
68.20
4.42
Table 2: Comparison of averaged accuracy on nine Zero-Shot tasks and perplexity on WikiText2. Results for SmoothQuant, GPTQ, OmniQuant, AWQ, SpinQuant and OSTQuant are from the OSTQuant paper, and QuaRot's results from the official code. ddenotes the methods that use the quantization-awar training to optimize R1.
4.1 Main results
Language Generation Task. We evaluate DFRot on a language generation task and com- pare it with SmoothQuant (Xiao et al., 2023), GPTQ (Frantar et al., 2022), OmniQuant (Shao et al., 2023), AWQ (Lin al., 2024a), SpinQuant (Liu et al., 2024) and OSTQuant (Hu et al., 2025). Table 2 shows the perplexity of LLaMA models. As shown, compared to QuaRot, DFRot achieves improvements in most cases. For example, DFRot achieves the most significant improvement on the LLaMA3-8B model with W4A4KV4 and W4A4KV16, outperforming QuaRot by 0.25 and 0.21, respectively. It is worth noting that DFRot has achieved near 1.00 PPL improvement on LLaMA3-70B, model known for its challenging quantization performance, even surpassing SpinQuant, which finetunes R1 on wikitext through coantizztion-awar--traine
Similar to QuaRot, DFRot does not require any retraining process and only needs a sample to optimize the rotation matrix. On a single NVIDIA A100 80G GPU, it only takes an extra 8 minutes for LLaMA-7B & LLaMA2-7B & LLaMA3-8B and 20 minutes for LLaMA2-13B, resulting in minimal overhead. Even for the 70B models, the additional time is less than 90 minutes, which is also acceptable. It demonstrates that DFRot has wide applicability and can serve as a cost-effective post-training method to enhance the quantization performance of rotated LLMs. Although DFRot does not achieve the best performance compared to the state-of-the-ard methods, like OSTQuant, we believe DFRot also help community to understand the fundamental performance gap between RO and RH.
Zero-Shot Tasks. We also evaluate DFRot on the following nine important zero-shot tasks: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2021), Open (Mihaylov et al., 2018), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), Arc (Easy and Challenge) (Clark et al., 2018) and LAMBADA (Radford et al., 2019). We use lm_ ev.5==0.4.5 (Gao et al., 2024) or our experiments. Table shows the average score of DFRot on the above tasks. As can be seen, DFRot consistently achieves improvements compared to QuaRot across all tasks, For example, DFRot achieves a 1.56% accuracy improvement compared to QuaRot on the LLaMA3-8B model with W4A4KV4 quantization settings.
4.2 Ablation studies
Choice of y. To further understand the effect of hyperparameters in DFRot, we conducted an ablation study on Wikitext-2 PPL to investigate the impact of different y settings for W4A4KV16. As seen in Figure 5, when y ranges between 50 and 200 DFRot achieves signif- icant improvements across various L.LMMA models using RH. Notably, on the LLLMA3-8B
8

---

Published as a conference paper at COLM 2025
Fa Carkettre
(a) LLaMA-7B
(b) LLaMA2-7B
(c) LLaMA2-13B
(d) LLaMA3-8B
Figure 5: Comparison of WikiText-2 perplexity results under different 7 for W4A4KV16. Rj is initialized with RH.
By Curhe
(a) LLaMA-7B
(b) LLaMA2-7B
(c) LLaMA2-13B
(d) LLaMA3-8B
Figure 6: Comparison of WikiText-2 perplexity results under different for W4A4KV16. R initialized with RO.
model, which is known for its quantization performance sensitiveness to massive activa- tions from Table 1, we observed PPL improvement of over 0.2 in Figure 5(d). If we set = and treat X and X X equally to minimize their quantization errors, it may reduce the quantization loss of X X but increase the quantization loss of X" ulti- mately resulting in performance decline on the LLaMA2-13B. Conversely, if we set 7 + oo and only optimize the quantization enror for X" it will increase the quantization error of Xtal X resulting in accuracy drop across the LLaMA-7B, LLaMA2-7B, LLaMA2-13B and LLaMA3-8B models.
Initialize with Randomized Orthogonal. We conducted an ablation to study the effective- ness of DFRot when R initialized with RO. We keep the same experimental settings as in the study with RH and optimize the rotation matrix with different 7 values. As shown in Figure 6, our method achieves considerable improvements in RO scenarios compared to using RH for initialization. Meanwhile, more effective for LLM whose quantization pe- formance is more sensitive to the massive activations, such as LLaMA3-8B and LLaMA3-70B. However, due the exceptional performance of RH, initialization and optimization using RH always yield superior final results compared to those obtained with RO. Therefore, we recommend using RH for initialization in practice achieve better performance.
Sample1 x64 2048)
Sample2 x 2048)
Sample3 (64x2048)
Sample4 (64x2048)
Sample5 (64x2048)
Model
LLaMA3-8B
7.78
7.76 6.12
7.79
7.74
7.76 6.14
LLaMA2-7B
6.13
6.15
6.11
Sample1 (48 x 2048)
Samplel (32x 2048)
Sample1 (24x
Sample1 (16x 2048)
Sample1
Model
x 2048)
LLaMA3-8B
7.78
7.78 6.15
7.77
7,80
7.86
LLaMA2-7B
6.14
6.15
6.12
6,20
Table 3: Comparison WikiText-2 perplexity results under different calibration samples for W4A4KV16.
4.3 Analysis of Calibration Set Sensitivity
We performed ablation studies W4A4KV16 on LLaMA3-SB and LLaMA2-7B along two dimensions: the choice of calibration samples and the num of calibration tokens and evaluate

---

Published as a conference paper at COLM 2025
results on WikiText. Samples are all sampled from WikiText-2 train. In selecting the number of tokens, we utilize the inputs to the first N transformer blocks as the calibration data source and demonstrate results in Table 3. For example, when 48x 2048 tokens are selected, the inputs to transformer blocks to 23 are used for calibration. Our results indicate that, for tuning the rotation matrices in LLaMA3- 3- and LLaMAA-7B, using x6x 2048 tokens is often sufficient. We believe that these reasons may all be the causes for the optimization of DFRot being relatively insensitive to the number of tokens in the calibration dataset:
1. These special tokens will appear in relatively shallow network layers (Sun al., 2024), therefore, a small number of layers are also sufficient to capture these tokens.
2. For model, tokens with massive activations in LLMs tend to exhibit only few similar data distributions because these tokens are often produced by out proj or down proj layers with large weights (Yu al., 2024).
3. GPTQ will use 128 samples with a length of 2048 to calibrate the weights, which reduces the impact of the sample size during rotation matrix calibration.
4.4 Results MMLU
W-A-KV
Methods LLaMA2-7B
LLaMA3-SB
QWen2-7B
Mistral-7B-v0.3
16-16-16
FP
41.85
62.23
69.47
59.11
44-16
QuaRot
34.83
51.43
62.67
52.82
44-16
DFRot
35.54
51.68
63.40
53.38
Table 4: Comparison of MMLU results under different methods.
We compare DFRot with QuaRot with W4A4KV16 quantization configuration with different models. As seen in Table 4, even though rotation matrix R₁ is refined with WikiText-2 dataset, DFRot also outperforms QuaRot in all models. It indicates that DFRot, which refines Rı by optimized long tailed quantization error, can be seen as general method. It also worth noting that even though DFRot achieves slight improvement with WikiText2 for LLaMA2-7B, it achieves 0.71% improvement with MMLU, which is significant. On the contrary, for the LLaMA3-SB, while DFRot achieves significant improvement with WikiText2, it only achieves 0.25% improvement with MMLU, which is slight. To sum up, we can know that the PPL with WikiText2 can not been seen as a good indicator of the model downstream performance. In the future, we will study how to design more robust quantization algorithms for downstream tasks to further enhance the capabilities of quantized models in downstream tasks.
Conclusion
Eliminating outliers in LLMs through rotational invariance can significantly improve model quantization accuracy. In this paper, find that in the context of 4-bit activation quan- tization, the fundamental reason for the effectiveness difference between RO and RH is their performance tokens with massive activations. Specifically, randomized Hadamard transformations perform better on these tokens than random Orthogonal transformation. Based on the observation that tokens with massive activations are rare and important in LLMs, we treat the problem as a long-tail optimization and construct simple yet effective weighted quantization loss function to balance the importance of tokens. Furthermore, by alternately employing orthogonal Procrustes transformations to refine the rotation matrix R and optimizing quantization parameters for X, our method, named DFRot, enhances the Rotated LLMs by achieving Dual Free, including Outlier-Free and Massive Activation-Frrs is worth noting that DFRot significantly improves model accuracy in 4-bit activation quantization with just single data sample, achieving PPL improvements of 0.98 and 0.95 on W4A4KV4 and W4A4KV16, respectively, for the LLaMA3-70B, which notable for its quantization challenge.
10

---

Published as a conference paper at COLM 2025
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar axXiv:2303.08774 2023.
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arx preprint arXio:2401.15024, 2024a.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. ar Xiv preprint ar arXin:2404.00455, 2024b.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pp. 7432-7439, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. ar Xiv preprint arXiv:1905.1004 10044, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arvi preprint arXi:: Sttr:1803.0555,, 2018
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318-30332 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXio preprint ar wwXi=:2200.17323 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenod:: htgg///cords/12286/00
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. aiviv preprint arXiv:2009.0333 2020.
Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting, arviv preprint arXit:2501.13987 2025.
11

---

Published as a conference paper at COLM 2025
Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. How good are low-bit quantized Ilama3 models? an empirical study, arviv preprint Ar1:::2404. 14047, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint ar urXi:22310.06225, 2023.
Fengyu Li, Yilin Li, Junhao Zhu, Lu Chen, Yanfei Zhang, Jia Zhou, Hui Zu, Jingwen Zhao, and Yunjun Gao. Aistorian lets ai be historian: A kg-powered multi-agent system for accurate biography generation. Xiv preprint ar arXiv:2503.11346 2025.
Jun Li, Li Fuxin, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. arXiv preprint ar xrXi::2002.01113, 2020.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangx- uan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024a.
Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient I1m serving. ar preprint ar axXim:2405.04532, 2024b..
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant-IIm quantization with learned rotations. arXiv preprint 7Xit:2405.16406 2024.
J Macqueen. Some methods for classification and analysis of multivariate obseroations. University of California Press, 1967.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel M mixture models, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. ar preprint arXiv:1809.02777 0009902789, 2018.
Yuhong Mo, Hao Qin, Yushan Dong, Ziyi Zhu, and Zhenglin Li. Large language model (Ilm) ai text generation detection based transformer deep learning algorithm. arXiv preprint ar Xii::2405.06652, 2024.
Stanley A Mulaik. Foundations of fuctor analysis. CRC press, 2009.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8),9 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications the ACM, 64(9):99-106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. ar preprint arXiv:1904.09728, 2019.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13333, 2023.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations large language models. Xiv preprint ar 4r2iv:240..1766, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. ar Xiv preprint arXit:2312.11885, 2023.
12

---

Published as a conference paper at COLM 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09228, 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#:: Even better I1m quantization with hadamard incoherence and lattice codebooks. arv preprint ar arXio:2402.04436, 2024.
Bohao Wang, Feng Liu, Changwang Zhang, Jiawei Chen, Yudi Wu, Sheng Zhou, Xingyu Lou, Jun Wang, Yan Feng, Chun Chen, et al. Llm4dsr: Leveraing large language model for denoising sequential recommendation. arviv preprint ar Xiv::2440 082008, 2024.
Bohao Wang, Feng Liu, Jiawei Chen, Xingyu Lou, Changwang Zhang, Jun Wang, Yuegang Sun, Yan Feng, Chun Chen, and Can Wang. Msl: Not all tokens what you need for tuning llm as a recommender. arviv preprint ar Xrit:2504.04178 2025.
Jiateng Wei, Quan Lu, Ning Jiang, Siqi Li, Jingyang Xiang, Jun Chen, and Yong Liu. Struc- tured optimal brain pruning for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, PP. 13991-14007, 2024.
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. Xrv preprint ar arXio:2304.0914. 2023.
Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen. A survey on large language models for recommendation. CoRR, abs /////2355.9960 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yux- iong He. Zeroquant: Efficient and affordable post-training quantization for large-scale post-trainnng transformers. Advances in Neural Information Processing Systems, 35:27168-22 1-11113,,, 2022.
Mengxia Yu, De Wang, Qi Shan, and Alvin Wan. The super weight in large language models. arv preprint 1rxi:22411.07111, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint ar arXiv:1905.07880, 2019.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint ar Xrio:2210.02411,, 2022.
Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: A case study. In International Conference on Machine Learning, pp. 41092-41110. PMLR, 2023.
Ying Zhang, Peng Zhang, Mincong Huang, Jingyang Xiang, Yujie Wang, Chao Wang, Yineng Zhang, Lei Yu, Chuan Liu, and Wei Lin. Qqq: Quality quattuor-bit quantization for large language models. ar Xiv preprint ar Xiv:2406. 099944 2024.
Maosen Zhao, Pengtao Chen, Chong Yu, Yan Wen, Xudong Tan, and Tao Chen. Pioneering 4-bit fp quantization for diffusion models: Mixup-sign quantization and timestep-aware fine-tuning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18134-18143, 2025.
13

---

Published as a conference paper at COLM 2025
A Calibration Data
In this section, we explain the reason why we only used a single data sample to calibrate the rotation matrix Rj in DFRot, and don not attempt use more data:
In LLMs, outliers and massive activations often appear in some fixed channels. Therefore, the process of optimizing the rotation matrix can be seen as an optimization of the distri- bution patterns of outliers and massive activations. We have simply ten samples to calibrate the rotation matrix for LLaMA2-7B, but no significant improvement in accuracy observed.
Our calibration data is sample with length of 2048 tokens. Since we obtain the calibration set from each MHA and FFN, taking LLaMA2-7B as an example, we can obtain 2048 x 32 x 2 131072 tokens as calibration tokens. This is relatively sufficient to statistically analyze the distribution patterns of outliers and massive activations.
B Limitations
Due to the discontinuity of the quantization function, our current optimization method prone to getting stuck local minima and often relies on initialization. As can seen from Figure 5 and Figure 6, the performance of RH is better than that of RO in most cases. Meanwhile, during the optimization of the rotation matrix, we also found that after iterative convergence, the final quantization error optimized with RH initialization is often better than that with RO.
C Algorithm
Algorithm Optimization of Quantization Parameters and Rotation Matrix
Require: Token initial rotation matrix R1, quantization function Q
Ensure: Optimized rotation matrix R₃ and quantization parameters 1: Initialize Rj with randomized Hadamard matrix, =
2: while t< 100 do
3: // Step 1: Optimize Quantization Parameters f/x
4: for each token x do
5:
Compute quantization parameters s, via arg mins, ||*R Q(xR) Sx, sx, 2x)113
6:
Update 1 = (x (x end for
7:
8:
// Step 2: Optimize Rotation Matrix R
9:
Solve the Procrustes problem to update R/: R arg ming |XR - 1/x l1
10: f=t+1
11:
end while
12: return Optimized R
14

---

Published as a conference paper at COLM 2025
D Quantization error for tokens with Massive activation in LLaMA-7B, LLaMA2-7B, LLaMA2-13B
More quantization results for LLaMA-7B, LLaMA2-7B and LLaMA2-13B:
Vonila
Figure Comparison 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA-7B from Figure 2.
Figure Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 2.
300
Merite 15.00
BHULDPRA
Figure 9: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure
15

---

Published as a conference paper at COLM 2025
E
Quantization error between NR, RO, RH and DFRot
More 2D quantization error visualization are shown as follows:
Vaniila Erer: 20.45
10 Quant Emon 112.3
Quant Brorr 12.77
DTRE Cuart Emer
Figure 10: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA-7B from Figure 2.
24.47
Quant
Quan
Quint
Figure 11: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 2.
1a) Vandia Quart Eroo: 37.35
R0 Quard 140.12
KM Cuand Bran 25.15
1d1 DERec Cuart Enon
Figure 12: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure
16

---

Published as a conference paper at COLM 2025
F
Visualization for Different layers
We visualize for more layer follwing:
Everilaa> ERGI ERH Everilla>> > ER > B90)
ERO1 > Elannis) EIRM)
EIRH) Elvan EROI
1980
2 2003
(al 7L
(0) LLaMA1278
[c) LLaMA2-133 138
LLaM333 88
Figure 13: The tokens are from layers w.wprsss.iiipuuttu layernor..
E(verili) > EORGI > EJKH) Ectarili)s > EgUIt (000)
ES1O1 > ElVnniee) > EUMO
EIRN) Eivan Bal EOUOI
Fraay 2003
2000
dnl LL
(b) LLaMA2-78
[cl LLaMA2-133
(d) LLMMA3-08
Figure 14: The tokens are from modelllayers attention layernorm.
Elorilla > ERRO ERH
Evaniial EiRH) EIRO)
EIRO) BVanioi > EIRHI
14
259
2300
Freeo nes
Nrnce, 2003
Sanna 2000
(ol LLaMA-78
(b) LLaMA2-78
[c] LLaM42-138
(d) L1 LMA3-88
Figure 15: The tokens are from
Edarltts) > ERO1 ERH)
EManllaa EIRH) EIRO
EIRG) BVanlall > EIRHI
24
253
258
Frawy, 2900
1330
Krisk 2006
Srnal 200
Oal LLaMAr7
(b) LLa7A2-70
[c] LLaMA2-130
LL
Figure 16: The tokens are from model.layers.8.ppll attention layernorm.
17

---

Published as a conference paper at COLM 2025
Everllb)> EROI ERH
Elvania) EIRK) EIR
EIRO) BVanila E(HH
220
21
600
3300
61000
M 2000
Gel LLaMA-78
(0) LLaMA2278
[c] LLaMA2-13B
(d) LLLMA3388
Figure 17: The tokens are from
Eltarili) p- EORO F38H)
EdVaniii) EIRR > EIRO)
FIRO) EVAnl > E(RH
256
pee
Wrnakn 2006
153
3230
25
Oel LLaM-778
(b) LLaMA2-78
[c] LLaMA2-138 38
LL4M3333
Figure 18: The tokens are from model.layers.1.. 12. attention layernorm.
Elorilla)> > EROI ERHI E(Wrdilia) > Egent x b3s)
EROI > Eloniiis) E0RM)
EIRK) > Blvan ERRI
226
silen
22
2030
3308 20o1 Srens)
350
2re 2008
(al LLaMA-78
(b) LLaMA4-78
[c] LLaMA2-138
(d) LL
Figure The tokens are from modeelayyrrs1liipptlleeeemmm
E(Wnnilss) > ERRO ERR E(Wnliti) E.00 (90))
EjROI > E(Vanitt) EDRIO
EIRG) BIROO E00nliss
13
1937
234
1086 Krent 2000
(al LLaMA-78
(0) LLaMA2-78
[c] LLaMA2-122 13B
(4) LLaM33-88
Figure 20: The tokens are from model.layerr..6.poo attention layernorm.
18

---

Published as a conference paper at COLM 2025
Everlls)>> > ERGI ERH E(Veritaa>>> > Ejtn biO))
ERO1 Elaniis) > EIRM
EIRO) BRM) EmVnlbl
Xr 2000
600
2088
61000
Gdl LL
(b) LLaMA2-78
[c] LLaMA2-13B
(d) LLLMA3388
Figure 21: The tokens are from
Eltarills) > ESRO1> F3RH) E(tarilie) y EgUIt (Uo))
ESRo > ElVeniin) > EURIO
FIRO) EIPIO Echnllss
13
250
150
Lo30 Frven
Krnw 2006
JE8Z SrA
Sr 2000
del LLAM-77
(b) LLaMA2-78
[c] LLaMA2-138
(d) LL LLAMA33-8
Figure 22: The tokens are from modelllayers.0.ppol. 20. attention layernorm.
Elorilla)> > EROI ERH! E(Wriilia) > Egent x bas)
ERR0 >e Elannitt) > EIRK
EIRK Edvan ERR0
E9101 a E(Rnn) > EtVanile)
EIAR) > EURO) ecVnnilll
23
1006
2006
1900 2203 SVANN)
2006
(al LLaMA-78
(b) LLaMA2-78
[c] LLaMA2-138
(d) LL
Figure 23: The tokens are from
E(wnnilss) > ERRO ERRI E(Wnlili) > E.iI (90)
EIROI > ElVani EDRIO Eaaai >- E(RIN) ElVenitt)
EIRN EWVan Bal ESROI E(RN) EIRO) Echnniss
b20
251
1838 Wrvw
2006
(al LLaMA-78
(0) LLaMA2-78
(cI LLaMA2-138 13B
(4) LLaM33-8
Figure 24: The tokens are from 24 po.t attention layernorm.
19

---

Published as a conference paper at COLM 2025
1000
30
500
190
2000
506
100
20m
Varl
Vari
Rilve
Figure 25: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA-7B from Figure 13.
201-
2:312
2313
302
300
1200
2030
21/3
1200
2215
200
205
20
220
Vhri
Vori la
K-
Krtus DPAL
Figure 26: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 13.
202
00
125
3000
100
150
1200
1200
1200
20
20
Variaa RO
Vari
80 va R-1
Briva
Figure 27: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure 13.
200
20/6
304
302
00
130
300
60
150
240
201a
Vari RD
Vari
80 va R-1
Rriiv DF
Figure 28: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 13.
20

---

Published as a conference paper at COLM 2025
1000
30
500
190
2001
50e
1000
Varl
Vari
Rilve
Figure 29: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA-7B from Figure 15.
201-
2:322
2313
302
320
1200
2030
27/3
1200
2215
200
205
20
200
250
Viri
Vori la
K-
Krtus DPA
Figure 30: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 15.
00
220
200
200
225
1020
200
20
225 215
180
Va RO
Variaa va
80 R-1
Briva
Figure 31: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure 15.
200
20/6
304
322
00
130
100
3000
1500
240
2016
Vari RD
Vari
80 va Rr1
Rriva DF
Figure 32: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 15.
21

---

Published as a conference paper at COLM 2025
1.00
30
580
15m
1000
1800
-80
3000
150
Warl
Rilve
Figure 33: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA-7B from Figure 17.
201-
2:322
2313
302
320
1200
2030
205
1200
2215
200
205
20
220 Krtuss DPAL
Vhri
Vori la
K-
Figure 34: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 17.
202
00
225
1000
100
230
200
20
100
120
200
200
ARC
100
200
212
200
Va 19.10 30
Verila va
RD va R1
131 Aff
Figure 35: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure 17.
200
20/6
304
302
00
130
3000
1500
240
2016
Vari va
Vari
80 va R-1
Rriva DFrce
Figure 36: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 17.
22

---

Published as a conference paper at COLM 2025
200
201
20.
1000
1000
30
30-
180
150
180
18
Va
Ven
PM
Figure 37: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA-7B from Figure 19.
201-
2:322
2313
302
320
1200
2030
27/3
1200
2215
200
205
20
220
Vhri la
Vori la
Krtus
Figure 38: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 19.
202
00
225
1000
100
230
200
20
101
120
200
200
ARC
20
200
Va 18. 10
Varila va
RD R1
121 Off
Figure 39: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure
200
20/6
302
302
00
130
3000
1500
240
2016
Vari va
Vari
80 va R-1
Ertiv DFAct
Figure 40: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 19.
23

---

Published as a conference paper at COLM 2025
190
1.00
150
30
200
30m
200
2000
300
64 Riive
Vanil
Van
Figure 41: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA-7B from Figure 21.
201-
2013
23322
1003
302
300
220
1200
100
20/3
100
120
1200
100
2012
1200
125
125
20
So.1
Va 15
HD HH
HH Dff
Figure 42: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 21.
00
30: 200
100
202 200 1300
300 200
1000
200 100
120
20
15
Va N1
Vari va
80 R-1
Rriva
Figure 43: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure 21.
200
20/6
304
322
00
130
2000
1500
240
2016
Vari va
Vari
80 va R-1
Rriia DFtct
Figure 44: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 21.
24

---

Published as a conference paper at COLM 2025
1.00
30
300
2:00
1500
360
20t
Rio Ril D.Ree
VenlaveRO
Figure 45: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA-7B from Figure 23.
201-
23322
302
380
100
2008
1100
2500
1500
300
2200
200
2000
1800
120
200
65
Vori la
Vania
NO
Anflus DEALE
Figure 46: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 23.
202
00
20 100
155 200 1800 202
200
280
1000
200
20
2012
20 E5
Vari
Vari
80 vB R-1
Briva
Figure 47: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure
200
20/6
304
302
120
220
200
200g
1100
200
Vari RD
Vari
80 va R-1
DFfct
Figure 48: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 23.
25

---

Published as a conference paper at COLM 2025
-25
-22
-02
(4) Varillla Quant
Quant Enor: 117.81
(c) Quant Enor:
Quart
Figure 49: Comparison of 4-bit quantization enror for the token with massive activation with
NR, RO, RH and DFRot for LLaMA-7B from Figure 13.
-20
40
(4) Varilll
051
6c) Quant
Quent
Figure 50: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 13.
03
(N) Vanilla Quant Emor 56.02
(5) RD Quant 156.12
[c] Quant Emor: 26.73
(30 DFRct Ouent Erer: 5.53
Figure 51: Comparison of 4-tit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure 13.
-30
-10
(2) Vanilla Quant Emor: 29.17
R5 Quant Emer: 97.50
(c) RH Quant 27.89
(30 DFR Quant Erant Eror: 0.99
Figure 52: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 13.
26

---

Published as a conference paper at COLM 2025
-25
-22
-02
(4) Varillla Quant
(B) Qaai 85.31
(c) Quant Enor
Quart
Figure 53: Comparison of 4-bit quantization enror for the token with massive activation with
NR, RO, RH and DFRot for LLaMA-7B from Figure 15.
-20
40
(4) Varilla Quant
97.19
6c) Quant Enor:
Quent
Figure 54: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 15.
03
(N) Vanilla Quant Emo: 55.34
(3) RD Emor: 132.00
[c] Quant Emor: 26.97
(30 DFRct Ouent Erer:
Figure 55: Comparison of 4-bit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure 15.
-30
-10
(2) Vanilla Quant Emor: 28.45
0 RD Quant Enor: 128.75
(c) RH Quant Ener: 28.4
(30 DFR.0 Quant QuantErrr: Eror: 0.64
Figure 56: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 15.
27

---

Published as a conference paper at COLM 2025
-25
-22
-02
(4) Varillla Quant
(B) Qami 91.75
(c) Quant Enor:
Quart
Figure 57: Comparison of 4-bit quantization enror for the token with massive activation with
NR, RO, RH and DFRot for LLaMA-7B from Figure 17.
-20
40
(4) Varillla
$1.25
6c) Quant Enor:
Quert
Figure 58: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 17.
DAVEE
03
(N) Vanilla Quant Emer 54.06
(5) RD Quant 111.12
[c] Quant Emor: 25.03
(30 DFRct Ouent Erer:
Figure 59: Comparison of 4-tit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure
-10
(2) Vanilla Quant Emor: 26.22
050 RD Quant Enor: 109.12
(c1 RH Quant 28.73
(30 DFR Quant Orant Eror: 0.44
Figure 60: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 17.
28

---

Published as a conference paper at COLM 2025
QUIEX
-25
-22
-02
(4) Vartilla Quant
Quant Enor: 118.12
(c) Quant Enor
Quart
Figure 61: Comparison of 4-bit quantization enror for the token with massive activation with
NR, RO, RH and DFRot for LLaMA-7B from Figure 19.
-20
40
(4) Varilll
91.19
6c) Quant Enor:
Quant
Figure 62: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 19.
03
(N) Vanilla Quant Emor 47
(5) RD Quant Emor: 122.56
[c] Quant Emor: 26.19
(30 DFRct Ouent Erer: 5.11
Figure 63: Comparison of 4-tit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure
-10
(2) Vanilla Quant Emor: 24.17
R8 Quant Emer: 85:31
(C) RH Quant Quant 28.48
(30 DFR Quant Erant Erio: 0.62
Figure 64: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 19.
29

---

Published as a conference paper at COLM 2025
QUIEX
-25
-22
-10
(4) Varillla Quant Enor
(B) Qaait 84.8
(c) Quant Enor:
Quart
Figure 65: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA-7B from Figure 21.
-20
40
(4) Varilll Quant
Quant
6c) Quant
Quant
Figure 66: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 21.
DANTA
03
(N) Vanilla Quant Emor 40.25
(5) RD Quant Emor: 127.12
[c] Quant Emor: 23.83
(30 DFRct Ouent Erer:
Figure 67: Comparison of 4-tit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure 21.
(2) Vanilla Quant Emor: 21.30
050 RD Quant
QuantErrr: Enor: 112.00
(c1 RH Quant
Enert Ener: 28.73
(30 DFR Quant 0.51
Figure 68: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 21.
30

---

Figure 69: Comparison of 4-bit quantization enror for the token with massive activation with Published as a conference paper at COLM 2025
QUIEX
-25
-22
-10
(4) Varillla Quant Enor:
Quant Enor: 126.88
RH
Quart
NR, RO, RH and DFRot for LLaMA-7B from Figure 23.
-20
40
Quanni
Quant 105.19
(c) Quant Enor:
Quent Ener:
Figure 70: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-7B from Figure 23.
DAVEE
03
(N) Vanilla Quant Emo: 23.45
(5) RD Quant Emor: 126.12
[c] Quant Emor: 22.83
(30 DFRct Ouent Erer: 6.11
Figure 71: Comparison of 4-tit quantization error for the token with massive activation with NR, RO, RH and DFRot for LLaMA2-13B from Figure 23.
-30
-10
(a) Vanilla Quant Emor: 19.72
R8 Quant Emor: 86.69
(c) RH QuantErce 26.88
(30 DFR Quant 0.53
Figure 72: Comparison of 4-bit quantization enror for the token with massive activation with NR, RO, RH and DFRot for LLaMA3-8B from Figure 23.
31

---

Published as a conference paper at COLM 2025
G
Full Results
Model
W-A-KV
#Bits
Method
ARC-c
(1)
ARC-e
(1)
BoolQ
(1)
HellaS.
(1)
(1)
OBQA
(1)
PIQA
(1)
SIQA
(1)
WinoG. (1)
Avg. (1)
Wiki2
(c)
16-16-16
Full Precision
46.42
74.33
77.71
75.94
73.69
44.20
79.16
45,91
69.53
65.21
5.47
RIN
25.34
28.03
50.52
27.7
1.01
26.20
50.62
33.93
48 38
32.44
SmoothQuant
28.33
49.39
27.28
1.18
23.40
48.80
33.62
50.75
32.13
nan
GPTQ
24.40
28.70
51.62
28.66
1.36
24,60
51.14
34.49
49.49
32.72
nan
4-4-16
QuaRot
45.99
72.73
77.8
75.92
73.45
43.8
78.02
46.16
68.03
61.66
5.45
DFRot
73.23
2-7B
SpinQuant'
37.54
62.58
71.16
70.48
67 16
34,80
75
39.76
60.62
57.37
6.78
OSTQuant
44.03
71,93
75.41
74.94
73.22
43.20
78.51
45.85
68.03
63,90
5.60
RTN
27.22
27.06
50.83
27.34
0.93
25. 80
49.51
34.85
50.51
32.67
Quant
26.37
25.63
47.71
27.15
1.11
26.40
34.49
48.38
32.12
nan
GPTQ
26.96
27.65
52.84
28.83
1.63
20
49.62
35.11
49.50
33.52
nan
4-4-
Omniquant
31.40
53.75
63.79
55.06
35.63
34.40
66.59
40.28
54.70
48.40
14.26
QuaRot
41.3
69.32
72.66
72.09
39.0
76.5
43.19
60.81
6.25
DFRot
43.52
70.83
73.30
72.62
68 46
41 40
76.82
44.17
65.11
61.80
6.25
SpinQuant'
40.44
71.08
74.40
73.51
70.66
41.80
76.88
43.50
65.82
62.01
5.96
OSTQuant"
42.92
72.56
74.71
73.14
71.76
77
44.98
66.77
63.18
16-16-16
Full Precision
49.15
77.53
80.58
79.39
76.62
45.20
80.63
47.49
71.90
67.61
4.88
RTN
27.99
26.81
38.50
26.08
0.00
23.60
48.20
34,90
51.62
30.86
8c3
SmoothQuant
30.87
3.67
55.01
35.31
34.26
GPTQ
27.82
26.77
37.92
25.67
0.00
21.80
47.77
35.11
48.15
30.11
4e3
QuaRot
46.42
73.86
78.10
75.68
74.31
43,00
79.05
44.37
71.35
65.13
5.35
DFRot
46.67
74.45
77.19
07.07
75.04
43.6
78.29
46.16
69.61
65.34
5.39
2-13B
SpinQuant'
43.77
69.99
76.57
74.63
72.81
41.60
77
44.27
68.19
63.23
5.24
OSTQuant"
47.78
74.66
80.03
77
75.94
44.40
79.38
46.06
70.32
66.24
5.14
RTN
27.82
26.32
38.58
26.27
0.02
26.00
79.78
"3..39
49.17"
30.93
783
Smanthhuant
24.49
33.00
45.84
30,70
2.70
23.80
53.81
34,80
51.07
33.36
GPTQ
27.90
26.39
37.95
26.16
0.00
27.00
48.26
34.39
50.43
27.85
3e3
4-44
Omniquant
32.85
55.13
61.34
60.13
42.85
33.40
68.17
39.76
56.51
50.35
12:30
QuaRot
73.16
DFRot
46.50
73.48
76.67
76.83
73.92
43.00
79.27
45.55
69.30
64.95
5.43
SpinQuant"
46.67
74.49
76.76
75.22
72.19
42.40
78.29
43.45
67.72
64.13
5.74
Quant"
77.46
78.67
65.41
16-16-16
Full Precision
53.50
77.74
81.10
79.18
75.74
44.80
80.63
47.08
73.01
68.09
6.14
RTN
25.72
36.50
31.26
3.13
27.66
552.72
"35.26
550.4"
35.42
662
SmoothQuant
23.29
28.
48.93
29.19
1.57
28.60
54,
33.37
49,64
33.04
1e3
GPTQ
23.46
32.07
43.79
30.10
2.41
28.00
53.97
34.14
32.96
4-4-16
QuaKot
44.03
69.74
71.90
73.16
67.26
42.4
76.71
45.04
66.46
61,86
8.11
DFRot
46.16
72.90
73.73
74.98
68.23
40.60
77.53
44.32
68.67
63.01
7.78
3-8B
SpinQuant"
76.36
75.98
77.37
64.11
OSTQuant*
48.81
73.48
79,82
75.97
72.62
42.40
78.18
45.75
69.22
65.14
7.24
RTN
23.72
30.56
46.18
29.83
2.70
28 60
52.45
34.39
50.20
33.18
7e2
Smoothiuant
23.55
28.90
29.40
32.96
GPTQ
23.38
32.74
44.34
29.72
2.39
29.80
54.95
34.75
51.30
33.71
6e2
4444
Omriqqunt
22,87
30.35
41.53
31.11
1,86
25.40
53.37
34.08
50433
32.33
4e2
QuaRot
43.17
72.66
72.53
66.66
39.20
76.06
44.83
61.38
8.28
DFRot
44.97
71.09
73.27
74.13
43.00
78.24
44.58
69.53
62.94
7.1
SpinQuant"
46.33
73.57
76.15
75.43
71.40
41.40
79.16
44.68
68.75
64.10
7.35
OsTQuant"
49.32
76.73
78.87
76.01
70.77
43.20
78.51
45.70
69.22
65.37
7.29
Table 5: Complete omparison the perplexity score on WikiText2 and averaged accuracy on Zero- shot Common Sense Reasoning tasks on LLaMA-2 &
32

---

Published as a conference paper at COLM 2025
Model
#Bits
Method
ARC-c
ARC-e
BoolQ (*)
HellaS,
LambA. (1)
OBQA
PIQA
SIQA
WinoG. (7)
Avg. (1)
Wiki2
W-A-KV
(f)
(1)
(?)
(f)
(1)
16-16-16
Full Precision
44.71
72.90
74.98
76.20
73.08
43.80
79.16
45.55
69.93
64.48
5.68
RTN
23.46
29.34
45.05
29.02
1.24
26.00
52.07
51.30
32.51
SmoothQuant
25.17
31.40
51.62
29.73
5.43
28.20
54.68
34.44
49.09
34.42
3e2
GPTQ
23.89
27.74
42.87
28.49
1.28
27. 27.40
51.00
36.23
50.20
32.12
le3
4-4-16
QuaRot
41.38
68.01
72.23
73.02
70.33
42.6
76.99
43.76
66.54
61.65
6.33
DFRot
42.06
7B
OSTQQaat"
SpinQuant"
42.58
40.19
68.43
70.79
72.87
72.35
72.91
74.06
70.77
70.68
41.20
43.40
77.75
77.69
44.17
45.04
67.25
68.67
61.82
62.72
6.04
6.08
RTN
23.89
29 59
46.67
37
1.13
26.40
'52.99
35,21
34
32.77
1e4
SmoothQuant
23.38
34.75
50.67
GPTQ
23.89
27.90
43.88
86
1.05
26.20
51.85
34.08
49.49
31,80
2e3
4444
Omniquant
31.40
54.84
61.00
56 98
38.29
31.90
66.59
39.30
55.17
48.46
11.26
QuaRot
40.70
68. 39
71.96
72.46
70.37
40.60
77.09
43.60
65.75
61.21
6.36
DFRot
40.87
43.86
SpinQuant'
39.08
68.18
73.06
72.82 87
70.46
40.60
77.42
42.68
61.32
6.12
OSTQuann"
42.92
70.33
72.11
73.77
70.66
42.42
77.91
44.93
67 88
62,55
6.07
16-16-16
Full Precision
47.87
74.49
77.86
79.10
76.03
44.40
80.30
46.72
73.24
66.67
5.09
RIN
25.85
26 26
42.05
50.33
34.60
50.67
31.63
2e4
SmoothQuant
25.43
29.29
51.56
28.12
2.02
26.00
53.32
34.34
49.57
33.29
6e2
GPTQ
24.66
27.78 78
40.80
83
0.70
24.20
51.31
36.65
51.70
31.51
3e3
QuaRot
48.04
72 18
76.02
76.90 90
72.56
43.60
79.43
44.47
70.24
64.83
5.57
DFRot
46.25
72 35
74.5
45
72.99
45.45
70.32
5.58
13B
OSTQQuan
SpinQuant"
48.04
45.73
74.07
72.56
77.13
75.38
76.86
77.22
74.58
73.28
45.00
43.60
78.89
78.62
46.16
44.63
71.35
70.40
65.80
64.59
5.40
RIN
26.28
27 27
42.35
25 85
0.19
26. 60
49.95
34.19
49 25
31.33
3e4
SmoothQuant
24.49
28
$1.65
27 91
2.08
26.00
52.56
35.41
50.39
33.28
5e2
GPTQ
23.63
27.31
89.85
26.17
0.56
26.00
51.66
35,62
49.57
30.63
3e3
4-4-
Omniquant
29.61
48 23
20
56.45
28.76
31.40
63.29
37.10
55.64
45.63
10.87
QuaRot
47.10
70.92
75.6
76.31
72.44
44.8
77.91
45.65
71.35
64.68
5.39
DFRot
46.16
70.88 88
74.22
76.88
73.18
42.40
78.13
46.06
70.40
64.26
5.62
SpinQuant'
45.99
70.71
76.51
77.16
73.63
45.60
79.00
45.65
70.32
64.95
OSTQQuant
45.90
75.25
76.94
77.21
74.23
43.40
79.43
45.91
70.56
65.43
5.40
16-16-16
Full Precision
52.99
80.39
62.75
82,62
77.59
48.00
82.26
47.75
75.69
70.00
4.10
RIN
25.00
27 .5.95
42.02
22
0.21
27.
49.1
34.65
50.91
31.57
283
SmoothQuant
23.63
30.68
54.86
31.91
3.80
28.
54.13
34.49
50.04
34.64
1e3
GPTQ
27.30
27
38. 69
26.75
0.17
25
49.02
35
47.75
30.89
2e3
4-4-16
QuaRot
52.05
75 84
81.28
80.78
75.33
46.2
80.36
45.91
28
67.79
4.7
DFRot
50.94
76.64
80.43
80.82
75.35
47.0
80.03
47.34
74.03
68.06
4.78
30B
SpinQuant"
50.06
77.06
$1.38
80.62
76.79
46.00
80.14
46.37
74
68.08
4.53
OSTIQuant
51.37
78.11
62.48
79.51
75.99
45.40
81.18
47.80
74.82
68.52
4.43
RIN
25700
"28.87"
14077
27.29
0.39
25.60
19.87
5513
49.33
316
265
SmoothQuant
22.61
32.87
55.05
31.28
3.40
28.00
53.75
34.65
50.28
34.65
le3
GPTQ
4444
Omniquant
29.10
53.79
54.95
52.44
26.45
30.60
65.56
38.54
53.91
45.04
10.33
QuaRot
51.96
76.22
80.21
80.96
74.52
46.40
79,87
46.67
74.51
67.92
4.77
DFRot
51.11
76.43
80.98
74.87
46.60
79,60
46.21
74.66
67.93
4.81
SpinQuant"
51.62
76.98
81.07
80.57
76.63
46.00
79.92
46.26
74.19
OSTQQuant
49.74
76.52
81.16
81.13
77.57
46.40
80.90
46.11
74.27
68.20
4.42
Table 6: Complete omparison of the perplexity score WikiText2 and averaged accuracy Zero- shot Common Sense Reasoning tasks LLLMA.
33

---

Published as a conference paper at COLM 2025
Model
W-A-KV
#Bits
Method
ARC=c
(f)
ARC-e (*)
BaolQ (")
HellaS. (1)
LambA. (f)
OBQA
(7)
PIQA (f)
SIQA
(1)
WineG. (7)
Avg. (T)
Wiki2
(4)
16-16-16
Full Precision
57.42
81.02
83.79
83.81
79.60
48.80
82.70
49.18
77.98
71.59
3.32
29.35
SmoothQuant
25.00
35.98
58.23
7.49
25.00
54.62
35.21
35,86
GPTQ
27.82
25.80
37.95
25 22.82
0.00
27.00
49.67
33.98
49.72
30.86
nan
QuaRot
55.55
80.18
62.57
81 .65
77.22
46.8
81.77
48.06
75.85
69.96
3.89
DFRot
55.46
78
82.32
79.19
76.87
46.4 86.40
81.23
46.21
75.93
3.99
2-70B
OSTQQuant
SpinQuant
56.61
55.38
78.96
80.51
$3,36
83.03
82.68
79.00
79.11
47.80
17.86
83.00
82.10
48.76
48.67
77.43
76.70
70.56
70.92
3.57
RIN
30.38
27.74
38.23
26.12
0.02
24 60
51.74
34
52.49
31.73
7e4
SmoothQuant
24.15
55.32
52.17
GPTO
28.75
26.39
37.86
26.40
50.00
34.44
50.04
31.09
nan
4444
QuaRot
55.35
79.55
62.02
81.39
78.03
47.2
80.79
47 49
77.58
69.96
3.92
DFRot
54,27
78.32
82.02
79.11
76.81
46.80
81.23
$6.88
73.56
68.78
4.02
SpinQuant"
56.31
83.55
47.20
82.21
OSTQuant
56.58
80.17
83.64
82.49
78.72
48.00
82.76
48.67
76.49
70.84
16-16-16
Full Precision
64.42
85.98
85. 14
84.95
79.47
48.46
84.39
50.82
80.66
73.81
2.86
RIN
27.47
25 88
37.83
26 26
0.00
27 20
55. 63
36.26
49.35
31.2
963
SmoothQuant
25.60
34.47
32.48
1.98
30.00
54.24
33.83
48.93
34.67
2e2
GPTQ
25.77
4-4-16
QuaRot
52.99
76.52
$1.59
79.35
75.1
47.8
79.87
46.37
74.66
68.25
5.92
DFRot
58.28
81.36
81.50
82.21
75.24
45.40
82.2
46.47
75.69
69.82
497
3-70B
SpinQuant'
53.84
77.69
80.24
78.19
73.06
45.00
78.67
43.24
73.01
66.99
OSTQuant
61.84
84.56
84.14
82.47
77.08
46.07
83.36
50.23
80.13
72.21
RTN
27.3
37.85
26.12
0.00
26.60
'0.76
55.66
43.38
3082
963
SmoothQuant
23.46
31.48
48.81
29.22
4.13
28.00
52.56
34.95
51
33.76
3e2
GPTQ
26.11
25.17
45.17
26.07
0.00
26.40
48.86
33.68
49.17
31.20
4e4
4-4-4
QuaRot
DFRot
58.02
81.10
81.13
81.59
74.79
47.40
81.83
46.57
74.19
69.62
5.03
SpinQuant"
51.88
76.39
80.98
76.50
71.43
43.46
79.27
44.17
72.69
66.31
6.24
OSTQuant"
61.29
82.39
83.25
75.90
48.93
8173
51.24
77.01
71.69
Table 7: Complete comparison perplexity score on WikiText2 and averaged accuracy on Zero- shot Common Sense Reasoning tasks for LLaMA2-70B and LLaMA3-70B.
34


Reviews:
[{"review_id": "4OK69zjhOw", "reviewer": "Reviewer_JUny", "summary": "This paper propose a method called DFRot to solve the outlier problem in the quantization of LLM. The method is initialized with Hadamard matrix and adaptively adjusted for LLaMA and Mistral models under W4A4 quantization. The authors find that the stochastic Hadamard transform is more effective in 4-bit activation quantization and based on this employ the QuaRot framework to learn to rotate the matrix to reduce the quantization error of outliers.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Authors verify the motivation and effectiveness of the method through a large number of experiments, which are reasonably and comprehensively designed, and the results support the advantages of the proposed method. By comparing the quantization errors of different rotation matrices when dealing with tokens with a large number of activations, the significant effect of the proposed method in improving the quantization accuracy of the model is visually demonstrated, which enhances the credibility of the paper.\n\n2. The paper focuses on the quantization problem of LLM, especially on solving the model scalability problem due to memory limitation in the quantization process. In the current context where LLM is widely used but faces deployment and computational resource challenges, the research results of this paper provide a reference for the field of model compression and quantization and promote the development of related technologies.", "reasons_to_reject": "1. The paper does not include analyses that explored DFRot tubing memory reduction or acceleration related.\n\n2. The average improvement of the method is not significant and stable enough in zero-shot tasks, such as LLaMA2-70B. And the fact that the zero-shot tasks in the paper contain several different types of tasks makes one doubt whether DFRot can achieve good results consistently in multiple tasks.\n\n3. The paper points out that tokens with large-scale activation have a significant impact on model performance, and experimentally verifies their existence and performance under different quantization methods. However, the paper does not provide an in-depth theoretical analysis of the nature of these special tokens and how they work in the model. This lack of analysis leads to an incomplete understanding of the optimization principles of the DFRot method, which may limit the possibility of further improving and extending the method.", "questions_to_authors": "1. The paper mentions that an alternating optimization approach is used for optimizing the rotation matrix, where the rotation matrix is optimized by fixing the quantization parameters and then the rotation matrix is optimized by fixing the quantization parameters. Is the rotation matrix used in the form of a diagonal block matrix or in the form of a complete D×D matrix? Does this alternating optimization approach lead to the optimization process falling into a local optimum in some cases?\n\n2. The paper mentions using only one data sample for calibration to optimize the rotation matrix, but also points out that there may be differences in the way calibration data are obtained for different models. In practice, if the distribution of the calibration data deviates from the distribution of the actual inference data, will this affect the performance of the DFRot?", "extra_scores": {}}, {"review_id": "Un4xXY0PoI", "reviewer": "Reviewer_gFVi", "summary": "This paper introduces a new LLM quantization method, specifically addressing the challenge of quantizing rare tokens with massive activations. Building upon the observation that RH outperforms RO in mitigating quantization errors for these tokens, the authors frame the issue as a long-tail optimization problem. They propose a weighted loss function that emphasizes minimizing errors for these rare tokens and employ an alternating optimization strategy using orthogonal Procrustes transformations to refine the rotation matrix. This approach enhances the distribution of rotated activations, making them more amenable to low-bit quantization.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. New insight into quantization: first to rigorously attribute RH’s advantage over RO to improved quantization error for rare tokens and validate via FP16 ablation.\n2. Effective post-training method: The weighted loss formulation and alternating update require no full fine-tuning or gradient storage, making DFRot highly practical.\n3. Comprehensive ablations: Varies the γ weight and initialization (RH vs. RO) to demonstrate method robustness and guide practitioners, with clear sweet spots and recommendations.", "reasons_to_reject": "1. Experiments focus solely on WikiText-2 and standard zero-shot NLP tasks. No evaluation on diverse downstream tasks (e.g., MMLU),  model families beyond LLaMA (e.g., Qwen or MoE models).\n2. The alternating algorithm lacks discussion of convergence guarantees, sensitivity to calibration set size.\n3. The paper also overlooks the potential benefit of tuning per-token clipping thresholds for these rare tokens, see Sec. 3.2 of “Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models”.", "questions_to_authors": "See above", "extra_scores": {}}, {"review_id": "jbL86ZktmO", "reviewer": "Reviewer_ZCwi", "summary": "This paper addresses the quantization of Large Language Models (LLMs) to low bit-widths (W4A4), aiming to improve existing rotation-based methods.\nIts main contribution is identifying why Randomized Hadamard (RH) transforms outperform Randomized Orthogonal (RO) transforms: RH better handles rare but critical massive activation tokens.\nBased on this, the authors propose DFRot, featuring a weighted loss function emphasizing these massive activations and an efficient, alternating optimization algorithm using the Procrustes solution to refine the rotation matrix.\nExperiments show DFRot significantly improves perplexity and zero-shot accuracy over baseline rotation methods (QuaRot) on various LLaMA models, including challenging ones like LLaMA3-70B, with minimal optimization overhead.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper provides an empirically supported explanation for the previously observed but unexplained performance difference between RH and RO transforms in low-bit activation quantization, attributing it to the handling of massive activations.\n- The proposed DFRot is an efficient post-training optimization technique that requires only a small calibration dataset and minimal computation time (no backpropagation through the LLM required for optimizing). This makes it a practical add-on to existing quantization pipelines.\n- It demonstrates significant improvements in perplexity and zero-shot task accuracy over baseline methods (QuaRot) across multiple LLaMA models, including notably difficult ones, validating the effectiveness of the approach.\n- The paper clearly identifies and analyzes the massive activation phenomenon as a key bottleneck, providing valuable insights beyond the proposed method itself.", "reasons_to_reject": "- While the empirical evidence for the RH vs. RO difference regarding massive activations is strong, the paper lacks a theoretical derivation or proof explaining why Hadamard matrices interact with quantization and massive activations in this specific beneficial way. The reason for the difference might actually be different.\n- While the explanation and the targeted optimization are novel, the core method builds directly upon existing rotation techniques (QuaRot) and standard optimization components (Procrustes analysis). The improvement, while significant over the baseline, might be viewed by some as incremental compared to other paradigms.\n- The alternating optimization algorithm guarantees convergence to a local optimum, not necessarily the global optimum for the rotation matrix R1. The dependence on initialization (RH is recommended) suggests results might vary.\n- The method requires online computation for the internal rotation matrices R3 and R4. Although mitigated by using efficient Hadamard transforms, the exact latency impact is not thoroughly analyzed, which might be a concern for latency-critical applications.", "questions_to_authors": "- Could you provide a more detailed analysis or estimation of the practical latency overhead introduced by the online computation of the R3 and R4 Hadamard transforms during inference?\n- While a full theoretical proof might be complex, could you elaborate further on any mathematical intuitions or properties of Hadamard matrices that might explain their advantageous behavior with massive activations under 4-bit quantization compared to general random orthogonal matrices?\n- How sensitive is the performance of DFRot to the specific choice of the single calibration sample used for optimizing R1? Have you experimented with multiple different samples? I guess that, if the sample does not produce some patterns of massive activations, does optimization would be bad.\n- The weight gamma for the massive activation loss term is chosen empirically. Have you considered methods to automatically tune or learn this hyperparameter? And, in the experiment, does the choice use the test set as validation for tuning?\n- Would you clarify how to judge massive activations?\n- Figure 2. Please improve the visibility. e.g., colors should differ.", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers agree (6,7,8) this solves an important mystery: why Hadamard beats orthogonal transforms in 4-bit quantization. The insight  to directly optimize the long-tail for large tokens is novel and well-validated. Authors ran all requested experiments including Qwen2/Mistral models and MMLU. The 0.98 perplexity improvement on LLaMA3-70B with just one calibration sample is impressive. Clear accept.
Decision: Accept