Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference
Authors: Ke Yi
Zengke Liu
jianwei zhang
Chengyuan Li
Tong Zhang
Junyang Lin
Jingren Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Large language model
Quantization
INT4 inference
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models have demonstrated promising capabilities upon scaling up parameters. However, serving large language models incurs substantial computation and memory movement costs due to their large scale. Quantization methods have been employed to reduce service costs and latency. Nevertheless, outliers in activations hinder the development of INT4 weight-activation quantization. Existing approaches separate outliers and normal values into two matrices or migrate outliers from activations to weights, suffering from high latency or accuracy degradation. Based on observing activations from large language models, outliers can be classified into channel-wise and spike outliers.
In this work, we propose Rotated Runtime Smooth (**RRS**), a plug-and-play activation smoother for quantization, consisting of Runtime Smooth and the Rotation operation. Runtime Smooth (**RS**) is introduced to eliminate **channel-wise outliers** by smoothing activations with channel-wise maximums during runtime. The Rotation operation can narrow the gap between **spike outliers** and normal values, alleviating the effect of victims caused by channel-wise smoothing.
The proposed method outperforms the state-of-the-art method in the LLaMA and Qwen families and improves WikiText-2 perplexity from 57.33 to 6.66 for INT4 inference.

Published as a conference paper at ICLR 2025
ROTATED RUNTIME SMOOTH: TRAINING-FREE ACTI- VATION SMOOTHER FOR ACCURATE INT4 INFERENCE
Ke Yi! Zengke Liu Jianwei Zhang Chengyuan Li?,
Tong Zhang"!, Junyang Lin" ,Jigrren Zhou
ABSTRACT
Large language models have demonstrated promising capabilities upon scaling up parameters. However, serving large language models incurs substantial computa- tion and memory movement costs due their large scale. Quantization methods have been employed to reduce service costs and latency. Nevertheless, outliers activations hinder the development of INT4 weight-activation quantization. Exist- ing approaches separate outliers and normal values into two matrices or migrate outliers from activations to weights, suffering from high latency or accuracy degra- dation. Based on observing activations from large language models, outliers can be classified into channel-wise -wise and spike outliers. In this work, we propose Rotated Runtime Smooth (RRS), a plug-and-play activation smoother for quantization, consisting of Runtime Smooth and the Rotation operation. Runtime Smooth (RS) is introduced to eliminate channel-wise outliers by smoothing activations with channel-wise maximums during runtime. The Rotation operation can narrow the gap between spike outliers and normal values, alleviating the effect of victims caused by channel-wit wise smoothing. The proposed method outperforms the state- of-the-art method in the LLaMA and Qwen families and improves WikiText-2 perplexity from 37.33 to 6.66 for INT4 inference.
0
INTRODUCTION
Large language models have demonstrated promising capabilities as parameters are scaled up. However, serving large language models is plagued by the high cost of computation and memory movement due to their scale. Consequently, many quantization methods are applied to reduce size and gain throughput improvement. From the service perspective, quantization can be categorized as weight-only quantization and reight-activation quantization. The former can focus on compressing the model's weights and saving costs related to memory movement, which is suitable for the memory- bound decoding stage. The latter quantizes both weight and activation to low bits and utilizes low-bit matrix multiplication kernels to achieve speedup. However, the existence of outliers activation stretches the quantization range, compressing the effective bits for normal values and thus hindering the development of low-bit acttvvttitivtioon quantization.
To address outliers, previous works such (Kim et al. 2023 Dettmers et al. 2022) separate outlier and normal values into two matrices. However, the implementation is not hard hardware-come core-clo and fails to expedite inference. To achieve acceleration and maintain accuracy under A8W8 quan- tization, SmoothQuant (Xiao et al. 2023) transfers appropriate outliers from activation to weight offline through channel-wise smoothing scales. Nevertheless, the offline smoothing scales would be ineffective when facing unmatched input, and the outlier sharing scheme makes weight difficult quantify. The aforementioned reason impedes the implementation of SmoothQuant for A4W4 quantization. QuaRot (Askbooos et al. 2024) utilizes the property that rotation can suppress outliers;
Work done during the internship Owen Group. denotes equal contribution. 1 denotes equal advising. South China Univesity of Technology Alibaba group Key Laboratory of System Software (CAS) and State Key Laboratory of Computer Science, Institute of Software. Chinese Academy of Sciences 4 University of Chinese Academy of Sciences. Project page:
g thub. i wttp:rrs222.......
io/

---

Published as a conference paper at ICLR 2025
2
32
0.5
2
2
2
32
0.5
2
2
2
2
2
Abx Max
Origin
32
-1
2
2
2
32
-1
2
2
2
2
2
-1
32
2
32
outlier
Chaunel-wise outlier
Spike outller
Xdiag(s')
= xdiag(s)
Xdiag(s)"
Smoothing
2
32
2
2
2
0,5
2
2
2
32
2
2
2
8
-1
0.5
-1
W dlag(s
Wi
diag(s)W
maxiX_cclio ccali
max|X|
max [W
max
Vietim
(9) Unmatched Scale
Outtiers migrated weight
(c)
by
Figure 1: Challenges of Smoothhuant faced with outliers. (a) The scale s', which does not match the channel- wise maximums of activations, is ineffective for smoothing purposes. (b) The migration scheme makes difficult quantize the smoothed aetivation/weight down bits. (c) Normal values are pruned as victims after smoothing due to the spike outlier. Note that only calibration but no quantization is involved in the above process.
hence, pairwise rotate the activation and weight with equivalent output. The rotated activation and weight tend to spread outliers internally, leading to effectiveness smoothness for A4W4 quantization. However, rotation cannot guarantee a smoother matrix, and a rotated matrix may still exhibit the shape of channel-wise outliers, depicted in Figure Therefore, deriving more robust, accurate, and training-free scheme for INT4 inference remains an open challenge.
Based the observation, outliers could be categorized into channel- outliers and spike outliers, as shown in Figure [ To handle channel-wise outliers, we propose Runtime Smooth. Firstly, reordered to gather outliers and normal values. Subsequently,
weights and activations activations and smooth activations by dividing group-wise maximums. The later quantized smoothed activation, weight, and group-wise maximums are inputs for the fused GEMM kernel. The entire process incurs minimal overhead compared to the original A4W4 pipeline. However, the existence of spike outliers causes the effect of victims after channel-wise smoothing, as shown in Figure To address both channel-wise outliers and spike outliers, we propose Rotated Runtime Smooth, where we rotate weights and activations following (Ashkboos al. 2024) and apply Runtime Smooth on rotated activations. The spike outlier is spread along with its token, leading to smoother token with consistent values. The consistent values are comparatively larger than the normal values, thereby serving smoothing scales for Runtime Smooth. The genesis of victims is abnormal smoothing scales, and the consistent smoothing scales across channels prevent the existence victims.
group up
To evaluate the proposed method, conducted experiments LLaMA families, Qwen families, Mistral, etc. On LLaMA3- Rotated Runtime Smooth can gain perplexity improvement from 57.33 to 6.66 under A4W4 quantization compared with the state-of-the-an oftthe-art. We summarize contributions as follows:
We comprehensively revisited the activation smoothing method for LLM quantization, concluding the reasons for success or failure under A4W4 quantization.
We propose Runtime Smooth, plug-and-play component that eliminates channel-wise outliers of activation in runtime without migrating outliers to weights, bringing negligible overhead for INT4 matrix multiplication.
We propose Rotated Runtime Smooth to overcome the spike outliers and enhance the robustness for channel-wise outliers. A comprehensive evaluation validates the effectiveness the proposed methods, which gain thorough improvement on various models for INT4 inference.
2

---

Published as a conference paper at ICLR 2025
Probability to generale more oulliers after rotation (%)
Wic
We
Random
Llama-2-138
50
Llama-2-
Llama-3-70B
Qwen1. 1,5BB
Llama-3.17700
SoltMax
3.3
20003
Activation Quantization
Wous
(b) Statistic analysis about Rotation
Rotated Activation
Woxr
RT
Wap
Origin Activation
Kela
Rotated and Quantized weights
W
Woows
R
Framework of rotation based method
(c) Rotated activations can sub -mossmo
Figure 2: Review of the rotation-basee method. (a) illustrates a simple implementation the rotation rotation-based
method. The output from projector is changed since Y (XR) (R "W" Xw" (b) explains the success of rotation-based method on LLMs, where activations have high confidentiality to smoothed after rotation compared with random matrix. (c) illustrates that activation with channel-wise outliers still maintains
sub-smooth after rotation, leaving space for further smoothing
2
PRELIMINARIES
2.1 QUANTIZATION
Quantization converts high-precision matrices into discrete elements with scaling factors, achieving a
lower bit per element. The process of quantization can be expressed as XINT = 1 where X represents the floating-pon point tensor and a is the scaling factor. The existence of outliers stretches the scaling factor and leaves few effective bits for normal values. Dividing the matrix into groups for quantization can mitigate the effect of outliers. In previous literature, the per-tensor quantization considers the entire matrix as a group; the per-channel quantization assigns different scaling factors to each row, and the sub-channel quantization divides rows into fine-grained groups. Although fine- -he-grained grouping can alleviate accuracy degradation, more scaling factors entail additional computation and storage costs. In this work, we adopt the per-channel scheme following (Xiao al. 20233] Ashkboos al. 2024 Liu al. 2024) for INT4 quantization.
max( (X]
2w-1-
2.2 CHANNEL- WISE SMOOTHING METHOD
Under the assumption that outliers persist in fixed channels of activations, SmoothQuant migrates the outliers by dividing the smoothing scale s E RK, where denotes channel dimension. For ensuring equivalence of output, would be multiplied to weight; the process be described as Y (Xdiag(s) (diiags))
XWr. The smoothing scales are computed as Sy = max X)) max([Wa (I, 1,2. K, fairly share the outliers between weights and activations. Since weights are mostly quantized offline (Frantar et al. 2022), directly multiplying during runtime would undermine the quantization property. Therefore, s is pre-computed using calibration set and merged into weights before quantization.
Although SmoothQuant is effective under the A8W8 scheme, it fails for INT4 inference three respects, as shown in Figure [ Firstly, the smoothing scales depending the calibration set are prone to being unmatched with online activations; hence, they cannot smooth outliers. Secondly, the outlier channels of activation failure in low-bit guantization. Thirdly, outliers are not always channel-wise consistent, where spike outliers exist, and normal values pruned 'victim' after smoothing.
not eliminated but partially migrated to weights, thus leading to
3

---

Published as a conference paper at ICLR 2025
Spike Outliers and Effect of Victim: Spike outliers stretch the smoothing scale. The normal values divided by abnormal smoothing scales are minimal compared to other elements the quantization group: hence, they become 'victims'. The existence of the victims leads to quantization error.
2.3 ROTATION BAATID METHOD
A rotation matrix is orthogonal matrix R satisfied RR = I and del = Quip (Tseng al. 22024) showed that multiplying a weight matrix on the left and right by an orthogonal matrix theoretically alleviate outliers, making matrices casier to guantize. QuaRot (Ashkboos et al.) 2024), employs a similar technique by multiplying weight or activations by only one rotation matrix, rotation
maintaining equivalent output depicted in Figure 2 (a). However, multiplying matrix could not theoretically guarantee a smoother weight or activation. To explain the success the snation-bbsed method, we computed the probability that the activation of different models becomes less smooth after rotation, as illustrated in Figure 2
(b). Following previous works, we measure the smoothness as μ abs (max (RMM(())))(((()) (t), where t denotes one token in activation, and RMS denotes root mean square. Rotating activation from LLMs consistently exhibits low probability of being less smooth compared with rotating random matrix. However, having chance be less smooth is a potential trouble. The appendix provides detailed explanation of probability calculation. On the other hand, activations with channel-wise outliers can be viewed as a collection of vectors with the same direction. From the rotation property, the rotated activation might be sub-smooth, as (c), leaving space for better smoothing.
shown in Figure 2
3 METHODOL.OGY
In this section, we introduce Runtime Smooth to eliminate channel-wise outliers 33. and how to implement it efficiently with kernel fusion (3.2p. To comprehensively eliminate outliers, propose Rotated Runtime Smooth, which addresses the effect of the victim and sub-smoothness of rotated activation 3.
3.1 RUNTIME SMOOTH
Challenges for the smoothing- method under INT4 inference are discussed in Section 2.2 One intuitive way mitigate this challenge is to obtain smoothing scale in runtime and not merge into weights. The process can be formulated as:
SmoothQuant Runtime Scale
Origin A4W4
SmoothQuant
Runtime Smooth
PPL
9e2
8c2
4e2
7e2
3c2
Sj max (X j).j 1,2,..., K
X Quantize (Xdiag(s) -) W Quantize(W)
18.2
10.9
10.9
K
W !
A4W4
A4W16
Y
Sjz
Figure 3: Preliminary ablation study
where s E R denotes the runtime smoothing scale, X E RNN denotes activations, W MX denotes weights, X, Nx1 denotes column of activation. We conducted ablation study with LLaMA3-8B and the WikiText-2 dataset to understand better the effect of unmatched scale and outlier shared scheme. As shown in Figure 3 merely applying the runtime smoothing scale could not make 44W4 feasible, whereas Runtime Smooth does, echoing the importance of not migrating outliers weights. To avoid the effect of quantization error from weight, further apply the A4W16 16 setting. The perplexity improvement, from 4e2 to 10.9, validates the effectiveness of adopting the runtime scale.
3.2 RUNTIME SMOOTH WITH KERNEL FUSION
However, the naive implementation cannot integrated into the GEMM pipeline. A GEMM kernel splits the input matrix into blocks by columns and conducts parallel block computation. where the inconsistent smoothing scale would make the block-wise computation complex. Intuitively, if the
4

---

Published as a conference paper at ICLR 2025
122
18
1.2
12
3.6
1.8
Group Max with Group Siiee =2
12
12
0.9
3,6
0.25
24
3,6
Activation
Smoothed Activation
1.2
(2) Group-wise smooth
ArgSort
Block
INT
INT4
Weight
Q-W
INT4
Runtime Scale
1.2
1.2
0.9
3.6
Q_X
Activation
(1) Reorder activations and weights
(3) Kernel fusion for Runtime Smooth
Figure 4: Pipeline of Runtime Smooth. (1) Reorder activations and weight according Channel-wise maximums of activation. Note that the reordering process would not change the final result since Y = >> X, @.?. and Y is irrelevant with the order of (2) Group up activations according to block size of matrix multiplication computation. The maximums of the group multiplication pipeline, quantized smoothed activations and weights are segmented into blocks. equivalent to the previous group size. Within block, tiled smoothed activations are multiplied by tiled quantized weights. The runtime smoothing scales are applied to the interim result.
the runtime smoothing scale of the group. (3) In the matrix block size
Recale
Channcl-siie
(b) Spilkc
Figure 5: Analysis of rotated activations with different outliers. (a) Activation with channel-wise outliers maintains channel-win wise consistency after rotation, hence being sub-smooth for quantization. (b) One spike outlier is spread on its token internal, where the smoothing scale is consistent without abnormal value, further preventing 'victim".
smoothing scale is the same within a block, equation 3.1 can be deduced to Y = The overhead would be minimized due to fewer muluplication operations. Roughly altering the smoothing scales be consistent would degrade the level of smoothness. Hence. we reorder the activations and weights according to the magnitude of smoothing scales, validated as hardware- friendly with negligible cost (Lin et al. 2024b). The reordered activations group up the outliers and normal values. The group size is configured to be identical to the block size. The maximum value of an activation group is set as the smoothing scale for the corresponding computation block.
W
As shown in Figure the pipeline of GEMM fused with Runtime Smooth can be described as 1. Reorder the activation and weight according to channel-wise maximums of activation; 2. Group up the activation and set the group wise maximum smoothing scales; 3. Calculate the matrix multiplication of the tiled block and multiply the runtime scale on the dequantized result, followed by reduction operation. It is worth noting that, compared with the A4W4 baseline, the extra runtime scale multiplication only brings negligible overhead, which will be discussed Section 4.5
3.3 ROTATED RUNTIME SMOOTH
In this section. propose Rotated Runtime Smooth (RRS) comprehensively eliminate outliers, including both channel-wise and spike outliers. Here, the activations are rotated and subsequently applied Runtime Smooth.
5

---

Published as a conference paper at ICLR 2025
For activations with channel-wise outliers, it maintains the channel consistency after rotation due to the rotation property, as shown in Figure Moreover, the rotated activations have a chance to raise the level of outliers with small probability. Compared with the pure rotation method. RRS comprehensively smooths channel-wise, leading to low quantization error. Compared with Runtime Smooth, RRS is robust activations with tiny spikes in the practical scenario.
As discussed in Section 2.2 the existence of spike outlier is the bottleneck of Runtime Smooth. Figure 5 shows that the spike outliers are spread within tokens after rotation. Hence, the smoothing scales are more consistent across channels, leaving fewer victims since all elements are divided by such a consistent scale. The process can be described as:
Ga t [E. ,,, E,,,,,,.... ".,,, ,El, R = VK
I
trotation t -t-R-
1c,10; t,, t.r ,c,Kk 0, +
(I)
K
smooth scale = max( trotationn))
.M. ,10.1]
VK
where the O denotes spike outliers and E denotes normal values. Cij E {-1,+1} denotes the elements in Hadamard rotation matrix. In a practical scenario, the normal values & are relatively small compared with spike outliers O; hence, they are omitted the smoothing scale as shown in Equation The genesis of victims is the abnormal smoothing scale; hence more consistent smoothing scale brought by RRS frees 'victims', Here, we only discuss the scenario where there only one spike in the token, and we conduct comprehensive analysis of multiple spikes for real scenarios in Section
I
We apply RRS to Linear Layers in Transformer blocks. We first offline rotate the weight matrix and insert online rotation operation before output and down projectors following previous works (Ashkboos al. 2024). The rotated weights are quantized offline with GPTQ (Frantar et al. (202)). During inference, we perform Runtime Smooth on the rotated activations of linear layers and apply activation quantization subsequently.
4 EXPERIMENTS
4.1
SETTINGS
We conduct experiments on mainstream L.MMs, including LLaMA families (Touvron et al. 2023b)
(LLaMA2-133) 13B, LLaMA2-70B, LLaMA3-8B. LLaMA3-70B, LLaMA3.1-88 L. LLaMA3
1-70B).
Qwen families Yang et al. 2024) (Qwen1.5-1) 1.5-7B Qwen 1,5--BB,) Mistral (Jiang et al. 2023) and
Mixtral (Jang et al. 2024). Activation quantization employs per-channel symmetric scheme with round-to- ne-aresttt (RTN) strategy. KV cache quantization employs sub-channel symmetrie scheme with groupsize 128 and round-to-nearess (RTN) strategy. In most cases, weight quantization employs per-channel symmetric scheme with GPTQ (Frantar et al. 2022) strategy, except for baseline 'RTN'. We apply standard GPTQ settings by using 128 samples from WikiText-22 with a sequence length of 2048 as the calibration set. We evaluate the performance of the models on WikiTeext22 perplexity and zero-shot Common Sense QA benchmarks. The Common Sense QA benchmarks include ARC-e, ARC-c CCark et al. 2018), BoolQ (Clark et al. 2019). and OBQA (Mihaylov et al. 2018).
4.2
MAIN RESULT
Runtime Smooth emphasizes activation smoothing for INT4 inference. The plug-ands and-ppay Runtime Smooth operators are employed before activation quantization. We conduct a comparison between Runtime Smooth and SmoothQuant (Xiao al. 2023), demonstrating promising improvement, For instance LLaMA2-70B: le2 -> 6.95, LLaMA3-8B: 8e2 ~> 10.47, Qwen @wenl.55-BB. 1.5-BB: 3e2 ->13.32 under the A4W4KV16 scheme, as shown in Table For certain models like LLaMA3-70B and LLaMA3. both Runtime Smooth and SmoothQuanl fail for the difficulty of weight quantization. To eliminate the influence of quantization error from weights, we carry out experiments under A4W 16KV16 settings. Under the activation-only quantization setting, Runtime Smooth consistently outperforms SmoothQuant and achieves 40x improvement on LLaMA3-8B, validating the effectiveness of Runtime Smooth. Here the group size of the smoothing scale is 1 to observe the upper bound performance.

---

Published as a conference paper at ICLR 2025
Table Comparison on WikiText-2 perplexity. evaluate models and methods on three quantization schemes: A4W4KV4, A4W4KV16, and A4W 16KV16. Results for SmoothQuant, GPTQ, and QuaRot were obtained by re implementation based on their publicly released codebase.
#Bits
LLaMA LLaMA
LLaMA
LLaMA
LLaMA LLLMMA
Qwen
Qwen
Method
Mixtral Mistral
W-A-KV
2-13B
2-70B
3-8B
3,1-8B
3-70B
3.1-770
1,57BB
1,5-148
16-16-16
FP16
5.00
3.30
6.13
6.24
2.80
2.81
3.84
5.94
7,95
7.44
RTN
5e3
Nan
4e2
2c2
3e4
lc4
8c2
7e2
6e3
2e4
SmoothQuant
97.04
Nan
5c2
2c2
74.23
48.59
88,68
47.92
2c2
lc2
16-4-16
RS
6.40
4.52
11.44
9.71
11:31
7.27
5.72
7.39
10.95
9.74
QuaRot
5.24
3.72
7.77
7.81
4.68
6.29
9.07
8.18
RRS
5.22
3.74
7.55
7.60
6.32
5.03
4.50
6.21
8.91
8.16
RTN
7c3
2c5
9c2
5c2
le5
2c4
8c2
7c2
9c3
3e4
SmoothQuant
34.50
le2
8e2
4e2
5e2
2+2
3e2
76.25
3e2
1e2
GPTQ
5e3
2e6
9c2
4e2
le5
2c4
le3
6e2
le4
3e4
4-4-16
RS
8.79
6.95
10.47
10.39
7e3
le4
7.37
8.16
13.32
10.38
QuaRot
5.39
3.85
8.38
8.38
57.33
6.26
4.80
6,38
9.34
8.32
RRS
5,36
3.86
8.11
8.12
6,66
5.56
4.63
6.31
9.17
8.29
RTN
7c3
2e5
1c3
6c2
2c4
9c2
7c2
le4
2e4
SmoothQuant
56.60
le2
le3
6e2
5e2
3e2
2e2
78.39
3e2
2e2
GPTQ
5c3
le6
lc3
5c2
le5
2c4
lc3
6c2
le4
3e4
4.44
RS
9.15
7.08
11.52
11.19
8c3
1c4
7.98
8.64
13.97
10.62
QuaRot
5.51
3.89
8.76
8.80
49.73
6.46
4.93
6.45
9.55
8.43
RRS
5.45
3,89
8,42
8.49
6,87
5.69
4.74
6.35
9.37
8.35
To further narrow the accuracy gap between INT4 inference and full precision inference, we propose Rotated Runtime Smooth. Here, the group size of the smoothing scale is set to 128, which is identical to the GEMM block size, enabling efficient implementation. Compared with the state-of-the-an QuaRot, Rotated Runtime Smooth consistently outperforms across different model sizes and model families. most instances, Rotated Runtime Smooth achieves 0.1 0.3 improvement in WikiText-2 perplexity. It is noteworthy that on LLaMA3-70B,, Rotated Runtime Smooth reduces perplexity from 57.33 6.66 under the A4W4KV16 setting and from 49.76 6.87 under the A4W4KVA setting. Notably, 57.33 and 49.77 76 are abnormal results. Hence, we analyze outliers from rotated LLaMA3-70B and 1- The distribution outliers is similar for the two models. More details provided in Section C This indicates that models have different sensitivities outliers. Further suppressing the outliers can enhance robustness.
We also conduct comparison between the proposed approaches and the -tate-of-the-an methods on the zero-shot common sense QA task, as presented Table 2 The Rotated Runtime Smooth consistently surpasses the baseline by approximately 3% in terms of average accuracy improvement. In certain cases, LLaMA338B and LLaMA3.1-888 1-8B, Runtime Smooth outperforms QuaRot. The latter requires complex online Hadamard rotation as described in (Ashkboos et al. 2024). More results listed in the Appendix
4.3 COMPARISON WITH TRAINING-BASED METHOD
SpinQuant [Liu at 2024) suggests that diverse Rotation matrices exhibit variations in their impact of smoothing. Consequently, it substitutes the origin fix Rotation matrix with trainable Rotation matrix and trains the rotated network. The training process is time-consuming, taking 1.5 hours for 7B model one A 110 GPU and 12 hours for 70B models eight A100 GPUs. We re-implement SpinQuant and compare it with our method as shown in Table 3 It is noteworthy that SpinQuan applies asymmetric quantization to activation and KV cache. We experiment with our method under the same settings. The result reveals that the training- based method degrades WikiText-2 perplexity compared to the training-free method and still has room for improvement.

---

Published as conference paper at ICLR 2025
Table 2: 0-shot accuracy (%) the Common Sense QA tasks. Each block is based on the same
foundation model specified in the row. We organize all results under different quantization schemes.
#Bits
Model
Method
OBQA
BoolQ
ARC E
ARC C;
Avg.
GPTQ
26.6
44.0
29.8
21.0
30.4
SmoothQuant
25.0
43.7
30.2
22.9
30.4
RS
42.4
65.6
72.2
46.3
56.6
LLaMA-3-8B
QuaRot
39.2
70.7
68.6
41.5
55.0
RRS
42.4
73.6
71.1
44.8
58.0
GPTQ
26.2
44.7
30.3
23.9
31.3
SmoothQuant
26.2
46.42
30.9
24.5
32.0
RS
44.6
68.8
72.5
47.6
58.4
LLaMA-3.1-8B
QuaRot
37.6
75.6
72.9
44.8
57.7
RRS
42.4
78.1
76.1
50.3
61.7
GPTQ
27.0
46.1
32.0
25.0
32.5
SmoothQuant
28.6
57.5
42.2
31.4
39.9
4-4-16
RS
35.2
80.2
72.4
51.1
59.7
Mistral
QuaRot
44.4
83.1
71.9
54.4
63.4
RRS
43.4
85.1
74.4
55.4
64.6
GPTQ
28.2
42.9
25.9
26.5
30.9
SmoothQuant
28.2
54.0
32.3
26.7
35.3
RS
37.6
72.6
56.9
39.0
51.5
Qwenl.5-7B
QuaRot
39.0
73.9
61.3
40.4
53.7
RRS
43.0
77.7
61.5
42.0
56.0
GPTQ
28.2
42.0
28.8
24.1
30.7
SmoothQuant
28.8
52.2
37.6
28.8
36.9
RS
43.2
65.7
67.2
41.5
54.4
LLaMA-3-8B
QuaRot
38.6
70.6
66.7
40.4
54.1
RRS
42.4
73.5
68.7
44.7
57.3
GPTQ
28.0
43.6
29.0
23.3
31.0
SmoothQuant
28.0
45.1
30.0
22.7
31.4
RS
43.4
65.3
71.1
46.1
56.5
LLaMA-3.1-8B
QuaRot
37.2
73.5
65.7
41.7
54.5
RRS
41.8
77.8
75.3
48.1
60.8
GPTQ
24.8
46.2
31.
25.8
32.0
SmoothQuant
27.0
56.6
40.6
29.5
38.4
4-4-4
RS
36.6
81.0
72.4
50.3
60.1
Mistral
QuaRot
43.2
83.0
72.8
52.5
62.9
RRS
45.6
84.3
73.2
54.6
64.4
GPTQ
30.0
42.8
26.2
26.5
31.4
SmoothQuant
28.0
53.1
30.2
25.1
34.1
RS
36.8
70.2
56.2
40.2
50.9
Qwen1.5-7B
QuaRot
39.6
74.3
62.1
42.0
54.5
RRS
40.4
76.6
61.8
42.5
55.3
Table 3: Comparison with the training- based method, SpinQuant, where the result was obtained by re-implementation based on its publicly released codebase
Method
LLaMA-2-7B
LLaMA-2-13B
LLaMA-3-SB
LLaMA-3.I-BB
Spinquant 6.37
5.55
7.99
7.93
QuaRot 6.03
5.35
7.91
7.89
RRS
5.99
5.29
7.79
7.76
8

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study of Group size of runtime smooth scale
Method
Model
32
64
128
256
Mistral
6.29
6,28
6.30
6.31
6.31
6.33
RRS
Qwen1.5-17
9.16
9.14
9.16
9.17
9.18
LLaMA3,1-8B
8.09
8.09
8.10
8.12
8.18
8.29
Mistral
8.16
11.17
11.42
18.48
36.46
154.19
RS
Qwen1.5-7B
13.32
15.67
18.08
22.83
41.55
LLaMA3,1-8B
10.39
13.51
20.74
51.63
214.88
2706.82
300
250
200
Tops
150
100
100
Ours
Ours
50
50
W4A4 Per-Chameel
W4A4 Per-Chamne
W4A4 Sub-Channel
W4A4 Sub-Channel
0
1000
2000
3000
4000
1000
2000
3000
4000
Batch size(#)
Batch size(#)
Figure 6: Performance evaluation of different quantization approaches. We set up the evaluation configuration aligned with the LLaMA-7b configuration and 1024 sequence length. Kernels are evaluated by NVBench
4.4 ABLATION STUDY
We conduct an ablation study on the group size of runtime smoothing scale for Mistral, Qwenl.5-77 and L 1-B,, as presented 4 Runtime Smooth can effectively minimize the gap between A4W4 and full precision through subtle group strategy. However, the accuracy deteriorates as the group size increases. Rotated Runtime Smooth employs the rotation technique to minimize the gap between outliers and normal values, thereby being robust to the coarse group scheme and enabling the implementation the fused kernel. It should be noted that in Qwen1.5 1.5-7B, the size of the input activation for Down projector is 1008, which does not support group size of 512.
4.5 EFFICIENCY EVALUATION
We evaluate the GEMM kernel fused with Runtime Smooth on NVBench (NVIDIA) 2024) with RTX 4070 Ti, as shown in Figure 6 The group size of the smoothing scale is 128, the same as the block size of the GEMM kernel. We implement Per-Channel A4W4 and Sub-Channel A4W4 as baselines. Compared with Per-Channel A4W4, Runtime Smooth fused Kernel brings limited overhead, including the movement of the smoothing scale (shape [1, KJ) and a multiplication between matrix and scalar. Sub-Channel A4W4 brings noticeable overhead, including the movement of group-wise quantization scale (shape of [N, LJ and [M, LJ) and multiplication between matrices. Hence, across different batch sizes and hidden dimensions. Runtime Smooth fused Kernel brings negligible overhead compared with A4W4 Ber-Chnneel quantization, which is also the setting of QuaRot and SpinQuant.

---

Published as a conference paper at ICLR 2025
5
RELATED WORK
5.1 LARGE LANGUAGE MODELS
Pre-trained language models have achieved remarkable progress through scaling. Open-source large language models (LLMs) can reach up to 405 billion parameters and offer promising few-shot/zero- shot results. The mainstream LLMs Yang et al. 2024) Touvron et al.) [023a] DeepSeek-AN et al.) 2024]Jiang et al. 2023) continuously provide models with large scales and enhanced capabilities. However, serving large language models for inference becomes costly and challenging as the models expand.
5.2 MODEL QUANTIZATION
Quantization represents an effective approach for reducing model size and expediting inference. From a serving perspective, quantization be classified into weight-only quantization and weight- activation quantization. Weight-only quantization compression employs low-bit representations for weight matrices, thereby saving memory movement in memory
scenarios, specifically the decoding stage. GPTQ (Frantar et al., 2022) used 4-bit to quantize the weight based on the approximate second-order information. AWQ (Lin et al.] 2023) further advanced aceuracy by preserving salient weights. QFA Yi et al.
20224) fine-tune supernetwork encompassing multiple mixed precision configurations and efficiently offer high-performance sub-networks for diverse scenarios. QuiP (Chee et al.) 2024 T Tseng al. 2024) successfully represents weights using 2 bits via an adaptive rounding method. Weight-activation quantization can further accelerate computation by leveraging low-bit GEMM kernel suitable for compute- bouud scenarios, namely the pre- pre-filling stage. Qserve (Lin et al., 2024b) further implement the A8W4KV4 and better accelerate. Among weight- activation quantization methods, the existence of outliers presents the most formidable problem, it can result in substantial drops in accuracy.
5.3 OUTLIERS CHALLENGE
Outliers can expand the quantization range and compress the information intensity for normal values. LLM.int8) (Dettmers et al. 2022) employs mixed INT8/FP16 decomposition to handle activation outliers. Nevertheless, such an implementation results significant latency overhead and can even be slower than FP16 inference. Subsequent work (Yuan al. 2023) rearranges the channels to reduce the variance within one quantization group. further enhancing accuracy. Atom (Zhao et al. 2024) integrates the reorder technique and mixed INT4/INTS precision maintain accuracy and accelerate compared to the FP16 baseline. SmoothQuant (Xiao et al. 2023) exchanges outliers between weights and activations find an optimal point that shares appropriate outliers in weights and activations, achieving A8W8 inference with minimal accuracy degradation. However, the outlier sharing scheme may not be suitable for the extreme A4W4 setting. To further smooth outliers, QuaRot (Ashkboos et al. 2024) pairwise rotates the activation and weight to suppress outliers and maintain output equalization, enabling INT4 inference with well-smoothed activations. SpinQuan (Liu et al. 2024) suggests substituting the origin fix Rotation matrix with a trainable Rotation matrix could improve accuracy. DuQuant (Lin et al. 2024a) involves double rotation and achieves a more smooth activation.
CONCLUSION
This work presents Rotated Runtime Smooth, plug-and-play runtime activation smoother that facilitates INT4 inference. In Rotated Runtime Smooth, Runtime Smooth effectively eliminates channel-wise outliers. Additionally, rotation operations migrate the negative impact from spike outliers. Rotated Runtime Smooth can be easily implemented with negligible overhead and is generalized across various large language models (LLMss.. Through comprehensive elimination of channel-wise and spike outliers, Rotated Runtime Smooth achieves significant enhancement compared to the prior channel-wise smoothing approach and outperforms staeeof-thhear methods on diverse models for INT4 inference.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ashkboos, S., Mohtashami, A., Croci, M. L.,, Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, (2024). Quarot: Outlier-free 4-bit inference in rotated llms.
Chee. J., Cai, Y., Kuleshov, V., and Sa, C. D. (2024). Quip: 2-bit quantization of large language models with guarantees.
Clark, C., Lee, K., Chang. M.-W. W., Kwiatkowski, T., Collins, M., and Toutanova, K. (2019). BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044
Clark, P.. Cowhey. I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018). Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint 1rXiv:1803.053.
DeepSeek-AI, Liu, A., Feng, B., Wang. B., Wang, B., Liu, B.. Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., Yang. D., Chen, D., Ji. D., Li, E., Lin, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Yang. H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L.., Liang. J., Guo, J., Ni, J., Li, J., Chen, J.. Yuan, J.. Qiu, J., Song, J., Dong, K., Gao, K., Guan, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L.,, Zhang. L.. Li, M., Wang, M., Zhang, M., Zhang, M., Tang. M., Li, M.. Tian, N., Huang, P., Wang, P., Zhang, P., Zhu, Q. Chen, Q. Du, Q. Chen, R. J., Jin, R. L., Ge, R., Pan, R., Xu, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ma. S., Wang, S., Zhou, S., Yu, S., Zhou, S., Zheng. S., Wang, T., Pei, T. Yuan, T., Sun, T., Xiao, W. L., Zeng, W., An, W., Liu, W., Liang. W. Gao, W., Zhang, W., Li, X. Q.. Jin. X., Wang. X., Bi, X., Liu, X., Wang, X., Shen, X., Chen, X., Chen, X., Nie, X., Sun, X., Wang, X., Liu, X., Xie, X., Yu, X., Song, X., Zhou, X., Yang, X., Lu, X., Su, X., Wu, Y., Li, Y. K., Wei, Y. X., Zhu, Y. X., Xu, Y., Huang. Y., Li, Y., Zhao, Y., Sun, Y., Li, Y., Wang, Y., Zheng, Y., Zhang, Y., Xiong, Y., Zhao, Y., He, Y., Tang, Y., Piao, Y., Dong, Y., Tan, Y., Liu, Y., Wang, Y., Guo, Y., Zhu, Y., Wang. Y., Zou, Y., Zha, Y., Ma, Y. Yan, Y., You, Y., Liu, Y., Ren, Z. Z Ren, Z., Sha, Z., Fu, Z., Huang, Z., Zhang, Z., Xie, Z., Hao, Z., Shao, Z., Wen, Z., Xu, Z., Zhang, Z., Li, Z., Wang, Z., Gu, Z., Li, Z., and Xie, Z. (2024). Deepseek-v22 A strong, economical, and efficient mixture-off-expert language model.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. (2022). Llm.int8)): 8-bit matrix multiplica- tion for transformers at scale.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2022). Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 13X.1:2210.13323.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. (2023). Mistral 7b.
Jiang, A. Q.. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L.. Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. (2024). Mixtral of experts.
Kim, S., Hooper, C., Gholami, A., Dong, Z.,, Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. (2023). Squeezellm: Dense-and- quantization. arXiv preprint arXiv:2306.076.
Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, L., Song, L., Sun, Z.,, and Wei, Y. (2024a). Duquant: Distributing outliers via dual transformation makes stronger quantized llms. arXiv preprint arXiv:240..07221.
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. (2023). Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00978
Lin, Y., Tang, H., Yang, S., Zhang, Z.,, Xiao, G., Gan, C., and Han, S. (2024b). Qserve: W4a8kv4 quantization and system co-design for efficient Imm serving.
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. (2024). Spinquant: Llm quantization with learned rotations.

---

Published as a conference paper at ICLR 2025
Mihaylov, T., Clark, P., Khot, T.,, and Sabharwal, A. (2018). Can suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.
NVIDIA (2024). Nvbench: Nvidia's benchmarking tool for gpus. Available online: https 7/github. COW.///D////DDDA//be....
Shazeer, (2020). Glu variants improve transformer.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E.,, Azhar, F., et al. (2023a). Llama: Open and efficient foundation language models. arXiv preprint rxXv:23022.13771
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. (2023b). Llama: Open and efficient foundation language models.
Tseng. A., Chee, J., Sun, Q., Kuleshov, V., and Sa, C. D. (2024). Quip: Even better Ilm quantization with hadamard incoherence and lattice codebooks.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. (2023). Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pages 38087-38099 PMLR.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, Ma, J., Yang. J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao. R., Lin, R., Wang. S., Bai, S., Tan, S., Zhu, T., Li, T. Liu, T. Ge. W.. Deng. X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y.. Zhang, Y., Wan, Y., Chu, Y. Liu, Y., Cui, 7.,, Zhang, 7.,, Guo, Z., and Fan, Z. (2024). Qwen2 technical report.
Yi, K., Xu, Y., Chang, H., Tang, C., Meng, Y., Zhang, T., and Li, J. (2024). One quantilm for all: Fine-tuning quantized Iims once for efficient deployments.
Yuan, Z., Niu, L.., Liu, J., Liu, W., Wang, X., Shang, Y., Sun, G., Wu, Q., Wu, J., and Wu, B. (2023). Rptq: Reorder-based post-training quantization for large language models.
Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikei, B. (2024). Atom: Low-bit quantization for efficient and accurate Ilm serving. Proceedings of Machine Learning and Systems, 2:196-209.
12

---

Published as a conference paper at ICLR 2025
A SUPPLEMENTARY EXPERIMENT
A. MORE DOWNSTREAM TASK
In this section, incorporate additional downstream tasks such as MMLU, GSM8K (Math), C-EVAL (Chinese), and HumanEval (code), utilizing 7/8B models from Qwen2.5, Mistral, LLaMA 3, and LLaMA 3.1. The results show that our method effectively smooths the activations and works with out-of-disse of-distributton (OOD) data like math and code, making the improvements over the baselines not so small.
Table 5: O-shot accuracy (%) on GSM8k, MMLU, C-Eval with Qwen2.5-7B, Mistral, LLaMA-3-88, LLaMA-3.1-8B.
Model
Method
GSM8K flex
GSM8k strict
MMLU
C-Eval
(Math)
(Math)
(Chinese)
Qwen2.5-7B
RRS
70.96
62.85
63.00
66.6
Qwen2.5-7B
QuaRot
62.77
52.99
62.15
66.1
Mistral
RRS
37.30
36.69
56.65
40.6
Mistral
QuaRot
31.15
30.62
55.16
39.3
LLaMA3-8B
RRS
31,46
31.38
53.51
35.5
LLaMA3-8B
QuaRot
24.71
24.56
50.85
31.3
LLaMA3.1-80
RRS
36.08
35.93
54.96
38.8
LLaMA3.1-SB
QuaRot
28.50
28.27
51.64
34.1
Table Evaluation on HumanEval with Qwen2.5-7B.
Model
Method RRS
HumanEval (Code)
Qwen2.5-7B
60.98
Qwen2.5-7B
QuaRot
49.39
A.2
MORE BASELINES
We compare method with two additional baselines: SpinQuant and DuQuant, well as their variants that incorporate Runtime Smooth (RS). The results show that our runtime smooth is well generalizable and valid, demonstrating the importance of runtime smoothing for various data from different domains.
Table 7: O-shot accuracy (%) the Common Sense QA tasks. Each block is based on the same foundation model specified in the row. We organize all results under different quantization schemes.
Method
GSM8k
GSM8k, Striet
MMLU ARC C
ARC E
BoolQ
OBQA
Avg.
Flex
RRS
31.4
31.3
53.5
44.8
71.1
73.6
42.4
57.9
DuQuant
24.4
24.0
49.4
45.8
68.7
71.3
42.2
57.0
DuQuant+RS
32.9
32.4
55.7
47.5
74.0
76.2
42.6
60.1
SpinQuant
22.9
22.6
48.4
43.2
67.3
72.7
38.8
55.5
SpinQuant+RS
23.1
23.2
52.2
45.1
70.6
73.6
39.4
57.1
B MEASUREMENT OF PROBABILITY
In this section, we explain how to obtain the result in Figure 2 We collect activations cosen-wise for each module in the WikiText-2 test set for LLMs. In order determine the probability of the metric, we first compute the smooth measure = abs max (t) /t), where t denotes one token in activation, and RMS denotes root mean square. Then rotate the activation with the default
13

---

Published as a conference paper at ICLR 2025
Hadamard matrix R and recalculate μ' = abs(max (RM))))(())( +@ Finally, we calculate the probability of < pl.
C OUTLIER
C.1
ACTIVATION WITH MULTIPLE SPIKE OUTLIERS
This section further analyzes how tokens with multiple outliers behave after rotation. The token with multiple outliers is defined as:
t-
O22 Oa,, ..
(2)
where denotes the number outliers, (2,,12,...1)) denotes the index of outliers. In this work, we set the Hadamard matrix the rotation matrix R = vK [8t.]] KxK. where St.j E {11, The rotated token can be described as:
rot
(3)
O
Σ
Σ Oial-
K
d=1
The construction of grot can be viewed as contributions of outliers, where outliers are canceled out by each other stacked to enlarge. The effect of victims refers the smoothness of normal tokens after smoothing. The process is defined as:
x = 111,1, 11.
(5)
scale [absmax
1,
8ig.1 Oie ,... absmmax
1,
Sig. Oig
(6)
d=1
d=1
where we assume normal tokens are filled up with 1. The effect of victims can be qualified with the equation:
Xsmooth = 1/scale, maxx(Xemooth )/ /RMS (MMSS(XxmmoxA)) emmooth))
To measure the effect of victims in the actual scenario, we first collect the activations from LLaMA3- 8B. For activations from the Down Projector, spike outliers are 1000x larger than the medium value, as shown in Figure where outliers in a channel-wise manner are not overly large.
To analyze the effect of smoothing rotated spike outliers, apply the Monte Carlo approach by generating token from the Gaussian distribution and inserting spike outliers according to statistics of spike outliers, then rotating, smoothing, and calculating as shown in Figure g Rotated tokens with multiple outliers are up and down across channels due to the effect of offset and stack. On the other hand, we can stack rotated tokens to obtain consistent large scale across channels. As shown in Figure normal tokens after smoothing mostly casy quantify but contrary when two outlier tokens are exhibited in one activation. The reason that two tokens cannot cover the whole channel, where more stacked tokens lead to lower Notably, the with only two outlier tokens is rare but could potentially trouble Rotated Runtime Smooth.
C.2 ANALYSIS EXTENT OF OUTLIER REMOVAL FOR DIFFERENT METHOD
We conduct experiments on mainstream models with different outlier smoothing approaches to analyze smoothness integrally rather than simulating with manual spike outliers. Specifically, we collect activations with models evaluated by WikiText-2 and apply different smoothing approaches. To measure the level of outliers; we set absmax (t) /||t /||t|/ |2. where t denotes one token. T specific impact of approaches on outliers on different LLM's components. For channel-wise consistent; hence, Runtime Smooth Down_
UP Projector, and Gate project, the activations outperforms rotation, where pure rotated activations are sub-smooth. For Down, Projector, the
intermediate activations contain spike outliers due to SwiGLU (Shazeer 2020) functions; hence, Runtime Smooth suffers from the effect of the victim and fails to smooth. Rotated Runtime Smooth solves two kinds of outliers and consistently outperforms other approaches.
14

---

Published as a conference paper at ICLR 2025
Layer 0
Layer 1
Layer 2
Layer 3
30"
10a
104
10"
10a
104
10-
100
10000
500
500
1000
500 Layer
1000
Layer
Layer
Layer
10a
10a
10a
10a
101
10
500
1000
100
500
1000
1000
500
1000
500
1000
Layer
Layer
Layer
10"
10
101
101
109
302
100
500
1000
100
500
1000
100
500
1000
1.00
500
1000
Layer 12
Layer 1.3
Layer 2.4
Layer 15
20"
10a
10a
10-
10a
10a
100
102
100
500
1000
100
500
1000
100
500
1000
1000
500
1000
Layer 16
Layer 17
Layer 18
Layer 19
10a
10a
10a
10'
102
107
101
201
3.0
109
10a
500
500
1000
500
Layer
Layer
Layer
Layer
2.0
10a
102
10"
20*
101
5.00
1000
500
1000
500
1000
500
1000
Layer 24
25
26
27
207
10a
102
10a
104
164
101
207
100
207
10a
100
5.00
1000
100
500
1000
100
500
1000
1000
500
1000
Layer 28
Layer 29
Layer 30
Layer 31
10a
10'
102
10'
10
10'
10a
109
10a
100
500
1000
100
500
1000
100
500
1000
100
500
1000
Figure 7: Collecting the activations as input of Down Projector with the full precision LLaMA3-8B model evaluating on WikiText-2 and counting the magnitude and number of spike outliers. The magnitude is calculated by a/ meeiimmm where token, and the element of the token. We separately count the spike outliers with different magnitude intervals.
15

---

Published as a conference paper at ICLR 2025
Layer 1
Layer
$
10
10
10
10
10
10
10
Layer
Layer 5
Layer
Layer 7
+
20
20
10
5
Layer
Layer
Layer
15
10
10
10
5
10
10
10
Layer
Layer 13
Layer
Layer 15
20
15
10:
10
10
10
Layer 16
Layer 17
Layer
Layer 19
30
20-
20
30
4
15
15
10-
10
10
Layer
Layer
Layer
30
20
10
10
10-
10
5
10
10
10
Layer 24
Layer 25
Layer 26
Layer 27
20
20
20
10
5
10
10
10
Layer 28
Layer 29
Layer
Layer 31
25
20
20
20-
10
10
10
Figure 8: Simulation for the effect of victims after smoothing with rotated spike outliers. The magnitude and number of spike outliers are configured according to Figure The X-axis denotes the number of spike tokens in activation. The Y-axis denotes of the normal tokens divided by smooth scales.
16

---

Published as a conference paper at ICLR 2025
QXV Projezter
Ouiput Projector
Up, Proj Gitee, Proj
Projector
R3
1000
10
1015
R033
JOCS
I Lbma2-7B
KS
1085
R
RUS
X
RR3
X
RS
P0SS
KRS
Lamm3-88
R
R3
KR3
R
RRS
X
R3
RRS
X
RS
RRS
R3
10SS
R
0US
RKS
10XS
Liama3-708
RS
R
RRS
X
RRS
X
RS
POS
Clma3.1- 1-70R
KRS
Figure 9: Statistic analysis of outlier removal with different smooth approaches. We collect the activations with full precision models evaluating on WikiText-2. 'X' denotes origin activations. 'R' denotes rotated activations, 'RS' denotes activations after Runtime Smooth, and 'RRS' denotes activations after Rotated Runtime Smooth.
17


Reviews:
[{"review_id": "VdjAOq2rtj", "reviewer": "Reviewer_fXyG", "summary": "A plug-and-play quantization method based on runtime smoothing and the rotation operation of activations has been proposed. Runtime smoothing is responsible for eliminating channel-wise outliers, while the rotation operation mitigate the gap between spike outliers and normal values, resulted by channel-wise smoothing. Experimenting with INT4 inference on different LLMs shows that the proposed method can reduce the perplexity compared to other approaches.", "questions": "Please see my comments above.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Combining two exiting ideas in the Quantization literature to overcome both channel-wise and spikes outliers\n- Comprehensive experiments with 3 different LLMs, including LLaMA, Qwen, Mixtral and Mistral models on WikiText-2 perplexity", "weaknesses": "Some notations need to be corrected. For example,\n- Line 168: identity matrix and number 1 should be distinguished. $|.|$ the norm needs to be determined. \n- Line 175: $absmax$ ---- > $abs(\\max)$\n- Equation (1): $\\mathrm{X_j}$ needs to be defined as the columns of matrix $\\mathrm{X}$.\n- Equation(2): $/$ is not a valid operation for matrices. You need to write the formula as a multiplication with a diagonal matrix.\n- line 214: The condition needs to be re-written: $s_j$ cannot be taken out of the sum.\n\nAnother issue is that the proposed method has been only tested for the  WikiText-2 perplexity. Can the approach is applicable for other modalities and tasks. For instance, ASR, speech translation, Image understanding, etc.\n\nThere is only a small improvement over QuaRot approach. I am wondering if your results are statistical significant?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NDjJfQbAfJ", "reviewer": "Reviewer_2Usn", "summary": "The paper proposes a quantization method for LLMs, utilizing a smoothing operation during runtime to mitigate the issue of spike outliers and normalize values. Experiments conducted under INT4 settings validate the effectiveness of this method.", "questions": "See Weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is clearly structured, making it easy to follow.\n2. The method’s motivation is substantiated by experimental results, lending it credibility. In particular, the experiments in Section 4.5 demonstrate the minimal overhead introduced by the method, highlighting its practicality.", "weaknesses": "1. **Figure 1 is confusing and hard to interpret.** The differences between panels (a) and (b) are unclear, as they are on completely different scales despite both sharing the same $ W $ and $ X $. Additionally, have you quantized $ \\hat{X} = X \\text{diag}(s)^{-1} $? The elements in $ \\hat{x} $ appear to be integers, though they should be floats.\n\n2. **Some terms and phrases are difficult to understand.** For instance, the caption for Figure 2(c) uses the phrase \"channel-wise consistency\" without providing any explanation, making it hard to grasp. Similarly, in point (1), what exactly is meant by \"unmatched scale\" in Figure 1?\n\n3. **The motivation closely resembles prior work.** For example, DuQuant [1] also begins by addressing layers with massive outliers (in this paper, it is named as \"spike\") and tackles both massive and normal layers. Similarly, the method in [2] addresses large activation outliers and provides more analysis on this topic than the present paper.\n\n4. **Some claims lack theoretical or experimental support.** For instance, the claim that \"smoothing scales depending on the calibration set are prone to being unmatched with online activations\" would benefit from experimental results demonstrating the likelihood of such a mismatch.\n\n5. **Some explanations are too brief.** For example, the term \"reorder\" is mentioned without much elaboration, described only as \"reordering the activations and weights according to the magnitude of smoothing scales.\"\n\n6. **The method lacks clarity in its description.** In Equation 3, $ \\hat{X} $ and $ \\hat{W} $ appear to be in INT4 format, while $ s_i $ is a float (assuming this is correct, as the authors do not specify). How are these terms compatible for multiplication? A DeQuantize operation may need to be included somewhere in Equation 4.\n\n7. **The ablation study feels incomplete.** The method employs several components—reordering, online smoothing, etc.—but the impact of each isn’t fully explored.\n\n8. **The performance is underwhelming.** The proposed method does not appear to outperform SpinQuant [3] or DuQuant [1], even though it uses online quantization, whereas SpinQuant and DuQuant are offline methods.\n\nMinors: \n1. In Equation 4, $ c_{i,j} $ should be $ s_{i,j} $.\n\n[1] Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, L., ... & Wei, Y. (2024). DuQuant: Distributing outliers via dual transformation makes stronger quantized LLMs. *arXiv preprint arXiv:2406.01721*.  \n\n[2] Yang, J., Kim, H., & Kim, Y. (2024). Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs. *arXiv preprint arXiv:2405.14428*.  \n\n[3] Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., ... & Blankevoort, T. (2024). SpinQuant—LLM quantization with learned rotations. *arXiv preprint arXiv:2405.16406*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7gSrW0hYk0", "reviewer": "Reviewer_f1tA", "summary": "This paper extends SmoothQuant [Xiao 2023] by:\n - Performing Quip-like [Tseng 2024] rotation before activation smoothing;\n - Obtaining smoothing scale in runtime and not merging it into weights;", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is generally well-written, with a comprehensive ablation study (RRS vs RS and RRS vs SmoothQuant) demonstrating the effectiveness, which is well motivated by the preliminary section.", "weaknesses": "The paper lacks an evaluation on the clock-time (instead of the operation count) overhead introduced by the method. This is a concern given that the method determines activation smoothing scale during runtime.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SILGr9GAN5", "reviewer": "Reviewer_MyVY", "summary": "The authors were the first to highlight that channel-wise and spike outliers cause unnecessary distributional expansion in the quantization of large language models (LLMs).\n\nThe authors proposed a concise and effective technique called Rotated Runtime Smooth, which reduces these outliers to ensure that the activation distribution for LLM quantization remains compact.\n\nThe authors functionally explained, along with the methodology, how Runtime Smooth addresses channel-wise outliers and Rotated Smooth effectively handles spike outliers.\n\nThe authors demonstrated effective performance improvements across various LLM models using this approach.", "questions": "Please refer to my comment for Weaknesses section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors focused on addressing channel-wise and spike outliers for the first time, which highlights a novel problem-solving approach and demonstrates significant performance improvements.\n\nThe methodology of Rotated Runtime Smooth (RRS) is simple, intuitive, and provides a clear and straightforward explanation of how it addresses each issue effectively.\n\nBy making their code publicly available, the authors have supported further research and made a valuable contribution to the ICLR community.", "weaknesses": "The explanation of the methodology is insufficient: For example, in Figures 1 and 4, which aim to explain Runtime Smooth, it is difficult to gain a proper understanding. The main text also lacks adequate explanation of these figures.\n\nFor instance, in Figure 4, while the activation and weight reorder process for Runtime Smooth is presented, there isn’t enough explanation about why this process does not alter or distort the final result. Similarly, in Figure 1, it is unclear what the value of s represents (in relation to the equations in the text) and how exactly it reduces channel-wise outliers and effectively resolves the victim issue caused by spike outliers.\n\nThese examples highlight the need for a more detailed and clear description of the methodology, possibly in the appendix. This would help ensure that readers can thoroughly understand the method. While the high-level intuition is conveyed, the reproducibility of the method should be clearly delivered to readers through the paper itself, without relying on the availability of the code.\n\nIt is essential to discuss why this study is limited to INT4. The limitations and advantages should be clearly articulated. For example, there is no comparison with recent trending quantization techniques, such as One-bit LLM. A thorough investigation of related work is necessary, and efforts should be made, at least indirectly, to demonstrate the applicability and effectiveness of the proposed method in relation to these latest quantization techniques. https://arxiv.org/abs/2310.11453\n\n======\n\nAssuming that the above points are adequately addressed, I will start with a borderline or higher score. However, if it is determined that these points are not sufficiently reflected, or based on the overall opinions of other reviewers or AC, I will consider adjusting my final score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cru8Tg0wlc", "reviewer": "Reviewer_fKdX", "summary": "This paper proposed a quantization technique designed to address outliers during Large Language Model (LLM) quantization, particularly for aggressive scenarios like W4A4 quantization. The method employs two steps: 1) mitigating channel-wise outliers through rotation, and 2) implementing online smoothing to further handle spike outliers. The authors demonstrate promising accuracy results across several benchmarks.", "questions": "Please refer the last item of Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors provide a comprehensive analysis of activation distribution patterns, offering well-reasoned solutions for each identified challenge.\n- They proposed a training-free approach that outperforms existing training-based methods on some benchmarks, reducing engineering complexity and showing potential for practical applications in industry scenarios.", "weaknesses": "- Limited novelty: The core methodology is primarily based on existing approaches like [Smooth quantization](https://arxiv.org/abs/2211.10438 ) and [QuaRot](https://arxiv.org/abs/2404.00456 ), while the reordering technique shows overlap with existing research such as [ATOM](https://arxiv.org/abs/2310.19102) and [LLM.int8()](https://arxiv.org/abs/2208.07339).\n- The experimental evaluation is limited in scope, focusing on common sense QA tasks. To better validate the method's effectiveness, the paper would benefit from:\n    - More comprehensive comparisons with training-based methods like [SpinQuant](https://arxiv.org/abs/2405.16406) on more complex tasks, like [MMLU](https://arxiv.org/abs/2009.03300)\n    - Detailed kernel performance comparisons with related research like [Qserve]( https://arxiv.org/abs/2405.04532), and [ATOM](https://arxiv.org/abs/2310.19102)\n\n- Several aspects of the paper's presentation could be improved:\n    - Figure 1: Show the challenges of existing methods, suggest adding visual explanations demonstrating how the proposed RRS technique addresses these challenges\n   - Figure 1(b): Highlight the activation grouping methodology in the diagram\n   - Figure 2(b): Provide a more detailed explanation of the probability metrics\n   - Figure 6: Add clear y-axis labels and justify the unusual batch size choices (>1000).\n   - Figure 9: Verify if the orange label represents `u > 8` instead of `u < 8`", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)