Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference

## Abstract

Abstract Large language models have demonstrated promising capabilities upon scaling up parameters. However, serving large language models incurs substantial computation and memory movement costs due to their large scale. Quantization methods have been employed to reduce service costs and latency. Nevertheless, outliers in activations hinder the development of INT4 weight-activation quantization. Existing approaches separate outliers and normal values into two matrices or migrate outliers from activations to weights, suffering from high latency or accuracy degradation. Based on observing activations from large language models, outliers can be classified into channel-wise and spike outliers.
In this work, we propose Rotated Runtime Smooth ( RRS ), a plug-and-play activation smoother for quantization, consisting of Runtime Smooth and the Rotation operation. Runtime Smooth ( RS ) is introduced to eliminate channel-wise outliers by smoothing activations with channel-wise maximums during runtime. The Rotation operation can narrow the gap between spike outliers and normal values, alleviating the effect of victims caused by channel-wise smoothing.
The proposed method outperforms the state-of-the-art method in the LLaMA and Qwen families and improves WikiText-2 perplexity from 57.33 to 6.66 for INT4 inference.

## 1 Introduction

Large language models have demonstrated promising capabilities as parameters are scaled up. However, serving large language models is plagued by the high cost of computation and memory movement due to their scale. Consequently, many quantization methods are applied to reduce size and gain throughput improvement. From the service perspective, quantization can be categorized as weight-only quantization and weight-activation quantization. The former can focus on compressing the model’s weights and saving costs related to memory movement, which is suitable for the memory-bound decoding stage. The latter quantizes both weight and activation to low bits and utilizes low-bit matrix multiplication kernels to achieve speedup. However, the existence of outliers in activation stretches the quantization range, compressing the effective bits for normal values and thus hindering the development of low-bit weight-activation quantization.

To address outliers, previous works such as (Kim et al.,, ; Dettmers et al.,, ) separate outlier and normal values into two matrices. However, the implementation is not hardware-compatible and fails to expedite inference. To achieve acceleration and maintain accuracy under A8W8 quantization, SmoothQuant (Xiao et al.,, ) transfers appropriate outliers from activation to weight offline through channel-wise smoothing scales. Nevertheless, the offline smoothing scales would be ineffective when facing unmatched input, and the outlier sharing scheme makes weight difficult to quantify. The aforementioned reason impedes the implementation of SmoothQuant for A4W4 quantization. QuaRot (Ashkboos et al.,, ) utilizes the property that rotation can suppress outliers; hence, pairwise rotate the activation and weight with equivalent output. The rotated activation and weight tend to spread outliers internally, leading to effectiveness smoothness for A4W4 quantization. However, rotation cannot guarantee a smoother matrix, and a rotated matrix may still exhibit the shape of channel-wise outliers, as depicted in Figure 2. Therefore, deriving a more robust, accurate, and training-free scheme for INT4 inference remains an open challenge.

Based on the observation, outliers could be categorized into channel-wise outliers and spike outliers, as shown in Figure 1.
To handle channel-wise outliers, we propose Runtime Smooth. Firstly, weights and activations are reordered to gather outliers and normal values.
Subsequently, we group up activations and smooth activations by dividing group-wise maximums.
The later quantized smoothed activation, weight, and group-wise maximums are inputs for the fused GEMM kernel.
The entire process incurs minimal overhead compared to the original A4W4 pipeline.
However, the existence of spike outliers causes the effect of victims after channel-wise smoothing, as shown in Figure 1.
To address both channel-wise outliers and spike outliers, we propose Rotated Runtime Smooth, where we rotate weights and activations following (Ashkboos et al.,, ) and apply Runtime Smooth on rotated activations.
The spike outlier is spread along with its token, leading to a smoother token with consistent values.
The consistent values are comparatively larger than the normal values, thereby serving as smoothing scales for Runtime Smooth.
The genesis of victims is abnormal smoothing scales, and the consistent smoothing scales across channels prevent the existence of victims.

Figure: Figure 1: Challenges of SmoothQuant faced with outliers. (a) The unmatched $s$ is useless for smoothing. (b) Based on the migration scheme, smoothed activation/weight is still hard to quantize to 4-bit. (c) Normal values are pruned as victims after smoothing due to the spike outlier.
Refer to caption: x1.png

To evaluate the proposed method, we conducted experiments on LLaMA families, Qwen families, Mistral, etc. We validate the performance on the WikiText-2 perplexity and Common Sense QA benchmarks. On LLaMA3-70B, Rotated Runtime Smooth can gain perplexity improvement from 57.33 to 6.66 under A4W4 quantization compared with the state-of-the-art. We summarize our contributions as follows:

- •
We comprehensively revisited the activation smoothing method for LLM quantization, concluding the reasons for success or failure under A4W4 quantization.
- •
We propose Runtime Smooth, a plug-and-play component that eliminates channel-wise outliers of activation in runtime without migrating outliers to weights, bringing negligible overhead for INT4 matrix multiplication.
- •
We propose Rotated Runtime Smooth to overcome the spike outliers and enhance the robustness for channel-wise outliers. A comprehensive evaluation validates the effectiveness of the proposed methods, which gain thorough improvement on various models for INT4 inference.

## 2 Preliminaries

### 2.1 Quantization

Quantization converts high-precision matrices into discrete elements with scaling factors, achieving a lower bit per element. The process of quantization can be expressed as $\mathbf{X_{INT}}=\lfloor\frac{\mathbf{X}}{\alpha}\rceil,\alpha=\frac{\max(|
\mathbf{X}|)}{2^{N-1}-1}$, where $\mathbf{X}$ represents the floating-point tensor and $\alpha$ is the scaling factor. The existence of outliers stretches the scaling factor and leaves few effective bits for normal values. Dividing the matrix into groups for quantization can mitigate the effect of outliers. In previous literature, the per-tensor quantization considers the entire matrix as a group; the per-channel quantization assigns different scaling factors to each row, and the sub-channel quantization divides rows into fine-grained groups. Although fine-grained grouping can alleviate accuracy degradation, more scaling factors entail additional computation and storage costs. In this work, we adopt the per-channel scheme following (Xiao et al.,, ; Ashkboos et al.,, ; Liu et al.,, ) for INT4 quantization.

Figure: Figure 2: Review of the rotation-based method. (a) illustrates a simple implementation of the rotation-based method. The output from projector is not changed since $\mathbf{Y}=(\mathbf{X}\mathbf{R})(\mathbf{R}^{-1}\mathbf{W}^{T})=\mathbf{X} \mathbf{W}^{T}$. (b) explains the success of the rotation-based method on LLMs, where activations have high confidentiality to be smoothed after rotation compared with a random matrix. (c) illustrates that activation with channel-wise outliers still maintains channel-wise consistency after rotation, leaving space for further smoothing
Refer to caption: x2.png

### 2.2 Channel-wise Smoothing Method

Under the assumption that outliers persist in fixed channels of activations, SmoothQuant migrates the outliers by dividing the smoothing scale $\mathbf{s}\in\mathbb{R}^{K}$, where k denotes channel dimension. For ensuring equivalence of output, $\mathbf{s}$ would be multiplied to weight; the process can be described as $\mathbf{Y}=(\mathbf{X}\text{diag}(\mathbf{s})^{-1})(\text{diag}(\mathbf{s})
\mathbf{W}^{T})=\hat{\mathbf{X}}\hat{\mathbf{W}^{T}}$. The smoothing scales are computed as $\mathbf{s}_{j}=\max(|\mathbf{X}_{j}|)^{\alpha}/\max(|\mathbf{W}_{j}|)^{1-
\alpha},j=1,2,\ldots,K,$ to fairly share the outliers between weights and activations. Since weights are mostly quantized offline (Frantar et al.,, ), directly multiplying $s$ during runtime would undermine the quantization property. Therefore, $s$ is pre-computed using a calibration set and merged into weights before quantization.

Although SmoothQuant is effective under the A8W8 scheme, it fails for INT4 inference in three respects, as shown in Figure 1. Firstly, the smoothing scales depending on the calibration set are prone to being unmatched with online activations; hence, they cannot smooth outliers. Secondly, the outlier channels of activation are not eliminated but partially migrated to weights, thus leading to failure in low-bit quantization. Thirdly, outliers are not always channel-wise consistent, where spike outliers exist, and normal values are pruned as ’victim’ after smoothing.

Spike Outliers and Effect of Victim: Spike outliers stretch the smoothing scale. The normal values divided by abnormal smoothing scales are minimal compared to other elements in the quantization group; hence, they become ’victims.’ The existence of the victims leads to quantization error.

### 2.3 Rotation-based Method

A rotation matrix is an orthogonal matrix R satisfied $\mathbf{R}\mathbf{R}^{T}=1$ and $|\mathbf{R}|=1$. Quip (Tseng et al.,, ) showed that multiplying a weight matrix on the left and right by an orthogonal matrix can theoretically alleviate outliers, making matrices easier to quantize. QuaRot (Ashkboos et al.,, ), employs a similar technique by multiplying weight or activations by only one rotation matrix, maintaining an equivalent output as depicted in Figure 2 (a). However, multiplying one rotation matrix could not theoretically guarantee a smoother weight or activation. To explain the success of the rotation-based method, we computed the probability that the activation of different models becomes less smooth after rotation, as illustrated in Figure 2 (b). Following previous works, we measure the smoothness as $\mu=\textrm{absmax}\big{(}\mathbf{t}\big{)}/\textup{RMS}(\mathbf{t})$, where $\mathbf{t}$ denotes one token in activation, and RMS denotes root mean square. Rotating activation from LLMs consistently exhibits a low probability of being less smooth compared with rotating a random matrix. However, having a chance to be less smooth is a potential trouble.
On the other hand, activations with channel-wise outliers can be viewed as a collection of vectors with the same direction. From the rotation property, the rotated activation maintains the channel-wise consistency, as shown in Figure 2 (c), leaving space for better smooth.

## 3 Methodology

In this section, we introduce Runtime Smooth to eliminate channel-wise outliers (3.1) and how to implement it efficiently with kernel fusion (3.2). To comprehensively eliminate outliers, we propose Rotated Runtime Smooth, which addresses the effect of the victim and sub-smoothness of rotated activation 3.3.

### 3.1 Runtime Smooth

Figure: Figure 3: Preliminary ablation study
Refer to caption: x3.png

Challenges for the smoothing-based method under INT4 inference are discussed in Section 2.2. One intuitive way to mitigate this challenge is to obtain smoothing scale $s$ in runtime and not merge $s$ into weights. The process can be formulated as:

$$ $\displaystyle\mathbf{s}_{j}=\max(|\mathbf{X}_{j}|),j=1,2,...,K$ (1) $\displaystyle\hat{\mathbf{X}}=\textup{Quantize}(\mathbf{X}/\mathbf{s}),\hat{ \mathbf{W}}=\textup{Quantize}(\mathbf{W}),$ (2) $\displaystyle\mathbf{Y}=\sum_{j=1}^{K}\hat{\mathbf{X_{j}}}\cdot\hat{\mathbf{W_ {j}}}^{T}\cdot\mathbf{s}_{j},$ (3) $$

where $\mathbf{s}\in\mathbf{R}^{1\times K}$ denotes the runtime smoothing scale, $\mathbf{X}\in\mathbf{R}^{N\times K}$ denotes activations, $\mathbf{W}\in\mathbf{R}^{M\times K}$ denotes weights.
We conducted an ablation study with LLaMA3-8B and the WikiText-2 dataset to understand better the effect of unmatched scale and outlier shared scheme. As shown in Figure 3, merely applying the runtime smoothing scale could not make A4W4 feasible, whereas Runtime Smooth does, echoing the importance of not migrating outliers to weights. To avoid the effect of quantization error from weight, we further apply the A4W16 setting. The perplexity improvement, from 4e2 to 10.9, validates the effectiveness of adopting the runtime scale.

### 3.2 Runtime Smooth with Kernel Fusion

However, the naive implementation cannot integrated into the GEMM pipeline. A GEMM kernel splits the input matrix into blocks by columns and conducts parallel block computation, where the inconsistent smoothing scale would make the block-wise computation complex. Intuitively, if the smoothing scale is the same within a block, equation 1 can be deduced to $\mathbf{Y}=\mathbf{s}_{j}\cdot\sum_{j=1}^{K}\hat{\mathbf{X_{j}}}\cdot\hat{
\mathbf{W_{j}}}^{T}$. The overhead would be minimized due to fewer multiplication operations.

Figure: Figure 4: Pipeline of Runtime Smooth. (1) Reorder activations and weight according to channel-wise maximums of activation. (2) Group up activations according to block size of matrix multiplication computation. The maximums of the group are set to the runtime smoothing scale of the group. (3) In the matrix multiplication pipeline, quantized smoothed activations and weights are segmented into blocks. The block size is equivalent to the previous group size. Within a block, tiled smoothed activations are multiplied by tiled quantized weights. The runtime smoothing scales are applied to the dequantized interim result.
Refer to caption: x4.png

Roughly altering the smoothing scales to be consistent would degrade the level of smoothness. Hence, we reorder the activations and weights according to the magnitude of smoothing scales, validated as hardware-friendly with negligible cost (Lin et al.,, ). The reordered activations group up the outliers and normal values. The group size is configured to be identical to the block size. The maximum value of an activation group is set as the smoothing scale for the corresponding computation block.

As shown in Figure 4, the pipeline of GEMM fused with Runtime Smooth can be described as 1. Reorder the activation and weight according to channel-wise maximums of activation; 2. Group up the activation and set the group-wise maximum as smoothing scales; 3. Calculate the matrix multiplication of the tiled block and multiply the runtime scale on the dequantized result, followed by a reduction operation. It is worth noting that, compared with the A4W4 baseline, the extra runtime scale multiplication only brings negligible overhead, which will be discussed in Section 4.5.

### 3.3 Rotated Runtime Smooth

Figure: Figure 5: Analysis of rotated activations with different outliers. (a) Activation with channel-wise outliers maintains channel-wise consistency after rotation, hence being sub-smooth for quantization. (b) One spike outlier is spread on its token internal, where the smoothing scale is consistent without abnormal value, further preventing ’victim’.
Refer to caption: x5.png

In this section, we propose Rotated Runtime Smooth (RRS) to comprehensively eliminate outliers, including both channel-wise and spike outliers. Here, the activations are rotated and subsequently applied Runtime Smooth.

For activations with channel-wise outliers, it maintains the channel-wise consistency after rotation due to the rotation property, as shown in Figure 5. Moreover, the rotated activations have a chance to raise the level of outliers with a small probability. Compared with the pure rotation method, RRS comprehensively smooths channel-wise, leading to low quantization error. Compared with Runtime Smooth, RRS is more robust to activations with tiny spikes in the practical scenario.

As discussed in Section 2.2, the existence of spike outlier is the bottleneck of Runtime Smooth. Figure 5 shows that the spike outliers are spread within tokens after rotation. Hence, the smoothing scales are more consistent across channels, leaving fewer victims since all elements are divided by such a consistent scale. The process can be described as:

$$ $\begin{gathered}\mathbf{R}=\frac{1}{\sqrt{K}}{\left[c_{i,j}\right]}_{K\times K },\mathbf{t}=[\varepsilon,\cdots,\varepsilon,O_{i},\varepsilon,\cdots, \varepsilon],\\ \mathbf{t}_{rotation}=\mathbf{t}\cdot\mathbf{R}=\frac{1}{\sqrt{K}}[c_{i,1}O_{i }+\varepsilon,c_{i,2}O_{i}+\varepsilon,\cdots,c_{i,K}O_{i}+\varepsilon],\\ \textup{smooth_scale}=\max(\left|\mathbf{t}_{rotation}\right|)\approx\frac{1} {\sqrt{K}}[\left|O_{i}\right|,\left|O_{i}\right|,\cdots,\left|O_{i}\right|], \end{gathered}$ (4) $$

where the $O$ denotes spike outliers and $\varepsilon$ denotes normal values. $s_{i,j}\in\{-1,+1\}$ denotes the elements in Hadamard rotation matrix. In a practical scenario, the normal values $\varepsilon$ are relatively small compared with spike outliers $O$; hence, they are omitted in the smoothing scale as shown in Equation 4. The genesis of victims is the abnormal smoothing scale; hence a more consistent smoothing scale brought by RRS frees ’victims’. Here, we only discuss the scenario where there is only one spike in the token, and we conduct a comprehensive analysis of multiple spikes for real scenarios in Section A.1.

We apply RRS to Linear Layers in Transformer blocks. We first offline rotate the weight matrix and insert online rotation operation before output and down projectors following previous works (Ashkboos et al.,, ). The rotated weights are quantized offline with GPTQ (Frantar et al.,, ). During inference, we perform Runtime Smooth on the rotated activations of linear layers and apply activation quantization subsequently.

## 4 Experiments

### 4.1 Settings

We conduct experiments on mainstream LLMs, including LLaMA families () (LLaMA2-13B, LLaMA2-70B, LLaMA3-8B, LLaMA3-70B, LLaMA3.1-8B, LLaMA3.1-70B), Qwen families (Yang et al.,, ) (Qwen1.5-7B, Qwen1.5-14B), Mistral (Jiang et al.,, ) and Mixtral (Jiang et al.,, ). Activation quantization employs per-channel symmetric scheme with round-to-nearest (RTN) strategy. KV cache quantization employs sub-channel symmetric scheme with group size 128 and round-to-nearest (RTN) strategy. In most cases, weight quantization employ per-channel symmetric scheme with GPTQ (Frantar et al.,, ) strategy, except for baseline ’RTN’. We apply standard GPTQ settings by using 128 samples from WikiText-2 with a sequence length of 2048 as the calibration set. We evaluate the performance of the models on WikiText-2 perplexity and zero-shot Common Sense QA benchmarks. The Common Sense QA benchmarks include ARC-e, ARC-c (Clark et al.,, ), BoolQ (Clark et al.,, ), and OBQA (Mihaylov et al.,, ).

### 4.2 Main Result

Runtime Smooth emphasizes activation smoothing for INT4 inference. The plug-and-play Runtime Smooth operators are employed before activation quantization. We conduct a comparison between Runtime Smooth and SmoothQuant (Xiao et al.,, ), demonstrating promising improvement. For instance LLaMA2-70B: 1e2 -> 6.95, LLaMA3-8B: 8e2 -> 10.47, Qwen1.5-7B: 3e2 ->13.32 under the A4W4KV16 scheme, as shown in Table 1. For certain models like LLaMA3-70B and LLaMA3.1-70B, both Runtime Smooth and SmoothQuant fail for the difficulty of weight quantization. To eliminate the influence of quantization error from weights, we carry out experiments under A4W16KV16 settings. Under the activation-only quantization setting, Runtime Smooth consistently outperforms SmoothQuant and achieves 40x improvement on LLaMA3-8B, validating the effectiveness of Runtime Smooth. Here the group size of the smoothing scale is 1 to observe the upper bound performance.

**Table 1: Comparison on WikiText-2 perplexity. We evaluate models and methods on three quantization schemes: A4W4KV4, A4W4KV16, and A4W16KV16. Results for SmoothQuant, GPTQ, and QuaRot were obtained by re-implementation based on their publicly released codebase.**
| #Bits | Method | LLaMA | LLaMA | LLaMA | LLaMA | LLaMA | LLaMA | Mixtral | Mistral | Qwen | Qwen |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| W-A-KV | 2-13B | 2-70B | 3-8B | 3.1-8B | 3-70B | 3.1-70B | 1.5-7B | 1.5-14B |  |  |  |
| 16-16-16 | FP16 | 5.00 | 3.30 | 6.13 | 6.24 | 2.80 | 2.81 | 3.84 | 5.94 | 7.95 | 7.44 |
| \hdashline16-4-16 | RTN | 5e3 | Nan | 4e2 | 2e2 | 3e4 | 1e4 | 8e2 | 7e2 | 6e3 | 2e4 |
| SmoothQuant | 97.04 | Nan | 5e2 | 2e2 | 74.23 | 48.59 | 88.68 | 47.92 | 2e2 | 1e2 |  |
| RS | 6.40 | 4.52 | 11.44 | 9.71 | 11.31 | 7.27 | 5.72 | 7.39 | 10.95 | 9.74 |  |
| QuaRot | 5.24 | 3.72 | 7.77 | 7.81 | 1e2 | 5.67 | 4.68 | 6.29 | 9.07 | 8.18 |  |
| RRS | 5.22 | 3.74 | 7.55 | 7.60 | 6.32 | 5.03 | 4.50 | 6.21 | 8.91 | 8.16 |  |
| \hdashline4-4-16 | RTN | 7e3 | 2e5 | 9e2 | 5e2 | 1e5 | 2e4 | 8e2 | 7e2 | 9e3 | 3e4 |
| SmoothQuant | 34.50 | 1e2 | 8e2 | 4e2 | 5e2 | 2e2 | 3e2 | 76.25 | 3e2 | 1e2 |  |
| GPTQ | 5e3 | 2e6 | 9e2 | 4e2 | 1e5 | 2e4 | 1e3 | 6e2 | 1e4 | 3e4 |  |
| RS | 8.79 | 6.95 | 10.47 | 10.39 | 7e3 | 1e4 | 7.37 | 8.16 | 13.32 | 10.38 |  |
| QuaRot | 5.39 | 3.85 | 8.38 | 8.38 | 57.33 | 6.26 | 4.80 | 6.38 | 9.34 | 8.32 |  |
| RRS | 5.36 | 3.86 | 8.11 | 8.12 | 6.66 | 5.56 | 4.63 | 6.31 | 9.17 | 8.29 |  |
| \hdashline4-4-4 | RTN | 7e3 | 2e5 | 1e3 | 6e2 | 1e5 | 2e4 | 9e2 | 7e2 | 1e4 | 2e4 |
| SmoothQuant | 56.60 | 1e2 | 1e3 | 6e2 | 5e2 | 3e2 | 2e2 | 78.39 | 3e2 | 2e2 |  |
| GPTQ | 5e3 | 1e6 | 1e3 | 5e2 | 1e5 | 2e4 | 1e3 | 6e2 | 1e4 | 3e4 |  |
| RS | 9.15 | 7.08 | 11.52 | 11.19 | 8e3 | 1e4 | 7.98 | 8.64 | 13.97 | 10.62 |  |
| QuaRot | 5.51 | 3.89 | 8.76 | 8.80 | 49.73 | 6.46 | 4.93 | 6.45 | 9.55 | 8.43 |  |
| RRS | 5.45 | 3.89 | 8.42 | 8.49 | 6.87 | 5.69 | 4.74 | 6.35 | 9.37 | 8.35 |  |

To further narrow the accuracy gap between INT4 inference and full precision inference, we propose Rotated Runtime Smooth. Here, the group size of the smoothing scale is set to 128, which is identical to the GEMM block size, enabling efficient implementation. Compared with the state-of-the-art, QuaRot, Rotated Runtime Smooth consistently outperforms across different model sizes and model families. In most instances, Rotated Runtime Smooth achieves a 0.1 - 0.3 improvement in WikiText-2 perplexity.

It is noteworthy that on LLaMA3-70B, Rotated Runtime Smooth reduces perplexity from 57.33 to 6.66 under the A4W4KV16 setting and from 49.76 to 6.87 under the A4W4KV4 setting. Notably, 57.33 and 49.76 are abnormal results. Hence, we analyze outliers from rotated LLaMA3-70B and LLaMA3.1-70B. The distribution of outliers is similar for the two models. More details are provided in Section A.2. This indicates that models have different sensitivities to outliers. Further suppressing the outliers can enhance robustness.

**Table 2: 0 0-shot accuracy (%) on the Common Sense QA tasks. Each block is based on the same foundation model specified in the row. We organize all results under different quantization schemes.**
|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| #Bits | Model | Method | OBQA | BoolQ | ARC_E | ARC_C | Avg. |
|  |  | GPTQ | 26.6 | 44.0 | 29.8 | 21.0 | 30.4 |
|  |  | SmoothQuant | 25.0 | 43.7 | 30.2 | 22.9 | 30.4 |
|  |  | RS | 42.4 | 65.6 | 72.2 | 46.3 | 56.6 |
|  |  | QuaRot | 39.2 | 70.7 | 68.6 | 41.5 | 55.0 |
|  | LLaMA-3-8B | RRS | 42.4 | 73.6 | 71.1 | 44.8 | 58.0 |
| \cdashline<br>2-8 |  | GPTQ | 26.2 | 44.7 | 30.3 | 23.9 | 31.3 |
|  |  | SmoothQuant | 26.2 | 46.42 | 30.9 | 24.5 | 32.0 |
|  |  | RS | 44.6 | 68.8 | 72.5 | 47.6 | 58.4 |
|  |  | QuaRot | 37.6 | 75.6 | 72.9 | 44.8 | 57.7 |
|  | LLaMA-3.1-8B | RRS | 42.4 | 78.1 | 76.1 | 50.3 | 61.7 |
| \cdashline<br>2-8 |  | GPTQ | 27.0 | 46.1 | 32.0 | 25.0 | 32.5 |
|  |  | SmoothQuant | 28.6 | 57.5 | 42.2 | 31.4 | 39.9 |
|  |  | RS | 35.2 | 80.2 | 72.4 | 51.1 | 59.7 |
|  |  | QuaRot | 44.4 | 83.1 | 71.9 | 54.4 | 63.4 |
|  | Mistral | RRS | 43.4 | 85.1 | 74.4 | 55.4 | 64.6 |
| \cdashline<br>2-8 |  | GPTQ | 28.2 | 42.9 | 25.9 | 26.5 | 30.9 |
|  |  | SmoothQuant | 28.2 | 54.0 | 32.3 | 26.7 | 35.3 |
|  |  | RS | 37.6 | 72.6 | 56.9 | 39.0 | 51.5 |
|  |  | QuaRot | 39.0 | 73.9 | 61.3 | 40.4 | 53.7 |
| 4-4-16 | Qwen1.5-7B | RRS | 43.0 | 77.7 | 61.5 | 42.0 | 56.0 |
|  |  | GPTQ | 28.2 | 42.0 | 28.8 | 24.1 | 30.7 |
|  |  | SmoothQuant | 28.8 | 52.2 | 37.6 | 28.8 | 36.9 |
|  |  | RS | 43.2 | 65.7 | 67.2 | 41.5 | 54.4 |
|  |  | QuaRot | 38.6 | 70.6 | 66.7 | 40.4 | 54.1 |
|  | LLaMA-3-8B | RRS | 42.4 | 73.5 | 68.7 | 44.7 | 57.3 |
| \cdashline<br>2-8 |  | GPTQ | 28.0 | 43.6 | 29.0 | 23.3 | 31.0 |
|  |  | SmoothQuant | 28.0 | 45.1 | 30.0 | 22.7 | 31.4 |
|  |  | RS | 43.4 | 65.3 | 71.1 | 46.1 | 56.5 |
|  |  | QuaRot | 37.2 | 73.5 | 65.7 | 41.7 | 54.5 |
|  | LLaMA-3.1-8B | RRS | 41.8 | 77.8 | 75.3 | 48.1 | 60.8 |
| \cdashline<br>2-8 |  | GPTQ | 24.8 | 46.2 | 31.3 | 25.8 | 32.0 |
|  |  | SmoothQuant | 27.0 | 56.6 | 40.6 | 29.5 | 38.4 |
|  |  | RS | 36.6 | 81.0 | 72.4 | 50.3 | 60.1 |
|  |  | QuaRot | 43.2 | 83.0 | 72.8 | 52.5 | 62.9 |
|  | Mistral | RRS | 45.6 | 84.3 | 73.2 | 54.6 | 64.4 |
| \cdashline<br>2-8 |  | GPTQ | 30.0 | 42.8 | 26.2 | 26.5 | 31.4 |
|  |  | SmoothQuant | 28.0 | 53.1 | 30.2 | 25.1 | 34.1 |
|  |  | RS | 36.8 | 70.2 | 56.2 | 40.2 | 50.9 |
|  |  | QuaRot | 39.6 | 74.3 | 62.1 | 42.0 | 54.5 |
| 4-4-4 | Qwen1.5-7B | RRS | 40.4 | 76.6 | 61.8 | 42.5 | 55.3 |

### 4.3 Comparison with training-based Method

SpinQuant (Liu et al.,, ) suggests that diverse Rotation matrices exhibit variations in their impact of smoothing. Consequently, it substitutes the origin fix Rotation matrix with a trainable Rotation matrix and trains the rotated network. The training process is time-consuming, taking 1.5 hours for a 7B model on one A100 GPU and 12 hours for 70B models on eight A100 GPUs. We re-implement SpinQuant and compare it with our method as shown in Table 3. It is noteworthy that SpinQuant applies asymmetric quantization to activation and KV cache. We experiment with our method under the same settings. The result reveals that the training-based method degrades WikiText-2 perplexity compared to the training-free method. This suggests that the current technique for optimizing the rotation matrix still has room for improvement.

**Table 4: Ablation study of Group size of runtime smooth scale**
| Method | Model | 1 | 32 | 64 | 128 | 256 | 512 |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Mistral | 6.29 | 6.28 | 6.30 | 6.31 | 6.31 | 6.33 |
|  | Qwen1.5-7B | 9.16 | 9.14 | 9.16 | 9.17 | 9.18 | - |
| RRS | LLaMA3.1-8B | 8.09 | 8.09 | 8.10 | 8.12 | 8.18 | 8.29 |
|  | Mistral | 8.16 | 11.17 | 11.42 | 18.48 | 36.46 | 154.19 |
|  | Qwen1.5-7B | 13.32 | 15.67 | 18.08 | 22.83 | 41.55 | - |
| RS | LLaMA3.1-8B | 10.39 | 13.51 | 20.74 | 51.63 | 214.88 | 2706.82 |

### 4.4 Ablation Study

We conduct an ablation study on the group size of runtime smoothing scale for Mistral, Qwen1.5-7B, and LLaMA3.1-8B, as presented in 4. Runtime Smooth can effectively minimize the gap between A4W4 and full precision through a subtle group strategy. However, the accuracy deteriorates as the group size increases. Rotated Runtime Smooth employs the rotation technique to minimize the gap between outliers and normal values, thereby being robust to the coarse group scheme and enabling the implementation of the fused kernel. It should be noted that in Qwen1.5-7B, the size of the input activation for Down_projector is 11008, which does not support a group size of 512.

### 4.5 Efficiency evaluation

We evaluate the GEMM kernel fused with Runtime Smooth on NVBench (NVIDIA,, ) with RTX 4070 Ti, as shown in Figure 6. The group size of the smoothing scale is 128, the same as the block size of the GEMM kernel. We implement Per-Channel A4W4 and Sub-Channel A4W4 as baselines. Compared with Per-Channel A4W4, Runtime Smooth fused Kernel brings limited overhead, including the movement of the smoothing scale (shape of [1, K]) and a multiplication between matrix and scalar. Sub-Channel A4W4 brings noticeable overhead, including the movement of group-wise quantization scale (shape of [N, L] and [M, L]) and multiplication between matrices. Hence, across different batch sizes and hidden dimensions, Runtime Smooth fused Kernel brings negligible overhead compared with A4W4 Per-Channel quantization, which is also the setting of QuaRot and SpinQuant.

Figure: Figure 6: Performance evaluation of different quantization approaches. We set up the evaluation configuration aligned with the LLaMA-7b configuration and 1024 sequence length. Kernels are evaluated by NVBench
Refer to caption: x6.png

## 5 Related Work

### 5.1 Large language models.

Pre-trained language models have achieved remarkable progress through scaling. Open-source large language models (LLMs) can reach up to 405 billion parameters and offer promising few-shot/zero-shot results. The mainstream LLMs (Yang et al.,, ; ; DeepSeek-AI et al.,, ; Jiang et al.,, ) continuously provide models with large scales and enhanced capabilities. However, serving large language models for inference becomes costly and challenging as the models expand.

### 5.2 Model quantization.

Quantization represents an effective approach for reducing model size and expediting inference. From a serving perspective, quantization can be classified into weight-only quantization and weight-activation quantization. Weight-only quantization compression employs low-bit representations for weight matrices, thereby saving memory movement in memory-bound scenarios, specifically the decoding stage. GPTQ (Frantar et al.,, ) used 4-bit to quantize the weight based on the approximate second-order information. SqueezeLLM (Kim et al.,, ) handled outliers through non-uniform quantization and used a sparse format to keep outliers and sensitive weights at high precision. AWQ (Lin et al.,, ) further advanced accuracy by preserving salient weights. QFA (Yi et al.,, ) fine-tune a supernetwork encompassing multiple mixed precision configurations and efficiently offer high-performance sub-networks for diverse scenarios. QuiP (Chee et al.,, ; Tseng et al.,, ) successfully represents weights using 2 bits via an adaptive rounding method. Weight-activation quantization can further accelerate computation by leveraging a low-bit GEMM kernel suitable for compute-bound scenarios, namely the pre-filling stage. SmoothQuant (Xiao et al.,, ) pushes the quantization scheme to A8W8, achieving speed up and maintaining accuracy. Qserve (Lin et al.,, ) further implement the A8W4KV4 and better accelerate. Among weight-activation quantization methods, the existence of outliers presents the most formidable problem, as it can result in substantial drops in accuracy.

### 5.3 Outliers Challenge.

Outliers can expand the quantization range and compress the information intensity for normal values. LLM.int8() (Dettmers et al.,, ) employs mixed INT8/FP16 decomposition to handle activation outliers. Nevertheless, such an implementation results in significant latency overhead and can even be slower than FP16 inference. Subsequent work (Yuan et al.,, ) rearranges the channels to reduce the variance within one quantization group, further enhancing accuracy. Atom (Zhao et al.,, ) integrates the reorder technique and mixed INT4/INT8 precision to maintain accuracy and accelerate compared to the FP16 baseline. In addition to the reorder and mixed-precision techniques, SmoothQuant (Xiao et al.,, ) exchanges outliers between weights and activations to find an optimal point that shares appropriate outliers in weights and activations, achieving A8W8 inference with minimal accuracy degradation. However, the outlier sharing scheme may not be suitable for the extreme A4W4 setting. To further smooth outliers, QuaRot (Ashkboos et al.,, ) pairwise rotates the activation and weight to suppress outliers and maintain output equalization, enabling INT4 inference with well-smoothed activations.

## 6 Conclusion

This work presents Rotated Runtime Smooth, a plug-and-play runtime activation smoother that facilitates INT4 inference. In Rotated Runtime Smooth, Runtime Smooth effectively eliminates channel-wise outliers. Additionally, rotation operations migrate the negative impact from spike outliers. Rotated Runtime Smooth can be easily implemented with negligible overhead and is generalized across various large language models (LLMs). Through comprehensive elimination of channel-wise and spike outliers, Rotated Runtime Smooth achieves significant enhancement compared to the prior channel-wise smoothing approach and outperforms state-of-the-art methods on diverse models for INT4 inference.

## Appendix A Outlier

### A.1 Activation with multiple Spike outliers

This section further analyzes how tokens with multiple outliers behave after rotation. The token with multiple outliers is defined as:

$$ $\mathbf{t}=[\cdots,O_{i_{1}},\cdots,O_{i_{2}},\cdots,O_{i_{l}},\cdots],$ (5) $$

where l denotes the number of outliers, $\{i_{1},i_{2},...i_{l}\}$ denotes the index of outliers.
In this work, we set the Hadamard matrix as the rotation matrix $\mathbf{R}=\frac{1}{\sqrt{K}}{\left[s_{i,j}\right]}_{K\times K},$ where $s_{i,j}\in\{-1,+1\}$. The rotated token can be described as:

$$ $\displaystyle\mathbf{t}^{rot}$ $\displaystyle=\mathbf{t}\cdot\mathbf{R}$ (6) $\displaystyle\approx\frac{1}{\sqrt{K}}[\sum_{d=1}^{l}s_{i_{d},1}O_{i_{d}},\sum _{d=1}^{l}s_{i_{d},2}O_{i_{d}},\cdots,\sum_{d=1}^{l}s_{i_{d},K}O_{i_{d}}].$ (7) $$

The construction of $t^{rot}$ can be viewed as contributions of outliers, where outliers are canceled out by each other or stacked to enlarge. The effect of victims refers to the smoothness of normal tokens after smoothing. The process is defined as:

$$ $\displaystyle\mathbf{x}=[1,1,1,...,1],$ (8) $\displaystyle\textup{scale}=[\mathrm{absmax}\left(1,\frac{1}{\sqrt{K}}\sum_{d= 1}^{l}s_{i_{d},1}O_{i_{d}}\right),\cdots,\mathrm{absmax}\left(1,\frac{1}{\sqrt {K}}\sum_{d=1}^{l}s_{i_{d},K}O_{i_{d}}\right)],$ (9) $$

where we assume normal tokens are filled up with 1. The effect of victims can be qualified with the equation:

$$ $\textup{x}_{smooth}=1/\textup{scale},\mathbf{u}=\max(|\textup{x}_{smooth}|)/ \textup{RMS}(|\textup{x}_{smooth}|)$ (10) $$

To measure the effect of victims in the actual scenario, we first collect the activations from LLaMA3-8B.
For activations from the Down Projector, spike outliers are 1000x larger than the medium value, as shown in Figure 7, where outliers in a channel-wise manner are not overly large.

To analyze the effect of smoothing rotated spike outliers, we apply the Monte Carlo approach by generating a token from the Gaussian distribution and inserting spike outliers according to statistics of spike outliers, then rotating, smoothing, and calculating $u$ as shown in Figure 8. Rotated tokens with multiple outliers are up and down across channels due to the effect of offset and stack. On the other hand, we can stack rotated tokens to obtain a consistent large scale across channels.
As shown in Figure 8, normal tokens after smoothing are mostly easy to quantify but contrary when two outlier tokens are exhibited in one activation. The reason is that two tokens cannot cover the whole channel, where more stacked tokens can lead to lower $u$. Notably, the case with only two outlier tokens is rare but could potentially trouble Rotated Runtime Smooth.

### A.2 Analysis extent of outlier removal for different method

We conduct experiments on mainstream models with different outlier smoothing approaches to analyze smoothness integrally rather than simulating with manual spike outliers. Specifically, we collect activations with models evaluated by WikiText-2 and apply different smoothing approaches. To measure the level of outliers; we set $\mu=\textrm{absmax}\big{(}\mathbf{t}\big{)}/\|\mathbf{t}\|_{2}$, where $\mathbf{t}$ denotes one token.
Figure 9 shows the specific impact of approaches on outliers on different LLM’s components. For QKV_Projector, UP_Projector, and Gate_project, the activations are channel-wise consistent; hence, Runtime Smooth outperforms rotation, where pure rotated activations are sub-smooth. For Down_Projector, the intermediate activations contain spike outliers due to SwiGLU (Shazeer,, ) functions; hence, Runtime Smooth suffers from the effect of the victim and fails to smooth. Rotated Runtime Smooth solves two kinds of outliers and consistently outperforms other approaches.

Figure: Figure 7: Collecting the activations as input of Down Projector with the full precision LLaMA3-8B model evaluating on WikiText-2 and counting the magnitude and number of spike outliers. The magnitude is calculated by $x/medium(t)$, where t is a token, and x is the element of the token. We separately count the spike outliers with different magnitude intervals.
Refer to caption: x7.png

Figure: Figure 8: Simulation for the effect of victims after smoothing with rotated spike outliers. The magnitude and number of spike outliers are configured according to Figure 7. The X-axis denotes the number of spike tokens in an activation. The Y-axis denotes $u$ of the normal tokens divided by smooth scales.
Refer to caption: x8.png

Figure: Figure 9: Statistic analysis of outlier removal with different smooth approaches. We collect the activations with full precision models evaluating on WikiText-2. ’X’ denotes origin activations, ’R’ denotes rotated activations, ’RS’ denotes activations after Runtime Smooth, and ’RRS’ denotes activations after Rotated Runtime Smooth.
Refer to caption: x9.png

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see my comments above.
soundness: 3
strengths: - Combining two exiting ideas in the Quantization literature to overcome both channel-wise and spikes outliers
- Comprehensive experiments with 3 different LLMs, including LLaMA, Qwen, Mixtral and Mistral models on WikiText-2 perplexity
summary: A plug-and-play quantization method based on runtime smoothing and the rotation operation of activations has been proposed. Runtime smoothing is responsible for eliminating channel-wise outliers, while the rotation operation mitigate the gap between spike outliers and normal values, resulted by channel-wise smoothing. Experimenting with INT4 inference on different LLMs shows that the proposed method can reduce the perplexity compared to other approaches.
weaknesses: Some notations need to be corrected. For example,
- Line 168: identity matrix and number 1 should be distinguished. $|.|$ the norm needs to be determined. 
- Line 175: $absmax$ ---- > $abs(\max)$
- Equation (1): $\mathrm{X_j}$ needs to be defined as the columns of matrix $\mathrm{X}$.
- Equation(2): $/$ is not a valid operation for matrices. You need to write the formula as a multiplication with a diagonal matrix.
- line 214: The condition needs to be re-written: $s_j$ cannot be taken out of the sum.

Another issue is that the proposed method has been only tested for the  WikiText-2 perplexity. Can the approach is applicable for other modalities and tasks. For instance, ASR, speech translation, Image understanding, etc.

There is only a small improvement over QuaRot approach. I am wondering if your results are statistical significant?

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: See Weaknesses.
soundness: 2
strengths: 1. The paper is clearly structured, making it easy to follow.
2. The method’s motivation is substantiated by experimental results, lending it credibility. In particular, the experiments in Section 4.5 demonstrate the minimal overhead introduced by the method, highlighting its practicality.
summary: The paper proposes a quantization method for LLMs, utilizing a smoothing operation during runtime to mitigate the issue of spike outliers and normalize values. Experiments conducted under INT4 settings validate the effectiveness of this method.
weaknesses: 1. **Figure 1 is confusing and hard to interpret.** The differences between panels (a) and (b) are unclear, as they are on completely different scales despite both sharing the same $ W $ and $ X $. Additionally, have you quantized $ \hat{X} = X \text{diag}(s)^{-1} $? The elements in $ \hat{x} $ appear to be integers, though they should be floats.

2. **Some terms and phrases are difficult to understand.** For instance, the caption for Figure 2(c) uses the phrase "channel-wise consistency" without providing any explanation, making it hard to grasp. Similarly, in point (1), what exactly is meant by "unmatched scale" in Figure 1?

3. **The motivation closely resembles prior work.** For example, DuQuant [1] also begins by addressing layers with massive outliers (in this paper, it is named as "spike") and tackles both massive and normal layers. Similarly, the method in [2] addresses large activation outliers and provides more analysis on this topic than the present paper.

4. **Some claims lack theoretical or experimental support.** For instance, the claim that "smoothing scales depending on the calibration set are prone to being unmatched with online activations" would benefit from experimental results demonstrating the likelihood of such a mismatch.

5. **Some explanations are too brief.** For example, the term "reorder" is mentioned without much elaboration, described only as "reordering the activations and weights according to the magnitude of smoothing scales."

6. **The method lacks clarity in its description.** In Equation 3, $ \hat{X} $ and $ \hat{W} $ appear to be in INT4 format, while $ s_i $ is a float (assuming this is correct, as the authors do not specify). How are these terms compatible for multiplication? A DeQuantize operation may need to be included somewhere in Equation 4.

7. **The ablation study feels incomplete.** The method employs several components—reordering, online smoothing, etc.—but the impact of each isn’t fully explored.

8. **The performance is underwhelming.** The proposed method does not appear to outperform SpinQuant [3] or DuQuant [1], even though it uses online quantization, whereas SpinQuant and DuQuant are offline methods.

Minors: 
1. In Equation 4, $ c_{i,j} $ should be $ s_{i,j} $.

[1] Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, L., ... & Wei, Y. (2024). DuQuant: Distributing outliers via dual transformation makes stronger quantized LLMs. *arXiv preprint arXiv:2406.01721*.  

[2] Yang, J., Kim, H., & Kim, Y. (2024). Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs. *arXiv preprint arXiv:2405.14428*.  

[3] Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., ... & Blankevoort, T. (2024). SpinQuant—LLM quantization with learned rotations. *arXiv preprint arXiv:2405.16406*.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: The paper is generally well-written, with a comprehensive ablation study (RRS vs RS and RRS vs SmoothQuant) demonstrating the effectiveness, which is well motivated by the preliminary section.
summary: This paper extends SmoothQuant [Xiao 2023] by:
 - Performing Quip-like [Tseng 2024] rotation before activation smoothing;
 - Obtaining smoothing scale in runtime and not merging it into weights;
weaknesses: The paper lacks an evaluation on the clock-time (instead of the operation count) overhead introduced by the method. This is a concern given that the method determines activation smoothing scale during runtime.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to my comment for Weaknesses section.
soundness: 3
strengths: The authors focused on addressing channel-wise and spike outliers for the first time, which highlights a novel problem-solving approach and demonstrates significant performance improvements.

The methodology of Rotated Runtime Smooth (RRS) is simple, intuitive, and provides a clear and straightforward explanation of how it addresses each issue effectively.

By making their code publicly available, the authors have supported further research and made a valuable contribution to the ICLR community.
summary: The authors were the first to highlight that channel-wise and spike outliers cause unnecessary distributional expansion in the quantization of large language models (LLMs).

The authors proposed a concise and effective technique called Rotated Runtime Smooth, which reduces these outliers to ensure that the activation distribution for LLM quantization remains compact.

The authors functionally explained, along with the methodology, how Runtime Smooth addresses channel-wise outliers and Rotated Smooth effectively handles spike outliers.

The authors demonstrated effective performance improvements across various LLM models using this approach.
weaknesses: The explanation of the methodology is insufficient: For example, in Figures 1 and 4, which aim to explain Runtime Smooth, it is difficult to gain a proper understanding. The main text also lacks adequate explanation of these figures.

For instance, in Figure 4, while the activation and weight reorder process for Runtime Smooth is presented, there isn’t enough explanation about why this process does not alter or distort the final result. Similarly, in Figure 1, it is unclear what the value of s represents (in relation to the equations in the text) and how exactly it reduces channel-wise outliers and effectively resolves the victim issue caused by spike outliers.

These examples highlight the need for a more detailed and clear description of the methodology, possibly in the appendix. This would help ensure that readers can thoroughly understand the method. While the high-level intuition is conveyed, the reproducibility of the method should be clearly delivered to readers through the paper itself, without relying on the availability of the code.

It is essential to discuss why this study is limited to INT4. The limitations and advantages should be clearly articulated. For example, there is no comparison with recent trending quantization techniques, such as One-bit LLM. A thorough investigation of related work is necessary, and efforts should be made, at least indirectly, to demonstrate the applicability and effectiveness of the proposed method in relation to these latest quantization techniques. https://arxiv.org/abs/2310.11453

======

Assuming that the above points are adequately addressed, I will start with a borderline or higher score. However, if it is determined that these points are not sufficiently reflected, or based on the overall opinions of other reviewers or AC, I will consider adjusting my final score.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer the last item of Weaknesses.
soundness: 3
strengths: - The authors provide a comprehensive analysis of activation distribution patterns, offering well-reasoned solutions for each identified challenge.
- They proposed a training-free approach that outperforms existing training-based methods on some benchmarks, reducing engineering complexity and showing potential for practical applications in industry scenarios.
summary: This paper proposed a quantization technique designed to address outliers during Large Language Model (LLM) quantization, particularly for aggressive scenarios like W4A4 quantization. The method employs two steps: 1) mitigating channel-wise outliers through rotation, and 2) implementing online smoothing to further handle spike outliers. The authors demonstrate promising accuracy results across several benchmarks.
weaknesses: - Limited novelty: The core methodology is primarily based on existing approaches like [Smooth quantization](https://arxiv.org/abs/2211.10438 ) and [QuaRot](https://arxiv.org/abs/2404.00456 ), while the reordering technique shows overlap with existing research such as [ATOM](https://arxiv.org/abs/2310.19102) and [LLM.int8()](https://arxiv.org/abs/2208.07339).
- The experimental evaluation is limited in scope, focusing on common sense QA tasks. To better validate the method's effectiveness, the paper would benefit from:
    - More comprehensive comparisons with training-based methods like [SpinQuant](https://arxiv.org/abs/2405.16406) on more complex tasks, like [MMLU](https://arxiv.org/abs/2009.03300)
    - Detailed kernel performance comparisons with related research like [Qserve]( https://arxiv.org/abs/2405.04532), and [ATOM](https://arxiv.org/abs/2310.19102)

- Several aspects of the paper's presentation could be improved:
    - Figure 1: Show the challenges of existing methods, suggest adding visual explanations demonstrating how the proposed RRS technique addresses these challenges
   - Figure 1(b): Highlight the activation grouping methodology in the diagram
   - Figure 2(b): Provide a more detailed explanation of the probability metrics
   - Figure 6: Add clear y-axis labels and justify the unusual batch size choices (>1000).
   - Figure 9: Verify if the orange label represents `u > 8` instead of `u < 8`

Meta-review: Large language models achieve impressive performance as their scale increases but incur high computational and memory costs. Quantization mitigates these costs but struggles with activation outliers, hindering INT4 quantization. This work introduces Rotated Runtime Smooth (RRS), a plug-and-play activation smoother combining Runtime Smooth (RS) to handle channel-wise outliers and a Rotation operation to address spike outliers. RRS outperforms state-of-the-art methods, achieving significant improvements, including reducing WikiText-2 perplexity from 57.33 to 6.66 in INT4 inference.


Summary of the strengths and weaknesses of the paper according to the reviewers: 

Strengths:

+The paper introduces a plug-and-play quantization method combining runtime smoothing and rotation to address channel-wise and spike outliers.

+This paper claims negligible overhead compared to existing quantization methods while achieving better perplexity scores.

+It also extends experimental evaluation to multiple downstream tasks and baselines, demonstrating robustness across different domains.

+Finally, the reviewers appreciated the clear structure and logical flow, that they thought was helpful in understanding the methodology.

Weaknesses:

-Limited Novelty: Several reviewers noted overlaps with prior work (e.g., SmoothQuant, QuaRot, ATOM), questioning the methodological originality.

-Incomplete Methodology Explanation: Some figures (e.g., Figures 1 and 4) and terms were inadequately explained, making parts of the paper challenging to follow.

-Lack of Diversity in Evaluation: Initially focused on a narrow set of tasks (e.g., WikiText-2), prompting concerns about the generalizability of the results.

-Minor Performance Gains: Marginal improvements over prior methods led to skepticism about the significance of the results.

In their rebuttal, the authors addressed reviewer concerns by expanding experiments to additional tasks (e.g., MMLU, GSM8k) and demonstrating consistent improvements across diverse domains. They provided detailed runtime efficiency metrics, showing negligible overhead for their method. The authors clarified the novelty of their approach, emphasizing how the integration of Runtime Smooth and rotation resolves issues with prior methods. They also explained unclear figures and terms, committing to improving the final manuscript for better clarity. These efforts successfully convinced some reviewers to raise their scores, though skepticism about novelty and deeper theoretical insights persisted.

Reviewer reactions to the rebuttal were mixed. Some reviewers appreciated the additional experiments and detailed clarifications, leading to improved scores (e.g., Reviewer MyVY). They acknowledged the authors' efforts in addressing concerns about evaluation scope, efficiency, and clarity. However, other reviewers (e.g., Reviewer 2Usn) remained skeptical about the novelty of the approach and the lack of deeper theoretical insights, with concerns about incremental contributions persisting. The paper is a bit borderline with polarizing reviews. But I do think the authors response clarified major issues. While other issues persist I think the ICLR community will benefit from the discussion and I thus recommend acceptance. That said, I recommend the authors address the remaining issues raised by the reviewers including novelty concerns.

Decision: Accept (Poster)