Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference

Authors: Ke Yi, Zengke Liu, jianwei zhang, Chengyuan Li, Tong Zhang, Junyang Lin, Jingren Zhou
Primary area: foundation or frontier models, including LLMs
Keywords: Large language model, Quantization, INT4 inference

Abstract:
Large language models have demonstrated promising capabilities upon scaling up parameters. However, serving large language models incurs substantial computation and memory movement costs due to their large scale. Quantization methods have been employed to reduce service costs and latency. Nevertheless, outliers in activations hinder the development of INT4 weight-activation quantization. Existing approaches separate outliers and normal values into two matrices or migrate outliers from activations to weights, suffering from high latency or accuracy degradation. Based on observing activations from large language models, outliers can be classified into channel-wise and spike outliers.
In this work, we propose Rotated Runtime Smooth (**RRS**), a plug-and-play activation smoother for quantization, consisting of Runtime Smooth and the Rotation operation. Runtime Smooth (**RS**) is introduced to eliminate **channel-wise outliers** by smoothing activations with channel-wise maximums during runtime. The Rotation operation can narrow the gap between **spike outliers** and normal values, alleviating the effect of victims caused by channel-wise smoothing.
The proposed method outperforms the state-of-the-art method in the LLaMA and Qwen families and improves WikiText-2 perplexity from 57.33 to 6.66 for INT4 inference.

Decision: accept (poster)
Reviewer scores: 5, 5, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1730250352502

thread_type:
official_review

<thread_object>
creator: reviewer_fkdx

<content>
summary:
This paper proposed a quantization technique designed to address outliers during Large Language Model (LLM) quantization, particularly for aggressive scenarios like W4A4 quantization. The method employs two steps: 1) mitigating channel-wise outliers through rotation, and 2) implementing online smoothing to further handle spike outliers. The authors demonstrate promising accuracy results across several benchmarks.

strengths:
- The authors provide a comprehensive analysis of activation distribution patterns, offering well-reasoned solutions for each identified challenge.
- They proposed a training-free approach that outperforms existing training-based methods on some benchmarks, reducing engineering complexity and showing potential for practical applications in industry scenarios.

weaknesses:
- Limited novelty: The core methodology is primarily based on existing approaches like [Smooth quantization](https://arxiv.org/abs/2211.10438 ) and [QuaRot](https://arxiv.org/abs/2404.00456 ), while the reordering technique shows overlap with existing research such as [ATOM](https://arxiv.org/abs/2310.19102) and [LLM.int8()](https://arxiv.org/abs/2208.07339).
- The experimental evaluation is limited in scope, focusing on common sense QA tasks. To better validate the method's effectiveness, the paper would benefit from:
    - More comprehensive comparisons with training-based methods like [SpinQuant](https://arxiv.org/abs/2405.16406) on more complex tasks, like [MMLU](https://arxiv.org/abs/2009.03300)
    - Detailed kernel performance comparisons with related research like [Qserve]( https://arxiv.org/abs/2405.04532), and [ATOM](https://arxiv.org/abs/2310.19102)

- Several aspects of the paper's presentation could be improved:
    - Figure 1: Show the challenges of existing methods, suggest adding visual explanations demonstrating how the proposed RRS technique addresses these challenges
   - Figure 1(b): Highlight the activation grouping methodology in the diagram
   - Figure 2(b): Provide a more detailed explanation of the probability metrics
   - Figure 6: Add clear y-axis labels and justify the unusual batch size choices (>1000).
   - Figure 9: Verify if the orange label represents `u > 8` instead of `u < 8`

questions:
Please refer the last item of Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697020304

thread_type:
official_review

<thread_object>
creator: reviewer_myvy

<content>
summary:
The authors were the first to highlight that channel-wise and spike outliers cause unnecessary distributional expansion in the quantization of large language models (LLMs).

The authors proposed a concise and effective technique called Rotated Runtime Smooth, which reduces these outliers to ensure that the activation distribution for LLM quantization remains compact.

The authors functionally explained, along with the methodology, how Runtime Smooth addresses channel-wise outliers and Rotated Smooth effectively handles spike outliers.

The authors demonstrated effective performance improvements across various LLM models using this approach.

strengths:
The authors focused on addressing channel-wise and spike outliers for the first time, which highlights a novel problem-solving approach and demonstrates significant performance improvements.

The methodology of Rotated Runtime Smooth (RRS) is simple, intuitive, and provides a clear and straightforward explanation of how it addresses each issue effectively.

By making their code publicly available, the authors have supported further research and made a valuable contribution to the ICLR community.

weaknesses:
The explanation of the methodology is insufficient: For example, in Figures 1 and 4, which aim to explain Runtime Smooth, it is difficult to gain a proper understanding. The main text also lacks adequate explanation of these figures.

For instance, in Figure 4, while the activation and weight reorder process for Runtime Smooth is presented, there isn’t enough explanation about why this process does not alter or distort the final result. Similarly, in Figure 1, it is unclear what the value of s represents (in relation to the equations in the text) and how exactly it reduces channel-wise outliers and effectively resolves the victim issue caused by spike outliers.

These examples highlight the need for a more detailed and clear description of the methodology, possibly in the appendix. This would help ensure that readers can thoroughly understand the method. While the high-level intuition is conveyed, the reproducibility of the method should be clearly delivered to readers through the paper itself, without relying on the availability of the code.

It is essential to discuss why this study is limited to INT4. The limitations and advantages should be clearly articulated. For example, there is no comparison with recent trending quantization techniques, such as One-bit LLM. A thorough investigation of related work is necessary, and efforts should be made, at least indirectly, to demonstrate the applicability and effectiveness of the proposed method in relation to these latest quantization techniques. https://arxiv.org/abs/2310.11453

======

Assuming that the above points are adequately addressed, I will start with a borderline or higher score. However, if it is determined that these points are not sufficiently reflected, or based on the overall opinions of other reviewers or AC, I will consider adjusting my final score.

questions:
Please refer to my comment for Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730732213086

thread_type:
official_review

<thread_object>
creator: reviewer_f1ta

<content>
summary:
This paper extends SmoothQuant [Xiao 2023] by:
 - Performing Quip-like [Tseng 2024] rotation before activation smoothing;
 - Obtaining smoothing scale in runtime and not merging it into weights;

strengths:
The paper is generally well-written, with a comprehensive ablation study (RRS vs RS and RRS vs SmoothQuant) demonstrating the effectiveness, which is well motivated by the preliminary section.

weaknesses:
The paper lacks an evaluation on the clock-time (instead of the operation count) overhead introduced by the method. This is a concern given that the method determines activation smoothing scale during runtime.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733926953

thread_type:
official_review

<thread_object>
creator: reviewer_2usn

<content>
summary:
The paper proposes a quantization method for LLMs, utilizing a smoothing operation during runtime to mitigate the issue of spike outliers and normalize values. Experiments conducted under INT4 settings validate the effectiveness of this method.

strengths:
1. The paper is clearly structured, making it easy to follow.
2. The method’s motivation is substantiated by experimental results, lending it credibility. In particular, the experiments in Section 4.5 demonstrate the minimal overhead introduced by the method, highlighting its practicality.

weaknesses:
1. **Figure 1 is confusing and hard to interpret.** The differences between panels (a) and (b) are unclear, as they are on completely different scales despite both sharing the same $ W $ and $ X $. Additionally, have you quantized $ \hat{X} = X \text{diag}(s)^{-1} $? The elements in $ \hat{x} $ appear to be integers, though they should be floats.

2. **Some terms and phrases are difficult to understand.** For instance, the caption for Figure 2(c) uses the phrase "channel-wise consistency" without providing any explanation, making it hard to grasp. Similarly, in point (1), what exactly is meant by "unmatched scale" in Figure 1?

3. **The motivation closely resembles prior work.** For example, DuQuant [1] also begins by addressing layers with massive outliers (in this paper, it is named as "spike") and tackles both massive and normal layers. Similarly, the method in [2] addresses large activation outliers and provides more analysis on this topic than the present paper.

4. **Some claims lack theoretical or experimental support.** For instance, the claim that "smoothing scales depending on the calibration set are prone to being unmatched with online activations" would benefit from experimental results demonstrating the likelihood of such a mismatch.

5. **Some explanations are too brief.** For example, the term "reorder" is mentioned without much elaboration, described only as "reordering the activations and weights according to the magnitude of smoothing scales."

6. **The method lacks clarity in its description.** In Equation 3, $ \hat{X} $ and $ \hat{W} $ appear to be in INT4 format, while $ s_i $ is a float (assuming this is correct, as the authors do not specify). How are these terms compatible for multiplication? A DeQuantize operation may need to be included somewhere in Equation 4.

7. **The ablation study feels incomplete.** The method employs several components—reordering, online smoothing, etc.—but the impact of each isn’t fully explored.

8. **The performance is underwhelming.** The proposed method does not appear to outperform SpinQuant [3] or DuQuant [1], even though it uses online quantization, whereas SpinQuant and DuQuant are offline methods.

Minors: 
1. In Equation 4, $ c_{i,j} $ should be $ s_{i,j} $.

[1] Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, L., ... & Wei, Y. (2024). DuQuant: Distributing outliers via dual transformation makes stronger quantized LLMs. *arXiv preprint arXiv:2406.01721*.  

[2] Yang, J., Kim, H., & Kim, Y. (2024). Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs. *arXiv preprint arXiv:2405.14428*.  

[3] Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., ... & Blankevoort, T. (2024). SpinQuant—LLM quantization with learned rotations. *arXiv preprint arXiv:2405.16406*.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730752702238

thread_type:
official_review

<thread_object>
creator: reviewer_fxyg

<content>
summary:
A plug-and-play quantization method based on runtime smoothing and the rotation operation of activations has been proposed. Runtime smoothing is responsible for eliminating channel-wise outliers, while the rotation operation mitigate the gap between spike outliers and normal values, resulted by channel-wise smoothing. Experimenting with INT4 inference on different LLMs shows that the proposed method can reduce the perplexity compared to other approaches.

strengths:
- Combining two exiting ideas in the Quantization literature to overcome both channel-wise and spikes outliers
- Comprehensive experiments with 3 different LLMs, including LLaMA, Qwen, Mixtral and Mistral models on WikiText-2 perplexity

weaknesses:
Some notations need to be corrected. For example,
- Line 168: identity matrix and number 1 should be distinguished. $|.|$ the norm needs to be determined. 
- Line 175: $absmax$ ---- > $abs(\max)$
- Equation (1): $\mathrm{X_j}$ needs to be defined as the columns of matrix $\mathrm{X}$.
- Equation(2): $/$ is not a valid operation for matrices. You need to write the formula as a multiplication with a diagonal matrix.
- line 214: The condition needs to be re-written: $s_j$ cannot be taken out of the sum.

Another issue is that the proposed method has been only tested for the  WikiText-2 perplexity. Can the approach is applicable for other modalities and tasks. For instance, ASR, speech translation, Image understanding, etc.

There is only a small improvement over QuaRot approach. I am wondering if your results are statistical significant?

questions:
Please see my comments above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732125743002

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Global response:
We are cheerful that the reviewers found our method well motivated (Reviewer f1tA), novel (Reviewer MyVY), and well presented (all reviewers). We extend our gratitude to the reviewers for their constructive feedback. We have addressed all the concerns raised by the reviewers in the revised manuscript. We have made the following changes in the revised manuscript:

A major concern most reviewers share is the absence of a wider variety of downstream tasks and baselines. In the previous version, we conducted comparisons between our methods and those of SmoothQuant, QuaRot on WikiText PPL, and Common Sense QA. In the revised version, we add more downstream tasks (MMLU, GSM8k (Math), CEVAL (Chinese), HumanEval (code)) and baselines (SpinQuant and DuQuant). It is important to note that our method provides an effective and efficient plug-and-play activation smoother, and we will continue to investigate the performance of future approaches that incorporate our method.

**More downstream tasks**

We evaluate our method on two more downstream tasks: MMLU and GSM8k, with 7/8B models from Qwen2.5, Mistral, LLaMA 3, and LLaMA 3.1. The results show that our method effectively smooths the activations and is robust to out-of-distribution (OOD) data, e.g., math and code, thereby achieving non-trivial improvements over the baselines.


| Model | Method | gsm8k_flexible (Math) | gsm8k_strict (Math) | MMLU | C-Eval (Chinese)|
|--------------|------------|----------------|--------------|-------| ------|
| Qwen2.5-7B | RRS (ours) | 70.96 | 62.85 | 63.00 | 66.6 |
| Qwen2.5-7B | QuaRot | 62.77 | 52.99 | 62.15 | 66.1 |
| Mistral | RRS (ours) | 37.30 | 36.69 | 56.65 | 40.6 |
| Mistral | QuaRot | 31.15 | 30.62 | 55.16 | 39.3 |
| LLaMA-3-8B | RRS (ours) | 31.46 | 31.38 | 53.51 | 35.5 |
| LLaMA-3-8B | QuaRot | 24.71 | 24.56 | 50.85 | 31.3 |
| LLaMA-3.1-8B | RRS (ours) | 36.08 | 35.93 | 54.96 | 38.8 |
| LLaMA-3.1-8B | QuaRot | 28.50 | 28.27 | 51.64 | 34.1 |
 

| Model | Method | HumanEval (code)|
|--------------|------------|--------|
| Qwen2.5-7B | RRS (ours) | 60.98 |
| Qwen2.5-7B | QuaRot | 49.39 |

Limited by computation resources, more downstream tasks and models are under testing.


**More baseline**

We compare our method with two more baselines: SpinQuant and DuQuant and their variants featured with Runtime Smooth (RS). The results show that our Runtime Smooth is well generalisable and valid, demonstrating the importance of runtime smoothing for various data from different domains.

| Method | gsm8k_flexible | gsm8k_strict | MMLU | arc_challenge | arc_easy | boolq | openbookqa | average |
|--------------|----------------|--------------|------|---------------|----------|-------|------------|---------|
| RRS | 31.4 | 31.3 | 53.5 | 44.8 | 71.1 | 73.6 | 42.4 | 57.9 |
| DuQuant | 24.4 | 24.0 | 49.4 | 45.8 | 68.7 | 71.3 | 42.2 | 57.0 |
| DuQuant+RS | 32.9 | 32.4 | 55.7 | 47.5 | 74.0 | 76.2 | 42.6 | 60.1 |
| SpinQuant+RS | 23.1 | 23.2 | 52.2 | 45.1 | 70.6 | 73.6 | 39.4 | 57.1 |
| SpinQuant | 22.9 | 22.6 | 48.4 | 43.2 | 67.3 | 72.7 | 38.8 | 55.5 |


Another concern is the clarity of the paper. We thank the reviewers for their feedback. We are actively addressing the concerns raised and refining our paper in the revised version to enhance its understanding for the wider community. Specifically;

1. We fix notations and typos in the paper. (R1, R2, R5)
2. We have included more related work for clarity. (R2, R5)
3. Provide more details in the caption for Figure 1. (R2, R4)
4. Clarify the process of reordering the caption in Figure 4. (R2, R4)
5. Clarify the detail of probability metrics in Figure 2(b). (R5)

Finally, we would like to express our great appreciation and excitement that several reviewers have recognized our  potential. We want to emphasize that our work contributes to the community not only by proposing a particular approach that works well with A4W4 inference, but also by providing a lightweight, robust activation smoother plugin. We believe that it is worth publishing to stimulate further discussion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Large language models achieve impressive performance as their scale increases but incur high computational and memory costs. Quantization mitigates these costs but struggles with activation outliers, hindering INT4 quantization. This work introduces Rotated Runtime Smooth (RRS), a plug-and-play activation smoother combining Runtime Smooth (RS) to handle channel-wise outliers and a Rotation operation to address spike outliers. RRS outperforms state-of-the-art methods, achieving significant improvements, including reducing WikiText-2 perplexity from 57.33 to 6.66 in INT4 inference.


Summary of the strengths and weaknesses of the paper according to the reviewers: 

Strengths:

+The paper introduces a plug-and-play quantization method combining runtime smoothing and rotation to address channel-wise and spike outliers.

+This paper claims negligible overhead compared to existing quantization methods while achieving better perplexity scores.

+It also extends experimental evaluation to multiple downstream tasks and baselines, demonstrating robustness across different domains.

+Finally, the reviewers appreciated the clear structure and logical flow, that they thought was helpful in understanding the methodology.

Weaknesses:

-Limited Novelty: Several reviewers noted overlaps with prior work (e.g., SmoothQuant, QuaRot, ATOM), questioning the methodological originality.

-Incomplete Methodology Explanation: Some figures (e.g., Figures 1 and 4) and terms were inadequately explained, making parts of the paper challenging to follow.

-Lack of Diversity in Evaluation: Initially focused on a narrow set of tasks (e.g., WikiText-2), prompting concerns about the generalizability of the results.

-Minor Performance Gains: Marginal improvements over prior methods led to skepticism about the significance of the results.

In their rebuttal, the authors addressed reviewer concerns by expanding experiments to additional tasks (e.g., MMLU, GSM8k) and demonstrating consistent improvements across diverse domains. They provided detailed runtime efficiency metrics, showing negligible overhead for their method. The authors clarified the novelty of their approach, emphasizing how the integration of Runtime Smooth and rotation resolves issues with prior methods. They also explained unclear figures and terms, committing to improving the final manuscript for better clarity. These efforts successfully convinced some reviewers to raise their scores, though skepticism about novelty and deeper theoretical insights persisted.

Reviewer reactions to the rebuttal were mixed. Some reviewers appreciated the additional experiments and detailed clarifications, leading to improved scores (e.g., Reviewer MyVY). They acknowledged the authors' efforts in addressing concerns about evaluation scope, efficiency, and clarity. However, other reviewers (e.g., Reviewer 2Usn) remained skeptical about the novelty of the approach and the lack of deeper theoretical insights, with concerns about incremental contributions persisting. The paper is a bit borderline with polarizing reviews. But I do think the authors response clarified major issues. While other issues persist I think the ICLR community will benefit from the discussion and I thus recommend acceptance. That said, I recommend the authors address the remaining issues raised by the reviewers including novelty concerns.