DuaRot: Dual Rotation for Advanced Outlier Mitigation in Rotated LLMs

Authors: Jingyang Xiang, Ying Zhang, Chi Ma, Yujie Wang, yulei, LiuChuan, Wei Lin, Yong Liu
Primary area: foundation or frontier models, including LLMs
Keywords: large language model, Walsh–Hadamard transform, reparameterization, hardware-aware, rotational invariance

Abstract:
By employing rotation, outliers in activations can be effectively mitigated without altering the output, thereby facilitating the quantization of large language models (LLMs). However, existing rotation-based methods only consider global activation distributions, leaving the finer-grained distributions underexplored. Additionally, these methods predominantly rely on the Walsh–Hadamard transform (WHT) to accelerate online rotation operations, while not fully considering performance between matrix multiplication~(Matmul) and WHT in actual runtime. These limitations hinder the rotation's ability to effectively reduce quantization errors and decrease inference speed. Therefore, improvements are needed in their performance regarding both accuracy and speed. In this paper, we propose a dual rotation method for rotation matrices, dubbed DuaRot, based on reparameterization. During training, DuaRot sequentially refines global and local features to achieve effective outlier mitigation. During inference, global and local rotations can be merged, which maintains rotational invariance without introducing additional computational overhead. Meanwhile, we propose a hardware-aware matrix configuration strategy, which determines whether the online Hadamard matrix should be expanded into a trainable parameter space by taking the runtime of the WHT and Matmul into account. This approach further enhances the reduction of quantization errors in online rotation operations without compromising inference speed. Extensive experiments demonstrate that DuaRot outperforms existing methods across various models and quantization configurations. For instance, when applied to LLaMA3-8B, DuaRot achieves WikiText-2 perplexities of 7.49 and 7.41 under W4A4KV4 and W4A4KV16 configurations with Round-to-Nearest (RTN), improving by 0.51 and 0.41 over the state-of-the-art, respectively. The code will be publicly available soon.

Decision: reject
Reviewer scores: 6, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730116367218

thread_type:
official_review

<thread_object>
creator: reviewer_5wop

<content>
summary:
This paper enhances SpinQuant using two techniques. First, it introduces learnable local rotation, which can be integrated with the original global rotation after training. Additionally, the paper notes that sometimes online Hadamard rotation is slower than same-dimension matrix manipulation. To address this, the paper suggests converting the slower online Hadamard rotation into trainable parameters for more accurate quantization.

strengths:
1. The writing is clear and easy to follow.
2. The paper provides comprehensive experiments and a detailed discussion of related works.

weaknesses:
1. What is the source of the QuaRot and SpinQuant results? Were these results reproduced by the authors? It would be beneficial to report the source of the comparison methods.
2. This method builds on SpinQuant with two new techniques. However, Table 2 shows that the accuracy improvement over SpinQuant is negligible in most cases. While Table 1 shows more significant improvement, it may result from overfitting to the WikiText2 dataset due to more trainable parameters. It would be more reliable to test WikiText2 perplexity on other datasets, such as C4.
3. The online Hadamard rotation matrix is shared across all blocks, while the hardware-aware strategy introduces additional parameters. The paper should discuss the additional parameter overhead.
4. The introduction mentions that "both QuaRot and SpinQuant slow down inference speed during the decoding stage." Does this paper address speeding up the inference during the decoding stage?

questions:
Please refer weakness for details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730579230687

thread_type:
official_review

<thread_object>
creator: reviewer_ssjm

<content>
summary:
This paper follows the rotation-based method QuaRot and proposes a method to learn dual rotation matrices for achieving smoother activation distributions.

strengths:
1. The hardware-aware matrix configuration strategy is well-motivated.
2. The illustration of rotational invariance is clear and enhances understanding.

weaknesses:
1. Although the authors mention that SpinQuant and QuaRot rely on GPTQ and aim to achieve smoother activation distributions, DuQuant [1] accomplishes this goal through two orthogonal transformations. There is a lack of discussion and experimental comparison with DuQuant, which directly addresses this motivation.
2. All evaluations are conducted on small-scale language models, leaving the effectiveness of DuaRot on larger models unexplored.
3. There is no measurement of speedup. Given that the authors claim improved efficiency through their matrix configuration strategy, this evalution should be included.
4. The optimization-based approach used to enhance existing baselines is not novel. 

[1] DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs. NeurIPS 2024

questions:
1. Could you discuss and compare DuaRot with DuQuant, which achieves competitive results and a smoother activation landscape without relying on GPTQ?
2. Including figures to illustrate the activation changes might strengthen the argument for DuaRot's effectiveness.
3. Please provide additional evaluation results on the LLaMA3-70B model.
4. Could you include evaluations of memory usage and inference speedup for DuaRot?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650227149

thread_type:
official_review

<thread_object>
creator: reviewer_vpvv

<content>
summary:
This paper introduces DuaRot, a method that trains global and local rotational matrices independently to effectively mitigate activation outliers.

strengths:
- The motivation for proposing a hardware-aware matrix configuration strategy is strong and well-supported.
- The paper is clearly written and easy to follow.

weaknesses:
- The paper does not provide measurements for speed and memory usage, which I believe are critical evaluations and should be included.
- The experiments are limited to small-scale language models, leaving the method’s effectiveness on larger models unexplored.
- It would be helpful to provide a more direct visualization of the reduction in activation outliers to better illustrate DuaRot’s effectiveness.

questions:
See Weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730741901059

thread_type:
official_review

<thread_object>
creator: reviewer_ejjf

<content>
summary:
The paper proposes a rotation-based method to alleviate issues with LLM quantization. The proposed method utilizes two strategies to enhance adaptability and achieves excellent performance even without GPTQ under the INT4 setting.

strengths:
1. The paper is well-structured and developed, making it easy for readers to follow.
2. The motivation for the proposed method is convincing.
3. The method is supported by numerous experiments, which verify its effectiveness from several perspectives.

weaknesses:
1. Some terms mentioned in the paper may be misleading. The phrase "hardware-aware configuration" suggests that the method can automatically adapt to specific hardware. However, it may be more accurately described as a hyperparameter. Additionally, there is results presented for only one type of hardware in the paper.
2. There is a lack of comparison between DuaRot and other baselines in **real runtime** for both training and inference matrices. Since the authors emphasize this point multiple times in the paper, it would be beneficial for them to provide more experimental results in this area.
3. The model size used in the paper is somewhat too small, which limits the persuasiveness of the method's effectiveness.
4. The ablation study is incomplete from my perspective. I believe it would be beneficial to verify the effectiveness of both global and local rotation matrices.

I would consider raising my score if my concerns are addressed.

questions:
1. As mentioned in the Introduction section, the lack of previous work indicates that "both QuaRot and SpinQuant slow down inference speed for the decoding stage." How does DuaRot address this issue? Is there any difference between DuaRot and the mentioned methods in this regard?
2. As noted in Weakness 1, I believe it would be beneficial if you conducted experiments on additional hardware, at least including the 3090 (or 4090). Otherwise, you may want to reconsider the naming of the strategy you used.
3. What do you mean by "w/o DuaRot" in Table 3 and Figure 5? Does it imply that the method degrades to SpinQuant or something else?
4. Could you specify the modules with the online matrix $R^{d\times d}$ where $d \ge 512$ in the LLaMA/Mistral models? If there is a scenario where the online matrix is entirely Hadamard due to the hardware-aware matrix configuration strategy, will this negatively affect performance?
5. It seems peculiar that in Figure 5, when the size of $R_L = 128$, the method shows similar performance with and without DuaRot. Could you please provide a simple explanation for this? Additionally, what do you mean by "instability during training"? Is there any situation where training cannot be completed due to this instability?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a dual rotation method aimed at mitigating outlier effects in quantization for large language models (LLMs). This approach seeks to enhance performance without compromising output integrity, marking it as a substantial advancement in the field. Upon review, opinions among the reviewers varied. While one reviewer expressed a borderline positive stance due to the paper's novelty and experimental support, the majority raised significant concerns regarding its limited experimental scope and marginal contributions.
Despite the authors' efforts to address these critiques in their rebuttal, key issues remained unresolved. Concerns regarding the restricted applicability of their findings, based on tests conducted primarily on smaller models, were acknowledged but inadequately mitigated. Notably, problems such as the inadequacy of their experiments to convincingly demonstrate improvements over existing methods, particularly the lack of comparisons with larger models or diverse datasets, persisted. Furthermore, the objection that the reported performance improvements may chiefly result from overfitting has not been satisfactorily addressed.
Ultimately, the consensus among reviewers leans toward recommending rejection due to these unresolved concerns. The promising idea presented holds potential, yet the current execution falls short of the standard required for acceptance. Further work is needed to substantiate the claims and validate the applicability of the proposed method across a broader range of contexts.