DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs

Authors: haokun  lin
haobo  xu
yichen  wu
jingzhi  cui
yingtao  zhang
linzhan  mou
linqi  song
zhenan  sun
ying  wei
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 15:17:43

Abstract: Quantization of large language models (LLMs) faces significant challenges, particularly due to the presence of outlier activations that impede efficient low-bit representation. Traditional approaches predominantly address Normal Outliers, which are activations across all tokens with relatively large magnitudes. However, these methods struggle with smoothing Massive Outliers that display significantly larger values, which leads to significant performance degradation in low-bit quantization. In this paper, we introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers. First, DuQuant starts by constructing the rotation matrix, using specific outlier dimensions as prior knowledge, to redistribute outliers to adjacent channels by block-wise rotation. Second, We further employ a zigzag permutation to balance the distribution of outliers across blocks, thereby reducing block-wise variance. A subsequent rotation further smooths the activation landscape, enhancing model performance. DuQuant simplifies the quantization process and excels in managing outliers, outperforming the state-of-the-art baselines across various sizes and types of LLMs on multiple tasks, even with 4-bit weight-activation quantization. Our code is available at https://github.com/Hsu1023/DuQuant.

Main paper: 
 introduction: 
 Large language models (LLMs) [52, 7, 51] have demonstrated exceptional performance across a wide range of natural language processing tasks. However, their billions of parameters present considerable deployment challenges on resource-constrained edge devices, particularly in terms of memory usage and inference speed [23, 15, 57] . In response to these challenges, network quantization methods [20, 24] have been extensively explored to minimize memory usage by converting floatingpoint parameters into low-bit formats [18, 33, 8] , and to expedite inference by quantizing both activations and weights for accelerating the matrix multiplication process [65, 35, 76] .Among LLM quantization methods, a primary issue is the presence of activation outliers, which enlarge the quantization step sizes and subsequently cause significant accuracy loss [60] . To mitigate this problem, current research has developed various methods to address Normal Outliers in activations, which are persistent in several channels across all tokens [13, 65] . However, besides Normal Outliers, there exists another type of activation outlier [49, 36] , termed Massive Outliers. These outliers are characterized by their exceedingly high values and limited occurrence in a subset of tokens, as depicted in Figure 1(b) . Unfortunately, existing LLM quantization methods struggle to effectively address these Massive Outliers. For instance, SmoothQuant [65] , despite using a smooth factor to shift some of the activation outliers to the weight part, still cannot effectively handle Massive Outliers with extremely large values, as shown in Figure 1(c)(d) . OmniQuant [48] and AffineQuant [40] , on the other hand, exhibit training instability issues [35] due to the presence of Massive Outliers. Consequently, there is a pressing need for an LLM quantization approach that effectively addresses both Normal and Massive Outliers.To tackle this challenge, we propose the Dual transformations Quantization (DuQuant) method. Our motivation is to redistribute the activation outlier values across different channels, facilitating easier quantization. Specifically, we construct the orthogonal rotation matrix and the orthogonal permutation matrix. By multiplying these matrices with the activations, we can effectively perform column transformations on the activations, which in turn allows for the redistribution of outliers. For the rotation transformation aspect, we first identify specific dimensions of outliers as the prior knowledge and employ a greedy algorithm to construct the rotation matrix. To enhance the multiplication efficiency, we utilize diagonal block-wise rotation matrices, with each matrix responsible for a small portion of the activations. However, this approach may result in uneven outlier magnitudes across different blocks. Therefore, we propose the zigzag permutation for reordering the activation channels, which promotes a more uniform distribution across different blocks. Concretely, we distribute the channels with the highest activations across the blocks in a back-and-forth pattern. After establishing blocks with uniformly distributed outlier magnitudes, we employ another rotation transformation to further redistribute the outliers within each block. Note that we multiply the weight matrix with the transpose of the rotation and permutation matrices at the same time, preserving the linear layer equivalence and smoothing weights. Theoretical analysis confirms that the rotation and permutation transformations greatly mitigate quantization challenges induced by outliers.As a result, DuQuant offers several clear advantages over QuaRot [2] : (1) DuQuant's optimal rotation matrix, derived through a greedy search guided by prior knowledge, surpasses QuaRot's Hadamard rotation in managing outliers; (2) our unique zigzag permutation significantly reduces activation variance across blocks, providing a distinct advantage for handling massive outliers; and (3) by jointly smoothing weights and activations, DuQuant avoids time-consuming GPTQ algorithm in QuaRot. Extensive evaluations demonstrate that our DuQuant approach significantly outperforms existing 4-bit weight-activation quantization baselines across various benchmarks. Notably, DuQuant achieves a 5% improvement in Commonsense QA tasks across all LLaMA model sizes and a 10% increase in zero-shot MMLU benchmarks for the Vicuna-v1.5-13B. Moreover, in practical applications with the LLaMA2-7B model, DuQuant not only accelerates pre-filling phase by up to 2.08× but also reduces memory usage during decoding phase by 3.50×, with minimal impact on performance: only a 0.61 increase in perplexity and a 2.71% drop in accuracy compared to the FP16 model. These results highlight the effectiveness of DuQuant in enhancing the efficiency and capacity of quantized LLMs. manifest large values across specific feature dimensions and are present in all token sequences [65] , which we refer to as Normal Outliers. Recently, a distinct type of outlier [49, 36] , termed Massive Outliers, has been observed in LLMs. The primary distinctions between normal and massive outliers are: 1) Normal outliers persist across all token sequences, whereas massive outliers are confined to a limited number of tokens. 2) Massive outliers exhibit significantly larger magnitudes, often surpassing 100 and being approximately 1000 times greater than the median of other activations [49] . In our study, we delve deeper into the impact of these two distinct types of outliers on quantization.Massive Outliers Exist at the Second Linear Layer of FFN Module. In contrast to previous studies [49, 36] that observe massive outliers at the output of Transformer blocks, we first discover that these extremely large activations exist at the input of the down-projection layer within the FFN module. As depicted in Figure 1 , the input of the down-projection layer in the LLaMA2-7B model Layer 1 contains a single activation of significant magnitude (approximately 1400). This activation is isolated to one token and therefore classified as one of massive activations. This phenomenon is consistently observed across different layers and sizes of models, as illustrated in Appendix I.Massive Outliers Enlarge Quantization Difficulty. Although previous studies [65, 48, 40, 1] have proposed various approaches to eliminate outlier features, they still face challenges in effectively managing massive outliers. SmoothQuant [65] , for instance, attempts to shift the quantization difficulty from activations to weights by dividing the activation by a per-channel smoothing factor and multiplying it to the weight matrix. Nevertheless, we observe that this transfer at the input of the down-projection layer can cause the weights of the down-projection to display noticeable outliers, as demonstrated in Figure 1 . This issue arises because massive outliers cause the smoothing factor to become significantly large. Moreover, extremely large outliers can lead optimization-based methods to encounter problems with loss explosion. Both OmniQuant [48] and AffineQuant [40] have had to exclude their learnable parameters for the down projection layer due to unstable gradients. Given the poor accuracy observed with 4-bit quantization, QUIK [1] opts to use INT8 quantization for the down projection layer and Atom [76] applies INT8 quantization for 128 outlier channels. Consequently, massive outliers introduce new challenges to the quantization process that existing methods cannot fully address. This observation has motivated us to develop rotation and permutation transformations, which effectively handles both massive and normal outliers and achieves state-of-the-art performance. 
method: 
 In this section, we delve into the distribution of outliers and introduce our proposed DuQuant method. The DuQuant method is built on two key components: 1) the block-diagonal rotation matrix, tasked with the local redistribution of feature outliers, and 2) the zigzag permutation, responsible for the global reordering of outliers across different blocks.As the common modules within each transformer block of LLMs, both Multi-head Self-Attention (MSA) and Feed-Forward Network (FFN) fundamentally consist of basic linear layers, which can be represented as, Y = X • W ∈ R T ×Cout . Here, X ∈ R T ×Cin is the activation input and W ∈ R Cin×Cout denotes the weight matrix. In this paper, we focus on integer uniform quantization [26] of both activation and weight, aiming to achieve better hardware support. Specifically, the b-bit quantization process maps the FP16 tensor X to low-bit integer X q :EQUATIONThe notation ⌊•⌉ means the nearest rounding operation, ∆ is the quantization step size and z represents the zero point. Following [65, 48, 35, 40] , we employ per-token quantization for activation and per-channel quantization for weight, which entails assigning different step sizes to individual tokens of activations (∆ X ∈ R T ×1 ) and different output channels of weights (∆ W ∈ R 1×Cout ).To address the Normal Outliers issue stated in Section 2, current quantization methods, such as SmoothQuant [65] and OmniQuant [65] , usually adopt the smooth technique. Concretely, it involves the utilization of a per-channel smoothing diagonal matrix, denoted as Λ, to scale the input activation and weight matrix. The adjustment allows us to rewrite the original linear layer as Y = X•W = (X• Λ)(Λ -1 •W). The diagonal element Λ j within Λ is computed as Λ j = max(|X j |) α /max(|W j |) 1-α ,where α is a hyper-parameter representing the migration strength. However, despite the ability of this smoothing technique to shift the quantization challenge from activations to weights, it still faces difficulties in effectively managing Massive Outliers, as depicted in Figure 1 . This challenge stems from the extremely large massive outliers inducing large scaling factors Λ j , which in turn introduce new outliers in the weight matrix and result in significant performance declines in 4-bit quantization.According to these findings, we propose the DuQuant method, which includes the Rotation and Permutation transformations based on the smooth technique. By combining rotation transformation and channel permutation, our DuQuant method aims to redistribute these features within the activation space, thereby mitigating the effects of both Normal and Massive Outliers.The Rotation Transformation. In contrast to the smooth technique, our aim is to apply a rotation matrix for row or column transformations, mitigating the impact of both Normal and Massive outliers. The ideal rotation matrix, denoted as R, should possess the following properties: 1) R is an orthogonal matrix satisfying RR ⊤ = I and |R| = ±1. This allows us to reformulate the linear layer within the transformer as Y = X • W = (XR)(R ⊤ W); 2) R should be capable of effectively target the positions of outliers and effectively mitigating them through matrix multiplication. However, due to the Massive Outliers are usually randomly distributed within the activation space, it is challenging to directly identify the optimal rotation matrix R capable of mitigating outliers through a single rotation transformation. To address this problem, we employ a greedy search with prior knowledge to compute a rotation matrix R, thereby approximating the ideal rotation matrix R. Specifically, the calculation of R involves the following steps,• Identify the feature dimension d (1) where the outlier are primarily concentrated, i.e., d (1) = arg max j (max i |X ij |). Here, X ij represents the element in the i-th row and j-th column of X.• Based on the searched dimensions d (1) , we construct the rotation matrix as follows,EQUATIONHere, E d (1) is the switching matrix used to swap the first and the d (1) -th columns of the activation, and R represents an orthogonal initialized rotation matrix, in which the first row is specifically uniformly distributed. The motivation behind this is to mitigate outliers in the first column after the transformation by E d (1) . To further increase the randomness of the rotation operation, we retain the first column, where outliers have been mitigated, and randomly rotate the other columns by multiplying them with a random orthogonal matrix Q ′ . • Let N denote the greedy search steps, then the approximated rotationmatrix R = R 1 R 2 • • • R n , where n = arg min k∈[1:N ] max i,j |(XR 1 • • • R k ) ij | . Each R i is constructed according to Eqn.(2) and the identified feature dimension d (i) . Appendix G provides detailed pseudo code.Through this construction manner, we can ensure that the approximated optimal rotation matrix R can effectively mitigate outliers with large magnitudes, as opposed to merely using a randomly selected orthogonal rotation matrix. Nevertheless, directly constructing the entire rotation matrix is timeconsuming and results in substantial memory overhead. For fast matrix multiplication, following [64] , we approximate the rotation matrix R ∈ R Cin×Cin in a block-wise manner,EQUATIONwhere Rbi ∈ R 2 n ×2 n denotes a square matrix of the i-th block, which is constructed following the three steps mentioned above. And the block numbers K is calculated by K = C in /2 n . The Permutation Transformation. Despite adopting the block-diagonal rotation matrix R for its time and storage efficiency, its focus on local information introduces a potential limitation in further reducing the outliers. This is because the rotation transformation, conducted within each small block, cannot integrate the information across different blocks to further minimize outliers. Consequently, one block may have relatively larger outliers while another block has smaller outliers, resulting in high variance among different blocks, as shown in Figure 2 . This limitation explains that merely utilizing the block-diagonal rotation matrix is insufficient to effectively reduce the overall outliers.To effectively mitigate the overall outliers, it is essential to balance the outliers' magnitudes among various blocks. Specifically, within each small block, we denote the largest outlier in dimension d j as O j . Meanwhile, M bi represents the mean value of all O j in the i-th block, where i = 1, 2, ..., K. Then the variance in activation magnitudes across various blocks can be expressed as,EQUATIONTo minimize this variance and further reduce the overall outliers, we introduce the zigzag permutation. Concretely, we generate a zigzag sequence that starts by assigning channels with the highest activations to the first block. The process continues by assigning channels with the next highest activations to the subsequent blocks in descending order until the end of block K. Upon reaching the final block, the order reverses, starting from the channel with the next highest activation and proceeding in ascending order. This back-and-forth patterning continues throughout all the blocks, ensuring that no single block consistently receives either the highest or lowest activation channels.It is worth noting that the constructed permutation is an orthogonal matrix, which we denote as P, satisfying the conditions PP ⊤ = I and |P| = ±1. By employing the zigzag permutation, we achieve a balanced distribution of outliers across different blocks. This allows us to use an additional rotation transformation to further smooth the outliers. Figure 2 provides an illustration of outlier mitigation.The Overall DuQuant Method. To effectively mitigate both Normal and Massive Outliers, we first employ the smooth technique to shift the quantization challenge from activations to weights. Next, we introduce the block-diagonal rotation matrix R to locally redistribute feature outliers within the activation space. We then propose the zigzag permutation matrix for globally balancing the outliers across different blocks, followed by another application of the block-diagonal rotation transformation.To sum up, the linear layers within the transformer can be rewrite as,EQUATIONwhere the notation P denotes the orthogonal permutation matrix learned via the zigzag manner, the R(1) and R(2) represent the first and second block-diagonal rotation matrix, respectively.Remark 1. It is worth noting that the proposed DuQuant method can simultaneously smooth the weight matrix. While the commonly adopted smooth technique is effective, it can cause the weight matrix of the down-projection layer to exhibit pronounced outliers, leading to performance degradation. However, in the proposed DuQuant method, the rotation transformation we designed is applied to not only the activation input but also the weight matrix. As a result, the outliers induced by the smooth technique can be mitigated through our approximated rotation matrix R, yielding a smoother, more quantization-friendly weight matrix. Moreover, this approach eliminates the reliance on complex weight quantization techniques, such as GPTQ [18] used in Atom [76] and QuaRot [2] .Remark 2. To further decrease the computation and memory costs, we initially construct the k-th block rotation matrix Rb k , with the k-th block containing the largest outlier. We then assign Rbi = Rb k for all 1 ≤ i ≤ K. This strategy not only effectively mitigates the impact of outliers, but also reduces the number of block rotation matrices from K to 1, significantly reducing computation and memory requirements. Importantly, incorporating the invertible matrix G from Eqn. (5) significantly eases the quantization challenges for X and W. Consequently, the quantization process acts asY = (XG)(G -1 W) = X • Ŵ ≈ ∆ X∆ Ŵ( Xq -z X)( Ŵq -z Ŵ).To further demonstrate the effectiveness of the proposed DuQuant method, we conduct a theoretical analysis of the rotation and permutation transformations. Theorem 1 shows that within each block, the constructed rotation matrix effectively mitigates the maximum outlier, thereby reducing the outlier magnitude through a greedy search. Theorem 2 reveals that the employed zigzag permutation ensures a balanced upper bound shared among different blocks. This suggests that the zigzag permutation effectively reduces the variance shown in Eqn. ( 4) and thus assists the rotation matrix in further decreasing the outliers. Please refer to Appendix B for detailed proofs. Theorem 1 (Rotation). For the activation input X ∈ R T ×Cin , R ∈ R 2 n ×2 n is a diagonal block matrix constructed as per Eqn. (3). For a specific block b i , let O j (•) represent the maximum outlier of the j-th dimension d j within the input. Then, we can deduce that,EQUATIONTheorem 2 (Zigzag Permutation). For the activation input X ∈ R T ×Cin , it can be divided into K blocks, where K = C in /2 n . Let O j denote the max outlier of the dimension d j in X, the reordered outliers from large to small is expressed as O (1) , O (2) , ..., O (Cin) . Moreover, the M bi represents the mean value of all O j in the i-th block, i = 1, 2, ..., K.Let δ := max{|O (i+1) -O (i) |}, i = 1, 2, ..., C in -1.Then, following the zigzag permutation described in Section 3.2, the mean value M bi within each i-th block consistently satisfies,EQUATION4 Experiment Models and Evaluations. We apply our DuQuant on pre-trained LLMs: LLaMA (7B-65B) [52] , LLaMA2 (7B-70B) [53] , LLaMA3 (8B, 70B), Mistral, Phi2 and instruction-tuned LLMs: Vicuna-v1.5 (7B-13B) [10] . We evaluate quantized pre-trained LLMs on language generation tasks and commonsense QA tasks. Specifically, we assess the perplexity on WikiText2 [41] and C4 [45] datasets, as well as the zero-shot accuracy on PIQA [6] , ARC [12] , BoolQ [11] , HellaSwag [71] , and WinoGrande [46] datasets. Moreover, we evaluate quantized Vicuna models on MMLU [22] and MT-Bench [78] benchmarks, as well as their long-form generative capabilities on LongBench [4] .Implementation Details. In line with prior studies [35, 48, 40] , we apply per-token activation quantization and per-channel weight quantization. Given that W8A8 quantization has been established as lossless in precision by SmoothQuant [65] , our primary evaluation in this paper focuses on 4-bit and 6-bit quantization for weights and activations. As for details, we quantize all intermediate activations, excluding the SoftMax output. Moreover, we have developed two types of quantized models, denoted as DuQuant and DuQuant+LWC . For DuQuant, we employ round-to-nearest quantization, using a clipping ratio of 0.9 for activations and 0.8 for weights. To improve weight matrix quantization, DuQuant+LWC integrates the learnable weight clipping (LWC) technique from Omni-Quant. Concretely, LWC adjusts weights by training parameters γ, β∈ [0, 1] to compute step size ∆ = γ max(X)-β min(X) 2 b -1in Eqn. (1) . Notably, the smoothing diagonal matrix and the learned weight clipping factor can be integrated into the quantized weights, introducing no additional computational or memory costs. More details and hyperparameters are left in Appendix C.Baselines. We compare with state-of-the-art (SOTA) weight-activation PTQ methods, including SmoothQuant [65] , Outlier Supression+ [60] , OmniQuant [48] , QLLM [35] , AffineQuant [40] , and Atom [76] . For Atom, we reproduce the results with no group-wise asymmetric quantization.Table 1 : Perplexity (↓) results under 4-bit weight-activation quantization. The results for W6A6 can be found in Table D8 . Atom and OmniQuant unprocessed group-query attention for LLaMA2-70B. Quantization of Instruction-tuned Models. We quantize Vicuna-v1.5 [10] models to assess the generalizability of our DuQuant. Table 3 illustrates that our quantized models surpass the baselines across all task categories on MMLU benchmark. For Vicuna-13B, our DuQuant+LWC surpasses Atom by 10.01% under zero-shot settings and 6.95% under five-shot settings. Moreover, we compare our DuQuant with Atom and OmniQuant using MT-Bench and utilize GPT-4 to evaluate the answers from quantized models. As shown in Figure 3 , DuQuant quantized models significantly outperform both Atom and OmniQuant in win rates. Specifically, for Vicuna-7B, DuQuant only lost 16 and 1 times to Atom and OmniQuant, respectively, while achieving 68 and 155 wins against them.EQUATIONEvaluation of Long-context Generation. To further evaluate the long-text generative capabilities, we follow [38, 34, 56] and conduct a comprehensive comparison of DuQuant against state-of-the-art baselines on the LongBench [4] , which includes a variety of generative tasks to provide a broader evaluation. We set the maximum sequence length to 3500 for Vicuna models, with results presented in Table 4 . DuQuant achieves performance comparable to FP16 models, demonstrating the effectiveness of our dual transformations. More detailed results on different subtasks are listed in Table D3 , D4.Quantization of LLaMA3 Models. LLaMA3, known for its superior performance in various tasks, faces significant degradation in low-bit quantization [25] . To address this, we apply our DuQuant to quantize LLaMA3-8B. Table 5 displays the perplexity and zero-shot accuracy results. Notably, under W6A6 setting, our DuQuant achieves performance comparable to FP16 model. Furthermore, unlike other methods that show weaker results under W4A4 setting, our DuQuant maintains competitive performance, indicating its robustness with LLaMA3. We attribute this success to the advanced handling of outliers achieved through dual transformations, which is not restricted to specific models. 
ablation study: 
 Module-wise Impact. We ablate four distinct operations within DuQuant: 1) only the smoothing technology like SmoothQuant; 2) one rotation following the smoothing operation; 3) a sequence of rotation, permutation, and another rotation without smoothing; and k4) full DuQuant approach. Table 6 shows that the smoothing operation plays a basic role in our DuQuant by shifting activation outliers to weight. The initial rotation significantly enhances model performance, yielding competitive PPL results. Finally, permutation combined with a second rotation further enhances the quantized model. Influence of Normal/Massive Outliers. In this section, we comprehensively explore the influence of massive and normal outliers on quantization. Notably, we observe that massive outliers primarily occur at the down-projection of the FFN module. To isolate their effect, we remove the rotation and permutation transformations, applying only the smoothing technique to all down-projection inputs.The resulting perplexity for LLaMA2-7B and LLaMA-13B showed significant degradation, presented in Table 7 . Conversely, when we eliminate the rotation and permutation transformations for normal outliers, the performance decrease was noticeable but less severe compared to massive outliers. These findings indicate that: 1) massive outliers exert a more substantial impact on quantization, corroborating our claims in Section 2; 2) the smoothing technique alone struggles to fully mitigate the influence of outliers, particularly massive ones; and 3) our rotation and permutation methods prove highly effective against both types of outliers, leading to superior performance. Vicuna-v1.5-13BComparison with QuaRot [2] In light of the recent introduction of Hadamard rotations by QuaRot [2] to eliminate outlier features, we have undertaken a detailed analysis to highlight the key differences between our DuQuant and QuaRot. To ensure a balanced evaluation, we have reimplemented QuaRot in accordance with our quantization settings. The results demonstrate that 1) the rotation matrix constructed by DuQuant outperforms QuaRot's approach of simply selecting a randomly initialized Hadamard matrix. As depicted in Figure 10 , our DuQuant more effectively smooths activations than QuaRot. This is attributed to the prior knowledge utilized by DuQuant to accurately target the outliers; 2) As demonstrated by the perplexity in Table 8 , QuaRot employs GPTQ for their weight quantization method, whereas our DuQuant, with its sophisticated outlier management, attains competitive results using RTN quantization. The superiority proves the effectiveness of our zigzag permutation to enhance capacity. For a more comprehensive comparison, please refer to Appendix F. Permutation Frequency. We conduct ablations on the rotation and permutation frequencies in DuQuant. As shown in Figure 5 , "Perm 1" (two rotations with one permutation) achieves stronger performance compared with "Perm 0" (no permutation), while incurring an additional 8.9% computational cost on LLaMA2-7B and 9.3% on LLaMA2-13B compared to the W4A4 setup. Considering the approximately 2× speedup and the impressive performance, these additional costs are deemed acceptable. Further permutations, like "Perm 2," do not improve performance and reduce inference efficiency. Consequently, "Perm 1" strikes the best balance between perplexity and inference speed, making it the optimal configuration for DuQuant.Inference Speedup. To assess the inference speedup delivered by our DuQuant, we adopt the measurement strategy and W4A4 kernel from [2] . We evaluate the layer-wise speedup of LLaMA2 models on one NVIDIA 3090 GPU, with results detailed in Table 9 and 10 . We set the pre-filling sequence length at 2048 and decode for 128 steps. In the pre-filling stage, DuQuant achieves a 2.08× speedup over FP16 for LLaMA2-7B and a 2.34× speedup for LLaMA2-13B, with slight variations across different batch sizes. In the decoding stage, batching the token generation phase yields high throughput without any downside [44] . Consequently, we enlarge the batch size to 64 and the results for LLaMA2-7B in Memory Consumption. We measure the peak memory usage of DuQuant with the W4A4 kernel on LLaMA2-7B using a single NVIDIA 3090 GPU. We process 2048 tokens for pre-filing and run 128 decoding steps, with the results listed in Table 11 . In the pre-filling stage, DuQuant, SmoothQuant, and QuaRot achieve up to 3.2× memory reduction, while QLLM performs worse. In the decoding stage, DuQuant maintains strong memory efficiency, with superior performance. Runtime. Our DuQuant stands out for its efficiency, surpassing other baselines [48, 35, 40, 76] . The blockwise rotation ensures fast multiplication between the rotation and activation matrices. Zigzag permutation, involving simple channel swaps, is much faster than complex algorithms like Simulated Annealing, as discussed in Appendix E.3. Moreover, the advanced management of outliers makes DuQuant not rely on GPTQ or gradient-based training. Hence, DuQuant enables a rapid quantization process shown in Table F24 , e.g., we successfully quantize LLaMA2-13B in just 71s with superior performance. 
conclusion: 
 In conclusion, this paper presents DuQuant, an innovative quantization strategy for large language models (LLMs) that effectively addresses the challenge of outlier activations. By integrating rotation and permutation transformations, DuQuant effectively mitigates the impacts of both massive and normal outliers. This strategic redistribution of outliers not only simplifies the quantization process but also leads to substantial improvements in model performance. Consequently, DuQuant establishes new state-of-the-art results in 4-bit weight-activation quantization scenarios. This advancement enhances the deployment of efficient LLMs in resource-constrained environments. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Please refer to Section 4 and Appendix C for experimental settings.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Our study used a fixed seed for all quantization operations, following standards in post-training quantization, and thus did not report statistical significance.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the broader impacts of our work in Appendix H.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This work poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite all datasets and models utilized in our experiments. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Theorem 1 (Rotation). For the activation input X ∈ R T ×Cin , R ∈ R 2 n ×2 n is a diagonal block matrix constructed as per Eqn. (3). For a specific block b i , let O j (•) represent the maximum outlier of the j-th dimension d j within the input. Then, we can deduce that,EQUATIONProof. In the case of a specific block b i , the potential maximum value O j (X bi Rbi ), where X bi ∈ R T ×2 n , can be achieved by ensuring that the different j-th column outliers O j (X bi ) are located in the same t-th row of the activation input X bi . This can be formally defined as follows,max 1≤j≤2 n O j (X bi Rbi ) = O 1 (X bi E d (1) ) • 1 √ M + O 2 (X bi E d (1) ) • δ 1 + ... + O M (X bi E d (1) )δ M ,where M = 2 n and ∥1 2 n + δ 2 2 + ... + δ 2 M ∥ = 1, ( Rbi is orthogonal)Without loss of generality, let's assume that δ i ≥ 0 and define m := arg max i δ 2 i . Then we can derive that δ m ≥ 1 √ 2 n . Consequently, we can obtain the following inequality,EQUATIONEQUATIONThe inequality (1) holds because the switch matrix E d (1) has swap the largest outliers O 1 (X bi ) in the first column, i.e., O 1 (X bi E d (1) ) > max 2≤m≤M O m (X bi E d (1) ).Theorem 2 (Zigzag Permutation). For the activation input X ∈ R T ×Cin , it can be divided into K blocks, where K = C in /2 n . Let O j denote the max outlier of the dimension d j in X, the reordered outliers from large to small is expressed as O (1) , O (2) , ..., O (Cin) . Moreover, the M bi represents the mean value of all O j in the i-th block, i = 1, 2, ..., K. Let δ := max{|O (i+1) -O (i) |}, i = 1, 2, ..., C in -1. Then, following the zigzag permutation described in Section 3.2, the mean value M bi within each i-th block consistently satisfies,EQUATIONProof. According to the zigzag permutation described in Section 3.2, and considering the reordered outliers O (1) , O (2) , ..., O (Cin) , we can redistribute the channels they occupy (i.e., Oc )across different blocks. Specifically, for the i-th block, it contains the following channels,EQUATIONSimilarly, we can deduce that all M bi (i = 1, 2, ..., K) share the same upper bound after applying our zigzag permutation.In this work, all experiments are done on NVIDIA RTX 3090 GPUs for small-scale models and NVIDIA A100 GPUs for large-scale models. We set sequence length to 2048 for all evaluation tasks.For calibration data, following [48, 40, 35] , we randomly select 128 sampled sequences from the WikiText2 dataset, with the sequence length of 2048. For rotation and permutation transformations, the rotation block size 2 n is set to 128, and maximum greedy search steps N equals 256. We adopt once permutation times for efficiency. We conduct detailed ablation studies in Appendix E.2, E.3, E.4.Regarding quantization details, for multiplications between activations in MSA, such as Query and Key, attention outputs and Value, we apply a Hadamard rotation matrix for rapid and straightforward processing. A Hadamard matrix is an orthogonal and symmetric matrix filled with elements ±1/ √ 2 n . For smooth parameter α, we set it to 0.6 for DuQuant and 0.5 DuQuant+LWC. We clip the maximum activation values in all projection blocks, and the clipping ratio is set to 0.9. For DuQuant we also clip the maximum values in weight matrices, with a clipping ratio of 0.8. For LWC, we keep the same default epoch numbers of 20, batch size as 1, learning rate as 5e-3, and zero weight decay, as [48] . Zero-shot QA Results for 4-bit LLaMA2 Models. Table D1 showcases the zero-shot commonsense QA results for INT4 quantized LLaMA2 models. Our DuQuant method excels across various model sizes and datasets, demonstrating state-of-the-art performance in commonsense reasoning tasks. For example, DuQuant outperforms Atom by 5.43% for the LLaMA2-7B model and by 5.18% for the LLaMA2-13B model. In contrast to Atom [76] , which relies on GPTQ for weight quantization and maintains 128 channels at INT8, thereby increasing memory usage, our method offers a rapid and more efficient weight-activation quantization solution through Rotation and Permutation.MMLU Results for 4-bit Vicuna-v1.5-7B. Vicuna-v1.5 models [10] , fine-tuned from LLaMA-2 models using high-quality user-shared conversations, are considered state-of-the-art chatbots. Table D2 displays the INT4 quantization results for Vicuna-v1.5-7B on the MMLU benchmarks. In comparison to SmoothQuant, OmniQuant, and Atom, our DuQuant method exhibits the smallest DuQuant largely surpasses other baselines, particularly with Mistral-7B. Regarding the Phi2-2.8B model, it often experiences instability in matrix multiplication between queries and values, leading to overflow issues and posing great challenges to quantization. However, while DuQuant may not perform as well as FP models, it still significantly outperforms other baselines. In addition, we have visualized the massive outliers in the down projection layer of the Mistral-7B model and the feature space after our dual transformations. These visualizations are shown in Figure I9 . It can be observed that our DuQuant perfectly eliminates these outliers. These results underscore the effectiveness of our dual transformation approach in addressing massive outliers across various types of LLMs. W6A6 Quantization Results. To thoroughly evaluate the effectiveness of our DuQuant models, we conduct comprehensive assessments under the W6A6 quantization setting. The perplexity results for language generation tasks are displayed in Table D8 , while the zero-shot accuracy for Commonsense QA tasks is detailed in Tables D9 and D10 . Our findings reveal that DuQuant not only surpasses other baselines but also achieves nearly lossless performance with FP16 models in these tasks. Interestingly, in several instances, DuQuant slightly outperforms DuQuant+LWC . This suggests that the Rotation and Permutation transformations alone are sufficient to create highly competitive quantized models under W6A6 settings, without the need for additional enhancements such as the learnable weight clipping (LWC) technique. These outcomes highlight the exceptional versatility and robustness of DuQuant across various quantization scenarios, confirming its potential as a leading solution in post-training quantization for large language models.Table D8 : Preplexity (↓) results on the WikiText2 and C4 datasets under 6-bit WA quantization. The current generation of LLMs usually splits into pre-filling and decoding phases and deploys on two separate machines [44] . Here, we present more speedup and memory-saving results for these two phases achieved with the LLaMA2-7B model on a single NVIDIA RTX 3090 GPU. We set the input sequence length to 2048 and the decoding steps to 256. End-to-end results for time speedup and memory savings during the pre-filling stage are shown in Table E11 and E12 . We can observe that DuQuant achieves a maximum speedup of 2.01× during the pre-filling phase, with speedup increasing as the batch size grows. From Table E12 , DuQuant demonstrates significant memory savings, effectively reducing memory usage by up to 3.20× through quantization. For the decoding phase, we enlarge the batch size to 64 and measure speedup along with memory usage for one LLaMA2-7B layer, constrained by the 24 GB memory of the GPU. As shown in Table E13 , DuQuant maintains speedup and memory usage comparable to QuaRot. These results underscore the efficiency of DuQuant in optimizing resource utilization, highlighting its potential to enhance performance and reduce costs in deploying large language models, particularly in resource-constrained environments. EQUATIONAblation of Rotation Block Size. To further explore the impact of rotation block size, we apply varying block sizes in the rotation matrices to both LLaMA2-7B and LLaMA2-13B models and evaluate the perplexity of the quantized models. The results, presented in Table E14 , indicate that increasing block sizes generally improves model performance. This improvement occurs because larger block sizes allow outliers to be distributed across more channels, evening out values throughout the activation/weight matrix thereby enhancing quantization accuracy and performance. Additionally, quantization runtime decreases with larger block sizes, likely due to more efficient transformations during the reshaping of original activation/weight matrices. Consequently, we adopt 128 as our rotation block size for all experiments for efficiency and effectiveness.Ablation of Rotation Times. Identifying the optimal rotation matrix R is a complex challenge, so we employ a greedy search algorithm to approximate the matrix as R. We conduct an ablation study on the number of greedy steps N and summarize the results in Table E15 . Initially, as N increases, the model performance improves, reflecting our ability to determine R more effectively. However, when N reaches 1024, the model begins to overfit. Consequently, we have chosen N = 256 for all our experiments, as it offers the optimal balance between model performance and time usage. As discussed in Section 3.2, rotation transformations within each block are limited and unable to redistribute outliers across different blocks. To address this, we introduce a permutation transformation aimed at balancing outliers more comprehensively. Our primary goal is to minimize the variance among different blocks, as outlined in Eqn. (4). We explore several optimization algorithms, with the results detailed in Table E16 . Note that the variance values are measured on activation values of the query project in the first layer of each model, and the time in the table represents the runtime of calibration. The Zigzag permutation notably reduces the variance to 3.0e-4, achieving this with minimal time expenditure and yielding competitive perplexity results. While Simulated Annealing slightly outperforms Zigzag in terms of perplexity for the LLaMA2-7B model, it was significantly more time-consuming, and the marginal gains did not justify the additional complexity. Therefore, we select Zigzag permutation as our preferred method, leading to smoother outlier distribution and more effective quantized models. Ablation of Different Calibration Datasets. We apply our DuQuant to quantize the LLaMA2-7B model using different calibration datasets, with results presented in Table E17 . It can be observed that the selection of calibration datasets has a relatively minor impact on quantization performance. This is because our method uses the calibration data solely to identify outlier channels, rather than for gradient-based parameter learning as seen in methods like OmniQuant [48] and AffineQuant [40] . This ablation study underscores the robustness of our DuQuant method. Calibration-free Quantization. To further explore the robustness of DuQuant under varying calibration conditions, we generate random calibration data within the vocabulary range of the model, setting the sample count to 256. The results, shown in Table E18 , indicate that even in calibrationfree settings, our method continues to perform well, achieving results that are competitively close to those obtained with actual calibration data. This demonstrates that DuQuant could provide a viable solution in real-world scenarios where obtaining specific calibration data is challenging or impossible. In addition, our findings suggest that outliers are inherent to certain model layers, reflecting characteristics of the model weights or modules, especially in recent LLMs. This aligns with concurrent research: [67] identified consistent massive outliers at the FFN down projection layer in GLU-based LLMs, such as LLaMA, Mistral, Mixtral, SOLAR, and Gemma, while [43] reported that, although OPT models are sensitive to different calibration sets, newer models demonstrate robustness to outliers and maintain stable activations. These insights reinforce the idea that outliers are more linked to the internal structure of model weights and modules than to calibration data. DuQuant's ability to deliver high performance without relying on traditional calibration data creates opportunities for deploying quantized models in environments with stringent privacy requirements or limited data availability. This highlights a promising direction for future research, focusing on improving model adaptability and deployment flexibility. Ablation of Different Numbers of Calibration Samples. We utilize our DuQuant to quantize the LLaMA2-7B model using varying numbers of calibration samples from the WikiText2 dataset, with results detailed in Table E19 . Interestingly, the quantization performance shows a low correlation with the number of samples, demonstrating the robustness of DuQuant Ṫhis stability arises because we utilize the mean activation values from these samples to construct our rotation matrices. Since we average the activations, the influence of any single, potentially non-representative sample is minimized, ensuring consistent performance. Notably, as we use mean values, the time cost of our quantization process remains constant regardless of the number of samples, enhancing the efficiency. • Efficient Channel Permutation: Our channel permutation not only further smooths outlier features but also benefits from rapid implementation, enhancing overall performance.• Simultaneous Weight Matrix Smoothing: Unlike QuaRot, DuQuant directly and efficiently smooths the weight matrix, avoiding the time-consuming GPTQ algorithm and accelerating the quantization process, as demonstrated high quantization efficiency in Table F24 .Experimental results underscore the superiority of DuQuant over QuaRot. We first summarize the experimental setting differences between the original paper of QuaRot with ours in Table F20 * . For a fair comparison, we reproduce the QuaRot under 4-bit per-channel weight and per-token activation asymmetric quantization. Table F21 displays the perplexity (PPL) and zero-shot accuracy for models LLaMA1-7B, LLaMA2-7B, and LLaMA3-8B. Our DuQuant method consistently outperforms QuaRot-RTN across all benchmarks, showcasing our advanced weight matrix management. Furthermore, compared to QuaRot-GPTQ, and DuQuant+LWC achieve better average accuracy across six QA tasks and demonstrate superior performance on the WikiText and C4 datasets, particularly for LLaMA3-8B. Moreover, we further provide the results of DuQuant under the setting utilized in the original paper of QuaRot in Table F22 . DuQuant still surpasses QuaRot by a large margin.Additionally, we assess the effectiveness of the rotation matrices utilized in DuQuant which incorporate prior knowledge against the Hadamard matrices used in QuaRot. We omit the permutation step in DuQuant and directly contrast it with QuaRot-RTN. Results in Table F23 show that our DuQuant without permutation outperforms QuaRot by a clear margin, which confirms that our rotation transformation is more effective than Hadamard by leveraging prior knowledge. It is worth noting that because a Hadamard matrix is orthogonal and symmetric, it multiplies by itself to yield the identity matrix. In other words, the Hadamard matrix is not suitable for greedy searches aimed at finding smaller outliers. These findings differentiate DuQuant from QuaRot and highlight the effectiveness of our approach in managing outliers for post-training quantization of large language models.Input: Pre-initialized rotation matrix R, greedy search steps N , activation matrix X with shape of [T, Cin] Output: Rotation matrix R function get_rotation_matrix (X, R, N ) 1: T , Cin = X.shape 2: R = eye(Cin) 3: a = maxi,j |Xij| 4: for k in 1, ..., N do 5: channel_max = X.abs().max(dim = 0).values 6: outlier_channel = arg max(channel_max) 7:Obtain randomly initialized orthogonal matrix Q ′ with shape of [Cin -1, Cin -1] 8:Q ′ = concat([zeros(Cin -1, 1), Q ′ ], dim=1) 9: Q = concat([zeros(1, Cin), Q ′ ], dim=0) 10: Q[0, 0] = 1 11: R ′ = matmul( R, Q) 12: R ′ [:, outlier_channel], R ′ [:, 0] = R ′ [:, 0], R ′ [:, outlier_channel] 13: R ′ [outlier_channel, :], R ′ [0, :] = R ′ [0, :], R ′ [outlier_channel] 14: R = matmul(R, R ′ ) 15: X = matmul(X, R ′ ) 16:if maxi,j |Xij| < a then 17: R = R 18: a = maxi,j |Xij| 19:end if 20: end for 21: return R * Following prior works [48, 35, 40] , we describe operations on Query, Key, and Value states as per-tensor quantization for consistency. To be precise, these operations are effectively applied on a per-head basis.Limitations. The primary limitation of our method is the lack of a specialized strategy for calibration data selection. We adhere to established practices [48, 40, 35, 76, 2, 1] by randomly selecting 128 samples from the WikiText2 dataset to compute the mean embeddings that inform our rotation matrix and zigzag permutation order. We also explore the possibility of calibration-free quantization and show some promising results. However, further investigating more tailored choices for calibration data can potentially enhance the performance of our quantized models. We leave this for future study.Broader Impacts. Our work identifies the presence of massive outliers in down-projection layer of FFN modules, which significantly complicates low-bit weight-activation quantization. To address this challenge, we implement a combination of rotation and permutation matrices to effectively smooth both massive and uniform outliers, proving both fast and effective. Consequently, we establish a new state-of-the-art for INT4 weight-activation post-training quantization methods. Our approach aims to accelerate large language models and reduce memory usage during deployment, offering substantial benefits to the field of LLM research. These advancements could lead to more efficient and accessible LLM applications, facilitating broader usage and enabling more sustainable AI implementations.We provide additional visualizations of normal and massive outliers in various models (LLaMA1, LLaMA2, Vicuna-v1.5, Mistral) from Figure I1 to Figure I9 . In each figure except Mistral, the left side illustrates changes in normal outliers before and after applying our rotation and permutation transformations, while the right side shows the changes in massive outliers before and after transformations. It is evident that massive outliers consistently occur in the down-projection layer of the FFN module across all models, supporting our findings discussed in Section 2. Conversely, normal outliers appear in different modules within the transformation block. For instance, Figure I3 shows normal outliers at the up-projection layer of the FFN module in LLaMA1-13B. Significantly, both massive and normal outliers are reduced markedly after our rotation and permutation transformations, leading to easier quantization of activations. This underscores the effectiveness of our DuQuant in managing outlier features across diverse LLM models. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: The authors propose a new method for weight and activation quantization in LLMs that targets the existing issue of large-magnitude activations and their effect on quantization. Their method applies a carefully-designed linear transformation to better distribute these outliers, with theoretical and practical demonstrations of success. Initial reviews of the paper were positive, and increased further after clarifications and additional results during the rebuttal and discussion phase. I concur with the reviewers and recommend acceptance. The authors should carefully incorporate feedback from the reviewers into the final revision.

Reviews: review: paper_summary: paper_summary: The paper proposes a new LLM quantization method named DuQuant (for Dual transformations Quantization). This method is able to quantize the weights and activations of an LLM to 4 or 6 bits without losing significant precision.

The paper identifies the issue of Massive outliers in the activations of a LLM. These are outliers within the outliers, having magnitudes in the order of 100-1000. Traditional quantization methods, such as SmoothQuant, cannot deal with these outliers, because they attempt to address the errors locally. DuQuant, on the other hand, uses a sequence of rotation, permutation, and another rotation to spread these outliers out across many weights, resulting in a more accurate capture.

The results are supported with theoretical results as well as experiments. DuQuant outperforms the baselines on perplexity, QA and MT-bench.
strength_weakness: strength_weakness: A shortcoming of the paper is the lack of close comparison to QuaRot. QuaRot is a very similar technique and it deserves to be in the related works section as well as be a baseline in the experiments. While there is a detailed comparison in the appendix, the work should also be mentioned in the main body.

The paper could be improved at places, but I don't consider any of these shortcomings a major issue:
* The paper has numerous typos (eg.: Line 11: Typo, Line 188: typo, Line 313: Figure reference missing). Please proofread.
* The paper is extremely dense, its sometimes difficult to distinguish the figure captions from the main body. Please use proper paddings for tables and figures where possible.
* The description of how the rotation matrix is constructed could be more detailed (Lines 141 - 147). Given that this is a core algorithmic contribution of the paper, it should be clear to the reader how it works. Perhaps a figure or pseudocode would be helpful here.
questions: questions: No outstanding questions.
limitations: limitations: No limitations.
paper_summary: paper_summary: The paper explores new approaches in LLM quantization. The work tries to address the performance degradation due to massive outliers in the weights. The work show competitive performance across different settings, up to 4-bit weight activation quantization. The work also provides solid experiments and visualization on the effectiveness of the proposed methods.
strength_weakness: strength_weakness: 1. I am a bit puzzled by the results in Table.1 vs Table.2. It is shown that lower perplexity does not necessarily reflect in higher benchmark accuracy. This conflict exists within DuQuant (DuQuant vs DuQuant_LWC) and other methods like AffineQuant,OmniQuant vs QLLM. The authors might want to consider adding some justification to this conflict to further improve the soundness of the tables.
questions: questions: From the benchmarks, it is shown that the proposed method almost matches the fp16 results (unquantized). It would be interesting to see the mt-bench of the quantized vs fp16 model to see if such an observation still holds.
limitations: limitations: NA
paper_summary: paper_summary: The paper presents a new post-quantization method (DuQuant) that targets low-precision (4-bit / 6-bit) weight and activation quantization. The authors show how the presence of massive outliers affects quantization when using existing methods (smoothing is not sufficient with SmoothQuant / OmniQuant training is not stable for layers exhibiting massive outliers). They propose their method for better handling both normal and massive outliers - by utilizing orthogonal rotation and permutation matrices, with a simple zigzag permutation scheme for a better / more even distribution of outliers. They provide theoretical proofs for both their rotation and zigzag-permutation operations grounding their proposed algorithm. They showcase the strength of the propsed methods as it can be enabled with simple quantization, and not rely on expensive quantization methods like GPTQ to achieve new state of art quantized models across a range of different models.
strength_weakness: strength_weakness: 1. Most of the evals (except MT-Bench) are logit-based evals (and not generative). This has a side effect of hiding some of the inherent limitations of low-precision quantization algorithms (i.e., error accumulation across generated tokens).
2. One limitation of the proposed benchmarks (for implementation performance) is that they measure the prefill performance, but do not show any generation performance. This typically dominates over prefill performance.
3. One thing is the reporting of results is not consistent across the models - for eg. some tables use Atom for reporting model quality at 4-bits, but some models do not report this performance. While the benefits of the method are clear given the higher accuracies on downstream tasks, it is difficult to judge how the differences translate on models of higher quality (e.g. Llama-2 vs Llama=3 70B).
questions: questions: 1. Many times the perplexity numbers and downstream performance are not 1:1 correlated. This most likely has to do with the standard error of the downstream tasks? If so, it will be good to clarify this for the reader.
2. Figure 5 is not clear in explaining the different settings - what do Perm 0, 1, and 2 correspond to? Also in the paragraph above the figure, in lines 312-313, it is not clear which Figure is being referred to.
3. For Table 6, it will be better to explain how to read the table. It took multiple passes to understand the full setup of the ablations and the associated results in the table.
4. It is surprising that the method is largely calibration free - indicating that the outliers are more a property of the model weights, and activations are suffering as a by-product of this? can the authors clarify their intuition around this.
limitations: limitations: The authors address the limitations and broader impacts of their work. One thing they do mention in the checklist is reporting of statistical signficance - which I do not see anywhere in the paper. Can authors point to where these results are? Or equivalently change the checklist to reflect this.
paper_summary: paper_summary: This work proposed a transformation (composition of orthogonal and permutation transformation) that makes LLMs more quantization-friendly (accounting for the presence of outlier features). The approach is validated on several modern LLMs from Llama-1,2,3 families.
strength_weakness: strength_weakness: The idea of applying rotation transformation for simplifying weight/activation quantization is not novel and (to my knowledge) was first introduced in QuaRot. The introduced method proposes a specific form of orthogonal/permutation matrices. QuaRot results for Llama-2-7b quantization reported in QuaRot paper are significantly better than the one presented in Table 7 and only marginally inferior to DuQuant numbers. I would suggest comparing with the numbers from original paper for fairness.
questions: questions: * Do the learned rotation possess incoherence property [1]? 

* Can one boost GPTQ performance further (as it is strictly stronger method compared to RTN) with DuQuant rotation matrices? 

* What is the context length for Llama-3-8B evaluation? Typically, one uses the training context length for comparison (8k for Llama-3 model family). 

[1] Chee, Jerry, et al. "Quip: 2-bit quantization of large language models with guarantees." Advances in Neural Information Processing Systems 36 (2024).
limitations: limitations: See Weaknesses.
paper_summary: paper_summary: The paper introduces a method called DuQuant, a new quantization method specialized for LLMs. The paper notes that "massive outliers" cause previous quantization approaches to be less effective or powerful, and then proposes a new quantization method which ameliorates the effect of such massive outliers. Theoretical analysis shows that the method has good properties, such as successfully spreading out the outlier mass directly before quantization, which facilitates better outcomes. Experiments show state-of-the-art performance across standard language modelling benchmarks on Llama and Llama-2 class models, preserving most of the base model's performance.
strength_weakness: strength_weakness: Several potentially desirable properties of the model may not be completely covered by the analysis. For example:
- The memory consumption reduction seems good, but the presentation of this result is postponed to the appendix E. Could you find a way to put this in the main body (maybe moving back the presentation of one of the several pure performance metrics such as PPL, or one of the ablation studies), and compare to other state-of-the-art models? In my opinion, since people usually only do quantization for better memory consumption or speeding up inference, measuring the performance on both of these axes is crucial.
- The speedup is only measured for pre-filling. Is it possible to compute end-to-end speedup for open-ended generation? It would be great to see the impact of the method on some more realistic workloads (again, across the axes of performance, memory consumption, and speedup), to supplement the thorough analysis of each component in the paper.
To address both these issues, along with an empirical study, an analysis of the asymptotic runtime/memory complexity of the quantization procedure would be great if possible.

Also a slight nitpick: at the bottom of page 2, massive activations are not the same phenomenon as attention sinks, though they can be related.
questions: questions: - Previous work on Massive Activations have shown their ubiquity in many types of LLMs, not just Llama, and even on certain types of vision models. Is it possible that this work can be done on other types of LLMs/transformers? Does the backbone model you pick significantly change the effectiveness of the method (e.g., Phi vs Llama vs Mistral)?
limitations: limitations: The authors are generally precise about the method's strengths and weaknesses, and include a limitations section.

Comments: title: To Reviewer Rbrg
Response to Long-Context Results
To Reviewer x516
Speedup for Decoding Stage and Results for Long-context Generation
Thank you for your reply
To Reviewer 8e4p
To Reviewer XTv7
To Reviewer nwuk
Reply to Rebuttal
comment: We are pleased to have addressed all concerns regarding the long-context properties of DuQuant. We appreciate the reviewer's decision to raise the score and will ensure to include these results in the main paper. Additionally, we plan to extend our experiments to LLaMA 3.1 series models to further validate the effectiveness of DuQuant. Again, we thank the reviewer for these invaluable suggestions and look forward to further enhancing our work with these additional evaluations.
These results look very reasonable. Because they resolve my concern about the long-context properties of DuQuant compared to other methods, I will raise my score. Please add these results to the main paper. Along with that, I would recommend to try the method on as many Llama-class models as possible (my understanding is that you don't need to rewrite the kernels for this), beyond just Vicuna. The results for Llama 3.1 8B or 70B (if hardware allows) would probably be the most practically relevant.
We are pleased that we have effectively resolved the key issues identified by the reviewer. We extend our sincere thanks for the reviewer’s thorough examination of our manuscript and the constructive feedback provided. We will revise the manuscript to include the details about the construction of the rotation matrix.
> - We appreciate the reviewer's detailed feedback, which is crucial for improving our work. The current generation of LLMs usually splits into pre-filling and decoding phases and deploys on two separate machines [1]. As we have already provided speedup/memory usage results for the pre-filling stage in the previous response and original paper, we further measure the **decoding stage speedup**.
>
> - In the decoding stage, batching the token generation phase yields high throughput without any downside [1]. Consequently, we enlarge the batch size to 64 in the decoding stage and measure speedup along with memory usage for one LLaMA2-7B layer, constrained by the 24 GB memory of an RTX 3090. We set the pre-filling sequence length at 2048 and decode for 128 steps. The results are presented below:
>
>   | One Layer, INT4, BS=64 | Time (ms) | Saving Factor | Memory (GB) | Saving Factor |
>   | ---------------------- | --------- | ------------- | ----------- | ------------- |
>   | FP16                   | 659       | -             | 3.550x       | -             |
>   | SmoothQuant            | 437       | 1.508x         | 1.669       | 2.127x         |
>   | QLLM                   | OOM       | -             | OOM         | -             |
>   | QuaRot                 | 457       | 1.442x         | 1.678       | 2.116x         |
>     | DuQuant                | 499       | 1.321x         | 1.677       | 2.117x         |
>
>     - From the table, the results demonstrate that DuQuant maintains speedup and memory usage comparable to QuaRot while delivering superior performance.
>
> - To further enhance real-world application speedup, we are grateful for the reviewer’s suggestion and committed to (1) developing more advanced W4A4 kernels to enhance decoding speedup in future work, or (2) combining our methods, which are compatible, with other decoding speedup techniques, such as speculative decoding, to substantially improve the overall speedup of DuQuant.
>
> - Responding to your interest in long-term generation results, we have included additional evaluations with LongBench, designed for long-context scenarios. With a **maximum generation length of 3500**, DuQuant significantly outperforms other baselines. We list the average results for Vicuna models as follows, while for the more detailed results please refer to the response to Reviewer XTv7 W1.
>
>   | Vicuna      | 7B (Avg.) | 13B (Avg.) |
>    | ----------- | --------- | ---------- |
>   | FP16        | 39.21     | 40.77      |
>    | SmoothQuant | 4.62      | 2.73       |
>   | OmniQuant   | 1.56      | 3.93       |
>    | Atom        | 33.19     | 30.61      |
>   | DuQuant     | **37.25** | **38.75**  |
>   
> [1] Patel, Pratyush, et al. "Splitwise: Efficient generative llm inference using phase splitting." *2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)*. IEEE, 2024.
Thank you for the detailed reply and for addressing my concerns. I think the construction of the rotation matrix will be an important clarification in the revised manuscript.
We are delighted to see that the major concerns raised by the reviewer have been successfully addressed. We would like to express our deep appreciation for the reviewer's dedicated time and effort in scrutinizing our paper and providing invaluable feedback.
We are pleased that the concerns raised by the reviewer have been addressed, and we will incorporate the additional experimental results and clarifications during our discussion into the revised version. Thanks again for the time and effort the reviewer has dedicated to reviewing our paper and providing valuable feedback.
We thank the reviewer for acknowledging our efforts to address the concerns. We are grateful for the decision to reconsider the score based on our responses. The constructive feedback has greatly enhanced our manuscript.
I thank the authors for the rebuttal and clarification. I think the rebuttal addresses my concern and I will raise the score.
Thanks for your detailed response and clarifications. My concerns regarding the fairness of the evaluation setup are resolved. Hence, I have decided to raise the score.
Thank you for the detailed response to the reviews and additional experimental results. After reading all reviews and response, and the overall global response, I will retain my score. 

I encourage the authors to do the following for the camera ready version:
    1. Add the generative evals (W1 response) and memory profile (W2 response) to the supplementary and link them in the main work for readers.
    2. Add the clarifications for Q 2 & 3 to the paper for easier reading
    3. Address the Q4 answers in the supplementary work.
Thank you for the detailed response to the reviews. After reading all reviews and responses as well as the global response, I will keep my score. I would have liked to see more results on long form generation (both memory, which was provided, as well as speedup and accuracy), but overall it's a strong work.