Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:21:26

Abstract: Modern large language models (LLMs) have established state-of-the-art performance through architectural improvements, but still require significant computational cost for inference. In an effort to reduce the inference cost, post-training quantization (PTQ) has become a popular approach, quantizing weights and activations to lower precision, such as INT8. In this paper, we reveal the challenges of activation quantization in GLU variants, which are widely used in feed-forward network (FFN) of modern LLMs, such as LLaMA family.
The problem is that severe local quantization errors, caused by excessive magnitudes of activation in GLU variants, significantly degrade the performance of the quantized LLM. We denote these activations as activation spikes. Our further observations provide a systematic pattern of activation spikes: 1) The activation spikes occur in the FFN of specific layers, particularly in the early and late layers, 2) The activation spikes are dedicated to a couple of tokens, rather than being shared across a sequence. Based on our observations, we propose two empirical methods, Quantization-free Module (QFeM) and Quantization-free Prefix (QFeP), to isolate the activation spikes during quantization. Our extensive experiments validate the effectiveness of the proposed methods for the activation quantization, especially with coarse-grained scheme, of latest LLMs with GLU variants, including LLaMA-2/3, Mistral, Mixtral, SOLAR, and Gemma.
In particular, our methods enhance the current alleviation techniques (e.g., SmoothQuant) that fail to control the activation spikes.

Main paper: 
 introduction: 
 Large language models (LLMs) have become a key paradigm in natural language processing, accelerating the release of variations within the community [49, 58] . Furthermore, latest LLMs establish state-of-the-art performance by training with increased scale, as well as by adopting architectural improvements such as GLU [40] , RoPE [41] , GQA [2] , and MoE [21] . Especially, GLU (Gated Linear Unit) variants (e.g., SwiGLU, GeGLU) has been adopted in the most of modern LLM architectures (e.g., LLaMA family [46] ), due to training efficiency [31, 40] . Although LLMs broaden foundational capabilities in natural language tasks and potential for various applications, billions of parameters in the large models impose considerable computational costs on end users in practice. To reduce GPU memory requirements and accelerate inference speed, post-training quantization (PTQ) offers an affordable solution by quantizing weights and activations into a lower precision (e.g., INT8) without a need for expensive retraining steps [17, 19, 30] . However, recent studies have revealed that large magnitude values at certain coordinates exist in the activations of LLMs, which are often called outliers, posing a key challenge in activation quantization [1, 12, 50, 51] . Another line of works attempts to explain the role of outlier values in the attention mechanism [9, 42] . Nevertheless, current research on the impact of evolving LLM architectures on the outliers remains insufficient.In this paper, we present our discovery that the GLU architecture in the feed-forward network (FFN) generates excessively large activation values, which are responsible for significant local quantization errors. Specifically, we observe that these problematic activation values occur in specific linear layers and are dedicated to a couple of tokens, which will be discussed in Section 3. To distinguish the excessive GLU activations from the outliers, we refer to them as activation spikes. In light of our observations, we propose two empirical methods to mitigate the impact of activation spikes on quantization: Quantization-free Module (QFeM) and Quantization-free Prefix (QFeP). QFeM aims to partially exclude quantization for linear layers (or modules) where large quantization errors occur, instead of quantizing the entire linear modules in the LLM. By scoring the extent of scale disparity, QFeM selects linear modules to exclude. On the other hand, QFeP identifies the prefix that triggers activation spikes and preserves its context as a key-value (KV) cache, thereby preventing the recurrence of activation spikes in subsequent tokens. It is noteworthy that both QFeM and QFeP rely on calibration results to capture activation spikes in advance, without any modifications to the target LLM. This indicates that our methods can be integrated into any existing quantization methods.In our comprehensive experiments, we demonstrate that recently released LLMs incorporating GLU variants struggle with activation spikes when applying activation quantization. Consequently, the proposed methods, QFeM and QFeP, substantially enhance the performance of the primitive quantization method, the round-to-nearest (RTN) method. Furthermore, we observe that current outlier alleviation methods [50, 51] are exposed to the activation spikes and benefit from our proposed methods. Compared to the strong baseline of fine-grained activation quantization [55] , our methods show competitive performance, achieving reduced latency and memory footprint.In summary, the contributions of our work are as follows:• We find that the GLU architecture in modern LLMs systematically generates excessive activation values, which are responsible for significant performance degradation in activation quantization.• Based on our observations, we propose two empirical methods, QFeM and QFeP, which effectively exclude the activation spikes during quantization, with negligible computational overhead and compatibility with any existing quantization techniques.• Our extensive experimental results validate the detrimental impact of the activation spikes on activation quantization, while our proposed methods consistently enhance the quantization performance. 
related works: 
 Outlier Values in LLMs. Previously, outlier values have been observed in the transformer-based language models such as BERT [14] and early GPT [36] models through numerous studies [8, 24, 27, 35, 45] . Since the advent of LLMs [10, 57] rooted in the GPT, recent studies by [1, 12, 51] have tackled the existence of outlier values in LLMs. According to them, these outliers exhibit a large magnitude of values at the shared dimensions of hidden states across tokens. More recently, [9, 42] explain that the outliers attribute to the vertical pattern in the attention mechanism [25, 52] , which influences the performance of LLMs. In particular, [42] claims a different type of outlier existing in the hidden states of specific tokens. However, prior studies merely focus on the superficial hidden states between the decoder layers. Our work provides a module-level investigation where quantization is applied practically, focusing on different LLM architectures.Post-training Quantization for LLMs. Post-training quantization (PTQ) refers to the quantization of a neural network model to low precision, such as INT8, without additional parameter updates [17, 19] . Especially for LLMs, this approach cost-effectively achieves inference with low memory usage and faster inference latency by quantizing the weights and activations used in matrix multiplication (e.g., linear layer). However, because of the challenges in activation quantization of LLMs, many recent works are mainly focused on the weight-only quantization [11, 13, 15, 23, 26, 39, 54] .Otherwise, the activation quantization faces inherent outliers, which hinder accurate quantization by reducing representation resolution. To address this challenge, [12] proposes a mixed-precision quantization method where the outlier dimensions are computed in high precision. [50, 51] approach migration of scale from activation to weights to alleviate the scale of outlier activations. Along this line of research, we propose to enhance the activation quantization based on our observations. Figure 1 : Calibration results on GLU-implemented and non GLU-implemented LLMs. We present the maximum magnitudes of input activations for each linear modules and layer-wise hidden states. For more results on different LLMs, see Appendix A.2, A.3.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [No] Justification: The limitation of our work is that our methods are based on the observations without theoretical validation. However, our extensive experimental results validate the effectiveness of our methods. 
activation spikes: excessive magnitude of glu activations: 
 For clarity, "hidden states" refer to the output tensor of a transformer layer (or block), while "input activations" or "activations" denote the input tensor of a linear layer (or module) in the remain of this paper. Recent work [42] has investigated a novel type of outlier existing in the hidden states across modern LLMs. Although these outliers of hidden states play a crucial role in the attention mechanism [9, 42, 52] , their relationship with input activations for quantization has not been fully explored.Importantly, because recent LLMs adopt Pre-LN [4, 53] , which normalizes hidden states before selfattention and feed-forward network (FFN) blocks, the scale of hidden states does not reflect the scale of input activations within the transformer block. Therefore, we focus on the input activations fed into each linear module within the transformer block to connect to activation quantization. Specifically, we examine the four linear (projection) layers: query (parallel to key and value), out, up (parallel to gate), and down modules. For detailed illustration of Pre-LN transformer, please see Appendix D.1.To analyze the input activations, we employ a calibration method, which is used to estimate the quantization factors such as scale and zero-point. For the calibration data, we use 512 samples randomly collected from the C4 [37] training dataset. Afterwards, we feed each sample into the LLM and monitor each hidden state and input activation through the decoder layers. To estimate the scale factor, we use absolute maximum value. The tested LLMs are listed in Appendix A.1.GLU-implemented LLMs exhibit activation spikes at specific layers. In Figure 1a , we display the calibrated scale factors for the LLMs that implement GLU variants (e.g., SwiGLU, GeGLU).Across models, we observe a shared pattern of scale from the results. Within the early and late layers, the down modules in the FFN show noticeable magnitudes of input activations. Note that these input activations are derived from the Hadamard Product within GLU. Thus, the GLU variants generate activation spikes at the specific layers. Interestingly, we notice a high correlation between the emergence of activation spikes and intermediate hidden states of large scale. This indicates that the FFN contributes to amplifying the hidden states via the addition operation in the residual connection [18] . Once the magnitude of the hidden states is exploded, it persists through layers until encounter the activation spikes at late layers.Non GLU-implemented LLMs show modest scale distribution. Figure 1b illustrates the calibration results for LLMs with the original feed-forward implementation in Transformer [48] . We observe that the LLMs continue to generate the large-scale hidden states, regardless of the GLU implementation. This corresponds to the observations in [42] . More importantly, our module-level results elaborate that the scale of hidden states is not transferable to the input activations of inner linear modules. Instead, we reveal that GLU variants are associated with the hidden states and generate activation spikes. This clarifies the quantization challenge of the GLU-implemented LLMs concentrated in the early and late layers. Because excessive scales of activation spikes have the potential to hinder the accurate quantization, we conduct an in-depth analysis to better understand these activation spikes in the following sections. In the previous section, we observed the excessive scale of the input activations derived from GLU activation. When quantizing the input activations, the variance of input activation scales for each token affects the quantization performance [55] . To delve into the disparity between token-wise scales in the activation spikes, we unroll them through the sequence of tokens. Figure 2 illustrates the individual input activation scales where the activation spike appears. Given a token sequence, the large magnitudes of input activations are observed in a couple of tokens, such as the BOS token, newline (\n), and apostrophe ('). These specific tokens coincide with the observations of [42] , which suggests that such tokens exhibit massive values in the hidden states. Thus, the activation spike is associated with the process of assigning a special role to these tokens in later transformer layers.However, the excessive scale of specific token hinders the estimation of scale factor for the other tokens, such as in per-tensor quantization. Additionally, the largest scale is dedicated to the first instance of the specified token, while the following usage exhibits a modest scale. This phenomenon makes the quantization more complicated, as the activation spikes dynamically occur depending on the current input sequence.We explore the impact of local quantization errors caused by activation spikes on LLM outputs. To identify the layers where activation spikes occur, we utilize a ratio between the maximum and median values of the token-wise input activation scales, instead of using the maximum scale value alone.The max-median ratio for linear layer m can be formulated as r (m) = max(S (m) ) median(S (m) ) , where S (m) represents the token-wise input activation scales incoming to module m ∈ M . This max-median ratio captures the extent to which maximum scale dominate the other token scales. For comparison, we choose the activation quantization targets as the top-4, middle-4, and bottom-4 modules, based on the max-median ratio in descending order. Then, we evaluate the perplexity and mean-squared error (MSE) using the calibration dataset. Here, the MSE is calculated for the last hidden states between the original (FP16) and partially quantized LLM. As shown in Table 1 , quantization on the top-4 rated modules solely degrades the LLM performance by significant margins, while the other cases exhibit negligible performance changes. We consider these quantization-sensitive input activations (inter alia activation spikes) to be the quantization bottleneck, which, in this paper, refers to the quantization error caused by outliers.Furthermore, the activation spikes are conditioned on the specific context of the input sequence as discussed in Section 3.2. Altogether, such dynamic bottlenecks must be handled with caution to enhance the quantization performance of LLMs. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
mitigating quantization quality degradation based on the observation: 
 To address the quantization bottleneck, our approach is based on the deterministic occurrence patterns of activation spikes. First, we utilize the observation that bottlenecks occur at a few specific layers.This implies that naive full quantization of LLMs is affected by these bottlenecks. Second, we exploit the phenomenon that the activation spike is derived from the first occurrence of specific tokens. Thus, the planned occurrence prevents recurrence in the subsequent and possibly future tokens. In the following sections, we propose two methods inspired the above insights.In the full quantization of LLM, all linear layers within the LLM are quantized. Among these linear layers, we propose omitting the quantization of input activations for linear layers where significant quantization errors are caused by activation spikes. To be noted, increasing the number of unquantized modules exhibits a trade-off between the inference latency and the model performance.Thus, determining which module should be quantized (or left unquantized) is crucial to retain the efficacy of quantization. Here, we use the max-median ratio r (m) and define a set of unquantized modules, denoted as M unq , where the ratio r (m) of each linear layer is larger than threshold α. For instance, all linear layers in M are quantized if α = ∞. For clarity, we treat sibling linear layers, such as query-key-value, as a single linear layer. To control the impact of activation quantization only, we leave the weight parameters in unquantized linear layers as INT8 and dequantize them into FP16 during matrix multiplication with the incoming activations, operating as weight-only quantization. 2 .Orthogonal to the QFeM, we propose Quantization-free Prefix (QFeP) that mitigates the quantization errors by precomputing the prefix (or short prompt) corresponding to activation spikes. This method is based on the observations presented in Section 3.2, which indicate that significant quantization errors result from the overestimated scale factor of the first instance within the restricted token set. Inspired by this occurrence pattern of activation spikes, we aim to construct a prefix which stabilizes the quantization scale factor of the tokens that come after the prefix. In other words, once the prefix is fixed at the beginning, the activation spikes consistently occur within the prefix.Afterward, we employ key-value (KV) caching mechanism to process the activation spikes in advance.In practice, KV cache is utilized to optimize the decoding speed of causal language models by storing precomputed key and value states of the previous tokens [32, 34] . This approach provides a bypass of the quantization including activation spikes, while preserving the context of prefix through the KV cache. The KV cache for the prefix is precomputed once through the offline inference of LLM without quantization. Then, this KV cache is exploited in the quantization phases, such as calibration or dynamic quantization, even for quantized inference. The process of QFeP is illustrated in Figure 3 .Prefix Search. To form a prefix of explicit activation spike, we first identify candidate token that represent the activation spike at the linear layer with the highest max-median ratio r (m) . For instance, the candidate token can be apostrophe (') token for LLaMA-2-70B model, as highlighted in red in Figure 2 . Once the candidate token is identified, we search the middle context token for between the BOS token and the candidate token in the prefix. This middle context provides dummy context, which is required to activate the candidate token. To find the middle context, we design a template[B, T 1 , C 1 , T 2 , C 2 ]where B, T i , and C i denote the BOS token, context token, and candidate token in the vocabulary V , respectively. Then, we select the context token T where C 1 triggers an activation spikes, while later instance of the same token C 2 does not. When the context token for the activation spikes is varied, we choose the token that maximizes the activation scale ratio between the C 1 and C 2 . Finally, we prepare the KV cache for searched prefix of [B, T, C]. Note that the latter sequence in the template can be replaced with sequences from dataset instead of repetition. with GLU, according to their paper and source code. We recognize recent LLMs, including LLAMA-2-{7B, 13B, 70B} [47] , LLaMA-3-{7B, 70B}, Mistral-7B [20] , Mixtral-8x7B [21] , SOLAR-10.7B [22] , and Gemma-7B [43] , utilize the GLU architecture. The LLMs with original FFN are not covered, as they suffer from the existing outliers rather than activation spikes. All models are sourced from the huggingface-hub2 repository. Quantization. In the experiments, we quantize both the input activations and the weights of linear layers for INT8 matrix multiplication operations. Note that in Table 2 , |M | denotes the total number of linear modules targeted for quantization. In these linear layers, we opt for dynamic per-tensor quantization as the quantization scheme of input activations, and per-channel quantization for weights, respectively. Regarding both input activations and weights, we symmetrically quantize the range using the absolute maximum value as the scale estimation function. For comparison, we use FP16 and per-token activation quantization [55] as baselines. We refer the reader to Appendix B for Batch Matrix-Multiplication (BMM) quantization, which involves quantizing tensors in the self-attention.Evaluations. We evaluate the quantized LLMs with two metrics: zero-shot evaluation accuracy and perplexity. For zero-shot evaluation, we use the four datasets: PIQA [7] , LAMBADA [33] ,HellaSwag [56] , and WinoGrande [38] . We utilize the lm-evaluation-harness library [16] to evaluate zero-shot tasks. To measure perplexity, we use the WikiText-2 [28] dataset. In all cases, we use the[BOS] token as the starting token for each input sequence by default. 
main results: 
 LLaMA-2 Models. We report the evaluation results of quantization on LLaMA-2 models in Table 3 .Compared to FP16 precision, quantizing both weights and activations (W8A8) degrades the overall performance. The results demonstrate that our proposed methods resolve the activation spikes and, surprisingly, restore the performance of the W8A8 close to that of FP16. For example, the LLaMA-2 7B model achieves less than a 1% performance drop from FP16. It is worth noting that the proposed QFeM and QFeP improve at comparable levels. This indicates that the activation spikes present a direct cause of the significant decrease in quantization performance. Because the proposed methods are orthogonal, the performance slightly increases when incorporating both QFeM and QFeP compared to applying them individually.Other GLU-implemented LLMs. For other LLMs that incorporate GLU, we investigated the effectiveness of our methods in mitigating the quantization bottleneck. As can be seen in Figure 5 , our methods consistently remedy the performance drop caused by activation spikes. Noticeably, the Mixtral model demonstrates robustness towards the performance degradation. This indicates that the mixture of experts architecture, which divides the MLP experts by tokens, helps to alleviate the impact of the activation spikes. Meanwhile, addressing the activation spikes is not a sufficient complement for the Gemma model compared to other models. We attribute this to the choice of activation function among GLU variants; specifically, Gemma uses GeGLU, while other models employ SwiGLU.While our method focuses on the activation spikes, the inherent outlier values in the input activations remain. Here, we combine the prior outlier alleviation methods, such as SmoothQuant (SQ) [51] and OutlierSuppressionPlus (OSP) [50] , to further improve the quantization error. In practice, our methods are utilized during the scale calibration phase of alleviation methods to mitigate the impact of activation spikes on scale migration between activations and weights. Table 4 demonstrates the evaluation results of applying the outlier alleviation methods solely and combining them with our methods. We find that there are cases where the alleviation method fails to recover the performance when quantizing the activations with per-tensor scheme. 3 This indicates that alleviating the outlier scales, including the activation spikes, is challenging. With the QFeM, the activation spikes are excluded, and the accurate alleviation is enabled. In addition, the QFeP also benefits from the SQ method, as seen in the case of LLaMA-2 70B. Exceptionally, the OSP successfully addresses the activation spikes in the 13B and 70B cases. For the QFeP, we designed a length-three prefix for the KV cache, including the BOS token, context token, and extra token for activation spike. Because the KV cache consumes the capacity of the pretrained sequence position, it raises a question about the length of the prefix. Therefore, we conduct ablation study for different prefixes for the KV cache. For the prefixes, we prepare random, BOS only, and both QFeP without and with the context token. We illustrate the results of ablation study in Figure 6 . In all cases, the random prefix showcases the lowest performance. While the KV cache with the BOS token demonstrates inconsistent performance, our QFeP LLaMA-2-13B (GPU=RTX4090, token length=2000) consistently shows significant improvement. Importantly, the results imply that the sufficient prefix for the models exhibits differences. However, we emphasize that our KV design for QFeP shows improvements by large margins across all models.The proposed methods require additional resources to evict the activation spikes. Therefore, we analyze the computational costs of the methods and compare them in various schemes. For comparison, we evaluate different activation quantization schemes: dynamic per-token, dynamic per-tensor, and static per-tensor, denoted as AQ1, AQ2, and AQ3, respectively. This distinction establishes strong baselines and demonstrates the potential of the methods. To calibrate the static scales, we estimate the absolute maximum value using the calibration dataset, which is used in Section 3.1.Inference Latency. For each setting, we present the accuracy of the zero-shot tasks and inference latency of the fixed token sequence, as shown in Figure 7 . While the fine-grained scheme (AQ1) shows a negligible accuracy drop, the counterparts (AQ2, AQ3) degrade with the quantization bottleneck.However, by applying our methods, the coarse-grained schemes achieve a competitive performance gain. For example, the combination of AQ2 and QFeM demonstrates the performance close to the AQ1 but with faster latency. The results signify that addressing the quantization bottleneck is important to accelerate the inference latency with coarser granularity. Specifically, the naive static quantization (AQ3), the fastest scheme, exhibits a significant decline. We hope that our work contributes to the future works, which address the remaining challenges in static quantization.Memory Footprint. In Table 5 , we record the maximum memory footprint of our methods. For QFeP, the additional memory is consistently required for the preserved KV cache. However, this memory overhead is much smaller than that used in the fine-grained quantization (AQ1), as QFeM utilizes only three tokens for the cache. Contrary to QFeP, QFeM shows inconsistent memory utilization. For example, the 7B model with QFeM exhibits memory usage similar to AQ2, while the 70B model with QFeM incur additional consumption for a sequence length of 1K. This is attributed to the use of W8A16 for the unquantization modules in QFeM. To tailor the memory usage or inference speed, an alternative strategy can be utilized for QFeM, such as applying fine-grained activation quantization to the unquantization modules instead of using W8A16. 
conclusion: 
 We explore the quantization challenge of GLU activations for modern LLMs. We find that the GLU variants generates excessive activation scales, which cause significant quantization bottlenecks at the specific layers. Based on the systematic generation pattern of the activation spikes, we propose methods that address the spikes in a layer-wise (QFeM) and token-wise manner (QFeP). In the experiments, we confirm that the proposed methods effectively resolve the quantization bottlenecks and result in a large performance gain. We expect that our work sheds light on the potential challenges in future studies regarding quantization and facilitates the development of efficient LLM systems. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide a computational cost analysis in Section 5.5. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the models and dataset used in the experiments (see Section 5.1). 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we provide details of LLMs when performing calibration, which is the step during quantization where the FP16 ranges are computed (Appendix A.1), and additional calibration resultsIn Section 3.1, we have performed the calibration method on various LLMs. We observe the calibration results by categorizing based on the presence of GLU in the LLMs. Table 6 shows the detailed structures of the LLMs. We refer notations for feed-forward implementiation from [40] . In the case of GLU-implemented LLMs, which is LLaMA-2, LLaMA-3, Mistral, Mixtral, SOLAR, StableLM-2, and Gemma, most models have SwiGLU for FFN activation, while only Gemma has GeGLU. On the other hand, in non GLU-implemented LLMs, most of them utilize GeLU for FFN activation, with the exception of OPT, which uses ReLU. Gemma-7B self-attn.q/k/v self-attn.out feed-forward.up/gate feed-forward.down hidden states To achieve faster inference latency, BMM operations in the self-attention also can be computed as INT8 operation [51] . This requires a quantization on the query, key, and value states including the cached context. Because activation spikes produce a large magnitude of latent values, it is important to confirm the extent of quantization errors from KV quantization. This confirmation is necessary to gain advantages from BMM quantization. In Table 7 , we examine the impact of BMM quantization on the W8A8 and QFeM. Regardless of the BMM quantization, the QFeM method consistently improves the quantization bottleneck. For example, the 13B and 70B models maintain their performance, while the 7B model shows a slight decrease. However, this decrease appears to be due to inherent quantization errors rather than a quantization bottleneck from activation spikes. As a result, we confirm that our QFeM method effectively improves the overall performance even in the BMM quantization scenario. In Table 8 , we provide additional results for Section 5.3 with coarse-grained quantization (i.e., per-tensor quantization) scheme for weight quantization. Compared to the results obtained with perchannel weight quantization in Table 4 , these results elucidate the negative impact of activation spikes on the performance of outlier alleviation methods. Furthermore, this suggests that the performance of OSP method resort to the weight quantization scheme. Nevertheless, the proposed methods, QFeM and QFeP, consistently improve the effectiveness of outlier alleviation methods by mitigating the impact of activation spikes. In Figure 11 , we illustrate the Pre-LN transformer architecture and each sub-modules. We highlight with the same color the linear modules that accept identical input activations. Note that the hidden states are normalized before forwarding into the query and up linear modules. We provide additional results for token-level scale analysis (Section 3.2). In Figure 12 and Figure 13 , the token for the activation spikes behind the BOS token does not exhibit the excessive activation scale. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators.Code is available at https://anonymous.4open.science/r/activation-spikes-EDF0.Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS2024). Do not distribute.https://huggingface.co/modelsIn their papers, the activations of LLaMA models are quantized using only a per-token scheme. 


Decision: The reviewers' criticism and recommendations are very consistent. One major issue was the lack of comparison to state-of-the-art LLM and the strong resemblance to a previously researched methods. Apparently, the authors were able to address these concerns in the rebuttal. Finally, all reviewers recommend borderline acceptance. 
After reading the paper I also think that -although the technical innovation of the paper is moderate- it is a good paper or sufficient interest to the NeurIPS community. However, after a discussion with the SAC, I change my recommendation to reject.

Reviews: review: paper_summary: paper_summary: This paper introduces activation quantization methods for GLU-based LLMs, which often face challenges due to activation spikes. To effectively manage these spikes and enable activation quantization using a PTQ-based approach, the paper proposes a Quantization-free Module (QFeM) and a Quantization-free Prefix (QFeP). Specifically, QFeM aims to partially bypass quantization for linear layers where large quantization errors occur. QFeP identifies the prefix that triggers activation spikes and preserves its context as a key-value (KV) cache, preventing the recurrence of activation spikes in subsequent tokens. The paper presents extensive experimental results to compare the accuracy of the quantized models.
strength_weakness: strength_weakness: 1) The perplexity/accuracy results of the baseline methods deviate from the results reported in previous papers.

2) The paper does not compare its method with the state-of-the-art LLM quantization method [1], which enables W4A4 quantization (partially using 8-bit operations) with a PTQ approach.

[1] Zhao, Yilong, et al. "Atom: Low-bit quantization for efficient and accurate llm serving." Proceedings of Machine Learning and Systems 6 (2024): 196-209.
questions: questions: 1) The perplexity and accuracy results of the baseline methods (SQ [1] and OSP [2]) in Table 4 are worse than the figures reported in the original SQ and OSP papers. For instance, the SQ paper reported successful preservation of LLM perplexity after W8A8 quantization (Table 7 of [1]), but Table 4 of this paper shows poor perplexity and accuracy results for SQ. Additionally, the OSP paper claimed successful preservation of LLM perplexity even after INT6 quantization and reported better perplexity results compared to SQ (Table 2 of [2]). Although OSP only evaluated LLaMA-1 and there is no LLaMA-2 data, we can reasonably expect similar trends for LLaMA-2 given that both use GLU-based activation functions. However, Table 4 of this paper shows poor perplexity and accuracy results for OSP, particularly for LLaMA-2-7B. Why are the evaluation results for the previous methods so different?

2) Since the prefix token is retrieved from the calibration set and the threshold alpha is determined from it, will these parameters remain consistent when inferring on a new dataset using the same model?

3) What are the advantages of the proposed method compared to Atom [3] in terms of perplexity/accuracy, latency, or other aspects?

[1] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." International Conference on Machine Learning. 2023.

[2] Wei, Xiuying, et al. "Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling." arXiv preprint arXiv:2304.09145 (2023).

[3] Zhao, Yilong, et al. "Atom: Low-bit quantization for efficient and accurate llm serving." Proceedings of Machine Learning and Systems 6 (2024): 196-209.
limitations: limitations: The proposed method is limited to GLU-based LLMs.
paper_summary: paper_summary: This paper addresses the precision challenges posed by the large language models (LLMs) quantization during inference, specifically focusing on the quantization errors in GLU-based feedforward networks. The authors identify that GLU variants in LLMs cause significant local quantization errors due to excessive activation magnitudes, referred to as activation spikes. They observe that GLU-implemented models have larger spikes than non-GLU-implemented models.  They propose two methods, Quantization-free Module (QFeM) and Quantization-free Prefix (QFeP), to isolate and mitigate these spikes during quantization. QFeM leave some linear layers unquantized (usually those layers that cause large activation spikes in the first several layers), and QFeP introduce an additional prefix before the inference process. Their extensive experiments show that these methods improve quantization performance and are compatible with existing techniques.
strength_weakness: strength_weakness: 1. My major concern is about the baseline of SmoothQuant reported in Table 4. For example, In Table 7 of SmoothQuant's original paper, they report that W8A8 SQ's PPL of Llama-7B on WikiText-2 dataset is 5.515, while the authors report a PPL of 9.907 on the same dataset. Is there a specific reason about this large gap?

2. In Table 3, the improvement brought by the QFeP method does not seem significant, especially when combining with the QFeM method.
questions: questions: 1. In the method of Quantization-free Prefix (QFeP), will the additional prefix decrease or increase the accuracy? That means you use the QFeP method but do not apply quantization, only introduce the additional prefix.

2. Where do the spikes of the GLU-based model come from in the element-wise multiplication? Do they mostly comes from the gate projection part, or the up projection part?
limitations: limitations: None
paper_summary: paper_summary: This paper identifies some of the underlying causes for why activation quantization (PTQ) could lead to low performance and suggests some methods to address these issues.
strength_weakness: strength_weakness: Please see the “Questions” section.
questions: questions: My review is as follows:

1) The results of Table 2 seem to suggest that SmoothQuant leads to an unacceptably high performance degradation. Table 6 of the SmoothQuant paper however shows that for Llamav1, the degradation is very small. I wonder if a difference in implementation is causing this discrepancy in results. I’m sorry if this is mentioned somewhere in the text and I missed it. Could you please clarify?

2) What is the latency for FP16 in Figure 7?

3) There are multiple recent works that suggest quantizing the weights to 4 bits can be done without a big impact to the accuracy. Since memory access (for weights) is typically the bottleneck when LLMs are deployed, it is possible that one may prefer W4A16 over W8A8. In my opinion, studying 4-bit weight quantization (in addition to the already studied W8 quantization) could make the paper more interesting.

4) QFeM method is essentially a mixed precision approach. A lot of quantization papers actually employ some form of mixed precision (even though it is sometimes only mentioned in the footnote). For instance, SmoothQuant uses FP16 for LayerNorm. Has this been taken into account when comparing against other methods? Also, have similar assumptions been made in the implementation of the proposed method?

Minor:

5) There is a type on line 90: “in the remain(der) of this 91 paper”
6) Typo in line 147 “dominate(s)”

Readability and presentation: The paper is mostly easy to understand. One thing I could say is that the idea of QFeM is easier to follow than QFeP; the introduction of the QFeP method may need to be improved and made more concise.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper pays attention to extremely large outliers in LLMs and further investigates the reasons behind these "attention spikes." Consequently, the authors propose two methods to enhance the performance of quantized models.
strength_weakness: strength_weakness: 1. The proposed QFeM method is not hardware-friendly, as it maintains some modules at high precision and cannot directly utilize low-bit INT General Matrix Multiply (GEMM) for activations and weights.

2. The proposed QFeP method bears a strong resemblance to a previously researched method, IntactKV[1], yet lacks a detailed comparative discussion.

3. The experimental settings are limited to W8A8 configurations, which previous research, such as SmoothQuant[2], has shown can nearly achieve lossless quantization for W8A8 models.

4. The authors have not included comparisons with state-of-the-art baselines, such as OmniQuant[3], AffineQuant[4], QLLM[5], and QuaRot[6].


[1]. Liu, Ruikang, et al. "IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact." arXiv preprint arXiv:2403.01241 (2024).

[2]. Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." International Conference on Machine Learning. PMLR, 2023.

[3]. Shao, Wenqi, et al. "Omniquant: Omnidirectionally calibrated quantization for large language models." arXiv preprint arXiv:2308.13137 (2023).

[4]. Ma, Yuexiao, et al. "Affinequant: Affine transformation quantization for large language models." arXiv preprint arXiv:2403.12544 (2024).

[5]. Liu, Jing, et al. "Qllm: Accurate and efficient low-bitwidth quantization for large language models." arXiv preprint arXiv:2310.08041 (2023).

[6]. Ashkboos, Saleh, et al. "Quarot: Outlier-free 4-bit inference in rotated llms." arXiv preprint arXiv:2404.00456 (2024).
questions: questions: 1. Could you provide a detailed analysis highlighting the differences between the QFeP method and IntactKV?

2. Could you expand the experimental results to include different quantization settings such as W4A4 and W4A8?

3. Could you offer a more detailed comparison with state-of-the-art (SOTA) baselines or conduct the ablation tests as outlined in Table 4?

If the authors can provide more comprehensive results, I am prepared to raise my evaluation scores.
limitations: limitations: Yes.

Comments: title: Minor questions about rebuttal
comment: We greatly appreciate your time and effort in reviewing our rebuttal. Your constructive questions have been instrumental in enhancing the quality of our work.
Thank you for your response and the additional experiments. You have addressed many of the questions I had, so I have decided to increase my score.
Thank you for your insightful reviews of our work and rebuttal during the review period. Your valuable feedback is greatly appreciated and will be incorporated into the final version.
Thanks for your detailed response and additional experiments. Some of my concerns are addressed and I have decided to raise my score.
We greatly appreciate your constructive response to our rebuttal! We are highly encouraged by your thoughtful feedback. Thank you.
Thank you for the detailed responses! It would be great the findings reported in the rebuttal could be explicitly incorporated in the revision. I'll increase my rating from 4 to 5 since some of my concerns regarding the numerical results are already addressed.
We are sincerely grateful for your effort and time in reviewing our rebuttal and for your valuable questions.

---

**Q5.** IntactKV pre-saves all system prompts as pivot tokens for Vicuna models, which includes all attention spikes, similar to QFeP.

**A5.**
The discussion continues from our previous response to your Q2, which highlighted the differences between QFeP and IntactKV [1].

Notably, IntactKV selects different pivot tokens depending on the type of LLMs.
- For pre-trained LLMs (e.g., LLaMA), IntactKV selects only the [BOS] token as a pivot token.
- For supervised fine-tuned LLMs (e.g., Vicuna), IntactKV stores all system prompts as pivot tokens.

We denote these methods as IntactKV[B] and IntactKV[P], respectively, according to [1].

In our previous rebuttal, we compared QFeP with IntactKV[B] due to the absence of system prompts for pre-trained LLMs.
When quantizing activations for pre-trained LLMs, QFeP's ability to address activation spikes more effectively than IntactKV becomes evident, as discussed in the previous response.

Nevertheless, further investigation into activation spikes for supervised fine-tuned (or instruction fine-tuned) LLMs would be valuable.
Previous works have found that pivot tokens persist after instruction fine-tuning [1, 2].
This finding implies that activation spikes are transferable and that our methods are also effective.
Indeed, we observed equivalent activation spikes in both LLaMA-2 and its fine-tuned models, such as Vicuna-v1.5 and LLaMA-2-Chat.
While IntactKV[P] may store long system prompts for pivot tokens, QFeP saves a more compact prefix with a length of 3.
Furthermore, QFeP is explicit and prompt-agnostic, providing flexibility for system prompts.
Finally, **QFeP can be applicable to more generalized and effective solutions for addressing activation spikes when applying activation quantization, regardless of whether the LLM is pre-trained or fine-tuned.**


[1] Liu, Ruikang, et al. "IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact." arXiv preprint arXiv:2403.01241 (2024).

[2] Sun, Mingjie, et al. "Massive activations in large language models." arXiv preprint arXiv:2402.17762 (2024).

---

**Q6.** I am curious as to why QFeP does not improve the performance of the 4-bit LLaMA2-13B model in Appendix 3. Could this be due to the influence of an additional prefix affecting the model's performance?

**A6.**
During the preparation of the rebuttal to reviewer geCr, we confirmed that introducing only an additional prefix of QFeP slightly degrades the performance of the FP16 model.

Based on this observation, we hypothesize the following regarding QFeP's contribution to the 4-bit LLaMA-2-13B:

- Case 1 (Atom [3] + QFeP): Given that quantization errors have been minimized via the fine-grained group quantization scheme of Atom, the prepended prefix of QFeP has the possibility to slightly degrade performance. Note that the fine-grained group quantization utilizes a finer granularity than per-token quantization.
- Case 2 (OmniQuant [4] + QFeP): At present, we are unable to determine the specific factor that degrades the perplexity for WikiText-2, although QFeP achieved a performance gain for C4.
To provide further evaluation results, we assessed four zero-shot tasks, as shown in the table below.
The results indicate that QFeP improves the OmniQuant baseline for three tasks, specifically the WinoGrande task.
This is similar to the results of W4A6 LLaMA-2-13B in Table 2 of the attached PDF.

While there are numerous potential factors that could explain the degradation (e.g., trainable components of OmniQuant or softmax quantization in low-bit), we are making an effort to include an ablation study, similar to Figure 6, to clarify the influence of an additional prefix. Thanks again for your questions.

| Model | #Bits | Method | PIQA($\uparrow$) | LAMBADA($\uparrow$) | HellaSwag($\uparrow$) | WinoGrande($\uparrow$) | Avg($\uparrow$) |
|---|---|---|:---:|:---:|:---:|:---:|:---:|
|LLaMA-2-13B|FP16| - | 79.49% | 76.54% | 60.20% | 72.38% | 72.15% |
|LLaMA-2-13B|W4A4|Atom|78.02%|75.80%|58.48%|70.56%|70.72%|
|LLaMA-2-13B|W4A4|$\quad$+QFeM|78.07%|75.43%|**58.49%**|70.40%|70.60%|
|LLaMA-2-13B|W4A4|$\quad$+QFeP|76.99%|**76.19%**|58.22%|70.80%|70.55%|
|LLaMA-2-13B|W4A4|$\quad$+QFeM+QFeP|**78.24%**|75.61%|58.34%|**71.11%**|**70.83%**|
|LLaMA-2-13B|W4A4|OmniQuant|67.25%|41.63%|44.95%|52.57%|51.60%|
|LLaMA-2-13B|W4A4|$\quad$+QFeM|**71.11%**|**48.01%**|46.86%|57.14%|**55.78%**|
|LLaMA-2-13B|W4A4|$\quad$+QFeP|68.93%|38.99%|45.95%|**58.48%**|53.09%|
|LLaMA-2-13B|W4A4|$\quad$+QFeM+QFeP|70.02%|44.89%|**46.93%**|57.46%|54.82%|

[3] Zhao, Yilong, et al. "Atom: Low-bit quantization for efficient and accurate llm serving." Proceedings of Machine Learning and Systems 6 (2024): 196-209.

[4] Shao, Wenqi, et al. "Omniquant: Omnidirectionally calibrated quantization for large language models." arXiv preprint arXiv:2308.13137 (2023).
I appreciate the authors' discussion and the additional evaluation results provided. However, I have a few minor points that require further clarification. Firstly, IntactKV pre-saves all system prompts as pivot tokens for Vicuna models, which includes all attention spikes, similar to QFeP. Secondly, I am curious as to why QFeP does not improve the performance of the 4-bit LLaMA2-13B model in Appendix 3. Could this be due to the influence of an additional prefix affecting the model's performance?
We appreciate your review of our rebuttal. We're pleased that our results addressed your concerns and improved your assessment of our work.
The results are very interesting.  I increase my score accordingly.