## 6 Conclusions  
We explored the quantization challenge of GLU activations for modern LLMs. We found that the GLU variants generate excessive activation scales, which cause significant quantization bottlenecks at the specific layers. Based on the systematic generation pattern of the activation spikes, we propose methods that address the spikes in a layer-wise (QFeM) and token-wise manner (QFeP). Our experimental results demonstrate significant improvements, showing approximately a 7-16% increased accuracy in quantized LLaMA-2 models compared to traditional quantization methods. Additionally, our methods achieved a substantial reduction in perplexity--up to 85.4% in LLaMA-2-13B, and over 50% in LLaMA-2-7B--closely approaching full-precision (FP16) performance. Furthermore, our analysis indicates that QFeM optimizes computational efficiency, reducing latency by up to 10.4% in LLaMA-2-13B, by selectively avoiding quantization in layers severely impacted by activation spikes, effectively balancing latency-performance trade-offs. Similarly, QFeP minimizes memory overhead, increasing memory usage by less than 0.05% in LLaMA-2-7B for 2 K tokens, while efficiently managing problematic activations through KV caching mechanisms. We expect that our work sheds light on the potential challenges in future studies on quantization and facilitates the development of efficient LLM systems.  
Conceptualization, J.Y.; methodology, J.Y.; software, J.Y. and H.K.; validation, J.Y. and H.K.; formal analysis, J.Y.; investigation, H.K.; resources, H.K.; data curation, H.K.; writing--original draft preparation, J.Y.; writing--review and editing, H.K., J.J. and Y.K.; visualization, H.K.; supervision, Y.K.; project administration, Y.K.; funding acquisition, Y.K. All authors have read and agreed to the published version of the manuscript.  
\begin{table}
\begin{tabular}{l c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**SeqLen**} \\ \cline{2-3}  & **1 K** & **2 K** \\ \hline \hline \multicolumn{3}{c}{LLaMA-2-7B} \\ \hline AQ1 & 8185 MiB & 9516 MiB \\ AQ2 & 8148 MiB & 9474 MiB \\ +QFeP & 8149 MiB & 9478 MiB \\ +QFeM & 8148 MiB & 9474 MiB \\ \hline \hline \multicolumn{3}{c}{LLaMA-2-70B} \\ \hline AQ1 & 67,756 MiB & 69,037 MiB \\ AQ2 & 67,648 MiB & 68,820 MiB \\ +QFeP & 67,651 MiB & 68,822 MiB \\ +QFeM & 67,838 MiB & 68,819 MiB \\ \hline \hline \end{tabular}
\end{table}
Table 5: Memory footprint (in MiB) for different quantization methods across sequence lengths of 1 K and 2 K. Results are shown for both LLaMA-2-7B and LLaMA-2-70B models.  
**Funding:** This research was financially supported by the Ministry of Trade, Industry and Energy (MOTIE) and Korea Institute for Advancement of Technology (KIAT) through the International Cooperative R&D program (Project No. P0025661). This work was also supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-00155885, Artificial Intelligence Convergence Innovation Human Resources Development (Hanyang University ERICA)).  
**Data Availability Statement:** The dataset and source code utilized in this study have been made publicly available in the GitHub repository ([[https://github.com/onnoo/activation-spikes](https://github.com/onnoo/activation-spikes)]([https://github.com/onnoo/activation-spikes](https://github.com/onnoo/activation-spikes))) and can be accessed at any time. (We last accessed on 17 April 2025).  
**Acknowledgments:** We sincerely thank the anonymous reviewers for their insightful and constructive comments, which greatly contributed to enhancing the content and quality of this paper.  
**Conflicts of Interest:** The authors declare no conflicts of interest.