## 1 Introduction  
Large language models (LLMs) have become a key paradigm in natural language processing, accelerating the release of variations within the community [1, 2]. Furthermore, the latest LLMs establish state-of-the-art performance by training with increased scale, as well as by adopting architectural improvements such as GLU [3], RoPE [4], GQA [5], and MoE [6]. Especially, GLU (Gated Linear Unit) variants (e.g., SwiGLU, GeGLU), which modulate hidden representations through element-wise gating between parallel linear transformations, have been adopted in most modern LLM architectures (e.g., LLaMA family [7]) due to their training efficiency [3, 8]. Although LLMs broaden foundational capabilities in natural language tasks and potential for various applications, billions of parameters in the large models impose considerable computational costs on end users in practice. To reduce GPU memory requirements and accelerate inference speed, post-training quantization (PTQ), which reduces bit-widths of weights and activations after training without updating parameters, offers an affordable solution by quantizing weights and activations into a lower precision (e.g., INT8) without a need for expensive retrainingsteps [9; 10; 11]. However, recent studies have revealed that large magnitude values at certain coordinates exist in the activations of LLMs, which are often called outliers, posing a key challenge in activation quantization [12; 13; 14; 15]. Another line of works attempts to explain the role of outlier values in the attention mechanism [16; 17]. In this paper, we present our discovery that the GLU architecture in the feed-forward network (FFN) generates excessively large activation values, which are responsible for significant local quantization errors. Specifically, we observe that these problematic activation values occur in specific linear modules and are dedicated to a couple of tokens, which will be discussed in Section 3. To distinguish the excessive GLU activations from the outliers, we refer to them as activation spikes. In light of our observations, we propose two empirical methods to mitigate the impact of activation spikes on quantization: Quantization-free Module (QFeM) and Quantization-free Prefix (QFeP). QFeM aims to partially exclude quantization for linear layers (or modules) where large quantization errors occur, instead of quantizing the entire linear modules in the LLM. By scoring the extent of scale disparity, QFeM selects linear modules to exclude. On the other hand, QFeP identifies the prefix that triggers activation spikes and stores its context using a key-value (KV) cache. This cache mechanism retains previously computed attention keys and values, allowing the model to bypass redundant computations during decoding. It is noteworthy that both QFeM and QFeP rely on calibration results to capture activation spikes in advance without any modifications to the target LLM. Thus, our methods can be integrated into any existing quantization methods such as [PERSON] et al. [13], [PERSON] et al. [15]. In our comprehensive experiments, we demonstrated that recently released LLMs incorporating GLU variants struggled with activation spikes when applying activation quantization. Consequently, the proposed methods, QFeM and QFeP, substantially enhance the performance of the primitive quantization method, the round-to-nearest (RTN) method, which quantizes activations by rounding each value to the closest discrete level based on fixed scaling. Furthermore, we observe that current outlier alleviation methods [13; 15] and state-of-the-art quantization methods [18; 19] are exposed to the activation spikes and benefit from our proposed methods. Compared to the strong baseline of fine-grained activation quantization [20], our methods show competitive performance, achieving reduced latency and memory footprint.