## 3 Activation Spikes: Excessive Magnitude of GLU Activations
### Effect of Quantization on Activation Spikes  
We explore the impact of local quantization errors caused by activation spikes on LLM outputs. To identify the layers where activation spikes occur, we utilize a ratio between the maximum and median values of the token-wise input activation scales instead of using the maximum scale value alone. The max-median ratio for linear layer \(m\) can be formulated as \(r^{(m)}=\frac{\max(\mathbf{S}^{(m)})}{\text{median}(\mathbf{S}^{(m)})}\), where \(\mathbf{S}^{(m)}\) represents the token-wise input activation scales coming to module \(m\in M\). This max-median ratio captures the extent to which the maximum scale dominates the other token scales. For comparison, we chose the activation quantization targets as the top-4, middle-4, and bottom-4 modules based on the max-median ratio in descending order. Then, we evaluated the perplexity and mean-squared error (MSE) using the calibration dataset. Here, the MSE was calculated for the last hidden states between the original (FP16) and partially quantized LLM. As shown in Table 1, quantization on the top-4-rated modules solely degrades the LLM performance by significant margins, while the other cases exhibit negligible performance changes. We consider these quantization-sensitive input activations (_inter alia_ activation spikes) to be the quantization bottleneck, which, in this paper, refers to the quantization error caused by outliers.  
Furthermore, the activation spikes are conditioned on the specific context of the input sequence as discussed in Section 3.2. Altogether, to address the influence of activation spikes--which vary across layers due to the GLU architecture--we propose in the following sections a layer-wise selective quantization method and a prefix-based approach for mitigating quantization errors.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{5}{c}{**Perplexity (\(\downarrow\))**} & \multicolumn{5}{c}{**MSE (\(\downarrow\))**} \\ \cline{2-9}  & **FP16** & **Top 4** & **Middle 4** & **Bottom 4** & **Top 4** & **Middle 4** & **Bottom 4** \\ \hline LLaMA-2-7B & 7.37 & **11.77** & 7.38 & 7.40 & **1908.80** & 1.03 & 12.90 \\ LLaMA-2-13B & 6.84 & **15.09** & 6.84 & 6.84 & **4762.11** & 0.91 & 10.38 \\ Mistral-7B & 8.35 & **69.45** & 8.35 & 8.36 & **218.60** & 0.02 & 0.18 \\ Gemma-7B & 10.85 & **85.83** & 10.94 & 10.87 & **213.93** & 1.60 & 1.07 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Perplexity and MSE of partial activation quantization of LLMs. Quantization is applied to the Top 4, Middle 4, and Bottom 4 layers, selected based on the max-to-median ratio of token-wise activation scales. Both perplexity and MSE are measured using the calibration dataset. MSE is computed between the last hidden states of the FP16 and partially quantized models.  
Figure 2: Token-wise scales in a specific layer with an activation spike. When quantizing the input activations using a per-tensor scale, the scale of the activation spike dominates the scales of the other tokens: token-wise scales in a specific layer with an activation spike. The y-axis represents the activation magnitude, and the x-axis corresponds to the input sequence tokens. The background color of each token indicates its role: yellow for general tokens, red for the first occurrence of a token that causes an activation spike, and green for a repeated occurrence of the same token that does not trigger a spike. For more examples, see Appendix C.