## 5 Experiments
### Main Results
#### 5.2.2 Other GLU-Implemented LLMs  
For other LLMs that incorporate GLU, we investigated the effectiveness of our methods in mitigating the quantization bottleneck. As can be seen in Figure 5, our methods consistently remedy the performance drop caused by activation spikes. Noticeably, the Mixtral model demonstrates robustness towards the performance degradation. This indicates that the mixture of experts architecture, which divides the MLP experts by tokens, helps to alleviate the impact of the activation spikes. Meanwhile, addressing the activation spikes is not a sufficient complement for the Gemma model compared to other models. We attribute this to the choice of activation function among GLU variants; specifically, Gemma uses GeGLU, while other models employ SwiGLU.  
Figure 5: The average accuracy of a zero-shot evaluation on other GLU-implemented LLMs. Most models recover significantly compared to W8A8, with performance close to FP16. The x-axis represents bar plots for each method: W8A8, QFeM, QFeP, and QFeM + QFeP. The y-axis indicates the zero-shot average accuracy across evaluation tasks. The red dashed line shows the performance of the original FP16 model for reference.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & **WikiText-2** & **PIQA** & **LAMBADA** & **HellaSwag** & **WinoGrande** & **Avg** \\  & **(ppl \(\downarrow\))** & **(acc \(\uparrow\))** & **(acc \(\uparrow\))** & **(acc \(\uparrow\))** & **(acc \(\uparrow\))** & **(acc \(\uparrow\))** \\ \hline \hline \multicolumn{7}{c}{ILaMA-2-7B} \\ \hline FP16 & 5.268 & 78.18\% & 73.67\% & 57.13\% & 69.46\% & 69.61\% \\ \hline W8A8 & 8.634 & 72.80\% & 62.27\% & 49.57\% & 63.69\% & 62.08\% \\ \hline +QFeM & 5.758 [\(-\)**2.876**] & 78.02\% & 73.86\% & 56.32\% & 68.35\% & 69.14\% [**4.06**] \\ +QFeP & 5.758 [\(-\)**2.876**] & 76.44\% & 73.57\% & 55.55\% & 69.22\% & 68.69\% [**46.61**] \\ +QFeM+QFeP & 5.573 [\(-\)**3.061**] & 77.86\% & 74.58\% & 56.05\% & 69.38\% & 69.47\% [**4.73**] \\ \hline \hline \multicolumn{7}{c}{ILaMA-2-13B} \\ \hline FP16 & 4.789 & 79.49\% & 76.54\% & 60.20\% & 72.38\% & 72.15\% \\ \hline W8A8 & 34.089 & 70.13\% & 49.66\% & 42.65\% & 58.72\% & 55.29\% \\ +QFeM & 5.241 [\(-\)**28.848**] & 77.58\% & 75.68\% & 59.13\% & 72.61\% & 71.25\% [**41.96**] \\ +QFeP & 6.000 [\(-\)**28.089**] & 77.53\% & 73.94\% & 57.23\% & 70.96\% & 69.91\% [**41.62**] \\ +QFeM+QFeP & 5.126 [\(-\)**28.963**] & 78.51\% & 75.86\% & 59.44\% & 72.61\% & 71.61\% [**41.32**] \\ \hline \hline \multicolumn{7}{c}{ILaMA-2-70B} \\ \hline FP16 & 3.218 & 81.45\% & 79.45\% & 65.29\% & 80.43\% & 76.65\% \\ \hline W8A8 & 8.055 & 74.05\% & 70.27\% & 55.21\% & 67.96\% & 66.87\% \\ +QFeM & 3.830 [\(-\)**4.225**] & 81.23\% & 77.66\% & 64.15\% & 78.14\% & 75.30\% [**+8.43**] \\ +QFeP & 6.007 [\(-\)**2.048**] & 77.64\% & 73.26\% & 63.40\% & 76.16\% & 72.62\% [**+5.75**] \\ +QFeM+QFeP & 3.708 [\(-\)**4.347**] & 81.23\% & 77.82\% & 64.65\% & 77.11\% & 75.20\% [**+8.33**] \\ \hline \hline \end{tabular}
\end{table}
Table 3: Perplexity and zero-shot evaluation for the quantization on ILaMA-2 models. FP16 denotes the original model precision, and W8A8 denotes the model quantized to INT8 for both weights and activations. Colored cells indicate results obtained by applying our proposed methods to existing quantization approaches. Column headers represent different benchmarks, with the values in parentheses denoting ppl (perplexity) and acl (accuracy), respectively. Downward arrows (\(\downarrow\)) indicate that lower values are better (e.g., for perplexity), while upward arrows (\(\uparrow\)) indicate that higher values are better (e.g., for accuracy). Bold text in the main table highlights the improvements over the baseline quantization method (W8A8 without our method).