Automated Fine-Grained Mixture-of-Experts Quantization
Authors: Zhanhao Xie, Yuexiao Ma, Xiawu Zheng, Fei Chao, Rongrong Ji, Wanchen Sui, Shen Li, Yong Li
Accepted at: acl25

Automated Fine-Grained Mixture-of-Experts Quantization

Zhanhao Xie1, Yuexiao Ma1, Xiawu Zheng1, Fei Chao1, Wanchen Sui2, Yong Li2, Shen Li2, Rongrong Ji1*
1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China
2Alibaba Cloud Computing

Abstract
The Mixture of Experts (MoE) architecture enables efficient model scaling through conditional computation, where only subset of parameters are activated per input. However, this distributed architecture poses unprecedented challenges for model compression, as conventional quantization methods optimized for dense networks prove inadequate. This paper introduces a specialized quantization framework for MoE architectures, motivated by our discovery that weight matrices across expert networks exhibit distinctive channel-wise outlier distributions, necessitating a more nuanced compression approach. Through theoretical analysis incorporating Fisher Information matrices and condition number characteristics, we establish a fundamental relationship between layer functionality and quantization sensitivity, demonstrating that down-projection layers inherently demand higher precision compared to up-projection layers. Leveraging these insights, we develop an automated channel-wise quantization framework that dynamically determines optimal bit-width allocations while maintaining minimal computational overhead through efficient statistical approximations. When evaluated on the Mixtral-8x7b-v0.1 architecture, our methodology demonstrates a 3.96% improvement over existing state-of-the-art approaches across natural language understanding benchmarks, while achieving superior compression ratios. Code is available at: https://github.com/ceiling4/Fine-Grained-MoE-Quantization

1 Introduction
In recent years, Large Language Models (LLMs) (Touvron et al., 2023a,b; Reid et al., 2024; Zhang et al., 2022; Yang et al., 2024) have demonstrated unprecedented advancements in natural language processing. However, this remarkable progress has been accompanied by an exponential increase in computational resource requirements. The Mixture of Experts (MoE) architecture (Jacobs et al., 1991; Fedus et al., 2022; Lepikhin et al., 2020) has emerged as a transformative paradigm in large language models, introducing efficient parameter scaling through selective activation of context-relevant parameter subsets. Through strategic partitioning of the parameter space into specialized expert networks coupled with dynamic routing mechanisms, MoE models achieve substantial performance gains while maintaining modest training costs and introducing only marginal computational overhead during inference (Jiang et al., 2024; Dai et al., 2024; Liu et al., 2024b). This innovative approach enables the processing of increasingly sophisticated tasks while optimizing resource utilization, marking a significant milestone in scaling language model capabilities (Rajbhandari et al., 2022; Chen et al., 2022).

The exponential growth of language models has heightened the critical importance of model compression. Various compression techniques, including quantization (Frantar et al., 2023; Yuan et al., 2023; Shao et al., 2023; Ma et al., 2024; Liu et al., 2024c), sparsification (Frantar and Alistarh, 2023;

![Comparison of different outlier metrics (W_{up}, W_{gate}, and W_{down}) across layers and experts in Mixtral-8x7B and DeepSeek-V2-Lite, revealing consistently elevated values in down-projection metrics across MoE architectures.](page_1012_370_377_246.png)

Figure 1: Comparison of different outlier metrics (\( W_{up} \), \( W_{gate} \), and \( W_{down} \)) across layers and experts in Mixtral-8x7B and DeepSeek-V2-Lite, revealing consistently elevated values in down-projection metrics across MoE architectures.
Sun et al., 2023), and knowledge distillation (Hsieh et al., 2023; Gu et al., 2024), have demonstrated remarkable efficacy in reducing model footprint while preserving performance. Among these approaches, quantization has emerged as particularly promising, with recent advances successfully addressing the challenging outlier problem in large dense models through sophisticated weight compensation and affine transformation techniques.

However, these quantization methodologies, originally designed for dense architectures, encounter unique challenges when applied to MoE-LLM architectures. The fundamental distinction lies in the dynamic routing mechanism (Shazeer et al., 2017), where input tokens are intelligently directed to specific experts based on semantic content. This sophisticated routing mechanism, combined with sparse computational patterns, renders conventional quantization techniques inadequate without compromising model performance (Li et al., 2023). Moreover, the inherent heterogeneity of expert structures indicates that individual experts within the MoE framework possess distinct parameters and specialized functionalities. Unlike traditional homogeneous neural networks, this architectural diversity suggests that each expert requires a tailored compression strategy, introducing additional complexity to the quantization process. The distinctive characteristics of MoE architectures present unprecedented quantization challenges, particularly in managing outliers effectively across heterogeneous expert networks. Recent research has begun addressing these challenges in MoE compression. (He et al., 2024) introduced a unified compression framework integrating expert slimming with quantization, marking an initial step toward comprehensive MoE compression. Additionally, (Li et al., 2024) investigated variable quantization bit-width allocation based on importance metrics across different structural granularities. However, these approaches predominantly focus on static structural metrics, without fully addressing the unique characteristics of MoE architectures. The field currently lacks a specialized quantization solution engineered specifically for the distinctive nature of MoE models, particularly in effectively managing heterogeneous expert structures.

In response to the unprecedented challenges inherent in quantizing MoE models ((Jiang et al., 2024; Li et al., 2024)), we present a novel automated mixed-precision quantization framework specifically designed for MoE-based Large Language Models. Our approach is fundamentally motivated by a critical empirical observation: weight matrices within MoE architectures exhibit distinctive channel-wise outlier patterns, with substantial heterogeneity across different channels within individual experts. As illustrated in Figure 1 and 2, this phenomenon is consistently observed across diverse MoE architectures, with particularly pronounced patterns in down-projection layers. These findings fundamentally challenge conventional layer-wise quantization paradigms and underscore the necessity for more granular quantization strategies. Through comprehensive theoretical analysis, we establish that various components within MoE architectures demonstrate differential susceptibility to quantization effects. Specifically, we prove that down-projection layers exhibit inherently greater sensitivity to quantization perturbations compared to their up-projection counterparts, as evidenced by their elevated condition numbers and Fisher Information metrics. This theoretical foundation provides principled guidance for optimal bit allocation across the model’s architectural components. To reconcile these theoretical insights with practical deployment constraints, we introduce computationally efficient statistical approximation techniques that facilitate automated bit allocation without introducing additional training complexity. Our methodology maintains rigorous theoretical guarantees while achieving the computational efficiency necessary for large-scale model deployment. Experimental results demonstrate significant improvements over existing approaches, achieving a 3.96% performance gain on the Mixtral-8x7B model under extreme low-bit configurations.

The primary contributions of this work are:

• We elucidate and systematically characterize the channel-wise heterogeneity in MoE weight distributions, revealing distinctive patterns that necessitate the development of fine-grained quantization methodologies.

• We establish a comprehensive theoretical framework that quantifies and explicates layer-wise sensitivity variations in MoE architectures, providing mathematical evidence for the increased quantization precision requirements in down-projection layers.

• We propose an integrated quantization framework that operationalizes these theoretical insights, automatically determining optimal bit-
width allocations while maintaining minimal computational overhead through sophisticated statistical approximations.

• We empirically validate our approach through extensive experimentation, demonstrating superior compression ratios while maintaining model performance across multiple MoE architectures.

2 Related Work

2.1 Mixture-of-Experts Models

Mixture-of-Experts (MoE) architectures have emerged as a significant innovation in Large Language Models (LLMs) (Jiang et al., 2024; Dai et al., 2024; Liu et al., 2024a), offering a balance between model capacity and computational efficiency. Originally proposed by (Jacobs et al., 1991), MoE has evolved substantially through key developments like (Shazeer et al., 2017)’s application to transformer models and (Fedus et al., 2022)’s introduction of sparse gating mechanisms for efficient routing. The theoretical foundations have been strengthened by works such as (Chen et al., 2022; Chowdhery et al., 2022), while recent advances like the Mixtral model (Jiang et al., 2024) have demonstrated that MoE can match full-parameter LLM performance while using significantly fewer active parameters. These developments have sparked growing interest in MoE optimization and compression techniques (Li et al., 2023; Rajbhandari et al., 2022), highlighting the architecture’s potential for efficient, large-scale language modeling.

2.2 Post-Training Quantization

Post-training quantization (PTQ) (Wei et al., 2022b; Yao et al., 2022; Ashkboos et al., 2024; Liu et al., 2024c; Sun et al., 2024) has emerged as an efficient technique for model compression, particularly beneficial for LLMs. Unlike quantization-aware training or fine-tuning (Tailor et al., 2020; Ding et al., 2022), PTQ operates on pre-trained models without extensive retraining (Liu et al., 2021; Fang et al., 2020). In computer vision, AdaRound (Nagel et al., 2020) optimizes weight rounding strategies, BRECQ (Li et al., 2021) introduces block-wise reconstruction, and QDROP (Wei et al., 2022a) enhances performance through activation substitution. For LLMs, GPTQ(Frantar et al., 2023) uses approximate second-order information for layerwise quantization, SmoothQuant (Xiao et al., 2023) tackles activation outliers, and AWQ (Lin et al., 2023) preserves critical weights’ precision. OmniQuant (Shao et al., 2023) integrates multiple strategies, combining mixed-precision quantization, outlier handling, and adaptive rounding. AffineQuant (Ma et al., 2024) introduces an affine transformation to adjust weight distribution, effectively reducing quantization errors. These advancements have significantly improved LLM deployment efficiency on resource-constrained devices (Kim et al., 2023; Chen et al., 2024).

3 Preliminary

Mixture of Experts. The Mixture-of-Experts (MoE) architecture represents a significant advancement in Transformer models (Fedus et al., 2022; Jiang et al., 2024), superseding the conventional Feed-Forward Network (FFN) sublayer with a more sophisticated dynamic structure. An MoE layer incorporates \( N \) parallel FFN modules, designated as experts \( E_1, E_2, \ldots, E_N \), alongside a router network (Shazeer et al., 2017) that facilitates dynamic input allocation through a specialized gating mechanism.

For an input vector \( x \in \mathbb{R}^{d_{in}} \), the MoE layer generates output \( y \in \mathbb{R}^{d_{out}} \) through the following computation:

\[
y = \sum_{i=1}^N r_i(x) E_i(x),
\]

where \( r_i = [router(x; G)]_i \) denotes the gating weight assigned to expert \( i \), \( G \) represents the gating function that determines routing probabilities based on input characteristics, and \( E_i(x) \) corresponds to the output of the \( i \)-th expert. Following architectural principles established in (Touvron et al., 2023a; Jiang et al., 2024), each expert implements an enhanced FFN structure defined as:

\[
E_i(x) = W_d^{(i)} (\mathrm{Act}(W_g^{(i)} x) \odot W_u^{(i)} x),
\]

where \( \mathrm{Act}(\cdot) \) denotes the activation function, and \( \odot \) represents element-wise multiplication. The expert parameters comprise up-projection matrices \( W_u^{(i)}, W_g^{(i)} \in \mathbb{R}^{d_{mid} \times d_h} \) and a down-projection matrix \( W_d^{(i)} \in \mathbb{R}^{d_h \times d_{mid}} \), where \( d_{mid} \) and \( d_h \) represent the intermediate and hidden state dimensions, respectively.

Quantization. For a given weight matrix \( W \), the \( b \)-bit quantization operation \( Q_b \) (Nagel et al., 2021)
(a) Outlier Proportion Across Projection Matrices
(b) L2 Norm Distribution of Outlier Channels

Figure 2: Channel-wise Distribution Analysis across MoE Architectures: (a) Outlier Proportion Across Projection Matrices reveals that outlier channels constitute only a small fraction of total channels, with down-projection matrices (\( W_{down} \)) consistently showing higher outlier proportions; (b) L2 Norm Distribution of Outlier Channels demonstrates a pronounced long-tail distribution, where outlier channels exhibit L2 norms orders of magnitude larger than typical channels

is formulated as:

\[
Q_b(W) = s \cdot \mathrm{clamp}\left(\frac{W}{s}\right),
\]
where \( s = \frac{\max(|W|) - \min(|W|)}{2^b - 1} \). \hspace{1cm} (3)

Here, \( s \) denotes the quantization step size, and \( \mathrm{clamp}(\cdot) \) represents a function that constrains values within a specified range. The quantization error, denoted as \( e_b(W) = W - Q_b(W) \), exhibits specific statistical properties.

Fisher Information Matrix. In the context of neural networks with parameters \( W \) and loss function \( L \), the Fisher Information Matrix is characterized by:

\[
F_W = \mathbb{E}\left[ \left( \frac{\partial L}{\partial W} \right) \left( \frac{\partial L}{\partial W} \right)^T \right],
\] \hspace{1cm} (4)

where the expectation is computed over the underlying data distribution. This matrix provides a principled measure of the local curvature in the loss landscape and quantifies the relative significance of individual parameters within the network architecture.

4 Method

4.1 Channel-wise Mixed-precision Quantization

Extensive empirical analysis of weight matrices in MoE architectures reveals a distinctive structural characteristic: the distribution of outliers exhibits pronounced channel-wise heterogeneity. Our investigation of \( W_{up} \), \( W_{gate} \), and \( W_{down} \) matrices across diverse expert networks demonstrates that significant outliers are consistently concentrated within a small subset of channels, while the majority maintain notably uniform distributions. As visualized in Figure 2, these channel-wise outlier patterns exhibit substantial variation, suggesting that conventional layer-wise or tensor-wise quantization approaches, which impose uniform quantization across all channels, are inherently suboptimal for MoE architectures.

Motivated by these empirical observations, we introduce a channel-wise mixed-precision quantization framework. For a weight matrix \( W \in \mathbb{R}^{m \times n} \) within expert \( E_i \), our approach independently quantizes each row vector \( w_j \) using bit-widths selected from a predefined set \( \mathcal{B} = b_1, b_2, ..., b_k \). The quantization operation for channel \( j \) with assigned bit-width \( b_j \in \mathcal{B} \) is formulated as:

\[
Q_{b_j}(w_j) = s_j \cdot \mathrm{round}(\frac{w_j}{s_j}),
\] \hspace{1cm} (5)

where the channel-specific quantization step size \( s_j \) is computed as:

\[
s_j = \frac{\max(|w_j|) - \min(|w_j|)}{2^{b_j} - 1}.
\] \hspace{1cm} (6)

This channel-wise quantization paradigm offers significant theoretical and practical advantages. By allocating higher bit-widths to channels containing substantial outliers, the framework preserves the model’s critical representational capacity in numerically sensitive regions. Simultaneously, channels exhibiting more uniform distributions can be efficiently quantized with lower bit-widths while maintaining numerical stability, leveraging their inherent robustness to quantization effects. This adaptive precision allocation enables optimal balance between model fidelity and memory efficiency
4.2 Theoretical Analysis of Layer-wise Quantization Sensitivity

A fundamental challenge in quantizing MoE architectures lies in understanding the differential sensitivity of various layer types to precision reduction. Our empirical investigations reveal a striking pattern: down-projection weights \( W_d \) consistently exhibit substantially higher magnitude and frequency of outliers compared to up-projection (\( W_u \)) and gating (\( W_g \)) weights across different experts. This systematic variation in weight distributions suggests an inherent asymmetry in the quantization requirements across different components of MoE architectures.

To rigorously characterize this empirical observation, we establish a theoretical framework that elucidates component-wise quantization sensitivity in MoE architectures. Our analysis synthesizes three fundamental aspects: the relationship between quantization perturbations and loss function dynamics through Fisher Information, the propagation patterns of quantization errors in MLP structures, and the derivation of optimal bit allocation strategies.

Proposition 4.1 (Loss Function Taylor Expansion). For sufficiently small perturbations \( \epsilon \) in the parameter space \( W \), the loss function \( L \) admits the following second-order approximation:
\[
L(W + \epsilon) = L(W) + \mathrm{tr}\left(\frac{\partial L}{\partial W} \epsilon^T\right) + \frac{1}{2} \mathrm{tr}\left(\epsilon H_W \epsilon^T\right) + O(|\epsilon|^3),
\]
where \( H_W \) denotes the Hessian matrix of \( L \) with respect to \( W \).

Lemma 4.2 (Fisher Information and Quantization). In practical neural network deployments where the loss landscape exhibits local convexity and quantization perturbations remain within the regime of Taylor approximation validity, the expected loss increase due to quantization necessarily satisfies:
\[
\mathbb{E}[\Delta L] \geq \mathrm{tr}\left(F_W \epsilon_b(W) \epsilon_b(W)^T\right).
\]
Proof. Leveraging the Taylor expansion from the previous proposition:
\[
\begin{align*}
\Delta L &= L(W + \epsilon_b(W)) - L(W) \\
&= \frac{1}{2} \mathrm{tr}(\epsilon_b(W), H_W, \epsilon_b(W)^T) \\
&\quad + \mathrm{tr}\left(\frac{\partial L}{\partial W}, \epsilon_b(W)^T\right) + O(|\epsilon_b(W)|^3).
\end{align*}
\]
The result follows naturally by taking expectation and applying the definition of the Fisher Information Matrix:
\[
\begin{align*}
\mathbb{E}[\Delta L] &\geq \mathbb{E}\left[\mathrm{tr}\left(\frac{\partial L}{\partial W}, \epsilon_b(W)^T\right)\right] \\
&= \mathrm{tr}\left(\mathbb{E}\left[\frac{\partial L}{\partial W}, \frac{\partial L}{\partial W}^T\right], \epsilon_b(W), \epsilon_b(W)^T\right) \\
&= \mathrm{tr}(F_W, \epsilon_b(W), \epsilon_b(W)^T).
\end{align*}
\]
(10)

To analyze the quantization sensitivity in FFN structures, we examine the error propagation through the projection layers. Taking the \( W_{down} \) and \( W_{up} \) matrices as representative examples, we establish the following characterization of quantization-induced errors.

Lemma 4.3 (Error Propagation in FFN Layers). For the standard FFN architecture incorporating up-projection matrix \( W_u \in \mathbb{R}^{d_h \times d} \) and down-projection matrix \( W_d \in \mathbb{R}^{d \times d_h} \) with \( d_h > d \), the quantization error propagation manifests as:
\[
|\Delta y|_2 \leq \kappa(W_d)|\epsilon_b(W_d)|_2|x|_2 + |W_d|_2|\epsilon_b(W_u)|_2|x|_2,
\]
where \( \kappa(W_d) = |W_d|_2|W_d^+|_2 \) represents the condition number of \( W_d \).

Proof. The quantization-induced output perturbation naturally decomposes as:
\[
\begin{align*}
\Delta y &= (W_d + \epsilon_b(W_d))(W_u + \epsilon_b(W_u))x \\
&\quad - W_d W_u x \\
&= W_d \epsilon_b(W_u)x + \epsilon_b(W_d) W_u x \\
&\quad + \epsilon_b(W_d) \epsilon_b(W_u)x.
\end{align*}
\]
Application of the triangle inequality yields:
\[
\begin{align*}
|\Delta y|_2 &\leq |W_d \epsilon_b(W_u)x|_2 + |\epsilon_b(W_d) W_u x|_2 \\
&\quad + |\epsilon_b(W_d) \epsilon_b(W_u)x|_2 \\
&\leq |\epsilon_b(W_d)|_2|W_u|_2|x|_2 \\
&\quad + |W_d|_2|\epsilon_b(W_u)|_2|x|_2 + O(|\epsilon_b(W)|^2).
\end{align*}
\]
(13)

The stated inequality follows from the architectural constraint \( |W_u|_2 \leq \kappa(W_d) \).

Theorem 4.4 (Optimal Bit Allocation). In typical neural network deployments where input distributions exhibit bounded moments and weight matrices maintain Lipschitz continuity, the optimal bit allocation ratio \( r^* \) between \( W_{down} \) and \( W_{up} \) necessarily satisfies:
\[
r^* = \frac{b_d}{b_u} \geq \sqrt{\frac{\kappa(W_d)}{\kappa(W_u)}} \cdot \frac{|FW_d F|}{|FW_u F|}.
\]
The detailed proof can be found in Appendix A.1.
Corollary 4.5. Under contemporary neural network architectures employing standard initialization schemes, the following relationships naturally emerge:

\[
\kappa(W_d) > \kappa(W_u) \quad \text{and} \quad |F_{W_d}|F| > |F_{W_u}|F|,
\]

(15)

These inequalities, in conjunction with Theorem 4.4, establish that the optimal bit ratio \( r^* \) between Wdown and Wup necessarily exceeds unity, indicating the imperative for higher bit allocation to the down-projection layer. This result derives from dimensional analysis and established initialization properties, providing theoretical guidance for bit allocation in mixed-precision quantization. The detailed proof is provided in Appendix A.2.

These theoretical findings provide rigorous justification for the empirically observed sensitivity of the down-projection layer. The analysis reveals that this sensitivity emerges from two fundamental factors: (1) the inherent dimensional compression in the down-projection operation, which engenders heightened sensitivity to quantization errors, and (2) its position in the computational graph, where perturbations can be amplified through the network structure. These insights naturally motivate our proposed bit allocation strategy that assigns higher precision to the down-projection layer to preserve model performance..

4.3 Theoretically-Guided Mixed-Precision Quantization Framework

Building on our layer-wise sensitivity analysis and channel heterogeneity characterization, we propose a mixed-precision quantization framework that bridges theoretical insights with deployment efficiency. Our method establishes dual-granularity bit allocation through theoretically-grounded sensitivity metrics.

Channel-wise Theoretical Sensitivity Metric

From Theorem 4.4 connecting quantization sensitivity with Fisher Information F and condition numbers \( \kappa \), we derive a channel-level sensitivity metric:

\[
S_c = \alpha \cdot \underbrace{\mathrm{tr}(F_c)\kappa_c}_{T_c} + (1 - \alpha) \cdot O_c
\]

(16)

where \( \alpha \in [0, 1] \) regulates the theoretical-empirical trade-off. The theoretical component \( T_c \) combines the cumulative gradient variance captured by the Fisher Information trace \( \mathrm{tr}(F_c) \) with the numerical stability quantified through the condition number \( \kappa_c \), which governs parameter perturbation sensitivity. The empirical component \( O_c \) implements our channel-wise distribution analysis in Section 4.1 by statistically quantifying quantization-critical outliers. This data-driven mechanism through an integrated analysis of the magnitude and frequency characteristics of outliers, thereby providing essential empirical compensation to theoretical sensitivity estimates.

Computationally Efficient Approximation

Direct computation of \( \mathrm{tr}(F_c) \) requires \( O(N_{params} \cdot N_{samples}) \) complexity due to gradient propagation. We develop theoretically-justified approximations:

\[
\mathrm{tr}(F_c) \approx V_c = \frac{\mathrm{var}(\mathbf{w}_c)}{\mathrm{mean}(\mathrm{var}(\mathbf{w}_i))}
\]
\[
\kappa_c \approx K_c = \frac{|\mathbf{w}_c|_2}{\min_j |\mathbf{w}_j|_2}
\]

(17)

The variance approximation \( V_c \) derives from maximum entropy principles under second-
<table>
  <tr>
    <th>#BITS</th>
    <th>Granularity</th>
    <th>Method</th>
    <th colspan="6">Accuracy (%)†</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th>BOOLQ</th>
    <th>PIQA</th>
    <th>HELLAS.</th>
    <th>WINO.</th>
    <th>ARC_E</th>
    <th>ARC_C</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>-</td>
    <td>-</td>
    <td>85.02</td>
    <td>82.59</td>
    <td>83.99</td>
    <td>76.56</td>
    <td>84.13</td>
    <td>56.83</td>
    <td>78.18</td>
  </tr>
  <tr>
    <td rowspan="4">2.86</td>
    <td>EXPERT</td>
    <td>FREQUENCY</td>
    <td>79.36</td>
    <td>78.62</td>
    <td>74.91</td>
    <td>70.88</td>
    <td>76.85</td>
    <td>45.82</td>
    <td>71.07</td>
  </tr>
  <tr>
    <td>METRIC</td>
    <td>W-OUTLIER</td>
    <td>79.42</td>
    <td>79.98</td>
    <td>77.65</td>
    <td>72.69</td>
    <td>78.54</td>
    <td>48.63</td>
    <td>72.82</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>W-OUTLIER</td>
    <td><b>82.02</b></td>
    <td>80.14</td>
    <td>78.18</td>
    <td>71.11</td>
    <td>77.78</td>
    <td>49.23</td>
    <td>73.10</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>OURS</td>
    <td>81.53</td>
    <td><b>80.69</b></td>
    <td><b>78.57</b></td>
    <td><b>71.69</b></td>
    <td><b>79.8</b></td>
    <td><b>50.43</b></td>
    <td><b>73.79</b></td>
  </tr>
  <tr>
    <td rowspan="4">2.68</td>
    <td>METRIC</td>
    <td>W-OUTLIER</td>
    <td>78.07</td>
    <td>78.45</td>
    <td>76.25</td>
    <td>70.32</td>
    <td>76.3</td>
    <td>48.12</td>
    <td>71.25</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>W-OUTLIER</td>
    <td><b>81.31</b></td>
    <td>78.84</td>
    <td>77.18</td>
    <td><b>72.06</b></td>
    <td>76.18</td>
    <td>44.37</td>
    <td>71.66</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>OURS</td>
    <td>80.58</td>
    <td><b>80.3</b></td>
    <td><b>77.8</b></td>
    <td>71.11</td>
    <td><b>78.2</b></td>
    <td><b>48.21</b></td>
    <td><b>72.68</b></td>
  </tr>
  <tr>
    <td rowspan="4">2.54</td>
    <td>EXPERT</td>
    <td>FREQUENCY</td>
    <td>73.55</td>
    <td>78.07</td>
    <td>72.51</td>
    <td>68.67</td>
    <td>72.69</td>
    <td>42.15</td>
    <td>67.94</td>
  </tr>
  <tr>
    <td>METRIC</td>
    <td>W-OUTLIER</td>
    <td>74.65</td>
    <td>78.07</td>
    <td>74.17</td>
    <td>71.43</td>
    <td>73.15</td>
    <td>42.75</td>
    <td>69.04</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>W-OUTLIER</td>
    <td>76.36</td>
    <td>78.24</td>
    <td>75.14</td>
    <td><b>72.06</b></td>
    <td>75.67</td>
    <td>44.8</td>
    <td>70.38</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>OURS</td>
    <td><b>80.49</b></td>
    <td><b>79.27</b></td>
    <td><b>76.58</b></td>
    <td>71.9</td>
    <td><b>77.57</b></td>
    <td><b>46.76</b></td>
    <td><b>72.10</b></td>
  </tr>
  <tr>
    <td rowspan="4">2.30</td>
    <td>METRIC</td>
    <td>W-OUTLIER</td>
    <td>71.59</td>
    <td>76.71</td>
    <td>72.14</td>
    <td>68.35</td>
    <td>72.81</td>
    <td>42.66</td>
    <td>67.38</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>W-OUTLIER</td>
    <td>76.3</td>
    <td>76.93</td>
    <td>73.79</td>
    <td>69.53</td>
    <td>73.19</td>
    <td><b>41.98</b></td>
    <td>68.62</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>OURS</td>
    <td><b>79.6</b></td>
    <td><b>78.67</b></td>
    <td><b>75.1</b></td>
    <td><b>70.8</b></td>
    <td><b>73.78</b></td>
    <td>41.64</td>
    <td><b>69.93</b></td>
  </tr>
  <tr>
    <td rowspan="4">2.20</td>
    <td>EXPERT</td>
    <td>FREQUENCY</td>
    <td>69.27</td>
    <td>77.31</td>
    <td>70.79</td>
    <td>69.3</td>
    <td>71.3</td>
    <td>39.51</td>
    <td>66.25</td>
  </tr>
  <tr>
    <td>METRIC</td>
    <td>W-OUTLIER</td>
    <td>66.27</td>
    <td>76.55</td>
    <td>70.22</td>
    <td>66.69</td>
    <td>70.12</td>
    <td>39.51</td>
    <td>64.89</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>W-OUTLIER</td>
    <td>72.45</td>
    <td><b>77.86</b></td>
    <td>72.97</td>
    <td>70.17</td>
    <td>73.32</td>
    <td>42.92</td>
    <td>68.28</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>OURS</td>
    <td><b>74.31</b></td>
    <td>77.69</td>
    <td><b>73.54</b></td>
    <td><b>71.03</b></td>
    <td><b>73.65</b></td>
    <td><b>42.92</b></td>
    <td><b>68.85</b></td>
  </tr>
</table>

Table 1: Zero-Shot Task Performance of Mixtral-8x7B using Automated Fine-Grained MoE Quantization. #Bits denotes bits for weight quantization, Granularity represents the level of mixed-precision application, and Method indicates the bit allocation strategy. "HellaS." is the short format of "HellaSwag" and "Wino." denotes "Winogrande".

moment constraints, while \( K_c \) preserves spectral stability relationships (see Appendix B for detailed derivations). This reduces computational complexity from \( O(N^2) \) to \( O(N) \). The final implementable metric becomes:

\[
S_c = \alpha(K_c V_c) + (1 - \alpha)O_c
\]

Layer-wise Bit Allocation Strategy

Theorem 4.4 establishes the optimal bit ratio between projection layers:

\[
\frac{b_d}{b_u} \geq \sqrt{\frac{\kappa(W_d)}{\kappa(W_u)}} \cdot \frac{|F_{W_d}|F}{|F_{W_u}|F}
\]

We implement this through practical approximations preserving theoretical guarantees:

\[
\beta_i = \sqrt{\frac{\kappa_d V_d}{\kappa_u V_u}}
\]

where \( \kappa_l = \max_j |w_j|_2 / \min_j |w_j|_2 \) captures numerical stability and \( V_i \) approximates Fisher norm \( |F_i|_F \). The allocation respects per-expert constraints:

\[
b_d^i + b_u^i + b_g^i = B^i
\]

where \( b_d^i, b_u^i, \) and \( b_g^i \) represent the bits allocated to \( W_{down}, W_{up}, W_{gate} \) respectively within expert \( i \), and \( B^i \) denotes the total bit budget for that expert.

The algorithm presented in Algorithm 1 integrates theoretical insights with practical efficiency considerations, operating at both the layer and channel granularities while maintaining minimal computational overhead through statistical approximations.

5 Experiments

5.1 Settings

Implementation Details. We implement our method following standard LLM quantization practices (Frantar et al., 2023; Lin et al., 2023; Shao et al., 2023; Ma et al., 2024), using a WikiText2 (Merity et al., 2016) calibration set (128 samples × 2048 tokens) and asymmetric group quantization (group size 128). Attention layers are quantized to 4-bit while maintaining full-precision routers. For FFN layer mixed-precision quantization, we set sensitivity balance coefficient \( \alpha \) to 0.5 and employ hardware-friendly 2-bit and 4-bit precision levels. To ensure fair comparison across granularity levels (Li et al., 2024), we maintain consistent ratios of 2-bit and 4-bit parameters while controlling the
<table>
  <tr>
    <th>#BITS</th>
    <th>GRANULARITY</th>
    <th>METHOD</th>
    <th colspan="6">ACCURACY (%) ↑</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th>PIQA</th>
    <th>HELLAS.</th>
    <th>WINO.</th>
    <th>ARC_E</th>
    <th>ARC_C</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>-</td>
    <td>-</td>
    <td>80.14</td>
    <td>77.28</td>
    <td>69.22</td>
    <td>73.19</td>
    <td>41.64</td>
    <td>68.294</td>
  </tr>
  <tr>
    <td rowspan="3">2.54</td>
    <td>LINEAR</td>
    <td>W-OUTLIER</td>
    <td>74.32</td>
    <td>67.88</td>
    <td>63.8</td>
    <td>69.2</td>
    <td>37.57</td>
    <td>63.06</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>W-OUTLIER</td>
    <td>75.3</td>
    <td>68.13</td>
    <td>64.33</td>
    <td>69.4</td>
    <td><b>38.74</b></td>
    <td>63.71</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>OURS</td>
    <td><b>75.46</b></td>
    <td><b>68.68</b></td>
    <td><b>64.64</b></td>
    <td><b>69.49</b></td>
    <td>38.71</td>
    <td><b>63.94</b></td>
  </tr>
  <tr>
    <td rowspan="3">2.30</td>
    <td>LINEAR</td>
    <td>W-OUTLIER</td>
    <td>73.99</td>
    <td>67.42</td>
    <td>62.98</td>
    <td>67.61</td>
    <td>36.52</td>
    <td>62.2</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>W-OUTLIER</td>
    <td>75.57</td>
    <td>67.72</td>
    <td><b>63.38</b></td>
    <td>68.22</td>
    <td>36.95</td>
    <td>62.76</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>OURS</td>
    <td><b>75.21</b></td>
    <td><b>68.89</b></td>
    <td>63.22</td>
    <td><b>70.22</b></td>
    <td><b>38.95</b></td>
    <td><b>63.79</b></td>
  </tr>
  <tr>
    <td rowspan="3">2.20</td>
    <td>LINEAR</td>
    <td>W-OUTLIER</td>
    <td>73.88</td>
    <td>66.32</td>
    <td>61.88</td>
    <td>64.9</td>
    <td>35.41</td>
    <td>60.66</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>W-OUTLIER</td>
    <td>73.94</td>
    <td>67.2</td>
    <td>62.67</td>
    <td>65.7</td>
    <td>36.69</td>
    <td>61.75</td>
  </tr>
  <tr>
    <td>CHANNEL</td>
    <td>OURS</td>
    <td><b>74.16</b></td>
    <td><b>67.54</b></td>
    <td><b>62.98</b></td>
    <td><b>69.07</b></td>
    <td><b>38.31</b></td>
    <td><b>62.88</b></td>
  </tr>
</table>

Table 2: Zero-Shot Task Performance of DeepSeek-V2-Lite using Automated Fine-Grained MoE Quantization. #Bits denotes bits for weight quantization, Granularity represents the level of mixed-precision application, and Method indicates the bit allocation strategy. "HellaS." is the short format of "HellaSwag" and "Wino." denotes "Winogrande".

overall parameter budget.

Baseline. All experiments were conducted on the Mixtral-8x7B-v0.1 (Jiang et al., 2024), DeepSeek-V2-Lite(Liu et al., 2024a) and Qwen-1.5-MoE(Team, 2024). Our method successfully quantizes this multi-billion parameter model, with all GPTQ (Frantar et al., 2023) experiments completed on a single NVIDIA RTX 3090 GPU with 24GB memory .

Evaluation. To assess the efficacy of our automated fine-grained quantization approach for MoE models, we conducted evaluations across five zero-shot tasks: PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2019), BoolQ (Clark et al., 2019), and ARC (Clark et al., 2018). We utilized the lm-eval-harness (Gao et al., 2021) framework to obtain individual task accuracies and the overall average accuracy. This comprehensive evaluation strategy enables us to gauge the impact of our quantization method on various aspects of model performance, providing insights into its generalization capabilities across diverse natural language understanding tasks.

5.2 Main Results

We conduct comprehensive zero-shot evaluations across various quantization configurations, including different bit-widths, granularity levels, and priority allocation strategies. Our primary baseline is derived from the extensive mixed-precision quantization experiments for MoE models presented in (Li et al., 2024). Their work explores two granularity levels: expert-level allocation, which distributes bits among experts within the same block, and metric-level allocation, which operates across all matrices (\( W_{down}, W_{up}, W_{gate} \)) throughout the model. For allocation strategies, they investigate frequency-based (frequency) prioritization according to router activation patterns and weight-outlier (w-outlier) based ordering. The specific bit-width assignment is determined by the mixed-precision ratio - for instance, a 2.54-bit configuration indicates that 25% of the FFN parameters are quantized to 4-bit precision, with the remaining parameters at 2-bit precision.

The experimental results in Table 1, Table 2 and Table 3 demonstrate the consistent superiority of our approach across various bit-width configurations. Our method achieves the highest average zero-shot accuracy under all bit-width settings, with particularly remarkable performance in lower bit. Notably, at 2.20 bits, our approach outperforms the best metric-level method by 3.96% (68.85% vs. 64.89%) and surpasses the frequency-based expert-level approach by 2.6% (68.85% vs. 66.25%) in accuracy, highlighting its competitive advantage in aggressive quantization scenarios. Furthermore, the effectiveness of channel-level granularity is evident even when using the same weight-outlier parameter selection strategy, consistently outperforming metric-level quantization across all configurations. This improvement suggests that finer-grained mixed-precision allocation enables more precise capture of parameter sensitivity and better preservation of model capabilities under strict bit-width constraints.
5.3 Ablation Study

Sensitivity Analysis of FFN Projections To validate Corollary 4.5 empirically, we conduct an ablation study by quantizing either the down-projection or up-projection layer while maintaining full precision for all other layers. As shown in Figure 3 We measure the mean squared error (MSE) between the original and quantized outputs across different transformer blocks. The results consistently show that down-projection layers exhibit higher quantization sensitivity across all transformer blocks, necessitating more bits in mixed-precision quantization. This experimental observation aligns with our theoretical analysis in Corollary 4.5, providing empirical support for our sensitivity-aware bit allocation strategy.

Analysis of Balance Parameter \( \alpha \) We conduct an ablation study on hyperparameter \( \alpha \) at 2.54-bit quantization level, varying it from 0.2 to 0.8. As shown in Figure 4, model performance peaks at \( \alpha \approx 0.5 \), indicating that balancing weight magnitude and gradient information is crucial for effective bit allocation in mixed-precision quantization. Lower \( \alpha \) values overemphasize weight magnitudes while higher values excessively prioritize gradient information, both leading to suboptimal quantization results.

![Layer-wise Mean Squared Error (MSE) comparison between down-projection and up-projection quantization, demonstrating consistently higher sensitivity in down-projection layers across transformer blocks.](page_176_1042_563_246.png)

Figure 3: Layer-wise Mean Squared Error (MSE) comparison between down-projection and up-projection quantization, demonstrating consistently higher sensitivity in down-projection layers across transformer blocks.

Analysis of Channel Quantization Direction
We investigate the impact of channel direction choice in quantization by comparing row-wise and column-wise quantization strategies, while maintaining the quantization level at 2.54-bit and using weightoutlier as the allocation criterion. Our experiments in Table 4 demonstrate that row-wise quantization consistently achieves superior performance. This finding aligns with the inherent function of FFN layers, as each row vector corresponds to an output dimension that represents a distinct feature transformation. By quantizing along the row direction, we preserve the integrity of individual feature transformations, allowing each output dimension to maintain its unique contribution to the model’s representation capacity. This result suggests that preserving the precision of output feature computations is more critical than maintaining the precision of input feature combinations.

6 Conclusion

This paper presents an automated fine-grained quantization framework for MoE models. Our key theoretical contribution establishes the relationship between Fisher Information, condition numbers, and quantization sensitivity, providing principled guidance for bit allocation across different model components. Through extensive empirical analysis, we demonstrate that MoE weight matrices exhibit distinct channel-wise outlier patterns, necessitating fine-grained quantization approaches. Our proposed framework successfully bridges theoretical insights with practical efficiency considerations through statistical approximations, enabling automated bit allocation without additional training overhead.

Limitations

While our work demonstrates promising results in MoE quantization, several limitations warrant discussion. First, our experimental validation, although thorough on Mixtral-8x7B, DeepSeek-V2 and Qwen-1.5-MoE, is constrained to a limited set of MoE architectures. Despite the block-wise quantization approach being computationally feasible on a single GPU, resource constraints prevented us from extending our evaluation to emerging models such as DeepSeek-V3, potentially limiting our understanding of the framework’s generalizability across different MoE architectures. Additionally, while our work focuses specifically on quantization-based compression, it does not explore potential synergies with other compression techniques such as pruning, knowledge distillation, or structured sparsification. Future work could investigate the integration of our fine-grained quantization framework with these complementary compression methods, potentially yielding more comprehensive and efficient MoE compression solutions.
References

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence.

Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, and 1 others. 2024-Db-llm: Accurate dual-binarization for efficient llms. arXiv preprint arXiv:2402.11960.

Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. 2022. Towards understanding the mixture-of-experts layer in deep learning. Advances in neural information processing systems, 35:23049–23062.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and 1 others. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazou Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, and 1 others. 2024. Deepseek-moe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066.

Shaojin Ding, Phoenix Meadowlark, Yanzhang He, Lukasz Lew, Shivani Agrawal, and Oleg Rybakov. 2022. 4-bit conformer with native quantization aware training for speech recognition. arXiv preprint arXiv:2203.15952.

Jun Fang, Ali Shafiee, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph H Hassoun. 2020. Post-training piecewise linear quantization for deep neural networks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 69–86. Springer.

William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39.

Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. International Conference on Learning Representations.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate post-training compression for generative pretrained transformers. In International Conference on Learning Representations.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muenninghoff, and 1 others. 2021. A framework for few-shot language model evaluation. Version v0. 0.1. Sept.

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minilm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations.

Shuai He, Daize Dong, Liang Ding, and Ang Li. 2024. Demystifying the compression of mixture-of-experts through a unified framework. arXiv preprint arXiv:2406.02500.

Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301.

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural computation, 3(1):79–87.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088.

Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2023. Squeezelm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629.

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668.

Pingzhi Li, Xiaolong Jin, Yu Cheng, and Tianlong Chen. 2024. Examining post-training quantization for mixture-of-experts: A benchmark. arXiv preprint arXiv:2406.08155.
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. 2023. Merge, then compress: Demystify efficient smoe with hints from its routing policy. arXiv preprint arXiv:2310.01334.

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. 2023. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978.

Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, and 1 others. 2024a. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024b. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024c. Spinquant–llm quantization with learned rotations. arXiv preprint arXiv:2405.16406.

Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. 2021. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092–28103.

Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. 2024. Affinequant: Affine transformation quantization for large language models. arXiv preprint arXiv:2403.12544.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197–7206. PMLR.

Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. 2021. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295.

Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International conference on machine learning, pages 18332–18346. PMLR.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641.

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2023. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695.

Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, and 1 others. 2024. Flatquant: Flatness matters for llm quantization. arXiv preprint arXiv:2410.09426.

Shyam A Tailor, Javier Fernandez-Marques, and Nicholas D Lane. 2020. Degree-quant: Quantization-aware training for graph neural networks. arXiv preprint arXiv:2008.05000.

Qwen Team. 2024. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters".

Hugo Touvron, Thibaut Lavril, Gautier Izardc, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, and 1 others. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. 2022a. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. arXiv preprint arXiv:2203.05740.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. 2022b. Outlier suppression: Pushing the limit of low-bit transformer language models. In Advances in Neural Information Processing Systems.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and I others. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.

Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Advances in Neural Information Processing Systems.

Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xing-gang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. 2023. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? CoRR, abs/1905.07830.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher DeWan, Mona Diab, Xian Li, Xi Victoria Lin, and I others. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
A Appendix A: Detailed Proofs for Optimal Bit Allocation

A.1 Proof of Theorem 4.4
Proof. The proof consists of three main steps:
Step 1: Error Propagation. As we have previously derived in Equation 12, from matrix perturbation analysis, we have:

\[
\Delta y = (W_d + \epsilon_b(W_d))(W_u + \epsilon_b(W_u))x \\
- W_d W_u x \\
= W_d \epsilon_b(W_u)x + \epsilon_b(W_d)W_u x \\
+ \epsilon_b(W_d)\epsilon_b(W_u)x.
\]

Step 2: Bounding the Expected Loss Increase. Using Lemma 4.2 and 4.3, we can bound the expected loss increase. First, note that for uniformly distributed quantization error, we have:

\[
\mathbb{E}[\epsilon_b(W)\epsilon_b(W)^T] = \frac{1}{2^{2b}} \mathrm{diag}(\mathrm{var}(W))
\]

For \( W_d \):

\[
\mathbb{E}[\Delta L_d] \geq \mathrm{tr}(F_{W_d}\epsilon_b(W_d)\epsilon_b(W_d)^T) \\
= \mathrm{tr}(F_{W_d}\mathbb{E}[\epsilon_b(W_d)\epsilon_b(W_d)^T]) \\
= \frac{1}{2^{2b_d}} \mathrm{tr}(F_{W_d} \mathrm{diag}(\mathrm{var}(W_d))) \\
= \frac{|F_{W_d}|_F}{2^{2b_d}}
\]

Similarly for \( W_u \):

\[
\mathbb{E}[\Delta L_u] \geq \mathrm{tr}(F_{W_u}\epsilon_b(W_u)\epsilon_b(W_u)^T) = \frac{|F_{W_u}|_F}{2^{2b_u}}
\]

By the independence of quantization operations between layers and the additivity of expectation:

\[
\mathbb{E}[\Delta L] = \mathbb{E}[\Delta L_d] + \mathbb{E}[\Delta L_u] \\
\geq \frac{|F_{W_d}|_F}{2^{2b_d}} + \frac{|F_{W_u}|_F}{2^{2b_u}}
\]

□

A.2 Proof of Corollary 4.5
Proof. Step 1: Dimensional Analysis. Recall \( W_d \in \mathbb{R}^{d \times d_h} \) and \( W_u \in \mathbb{R}^{d_h \times d} \), where \( d_h > d \).
Step 2: Xavier Initialization. Under Xavier initialization, weights are typically sampled as:

\[
W_{ij} \sim \mathcal{N}\left(0, \frac{2}{n_{in} + n_{out}}\right).
\]

Step 3: Condition Number Comparison. For \( W_d \), this leads to:

\[
\kappa(W_d) \approx \frac{\sqrt{d_h}}{d} \frac{\max \sigma_i}{\min \sigma_i},
\]

while for \( W_u \):

\[
\kappa(W_u) \approx \frac{\sqrt{d}}{d_h} \frac{\max \sigma_i}{\min \sigma_i},
\]

so \( \kappa(W_d) > \kappa(W_u) \) since \( d_h > d \).
Step 4: Fisher Information Comparison. Due to the bottleneck from \( d_h \) to \( d \), \( W_d \) compresses information more aggressively, leading to:

\[
|F_{W_d}|_F > |F_{W_u}|_F.
\]

Hence, both terms in the inequality of Theorem 4.4 exceed 1, implying \( r^* = \frac{b_e}{b_c} > 1 \). □

B Appendix B: Approximation Derivation
The approximations are derived from fundamental theorems in optimization and matrix theory:

B.1 Fisher Information Trace Approximation
1. Starting with the exact definition:

\[
\mathrm{tr}(F_c) = \mathbb{E}[|\partial L / \partial w_c|^2]
\]

2. Applying optimization theory and Taylor expansion near optimality:

\[
\partial L / \partial w_c = \delta_c \otimes x \approx H_c(w_c - w_c^*)
\]

where \( H_c \) is the Hessian and \( w_c^* \) are optimal parameters.

3. This leads to the key approximation:

\[
\mathrm{tr}(F_c) = \mathbb{E}[||H_c(w_c - w_c^*)||^2] \\
= \mathbb{E}[\delta_c^2] \mathbb{E}[||x||^2] \propto \mathrm{var}(w_c)
\]

4. The normalization term emerges naturally; mean(\mathrm{var}(w_c)) ensures scale invariance across layers

B.2 Condition Number Approximation
1. From matrix perturbation theory:

\[
\kappa_c = \sigma_{\max}(w_c) / \sigma_{\min}(w_c)
\]

2. For channel vectors, this simplifies to:

\[
\kappa_c = \|w_c\|_2 / \min_j \|w_j\|_2
\]

preserving the relative scaling relationships between channels.

C Other experiment results
<table>
  <tr>
    <th>#BITS</th>
    <th>GRANULARITY</th>
    <th>METHOD</th>
    <th colspan="6">ACCURACY (%) ↑</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th>PIQA</th>
    <th>HELLAS.</th>
    <th>WINO.</th>
    <th>ARC_E</th>
    <th>ARC_C</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="3">2.69</td>
    <td rowspan="3">CHANNEL</td>
    <td>LINEAR</td>
    <td>80.14</td>
    <td>77.28</td>
    <td>69.22</td>
    <td>73.19</td>
    <td>41.64</td>
    <td><b>68.294</b></td>
  </tr>
  <tr>
    <td>W-OUTLIER</td>
    <td>76.71</td>
    <td>72.60</td>
    <td>65.43</td>
    <td>59.60</td>
    <td>32.34</td>
    <td>61.336</td>
  </tr>
  <tr>
    <td>OURS</td>
    <td><b>77.48</b></td>
    <td><b>73.67</b></td>
    <td><b>62.59</b></td>
    <td><b>63.49</b></td>
    <td><b>35.41</b></td>
    <td><b>62.528</b></td>
  </tr>
  <tr>
    <td rowspan="3">2.58</td>
    <td rowspan="3">CHANNEL</td>
    <td>LINEAR</td>
    <td>77.53</td>
    <td>72.77</td>
    <td>62.12</td>
    <td>61.07</td>
    <td>31.23</td>
    <td>60.944</td>
  </tr>
  <tr>
    <td>W-OUTLIER</td>
    <td>76.33</td>
    <td>72.78</td>
    <td>63.33</td>
    <td>61.62</td>
    <td>34.39</td>
    <td>61.690</td>
  </tr>
  <tr>
    <td>OURS</td>
    <td><b>77.84</b></td>
    <td><b>73.24</b></td>
    <td><b>63.38</b></td>
    <td><b>63.56</b></td>
    <td><b>33.36</b></td>
    <td><b>62.276</b></td>
  </tr>
  <tr>
    <td rowspan="3">2.47</td>
    <td rowspan="3">CHANNEL</td>
    <td>LINEAR</td>
    <td>75.90</td>
    <td>72.60</td>
    <td>62.12</td>
    <td>61.871</td>
    <td>31.28</td>
    <td>60.7542</td>
  </tr>
  <tr>
    <td>W-OUTLIER</td>
    <td>75.97</td>
    <td>72.46</td>
    <td>61.56</td>
    <td>62.09</td>
    <td>32.79</td>
    <td>60.974</td>
  </tr>
  <tr>
    <td>OURS</td>
    <td><b>75.97</b></td>
    <td><b>73.21</b></td>
    <td><b>62.19</b></td>
    <td><b>62.18</b></td>
    <td><b>33.59</b></td>
    <td><b>61.428</b></td>
  </tr>
</table>

Table 3: Zero-Shot Task Performance of Qwen-1.5-MoE using Automated Fine-Grained MoE Quantization. #Bits denotes bits for weight quantization, Granularity represents the level of mixed-precision application, and Method indicates the bit allocation strategy. "HellaS." is the short format of "HellaSwag" and "Wino." denotes "Winogrande".

<table>
  <tr>
    <th>DIRECTION</th>
    <th colspan="6">ACCURACY (%) ↑</th>
  </tr>
  <tr>
    <th></th>
    <th>PIQA</th>
    <th>HELLAS.</th>
    <th>WINO.</th>
    <th>ARC_E</th>
    <th>ARC_C</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>DOWN ROW</td>
    <td>76.55</td>
    <td>73.52</td>
    <td>69.61</td>
    <td>75.34</td>
    <td>43.86</td>
    <td>67.78</td>
  </tr>
  <tr>
    <td>DOWN COLUMN</td>
    <td><b>78.84</b></td>
    <td>75.25</td>
    <td>68.98</td>
    <td>74.75</td>
    <td>44.2</td>
    <td>68.40</td>
  </tr>
  <tr>
    <td>ALL COLUMN</td>
    <td>78.13</td>
    <td><b>75.48</b></td>
    <td>71.51</td>
    <td>75.38</td>
    <td>44.28</td>
    <td>68.95</td>
  </tr>
  <tr>
    <td>ALL ROW</td>
    <td>78.24</td>
    <td>75.14</td>
    <td><b>72.06</b></td>
    <td><b>75.67</b></td>
    <td><b>44.8</b></td>
    <td><b>69.19</b></td>
  </tr>
</table>

Table 4: Performance comparison of different channel quantization directions in FFN layers of Mixtral-8x7B-v0.1. All Row represents using row-wise quantization for all channel quantization matrices, Down Row represents row-wise quantization for \( W_{down} \) only, column-wise for \( W_{up} \) and \( W_{gate} \). "HellaS." is the short format of "HellaSwag" and "Wino." denotes "Winogrande".

![Line plot showing the impact of balance parameter α on zero-shot accuracy under 2.54-bit quantization, demonstrating optimal performance at α nearby 0.5.](page_1012_1342_377_246.png)

Figure 4: Impact of balance parameter \( \alpha \) on zero-shot accuracy under 2.54-bit quantization, demonstrating optimal performance at \( \alpha \) nearby 0.5.

Reviewer paper summaries: This paper introduces an automated fine-grained quantization framework for MoE-based LLMs. It addresses the inherent challenges posed by MoE architectures, particularly the distinctive outlier distributions across expert networks. By conducting theoretical analysis, the authors identify higher quantization sensitivity in down-projection layers compared to up-projection layers, motivating their channel-wise quantization strategy. The authors leverage statistical approximations to allocate optimal bit-widths dynamically and validate their approach empirically on models such as Mixtral-8x7B-v0.1 and DeepSeek-V2-Lite, showing significant performance improvements over state-of-the-art quantization methods.

This paper introduces a quantization method specifically for Mixture of Experts (MoE) architectures. It reveals that weight matrices across experts exhibit distinctive channel-wise outlier distributions. Based on this observation, it proposes a channel-wise mixed-precision quantization framework. The paper also provides theoretical analysis and further proposes a channel-wise mixed-precision quantization framework. Evaluations are conducted on Mixtral-8x7B and DeepSeek-V2-Lite models.
Summary of strengths: 1. The paper present a novel method for MoE quantization with theoretical and empirical foundations.

2. Clear explanations and detailed algorithms are provided, which facilitate replication of results.

1. The motivation is clear, and the preliminary channel-wise analysis effectively illustrates the rationale behind the proposed method.

2. The theoretical analysis is both sufficient and insightful, providing a solid foundation for the mixed-precision quantization framework.

3. The automated allocation strategy for mixed-precision quantization is novel.
Summary of weaknesses: 1. The paper only experiments on two MoE models: Mixtral-8x7B-v0.1 and DeepSeek-V2-Lite. The generalization to other models is questionable.

2. While the authors claim minimal computational overhead, more explicit comparisons or detailed analysis of runtime and computational costs relative to baselines would enhance clarity.

3. Provide clearer justifications or intuitive explanations for choosing specific empirical approximations, particularly in equations (17) and (18).

The experimental analysis is relatively weak compared to the theoretical discussion. Including more experimental insights into the quantization of the experts in the main text, while moving some of the theoretical proofs to the appendix, could strengthen the paper.
Comments, suggestions and typos: 1. Consider additional experiments on more diverse datasets or tasks beyond zero-shot benchmarks to demonstrate broader applicability.

2. Typo: Section 5.1, line 489, "WikiText2" is cited incorrectly

None
Overall assessment scores: 3

2
Metareview score: 3
Metareview: This paper proposed a quantization method for Mixture of Experts. Specifically, it proposed a channel-wise mixed-precision quantization framework and provided some theoretical analysis on it. The experiments on several datasets show the effectiveness of the method.
Summary of reasons to publish: 1. The idea of the method is novel.
2. The experimental results look promising.
3. It provides theoretical analysis.