MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
Authors: Zhixuan Chen
Xing Hu
Dawei Yang
Zukang Xu
XUCHEN
Zhihang Yuan
Sifan Zhou
JiangyongYu
Venue: ICML 2025 poster (2025)
Keywords: Mixture of Expert
Large Language Models
Quantization
Primary area: deep_learning->large_language_models

Abstract: Mixture-of-Experts (MoE) large language models (LLMs), which leverage dynamic routing and sparse activation to enhance efficiency and scalability, have achieved higher performance while reducing computational costs. However, these models face significant memory overheads, limiting their practical deployment and broader adoption. Post-training quantization (PTQ), a widely used method for compressing LLMs, encounters severe accuracy degradation and diminished generalization performance when applied to MoE models. This paper investigates the impact of MoE’s sparse and dynamic characteristics on quantization and identifies two primary challenges: (1) Inter-expert imbalance, referring to the uneven distribution of samples across experts, which leads to insufficient and biased calibration for less frequently utilized experts; (2) Intra-expert imbalance, arising from MoE's unique aggregation mechanism, which leads to varying degrees of correlation between different samples and their assigned experts. To address these challenges, we propose MoEQuant, a novel quantization framework tailored for MoE LLMs. MoEQuant includes two novel techniques: 1) Expert-Balanced Self-Sampling (EBSS) is an efficient sampling method that efficiently constructs a calibration set with balanced expert distributions by leveraging the cumulative probabilities of tokens and expert balance metrics as guiding factors. 2) Affinity-Guided Quantization (AGQ), which incorporates affinities between experts and samples into the quantization process, thereby accurately assessing the impact of individual samples on different experts within the MoE layer. Experiments demonstrate that MoEQuant achieves substantial performance gains (more than 10 points accuracy gain in the HumanEval for DeepSeekMoE-16B under 4-bit quantization) and boosts efficiency.

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
Zhixuan Chen 1 Xing Hu
Dawei Yang
Zukang Xu
Chen Xu Zhihang Yuan
Sifan Zhou 12 Jiangyong Yu!
Abstract
performance gains (more than 10 points accuracy gain in the HumanEval for Moe- under 4-bit quantization) and boosts efficiency.
Mixture-of-Expers (MoE) large language mod- els (LLMs), which leverage dynamic routing and sparse activation to enhance efficiency and scala- bility, have achieved higher performance while re- ducing computational costs. However, these mod- els face significant memory overheads, limiting their practical deployment and broader adoption. Post-training quantization (PTQ), widely used method for compressing LLMs, encounters se- vere accuracy degradation and diminished gener- alization performance when applied to MoE mod- els. This paper investigates the impact of MoE's sparse and dynamic characteristics on quantiza- tion and identifies two primary challenges: (I) Inter-expert imbalance, referring the uneven distribution of samples across experts, which leads to insufficient and biased calibration for less frequently utilized experts; (2) Intra-expert imbalance, arising from MoE's unique aggre- gation mechanism, which leads to varying de- grees of correlation between different samples and their assigned experts. To address these chal- lenges, propose MoEQuant, novel quantiza- tion framework tailored for MoE LLMs.. MoE- Quant includes two novel techniques: 1) Expert- Balanced Self-Sampling (EBSS) is an efficient sampling method that efficiently constructs a cali- bration set with balanced expert distributions by leveraging the cumulative probabilities of tokens and expert balance metries guiding factors. 2) Wfiinty--uiied Quantization (AGQ), which in- corporates affinities between experts and samples into the quantization process, thereby accurately assessing the impact of individual samples on dif- ferent experts within the MoE layer. Experiments demonstrate that MoEQuant achieves substantial
1. Introduction
Recent advances natural language processing have been profoundly influenced by the success of large language models (LLMs). Among these, Mfxture-of-Experts (MoE) LLMs, which leverage the dynamic routing mechanisms and scalable capabilities of MoE layers, have demonstrated superior performance and achieved state-of-the-art results, garnering significant attention from the research commu- nity (Jiang al., 2024; Qwen, 2024; Liu et al., 2024b). However, during deployment, MoE LLMs face only the same memory bandwidth constraints as conventional LLMs (Kim et al., 2023; Dettmers et al., 2022) but also substantially higher storage requirements. For example, in Qwen-MoB-AN M.MMMMo-A2.7B--- BB-1BB (Qwen, 2024), only 2.7 billion pa- rameters are activated during the generation phase, yet all 14 billion parameters must reside in memory, significantly in- creasing inference costs. Furthermore, MoE layers account for most of the parameter footprint within the transformer blocks: approximately 80% when considering activated experts and up to 97% when including all experts. Conse- quently, compressing MoE LLMs, particularly their MoE layers, is critical for reducing inference costs and enabling deployment on resource-constrained devices with limited memory capacity and bandwidth.
Post-Training Quantization (PTQ), which quantizes weights into low-precision format, effectively reduces model size and memory footprint, achieving notable success in con- ventional large language models (LLMs). For example, AWQ (Lin et al., 2023) and GPTQ (Frantar et al., 2022) compress model weights by up to four times without requir- ing additional training. while maintaining nearly lossless performance. However, when these methods are applied directly to MoE LLMs, they often lead to overfitting and significant performance degradation, particularly in terms of generalization. This is because they focus on layer-wise quantization while overlooking the unique architecture of MoE, which routes samples to limited number of experts and aggregates their outputs through weighted combinations,
'qqual contribution 'Houmo AI 'Southcast University. Corre- spondence to: Dawci Yang <dawei. ho
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models
70
GPTQ-gsmäk
+4.19%
Normalized Distribution
0.06
C4
MoEQuanl(GPTO)-gamng
Wikitext2
0.05
EBSS
+3.79%
6
0.04
+46.60%
+5.90%
0.03
Accuracy
+11.77%
0.02
40
0.01
+62.48%
30
+11.62%
+6.49%
10 20 30 40 50
50
+10.82%
Expert Index
20
+113.72%
+5.85%
Figure 2. Sample distribution on the first MoE layer of Qwen-MoB- A2.7B-14B for different calibration sets. For C4 and WikiTex12, 128 x 512 tokens were sampled, for our EBSS, samples were generated through the model's self-sampling method.
+99.91%
GPTQ (rQ0-vaaaanvalll
MoECUanntGPPQQ--umanees
OwenMoE-1
Deepseek-
Deepseek
OwenMoE
Mixtral
Mixtral-I
Models
Figure 1. The relative accuracy gains of GPTQ across various mod- els for two generative tasks, HumanEval and GSM8K, before and after applying MoBQuant are presented. The suffix "T" denotes the instruction fine-tuned version.
(EBSS) and Affinity-Guided Quantization (AGQ). EBSS constructs calibration sets based on the self-sampling ca- pabilities of L.LSs and incorporates cumulative probabil- ity and expert balance metries to guide the sampling pro- cess. This guidance significantly reduces search complex- ity. Additionally, it ensures that calibration samples evenly distributed among experts and consistent with the pretraining data distribution. AGQ addresses the imbalance in token-expen affinity during expert quantization by inte- grating affinity into layer-wise calibration and constructing weighted quantization errors. This approach adapts to the dynamic characteristics of MoE, enabling more accurate cal- culation of quantization errors and sensitivity. By integrat- ing these two methods, we present MoEQuant, framework that bridges existing quantization techniques with MoE ar- chitectures, taking a crucial step toward reconciling the effi- ciency of quantized systems with the unique requirements of MoE LLMs. As shown in Figure 1, MoEQuant achieves per- formance improvements of varying degrees across different models, highlighting its effectiveness and broad applicabil- ity for enhancing MoE language models. Our contributions are summarized as follows:
In addition, they fail to account for the inherent sparsity and heterogeneity introduced by the MoE structure.
We perform a comprehensive analysis of the key factors that affect the quantization performance of MoE LLMs and identified two inherent imbalances within the MoE archi- tecture as the primary contributors. Firstly, there an imbalance in the distribution of samples across different experts. As highlighted in DeepSeek (Liu et al., 2024b), various techniques have been developed to maintain load balance among experts, which is equally critical during the calibration phase. However, calibration sets are often do- main specific, and the gating mechanism can result in some experts being overloaded while others remain underloaded. Underloaded experts naturally receive insufficient calibra- tion, leading significant quantization errors. As shown in Figure 2, both of the two most commonly used calibration sets exhibit this imbalance. Secondly, there is an imbalance in the affinities between samples and their assigned experts. Unlike traditional I.MMs, where all samples are processed by a single feedforward network, MoE architectures use a gating mechanism to express the output as a weighted sum of results from multiple experts. Consequently, from the perspective of each expert, samples exhibit varying levels of affinity, defined as the correlation between a sample and its assigned expert. Existing PTQ methods (Xiao et al., 2022; Lin et al., 2023; Ashkboos et al., 2024) fail to account for this affinity during expert quantization. For example, GPTQ (Frantar et al., 2022) disregards the impact of the gating unit when collecting Hessian information, resulting in an inaccurate assessment of the importance of individual samples for each expert. This oversight distorts the Hessian information and significantly degrades the performance of the quantized model.
We identify two critical imbalances Dibalances-iiteereexprrr inter-expert and intra-expert-in the quantization of MoE models: sam- ple distribution imbalance among experts and token- expert affinity imbalance.
We propose Expert- Balanced Self- Self-Sampling to effi- ciently generate a balanced calibration dataset, ensur- ing equitable utilization of all experts. We also propose Affinity-Guider Quantization to introduce token-expert affinities into the quantization process, thereby improv- ing weight update accuracy and reducing quantization errors.
We develop MoFQuant, which seamlessly integrates EBSS and AGQ with existing PTQ methods, signifi- cantly enhancing the quantization performance of MoE LLMs. As one of the first studies in this area, we will release the code to encourage further exploration and drive progress in this field.
To address the aforementioned two imbalances, this paper introduces two methods: Expert-Balee Balanced Self-Sampling
2

---

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models
2. Related Work
Block x L
shared expert
routing expert
2.1. Mixture-of-Experts Large Language Models
The Mixtureoof-Experts (MoE) model, first introduced by (Jacobs et al., 1991) and (Jordan & Jacobs, 1994), has been extensively explored in the various contexts (Eigen al., 2013; Theis & Bethge, 2015; Deisenroth & Ng. 2015; Aljundi et al., 2017). In MoE LLMs, each MoE layer com- prises multiple expert networks and a gating network. The gating network, typically implemented as a linear layer with a softmax function, directs inputs to the appropriate expert networks and aggregates their outputs. Different mod- els employ various configurations. For example. Switch- Transformer (Fedus al., 2022) introduces top-1 gating strategy, achieving competitive results for specific model sizes. Mixtral-8x7B (Jiang et al., 2024) combines MoE with infrastructure innovations, utilizing two experts per layer to achieve excellent performance while maintaining low computational cost. DeepSeekMoE (Dai et al., 2024) refines expert segmentation by subdividing the intermediate hidden dimensions of FFNs, increasing the number of ex- perts, and activating more of them to improve knowledge decomposition and capture. It also introduces shared ex- perts, which are always activated to consolidate common knowledge across contexts, reducing parameter redundancy in souting-specific experts. DeepSeekv2 (Liu et al., 2024a) and DeepSeekv3 (Lu, 2025) further enhance performance with refined designs. Qwen-Moe (Qwen, 2024) replaces traditional FFN layers entirely with MoE layers, employ- ing four shared experts alongside four unshared experts se- lected from a pool of 60. During training, Qwen-MoE first adapts the existing Qwen-1.88 1.8B model to create Qwen .-5- MoE-A2.7B- 16B, achieving better overall pretraining per- formance.
MoE
x
Eg1
Y 2OO
Esg
Norm
Esm
Gating
E
Attn
COOD
E2
topk k
Norm
pn-1
Figure 3. The MoB structure in LLMss The router selects all non- shared experts and & shared experts with highest confidence. The predictions from all experts are then aggregated and weighted.
quantization schemes from a vector perspective, offering better adaptability to weight distributions.
However, these methods overlook the unique challenges posed by MoE architectures, resulting in significant accu- racy drop. Our proposed MoEQuant, rooted in the relation- ship between calibration samples and experts, is orthogonal to existing PTQ methods, enabling seamless integration for the effective quantization of MoE-based LLMs.
3. Preliminaries
As illustrated in Figure an MoE layer comprises m shared and routing experts, along with gating network that as- signs different experts and their corresponding probabilities to each token. Only the top k shared experts with the highest affinities are utilized. For a given input token æ, the output y is computed as a weighted sum of the outputs from the top k routing experts and all shared expert:
2.2. Post-Training Training Quantization for LLMS
Most LLMs are built upon Transformer( Vaswani et al., 2017) architecture, which is inherently memory-intensivv Post-training quantization (PTQ) has become a widely adopted approach compress LLMs, effectively reducing memory consumption while maintaining model accuracy. Two prominent PTQ methods, GPTQ (Frantar et al., 2022) and AWQ (Lin et al., 2023), have been extensively stud- ied. GPTQ employs Hessian-bee baassd error compensation to minimize quantization errors and achieve high compression rates. AWQ, on the other hand, accounts for the impact of activation distributions on weight quantization, thereby improving the performance of quantization. Beyond these methods, several advanced techniques have emerged to fur- ther enhance PTQ. Quarot (Ashkboos et al., 2024) applies Hadamard transformations to remove outliers without alter- ing the output, thus enhancing the effectiveness of GPTQ. GPTVQ (van Baalen et al., 2024) explores non-uniform
y = Σ (z))((()) + (a))))))
(I)
JEK
shared experts
top & routing experts
where X. = topk (19(()) i (1....,m,) .m])), gi())) repre- sents the weight assigned to the i-th expert.
Perplexity (PPL.) is common metric for evaluating the quality of language models. A lower PPL indicates better predictive accuracy and closer alignment with the model's true distribution. For sequence composed of n tokens D = (d1, dd)),,)), the perplexity with respect to model
3

---

MoEQuant: Enhancing Quantization for Mixture-of-Expert Large Language Models
M is defined as:
4. Method
PPL(D IM exp (-x Em log PM (d. di, de...... dz-1))
4.1. MoEQuant
(2)
In this paper, we present MoBQuant, a framework designed to efficiently quantize LLMs utilizing MoE architectures. MoEQuant addresses the critical challenge of expert im- balance, both inter- and intra-expert, which arises during the quantization process. Morresss tackles the imbalance from two perspectives: the generation of appert-balanced calibration datasets and the token-expert correlation dur- ing expert calibration. Correspondingly. it incorporates two solutions: Expert-Balanced Samp-ign (EBSS) and Affinity-Guided Quantization (AGQ). EBSS generates cal- ibration samples that ensure the balanced engagement of all experts within MoE architectures. AGQ, on the other hand, addresses the correlation disparities between samples introduced by gating units in MoE layers.
where P (d; d1,
represents the probabil-
ity predicted by the model for di, given the context
d,, .2..... d:-1-
Expert balance is evaluated by the standard deviation in the frequeney of expert usage across all layers. We define it as:
21=1
Oi
0
(3)
L
01 1
E
Ui 4z)2
(4)
6=1
where L denotes the number of layers in the MoE model, E' represents the total number of experts in a layer, and di refers to the standard deviation of the I-th layer, which is calculated based on the usage frequency of of each expert and the average frequency in.
Both methods are plug-and-play and can be seamlessly in- tegrated with other quantization techniques to improve the performance of MoE LLMs. Detailed descriptions of them are provided in Sections 4.2 and 4.3.
4.2. Expert-Balanced Self-Sampling
Quantization typically involves mapping a floating-point number to discrete interval using integer values. For weight quantization, we focus on the most commonly used per-channel symmetric uniform quantization. The quantiza- tion process is expressed as follows:
Current PTQ methods typically rely on domain -ppeifc calibration datasets, such as WikiText-2. Although these calibration datasets can preserve reasonable generalization capabilities for standard LLMs, their direct application to MoE LLMs often leads to significant performance degra- dation. This degradation occurs because calibration datasets result in an uneven sample distribution among experts. As illustrated in Figure 2, relying on a single calibration set usually produces long-tailed distribution of samples among different experts.
@(W) clamp (1 W
Amin Qmax
8
where W E R represents the weight matrix, s E IR" denotes the channel-wise quantization step, and Gminn Qmaa represent the quantization bounds. To facilitate the evalua- tion of quantization error, we typically perform a dequanti- zation operation:
An intuitive approach is to construct a Homain-balanced calibration set by sampling data from multiple domains. However, the virtually infinite number of possible domains makes achieving true domain balance both complex and impractical. Moreover, as shown in Figure 4, even high- quality datasets often exhibit high perplexity, indicating a misalignment between the selected data and the model's inherent distribution.
W = Q(W)
(6)
For linear layer, the loss caused by quantizing W can be formulated
c(w) = Wx - Wx 2 F²
Problem Definition Based on the above, the objective is to identify dataset D that satisfies two key properties:
Low perplexity. The samples in D* should align closely with the inherent distribution of the model M, which corresponds minimizing perplexity.
where X E bxe represents the activation of the calibration data this layer. AWQ (Lin et al., 2023) utilizes Equation 7 to guide the selection of smoothing coefficients and weight pruning. GPTQ (Frantar et al., 2022) follows OBQ (LeCun et al., 1989), which uses the Hessian to compensate for the quantization error. In conjunction with Equation 7, the Hessian can be effectively computed as:
Expert balance. The samples should be evenly dis- tributed among experts in MoE LLMs, ensuring that no expert is overused or underused.
This dual requirement can be formulated as a joint optimiza-
H X
(8)

---

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models
tion problem, which be formulated as:
where is the length of S. During the sampling phase,
perplexity be easily calculated by Rg and the predicted probability P(v|S) by
((MMD)
D arg min PPL(M. exp
(9)
(R. (R, + P(r(s)) (
where exp <(M.D) represents the reciprocal of normal- ized imbalance, exp is used to normalize o, and T is nyper-parameter controlling the impact of expert imbalance. Regarding the influence of the hyperparameter and the objective function, we provide a detailed derivation of the complete process in the Appendix A.I.
PPI((M, = exp n+
(12)
where denotes concatenation.
7.10
7-
6
The perplexity optimization corresponds to an optimal sub- set selection problem, while expert balancing is analogous boad-balancing problem. Both are NP-hard, making di- rect solution computationally infeasible. To enable practical analysis, the problem can be reformulated in combination with Equation 2 as
5
Perplexity
4.
3.05
2.88
(log (P(Di(Dss1)) (MM.D)
2
D = arg min
1.20
D
N
subject to D = V V
Wikifext2 HumanEval
GSMBK
CEval
MMLU
EBSS
times
Datasets
(10)
Figure 4. Perplexity performance on DeepSeck-MoE-Il of dif- ferent datasets.
where V = {U1, V2,, ...,.... , J represents the vocabulary that contains m tokens, P denotes probabilities predicted by M. n is the sequence length and @ denotes the Cartesian product. In this context, optimization can be viewed as searching for optimal path within an n-dimensional vocabulary space.
Probability- Guided Path Pruning. When predicting the next token in self-sampling approach, the head layer out- puts probabilities for all possible candidates, which typi- cally exhibit a multimodal distribution. Tokens with low probabilities often result in incoherent or semantically in- correct sequences, Based on this observation, we propose probability- path pruning method effectively improve search efficiency. The core idea is ignore low- probability branches during the search process.
Challenges. One challenge lies in the availability of limited datasets. Since only small amount of data is typically accessible for calibration, it is difficult to ensure ideal do- main balance or alignment with the pre-training distribution, which can adversely affect the final quantization perfor- mance. Another challenge is the computational cost of searching through the vast space of potential calibration sets. brute-force search would require exploring m" possibili- ties, which is infeasible. Greedy search strategies, although more efficient, may suffer from local optima, highlighting the need for more sophisticated but efficient search methods.
Specifically, during the calibration dataset search, we retain only w branches S = { ...,,...,S"" each with a length of l. When generating candidate sequences length / + 1, each branch St expands to potential sequences within the space St @ V. The pruning evaluation metric for these sequences is defined as:
Self-Sampling. To address the challenge of data availability, we leverage the self-sampling capabilities LLMs to con- struct calibration data. This data-free approach relies solely on the model's vocabulary and is naturally consistent with the model's learned language distribution (Liu et al., 2023). Furthermore, during the self-sampling process, historical probabilities and expert distributions are cached, eliminating redundant computations for perplexity and expert balance metrics. We define the historical cumulative preoobbbbyy of a sequence S as:
score(S]lu) = 1+ (Rs + log P(u[S)) + (M,
T
subject to V, (13)
where S one of S, and Rg corresponds to the cumu- lative probability defined Equation 11. The process of generating new set of w candidate sequences, S { S S.. is expressed as:
1
$ - arg lopk (w, score(()S)))) S|l
Rg
log(P(S) S1.-1,,
(11)
(14)
subject to V and 5 E 45,S" " sw),
5

---

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models
where arg lopk, (J(,, denotes the of w values of x that maximize (x)).
experts are selected and their outputs are weighted accord- ing their assigned probabilities. Traditional layerwise quantization employs Equation 7 to uniformly minimize the quantization error but overlooks the probabilities between samples and experts. We define this correction as affinity, asserting that it is equally important and should be taken into account during the quantization process.
As indicated in Equation 14 and Figure 5, the scores of can- didate sequences generated from the same input sequence S are influenced by the probability distribution produced by the LLM. Additionally, the expert balance metric affects all candidate sequences derived from S. By introducing an efficient search method for the calibration set, the search complexity is significantly reduced from (mm") to Owm)), and the risk of local optima is effectively mitigated.
Let E' be a specific expert and denote the set of tokens routed to this expert as X { 22 ...., Hbb, ), with the cor- responding affinity scores the gating network. The output for the i-th token processed by this expert can be expressed as
{(i, C2, ..., (b) provided by
Deferred Expert Imbalance Calculation. It is important to note that during the pruning process, as shown in Equation 14, the evaluation metric does not incorporate the candidate token v in the assessment of expert balance. This approach justified for several reasons:
c, E(r;).
(15)
E is FFN, which can be expanded into sequence of linear layers and an activation function such as ReLU. Here, we consider a representative FFN structure, leading to:
Unlike perplexity, which can be directly computed from the probability distributions output by LLMs, cal- culating the expert distribution for each token in the vocabulary requires iterating over all possible tokens, computationally expensive process. Since the distribu- tion of the current sequence is already known, deferred computation incurs minimal additional cost.
y = cs ((xW o f(xWoat))) Wdown), (16)
where f denotes the activation function, W denotes the pa- rameter matrix. Because of the predominantly linear nature of the FFN and the quasi-linear property of ʃ, the expression above can be reformulated as follows:
The pruning process relies primarily on the predicted probabilities of the LLM rather than on expert balance.
Including expert distributions during pruning of the next token is inappropriate, as it may lead to semantic misalignment or incoherence.
y/: = ((,,WW @ f( W dowwn
y: ((=Wup) @
of f(x [aooe))) (c Wroww))
(17)
As demonstrated in Equation 14, the deferred calcula- tion actually performs branch-level pruning, thereby ensuring the creation of an expert-balanced balanced calibration set on the premise of maintaining the perplexity.
y: = ((xWPP) @ f(e.: Wdown
This shows that the token-expert affinity G propagates through every layer of the expert network. When focus- ing specific linear layer, G can be directly integrated into the layer's operations. In other words, different tokens exert a gate-aware influence on the weights of the same expert, with e acting as a scaling factor that modulates the contribution of each token's 's input features to the linear layer.
frequency.
pruned
0.0
0.1
selected
0.5
0.1,0.9 -.- probability
0.5 0.9
expert
0.4
Affinity-aware quantization error. Traditional quantiza- tion methods for LLMs have not taken into account the affinity awwre property. Here, we incorporate the gating coefficients into layer-wise calibration for the first time by redefining the quantization loss for W as
0.5
0.9 0.9
0.0
0.1
0.1
0.9
0.5
0.8
0.0
0.5
0.9
0.5
0.8
0.5
0.9 0.5
n
Wa, 2
resuency
0.4
0.9
0.5
0.8
0.4
L(W)
Σ c Wxx 1=0
(18)
0.0
expert
For PTQ methods based on quantization error, such as AWQ, Equation 18 incorporates token-expert affinity into the quantization process. Unlike the original implemen- tation, which treats all tokens equally during calibration, our affinity-aware metric emphasizes tokens with higher affinities, thereby reducing the overall quantization error for influential tokens.
Figure 5. Illustrative diagram of EBSS. The expert distribution and cumulative probabilities jointly guide the path searching.
4.3. Affinity-Gudedd Quantization
In an MoE layer, the gating network assigns a probability score to each expert based on the input. The most relevant
6

---

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models
Table Results RTN, Omniquant, AWQ, GPTQ. Quarot+GPQQ and ours MoFQuant with 4-hit Weight Quantization among Tasks on Owen-MoF-144B, 14B, DeepSeckMooF-168 and -extral-&77B. where denotes MoFQuant based AWQ, ++ denotes MoFQuant based on Quarot+GPTQ. Notably, except for our proposed MoËQuant, other methods utilize Wikitext2 as the calibration dataset, which leads to overfitting Wikitext2. Perplexity measured the C4 dataset more accurately reflects the performance of different methods.
PPL
ACCURACY
MODEL
METHOD
WIKI
HUMAN EVAL
HELLA SWAG
OPEN BOOKQA
MATH
C4
MMLU
GSM8K
BOOLQ
AVG.
TEXT2
QA
7.22
9.30
59.60
32.32
62.55
79.82
57.96
30.40
35.77
51.20
RTN
10.83
12.49
48.10
14.63
16.07
72.11
51.42
25.80
30.08
36.89
QWEN-
OMNIQUANT
7.67
9.98
56.30
31.71
52.39
78.20
56.58
29.40
33.63
48.31
MoE-14B
AWQ
8.59
10.93
51.63
20.73
36.77
71.96
54.78
30.40
31.39
42.52
MOEQuant*
8.77
10.67
52.33
22.10
42,22
74.52
54.92
30.40
33.44
44.27
GPTQ
8.00
10.99
53.70
20.73
22.82
73.52
52.70
29.40
28.27
40.16
QUAROT+GPT
7.43
10.11
57.90
28.05
56.25
78.77
56.54
29.00
36.48
49.00
MOEQuant
7.55
9.62
58.30
29.87
58.38
78.04
56.87
30.20
35.50
49.59
FP
6.51
9.04
44.60
26.83
26.16
72.72
38.06
32.20
31.49
40.86
RTN
7.47
10.01
36.10
18.90
10.54
70.21
55.76
30.60
28.87
35.85
DEEPSEEK- MoE-16B
OMNIQUANT
6.79
9.49
43.50
21.95
18.65
73.82
56.67
32.40
31.02
39.72
AWQ
6.80
9.50
40.57
25.00
17.06
71.65
56.42
32.20
31.76
39.23
MOEQuant*
6.94
9.32
41.20
25.00
18.90
71.98
56.79
32.12
31.82
39.68
GPTQ
6.82
10.35
39.60
21.34
11.60
72.14
56.05
30.60
30.35
37.38
QUAROT+GPTV GPTQ
6.66
9.39
40.60
22.56
19.18
72.17
57.03
30.60
30.95
39.01
MOEQuant
6.78
9.22
42.20
25.00
19.18
73.49
57.20
31.40
31.66
40.01
FP
3.84
6.87
70.50
32.93
65.88
85723
64.88
35.80
42.4T
56.80
RTN
5.41
8.13
62.20
28.05
27.90
80.85
61.73
32.20
37.35
47.18
MIXTRAL- 8x7B
OMNIQUANT
4.19
7.20
68.10
34.75
57.01
84.13
63.03
33.00
41.91
54.56
AWQ
5.01
7.98
62.75
25.00
38.67
79.97
62.11
33.60
38.43
48.64
MOEQuant*
5.15
7.84
64.66
25.45
50.66
81.03
62.73
34.00
39.77
51.19
GPTQ
4.84
8.08
64.30
24.39
42.15
83.03
58.50
32.00
37.52
48.84
QUAROT+GPTQ
4.03
7.67
68.50
27.60
57.92
84.22
64.08
30.60
41.07
53.42
MOEQuant"
4.12
7.34
69.60
32.15
61.79
84.98
64.05
33.60
42.95
55.58
Gate-aware Hessian statisties. contrast to Equation 7,
DeepSeck-MoE-I6B (Dai et al., 2024), Qwen-MoE-
which assumes equal contributions from all tokens to the Hessian, the affinity-aware quantization loss (Equation 18) leads to a more reasonable Hessian computation:
14B (Qwen, 2024) and Mixtral-8x7B (Jiang et al., 2024). In addition, we compare instruction-tuned models to demon- strate the effectiveness of our method. Beyond standard per- plexity evaluations on Wikitext2 (Merity, 2016) and C4 (Raf- fel et al., 2020), we evaluate the proposed MoEQuant on various reasoning tasks, including MMLU (Hendrycks et al., 2020), BoolQ (Clark et al., 2019). HellaSwag (Zellers et al., 2019), Openbookqa (Mihaylov et al., 2018), and MathQA (Amini et al., 2019). Furthermore, MoFQuant using the HumanEval (Chen et al., 2021) and GSM8K (Cobbe et al., 2021). HumanEval evaluates code generation capabilities, while GSM8k assesses multistep mathematical reasoning skills.
= X
/ X Ve) (x x T
(19)
For Hessian- based PTQ methods (e.g., GPTQ), the im- proved Hessian incorporates token-specific weighting better capture the operational dynamics of MoE layers. As result, tokens with higher gating coefficients exert greater influence when computing sensitivity metrics, which guide weight updates and help minimize quantization error.
evaluate
The Full Algorithm. Finally, we present the full pscu- docode for EBSS and AGQ in Algorithm and Algorithm 2, including the optimizations discussed above.
Baseline. Our primary baselines consist of vanilla RTN and the PTQ methods for LLMs: AWQ (Lin et al., 2023) and GPTQ (Frantar al., 2022). Despite requiring parameters training, we still incorporate OmniQuant (Shao et al., 2023) as a baseline for comparison. For calibration, 128 segments from the Wikitex12 dataset are selected. Floating-point re- sults are provided as references.
5. Experiments
5.1. Setup
We employ weight quantization for LLMs using symmet- ric uniform quantization with per-channel granularity. All experiments are performed on NVIDIA A6000 GPUs. As MoEQuant is an efficient post-training quantization (PTQ) framework, it obviates the need for any fine-tuning.
Implementation Details. For the three complex reason- ing tasks, MMLU, GSM8K, and HumanEval, conduct evaluations based their official repository. For sev- eral other shot tasks, we use the open-source tool
Models and Datasets. We conduct experiments on

---

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models
(version 0.4.4) (Gao al., 2024) for assessment. In experiments involving AWQ and OminiQuant, adapt their official repository to support the three MoE models. In addition to the official GPTQ re- sults in the MoE architecture, we also combine GPTQ with an equivalent Hadamard transformation to eliminate out- liers in the weights, consistent with the implementation in QuaRot (Ashkboos et al., 2024), while avoiding any online transformations.
pabilities of the model and has become necessary pro- for deployment of LLMs in different scenarios. The quantization of instruction-tuned models is often more chal- lenging than that of base models. We perform benchmark tests on Qwen-MoF-- 14B-Chat (Qwen, 2024) and DeepSeek- MoE-I6B-Chal (Dai et al., 2024), covering three tasks. For Qwen-MoE- 14-6--14-Chaal MoEQuard
consistently main- tains more than 94% full- pull-sreeiiion performance, with most of the original reasoning ability effectively restored. As shown in Table 2, previous methods face more significant accuracy degradation on instruction-tuned models for code generation and mathematical reasoning tasks. For ex- Table 3. Avg score of our methods ablation study on Qwen, DeepSeck and Mixtral MoE models across tasks, the baseline method is GPTQ.
5.2. Results
Comparison results. We conduct a comprehensive com- parison of quantization performance across various LLMs and datasets. As shown in Table I, the results of nine tasks demonstrate that our method, MoEQuant, exhibits superior performance compared to other methods for MoE LLMs. Notably, although GPTQ achieves lower perplexity on Wiki- text2(likely due to overfitting from using Wikitext2 for cali- bration), its performance on C4 and other tasks notably weaker. In contrast, MoEQuant outperforms GPTQ and AWQ in most tasks, showing substantial improvements in both perplexity and task-specific scores. On average, MoE- Quant exceeds the original performance by 1% across all three models, as measured by the average score over seven tasks. In particular, on HumanEval and GSM8k, where other methods degrade the model's reasoning ability after quantization, integrating MoEQuant effectively preserves this ability in generation tasks, achieving results comparable to full-precision models. This is particularly important reasoning in complex tasks such as HumanEval is crucial for real-world applications, further highlighting the practical relevance of MoFQuant's performance.
MODEL
EBSS AGQ
AVG ACCURACY
FP
51.20
X
X
49.00
QwEN-MoE-14B
X
4
49.02
X
49,21
V
49.59
FP
40.86
X
X
39.01
DEEPSEEK MEEEEK--OE-LLLE 16B
X
7
39.50
39.87
40.01
FP
56.80
X
X
53.42
MIXTRAL-8X7B
54.24
55.15
55.58
ample, Quar experienced a 29% accuracy drop in HumanEval for Owee--MoE-14B-Chan With the integration of MoEQuant*+ the accuracy surpasses the full- precision model, further demonstrating the effectiveness of EBSS and AGQ in improving quantization performance. More detailed results on perplexity and reasoning tasks can be found in the Appendix Table 10.
Table Results of RTN, AWQ, GPTQ and MoEQuant with 4-bit weight quantization among 3 tasks on Qwen, DeepSeek and Mix- tral MoF. instruction-tuned models, where denotes MoFQuant based on AWQ. ++ denotes MoEQuant based on GPTQ.
MODEL
METHOD
MMLU
GSM8K
Ablation results. MoEQuant enhances generalization and reasoning abilities MoE LLMs through two primary methods: EBSS and AGQ. We conduct decomposition ex- periments in Table 3. The results demonstrate the effec- Table 4. Impact of Calibration Datasets on Expert Balance and Quantization Performance deepseek- Mooeeek-MoE-lle 16b LLMs. The ex- pert balance std denotes the standard deviation on frequency of expert.
EVAL
FP
59.00
21.34
30.71
RTN
43.00
7.32
9.70
QWEN-
AWQ
52.06
12.20
17.74
MoE-
MoEQuand
53.22
18.92
22.34
CHAT
GPTQ
51.30
10.98
16.22
QUAROT+ +GPT
57.30
15.24
26.08
MoEQuantth
58.00
21.95
29.11
FP
48.90
24.39
54.28
CALIB DATASET
EXPERT
AVG ACCURACY
RTN
41.40
10.41
28.88
MODEL
DEEPSEEK-
AWQ
46.33
18.90
39.88
BALANCE STD
M6B-16B-
MoEQuant*
46.80
19.20
47.42
FLOAT
40.86
CHAT
GPTQ
43.80
32.93
35.78
RTN 0.0427
35.85
QUAROT+GPTQ
46.60
13.41
47.08
DEEPSEFK- MOE-16B
WIKTEXT2
39.01
MoEQuantt+
47.60
21.95
48.97
HUMANEVAL
0.0877
38.90
GSM8K
0.0928
38.88
EBSS
0.0052
39.87
Experiments of instruction-tuned models. Instruction fine-tuning can significantly improve the application ca-
tiveness of both proposed methods individually, each con-
8

---

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models
tributing improved accuracy. Notably, their combination yields synergistic effect, leading significant enhance- ment in average precision that surpasses the sum of their individual contributions. For EBSS, we perform an ablation study to examine the impact of two key hyperparameters: temperature т and branch number w. The best performance achieved when is set to 1.2, and we set w to balance effectiveness and efficiency. More detailed results can be seen in Appendix A.2. Meanwhile, we compare the per- formance between the fixed dataset and our EBSS method in terms of mean accuracy. The results in Table 4 clearly demonstrate that EBSS significantly improves the balance of expert sampling, as evidenced by notable reduction in standard deviation, while also achieving the highest mean accuracy.
Speedup and memory savings. The motivation behind MoEQuant is compress MoE. LLMS to lower bitwidth, thereby reducing both latency and GPU memory usage dur- ing inference while preserving accuracy to the greatest ex- tent, ensuring practical applicability. As shown in Table 6, MoEQuant achieves an average inference speedup of over 1.2x and memory savings exceeding 3.2x, demonstrating significant improvements in inference efficiency. These advancements enable the deployment of MoE LLMs consumer-level devices, such the Nvidia 4090 GPU.
6. Conclusion
We propose MoEQuant, a framework designed to address the unique challenges of quantizing MoE LLMs. By in- corporating Expert-Balanced Self-Sampling and Affinity- Guided Quantization, MoEQuant extends traditional quanti- zation methods effectively handle both the uneven distri- bution of calibration samples among experts and the token- expert affinity variations introduced by gating units. Experi- mental results show that MoEQuant achieves near-floating- point accuracy even with low-bit quantization and signifi- cantly improves generalizability, particularly in instruction- finetuned models. These results underscore potential to substantially reduce model size and computational require- ments, making MoE LLMs more feasible for deployment in resource-constraieed environments.
Table Average scores 3-bit on DeepSeek Mixtral MoE models, where + denotes MoEQuant based on AWQ. ++ denotes MoBQuant based on GPTQ
MODEL
BITWIDTH
METHOD
AVG.
FP
40.86
3
RTN
20.17
DEEPSEEK- MoF-16B
3
AWQ
22.20
3
MoFQuant"
26.65
QEAROT+GPTO
35.85
3
MoEQuant
36.47
56.80
18.64
MIXTRAL-
3
AWQ 36.05
8x7B
3
MoEiQunnt 39.30
3
QUAROT+GPTQ 45.03
3
MoFQuant 49.75
Lower bitwidth. We evaluate the generalizability of our approach under lower bitwidth settings for DeepSeek-MoE: 16B and Mixtral-8x7B. As shown in Table 5, among the tested methods, MoEQuant* and MoEQuant** consis- tently achieve the highest average scores. These findings demonstrate that MoEQuant provides superior performance compared other quantization methods, effectively main- taining higher accuracy even at lower bitwidth. Full results provided in Appendix 5.
Table Speedup and memory saving of 3 MoE LLMs, compared between our 4-bit implementation and FP16. All tests were con- ducted Nvidia A6000 GPUs.
DECODER SPEED (TOKENS/SEC)
MODEL
SPEED
FP
QUANTIZED
UP
QWEN-MOE-144 14B
8.35
10.60
1.27
DEEPSEEK- SEREEEK-MOEIIIE
20.81
24.45
1.17
MIXTRAL-8X77
10.24
21.25
2.08
MEMORY USE (GB)
MODEL.
MEMORY
FP
QUANTIZED
SAVING
OWEN-MOF-14B
27.88
8.51
3.28
DEEPSEEK-MON MOE-166
32.23
9.87
3.27
MIXTRAL-8X7B
89.64
23.97
3.74

---

MoEQuant:
Enhancing Quantization for Mixtere-of-Experts Large Language Models
Impact Statement
Deisenroth, M. and Ng. J. W. Distributed gaussian pro- cesses. In International conference on machine learning, pp. 148 1481-1490, PMLR, 2015.
MoEQuant addresses the unique quantization challenges of Mixture-of- Expeeetss (MoE) LLMs by tackling inter-expert and intra-expert imbalances, ensuring efficient low-bit quan- tization while preserving model accuracy. By integrat- ing Expert- Balanced Self-Sampling (EBSS) and Affinity- Guided Quantization (AGQ), MoEQuant significantly en- hances calibration balance and token-expert interaction mod- eling, outperforming existing PTQ methods in generaliza- tion and reasoning tasks. Experimental results demonstrate that MoEQuant achieves 3.2x memory savings, 1.2x in- ference speedup, and substantial accuracy gains, making MoE LLMs more practical for deployment on consumer- grade GPUs like the Nvidia RTX 4090. This work advances the scalability and accessibility of MoE models, bridging the gap between peeh-performacee language modeling and efficient deployment.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, Lim. int8 (): 8-bit matrix multiplication for transformers scale. arXiv preprint arXiv:2208.077 070000.,.73.. 2022.
Eigen, D., Ranzato, M., and Sutskever, I. Learning fac- tored representations in deep mixture of experts. arXiv preprint arXiv: 1312.431 2013.
Fedus, W. Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120):1-39 2022.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323, 2022.
References
Aljundi, R., Chakravarty, P., and Tuytelaars, Expert gate: Lifelong learning with network of experts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 10666-337,,
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, I.., Hsu, J., Le Noac'h, h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C.,, Phang,
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika,
L.. Tang, E., Thite, A., Wang, B., Wang. K., and Zou, A. A framework for few-shot language model evaluation,
Amini, A., Gabriel, S., Lin, P., Koncel-Kedee Kedziorski, R., Choi, Y., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. arXiv preprint arXiv: 13319, 2019.
07 2024. URL https: zenodo. org/ httgg//cord/////////// 12608602.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D.., and Steinhardt, J. Measuring sive multitask language understanding. arXiv preprint 2:Xiv:2009.03300 2020.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456, 2024.
Jacobs, R. A., Jordan, M. I., Nowlan, J., and Hinton, G. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 87, 1991.
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, P.D.D. Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377. 2021.
Jiang. A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, S., Casas, D. d. I., Hanna, B., Bressand, F., al. Mixtral of experts. arXiv preprint arXiv:2401.00088 2024.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 10044, 2019.
Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2): 181 181-214, 1994.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, I.., Plappert, M., Tworek, J., Hilton, J., Nakano, R., al. Training verifiers to solve math word problems. arXiv preprint 2r. 2021.
Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.00662. 2023.
Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., J., Zeng, W., Yu, X., Wu, Y., et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts lan- guage models. arXiv preprint arXiv:2401.06066, 2024.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
10

---

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models
LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. In Advances in Neural Information Processing Systems, 1989.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 2017.
Lin, J. Tang, J., Tang. H., Yang, S., Dang, X., and Han, S. Awq: Activation-aware weight quantization for IIm compression and acceleration. arXiv preprinl arXiv:2306.00978 2023.
Xiao, G., Lin, J., Seznec, M., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10433 2022.
Liu, A., Feng, B., Wang. B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04444 2024a.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint
2019.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., al. Deepseck- v3 technical report. arXiv preprint arXiv:2412.19437, 2024b.
Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V. LIm-qat: Data-free quantization aware training for large language models. arXiv preprint 2rXiv:2305./7888 2023.
Lu, C.-P. The race to efficiency: new perspective on ai scaling laws. arXiv preprint wrXiv:2501.02156, 2025.
Merity, The wikitext long term dependency language modeling dataset. Salesforce Metamind, 2016.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv: (809:00899078 2018.
Qwen. -weel.55moeea22.75 hitps: httss///hgggnngaeeeee co/ Ses/wwe//Qwees M-E---E---- A2. 7B, 2024. Accessed: [n. d.].
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M.. Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 2020.
Shao, W., Chen, M., Zhang. Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large lan- guage models. CORR, abs/2308. 110088.31133 13137, 2023.
Theis, L. and Bethge, M. Generative image modeling using spatial Istms. Advances in neural information processing systems, 28, 2015.
van Baalen, M., Kuzmin, A., Nagel, M., Couperus, P., Bas- toul, C., Mahurin, E., Blankevoort, T., and Whatmough, P. Gptvq: The blessing of dimensionality for Ilm quanti- zation. arXiv preprint arXiv:2402, 153/9, 2024.

---

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models
A. Appendix
A.I. Derivation T and objective function
EBSS optimizes:
El0.1|
t00)
D arg minp D) (M,
D)
where т balances pretraining distribution alignment (via PPL) and expert balance (via σ). Smaller 7 prioritizes expert balance but may increase PPL.
There is no risk of ballncilnn as the cost of PPL because PPL is bounded, we prove it below:
When ignoring a, we denote PPL p. p typically remains close 1 due self-sampling. We have the following:
D PPL((M, (MMD) sp+
since 2 0,
we have: PPL(M, D) < -
The hyperparameter r is determined via ablation studies below.
A.2. Ablation study
In this section, we provide the complete comparison of results for our method EBSS and AGQ. As shown in Table 7, taking DeepSeck-MoE-166 as an example, when applied alone, EBSS brings nearly 1.3% improvement, while AGQ brings about 2%. When both techniques are combined, the performance improves significantly by 2.6%, which is similar on the Qwen-MoE-14B, This demonstrates the benefit of combining EBSS and AGQ, as the combined method outperforms both individual methods. It inevitable that for Mixtral 8x7b, the result of AGQ is not better than that of convential GPTQ, but the combination result is still the optimal one.
Table 7. Complete comparison of our methods ablation study Qwen, DeepSeek and Mixtral MoE models across 9 tasks, the baseline
method is GPTQ.
SCORE
MODEL
EBSS
AGQ
WIKI TEXT2
HUMAN EVAL
HELLA SWAG
OPEN BOOKQA
MATH
C4
MMLU
GSM8K
BOOLQ
AVG.
QA
FP
7.22
9.30
59.60
32.32
62.55
79.82
$7.96
30.40
35.77
51,20
X
7.43
10.11
57.90
28.05
56.25
78.77
56.54
29.00
36.48
49.00
QWEN-
7.44
10.09
57.30
29.27
56.41
76.45
56.86
31.00
35.87
49.02
M4E-14B
7.56
9.62
58.80
27.44
56.71
78.77
56.73
30.80
35.27
49.21
7.55
9.68
58.30
29.87
58.38
78.04
56.87
30.20
35.50
49.59
FP
6.51
9.04
44.60
26.83
20.16
72.72
38.06"
32.20
31.49
40.86
6.66
9.39
40.60
22.56
19.18
72.17
57.03
30.60
30.95
39.01
DEEPSEEK-
6.66
9.38
41.60
23.17
17.89
74.52
57.30
31.20
30.88
39.50
MoE-16B 16B
6.77
9.22
44.00
23.78
18.19
73.24
$7.21
31.80
30.92
39.87
6.78
9.25
42.20
25.00
19.18
73.49
$7.20
31.40
31.66
40.01
FP
3.84
6.87
70.50
32.93
65.88
85.23
64.88
35.80
42.41
56.80
4.03
7.67
68.50
27.60
57.92
84.22
64.08
30.60
41.07
53.42
MIXTRAL-
4.04
7.64
68.30
29.54
60.12
83.36
64.04
32.80
41.54
54.24
8x7B
4.10
7.38
69.10
31.19
60.50
84.83
64.21
34.20
42.01
55.15
4.12
7.38
69.60
32.15
61.79
84.98
64.05
33.60
42.95
55.58
In EBSS, we conduct ablation study to examine the impact of two key hyperparameters: temperature 7 and branch number w. The controls the significance of expert balance in the sentence probability distribution, while w determines the diversity the generated sentences. Although increasing w improves sentence diversity, also incurs higher computational across seven tasks, as shown in Table 8 and Table 9. When 1.2, the average score across datasets is maximized, Similarly, setting w yields optimal results, with further
costs. The experiments are performed on DeepSeck-
12

---

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models
increases in offering only marginal score improvements while significantly increasing generation time.
Table 8. Different avg scores across 7 tasks for DeepSeek-MoF-i with MoFQuant
1.0
1.1
1.2
1.4
1.5
AVG. 39.82
39.89
40.01
39.98
39.69
39.71
Table Different branch mumber w on avg scores across tasks DeepSeek-Mon MoE- 16B with MoEQuant+++
w
2
7
8
20
30
40
50
AVG.
39.77 39.80 40.01 39.98 40.01 40.00
40.00 40.01
40.00
40.10
40.07
40.08
40.11
A.3. Full results
In this section, we provide a comprehensive presentation of our results across various datasets to complement the main paper. Specifically, the results include the following.
Complete comparison on two perplexity and seven accuracy tasks for instruction-tuncd MoE LLMs: Qwen-MoE-14B- chat, and DeepSeek-Mon 105-166-chtt.
Complete comparision with the lower bit 2 perplexity and 7 accuracy tasks for DeepSeek-MoE-11e and Mixtral- 8x7B.
Table 10. Complete comparison of RTN, AWQ, GPTQ, Quarot+GPTQ and ours MoEQuant with 4-bit weight quantization among 9 tasks on Qwen, DeepSeek and Mixtral MoE instruction-tune models, where denotes MoEQuant based on AWQ. +++ denotes MoEQuant based on GPTQ.
PPL
SCORE
MODEL
METHOD
WIKI
C4
MMLU
HUMAN
GSM8K
BOOLQ
HELLA
OPEN BOOKQA
MATH
AVG.
TEXT2
EVAL
SWAG
FP
8.07
9.74
59.0
21.34
30.71
81.31
59.33
31.00
34.91
45.37
RTN
12.81
14.03
43.00
7.32
9.70
71.13
51.41
24.40
28.81
33.68
QWEN-
AWQ
9.97
11.90
52.06
12.20
17.74
74.74
55.37
30.40
31.46
39.14
MOE-145
MOEQuant
10.12
11.55
55.34
13.60
20.87
76.22
56.64
30.60
32.50
40.82
CHAT
GPTQ
9.16
12.24
51.30
10.98
16.22
72.91
51.97
28.00
26.67
36.86
QUAROT+GPTO
8.38
10.78
57.30
15.24
26.08
78.92
58.72
31.40
34.17
43.19
MOEQuard
8.65
10.21
58.00
21.95
29.11
79.11
58.53
33.20
34.77
44.95
FP
7.35
9.96
48790
24.39
34.28
79.8T
60069
33.40
34.27
47.96
RTN
8.63
11.06
41.40
10.41
28.88
75.84
$7.59
31.40
29.04
39.22
DEEPSEEK-
AWQ
7.72
10.49
46.33
18.90
39.88
78.20
58.97
33.80
32.86
44.13
M6B-16B-
MOEQuant*
7.85
10.23
46.40
18.90
45.41
78.20
59.03
33.60
33.14
44.95
CHAT
GPTQ
7.72
11.52
43.80
32.93
35.78
76.82
57.21
33.60
33.30
44.78
QUAROT+GPTQ
7.55
10.24
46.60
13.41
47.08
78.87
59.64
33.20
32.76
44.50
MOEQuant"*
7.70
10.08
47.60
21.95
48.97
79.20
59.30
33.80
32.60
46.20
13

---

MoEQuant: Enhancing Quantization for Mixture-of-Expert Large Language Models
Table 11. Complete results of of lower-bit among 9 tasks on DeepSeek and Mixtral MoE models, where + denotes MoBQuant based on
AWQ. ++ denotes MoFQuant based GPTQ.
BIT WIDTH
PPL
SCORE
MODEL
METHOD
HUMAN EVAL
MATH
C4
MMLU
GSM8K
BOOLQ
AVG.
TEXT2
SWAG
BOOKQA
QA
6.51
9.04
44.60
26.83
20.16
72.72
58.06
32.20
31.49
40.86
3BIT
RTN
26352
32357
24.8
0.00
1.59
51.62
26.18
15.60
21.44
20.17
DEEPSEEK-
3BIT
AWQ
4622
5505
27.80
1.90
2.88
53.20
27.97
17.80
23.86
22.20
MoE-16B
3BTT
MOEQuante
5100
4924
33.20
8.72
10.44
59.24
29.22
20.60
25.14
26.65
QUAROT+GPTQ
7.17
11.66
37.30
17.68
11.60
72.31
53.68
27.80
29.72
35.85
3BIT
MOEQuant
7.55
10.88
40.00
20.12
12.81
69.72
54.09
29.00
29.61
36.47
3.84
6.87
70.50
32.93
65.88
85,23
64.88
35.80
42.41
56.80
3BIT
RTN
44944
51241
25.30
0.00
0.00
41.52
25.61
18.40
19.66
18.64
MIXTRAL-
3BIT
AWQ
7.38
13.13
45.80
10.37
10.39
75.23
53.04
28.00
29.55
36.05
8x7B
3BIT
MOEQuantt
8.77
11.44
49.40
14.44
17.29
77.22
54.29
30.10
32.34
39.30
3BIT
QUTR
4.64
9.12
57.80
22.56
22.59
79.82
61.30
30.40
40.80
45,04
3BIT
MOEQuant
4.90
8.24
64.10
28.05
43.21
82.81
60.07
31.20
38.82
49.75
14

---

end for MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models
Algorithm 1 EBBSS-based Sentence Generation with Expert-Balancee Pruning
input M: MoE-LLM model
NS: Number of sentences
Seqlen: Sequence length
V: Vocabulary size
EBSS width
7: Expert balance temperature
output Fiinalssentenee Generated sentence list
1: Final sentences 4-0 0
2: for [1, do
3: First.toke < Random(V)
4: candidales < [first_token]
UInitialize candidate pool)
5: for j [1, Seglen - 1] do
6: expanded aepaanded_candidatees condidates t- 0
7: for each candidate E candidates do
8: logils, EB scores &- M (candidaal)
9: RS - CCmppurCCumlaaivPPooiiitttantit
{Eq. 113
for each v E V do
11:
new canddddde condidale prob Softmax (logits))
12:
13:
score +
RS (o
(M.new candidare)
(Eq. 13)
i+1
14:
15:
end for end for candidates Tooooe(rpedddd..cnnniii htlfe/ppmmdddccmmdddteeeeee candidates, w, key score)
(Keep top-w candidates)
19:
best_sentence arg maXeecondidates Score(c)
20: Final Snnaleeentenccessppeee(cece
21: end for
22: return Final senlences
23: Subroutine: ComppueeCCmmlaaiieeoohhdiitgseee
24: return E lsequencel log sesquueecceeeeeeeeeee 1..6 - 1))
(Eg. 113
25: Subroutine: ComputeEEpperrBaaaeeeseqennneee
26:
expert.usage t- cm (espe)) 21)
27: a
{Eg. 4}
28: return
29: Subroutine: Score(candidate)
30: RS +- CompuueCCmmlaaieeeooobbbiiccccad
32: return
R&
(Combined scoring function]
candidate
15

---

MoEQuant: Enhancing Quantization for Mixture-of-Expert Large Language Models
Algorithm 2 Enhanned GPTQ Quantization for MoE LLMs
input M: Pre-trained MoE-LLM
Deat: Calibration dataset (EBSS-generated)
bits: Target quantization bits (c.g.. 4)
7: Expert balance temperature
w: EBSS width parameter
output Quantized model M,
1: for each MoE layer in M do
2: & + Get expert list of layer I
3: H. Initialize weighted Hessian matrix
4: C Initialize affinity cache
5:
for each sample Deat do
6: gates + Compute gating scores (Ir) for &l
{Expert affinities)
7: topk experts t- Select top-k experts based on gates
8:
for each expert E E topk.erperts do
9:
C gates EE] a E(x)
(Get expert affinity) (Expett activation {Cache affinities) {Eq. 19}
11:
E + C U (c) [ + H E + aa
12:
13:
end for
14:
end for
15:
for each expert E in & do
WE E + Get expert weights
Sg + Compute sensitivity scores using H,[b]
18:
Qg Quantize We with:
(AGQ-GPTQ parameters
19:
Bit-width bils
20:
Sensitivity Sg
21:
Afflinity-wiighted Hessian H[[E] Replace E's weights with Qg
22:
end for
end for
25:
return M,
16


Reviews:
[{"review_id": "VsgKmGQxVI", "reviewer": "Reviewer_eTCn", "summary": "This paper introduces a framework named MoEQuant which designed to efficiently quantize MoE large language models while addressing calibration imbalances. It tackles two key challenges: the unequal distribution of calibration samples across different experts, and the varying affinities between tokens and their assigned experts. By integrating Expert-Balanced Self-Sampling to generate a balanced calibration dataset and Affinity-Guided Quantization to incorporate token-expert relationships into the quantization process, MoEQuant significantly reduces quantization errors. Extensive experiments demonstrate that MoEQuant not only maintains near-full-precision accuracy under low-bit quantization but also achieves faster inference speeds and substantial memory savings.\n\n## Update after rebuttal.\nThe additional explanation has indeed resolved most of my doubts and provided more comprehensive support for the arguments in this paper. I am inclined to accept this paper, but since I initially gave it a score of 3, which means leaning towards acceptance, I will maintain the score of 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.\n\n---\n>Why do some tasks (like HumanEval) benefit significantly from MoEQuant, while others show minimal or no improvement?\n\nThanks for your insightful observation. We classify tasks into:\n* **Simple  Retrieval Tasks** (e.g., BOOLQ HellaSwag) rely on direct answer based on surface-level knowledge and exhibit limited benefits from multi-expert collaboration. **However, we still reduces the FP accuracy gap by up to 54%(e.g., on DeepSeek-MoE-16B).**\n* **Challenging Resoning Tasks** (e.g., HumanEval GSM8K) involve continuous multi-step logical/mathematical reasoning, code generation, or structured problem-solving. MoE architectures excel here by assigning specialized experts to sub-problems, enhancing performance through collaborative reasoning.\nHowever, Applying PTQ to MoE LLMs for these tasks introduces two key challenges:\n    * Each expert must be sufficiently calibrated to ensure both functional accuracy and diversity.\n    * Mitigating error accumulation during multi-step decoding, which severely degrades accuracy.\n\n    **MoEQuant addresses them via EBSS(Expert-Balanced Self-Sampling) for comprehensive calibration of each expert and AGQ(Affinity-Guided Quantizatin) for improved quantization error modeling, achieving 5–24% relative accuracy gains.**\n>Why does AGQ alone underperform GPTQ on certain tasks, but perform better when combined with EBSS?\n\nAGQ optimizes quantization error via token-expert affinity weighting. However, **when used alone, imbalanced sample distribution among experts leads experts to receive fewer low-confidence samples,** increasing noise and variance in Hessian and causes evaluation variance. Despite this, AGQ still improves overall performance(1.2% relative gain on Mixtral).\n\n**Combining AGQ with EBSS ensures sufficient calibration for each expert, yielding a more stable Hessian with low variance**, enabling 4% relative gain on Mixtral. This synergy outperforms either method alone.\n> The paper notes dramatic accuracy drops with methods like RTN at 3-bit quantization. Could you discuss the reasons behind it and explain how MoEQuant effectively mitigates these issues?\n\nAs shown in [Table R1](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R1_eTCn.md), 3-bit quantization, limited to just 8 discrete values, constrains RTN performance. **Moreover, MoE LLMs suffer from inter- and intra-expert imbalances, exacerbating degradation.**\nMoEQuant addresses these imbalances via:\n1. EBSS: Enables comprehensive calibration of each expert, mitigating inter-expert imbalance.\n2. AGQ: Incorporates token-expert affinities into quantization loss, correcting intra-expert imbalance.\n\n**This dual approach in MoEQuant delivers SOTA 3-bit PTQ results**, yielding up to a 10.5% accuracy gain on Mixtral-8x7B (Table 3). \n>Could you provide additional theoretical or intuitive insights into how the temperature parameter (τ) affects the sampling process in EBSS?\n\nEBSS optimizes the objective:  $$\\mathcal{D}^* = \\arg\\min_{\\mathcal{D}} \\{ \n\\overbrace{\\text{PPL}(\\mathcal{M},\\mathcal{D})}^{\\in[0,1]} + \\overbrace{\\frac{\\sigma(\\mathcal{M},\\mathcal{D})}{\\tau}}^{\\in[1,+\\infty)}\n\\}$$where $\\tau$ balances distribution alignment (via PPL) and expert balance (via $\\sigma$). **Smaller τ prioritizes expert balance but may slightly increase PPL. However, PPL remains bounded, as proved below:** \n- When ignoring $\\sigma$，we denote $\\text{PPL}=p$, $p$ typically remains near 1 due to self-sampling(see Figure 4). We derive:$$\\mathcal{D}^* = \\text{PPL}(\\mathcal{M,D})+ \\frac{\\sigma(\\mathcal{M},\\mathcal{D})}{\\tau}\\leq p+\\frac{1}{\\tau}$$ since $\\sigma>=0$, we derive: $$\\text{PPL}(\\mathcal{M,D})\\leq p+\\frac{1}{\\tau}$$\n\nThe hyperparameter $\\tau=1.2$ is determined via ablation studies. Further experiments in [Table R2](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R1_eTCn.md) confirm its generalizability. \n> How might the quantization performance be affected if the activations deviate from this quasi-linear behavior?\n\nFor nonlinear function SiLU used in MoE LLMs, Equation 17 and AGQ remain effective. We support it by corresponding experiments：\n- **Numerical Error.** As shown in [Table R3](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R1_eTCn.md), the reformulated gate output: $$y_i\\approx\\left((xW^{up})\\odot f(c_ixW^{gate})\\right)W^{down}$$ remains high cosine similarity with true outputs across c values, approaching 1 for high-c experts . Top-k expert selection inherently prioritizes high-affinity tokens, further minimizing cumulative error.\n- **Task Performance Validation.** As Shown in [Table R4](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R1_eTCn.md), applying AGQ to the gate layers (which governs non-linear interactions) yields significant gains in both perplexity and downstream accuracy(1.2% gain for Qwen-MoE), confirming its effectiveness.", "claims_and_evidence": "The claims in the paper are generally well-supported by evidence. The authors clearly demonstrate that existing quantization methods perform poorly on MoE models, showing substantial performance degradation compared to full precision models. The paper provides compelling evidence that their proposed MoEQuant framework improves quantization performance across multiple MoE models (Qwen-MoE-14B, DeepSeek-MoE-16B, Mixtral-8x7B) and evaluation tasks. The performance improvements are substantial - showing gains of more than 10 points on HumanEval for DeepSeek-MoE-16B under 4-bit quantization. The ablation studies effectively demonstrate the individual contributions of EBSS and AGQ, showing that their combination yields the best results. The efficiency claims (3.2× memory savings and 1.2× inference speedup) are supported by benchmark data. **I did not find any claims without evidence**.", "methods_and_evaluation_criteria": "The proposed methods (EBSS and AGQ) and the evaluation criteria (datasets and tasks) introduced in the paper are reasonable, relevant, and well-suited to the specific challenges faced when quantizing Mixture-of-Experts (MoE) large language models. However, upon careful review, there is an aspect where the paper could be improved or clarified:\n- The paper supports its methods with comprehensive ablation studies that analyze the impact of key hyperparameters (e.g., temperature and branch number in EBSS) and comparisons across multiple models and bitwidth settings. This thorough experimental validation reinforces that the proposed methods are well-tailored to address the quantization challenges in MoE architectures.", "theoretical_claims": "I examined the derivations supporting the affinity-aware quantization error and the modified Hessian computation. These proofs extend standard PTQ formulations by incorporating token-expert affinity—where gating coefficients scale individual token contributions—into the error and sensitivity assessments. Overall, the algebraic steps appear sound under the assumption that the feedforward operations in the expert networks are quasi-linear (a reasonable approximation for activations like ReLU); however, this assumption might not hold for all activation functions, potentially limiting the generality of the results. No explicit calculation errors were found, though the reliance on these approximations suggests that further empirical validation is warranted to ensure that any deviations from quasi-linearity do not adversely impact the quantization performance.", "experimental_designs_or_analyses": "While the overall setup is well-crafted and appropriate, there are several areas that could benefit from further analysis, clarification, or improvement:\n- The authors present detailed quantitative results (Tables 1, 2, and 9), but certain observed phenomena are not thoroughly explained. For example: i. Why do some tasks (like HumanEval) benefit significantly from MoEQuant, while others show minimal or no improvement?\nii. Why does AGQ alone underperform GPTQ on certain tasks, but perform better when combined with EBSS?\n- Severe accuracy drops occur for some methods at 3-bit quantization (e.g., RTN). However, the authors do not explicitly analyze or discuss the reasons behind these dramatic performance degradations, nor clarify how MoEQuant mitigates these drops.\n- Temperature parameter τ selection: In the appendix, the authors show results for different τ values, but provide no theoretical justification for why 1.2 is optimal, nor clearly explain the physical meaning of how this parameter affects the sampling process.", "relation_to_broader_scientific_literature": "The paper’s contributions build directly upon and extend key ideas from both the quantization literature and prior research on MoE models. It leverages established methods such as GPTQ and AWQ—originally designed for traditional, dense LLMs—while addressing unique challenges inherent to MoE architectures, such as inter- and intra-expert imbalances. The proposed EBSS innovates on earlier load-balancing strategies by ensuring that calibration data more evenly represents the diverse set of experts in an MoE model, a concern that has been highlighted in previous MoE studies. Meanwhile, AGQ draws on and refines existing error-compensation techniques by incorporating token-expert affinity information, which is critical for accurately adapting quantization to the dynamic routing nature of MoE layers. Thus, by integrating ideas from both expert balancing and sensitivity analysis, the work not only situates itself within the broader context of quantization and model compression but also advances these fields by providing tailored solutions for efficient and accurate quantization of MoE-based LLMs", "essential_references_not_discussed": "I think the paper adequately covers the essential references needed to understand the context for its key contributions. But I am not very familiar with this domain.", "other_strengths_and_weaknesses": "The work robustly tackles calibration imbalances and reduces quantization errors, achieving near full-precision performance even under low-bit quantization. Extensive experiments demonstrate significant gains in accuracy, memory savings, and inference speed, while the integration of theoretical insights with practical validations underscores the overall significance and real-world applicability of the approach. But it also has some weaknesses:\n\n- The derivations assume quasi-linearity in expert network activations (e.g., with ReLU). Although this is reasonable in many cases, it might not hold for all activation functions, potentially limiting the generality of the theoretical results.\n- There are certain discrepancies in the experimental results that require further clarification:\nAs I said in the **Experimental Designs Or Analyses**, the paper does not fully explain why some tasks benefit more significantly from MoEQuant than others.", "other_comments_or_suggestions": "The paper is overall well-prepared, but here are a few additional suggestions and comments:\n\n- Further explain how the temperature parameter (τ) directly affects the sampling process in EBSS, and consider providing more intuition behind the choice of a τ value of 1.2.\n- Some experimental results, such as the variations observed in different tasks and the severe drops in accuracy for certain methods at low-bit quantization, could benefit from deeper analysis or discussion.", "questions_for_authors": "- The experiments indicate that MoEQuant leads to significant improvements on certain tasks while showing minimal gains on others. Can you elaborate on the underlying reasons for these variations? For example, are there characteristics of specific tasks that better exploit the strengths of MoEQuant?\n\n- The paper notes dramatic accuracy drops with methods like RTN at 3-bit quantization. Could you discuss the reasons behind these severe drops and explain how MoEQuant effectively mitigates these issues?\n\n- Could you provide additional theoretical or intuitive insights into how the temperature parameter (τ) affects the sampling process in EBSS? Specifically, how does varying τ influence the balance of calibration samples across experts, and why was a value of 1.2 deemed optimal?\n\n- Your theoretical derivations assume quasi-linearity in expert network activations (e.g., with ReLU). How might the quantization performance be affected if the activations deviate from this quasi-linear behavior (e.g., when using non-linear activations)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NY3RlKNxJ4", "reviewer": "Reviewer_NkUk", "summary": "MoEQuant is a novel quantization framework designed for Mixture-of-Experts (MoE) large language models (LLMs). It addresses the challenges of accuracy degradation encountered in traditional post-training quantization (PTQ) methods. The framework introduces two techniques: Expert-Balanced Self-Sampling (EBSS) and Affinity-Guided Quantization (AGQ). EBSS efficiently constructs balanced calibration datasets, while AGQ incorporates expert-sampling affinities into the quantization process. Experimental results demonstrate significant performance gains and efficiency improvements, making MoE models more practical for deployment on resource-constrained devices.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.\n\n---\n>The study focuses solely on weight quantization and does not address activation quantization. \n\nMemory constraints are the decisive factor for deploying MoE LLMs, which require orders of magnitude more memory than dense models (e.g., DeepSeek-MoE-14B retains 14B params in memory despite activating only 2.7B). **Therefore, we prioritize weight quantization because it significantly alleviates memory pressure,** achieving a 3.74× memory saving and a 2× speedup (see Table 4) for Mixtral-8x7B, thereby enabling deployment on consumer GPUs.\n\nTo further address your concern, we add W4A4 experiments. [Table R1](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R2_NkUk.md) show that MoEQuant improves accuracy by over 3% compared to QuaRot, demonstrating its versatility across diverse quantization scenarios.\n>MoEQuant is primarily an enhancement of existing quantization methods like AWQ and GPTQ. While it introduces novel techniques\n\nMoEQuant is a novel PTQ framework tailored for MoE LLMs rather than an enhancement, addressing key limitations of existing methods through three contributions:\n- **Problem Identification:** As shown in Table 1, previous PTQ methods to MoE LLMs causes severe accuracy drops(up to 17%), We attribute it to 2 key issues: inadequate calibration across experts(inter-expert imbalance) and inaccurate quantization error modeling(intra-expert imbalance) inherent in MoE quantization.\n- **Technical Innovation:** EBSS(Expert-Balanced Self-Sampling) ensures adequate expert calibration via probability-guided pruning and deferred imbalance evaluation, maintaining alignment with fp models while reducing search complexity (Sec. 4.2). AGQ(Affinity-Guided Quantization) incorporates token-expert affinities into quantization loss and Hessian statistics, enabling precise sensitivity analysis for MoE’s dynamic routing (Sec. 4.3).\n- **Empirical Validation:** MoEQuant seamlessly integrates with existing PTQ methods(AWQ,GPTQ,QuaRot), significantly improving accuracy. As validated in [Table R7](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R2_NkUk.md), it outperforms QuaRot by 5–23% on complex tasks like HumanEval and GSM8K, effectively closing prior performance gaps.\n\n>The baselines are not the latest and AWQ+GPTQ would be much better\n\nWe would like to clarify that, as described in Line 383, our GPTQ implementation equal 'QuaROt+GPTQ', a current SOTA training-free PTQ method, this ensures rigorous baseline. \n\nTo further address your concern, we have added a comparison with Omniquant in [Table R4-6](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R2_NkUk.md) and MoEQuant still works better.  We also manually implemented GPTQ+AWQ but obtained worse performance.\n> the ablation study in Table 5 shows that EBSS and AGQ improve the accuracy very little\n\nWhile Table 5 shows modest overall gains(1.2–4.1% relative), we emphasize that **MoEQuant delivers significant gains in instruction-tuned models and complex reasoning tasks(sensitive to quantization and critical for real applications).** As shown in [Table R2](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R2_NkUk.md):\n- On code generation(HumanEval) and mathematical reasoning(GSM8K) , MoEQuant achieves 4.69–9.84% relative accuracy gains in non-instruction-tuned models.\n- For instruction-tuned models, which better reflect deployment scenarios, MoEQuant demonstrates 17.22–23.57% relative gains.\n\n> Could you provide comparison of the time cost for MoEQuant and baselines?\n\nWe add a comparison of the time costs for MoEQuant and GPTQ. As shown in [Table R3](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R2_NkUk.md), Due to the incorporation of three cost-reducing techniques in EBSS—namely Probability-Guided Path Pruning, Cumulative Probability, and Deferred Expert Imbalance Calculation—**MoEQuant only increases a small amount of time overhead compared to GPTQ.**\n>Are the router layers of MoE quantized? how do these layers affect the performance of the quantized model?\n\nRouter layers are quantized. We did not find an obvious impact on the final performance. This may be due to the robustness of gating and the redundancy of expert knowledge.\n>Could you show the latest weight-only quantization methods on MoE models, such as OmniQuant and EfficientQAT\n\nIn [Table R4-R6](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R2_NkUk.md), we provide additional results comparing MoEQuant with OmniQuant. **While a direct comparison may be unfair since MoEQuant is designed for PTQ and OmniQuant involves training, MoEQuant consistently outperforms OmniQuant across all tested models.** This advantage arises because OmniQuant relies on a fixed calibration set, disregarding expert balance. EfficientQAT, being a global QAT algorithm, is beyond the scope of our comparison.", "claims_and_evidence": "There is clear and convincing evidence of the claims made in the submission.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the deployment of Mixture-of-Experts large language models.", "theoretical_claims": "This paper do not contains any  theoretical claims. In the section of  Expert-Balanced Self-Sampling,  this paper presents the optimization process, which is enough to demonstarte the challenge.", "experimental_designs_or_analyses": "The analysis of Quantization on MoE model is convinced, the datasets and benchmarks choosed in this paper are common.", "relation_to_broader_scientific_literature": "Previous LLM-PTQ methods (such as AWQ, GPTQ) focus on dense large language models, this paper proposes method for MoE-LLMs, which are different.", "essential_references_not_discussed": "[1] Shao, Wenqi, et al. \"Omniquant: Omnidirectionally calibrated quantization for large language models.\" arXiv preprint arXiv:2308.13137 (2023).\n\n[2] Chen, Mengzhao, et al. \"Efficientqat: Efficient quantization-aware training for large language models.\" arXiv preprint arXiv:2407.11062 (2024).", "other_strengths_and_weaknesses": "Strengths:\n1. MoEQuant introduces two innovative methods—Expert-Balanced Self-Sampling (EBSS) and Affinity-Guided Quantization (AGQ)—which address the unique challenges of quantizing MoE models, such as inter-expert and intra-expert imbalances.\n\n2. The framework achieves substantial performance gains, particularly in complex reasoning tasks like HumanEval, and enhances efficiency by reducing model size and computational requirements. This makes MoE models more practical for real-world applications.\n\n3. MoEQuant significantly improves the generalization ability of MoE models, especially when dealing with instruction-tuned models. It demonstrates robust performance across various tasks and datasets, outperforming conventional PTQ methods.\n\n4. By effectively compressing MoE models to lower bitwidths, MoEQuant facilitates the deployment of these models on consumer-grade devices with limited memory and computational resources, thus broadening their accessibility and scalability.\n\nWeaknesses：\n\n1. The study focuses solely on weight quantization and does not address activation quantization. This limited scope may overlook potential benefits and challenges associated with activations, which could be critical for overall model efficiency and performance.\n\n2. MoEQuant is primarily an enhancement of existing quantization methods like AWQ and GPTQ. While it introduces novel techniques, it does not represent a fundamentally new approach. This raises questions about the originality and potential for significant breakthroughs in quantization strategies.\n\n3. The experimental results are not much competitive, and the baselines are not the latest, and AWQ+GPTQ would be much better. And the ablation study in Table 5 shows that EBSS and AGQ improve the accuracy of the quantized model very little", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Could you provide comparison of the time cost for MoEQuant and baselines?\n\n2. Are the router layers of MoE quantized?  how do these layers affect the classification accuracy of MoE and the performance of the quantized model?\n\n3. Could you shows the latest weight-only quantization methods on MoE models, such as OmniQuant [1] and EfficientQAT [2].\n\n[1] Shao, Wenqi, et al. \"Omniquant: Omnidirectionally calibrated quantization for large language models.\" arXiv preprint arXiv:2308.13137 (2023).\n\n[2] Chen, Mengzhao, et al. \"Efficientqat: Efficient quantization-aware training for large language models.\" arXiv preprint arXiv:2407.11062 (2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3plkLTAG4j", "reviewer": "Reviewer_dN5E", "summary": "This paper proposes the MoEQuant framework, which aims to address the key challenges in the quantization of large language models (LLMs) with mixture of experts (MoE): inter-expert imbalance and intra-expert imbalance. Through two innovative methods, expert balanced self-sampling (EBSS) and affinity-guided quantization (AGQ), EBSS balances the expert utilization by self-generating calibration sets; AGQ introduces the affinity between samples and experts into the calculation of quantization error to optimize weight updates. Experiments show that MoEQuant significantly improves model performance under 4-bit quantization and achieves 1.2 times acceleration and 3.2 times compression in inference speed and memory usage, respectively, verifying its efficiency and universality.\n\n## update after rebuttal\nThe Authors have addressed all of my concerns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.\n\n---\n>paper does not provide direct experimental or theoretical proof that the poor MoE quantization effect is caused by load imbalance\n\nWe provide additional theoretical and experimental justifications below:\n- Inter-Expert Imbalance\n  - Theoretical Basis: [Ours derivation](https://anonymous.4open.science/r/MoEQuant-DDFD/asserts/inter-expert_balance.md) proves that expert balance is helpful for accurate Hessian.\n  - Experimental: [Table R1](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R3_dN5E.md) shows a positive correlation between expert balance and accuracy. EBSS(Expert-Balanced Self-Sampling) achieves 10x imbalance reduction and best accuracy. Table 5 shows EBSS reduces the accuracy gap by 49%.\n- Intra-Expert Imbalance\n  - Theoretical: Eq 15 formalizes expert aggregation, showing token-expert affinity varies. Eq 17 leverages linear equivalence for layer-wise quantization to anticipate affinity shifts.\n  - Experimental: Table 5 shows that on DeepSeek-MoE-16B, AGQ(Affinity-Guided Quantization) alone reduces the 'Quarot+GPTQ' vs. FP gap by 30%, and combined with EBSS, by 60%.\n\n>Supplementary Material \n\nWe provide pseudocode for EBSS and AGQ in [Algorithm 1-2](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R3_dN5E.md).\n\nExperiments cover code generation(HumanEval) and complex math reasoning(GSM8K). Table 2 shows we improve HumanEval over others by 44%-63%. \n\nFigure 2 shows that EBSS significantly reduces inter-expert imbalance.\n>GW-MoE is not explicitly discussed in the paper but relevant to its context\n\nWe appreciate GW-MoE's insights in identifying routing uncertainty, which highlights load balancing. We've incorporated this into our revision for greater scholarly depth.\n>EBSS may introduce additional computational burden\n\nWe appreciate your concern, but EBSS reduces complexity significantly through three innovations:\n- **Probability-Guided Path Pruning** eliminates unlikely branches using cumulative probability and expert balance, reducing candidates from $V^n$(exponential) to $mn$(linear).\n- **Cumulative Probability** stores only $m$(e.g., 4) scalars, cutting storage overhead from $mV$ to $m$, enabling efficient PPL computation via Eq 12.\n- **Deferred Imbalance Calculation** avoids iterating over all tokens (e.g., 150K) by leveraging known expert distributions from the current sequence.\n\nExperiments in [Table R6](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R3_dN5E.md) shows EBSS's efficency(just 17–42 mins).\n>How does EBSS adjust the trade-off between PPL and expert balance? is there a risk of over-balancing experts at the cost of PPL?\n\nEBSS optimizes: $$D^* = \\arg\\min_{D}\\{ \n\\overbrace{\\text{PPL}(M,D)}^{\\in[0,1]} + \\overbrace{\\frac{\\sigma(M,D)}{\\tau}}^{\\in[1,+\\infty)}\n \\}$$where $\\tau$ balances data distribution alignment (via PPL) and expert balance (via $\\sigma$). **Smaller τ prioritizes expert balance but may increase PPL.** \n    $\\tau=1.2$ is determined through ablation in the manuscript. [Table R2](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R3_dN5E.md) further validate its generalizability in more models.\n\n**There is no risk of over-balancing as the cost of PPL because PPL is bounded, we prove it below:** \n* when ignoring $\\sigma$，we denote $\\text{PPL}=p$, $p$ typically remains close to 1 due to self-sampling(see Fig 4).   We have:$$D^* = \\text{PPL}({M,D})+ \\frac{\\sigma(M,D)}{\\tau}\\leq p+\\frac{1}{\\tau}$$ since $\\sigma \\geq 0$, we have: $$\\text{PPL}(M,D)\\leq p+\\frac{1}{\\tau} - \\frac{\\sigma}{\\tau}\\leq p+\\frac{1}{\\tau}$$\n\n>Can you provide a sensitivity analysis of all EBSS hyperparameters\n\nAdditional ablations in [Table R2-R3](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R3_dN5E.md), which shows：\n-  **τ = 1.2 is also generalizable and effective on other models.**\n- Accuracy improves with increasing $w$, though marginal gains diminish significantly beyond **w=4, offering an effective trade-off between performance and efficiency.**\n\n>non-linear activation functions and residual connections in expert networks may break this assumption\n\nFor nonlinear SiLU in MoE LLMs, Eq 17 and AGQ remain effective. [Table R4 and R5](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R3_dN5E.md) verify them from respectively: \n- **Numerical Error.** The reformulated gate output:$$y_i\\approx\\left((xW^{up})\\odot f(c_ixW^{gate})\\right)W^{down}$$ remains high cosine similarity(higher c close to 1) with true outputs. Moreover, top-k selection inherently prioritizes high-affinity tokens, further minimizing approximate error.\n- **Task Performance.** Applying AGQ to gate layers yields significant gains in both PPL and accuracy(1.2%+ gain), confirming its effectiveness.\n\nAGQ and residual connections (outside MoE layers) are decoupled. We agree that affinity masks hold promise and will explore it as MoE evolves.", "claims_and_evidence": "The following claims have no clear and  convincing evidence because no experiment study, theoretical proof and reference are provided. \n\" two primary challenges: (1) Inter-expert imbalance, referring to the uneven distribution of samples across experts, which leads to insufficient and biased calibration for less frequently utilized experts; (2) Intra-expert imbalance, arising from MoE’s unique aggregation mechanism, which leads to varying degrees of correlation between different samples and their assigned experts.\"", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "This paper does not provide direct experimental or theoretical proof that the poor MoE quantization effect is caused by load imbalance.", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "1. MoE Architecture and Core Components: The paper's discussion of MoE's modular structure, comprising router and experts, aligns with foundational works that systematically explain how MoE decomposes tasks and allocates them to specialized sub-networks. This modular design has been a cornerstone in scaling large models while maintaining efficiency.\n2. Efficiency and Specialization: The paper highlights MoE's advantages in computational efficiency (Mixtral 8x22B's sparse activation) and specialized expert roles (handling specific token types like punctuation or visual data) . These ideas build upon prior research demonstrating MoE's superiority over dense models in resource utilization.\n3. Challenges and Mitigation: The paper addresses challenges such as load imbalance and router uncertainty, which are critical issues in MoE training and inference. For instance,  proposes a method (GW-MoE) to resolve uncertainty in router decisions by leveraging global workspace theory, a concept not explicitly discussed in the paper but relevant to its context.", "essential_references_not_discussed": "GW-MoE (Global Workspace Theory for MoE Routers) is a recent work which introduces a method to mitigate router uncertainty by broadcasting tokens across experts during fine-tuning. While the paper addresses uncertainty implicitly, GW-MoE provides a direct and complementary approach that could enhance the robustness of the proposed methods.\nAt the same time, the article can introduce how MoE currently solves the load imbalance of inter-experts and intra-experts.", "other_strengths_and_weaknesses": "Strengths：\n1. The EBSS and AGQ methods proposed in this article effectively solve the shortcomings of the MoE model quantization compression that leads to a sharp loss of performance. At the same time, the above methods can also be used to solve the problem of unbalanced load between experts and within experts in the general MoE model, optimize expert training, and improve resource utilization efficiency.\n2. As a plug-in framework, MoEQuant seamlessly integrates existing PTQ methods (such as AWQ and GPTQ), and can improve the quantitative effect of the MoE model without modifying the underlying code.\n\nWeaknesses：\n1. EBSS's self-sampling relies on probability-guided path pruning, which requires maintaining candidate sequence branches and calculating cumulative probabilities, which may introduce additional computational burden, especially limiting efficiency when generating long sequences.\n2. The performance of the method is highly dependent on hyperparameters (such as temperature τ and number of branches w). Although the paper gives the optimal parameters (τ=1.2, w=4), in actual deployment, fine-tuning is required for different models, which increases the complexity of use.\n3. AGQ assumes that the linear relationship between samples and experts is approximately valid (Equation 17), but in extremely sparse or nonlinear routing scenarios, affinity modeling may not be able to fully capture the complexity of interactions between experts, resulting in residual quantization errors.", "other_comments_or_suggestions": "1. While EBSS aims to balance expert activation during calibration, it may inadvertently bias the model towards easier expert routes during self-sampling. Suggest adding a mechanism to explicitly control the trade-off between perplexity minimization and expert balance (e.g., via a dynamic weighting factor for the σ term in Equation 9).\n2. The current AGQ formulation assumes linear propagation of affinities through experts (Equation 17). However, non-linear activation functions (e.g., ReLU) and residual connections in expert networks may break this assumption. Propose extending AGQ to account for non-linear interactions (e.g., via layer-wise affinity masks).", "questions_for_authors": "1. How does EBSS dynamically adjust the trade-off between minimizing perplexity and balancing expert activations? Specifically, the paper mentions using a temperature parameter τ (Equation 9) to control this balance, but the optimal τ=1.2 is fixed in experiments. How was τ=1.2 chosen, and is there a risk of over-balancing experts at the cost of perplexity? If τ is highly model-dependent or requires extensive tuning, this could limit EBSS’s practicality. Conversely, if τ can be generalized across models, it strengthens EBSS’s appeal.\n2. While the paper fixes τ=1.2 and w=4, other hyperparameters (e.g., pruning threshold for candidate sequences) also will influence performance. Can you provide a sensitivity analysis of all EBSS hyperparameters (e.g., temperature range, branching factor) across different model scales (e.g., 14B vs. 6B parameters)? If EBSS requires case-by-case hyperparameter tuning, its adoption cost increases.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gWcGb1myrn", "reviewer": "Reviewer_ZiDj", "summary": "This paper proposes a customized quantization method for MoE-LLMs, targeting two key challenges: inter-expert and intra-expert imbalance. To address these issues, the authors design a self-sampling approach to construct a balanced calibrated dataset and introduce an affinity-guided quantization loss to ensure balanced weight updates. Extensive experiments validate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.\n\n---\n>The proposed method shows marginal improvements against GPTQ\n\n>When applying both proposed modules to QWENMOE-14B, the performance does not show a clear improvement\n\nWhile Table 1 shows modest overall gains (1.2–4.1%), **MoEQuant delivers substantial improvements in instruction-tuned models and complex multistep reasoning tasks(sensitive to quantization and critical for real applications).** As shown in [Table R1](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R4_ZiDj.md):\n- On code generation(HumanEval) and mathe reasoning(GSM8K), MoEQuant achieves 4.7–9.8% relative accuracy gains in base models.\n- In instruction-tuned models—more representative of deployment scenarios—it achieves 17.2–23.6% gains.\n\nAdditionly, we're sorry for potentially misleading readers about GPTQ in manuscript. As Line 353 states, it actually uses the initialization, similar to the competitive QuaRot.\n>However, I have concerns regarding its scalability to larger MoE models\n\nTable 1 already demonstrates MoEQuant's effectiveness on Mixtral-8x7B (56B params). To further address your concern, we provide results for DeepSeekV2 in [Table R2](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R4_ZiDj.md), where MoEQuant achieves a consistent 3.75%+ relative gain.\n>the speedup observed in Table 4 is not significant\n\nThe modest speedup for Qwen-MoE-14B and DeepSeek-MoE-14B stems from lower bandwidth pressure with fewer activated parameters, a trend also observed in [OSTQuant]. For Qwen-MoE-14B and DeepSeek-MoE-16B, where 2.7B and 2.8B params are active per step, acceleration remains low. **In contrast, Mixtral-8×7B achieves a 2.1× speedup. Notably, after quantization, Qwen-MoE-14B requires only 8.51 GB of memory, enabling MoE-LLM deployment on edge devices (e.g., RTX 3060).**\n        \n[OSTQuant]:Refining Large Language Model Quantization with Orthogonal and Scaling Transformations\n>when combining MoEQuant with GPTQ, the paper does not report the corresponding latency\n\nWe appreciate the your suggestion and add a comparison of time overhead in [Table R3](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R4_ZiDj.md). Since EBSS reduces search complexity from exponential to linear through Probability-Guided Pruning and AGQ remains lightweight with negligible overhead, **the additional time increase is minimal(only 11 minutes for DeepSeek).**\n>it would be beneficial to give more analyses about the synthetic data\n\nWe appreciate your constructive feedback and provide synthetic cases in [Table R4](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R4_ZiDj.md), which highlight two key characteristics:\n1. **Coherence and Distributional Alignment:** Generated sentences maintain coherence and align with the original models' distribution, also evidenced by consistently low PPL in Figure 4.\n2. **Diversity:** The data spans diverse domains, including reasoning and code generation. This diversity alleviates expert imbalance and drives significant performance gains (relative 5–24%) on complex reasoning tasks(Table 2).\n\n> Paper only demonstrates the existence of imbalance among experts, however, it lacks empirical or theoretical analysis on how this imbalance affects the quantization results\n\nThanks for your insightful feedback. we provide both empirical and theoretical analysis below:\n-  As shown in [Table R5](https://anonymous.4open.science/r/MoEQuant-DDFD/Tables_for_R4_ZiDj.md), **expert balance is positively correlated with final accuracy.** MoEQuant reduces imbalance by 90% compared to others, achieving the best accuracy by encouraging even token distribution among experts.\n- In GPTQ, the Hessian matrix quality directly impacts quantization precision. We estimate it via the finite difference method:\n$$H_{ij} \\approx \\frac{\\partial^2{L}}{\\partial w_i \\partial w_j} \\approx \\frac{L(w+\\epsilon e_i+ \\epsilon e_j)-L(w + \\epsilon e_i)-L(w+\\epsilon e_j)+L(w)}{\\epsilon^2}$$\n$\\epsilon$ is the tiny perturbation, $e_i$ is the i-th standard basis vector, $L$ is the loss. A fourth-order Taylor expansion yields:\n$$\\hat{H_{ij}} \\approx H_{ij} + \\frac{\\epsilon^2}{12}(\\nabla_{iiii}L+6\\nabla_{iijj}L+\\nabla_{jjjj}L)+O(\\epsilon^4)$$ Assume the estimated variance of the loss is $Var(L)=\\frac{\\sigma^2}{n}$ , $\\sigma^2$ is the population variance, and $n$ is the sample size. Then the expectation of the Hessian estimation:\n$$E[\\hat{H_{ij}}] = H_{ij}+O(\\epsilon^2)+O(\\frac{\\sigma^2}{n\\epsilon^2})$$\nThen, the bias of the Hessian estimation is:\n$$Bias(\\hat{H_{ij}}) = E[\\hat{H_{ij}}]-H_{ij} \\approx O(\\epsilon^2)+O(\\frac{\\sigma^2}{n\\epsilon^2})$$\n**The formula's second term is inversely proportional to sample size n: larger n means smaller statistical deviation, and vice versa. Thus, maintaining expert balance ensures all experts are well-calibrated, leading to better performance.**", "claims_and_evidence": "The claims in Lines 60-63 are not well supported by clear evidence. As shown in Table 1, previous method like GPTQ does not exhibit consistently poor performance across different datasets. The proposed method shows marginal improvements against GPTQ on several datasets.", "methods_and_evaluation_criteria": "The proposed method addresses an important issue—MoE-LLM quantization. However, I have concerns regarding its scalability to larger MoE models, such as DeepSeek-V2. The paper lacks experiments on such models, leaving it unclear whether expert imbalance becomes more pronounced in larger architectures and how this might affect the performance of other post-training methods.", "theoretical_claims": "There are no formal proofs in the paper. I have checked the formulas. Formulas (18,19) are based on the assumption that the Hessian matrix H  is Positive Semi-Definite. This assumption comes from GPTQ.", "experimental_designs_or_analyses": "The experiments are extensive and sound. However, several experiments are not well analyzed. \n1. Even though the proposed method employs 3- or 4-bit weight quantization, the speedup observed in Table 4 is not significant. This questions the practical benefits of the proposed method. I'm not sure if it's due to the lack of activation quantization or expert routing overhead overshadow the benefits of weight quantization. Additionally, when combining MoEQuant with GPTQ, the paper does not report the corresponding latency. A comparison under this setting would be valuable for understanding the practical benefits.\n2. The proposed method can be viewed as data-free quantization, it would be beneficial to give more detailed analyses about the synthetic data like sentence coherence and data diversity? I think this can increase the soundness of the proposed method.", "relation_to_broader_scientific_literature": "The self-sampling method primarily relies on LLMs to generate synthetic data, a concept that has been explored in previous quantization, works, like [1]. The key difference here is that this method specifically selects data that ensures a more balanced expert utilization.\n[1] LLM-QAT: Data-Free Quantization Aware Training for Large Language Models\n\nThe Affinity-Guided Quantization achieves a more reasonable Hessian computation by incorporating affinity scores provided by the gating network. Leveraging approximate Hessian information has been explored in previous work, like GPTQ.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The paper is well illustrated and the method design is clearly stated.\nThe paper studies a timely problem, which is to reduce the memory consumption of large MoE models without incurring significant performance degradation. It offers an efficient post-training quantization (PTQ) framework, and obviates the need for any fine-tuning.\n\n\n## UPDATE AFTER REBUTTAL\nMy concerns about performance improvements and scalability are mostly addressed.  Quantization on MoE models is a timely research topic. Thus, I remain positive about this paper.", "other_comments_or_suggestions": "Previous post-training methods are mostly one-shot, requiring domain-specific datasets for fine-tuning. The proposed method uses self-sampling to bypass the need for collecting fine-tuning datasets. This approach eliminates the dependency on finetuning datasets, the method also offers advantages in terms of data privacy.", "questions_for_authors": "The paper only demonstrates the existence of imbalance among experts in LLMs; however, it lacks empirical or theoretical analysis on how this imbalance affects the quantization process and results. Are there any analysis on the relationship between expert imbalance and quantization performance?\nThe code respiratory is not availiable.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes new MoE quantization methods called Expert-Balanced Self-Sampling (EBSS) and Affinity-Guided Quantization (AGQ) to address imbalance problem of MoE quantization. EBSS provides a better calibration dataset selection to balance the expert usage, and AGQ uses affinities between experts and samples into the quantization process.

The reviewers generally agreed on the effectiveness of the proposed methods supported by the presented evidence. Therefore, all the reviewers agreed to accept the paper in the conference (1 accept and 3 weak accepts). Reviewers acknowledge the importance of the contributions - "MoEQuant introduces two innovative methods", "The paper is well illustrated, and the method design is clearly stated. The paper studies a timely problem" together with the strong empirical results.

Some reviewers questioned about the task sensitivity, hyper-parameter sensitivity and more detailed comparisons, and the authors provided a thorough investigation to address the issues. The reviewers explicitly and implicitly acknowledged that most of the questions were resolved by the rebuttal.

Given the consensus among reviewers, the paper provides an important contribution to the community, and it is recommended to be appeared in ICML 2025.
Decision: Accept (poster)