Discovering Important Experts for Mixture-of-Experts Models Pruning Through a Theoretical Perspective
Authors: Weizhong Huang
Yuxin Zhang
Xiawu Zheng
Fei Chao
Rongrong Ji
Liujuan Cao
Venue: NeurIPS 2025 poster ()
Keywords: Mixture-of-Experts Models
Network Pruning
Primary area: deep_learning
TL;DR: We propose a superior MoE pruning framework that determines the importance of experts in MoE models through a theoretical perspective.

Abstract: Mixture-of-Experts (MoE) architectures enable efficient scaling of large language models but face prohibitive memory demands due to massive parameterization. Existing pruning methods rely on heuristic metrics or impractical enumeration of expert subsets, leading to suboptimal performance or scalability. In this paper, we propose Shapley-MoE, an efficient pruning method for MoE models inspired by cooperative game theory. By quantifying each expert’s contribution via Shapley value, our method identifies important experts without exhaustive combination evaluations. To overcome the NP-hard complexity of exact Shapley computation, we introduce a Monte Carlo sampling strategy for efficient approximation that reduces complexity to quadratic time. However, vanilla Monte Carlo sampling still faces issues of insufficient estimation accuracy and low sampling efficiency. To address these issues, we further propose two novel methods to improve sampling accuracy and efficiency: (1) Early Truncation, which early terminates unstable sampling steps caused by overly small expert subsets, and (2) Router-Guided Importance Sampling, which prioritize sampling important expert subsets using gating activation probabilities. Both theoretical and experimental analyses show that both methods can accelerate Shapley value estimation and improve accuracy. Extensive empirical evaluations demonstrate that our pruned MoE models outperform existing expert pruning methods. Notably, when applied to the Qwen2-57B-A14B model, our method reduces the number of experts by 25% with only a 0.92 increase in perplexity and over 96.4% of the average zero-shot accuracy is maintained.

Discovering Important Experts for Mixture-of-Experts Models Pruning Through a Theoretical Perspective
Weizhong Huang" 1.2
Yuxin Zhang'
Xiawu Zheng 1.2.3
Fei Chao
Rongrong Ji
Liujuan Cao 1.2
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. 2 Institute of Artificial Intelligence, Xiamen University.
Peng Cheng Laboratory, Shenzhen, China.
Abstract
Mixture-of-Expert (MoE) architectures enable efficient scaling of large language models but face prohibitive memory demands due to massive parameterization. Existing pruning methods rely on heuristic metries impractical enumeration of expert subsets, leading to suboptimal performance or scalability. In this paper, we propose Shapley-MoF, an efficient pruning method for MoE models inspired by cooperative game theory. By quantifying each expert's s contribution via Shapley value, our method identifies important experts without exhaustive combination evaluations. To overcome the NP-hard complexity of exact Shapley computation, we introduce a Monte Carlo sampling strategy for efficient approximation that reduces complexity to quadratic time. However, vanilla Monte Carlo sampling still faces issues of insufficient estimation accuracy and low sampling efficiency. To address these issues, further propose two novel methods to improve sampling accuracy and efficiency: (1) Early Truncation, which early terminates unstable sampling steps caused by overly small expert subsets, and (2) Router-Guide Im- portance Sampling, which prioritize sampling important expert subsets using gating activation probabilities. Both theoretical and experimental analyses show that both methods can accelerate Shapley value estimation and improve accuracy. Extensive empirical evaluations demonstrate that our pruned MoE models outperform existing expert pruning methods. Notably, when applied to the Qwen2-57B-Ar A14B model, our method reduces the number of experts by 25% with only 0.92 increase perplexity and over 96.4% of the average zero-shot accuracy is maintained.
Introduction
Mixture-of-FFxe -xperts (MoE) 40 16, 79 architectures have emerged as a popular architecture for large language models (LLMs)
enabling efficient scaling and superior performance on complex tasks [22 5 36). However, its sparse activation paradigm introduces massive parameterization. posing prohibitive memory demands. To address the above issues, various MoE model compression strategies have been developed, including pruning [56]
K 74,
6
24
quantization
and knowledge 30
distillation 37 65).
can remove less important parameters from MoE models, allowing reducing the memory footprint and computational complexity of MoE models without significantly sacrificing performance. Recent studies have shown that there significant redundancy among experts in MoE models 9 49 and MoE models can still maintain
Among these methods, network pruning 51 75 66
33
70
32 52
"Corresponding author: caoliujuan xmu.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Monte Carlo estimation
9.9
Input
E2
E1
E4
PPL
4x
Router-guided
importance sampling
Activation frequency
Router
E3
E2
E1
E4
PPL
E3
0.2
0.7
0.1
E2
E2
E1
E4
PPL
Expert 1
Expert
Expert
Expert 4
0.1
Shapley value
E4
PPL
9.5
E4
Output
E1
E4
PPL
30.7
Early truncation
MoË model
Figure I: Shapley-MoE prunes experts by calculating their Shapley values via router-guided Monte Carlo sampling. It analyzes expert activation frequencies to prioritize important experts during sampling, iteratively removes experts to compute their marginal contributions, and applies early truncation for insignificant performance. Experts are then pruned based on their ranked Shapley values.
superior performance after redundant experts removed. Therefore, many expert pruning methods have been proposed to remove unimportant experts from MoE models. These expert pruning methods can mainly be divided into the following two categories:
Metric based methods. These methods determine the importance of experts MoE models based on manually designed metries. For example, SEER-MoE (56) proposes pruning MoE models based on the activation frequency of experts, removing those with lower activation frequencies. He et al. [4 proposed to calculate the averaged routing score of each expert, then retain the expert with highest score and remove the rest. Although these metric-based methods have demonstrated their effectiveness through experiments, extensive validation is required to ensure that the manually designed metrics are indeed effective. More importantly, most of these heuristic methods lack theoretical analysis, which can easily lead suboptimal performance.
Enumeration based methods. These methods determine the pruned MoE model by enumerating retained expert subsets. For example, NAEE 49 enumerates all possible combinations of retained experts and selects the combination that minimizes the Frobenius norm of the output difference before and after pruning. CD-MoE [6] uses greedy search method to obtain the pruned model. It starts with an empty subset of retained experts and iteratively adds the expert that minimizes the Jensen-Shannon divergence between the outputs of the pruned and original models, repeating this process until the desired number of experts retained. However, for fine-grained large-scale MoE models with hundreds of experts, these methods are impractical in practice, as they require tens of thousands or even trillions cnumerations, which is obviously infeasible.
In this paper, we propose a MoE pruning method (Shapley-Mon inspired by cooperative game theory 15 which does not require enumerating all possible expert combinations and can obtain nigh-performance pruned MoE model in just tens of minutes. We observe that in MoE models, experts are dynamically involved in collective decision pocciion-mmainng making through a gating mechanism that enables conditional activation. These experts function not only independent computational units but also as collaborative combinations interacting with each other. To handle the complex relationships among experts, we leverage the Shapley value [63] I. a key solution concept in cooperative game theory for allocating contributions to participants. By considering all possible combinations, the Shapley value quantifies each expert's contribution, thereby effectively identifying those experts that are highly relevant to task performance.
Since the exact computation of Shapley values is an NP-hard problem, we employ a Monte Carlo sampling [9 20) approach efficiently approximate Shapley values by sampling from the set of possible expert permutations. This successfully reduces the computational complexity from exponential to quadratic, making it feasible to compute Shapley values for experts in large-scale MoE models. However, using vanilla Monte Carlo sampling to estimate the Shapley values of MoE experts still suffers from insufficient estimation accuracy and low sampling efficiency. To enhance the accuracy and efficiency of Monte Carlo sampling, we propose the early truncation and router-

---

guided importance sampling (RGIS) methods. In the early truncation, we observe that sampling steps involving subsets with too few experts can lead drastic model performance degradation. resulting in unstable sampling outcomes. We propose to terminate such sampling steps early to reduce unnecessary computation and ensure accurate Shapley value estimation. Additionally, we introduce the RGIS method. This method leverages the activation probabilities of experts from the gating mechanism, prioritizing key expert subsets with higher activation frequencies during Monte Carlo sampling, and accelerates sampling through importance weighting. With the same number of sampling steps, this method can estimate the Shapley values more accurately. We conducted theoretical analysis and ablation experiments on the two proposed methods, both of which demonstrate that these methods significantly improve the estimation accuracy and increase the sampling efficiency.
To evaluate the effectiveness of our proposed Shapley-Mot method, we conducted systematic experiments based on several mainstream open-source MoE model architectures, including the Qwen 3. DeepSeek [3], and Mixtral [35] series. The evaluation metrics cover perplexity, average zero- shot accuracy on seven tasks as well as performance on domain-specific tasks such as knowledge reasoning, arithmetic, and code generation. Experimental results show that Shapley-MoE has obvious advantages over existing MoE expert pruning methods. Specifically, when applied to the Qwen2-57B- A14B 744 model with 25% reduction in the number of experts, its perplexity increases by only 0.92 and the average zero-shot accuracy is maintained at up to 96.4%. Furthermore, the pruned model can achieve 1.x reduction in GPU memory usage and a 1.26-2.92 x increase in inference speed with the pruning rate ranged from 25% to 75%, while the pruning process only takes 36 minutes. Additionally, we further explored the application of the Shapley-MoE method on multimodal MoE models and performed further performance optimization on the pruned models by integrating LoRA 29 fine-tuning and quantization techniques.
2 Related Work
Pruning the Expert of MoE Models. Researchers have proposed numerous MoE pruning algo- rithms to remove unimportant experts. There are some works that achieve the effect of pruning MoF
models through expert merging 42 47 78. Other works achieve the effect of pruning MoE models by directly discarding experts. For instance, SEER- MoE (56) proposes to prune the MoE model based on the activation frequency of experts. He et al. (24 proposed to calculate the averaged routing score of each expert and retain the expert with the highest score. NAEE [4] determines the subset of re- tained experts by enumerating all possible combinations of experts. CD-MoE (6) introduces greedy search algorithm that selects experts to retain based the smallest Censen-Shannee divergence (53). Existing MoE pruning methods rely on heuristics or impractical enumeration. We propose Shapley value-based pruning, offering theoretical performance guarantees and efficient model pruning.
Shapley Value. Shapley value 69 is game theory concept that quantifies each participant's average marginal contribution across all possible coalition combinations in cooperative scenarios. Shapley value are often used to assess the importance of model components 220 2 71). Neuron
Shapley (20] proposed using Shapley value to quantify the contribution of individual neurons to the predictions and performance of deep networks. Ancona et al. [2] proposed Shapley value as a metric for structured pruning in convolutional neural networks. Moreover, since accurately calculating Shapley value is an harhard problem, many methods have been proposed to approximate Shapley value, such Monte Carlo sampling [9] 20), kernel-based approximation algorithms [50]. and stratified sampling techniques [77]. In this paper, we innovatively extend Shapley value to evaluate the importance of experts and propose Monte Carlo sampling approach for approximating Shapley value, which incorporates early truncation and router-guided importance sampling.
3 Methodology
Notation. In this study, we adopt the following notation conventions: bold typeface indicates vectors (e.g., æ. y) and matrices (e.g., X, Y), while calligraphic font denotes loss function and set (e.g.. L, L)
3

---

3.1 Mixture of Experts
Given MoE model 40 16 comprising expert layers, let the I-th expert layer (1. ,.....,,) L}) contain n routing experts { {F1, F2 ... F2 |
Each expert constitutes independent feedforward neural network (FFN). For input vector E IRd with hidden dimension d, the output computation at the (-th expert layer follows:
Σ ()))
where G' (2x); [0, 1] denotes the output of gating network for expert i, and E (2) represents the i-th expert's output. The gating network is formally defined as:
(x) Softmax (Topk (zW:)):
where Wi E Rdxx denotes the routing matrix, and (op operator generates a sparse routing 9 pattern by preserving only the top-k values while setting others to -00, thereby activating out of experts per input token.
3.2 Shapley Value MoE Experts
Motivation. In MoE architectures, experts participate in collective pecision-making through con- ditional activation governed by gating mechanism. These experts not only function as independent computational units but also form interdependent subsystems, and there is collaborative relationship among them. To rigorously quantify individual expert contributions within this complex cooperative system, employ Shapley Value [63 [60 []] analysis from cooperative game theory [15 39]. Shapley Value a widely used mathematical tool for allocation problems, capable fairly quantifying the marginal contribution of each participant in a collaborative process.
Shapley Value Formulation. The process of experts dynamically participate in decision- through gating mechanism can be formally conceptualized where denotes the expert set and V 2 + I represents the model performance. Perplexity is commonly used to measure the performance of LLMs and lower perplexity indicates better model performance. Therefore, we use the reciprocal of perplexity as the metric for evaluating the performance of the MoE model, that is, V 1/PPL. Each expert E acts player whose Shapley Value [63] quantifies its average marginal contribution to model performance. Consider a MoE model with L expert layers containing routing experts per layer. Let the complete expert be E = {E| < < L, < < n) with cardinality = x n. The Shapley Value Φ for expert E
cooperative game I (E, V)
is defined as:
1
Σ
((SU - V(S)
N
(N-))
SCEV(F)
[S]
where (S) measures model performance when retaining subset S. The Shapley Value ΦE! computes the weighted average of E 's marginal contributions ((S precisely quantifies how each expert affects model performance through all possible collaborative scenarios. In addition, we present the properties of the Shapley value in Sec. C. which ensure that the contributions of MoE experts are fairly quantified across all coalition interactions.
over a C E {.)) This
3.3 Approximation of Shapley Value
Computational Challenges. While the Shapley value theoretically provides a metric for quantifying expert contributions, exact computation through Eq. 3 requires evaluating all 2 expert subsets, which is a prohibitive proposition for modern MoE architectures where total expert count N often exceeds 10". This exponential complexity motivates the development of efficient approximation methods.
"Some advanced MoE architectures (e.g., DeepSeekMoE- 16B 13] Qwen2-57B-A14g 73 incorporate
shared experts alongside routing experts. Our pruning strategy specifically removes less important routing experts while preserving all shared experts. For notational simplicity, we omit explicit representation of shared experts in this formulation.

---

Monte Carlo Estimation. We employ the Monte Carlo sampling approach 77 to estimate Shapley value through empirical expectation over expert permutations. For expert E E &, the estimator constructed as:
1 M Σ ((((m) U (EE)) V(((m))
O (V:
(4)
where M denotes the number of Monte Carlo samples. Each subset ( CE\ (E.) is generated by sampling a random permutation ((m) of all experts and taking S(m) Pref((((m)) which is the set of experts preceding E in permutation (m)) This stratified sampling strategy ensures uniform
coverage of coalition spaces while maintaining estimator unbiasedness [62 28. By averaging over M
permutations, this method reduces the computational complexity from (2 to O(MN), making feasible for large-scale MoE models.
Early Truncation. During Monte Carlo sampling, we observe that subsets S(m) with insuffi cient expert participation (i.e.. [S(m]] < N) frequently lead to catastrophic performance collapse (V(S(m)) + 0), introducing high variance in Shapley value estimation. Using this fact, for sampled permutation (((),, can avoid computing the marginal contributions of the earlier elements. There- fore, we implement an early truncation [19] mechanism during Monte Carlo estimation. Formally, the evaluation terminates prematurely when:
v(S(m))< <T VVE),
(5)
where - E (0, 1) is the threshold. This adaptive truncation achieves dual benefits: 1) It eliminates uninformative sampling steps where model functionality has collapsed, thereby reducing estimator variance. 2) It preserves computational resources by carly-exiting, Our ablation studies in Sec. 4.6 show that this technique reduces the pruning cost of Shapley-MoE by nearly half and improves the accuracy of Shapley-MoE. In addition, the error bound of early truncation is given by the following theorem:
Theorem 1 (Error Bound of Early Truncation Shapley Estimation). Let Φet denote the true Shapley denote its Monte Carlo estimator with early truncation threshold т. value for expert E, and B!
Assume that for any subset S C E {E satisfying V(S) < (E),), the marginal contribution is
bounded as (V(S U (E!)) - V(S) S c. Then, for any > 0, with probability at least 6, the
estimation error satisfies:
log(2/0)
16
traee o E
Se ((V((()) < <VV(E))
(6)
2M
where M is the number of Monte Carlo samples, and the probability P(V(S) < TV())) is taken the uniform distribution of expert permutations.
The proof of the above theorem is provided in Sec. D The theorem guarantees that early truncation preserves estimation accuracy through two mechanisms: 1) The first term PPV(SS) < +V) bounds errors from truncated subsets by their contribution probability and bounded marginal impact e.
2) The second term /oog(2/0)/(C) (2M)) controls Monte Carlo sampling error, which diminishes with sample size M. The above theorem shows that the error of the early truncation is bounded, which justifies the truncation: it discards minimally influential computations while preserving estimator consistency, enabling efficient approximation.
Router-Guided Importance Sampling. To further accelerate the calculation of the Shapley values of the MoE experts, we further propose Router-Guided Importance Sampling (RGIS) method based on the characteristics of the router network. This method leverages expert activation probabilities from the gating mechanism to prioritize critical coalitions during Monte Carlo estimation, achieving sampling acceleration through importance weighting. First, we profile each expert's activation frequency over calibration dataset D:
p
1
E RED
D
where / represents the empirical activation probability for expert E.. These probabilities establish prior distribution reflecting expert importance. Next, we generate expert permutations through
5

---

a Plackett-Luce model 58 parameterized by {pr}. For each Monte Carlo sample we sample permutation x(m) by sequentially selecting experts without replacement with probability:
P
E ATmh
P =
(8)
ExF E Pj
E
where A!T C & denotes the available expert set at step i, and (7(0)(( is the 1-th selected expert. This sampling prioritizes high-activation experts. Finally, we compute importance weights correct sampling bias. Let ((r(m)) denote the permutation probability under Plackett-Luce model:
Q(a(m))
Pa [
(9)
N s=t Pstm0]s]
==1
The importance weight becomes wm()
1/N
The RGIS Shapley estimator is:
RRISS E
1 M
Σ [(((S)m) U (EEE)) v()( wm(m) m[m]
(10)
where S(m) contains experts preceding E in n ((m), and I(m) E {0,1) indicates whether E was evaluated before truncation in sample
We propose the following theorem demonstrate the theoretical properties of the RGIS estimator. First, it guarantees that our router- outer-guided sampling strategy does not introduce systematic estimation bias through careful importance weighting. Second, demonstrates that leveraging router activation patterns as importance priors fundamentally reduces estimation variance compared to naive Monte Carlo sampling.
Theorem 2 (Unbiasedness and Variance Reduction of RGIS). The Guider-Guided Importance Sampling (RGIS) estimator RRGS defined in Eq. 10 satisfies the following properties:
RGIS E over the Plackett-Luce sampling distribution Q.
Unbiasedness: For any expert E E, E
2eg, where the expectation is taken
Variance Reduction: the activation probabilities {r.} are positively correlated with the true Shapley values {+ then for the same number of samples M, Var
(RRGIS)
S Var (MC
F
where dMc denotes the standard Monte Carlo estimator in Eq.
We have provided the proof of the above theorem in Sec. E Theorem 2 establishes that RGIS provides an unbiased Shapley value estimation while achieving variance reduction when expert activation frequencies reflect their true contributions. This occurs because the router's gating mechanism naturally prioritizes important experts, and frequently activated experts tend to have larger marginal impacts on model performance. By aligning the sampling distribution with these empirical importance measures, RGIS concentrates computational resources evaluating the most impactful coalitions. thereby improving estimation efficiency. As shown in the ablation experiments in Sec. [.6) RGIS improves the accuracy of Shapley-MoE. This means that it accelerates Monte Carlo sampling when given the same sampling number.
3.4 Shapley-MoF algorithm for MoE Pruning
We have presented our proposed Shapley-MoE for MoE expert pruning in Algorithm [ Specifically, our Shapley-MoE algorithm prunes MoE models by estimating expert Shapley value through router- guided Monte Carlo sampling. It first profiles expert activation frequencies to prioritize critical experts during permutation sampling. For each sampled permutation, incrementally adds experts, computes marginal performance gains, and applies early truncation when performance drops below a threshold. Finally, experts are ranked by their Shapley Values, and the lowest-ranked routing experts are pruned. Extensive experiments in Sec. 3 demonstrate that Shapley-Mot significantly outperforms current MoE pruning methods. Our method effectively reduces the memory footprint and computational cost of MoE models while maximizing the preservation of their performance.

---

Algorithm Shapley-MoE algorithm for MoE models Pruning
Require:
MoE model with expert set &, calibration dataset D, Monte Carlo samples M, truncation
threshold т, pruning ratio
Ensure: Pruned MoE model
Compute full model performance Viull V(E) on D
2: Profile expert activation probabilities {po} over the 'D (Eq.)
3: for each expert E do
4: Initialize Ø
5: end for
6: for m = 1 to M do
>> Monte Carlo loop
7:
Sample permutation x ((m) via Plackett-Luce model with probabilities {p'} (Eq. 8) Router-guided importance sampling
8: Compute permutation probability Q(T(m)) and weight w(m)) + ( (Eq.
I/N
9:
Initialize Vo <- ViaI
for to N do
Permutation traversal
11:
if V 2 VVan then
12:
Evaluate V/= + V(f=(m) + 11, (mm) [k+ + 2). x ((m) [N]}) on D else
14:
VE +- VR-1
> Early truncation
end if
16: Px [R] &- Pat + (Va-1 Va) (()) (Eq. 10
17: end for
18: end for
19: for each expert E E E do
20: dgg + dz! /M (Eq. 10)
Final approximate Shapley value
21: end for
Sort routing experts by d remove the y ratio experts with the lowest ranking
return Pruned MoE model
3.5 Performance Preservation Theorem
In this section, we propose the following theorem, which establishes a performance preservation bound for our Shapley-MoE method.
Theorem 3 (Performance Preservation Bound). Let & denote the complete of experts in the MoE model, and Eprined = E (E de < el (1,2...., L), E (1,2....,,)}) be the pruned expert set with Shapley value threshold 0. Under the following conditions:
The Shapley value estimator satisfies (0 d5 S for all E E E with probability
2 8. 8.
The cooperative game is additive, iee., V(S) = EEEE dgg for any S C E.
Then with probability at least - 8, the performance degradation after pruning satisfies: V(E) V (Eprunes) < NN 71), where N- = |8) the total number of experts.
We have provided the proof of the above theorem in Sec. The above theorem provides a crucial theoretical guarantee for our method. It formally establishes that the performance degradation of the model after pruning is controllably bounded. This bound is directly linked to two practical factors: the pruning threshold (c) we choose and the estimation accuracy our Shapley value calculation (1)). The theorem demonstrates that by removing only experts with verifiably low contributions and ensuring a precise estimation, the overall performance loss can be provably limited, thus validating the reliability and effectiveness of our pruning methodology.
7

---

Experiments
4.1 Experimental Setup
Models. We conducted experiments on popular open-source MoE models, including the Qwen series (Qwenl.5-MoE-AN) (3], Qwen2-57B-A144 (73), and Qwen3-30B-A3H [68], the DeepSeek
series (DeepSeekMoO-116 133 and DeepSeek-V2-1e V2-Lite 45 ) and Mixtral series (Mixtral-8x7B
35
The basic architecture information of these models are summarized in Sec.
Baselines. We compare strong baselines for MoE expert pruning directly related to this study, including Random [49]. Gating Score [4]. Frequency [56]. NAEE (4), Expert Trim (24] and
CD-MoE [6] Additionally, since NAEE 49 and CD-MoE (6) determine the pruned expert
through enumeration, they are not suitable for MoF models with dozens to hundreds of experts (see
analysis in Sec. H.. Therefore, only compare these methods Mixtral-8x7B 35
Benchmarks. We follow the evaluation settings of previous MoE pruning methods assess the zero-shot learning and language modeling capabilities of pruned MoE models. Specifically, we evaluate the zero-shot performance on seven downstream tasks: HellaSwag (76), WinoGrande [6]],
PIQA 4. OpenbookQA [55]. ARC Easy and Challenge [12], and BoolQ [1]. The above accuracies were obtained using the EleutherAl language model evaluation framework [18] Additionally. we
evaluate the perplexity of the pruned models the WikiText-2 (54] validation set. Furthermore,
assess the pruned MoE models' knowledge reasoning, arithmetic, and code generation capabilities, reporting 5-shot accuracy on the MMLU dataset [25]. 8-shot accuracy on the GSM8K dataset [64).
and O-shot accuracy on the HumanEval dataset
8
Implementation Details. We follow the settings of NAEE 49 and CD-MoE 6 method, randomly sampling 128 examples from the C4 training dataset [59 calibration data. Additionally, set the number of Monte Carlo sampling M to 20 and the early truncation threshold r to 0.5.
More Models, Baselines and Benchmarks. In Sec. ! we present additional experimental results, including pruning multimodal MoE models using our Shapley-MoE method, integrating pruned MoE models with quantization technique, further enhancing the performance of pruned MoE models through LoRA fine-tuning.
4.2 Zero-shot Tasks
Table
presents the average accuracy of pruned MoE models across seven zero-shot tasks after pruning 25% and 50% of the experts. The results clearly demonstrate that our proposed Shapley-MoF method significantly outperforms existing MoE pruning approaches. For instance, in the case of the model with pruning rate of 50%, the accuracy achieved by the Shapley-MoE method is 2. 2.17% higher than that of the best Frequency method. This substantial improvement underscores the effectiveness and superiority of our approach.
Dwenl.55MMo-AA-
Table 1: The average zero-shot accuracy across 7 tasks of pruned MoE models.
Pruning ratio
Qwen 2-57B-A14B
DeepSeek MoE-16B
Method
1.5-MoE-A2.7B
3-30B-A3B
V2-Lite
0%
None
61.70
65.39
66.36
61.92
64.16
Random
$4,79
61.76
$7.69
$3.37
$4.57
Gating Score
62.16
58.27
54.23
25%
Frequency
55.63
62.29
58.68
54.58
55,90
Expert Trim
55.16
62.06
58.27
53.75
55.44
Shapley-MoE
57.10
63.05
60.02
56.31
$7.29
Random
45.25
58.44
44,96
41.50
44.20
Gating Score
46.30
58.98
46.22
42.51
45.60
50%
Frequency
46.84
59.33
46,54
42.84
45.96
Expert Trim
46.35
59.15
46.27
42.67
45.85
Shapley-MoE
60.65
48.28
44.89
47.08

---

4.3 Language Modeling
Quantitative Evaluation. Table shows the WikiText-2 perplexity of MoE models after pruning 25% and 50% of experts, respectively. The results indicate that all the pruned models obtained by our proposed Shapley-MoF method have significantly lower perplexity than existing MoE pruning methods. For example, for the DeepSeek-MoF- 16B model with pruning rate of 50%, the perplexity of the Shapley-MoE method is 3.17 lower than that of the best Frequency method. The above results indicate that Shapley-Mot method not only preserves the zero-shot performance of the model well but also effectively retains the model's language modeling capability.
Table 2: The WikiText-2 perplexity of pruned MoE models.
Pruning ratio
Method
Qwen 2-57B-A14B
DeepSeek
LS-MoE-A2.7B
3-30B-A3B
MoE-16B
V2-Lite
0%
None
7.01
5.86
8.45
6.55
6.35
Random
11.01
7.85
13.72
10.61
10.52
Gating Score
7.15
12.39
9.57
25%
Frequency
10.08
7.11
11.97
9.46
9.56
Expert Trim
10.28
7.12
12.29
9.67
9.76
Shapley-MoF
6.78
11.02
Random
21.87
9.93
35.74
37.18
24.92
Gating Score
20.10
8.58
32.03
35.12
23.10
50%
Frequency
19.78
8.35
31.23
34.27
22.25
Expert Trim
19.98
8.65
32.26
35,20
23,01
Shupley-MoF:
17.57
7.98
28.10
31.10
19.96
we present the WikiText-2 perplexity performance of the pruned Mixtral-8x7B model under wider range of pruning rates, from 12.5% to 75%. The experi- mental results show that across different pruning rate settings, the perplexity of the pruned models obtained by our Shapley-Moft method consistently remains lower than that of pruned models obtained by other methods, which fully demonstrates the robustness of our Shapley-Mof method.
Varying Sparsity Rates. Table 3
Table WikiText-2 perplexity at different pruning rates.
Table 4: Ablation study of the effective-
Method
12.5%
25%
37.5%
50%
62.5%
75%
ness of early truncation and RGIS.
Random
5.32
6.45
8.15
13.42
15.21
27.25
Early Truncation
Pruning Cost
Gating Score
5.13
6.35
7.80
13.03
14.97
26.38
RGIS
PPL
(mins)
Frequency
5.10
6.20
7.75
12.90
14.83
26.31
Expert Trim
5,23
6.18
7.70
12.98
14.67
26.16
21.06
NAEE
5.02
6.02
7.71
14.46
26.03
19.24
18.6 18.65
35
CD-MoE
5.01
5.98
7.69
12.76
14.35
25.70
17.57
Shapley-MoE
4.82
5.60
7.11
12.01
13.26
23.86
4.4 Knowledge Reasoning, Arithmetic and Code Generation Task
We further evaluate the broader capability retention of the pruned MoE model. This includes assessing the pruned models' performance multi-task and cross- disciplinary language understanding and reasoning, arithmetic, and code generation. We prune 25% of the experts in the Qwenl -5-MoE-A2.78 model, and the results presented in Table Our method sig- nificantly outperforms other Mož. pruning methods, further demonstrating its effectiveness in preserving the various capabilities of pruned MoE models.
Table 5: Accuracy of the pruned model on the GSM8K, MMLU and HumanEval datasets.
Method
MMLU
GSMBK
HumanEval
None
62.50
61.50
34,20
Random Gating Score
46.20
24.18
17.34
47.20
27.14
18.21
Frequency
50.30
28.89
18.43
Expert Trim
49.60
27.09
17.89
Shapley-MoE
53,80
31.02
19.24
4.5 More Results
provide additional experimental results in appendix. Sec. [.I we show the performance of
pruned multimodal MoE models. In Sec. [.2 we demonstrate the performance improvements of pruned MoE models with LoRA fine-tuning. Additionally, Sec. we combine the pruned MoE models with quantization techniques to further compress pruned model.

---

4.6 Ablation Study
Effectivenes of Early Truncation and RGIS. We present an ablation study in Table evaluate the effectiveness of early truncation and RGIS, assessing the contributions of these two components to Monte Carlo sampling. Specifically, we report the WikiText-2 perplexity of the Dwenl.5-MoE-AA.. model pruned by 50% experts using Shapley-MoF. We observe that removing either early truncation or RGIS leads degraded performance of the final pruned model. Additionally, removing early truncation increases the pruning cost. This is because early truncation eliminates sampling steps when the model function collapses, reducing the variance of the estimator and saving computational resources by exiting early, thereby accelerating the Monte Carlo sampling process. RGIS leverages the expert activation probabilities in the gating mechanism to prioritize key coalitions during Monte Carlo estimation, and achieves accelerated sampling. Under the same number of sampling steps, this allows for more accurate Shapley value estimation.
Influence of Sampling Number M and Truncation Threshold We also demonstrate the impact the number of Monte Carlo samples M and the early truncation threshold T on the final accuracy of the pruned model. Fig. 2 shows WikiTexx-2 perplexity and pruning cost (minutes) of Shapley-Mol under different settings of M and т. Although reducing M can effectively decrease pruning cost, it may lead to performance degradation due to insufficient sampling, which affects the accuracy of Shapley value estimation. However, when the number of samples M exceeds 20, the final performance of the pruned model becomes insensitive to changes in M. Considering both search efficiency and model performance, we ultimately set M = 20 as the number of samples. In addition, moderate truncation threshold achieves better trade-off between accuracy and computational complexity, as it both suppresses the variance in sampling results and further reduces pruning cost.
Shroug Rahorsas
Papp
300
Namber Samp
Tuncacion
2%
23%
35%
75%
(a) Sampling Number M
(b) Truncation Threshold
Figure 3: Memory footprint and inference speed of pruned Qwen1.5-MoE-A2.775
Figure 2: Influence of sampling number M and truncation thresh- old
Inference Speedup. We measure the memory reduction and inference acceleration performance of the pruned 1.5-MoE--22.... model, with results shown in Figure 3 The results in the table are measured using the vLLM inference engine [3 on an NVIDIA A100 80GB GPU. Compared to the original model, when the pruning rate ranged from 25% to 75%, the inference speedup is improved by 1.26x to 2.92 x, and the memory usage on the GPU is also reduced by 1.25 to 2.46 x.
4.7 More Ablation Study
We have provided additional ablation results in the appendix. Specifically, in Sec. J.I we show the time required by Shapley-MoE to obtain different pruned models. In Sec. J.2 we analyze the impact of the calibration dataset the accuracy of the pruned models, including the number of calibration we demonstrate
samples and the use of domain-specific calibration datasets. Additionally, in Sec. [.3 the robustness of Shapley-Mot method under different random seeds.
5 Conclusion
This paper introduces Shapley-MoF, an efficient pruning method for MoE. models that leverages Shapley value quantify expert contributions. By integrating Monte Carlo sampling with early truncation and router- guided importance sampling, it achieves scalable and accurate pruning. Em- pirical results demonstrate superior performance over existing methods, enabling resource-efficient deployment of high-quality pruned MoE models.
10

---

Acknowledgments
This work was supported by the National Science Fund for Distinguished Young Scholars NN.662056033, the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. U23A20383, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No.
62072389, No. 62002305 and No. 62272401, No.624B21199 19), the Natural Science Foundation of Fujian Province of China (No. 2021J06003, No.2022106001,, and the Fundamental Research Funds for the Central Universities.
References
[1] E. Algaba, V. Fragnelli, and Sánchez-Soriano. Handbook of the Shapley value. CRC Press, 2019.
[2] M. Ancona, Öztireli, and M. Gross. Shapley value as principled metric for structured network pruning. arXiv preprint arXiv:2006.01795, 2020.
[3] Bai, Bai, Y. Chu, Z. Cui, K. Dang. Deng, Y. Fan, W. Ge, Y. Han, Huang, al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Y. Bisk, Zellers, R. Le bras, J. Gao, Y. Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pages 7432-7439, 2020.
[5] W. Cai, J. Jiang, F. Wang, J. Tang, S. Kim, and J. Huang. A survey on mixture of experts. arXiv preprint arXiv:2407.06204, 2024.
[6] M. Cao, G. Li, J. Ji, Zhang, X. Ma. S. Liu, and L. Yin. Condense, don't just prune: Enhancing efficiency and performance in moe layer pruning. arXiv preprint arXiv:2412.00069 2024.
[7] J. Castro, D. Gómez, and J. Tejada. Polynomial calculation of the shapley value based on sampling. Computers operations research, 3665):1726-11
2009.
[8] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.003:
2021.
[9] z. Chi, L. Dong, S. Huang, D. Dai, Ma, B. Patra, S. Singhal, P. Bajaj, X. Song, X.-L. Mao, al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 35:34600-3466 2022.
[10] M. N. R. Chowdhury, M. Wang. K. El Maghraoui, N. Wang, P.- Chen, and C. Carothers. A provably effective method for pruning experts in fine-tuned sparse mixture-of-expprts In Proceedings of the 41st International Conference on Machine Learning, pages 8815-8847, 2024.
[11] C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski. M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXiv:1905.10044, 2019.
[12] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint 2018.
[13] D. Dai. C. Deng, C. Zhao, R. Xu, Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:240) 006066, 2024.
[14] Z. Dong, H. Peng, P. Liu, W. X. Zhao, D. Wu, F. Xiao, and Z. Wang. Domain-specific pruning of large mixture-of-experts models with few-shot demonstrations, arXiv preprint arXiv:2504.0699. 2025.

---

[15] E. Elkind and J. Rothe. Cooperative game theory. Economics and computation: an introduction to algorithmic game theory, computational social choice. and fair division, pages 135-193. 2016.
[16] W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1 1-39, 2022.
[17] E. Frantar and D. Alistarh. Qmoe: Practical sub-1-bit compression of trillion-pant parlmmeeerrmtteerr models. arXiv preprint arXhv:2310.1679) 2023.
[18] L. Gao, J. Tow, B. Abbasi. S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac'h h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa. J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. org/records/
The language model evaluation harness, 07 2024. URL https://zents 12608602
[19] A. Ghorbani and J. Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pages 2242-22511 PMLR, 2019.
[20] A. Ghorbani and Y. Zou. Neuron shapley: Discovering the responsible neurons. Advances in neural information processing systems, 33:5922-5932, 2020.
[21] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904 6904-6913, 2017.
[22] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12940 2025.
[23] S. Hart. Shapley value. In Game theory, pages 210-216. Springer, 1989.
[24] S. He, D. Dong. L. Ding, and A. Li. Towards efficient mixture of experts: A holistic study of compression techniques. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL http::://pppereeiiieet http:e/vwwwwweeefftereeeeee met:/forumidd-HTyypll
[25] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Mea- suring massive multitask language understanding. In International Conference Learning Representations, 2021. URL https::
met/ffoumm/e ddkB jmI3GmQ
[26] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty- fiftt Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[27] W. Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pages 409-426, 1994.
[28] T. Homem-de Mello and G. Bayraksan. Monte carlo sampling-based methods for stochastic optimization. Surveys in Operations Research and Management Science, 19(1):56-85, 2014.
[29] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhr Zhu, Y. Li, S. Wang, L. Wang. and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:21066 09685, 2021.
[30] W. Huang, Y. Liao, J. Liu, R. He, H. Tan, S. Zhang, H. Li. S. Liu, and X. Qi. Mc-moe: Mixture compressor for mixture-of-expere ex-extssss 1lms gains more. arXiv preprint arXiv:2410.06270, 2024.
[31] W. Huang, Y. Zhang, X. Zheng, F. Chao, and R. Ji. Determining layer-wise sparsity for large language models through a theoretical perspective. In Forty-second International Conference on Machine Learning, 2025. URL httpssts:: httttp::///openevviii
[32] W. Huang, Y. Zhang, Zheng, F. Chao, and R. Ji. Towards efficient automatic self-pruning of large language models. arXiv preprint 250v 2025.
12

---

[33] W. Huang, Y. Zhang, X. Zheng, Liuyang. J. Lin, Y. Yao, and R. Ji. Dynamic low-rank sparse adaptation for large language models. In The Thirteenth International Conference on Learning Representations, 2025. URI. /tttps://openreeiex httpa/rwwwwwwweeeeeeeeeeeeeeeeee
[34] D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6700 2019.
[35] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. I. Casas, E. B. Hanna, F. Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401.0444 .40088, 2024.
[36] S. Jie, Tang, K. Han, Y. Li, D. Tang, Z.-H. Deng, and Y. Wang. Mixture of lookup experts. arXiv preprint arXiv:2503.15799 2025.
[37] G. Kim, G. Chu, and E. Yang. Every expert matters: Towards effective knowledge distillation for mfxture-of of-eeeetssessssseeeeeeee language models. arXiv preprint arXiv:2502.1294/7 2025.
[38] W. Kwon, 7. Li, S. Zhuang, Y. Sheng, I.. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611-626, 2023.
[39] J. Lemaire. Cooperative game theory and its insurance applications. ASTIN Bulletin: The Journal of the IAA, 21(1):17-40, 1991.
[40] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[41] Li, X. Jin, Y. Cheng, and T. Chen. Examining post-training quantization for mixture-of- experts: A benchmark. arXiv preprint 2xXiv:2406008/5: 2024.
[42] P. Li, Z. Zhang. P. Yadav, Y.-.. Sung, Y. Cheng, M. Bansal, and T. Chen. Merge, then compress: Demystify efficient SMoe with hints from its routing policy. In The Twelfth International Conference on Learning Representations, 2024. URL id=ePWG9Cy3WK
[43] B. Lin, Z. Tang, Y. Ye, J. Cui, B. Zhu, P. Jin, J. Huang. J. Zhang, Y. Pang, M. Ning, et al. Moe- Ilava: Mixture of experts for large vision- langun-a models. arXiv preprint arXiv:2401.15947, 2024.
[44] J. Lin, J. Tang, H. Tang, S. Yang. W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han. Awq: Activation-awan aware weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[45] A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo, et al. Deepseek-v2: strong, economical, and efficient mixture-of-expprts language model. arXiv preprint ar arXiv:2405.04434 2024.
[46] C. Liu and X. Wan. Codeqa: A question answering dataset for source code comprehension. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2-32223, 2021.
[47] E. Liu, J. Zhu, Z. Lin, X. Ning, M. B. Blaschko, S. Yan, G. Dai, H. Yang, and Y. Wang. Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs, arXiv preprini arXiv:2407.0000 00945, 2024.
[48] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
[49] X. Lu, Q. Liu, Y. Xu, A. Zhou, S. Huang, B. Zhang, Yan, and H. Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-expert large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6159-6172. 6172, 2024.
13

---

[50] S. M. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions. Advances neural information processing systems, 30, 2017.
[51] J. Luo, J. Wu, and W. Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 5058-5066, 2017.
[52] Y. Luo, W. Chen, X. Zheng, W. Huang, S. Yin, H. Lin, C. Fu, J. Huang, J. Ji, J. Luo, al. Quota: Query-oriented token assignment via cot query decouple for long video comprehension. arXiv preprint arXiv:2503. 250.0888999., 2025.
[53] M. L. Menéndez, J. A. Pardo, L. Pardo, and M. d. C. Pardo. The jensen-shannan divergence. Journal of the Franklin Institute, 334(2):007-318, 1997.
[54] S. Merity, C. Xiong. Bradbury, and R. Socher. Pointer sentinel mixture models, arXiv preprint arXiv:1609.000 1609.07443, 2016.
[55] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint (xXiv:/809002899 2018.
[56] A. Muzio, A. Sun, and He. Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts, arXiv preprint arXiv:2404.05089, 2024.
[57] B. Peng. C. Li, P. He, M. Galley, and J. Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.033 00000,,03777222 2023.
[58] R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193-203 1975.
[59] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 67, 2020.
[60] B. Rozemberezki, L. Watson, P. Bayer, H.-I. Yang. O. Kiss, S. Nilsson, and R. Sarkar. The shapley value in machine learning. In The 31st International Joint Conference on Artificial Intelligence and the 25th European Conference on Artificial Intelligence, pages 3572-5579, International Joint Conferences on Artificial Intelligence Organization, 2022.
[61] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
[62] A. Shapiro. Monte carlo sampling methods. Handbooks in operations research and management science, 10:353-425, 2003.
[63] L. S. Shapley. A value for n-person games. 1953.
[64] Shi. M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D). Zhou, D. Das, and J. Wei. Language models are multilingual -f reasoners. In The Eleventh International Conference on Learning Representations, 2023. URL IXp
httpp:://pperre/eev.neee///// www.:/enneetttoommmdmm//fa
[65] Shu, Y. Liao, L. Zhuo, C. Xu, L. Zhang, G. Zhang, H. Shi, L. Chen, Zhong, W. He, et al. Llava-mod: Making llava tiny via moe knowledge distillation. arXiv preprint arXiv:2408. 2024.
[66] M. Sun, 7. Liu, A. Bair, and J. Z. Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.1166) 2023.
[67] X. Sun, Y. Chen, Y. Huang, R. Xic. J. Zhu, K. Zhang, S. Li, Z. Yang. J. Han, X. Shu, et al. Hanguan-large: An open-source moe model with 52 billion activated parameters by tencent. arXiv preprint arXiv:24/1.02266, 2024.
[68] Q. Team. Qwen3, https: com/ Homm//weLMM//MM QQwenLM//eenn 2025.
14

---

[69] E. Winter. The shapley value. Handbook of game theory with economic applications, 3: 2025-2055 2002.
[70] Y. Wu, Xie, W. Huang, Zheng, F. Chao, and R. Ji. Mitigating gradient interference for efficient sparse fine-tuning of large language models, 2025. URL http:t::://ppeeeeiiee.... net/ 533Mei12mm
[71] H. Xiao, Z. Wang. Z. Zhu, J. Zhou, and J. Lu. Shapley-nas: Discovering operation contribution for neural architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 11 199--11901, June 2022.
[72] Z. Xie, Y. Zhang, C. Zhuang, Q. Shi, Z. Liu, J. Gu, and G. Zhang. Mode: A mixture-of-experts model with mutual distillation among the experts. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 16067-16075, 2024.
[73] A. Yang, B. Yang, B. Hui, B. Zheng. B. Yu, C. Zhou, C. Li, C. Li, D. Liu, and E. Huang. Qwen2 technical report. arXiv preprint arXiv:2407. /0671, 2024.
[74] A. Yang. B. Yang. B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15113 2024.
[75] H. Yang, H. Yin, M. Shen, Molchanov, H. Li, and J. Kautz. Global vision transformer pruning with hessian-aware saliency. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18547-18557, 2023.
[76] R. Zellers, Holtzman, Y. Bisk, Farhadi, and Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.07000 2019.
[77] J. Zhang, Q. Sun, J. Liu, L. Xiong, J. Pei, and K. Ren. Efficient sampling approaches to shapley value approximation. Proceedings of the ACM on Management of Data, 1(1):1-24, 2023.
[78] Z. Zhang, X. Liu, H. Cheng, C. Xu, and Gao. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts arXiv preprint arXiv:2407.09599, 2024.
[79] B. Zoph, I. Bello, S. Kumar, N. Du Y. Huang, J. Dean, N. Shazeer, and W. Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:220.008906, 2022.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction of this paper accurately reflect the contributions and scope of this paper, which is supported by the proposed methods and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of this paper and future work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification: We have provided a full set of assumptions each theorem section and provide complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided all the detailed information needed to reproduce the main experimental results of this paper in the experimental setup section in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The data used in this paper are all publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have elaborated all the experimental details such as datasets and hyperpa- rameters in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Following the experimental setup of previous methods, and because the experiments requires the use of a large number of resources, we only conducted a single experiment for the main results. However, we have shown the the error bar of set of experiment when running multiple times with different random seeds in Section J.3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the detailed description of the computational resources required to reproduce the experiments in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our research is conducted in accordance with the NeurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the social impacts, see the Section B for details.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
This paper does not present such risks because it does not publish risky data or models.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original paper that generated the code package or dataset is properly cited in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
The LLM models involved in this paper are all open source and available to the public. Our approach is used prune the open-source MoE. large language models, and we describe their use in Section 4.1
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

Appendix
Limitations and Future Work
23
Impact Statements
23
Properties of Shapley Value
24
D Proof of Theorem
24
Proof of Theorem 2
25
Proof of Theorem 3
26
Detailed Architecture Information of Different MoE Models
26
H
Analysis of Enumeration-Based MoE Pruning Methods
26
More Results
27
Multimodal Tasks
27
1.2 Using LoRA to Fine-tune the Pruned MoE Models
27
1.3 Integrate with Quantization Technique
28
J
More Ablation Study
28
J.1
Pruning Efficiency
28
Calibration Datasel
28
J.3 Robustness of Shaplcy-MoE under Different Random Seeds
29
Detailed Results for Zero-shot Tasks
29
A Limitations and Future Work
In this paper, propose novel MoE pruning framework based the Shapley value, which effectively prunes unimportant experts, reduces memory usage, and improves inference speed. However, under high pruning ratio settings, the accuracy of the pruned MoE model drops significantly, resulting in a performance gap compared to the original model. Although using LoRA fine-tuning helps narrow this gap, there still room for improvement to achieve high-ratio lossless pruning for MoE models. In the future, we plan to explore more efficient MoE compression techniques to further close the performance gap between compressed model and the original model under high compression ratio settings.
B Impact Statements
This paper proposes Shapley value-based framework for pruning MoE models. We have not found any direct negative social impacts caused by the algorithm itself. In fact, we believe that introducing our method to the community has tremendous social value. By pruning unimportant MoE experts, can significantly reduce the number of parameters in MoE models while retaining their functionality. Therefore, this helps to reduce the computational resource consumption of MoE models and contributes to lowering carbon emissions caused by GPU computation.
23

---

C
Properties of Shapley Value
The Shapley value satisfies four axiomatic requirements 663
Efficiency: Total Shapley Value equal the model's full-set performance gain: Σ Extes PEς
V(E) - -V))..
Symmetry: Experts contributing equally to all coalitions receive identical values: dE! = 12
VS _ {E), ), V(S u {E!)) = ((S u (E)
Null Player: Experts with zero marginal impact get Shapley value of 0: ΦE
vS S (F)) V(S).
Linearity: Values are preserved under linear combinations: dF (aV + BW) v - (W).
The above Shapley Value's axiomatic guarantees ensure expert contributions are quantified fairly across all coalitional interactions. This enables precise identification of redundant or underperforming experts (via Null Player property) while preserving critical collaborative experts (via Efficiency, Symmetry and Linearity property). forming a theoretically grounded basis for expert pruning in MoE models.
D Proof of Theorem 1
Proof.
The Shapley value OF can be decomposed into contributions from two disjoint events:
OF
E A
P(22)
E| -22 P(-2))
(11)
Non truncated regions
Truncated regions
where A ((S VSS). and Ω denotes the event V(S) +V(E).
The truncated estimator
trunc approximates the first term through Monte Carlo sampling:
Fy
M Σ
A (m)
19
(12)
EE
The estimation error can be bounded by:
Atunne
A
-
PP(-2)) + E[A [] P())
trunc o
(13)
E
E
Bias
Variance term
Bias Analysis: By the theorem's assumption, [E[A| -22 S c. Therefore, E -Q] P(-2)) S PP(())).
(14)
Variance Analysis: The variance term corresponds to the Monte Carlo estimation error of [E|A Q] - P(2). Since A bounded within [-c,e] the truncated regions and potentially within larger range [-C, C) in non-truncated regions (where C max (V(S U (E!)) V(S))), apply Hoeffding's s inequality (27). For M independent samples, with probability 2 1 - 5:
C2 log(2/8)
E [2] P(Ω)
Itrme D
(15)
2M
Assuming A E [- for constant C' (e.g., C V(E)), the variance term is bounded by
log(2/5) 2M
after normalizing C.
Combining both terms yields the stated error bound:
1og(2/6)
|dE;
Srrnee
S e P(V(()<+VV(E)
(16)
2M
24

---

E Proof of Theorem 2
Proof. Proof of Unbiasedness. Let y denote the set of all permutations of the expert set & The true Shapley value can be expressed as:
EE
Σ
[ u (E)) V(S
(17)
N
xelly
where S is the set experts preceding E in permutation т.
The RGIS estimator computes:
M [( U (EE))
RRGIS
Σ M
1
-E) v(s(m)) w(()) mmm]
(18)
E
m=
where (m()))
1/N Q(a()))
is the importance weight for permutation ((m), and mm) indicates whether
E was evaluated before truncation.
Taking expectation over the sampling distribution Q, and assuming truncation preserves unbiasedness preserving unbiasedness through controlled approximation):
(truncation error bounded by Theorem |
Eq
RGIS E
EQQ [(V(SU - V(S)) I.]
Σ Q(T) (v(s u - v(( 1/N
Q(*) L(a))
Σ TENl
V(S ( x
(EE)) - v(( L(T).
(19)
Under the truncation condition in Eq. 5 (x) equals for permutations where E, is placed before performance collapse. Since truncation only affects coalitions with negligible contributions (as shown Theorem T.. we have:
EQ
RGIS E
Σ Nelly
[VS U (EE)) - V(S = dE.
(20)
N
Thus, the RGIS estimator is unbiased.
Proof of Variance Reduction. The variance of the standard Monte Carlo estimator is:
Var PF: = M Varæ-Uunffom (v(s)) (EE V(((
(21)
For the RGIS estimator with importance sampling:
Var
RGISS E
EQQ
((V(S (E)) ((V() w²)
M *
M
(V(s U (E()) - V(S )²
Σ
Q(T))
M o2
M
N V(Q(T)
xellx
(Vis U(E - V(S!")) 2
1
1 N
Σ Relly
1
(22)
M
Q(T)
M
When activation probabilities {pt} correlate with Shapley values, the Plackett-Luce distribution Q(TT) assigns higher probability permutations where important experts (with larger |d ) appear
earlier. This makes Q(Tr) inversely proportional (V(S.) U (E()) v(s! for critical experts,
thereby reducing the summation term ΣS ((S("
Consequently:
Q(7)
RRIS E
Var (3MC)
Var
(23)
completing the proof.
25

---

Proof of Theorem 3
Proof. Under the additivity assumption,
V (pruned))
Σ PE
(24)
E Eppuse
Therefore, the performance degradation is
V(E) ((ppnn))))
PE!
(25)
E E.
For each pruned expert E Eprand (i.e., dø < E), we have Φg SØg Ø +
(26)
with probability at least 8. Let R = |(E) & Epruned } be the number of pruned experts. Then,
Σ
O < R(e + 7) S N(c + n))
(27)
E
Therefore complete the proof.
G Detailed Architecture Information of Different MoE Models
We have summarized the architecture information of different MoE models in Table including the total number of model parameters, the number of parameters activated per token during inference. the number of routing experts in each layer, and the number of experts in each layer activated per token during inference.
Table 6: Detailed architecture information of different MoE models
Model
Total params
Activated params
Experts num
Activated experts num
14.3B
2.7B
60
Qwen2-57B-A14R
57.48
14.08
64
8
Qwen3-30B-A33
30.58
3.38
128
DeepSeckMoE-16B
16.4B
2.8B
64
6
DeepSeek- V2-Lite
15.7B
2.4B
64
6
Mixtral-8x7B
46.78
13.0B
8
H Analysis of Enumeration-Based MoE Pruning Methods
The previous NAEE 49 and CD-MoE 6 methods proposed determining which experts to prune based the loss difference between the pruned MoE model and the original MoE model. They identify the optimal subset of experts by enumerating all possible combinations to achieve the best accuracy. However, in the NAEE method, it requires C'N enumerations to determine the optimal subset of experts, where N and Nprrnne represent the number of experts in the original and the pruned MoE models, respectively. For the CD-MoE method, requires (N NTprun)) N/2 enumerations. This is practically infeasible for large-scale MoE models with large number of experts. For example, for the Qwwn3-30B-Ah this would require approximately 2.4 x 1037 and 12, 288 enumerations for the NAEE and CD-MoEE methods, respectively. In contrast, our Shapley-MoF method can obtain the pruned MoE model within just 36 minutes, which is highly efficient.
model with 128 experts, when pruning the model to retain 64 experts,
26

---

I
More Results
L.1 Multimodal Tasks
We further apply our Shapley-MoB method prume multimodal MoE models, demonstrating the applicability of our approach in multimodal scenarios. Specifically, we our approach to prune the MoE-LLaVA SarAA-QQe--lae 1.8B-4e model [43] a multimodal MoE model with 12 MoE layers, each containing 4 experts, where only 2 experts are dynamically activated per token. The total parameter count the MoE-LLaVA-Qwen 1.8B-4c model 3.IB, with 2.2B parameters dynamically activated per token. We prune 25% of the model's experts and evaluate the performance of the pruned model on various visual question answering and reasoning benchmarks, including VQAv2 [2I], GQA [4] and ScienceQA-IME 48 The results in Table 7 show that our method outperforms other MoE pruning methods, achieving improvements of 1.7%, 2.0%, and 2.4% over the best- pest-performing method on the VQAv2, GQA and ScienceQA-IMG datasets, respectively.
Table 7: Performance of the MoE-LLaVA-QQen--..BB-- model with 25% experts pruned on VQA, VQAv2 and ScienceQA-IMG datasets.
Method
VQAv2
GQA
Science(AAIIMG
None
76.20
61.50
63.10
Random
66.90
51.20
52.70
Gating Score
68.10
53.90
54.70
Frequency
68.40
54.70
55.80
Expert Trim
67.80
54.30
55.10
Shapley-MoE
70.10
58.20
I.2 Using LoRA to Fine-tune the Pruned MoE Models
To address the notable accuracy degradation observed in pruned MoE model under high pruning rates, we further validate the effectiveness of applying LoRA [29 fine-tuning mitigate the accuracy gap between highly pruned MoE model and their original counterpart. Specifically, we select 1000 samples from the Alpaca-GPTI [57 dataset to fine-tune Qwenl.5-MoffA22..7 model, which has undergone pruning with 50% of its experts removed. During fine-tuning we set the LoRA rank to We evaluate the perplexity and average zero-shot accuracy pruned MoE model generated via different pruning methods, both with and without LoRA fine-tuning. As shown in Table g LoRA fine-tuning significantly enhances the accuracy of the pruned MoE model, effectively narrowing the performance gap between the pruned model and the original model. Furthermore, since the pruned model produced by our proposed Shapley-Me MMo method demonstrate superior accuracy, they exhibit greater capacity for performance recovery through LoRA fine-tuning Notably, the performance advantage of our Shapley-Mof method persists even after fine-tuning.
Table The comparison perplexity and average zero-shot accuracy of the Qwenl 1.eee...-M---2.. model after pruning 50% experts and performing LoRA fine-tuning.
Method
Fine-tuning
Perplexity ())
Accuracy (f)
None
N.A.
7.01
61.70
Random
N.A.
21.87
45.25
Random
LoRA
13.66
50.24
Gating Score
N.A.
20.10
46.30
Gating Score
LoRA
12.90
51.20
Frequency
N.A.
19.78
46.84
Frequency
LoRA
12.60
51.45
Expert Trim
N.A.
19.98
46.35
Expert Trim
LoRA
12.97
51.21
Shapley-MoF Shapley-MoE
N.A
17.57
49.01
LoRA
11.04
53.20
27

---

I.3 Integrate with Quantization Technique
We further demonstrate the additional benefits of compression and acceleration brought by quantizing the pruned MoE model. Specifically, we first prune 25% experts of the Qwen svenl.S-MoB. .2.MoE-A2..7 7B model and then apply the AWQ 44 method quantize the pruned model 4 bits. We also measure NVIDIA A100 80GB GPU.
the memory usage and inference speed of the compressed model The results in Table and Table TO show that quantizing the pruned model to bits further reduces memory usage and improves inference speed. Additionally, it is worth noting that the pruned model obtained by our method maintains optimal performance even after quantization, which indicates that better-performing pruned models are more compatible with other compression techniques, resulting in superior compression models.
Table 9: The WikiText-2 perplexity performance of the Qwenl.5-MoE-A2.77 model after combining the pruning and quantization technique.
Pruning Method
bits
PPL
Random W/AWQ
11.01
12.04
Gating Score W/AWQ
10.39
11.12
Frequency W/AWQ
10.08
10.89
Expert Trim
16
10.28
W/AWQ
11.02
Shapley-MoB
16
9.69
W/AWQ
10.34
Table 10: The GPU memory usage and inference speed of quantized pruned Qwenl 55M-0--..7 model.
bits
Memory (GB)
Reduction
Throughput (tokens/s)
Specdup
Unpruned x 16 bit
29.30
1.00x
149.90
1.00
Pruned x 16 bit
23.50
1.25x
188.50
1.26x
Pruned 4 bit
7.70
4.13x
593.80
3.96
J More Ablation Study
J.1 Pruning Efficiency
we report the time required to prune different MoE models using our Shapley-MoE method. The results were obtained NVIDIA A 100 80GB GPUs. For MoE models with different parameter sizes, Shapley-MoE method only takes tens of minutes to obtain the pruned model. This demonstrates that our Shapley-MoF. method is highly efficient and can quickly produce pruned MoE models. Furthermore, we believe that the efficiency advantage of our pruning method will remain for MoE models with even larger parameter sizes.
In Table I1
Table 11: Pruning cost of our Shapley-Me -MoE method (in minutes). Qwen2-57B
Qwen1.5-MoE
Qwen3-30B
DeepSeekMoE
DeepSeek -V2-Lite
Mixtral
Model
-A2.7B
-A14B
-A3B
-16B
-8x7B
Cost
36
112
172
50
40
42
J.2 Calibration Dataset
Number of Calibration Samples. We illustrate the impact of the number of calibration samples on the WikiText-2 perplexity of the pruned MoE. model Figure 4 Specifically, we present the perplexity of the Qwenl.5-MoE-A2. A2.7B model with 25% of experts pruned. We observe that as the
28

---

number of calibration samples increases, the perplexity of the pruned model decreases gradually. However, when the number of calibration samples reaches 256, the perplexity of the model does not decrease further. Considering that increasing calibration samples leads to longer pruning process time and the perplexity gain brought by the increase calibration samples, we choose 128 samples for calibration.
10.8
Perplexty
10.4
10.0
9.8
16
Calibration Samples
Figure 4: The WikiText-2 perplexity of the Seeenl.5MMoE-2... model with 25% experts pruned under different calibration samples.
Dpmaif-specific Calibration Datasets. We investigate the impact of using domain-specific datasets for calibration on the performance of pruned models in specialized domains. Specifically, We construct domain-specific calibration dataset containing 128 examples by randomly sampling sentences from the MATH [26) and CodeQA [46] training sets. These datasets represent the arithmetic and code-generation domains and are used to calibrate the pruned w.wenl.5-Mo-A2.. which 25% of the experts have been pruned. The performance of the pruned model is subsequently evaluated on the GSM8K dataset for 8-shot accuracy and the HumanEval dataset for O-shot accuracy, with the results summarized in Table 12 The results indicate that, compared using the C4 dataset for pruning model calibration, employing domain-spednt improvements in the pruned model's accuracy on the GSMSK and HumanEyal tasks. This indicates that when handling tasks specific domains, using datasets designed for these specific tasks for calibration achieve better MoE model pruning results compared to using general pretraining datasets.
model, in
calibration datasets yields notable
Table 12: Accuracy of the 1 .5-MoE-A2.7B model with 25% experts pruned on the GSM8K and HumanEval datasets when using domain-specifee calibration datasets.
Calibration
GSMSK
Calibration
HumanEval
Unpruned model
61.50
Unpruned model
34.20
C4
31.02
19.24
MATH
43.76
CodeQA
26.56
J.3 Robustness of Shapley-MoB under Different Random Seeds
To demonstrate the robustness of our method, we report the WikiText2 perplexity of the Qwenl.5- MoE-A2.7B model pruned by 50% experts using Shapley-Mod across five random seeds and different calibration sets in Table [3] From the experimental results, we can observe that the variance among different random seeds is very low, and Shapley-MoF consistently outperforms other MoE pruning algorithms for different random seeds, indicating the robustness of Shapley-MoE.
K Detailed Results for Zero-shot Tasks
In this section, we present the detailed results of the shot accuracy of the pruned MoE model on seven tasks.
29

---

Table 13: WikiText2 perplexity of the Qwenl.5-MoE-A22.7 model with 50% experts pruned across
five random seeds.
Method
Perplexity
Random
21.87 (-) 0,13
Gating Score
20.10 (-)00(())))) 0.122
Frequency
19.78 (-) 0.08
Expert Trim
19.98 (=) 0.08
Shapley-MoE
17.57 (=) 0.06
Table 14: Zero-shot task results of MoE models with 25% experts pruned.
Model
Method
HellaSwag
Winogrande
BoolQ
OBQA
PIQA
ARC-e
ARC-c
Mean
None
$7.92
68.75
79.78
30.40
80.09
73.06
41.89
61.70
Random
$1.20
62.38
71.35
25.60
72.88
63.98
36.11
54.79
Owenl.5-MoE-A2.78
Gating Score
52.10
63.01
72.41
25.80
73.01
64.35
37.24
55.42
Frequency
$2.21
65.31
37.78
55.63
Expert Thim
$1.89
63.12
71.90
25.00
73.20
63.90
37.14
55.16
Shapley-MoE
53.90
65.68
74.89
27.00
74.09
65.01
39.12
57.10
None
62.86
74.03
86.39
32.60
80.96
74.75
46.16
65.39
Random
$7.86
70.92
82.74
30,20
76.89
71.53
42,20
61.76
Qwen2-57H-A14h
Gating Score
58.16
71.91
82.89
30.40
76.98
71.98
42,81
62.16
Frequency
58.02
72.25
83.10
30.50
77.05
72.45
42.65
62.29
Expert Trim
$7.78
72.18
42.16
62.06
Shapley-MoF
59.15
72.97
84.56
31.00
78.12
72.56
43,02
63.05
None
59.58
70.24
88.62
34.40
79.25
52.82
66.36
Random
49.52
62.77
69.24
42.43
Qwen3-30B-A3B
Gating Score
50.23
62.98
82.18
29.20
70.03
70.23
43.01
58:27
Frequency
50.68
63.20
82.56
29.60
70.27
70.56
43.90
58.68
Expert Trim
50.64
63.37
82.01
29.10
69.89
69.79
43.09
58.27
Shapley-MoE
52.35
71.56
71.23
45.80
None
58.09
70.40
72.91
32,20
78.67
75.84
45,31
61.92
Random
46.42
64.04
69.46
24.80
70.33
63.60
34,92
53.37
DeepSeekMoE-Ion
Gating Soore
47.09
65.35
70.56
25.30
71.26
64.32
35.75
54.23
Frequency
47.67
66.09
71.05
25,40
7..87
64.09
35,90
54.58
Expert Trim
46.45
64,95
69.89
24.80
70.43
63,89
35,86
53.75
Shapley-MoE
50.13
67.15
71.45
28.00
72.09
66.45
38,90
56.31
None
58.68
71.35
79.72
34.20
80.14
78.37
46.67
64.16
Random
47.48
66.06
72.60
25.00
70.89
64.65
35,32
54.57
DeepScck-V2..e V2-LLte
Gating Score
49.02
66.25
72.90
25.40
71.67
64.91
37.31
55.35
Frequency
50.21
67.24
73.00
25.10
71.98
65.78
38.02
Expert Trim
49.43
73.24
24.30
70.91
65.29
38.00
55.44
Shapley-MoF
51.20
67.29
76.01
26.30
73.10
67.67
39.45
57.29
30

---

Table 15: Zero-shot task results of MoE models with 50% experts pruned.
Model
Method
HellaSwag
Winogrande
BoolQ
OBQA
PIQA
ARC-e
ARC-c
Mean
None
57.92
68.75
79.78
30.40
80.09
73.06
41.89
61.70
Random
40.27
57.72
60.25
18.80
62.49
49.88
27.33
45.25
Owenl.5-MoB-Ad
Gating Score
42,90
58.93
61.65
19.40
62.90
50.28
28.01
46.30
Frequency
43.88
59.46
50.80
28.69
46.84
Expert Thim
42.98
58.57
61.89
19.10
62.96
50.57
28.35
46.35
Shapley-MoE
45.78
60,90
65.09
21.50
64.97
53,96
30.89
49.01
None
62.86
74.03
86.39
32,60
80.96
74.75
46.16
65.39
Rundom
53.57
68.13
78.78
27.20
73.97
67.51
39.87
58.44
Qwen2-57B-A14B
Gating Score
54,13
69.14
79.34
27.60
74.23
68.14
40.26
58.98
Frequency
54.46
69.45
79.67
27.90
74.67
68.56
40.57
59.33
Expert Trim
54.36
69.01
79.02
27.30
74.87
40.87
Shapley-MoF.
56.78
70.92
81.13
28.90
75.78
69.18
41,89
60.65
None
59.58
70.24
88.62
34.40
79.60
79.25
52.82
66.36
Random
36.36
66.16
20.00
48.78
28.69
Qwen3-30B-A3B
Gating Score
38.89
$6.89
68.98
20.70
58.87
49.56
29.65
46.22
Frequency
39.45
57.90
67.90
20.80
58.98
49.79
30.98
46.54
Expert Trim
38.87
57.10
69.12
20.90
58.75
49.16
29.98
46.27
Shupley-Mok
41.95
51.24
31.98
None
58.09
70.40
72.91
32:20
78.67
75.84
45.31
61.92
Random
33.67
55.35
58.93
16.00
59.55
43.49
23.49
41.50
DeepSeekMoE-16ft
Gating Soore
34.67
56.36
60.01
17.10
60.12
44.23
25.09
42.51
Frequency
33.90
57.76
60.13
18.00
59.89
44.21
26.00
42.84
Expert Trim
34.56
57.01
59.79
18.10
59.81
43,13
26.29
42.67
Shapley-MoE
36.98
$9.01
61.34
19,30
62.02
46.67
28.90
44,89
None
58.68
71.35
79.72
34,20
80.14
78.37
46.67
64,16
Random
34.72
57.41
63.31
21.80
60.53
47.11
24.51
44.20
DeepSeek-VV V2-Lite
Gating Score
36.10
59.10
64.57
22,30
61.94
49.01
26.19
45.60
Frequency
37.64
59.57
65.08
23.00
62.45
48.45
25.56
Expert Trim
38.58
59.21
65.12
23.10
61.23
48.67
25.06
45.85
Shapley-MoE
39.45
60.34
67.13
24.00
62.86
49.05
26.76
47.08
31


Reviews:
[{"review_id": "kYvuB0mDXM", "reviewer": "Reviewer_ysJ4", "summary": "This paper proposes Shapley-MoE, a pruning framework that quantifies each expert’s importance using Shapley values. The authors estimate each expert’s marginal contribution to model performance via Monte Carlo approximation, and incorporate two key optimizations, Early Truncation and Router-Guided Importance Sampling, to reduce the theoretical exponential complexity to an approximate quadratic level while preserving accuracy. Experiments are conducted on various public MoE models including Qwen, DeepSeek, and Mixtral, demonstrating improvements in perplexity and zero-shot task accuracy under 25%–75% pruning rates, as well as reductions in GPU memory usage and inference latency.", "questions": "1. How do expert pruning, channel pruning, quantization, and their combinations compare in terms of memory-performance and speed-performance trade-off curves?\n2. How does Shapley-MoE perform against baseline methods (e.g., Random, Frequency) under random datasets and task distributions, where its calibration strategy may not align with task-specific characteristics?", "limitations": "Please refer to the weaknesses above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Applying Shapley values to assess expert importance in MoE is a novel and well-motivated approach, showing promising results in both efficiency and performance.\n- The paper includes comprehensive experiments and ablation studies to support its conclusions.\n\nWeaknesses\n- The paper lacks a thorough discussion on the necessity of expert pruning. Given that the core motivation is to reduce memory usage and improve inference speed, it would be more convincing to include comparisons with alternative approaches such as channel pruning and quantization. These methods may achieve similar resource reductions with potentially smaller performance degradation. Although the authors suggest that Shapley-MoE can be combined with such methods, it remains unclear whether expert pruning lies on the Pareto frontier when considering the trade-off between performance and efficiency.\n- The paper acknowledges that different tasks may require different calibration datasets for expert pruning, which aligns with the conceptual understanding of MoE experts. However, this raises questions about the fairness of comparisons with baseline methods like Random or Frequency-based pruning. Specifically, under randomly selected datasets and tasks, does Shapley-MoE still demonstrate consistent advantages over prior approaches?", "was_revised": true, "final_justification": "The authors’ response has addressed my concerns well, and I will maintain my score.", "rebuttal": "**We greatly appreciate your time, detailed comments, and valuable suggestions. We understand the concerns you raised and we're pleased to address these concerns.**\n\n> Weakness & Question 1: The paper lacks a thorough discussion on the necessity of expert pruning. Given that the core motivation is to reduce memory usage and improve inference speed, it would be more convincing to include comparisons with alternative approaches such as channel pruning and quantization. These methods may achieve similar resource reductions with potentially smaller performance degradation. Although the authors suggest that Shapley-MoE can be combined with such methods, it remains unclear whether expert pruning lies on the Pareto frontier when considering the trade-off between performance and efficiency. How do expert pruning, channel pruning, quantization, and their combinations compare in terms of memory-performance and speed-performance trade-off curves?\n\nThank you for your thoughtful and constructive feedback. According to your suggestions, we demonstrate the compression effects of expert pruning, channel pruning, and quantization on the Qwen2-57B-A14B model. Specifically, we pruned 25% of the experts using our method, pruned 25% of the channels in each expert using magnitude pruning, and performed 4-bit quantization for the Qwen2-57B-A14B model using the GPTQ method. The experimental results are as follows:\n|Method|WikiText-2 perplexity|Average zero-shot accuracy|Memory (GB)|Speedup|\n|-|-|-|-|-|\n|Qwen2-57B-A14B|5.86|65.39%|110.29|1.00$\\times$|\n|GPTQ quantization|6.14|65.01%|30.38|1.27$\\times$|\n|Channel pruning|7.99|61.40%|90.14|1.21$\\times$|\n|Shapley-MoE|6.78|63.05%|88.46|1.26$\\times$|\n|Shapley-MoE w/GPTQ quantization|7.12|62.78%|25.37|1.51$\\times$|\n\nFrom the data in the table, we can observe the following:\n\n(1) **Our method has advantages over channel pruning:** Our Shapley-MoE method outperforms the channel pruning method, with a perplexity that is 1.21 lower and an average zero-shot accuracy that is 1.65% higher. We believe that although channel pruning operates at a finer-grained level to prune the model, it disrupts important experts, leading to a more severe performance degradation. In contrast, our method preserves the full capabilities of relatively important experts, which enables it to achieve better performance.\n\n(2) **Compared with the original model, our method has only a slight performance drop, while it can significantly reduce the memory usage and improve the inference speed:** Compared with the original model, our method only increases the perplexity by 0.92 and decreases the average zero-shot accuracy by merely 2.34%. Meanwhile, it can achieve a 19.79% reduction in memory usage and a 1.26$\\times$ improvement in inference speed. Compared with quantization, the aforementioned significant reduction and inference speed improvement can be obtained on any device without the need for a specific inference engine.\n        \n(3) **Our method still has advantages over quantization:** The perplexity of our method is only 0.64 higher than that of the quantization method, and the average zero-shot accuracy is only 1.96% lower. Furthermore, combining our method with the quantization method can further advance the Pareto frontier of MoE model compression. Shapley-MoE with AWQ quantization only increases the perplexity by 1.26 and decreases the average zero-shot accuracy by 2.61% compared with the original model, while achieving a 77% reduction in memory usage and a 1.51$\\times$ improvement in inference speed. This further demonstrates the superiority of our method.\n\nFurthermore, we would also like to make the following clarifications regarding the necessity of expert pruning and its comparison with channel pruning and quantization methods.\n1. **On the necessity of expert pruning**: We agree that a more explicit discussion of why expert pruning is essential for MoE would enhance the paper. MoE architectures, as noted in our methodology section, feature a large number of sparsely activated experts (often exceeding 1000 in models like DeepSeekMoE-16B or Qwen2-57B-A14B), leading to significant memory and computational overhead during inference. Unlike dense models, MoE's conditional activation means many experts contribute marginally or redundantly, making expert-level pruning a natural and efficient way to reduce parameters while preserving the model's sparse computation benefits. This is particularly crucial for deployment on resource-constrained devices.\n2. **Comparison with pruning and quantization method**: You are correct that methods like channel pruning (e.g., removing less important channels in FFNs) and quantization (e.g., 4-bit or 8-bit weight compression) can achieve similar resource reductions. However, these approaches are orthogonal to expert pruning: channel pruning operates at a finer granularity within experts, potentially leading to higher performance degradation in sparse MoE settings due to disruption of expert specialization, while quantization reduces bit-width but specific quantized kernels are required to achieve model memory reduction and improved inference speed. In contrast, expert pruning is hardware-agnostic and can reduce model memory usage and enhance inference speed on any device.\n\nThank you again for your constructive suggestions. We will include the above results in the final version of our paper.\n\n> Weakness & Question 2: The paper acknowledges that different tasks may require different calibration datasets for expert pruning, which aligns with the conceptual understanding of MoE experts. However, this raises questions about the fairness of comparisons with baseline methods like Random or Frequency-based pruning. Specifically, under randomly selected datasets and tasks, does Shapley-MoE still demonstrate consistent advantages over prior approaches? How does Shapley-MoE perform against baseline methods (e.g., Random, Frequency) under random datasets and task distributions, where its calibration strategy may not align with task-specific characteristics?\n\nWe appreciate the reviewer’s insightful comment regarding the role of calibration datasets in expert pruning. According to your suggestions, we verify the effectiveness of our method on random datasets, we randomly selected total 128 examples from C4, Pile, and PTB to form the calibration dataset. These datasets are also commonly used as calibration datasets in previous compression methods (such as Wanda [3], SparseGPT [4], and OmniQuant [5]). We pruned 25% of the experts in Qwen1.5-MoE-A2.7B model. The experimental results are as follows:\n|Method|WikiText-2 perplexity|Average zero-shot accuracy|\n|-|-|-|\n|Random|11.01|54.79|\n|Gating Score|10.51|55.12|\n|Frequency|10.17|55.51|\n|Expert Trim|10.41|55.01|\n|**Shapley-MoE**|**9.80**|**57.03**|\n\n**Our Shapley-MoE method outperforms other baseline methods.** For example, compared with the best Frequency baseline, Shapley-MoE has a perplexity that is 0.37 lower and an average zero-shot accuracy that is 1.52% higher. We believe Shapley-MoE maintains advantages due to its game-theoretic foundation: it quantifies experts' marginal contributions to overall model performance, capturing inter-expert collaborations that random or frequency-based methods overlook. Frequency-based approach only consider isolated activation counts and gating score-based approach only consider isolated gating weight magnitude, which may not reflect true importance in diverse coalitions, while random pruning ignores any structure and task. Therefore, our method has advantages in any data distribution.\n\nFurthermore, we would also like to make the following clarifications regarding the use of the calibration dataset in our paper.\n1. **To ensure fairness in our comparisons, all baseline methods (including Random and Frequency-based pruning) were evaluated using the same calibration dataset as Shapley-MoE. We follow the settings of Expert pruning methods for MoE models (such as NAEE [1] and CD-MoE [2] method), randomly sampling 128 examples from the C4 training dataset as calibration data.**\n\n2. In addition, the calibration datasets used in our paper are all datasets frequently employed in previous compression works. C4 dataset has been widely utilized in methods such as Wanda [3], SparseGPT [4] and OmniQuant [5]. \n\n3. Furthermore, NAEE [1] found that using the MATH dataset for calibration during the expert pruning process of MoE models can improve the accuracy of the pruned models on mathematical tasks, as this enables the identification of experts that are more important for mathematical tasks. Our paper follows this setup and employs domain-specific calibration datasets to improve the accuracy of the pruned model on domain-specific tasks.\n\n[1] Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models. ACL 2024.\n\n[2] Condense, don’t just prune: Enhancing efficiency and performance in moe layer pruning. arXiv.\n\n[3] A Simple and Effective Pruning Approach for Large Language Models. ICLR 2024.\n\n[4] SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. ICML 2023.\n\n[5] Omniquant: Omnidirectionally calibrated quantization for large language models. ICLR 2024.\n\n**Finally, we hope that our response has addressed your concerns, and we kindly ask for a more favorable evaluation of our work. We will incorporate the above clarification into the final version of our paper. Thank you!**"}, {"review_id": "AF2AZG562j", "reviewer": "Reviewer_ovkS", "summary": "This paper proposes a new pruning method for Mixture-of-Experts models to discover more important experts. The authors introduce the Shapley value to estimate the importance of each expert and approximate the Shapley value by using a Monte Carlo sampling strategy. Early Truncation and Router-Guided Importance Sampling are also developed for the improvement of sampling accuracy and efficiency. Theoretical and experimental analyses demonstrate that the proposed method showcases a promising way for MoE model pruning.", "questions": "- Please provide more experimental results for different percentages of pruning.\n- Please illustrate the motivation for using Monte Carlo sampling to approximate the Shapley estimation\n- Can this method be employed in other LLM architectures? If yes, how to achieve this goal?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper innovatively extends Shapley value to evaluate the importance of experts and proposes a Monte Carlo sampling approach for approximating Shapley value, which incorporates early truncation and router-guided importance sampling.  \n- The authors provide theoretical guarantees for the error bound of early truncation Shapley estimation as well as the unbiasedness and variance reduction of RGIS.\n- The experimental results illustrate the robustness and the effectiveness of the proposed method.\nWeaknesses\n- The differences between the ground-truth and estimated value: The authors use Monte Carlo sampling to approximate the Shapley value, and calculating the ground-truth Shapley value is an NP-hard problem. Despite this, can the authors measure the differences between the ground-truth Shapley value and the estimated value?\n- More experimental results: The experiments are conducted in the setting of 25% and 50% pruning, what about 75% pruning or else? Can you provide a performance trend in terms of different percentages of pruning?", "was_revised": true, "final_justification": "After reading the rebuttal, I think the authors clearly explain their contributions and address some of my concerns, especially the comparison of the ground-truth Shapley value and the estimated value. The motivation of using Monte Carlo sampling presented in the main paper would be helpful. The question 3 I asked is about the application of other LLM architectures with no expert, which may be presented as potential limitations. I'm maintaining my overall score and I hope these suggestions are helpful for improving clarity and presentation.", "rebuttal": "**We appreciate the time and effort you put in reviewing our work, as well as your detailed comments and valuable questions. We understand the concerns you raised and we're pleased to address these concerns.**\n\n> Weakness 1: The differences between the ground-truth and estimated value: The authors use Monte Carlo sampling to approximate the Shapley value, and calculating the ground-truth Shapley value is an NP-hard problem. Despite this, can the authors measure the differences between the ground-truth Shapley value and the estimated value?\n\nThank you for your suggestions. We have conducted experimental measurements below. Considering that the time complexity of calculating the ground-truth Shapley value using Equation. 3 is $\\mathcal{O}(2^N)$, where $N$ is the total number of experts in the MoE model, and the number of experts in each MoE layer of the Qwen series and DeepSeek series MoE models exceeds 60. Therefore, we used Equation. 3 in our paper to calculate the ground-truth Shapley value of the 8 experts in the first MoE layer of the Mixtral-8x7B model, and adopted our proposed Shapley-MoE method to calculate the estimated Shapley value. The experimental results are as follows:\n|Expert id|0|1|2|3|4|5|6|7|\n|-|-|-|-|-|-|-|-|-|\n|Ground-truth|2.54e-4|3.03e-4|-2.94e-4|-1.19e-5|2.68e-3|7.24e-5|8.08e-5|-6.54e-5|\n|Estimated|2.66e-4|3.12e-4|-2.74e-4|-1.45e-5|2.70e-3|7.67e-5|8.00e-5|-6.14e-5|\n|Difference|+0.12e-4|+0.09e-4|+0.20e-4|-0.26e-5|+0.02e-3|+0.43e-5|-0.08e-5|+0.40e-5|\n\nFrom the above experimental results, we can draw the following conclusions:\n1. **The difference between the ground-truth Shapley value and the estimated value is very small, which proves that using our Shapley-MoE method to estimate the Shapley value is highly effective.** In addition, since calculating the ground-truth Shapley value is an NP-hard problem, it is not feasible in practice. However, our method provides an effective reference scheme for estimating Shapley value in practice.\n2. Both the ground-truth and estimated Shapley values follow the descending order of 2/4/3/5/6/7/1/0. **This means that the experts of MoE pruned based on the estimated Shapley value are the same as those pruned based on the ground-truth Shapley value.** In this case, the accuracy of the pruned model obtained by Shapley-MoE is the same as that of the pruned model based on the ground-truth Shapley value. This proves that the Shapley value estimated by our Shapley-MoE method will not affect the final pruning accuracy.\n\nThank you again for your insightful opinions. We will incorporate the above experimental results into the final version of our paper.\n\n> Weakness 2 & Question 1: More experimental results: The experiments are conducted in the setting of 25% and 50% pruning, what about 75% pruning or else? Can you provide a performance trend in terms of different percentages of pruning?\n\nThank you for your suggestion. **We have shown the experimental results under different pruning rates in Section 4.3. In Table 3, we show the WikiText-2 perplexity performance of the pruned Mixtral-8x7B model under a wider range of pruning rates, from 12.5% to 75%.** The experimental results show that:\n1. As the pruning rate increases, the perplexity of the pruned model shows an upward trend, which indicates that the performance of the pruned model shows a downward trend with the increase of the pruning rate.\n2. Across different pruning rate settings, the perplexity of the pruned models obtained by our Shapley-MoE method consistently remains lower than that of pruned models obtained by other methods, which fully demonstrates the robustness and effectiveness of our Shapley-MoE method.\n\n> Question 2: Please illustrate the motivation for using Monte Carlo sampling to approximate the Shapley estimation.\n\nThank you for your question. **The motivation for using Monte Carlo sampling to approximate the Shapley estimation is to make the calculation of Shapley value feasible for large-scale MoE models.** While the Shapley value theoretically provides a metric for quantifying expert contributions, exact computation through Eq. 3 requires evaluating all $2^N$ expert subsets, which is a prohibitive proposition for modern MoE architectures where total expert count $N$ often exceeds $10^3$. This exponential complexity motivates the development of efficient approximation methods. Therefore, we employ the Monte Carlo sampling approach to estimate Shapley value through empirical expectation over expert permutations. By averaging over $M$ permutations, Monte Carlo sampling reduces the computational complexity from $\\mathcal{O}(2^N)$ to $\\mathcal{O}(MN)$, making the calculation of Shapley value feasible for large-scale MoE models.\n\n> Question 3: Can this method be employed in other LLM architectures? If yes, how to achieve this goal?\n\nThank you for your question.\n1. Our method is applicable to expert pruning for various MoE model architectures. **In Sections 4.2 and 4.3, we present the experimental results of our method on various popular open-source MoE architectures,** including the Qwen series (Qwen1.5-MoE-A2.7B, Qwen2-57B-A14B, and Qwen3-30B-A3B), the DeepSeek series (DeepSeekMoE-16B and DeepSeek-V2-Lite), and the Mixtral series (Mixtral-8x7B). **In addition, our method is also applicable to expert pruning for multimodal MoE models.** In Section I.1, we present the experimental results of the MoE-LLaVA-Qwen-1.8B-4e model.\n2. Our method is applicable to expert pruning for all models with mixture-of-experts architectures that include experts. However, for models with other dense LLM architectures, such as the LLaMA-1/2 series, our method is not applicable because these models do not contain experts.\n\n**Finally, we hope that our response has addressed your concerns, and we kindly ask for a more favorable evaluation of our work. We will incorporate the above clarification into the final version of our paper. Thank you!**"}, {"review_id": "X0drO7Rfcs", "reviewer": "Reviewer_3TXS", "summary": "The paper tackles the problem of pruning Mixture-of-Experts (MoE) language models, which often waste memory and compute on redundant experts. Instead of relying on heuristics or exhaustive search, the authors frame expert selection as a cooperative game and propose Shapley-MoE—a pruning framework that scores each expert by its Shapley value, i.e., its average marginal contribution to the model’s performance (measured as $1/\\text{perplexity}$). Each expert is treated as a “player,” and the model’s performance over any subset of experts acts as the payoff function. The Shapley value formula precisely captures an expert’s average benefit across all coalitions.Exact Shapley values require evaluating $2^N$ subsets and are intractable for hundreds of experts. The authors approximate them by sampling permutations and computing marginal gains, reducing the complexity from exponential to $\\mathcal{O}(MN)$.Theorem 1 quantifies how much accuracy is lost when Monte Carlo sampling stops early once a sampled expert subset degrades performance below a threshold. Theorem 2 analyzes the estimator that draws permutations according to router activation probabilities.", "questions": "It would be helpful if the authors could further clarify the novelty of their approach, especially in relation to prior work that employs Shapley values for pruning neurons or convolutional filters. In addition, providing some intuition or guidance on the sample complexity would make it easier to assess the method’s practicality. It would also be useful to offer theoretical or empirical justification for the assumption that activation frequency reflects an expert’s true contribution. Moreover, it remains unclear whether expert rankings based on the 1/PPL objective are indicative of broader model importance. Finally, some insight into how often early truncation occurs and the extent to which it may bias perplexity estimates would be appreciated.", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper is methodologically sound and well‑executed. The use of Shapley values is mathematically justified, and the combination of Monte Carlo estimation with early truncation and router‑guided importance sampling (RGIS) forms a coherent, scalable pipeline. The writing is clear and the empirical evaluation is thorough. However, the core idea lacks novelty. The use of Shapley values for model pruning has appeared in prior work on neurons and convolutional filters, and extending this to MoE experts is a conceptually straightforward adaptation. The main contributions—early truncation and RGIS—are useful heuristics, but they are presented as practical add‑ons rather than fundamental algorithmic innovations. The theoretical analysis also has limitations. Theorem 2 only provides a one‑sided variance bound, without quantifying the expected gains, making the practical benefit of RGIS hard to predict. Theorem 1 ignores the additional variance introduced by the importance weights in RGIS, which may lead to large hidden constants. Furthermore, there is no theoretical justification that Shapley values computed with respect to 1/PPL will transfer to downstream tasks. Finally, despite reduced cost, the method still relies on Monte Carlo estimates.", "was_revised": true, "final_justification": "Many thanks to the authors thorough response and clear clarifications.\n\nI will remain my score.", "rebuttal": "**Thanks for your careful review and comments!**\n\n> W1 & Q1: Core idea lacks novelty. The use of Shapley values for model pruning has appeared in prior work on neurons and convolutional filters, and extending this to MoE experts is a conceptually straightforward adaptation. The main contributions—early truncation and RGIS—are useful heuristics, but they are presented as practical add‑ons rather than fundamental algorithmic innovations.\n\nWe appreciate the reviewer's insightful comment regarding prior applications of Shapley value in neurons or convolutional filters pruning. The similarity between these works and ours is that both determine the importance of model components based on the Shapley value, thereby pruning the unimportant model components.\n\n**However, we respectfully argue that extending Shapley values to MoE pruning is far from a straightforward adaptation due to the unique structural and computational characteristics of MoE architectures.** Unlike dense neural networks or convolutional layers, where components (e.g., neurons or filters) are always active and contribute uniformly, MoE experts operate in a sparse, conditionally activated manner governed by gating mechanisms. This introduces interdependencies among experts that create a fundamentally different \"cooperative game\" compared to prior settings. **Directly applying existing Shapley-based pruning methods would overlook these dynamics, leading to inefficient or inaccurate importance estimation.** The results of our ablation experiments in Table 4 show that our method has higher accuracy and lower pruning cost compared with the original Shapley value pruning method based on Monte Carlo sampling.\n\n**Moreover, our main contributions—early truncation and Router-Guided Importance Sampling (RGIS)—are not merely practical \"add-ons\" but principled innovations designed to address the exponential computational challenges unique to large-scale MoE.** Early truncation leverages the observation of performance collapse in low-expert coalitions and is backed by a novel error bound theorem (Theorem 1), which theoretically justifies its use and demonstrates bounded estimation error. Similarly, RGIS exploits MoE's gating probabilities as a natural prior for importance sampling via a Plackett-Luce model, with proven unbiasedness and variance reduction (Theorem 2}). These are not generic heuristics; they are tailored to MoE's router-driven sparsity, reducing computational complexity while improving accuracy, as validated in our ablation studies and experiments, where Shapley-MoE outperforms baselines by significant margins.\n\n**In essence, while building on the foundational idea of Shapley value, our work introduces a novel framework that adapts and extends them to the distinct challenges of MoE pruning, with theoretically grounded estimate accelerations that enable scalability to MoE.** We believe this represents a meaningful advancement in efficient MoE deployment.\n\n> W2: Theoretical analysis also has limitations. Theorem 2 only provides a one‑sided variance bound, without quantifying expected gains, making practical benefit of RGIS hard to predict.\n\nThank you for your insightful feedback on Theorem 2. However, we would like to make the following clarification.\n1. **Expected gains of RGIS**: We agree that a more precise quantification of variance reduction would strengthen the analysis. The one-sided bound was designed to offer a rigorous guarantee: under the reasonable assumption of positive correlation between activation frequencies and true Shapley value, RGIS ensures variance is no worse than standard Monte Carlo sampling. However, we also acknowledge that this bound does not specify the magnitude of improvement, which depends on factors such as strength of correlation, specific models/datasets, etc., and this increases the difficulty of our theoretical analysis.\n2. **Practical benefits of RGIS**: To address above, our ablation studies in Table 4 provide empirical quantification of RGIS's benefits: RGIS improves the accuracy of our Shapley-MoE method. Removing RGIS would lead to an increase of 1.08 in perplexity. This means that RGIS can accelerate Monte Carlo sampling, as Monte Carlo sampling requires more sampling steps to achieve the same level of accuracy.\n\n> W3: Theorem 1 ignores additional variance introduced by importance weights in RGIS, which may lead to large hidden constants.\n\nWe appreciate reviewer's astute observation regarding potential impact of importance weights on variance in Theorem 1, which indeed focuses on early truncation mechanism and does not explicitly account for RGIS-induced variance, which may lead to large hidden constants. However, we would like to make the following clarification.\n\n1. **Clarify the theorem's scope**: Theorem 1 specifically bounds the estimation error for early truncation in standard Monte Carlo sampling, without incorporating RGIS. Its purpose is to justify truncation mechanism's efficiency and bounded error independently.\n2. **Theoretical mitigation of variance**: Theorem 2 demonstrates that RGIS reduces variance compared to standard Monte Carlo under premise of assuming a complete sampling process, which mitigates additional variance introduced by importance weights in RGIS as well as magnitude of hidden constants.\n3. **Empirical validation**: Table 4 provides empirical validation for early truncation and RGIS. When both early truncation and RGIS are removed, perplexity increases by 3.49. Our ablation studies further demonstrate that RGIS with truncation yields accurate Shapley estimates and better pruning results than naive Monte Carlo, suggesting combined approach is robust empirically even if hidden constants are not fully minimized.\n\n> W4: There is no theoretical justification that Shapley values computed with respect to 1/PPL will transfer to downstream tasks.\n\nWhile Shapley values in our method are computed using V=1/PPL (an intrinsic measure of language modeling proficiency), we justify its transferability via both theory and empirics:\n1. **Theoretically**, perplexity is a standard, intrinsic evaluation metric for LLMs, directly measuring a model's ability to predict held-out data and reflecting its overall language modeling quality. By using V=1/PPL, we quantify expert contributions to the model's core generative capabilities, which form the foundation for downstream performance. This choice aligns with prior work in LLM pruning and quantization, where perplexity is commonly used as proxies for compressed LLM performance.\n2. **Empirically**, we evaluate zero-shot performance on seven downstream tasks and knowledge reasoning, arithmetic, and code generation capabilities in experimental section. experiments show that the Shapley value calculated based on 1/PPL reflects the expert's contribution well, and the MoE model with a higher Shapley value has higher accuracy on downstream datasets.\n\n> W5: Despite reduced cost, the method still relies on Monte Carlo estimates.\n\nThe exact computation of Shapley values has time complexity $\\mathcal{O}(2^N)$, which is infeasible for MoE where $N$ typically exceeds $10^3$. Monte Carlo sampling approximates it, reducing complexity to $\\mathcal{O}(MN)$, which $M$ is the number of samples. Thus, Monte Carlo estimation is essential to significantly lower computational costs. In addition, in our response to Weakness 1 from Reviewer ovkS, we also demonstrated that error between Shapley value estimated using Monte Carlo sampling and ground-truth Shapley value is very small.\n\n> Q2: Providing some intuition or guidance on sample complexity would make it easier to assess the method’s practicality.\n\nThe Monte Carlo-based Shapley value estimation has time complexity $\\mathcal{O}(MN)$, where $M$ is the sample number and $N$ is the total number of experts in MoE. Figure 2 shows pruning costs under different $M$ settings; increasing $M$ raises costs. Thus, balancing search efficiency and model performance, we set $M=20$. Our method's end-to-end time is only 36 minutes, indicating good practicality.\n\n> Q3: It would also be useful to offer theoretical or empirical justification for assumption that activation frequency reflects an expert’s true contribution.\n\nThe relationship between activation frequency and expert contribution is justified as follows:\n1. **Theoretically**, the gating network in MoE routes inputs to the most appropriate experts, prioritizing those that contribute most to performance. Frequently activated experts are those the model deems essential for handling inputs. Therefore, the above theoretical evidence supports the assumption that activation frequency reflects an expert’s true contribution.\n2. **Empirically**, we present extensive results of pruning experts in MoE models based on activation frequency in Section 4. Frequency-based pruning outperforms random-based pruning, which empirically validates that activation frequency can indeed reflect an expert’s true contribution.\n\n> Q4: It remains unclear whether expert rankings based on 1/PPL objective are indicative of broader model importance.\n\nExtensive experiments demonstrate that expert rankings based on 1/PPL and Shapley value effectively reflect importance in broader MoE models. This includes results of our method on popular open-source MoEs (Qwen, DeepSeek and Mixtral series) in Sections 4.2 and 4.3, and multimodal MoE in Section I.1.\n\n> Q5: Some insight into how often early truncation occurs and the extent to which it may bias perplexity estimates would be appreciated.\n\nEarly truncation is applied during each Monte Carlo sampling when catastrophic performance collapse occurs. Table 4 presents ablation results: removing it increases pruning cost (time rises from 36 to 62 minutes) and degrades pruned MoE performance (perplexity increases by 1.67).\n\n**We hope that our response has addressed your concerns, and we kindly ask for a more favorable evaluation of our work.**"}, {"review_id": "V1dNJt050e", "reviewer": "Reviewer_vwFD", "summary": "This paper proposes a metric (namely, Shapley value) for measuring the relative importance of the experts in Mixture-of-Experts (MoE) models, to prune unimportant experts from the model. Due to the large computational complexity of the exact evaluation of the proposed metric, the paper introduces a Monte Carlo-based sampling strategy to approximate the metric for each expert. Theoretical support has been provided for the proposed sampling strategy, along with empirical evidence.", "questions": "1. I am confused about the results in Theorem 1. With τ→0 (i.e., no early truncation), the first term on the right side of equation (6) diminishes, i.e., we have a tighter bound. Then, how does the early truncation help, as we are having a looser bound for the case?\n\n2. In equation (7), over which distribution is $\\mathbb{E}[G(x)]$ calculated?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe proposed method outperforms other baselines evaluated in this paper\n2.\tIn terms of model and task diversity, the empirical results are adequate\n3.\tThe paper is well-written and easy to follow\n\nWeaknesses:\n\n1.\tThe designed metric (Shapley value) does not seem completely novel to me. Although the metric considers all possible combinations of unpruned experts when evaluating the importance of a given expert, it still measures importance based on the change in model performance when that expert is pruned versus retained, using a calibration dataset—similar to the approach in [1]. The consideration of all possible combinations of unpruned experts while evaluating the metric is to capture the interdependency of the experts. However, no empirical evidence is provided to justify that there exists a sufficient level of interdependency. Moreover, it is not clear whether the interdependency is inter-layer or intra-layer.\n\n2.\tThe theoretical results are only to support the sampling strategy to approximate the metric. However, no theoretical support is provided to show that the metric is itself effective.\n\n3.\tThe paper compared the proposed method’s performance against some baselines. Similar to the proposed method, all those baselines are calibration data dependent, either evaluating the sensitivity of the individual experts over the calibration dataset or using the routing statistics (e.g., Gating score, Frequency) on the dataset. Comparison with data-free approach as in [2] would better evaluate the effectiveness of the proposed method.\n\n[1] Lu, Xudong, et al. \"Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models.\" Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.\n\n[2] Chowdhury, Mohammed Nowaz Rabbani, et al. \"A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts.\" International Conference on Machine Learning. PMLR, 2024.", "was_revised": true, "final_justification": "The authors clarified a major part of the weaknesses. Therefore, I updated the rating.", "rebuttal": "**Thanks for your careful review and comments!**\n\n> W1: The designed metric (Shapley value) does not seem completely novel to me. Although the metric considers all possible combinations of unpruned experts when evaluating the importance of a given expert, it still measures importance based on the change in model performance when that expert is pruned versus retained, using a calibration dataset—similar to the approach in [1]. The consideration of all possible combinations of unpruned experts while evaluating the metric is to capture the interdependency of the experts. However, no empirical evidence is provided to justify that there exists a sufficient level of interdependency. Moreover, it is not clear whether the interdependency is inter-layer or intra-layer.\n\nThank you for your thoughtful feedback. We appreciate the opportunity to clarify and strengthen our contributions.\n1. **Novelty of Shapley value metric and comparison with [1]**:\n    - Although both NAEE and our method measures importance based on the change in model performance when that expert is pruned versus retained, NAEE [1] enumerates all possible combinations of retained experts and selects the combination that minimizes the Frobenius norm of the output difference before and after pruning. **For fine-grained large-scale MoE with hundreds of experts, NAEE [1] method are impractical in practice, as they require tens of thousands or even trillions of enumerations, which is obviously infeasible.**\n    - **On the contrary, our method does not require enumerating all possible expert combinations and can obtain a high-performance pruned MoE in just tens of minutes.** By quantifying the contribution of each expert through Shapley value, thereby effectively identifying those experts that are highly relevant to task performance. In addition, we use Monte Carlo sampling to approximate Shapley value. **Crucially, we have improved the accuracy and efficiency of this approximation through novel early truncation and Router-Guided Importance Sampling (RGIS) techniques. These constitute the core algorithmic innovations we propose.** Therefore, our method has significant novelty compared to previous approaches.\n2. **Empirical evidence for expert interdependencies:** We acknowledge that direct evidence for the level of interdependency was not explicitly quantified in our paper. We believe that the experts in MoE naturally have interdependent relationships, which include both intra-layer and inter-layer relationships. Intra-layer interdependency occur when experts within a layer compete/cooperate via top-k routing, while inter-layer ones arise from sequential processing (e.g., an expert in layer $l$ relying on outputs influenced by experts in layer $l-1$). **Therefore, our formulation treats the expert set $\\mathcal{E}$ globally (across all layers), naturally capturing inter-layer and intra-layer interdependency.** For empirical evidence, please see our superior performance over baselines that do not model interdependencies (e.g., Frequency and Gating Score in Tables 1, 2 and 3). This gap indirectly evidences interdependencies, as these metrics underperform by pruning \"important\" experts in isolation without considering model interdependencies.\n\n> W2: The theoretical results are only to support the sampling strategy to approximate the metric. However, no theoretical support is provided to show that the metric is itself effective.\n\nThe effectiveness of Shapley value itself as a metric rests on its axiomatic guarantees from cooperative game theory, which we have shown in Section C. These properties ensure it is uniquely suited for expert contribution assessment in MoE:\n1. **Efficiency** ensures the sum of Shapley values equals the total performance gain achievable by the expert set. Pruning low Shapley value experts thus minimizes degradation, as their absence has little impact on the overall utility.\n2. **Symmetry** ensures equitable treatment of experts with identical impacts, preventing bias in pruning decisions.\n3. **Null Player Property** identifies experts with zero marginal impact (Shapley value is 0), enabling safe pruning without performance loss.\n4. **Linearity** preserves the metric's validity under combinations of games, which is relevant for multi-layer MoE where performance can be viewed as additive across layers.\n\nThese axioms collectively establish Shapley Value as a theoretically grounded metric for fair attribution in cooperative settings and ensure its effectiveness.\n\n> W3: The paper compared the proposed method’s performance against some baselines. Similar to the proposed method, all those baselines are calibration data dependent, either evaluating the sensitivity of the individual experts over the calibration dataset or using the routing statistics (e.g., Gating score, Frequency) on the dataset. Comparison with data-free approach as in [2] would better evaluate the effectiveness of the proposed method.\n\n1. **Comparison with [2].**\n    - [2] proposes to prune experts based on the change in the $l_2$ norm of the router in a fine-tuned MoE compared to that in pre-trained MoE. For example, when pruning a vision MoE fine-tuned on CIFAR-10, the pruning metric is $l_2$ norm difference of the router between this fine-tuned model and the model pre-trained on ImageNet-21k. It should be noted that although this pruning process does not require a dataset, the process of obtaining the fine-tuned model using a fine-tuning dataset does require a dataset. **Therefore, strictly speaking, the method proposed in [2] is not a data-free method. In addition, the method proposed in [2] is not applicable to most scenarios of pruning Mixture-of-Experts LLMs, because there is no corresponding pre-trained model for pruning in such cases. In contrast, our method can directly prune any MoE without comparing the $l_2$ norm difference with the pre-trained model.**\n    - The method proposed in [2] cannot be used to prune MoE model mentioned in our paper, so it cannot be directly compared with our method. We migrated our method to prune the V-MoE model mentioned in [2]. We used the training set of CIFAR-10 as the calibration dataset, evaluated CIFAR-10 test set accuracy, and pruned 50% experts without fine-tuning after pruning. The results are as follows:\n|Method|Accuracy|\n|-|-|\n|Method in [2]|98.13%|\n|**Ours**|**98.32%**|\n\n        **Our method outperforms the method proposed in [2], which once again demonstrates the effectiveness of our method.** We will include the above comparison results with the data-free approach [2] in the final version of our paper.\n\nFurthermore, we would like to make the following clarifications regarding the calibration dataset.\n1. **The use of a calibration dataset is a common practice in expert pruning for MoE.** Our method does require the use of a calibration dataset, but the use of a calibration dataset is a common practice in expert pruning for MoE. All the expert pruning methods compared in our paper, including Gating Score, Frequency, NAEE, Expert Trim, and CD-MoE, adopt calibration datasets.\n\n2. **The use of calibration datasets is also a common practice in compressing LLMs.** Many previous SOTA methods for LLMs pruning (e.g., SparseGPT [A1], and LLM-Pruner [A2]) and quantization (e.g., AWQ [A3] and SmoothQuant [A4]) have employed calibration datasets. By using calibration dataset and running the forward propagation of the model, we can measure the weights that have a significant impact on the final output of the model, thereby reducing compression error.\n\n[A1] SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. ICML 2023.\n\n[A2] LLM-Pruner: On the Structural Pruning of Large Language Models. NeurIPS 2023.\n\n[A3] AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. MLSys 2024.\n\n[A4] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML 2023.\n\n> Q1: With $\\tau→0$ (i.e., no early truncation), the first term on the right side of equation (6) diminishes, i.e., we have a tighter bound. Then, how does the early truncation help, as we are having a looser bound for the case?\n\n1. It is correct that the first term in the error bound on the right side of equation 6 approaches zero as $\\tau→0$ (approaching no truncation), resulting in a tighter theoretical bound that reduces to the standard Monte Carlo sampling error term. However, the primary purpose of Theorem 1 is not to claim that early truncation tightens the bound compared to no truncation, but rather to provide a rigorous guarantee that the truncated estimator remains accurate (with bounded error) despite skipping certain low-information sampling steps.\n2. The key benefit of early truncation is computational efficiency, which indirectly leads to improved estimation accuracy in practice. Without truncation, each Monte Carlo sample requires evaluating the full permutation of $N$ experts, leading to $\\mathcal{O}(MN)$ complexity that is often prohibitive. Truncation allows us to early-exit permutations where performance has collapsed, significantly reducing the average number of evaluations per sample. This enables us to afford a larger number of samples $M$ within the same computational budget, which in turn reduces the second term in the bound (the sampling error). As shown in our ablation studies (Section 4.6), this technique reduces the overall pruning cost by nearly half while improving accuracy.\n\n> Q2: In equation (7), over which distribution is $\\mathbb{E}(G(\\boldsymbol{x}))$ calculated?\n\nThank you for your careful reading and this was an oversight on our part. $\\mathbb{E}(G(\\boldsymbol{x}))$ is not calculated over any distribution and the expectation symbol $\\mathbb{E}$ is redundant, and we will remove $\\mathbb{E}$ in the final version of the paper.\n\n**We hope that our response has addressed your concerns, and we kindly ask for a more favorable evaluation of our work.**"}]

Author rebuttal:

Meta-review: This paper proposes Shapley-MoE, an efficient pruning method for MoE models inspired by cooperative game theory. The authors estimate each expert’s marginal contribution to model performance via Monte Carlo approximation, and incorporate two key optimizations, Early Truncation and Router-Guided Importance Sampling, to reduce the theoretical exponential complexity to an approximate quadratic level while preserving accuracy. Experiments on Qwen, DeepSeek, and Mixtral demonstrate its effectiveness.

Stengths:
1. novel methods with clear motivations
2. comprehensive evaluation results across different model families and datasets

Weaknesses:
1. Some reviewers pointed out that some empirical/theoretic analyses should be carried out for better understanding the internal working mechanism of the proposed method.
Decision: Accept (poster)