Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
Authors: zican Dong
Han Peng
Peiyu Liu
Xin Zhao
Dong Wu
Feng Xiao
Zhifeng Wang
Venue: NeurIPS 2025 poster ()
Keywords: Mixture-of-Experts
Expert Pruning
Primary area: deep_learning
TL;DR: We investigate the phenomenon of few-shot expert identification in large Mixture-of-Experts models and propose EASY-EP, a simple yet effective method for expert pruning.

Abstract: Mixture-of-Experts (MoE) models achieve a favorable trade-off between performance and inference efficiency by activating only a subset of experts. However, the memory overhead of storing all experts remains a major limitation, especially in large-scale MoE models such as DeepSeek-R1 (671B). 
In this study, we investigate domain specialization and expert redundancy in large-scale MoE models and uncover a consistent behavior we term~\emph{few-shot expert localization}, with only a few in-domain demonstrations, the model consistently activates a sparse and stable subset of experts on tasks within the same domain.
Building on this observation, we propose a simple yet effective pruning framework, \textbf{EASY-EP}, that leverages a few domain-specific demonstrations to identify and retain only the most relevant experts.
EASY-EP comprises two key components: \textbf{output-aware expert importance assessment} and \textbf{expert-level token contribution estimation}. The former evaluates the importance of each expert for the current token by considering the gating scores and L2 norm of the outputs of activated experts, while the latter assesses the contribution of tokens based on representation similarities before and after routed experts.
Experiments on DeepSeek-R1 and DeepSeek-V3-0324 show that our method can achieve comparable performances and $2.99\times$ throughput under the same memory budget as the full model, with only half the experts. Our code is available at https://github.com/RUCAIBox/EASYEP.

Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
Zican Dong" Han Peng .2, Peiyu Liu Wayne Xin Zhao :21
Dong Wu Feng Xiao Zhifeng Wang' Gaoling School of Artificial Intelligence, Renmin University of China
Beijing Key Laboratory of Research on Large Models and Intelligent Governance
University of International Business and Economics
4 YanTrro Technology Co. Ltd EBTech Co. Ltd {dongzican, panospeng) het.://uuuu.euuccrm Liupeiyustu@163. com, batmanfly@@mail.comm com, wudong@yantronic.. com, {fengx, 2hifengx//@btecch.com
Abstract
Mixture-of-Expert (MoE) models achieve favorable trade-off between perfor- mance and inference efficiency by activating only a subset of experts. However, the memory overhead of storing all experts remains a major limitation, especially in large-scale MoE models such as DeepSeek-RI (671B). this study, we investi- gate domain specialization and expert redundancy in large-scale MoE models and uncover a consistent behavior we term few-shot expert localization, with only a few in-domain demonstrations, the model consistently activates sparse and stable subset experts on tasks within the same domain. Building on this observation, we propose simple yet effective pruning framework, EASY-EP, that leverages a few domain-specific demonstrations to identify and retain only the most rele- vant experts. EASY-EP comprises two key components: output-aware expert importance assessment and expert-level token contribution estimation. The former evaluates the importance of each expert for the current token by considering the gating scores and L2 norm of the outputs of activated experts, while the latter assesses the contribution of tokens based on representation similarities before and after routed experts. Experiments on DeepSeek-RI RI and DeepSeck-V3- V3-0324 show that our method can achieve comparable performances and 2.99 throughput under the same memory budget as the full model, with only half the experts. Our code is available http:///gthab.om///UCUA////EEEE
Introduction
Mixture-of-Expee -f-eeeers-s-xe (MoE) architectures have been widely adopted as the backbones of various large language models (LLMs) due to their efficiency of scaling parameters without proportional computational overhead [ However, the deployment of large MoE models imposes substantial
memory requirements. Taking DeepSeek-RI (671B) I as an example, it takes about 1500 GB under BF16 precision and 750 GB under FP8 precision, necessitating 4x8 A800 or 2x8 H800 GPU configurations, respectively. This underscores the critical need to explore lite deployment strategies for large-scale MoE models like DeepSeek-R1.
Various training-free approaches have been proposed to alleviate the inference memory demands of MoE models. Expert pruning reduces memory by removing less important experts. Among them,
Equal Contribution.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Throughput vs Performance with Different Experts
1750
80
1500
Performance on AIME2024
Thronghput (token/s)
1250
1*8*0800
2*80
70
1000
Throughpul
60
Performance
750
50
250
40
0 32
128 Number of Experts
192
224
Figure 1: Throughput and performance comparison of DeepSeek-RI AIME2024 with varying expert numbers using EASY-EP. We deploy DeepSeek-RL with two 8x H800 for 224 and 256 experts, while 8x H800 for others. The throughputs of the latter configurations are multiplied by 2.
5 while perturbation- based ones select expert subsets to minimize hidden state drift [6] 7 However, the former may fail identify key experts, and the latter expensive as the number experts increases. Expert merging combines similar experts to reduce their number 8 but can cause neuron misalignment of experts MoE models trained from scratch [T] [1] Moreover, existing methods are primarily designed for MoEs with few experts layer (e.g. Mixtral 8 7B 2 66. This highlights the need for more scalable and accurate pruning strategies tailored to large-scale MoE models. While such scale poses challenges for current approaches, it also brings new opportunities: the increased granularity and domain specialization of experts in models like DeepSeck-R [12 make them especially amenable to domain-specinf pruning high compression ratios [14
router-based methods use expert activation statistics to estimate importance
In this study, we analyze how expert activations large MoE model vary across domains and respond to small number of demonstration samples. We observe consistent behavior: given just a few Nomain-speciine
examples, model tends to activate stable and sparse subset of experts that are highly relevant to the target domain. We refer to this phenomenon few-shol expert localization. To better understand mechanism behind this behavior, we focus on two factors: the domain specialization of experts and the sufficiency of limited demonstrations. Our key findings are as follows: (1) High gating-valls value experts are strongly gemain-specific, consistently dominating activations within their respective domain while remaining inactive in unrelated ones. (2) A small number demonstrations suffices to reliably trigger these experts, and they generalize well to other unseen datasets within the same domain.
Based our observations, we introduce domain domain-specific pruning framework that leverages few-shot demonstrations to address memory constraints in large-scale MoE models. Our approach begins by sampling a small number of task demonstrations from specific domain and generating responses with the original model, serving as calibration set. To identify and retain the most critical experts, we then develop a pruning method, Expert Assessment with Simple Yet-effective scoring for Expert Pruning, a.k.a., EASY-EP. This method estimates expert importance and retain fixed-size subset of top-scoring experts. EASY-EP consists two complementary components: (1) output-aware expert importance assessment, which combines gating values and the L2 norm of expert outputs to estimate per-token expert importance, and (2) expert-level token contribution estimation, which measures the similarity between the input and the residual-connecte token's ' contribution to the overall expert score. Notably, the entire scoring process requires only a single forward pass, eliminating the need for backpropagation repeated evaluations.
output of routed experts measure each
To evaluate the efficacy of our approach, we conducted systematic experiments on DeepSeek-BB and DeepSeek- S33SScck-V3-3322 V3-0324 using eight benchmark datasets covering math, coding, science, finance,

---

medicine, and agent execution capabilities, Specifically, with only retaining 50% experts, our method can keep comparable performances under the domain-specifie pruming settings while achieving over 90% of the full model's performances on DeepSeek-RI and even better performances V3-0324 under the mixed-domain pruning settings. shown in Figure [ the performances degrade only slowly with the increase of compression ratio, indicating robustness to pruning. Additionally, under identical memory constraints, pruning 50% of the experts yields a x.99 x increase in inference throughput for sequences of IK input and IK output lengths, highlighting the practical utility of our framework real-world deployments.
DeepSeek-
2 Background
MoE architectures introduce MoE modules where parameters are dynamically activated to replace feedforward networks (FFNs) [. Specially, in the (-th layer, MoE module contains a router G(.) and N routed experts (hi..... 1 }, 4h4, E PR, the router computes the logit of each expert for the t-th token and applies a gating function on the Top-K logits to obtain the gating values g¹. (the gating values of deactivated experts Ii,t 0). The Top-K experts are activated, and their outputs are aggregated via weighted summation. The final output is obtained by residual connections of the input and output of experts:
E.,)....,,,( R Given input representation sequence H'
N Σ Sh (h(,),, hi = h² hi
With the gating values of the router, we define two metries assess the importance of each expert, i.e., frequency and gating scores [5]. For all tokens in a calibration set and an expert E,,, we define the frequency Jj the number of times each expert is activated, while the gating scores r! is defined
as the total sum of the gating values when each expert is activated, as shown in Equation 2 Here, M is the size of the calibration set, and Tn denotes the number of tokens per demonstration.
S 0),
I
(2)
3 Empirical Analysis of Experts
Previous work has demonstrated that the expert distributions of MoEs with few experts (e.g.. Mixtral 8 x 7B) mainly depend on the syntax structures instead of domains (2). However, recent large-size MoE models are equipped with various fine- -en-ggai experts (e.g.. DeepSeek-Re R1 has 256 experts per layer), which may be more specialized and store distinct knowledge and capacities in their parameters. Motivated by this, we empirically study a phenomenon we term few-shot expert localization, where domain-specific experts can be reliably identified using only handful of demonstrations.
Ovella I5 15 Expeits
(B) Overlep of Top- Exper'
(C) Osodaa Top 125 Experts
(Di Ovedls eCTop 128 Experts
19
EM00AS ADUEDS ADAEUE AIME20
1:00
0.31
3.40
Stah
1.00
1x
1.00
0.76
6.75
0.75
1.80
Dheak
0.76
1,02
8:50
0.53
8:23
1.00
9.28
0.70
25-
Codke
Codee
1.00
Mar
Sciaice
Codiag
Sdinnce
Codia
DDussin
Dusuia
Figure 2: (A), (B): Overlap ratios of experts with top gating scores on different datasets. (C): Overlap ratio of top-128 128 experts with different numbers of demonstrations. (D): Overlap ratio of top-128 experts pruned with different math datasets.
SShared experts employed in some MoE models [ 16) but are considered in this work.
3

---

Table 1: Results of removing domain experts. Bold denotes in in-domain results.
Domain
AIME24
GPQA
LiveCodeBench
Full
77.08
70.91
63.32
Math
67.33 (-9.75)
69.19 (-1.72)
65.27 (+1.95)
Code
78.67 (+1.59)
71.72 (+0.81)
55.68 (-6.64)
Science
79.33 (+2.25)
59.09 (-11.82)
61.07 (-2.25)
3.1 Expert Specialization Across Domains
To assess whether experts in large MoE models exhibit domain-specifie specialization, we select AIME-2023 [7]. GPQA-main [18], and LiveCodeBench-Ve V3 [19] as calibration datasets for the domains of math, science, and coding, respectively. We conduct experiments using the representative model DeepSeek-R R1 on these datasets and extract the gating scores across different domains (as described in Section 2 By analyzing the expert activation distributions across these domains, aim to reveal how expert utilization varies and assess the degree of specialization.
Distinct Expert Distribution Across Domains. We first rank all experts each layer by their gating scores and select the top-16 and top-1 experts for each dataset. Subsequently, we measure the overlap of top-ranked experts across different domains and visualize the overlap ratios in Figure 2 where (A) corresponds to top-16 and (B) to top-128. We observe that the top- 16 experts are largely disjoint across datasets. When expanding the selection top-128, the degree of overlap increases, but a significant portion of the experts remains domain-specifer This indicates that large MoE models contain domain-speen specciai--seiiiieeddeeee experts that are predominantly activated their respective domains. We show experiments on different numbers of top experts and layer variations in Appendix A
Impact of Removing Dpmein-Specific Experts. order to explore the importance of domain- specific experts, we remove those that appear in the top-128 (by gating score) in specific domain but not in any others. We then evaluate each pruned model on tasks from the same domain as the calibration data (i.e., in-domain), and on tasks from other domains (i.e., out-of--omain), to assess the generalization behavior the remaining experts. As shown Table [ pruning these experts leads to significant performance degradation out-of-domain tasks. These results suggest that domain-spectte experts play a critical role in the relevant domain but are redundant for other domains.
in in-domain tasks while having minimal impact on
3.2 Expert Locality Within One Domain
Beyond examining the expert specialization across different domains, we examine the locality and stability of expert activation within single domain. We investigate how the number of demonstrations and the choice of calibration set influence expert selection patterns under the same domain setting.
Effect of Calibration Set Size. Given the presence of domain-specinf experts in DeepSeek-Rt a fundamental question arises: How many demonstrations are necessary to accurately identify these key experts? To answer this, sample varying numbers of demonstrations (i.e., 5, 25, 100) from LiveCodeBench--33 [9] For each setting, we compute the average gating scores and retain the top 128 experts. Based on these selections, then calculate the pairwise overlap ratios between expert sets derived from different demonstration sizes. As illustrated Figure 2 (C), even with just five demonstrations, over 90% of the critical experts can be effectively identified. Furthermore, 25 demonstrations are sufficient to capture all domain-specific experts (99%), with additional demonstrations yielding only marginal improvements. These results underscore the feasibility of few-shot domain-specife expert pruming.
Consistency of Expert Activation Across Datasets. To investigate the consistency of domain- specific experts across datasets within the same domain, we conduct experiments with DeepSeek- RI on four math datasets: AIME-2023, AIME-2024, AIME-2025, and HMMT-Feb 2025 [17]. Specifically, we perform expert pruning under the 25-shot setting for each dataset and retain Top- experts based on their average gating scores. We then compute the pairwise overlap ratios between each pair of datasets. As shown in Figure 2 (D), the overlaps exceed 84% across all math datasets,

---

revealing high degree of consistency in domain domain-specific expert activation. This indicates that domain-specific expert activation patterns are largely transferable within the same domain. Despite some sppeset- differences, the overall expert overlap remains strong and stable.
Empirical analysis with another metric is shown in Appendix B
Pas
Output Hidden
Euampic
Reepeme 1
Selost Tup 120 Expora
Exampl:
Respenee Requereed
Maddle Hidden
Doanple
Narnryy: 330G2
344B
Middle
- dis (illll
Impertiner
Ontgut Aware
E
Hel.ll - -
Sanchi.))
Router
HiputHilienk
Eisz
Eizz
Easa
Pxpertizntt Tolien Tosteibuauttttua
Input Uidden at
Router
Math
Coding
Science
Five man and nine wamen stand equally cirde random othinkoinnnaay problem <tthinks the derommGGrmpeettaaaaii
hours <thinks Oleay, Given problem sunder
What the expected frequenncy mergers... <thinks expected... elthink... W0oeed(10. (Hb) Woret to (roe(
pairs (t)..-eyhrnnee... "pehhel,ncasss Solutioer)
countt""
Figure 3: Overall framework of EASY-EP. Given calibration set consisting of input and responses by the model, EASY-EP leverages output-aware expert importance assessment and expert-level token contribution estimation compute the expert score the domain and returns the pruned expert sets.
4 Method
4.1
Overview
Motivated by our earlier observation of few-shot expert localization phenomena in Section 3 propose an expert pruning framework to reduce memory costs. use a small set of target-domain demonstrations to run the MoE model and collect expert activation statistics, considering both inputs and outputs for pruning. To effectively identify -ppeiii experts large MoE models, we introduce simple yet effective expert pruning method, EASY-EP. Specifically, product the expert output L2 norm and its corresponding gating value as the output-aware expert importance d,,- Next, we determine the expert-level token contribution s based the similarity of representations before and after expert computation. The final expert score 7(E)) is obtained by aggregating the product of two terms all tokens:
first compute the
T
I (E
Our method also supports mixed-domain pruning by averaging the normalized expert scores of all target domains. Thus, we can prune a single model to handle tasks across multiple domains:
mix
(E
(U=(E)))
/ (F())) 1=1
TET
Based the expert scores computed from small subset of data, efficiently select the Top-M experts with the highest scores as retained experts while pruning other experts to reduce memory costs. The overall framework of our method is illustrated in Figure
4.2 Oulput-Aware Expert Importance Assessment
To assess the importance of each expert, prior router-hased expert pruning methods assume activated expert gating scores reflect their importance (5). However, this assumption has not considered
5

---

the influence of experts. To further assess the contribution of each routed expert, we example the aggregated output h2 from all routed experts for given token:
h1 Σ
c Σ H
(5)
|
where ||-|| denotes the L2 norm of vector, e/,t (h') represents the output of the expert, and denotes the unit vector in the direction of the expert's output. Further, we can compute the upper bound of the L2 norm of the expert outputs as follows:
e/e || || E
|
Σ ghallee,.
(6)
This indicates that each expert's contribution to the final output is bounded by the product of its gating value and the 1.2 norm of its output, sf,rlle,,rll. An expert with large gating score may still produce outputs with low L.2 norm, ultimately resulting in a limited influence on the final output, which are empirically verified in Appendix C.I] Therefore, instead of using only the gating value, we define the importance of an expert for given token as the product its gating value and L2 norm of output, formalized in the following equation:
= Vgl.e > 0.
4.3 Expert-Level Token Contribution Estimation
When calculating the statistical metries for evaluating the importance of experts, prior work often directly averages the scores across all tokens (6 However, in practice, the influence of routed experts' outputs on the residual stream varies significantly across tokens (as shown in Appendix C.2). Intuitively, when dealing with tokens exhibiting low similarity before and after the MoE module, adjusting their routed experts will induce substantial distributional shift in their representations. In contrast, for tokens with high similarity, such adjustments will lead only minimal drift in their representational distributions [20 Inspired by these, we propose a similarity- bhased token impor- tance assessment method which give greater weights for the former tokens. Given the representations before and after the routed expert modules hi and h,, compute the cosine similarity between these representations. The token importance score sf is then defined as one minus this similarity, capturing the extent of change induced by the routed expert module:
st 1 1--Sin -Sim (h/ ht).
(8)
Experiments
5.1 Experimental Settings
Evaluation Benchmarks. To systematically assess the effectiveness of our proposed method, we
conduct experiments across eight benchmark datasets: AIME-2024 AIME-2025, HMMT-Feb 2025, LiveCodeBench [19, GPQA-Diamond [18], USMLE (22), FinancelQ 23 and AgentBench-OS 24). These benchmarks encompass six fundamental domains and tasks of LLMs: math, coding, science, medicine, finance, and agent-based task execution.
Experiment Settings. We select DeepSeek-RI R1 I and DeepSeek- V3-0324 15 our evaluated models and consider domain-spectie speaiffc and mixed-domanl pruning settings. For each domain, we randomly sample 25 instances and construct a calibration set by concatenating their inputs with the target model's 's outputs. We then evaluate the expert scores on the calibration data and select the top 64 and 128 experts with the highest scores each layer, respectively. We also average the normalized expert scores on different domains to evaluate the mixed-domain pruning performances. Details regarding the candidate sets and evaluation settings are provided in Appendix D

---

Table 2: Comparison of the performances of different expert pruning methods. HMMT denotes HMMT-Feb 2025, GPQA denotes GPQA-Diamond, A-OS denotes AgentBench-On OS, and FinIQ denotes FinancelQ.
Model
Method
Mix
#B
AIME-24
AIME-25
FMMT
LiveCode
GPQA
USMLE
FinlQ
A-OS
Avg
Full
256
77.08
66.67
44.38
63.32
70.91
92.66
82.1
40.51
67.20
Random
X
64
0.00
0.00
0.90
0.00
26.09
0.00
0.00
0.00
3.26
Frequency
17.68
Galing Score
X
64
2.67
1.35
2.67
14.97
46.83
0.86
0.00
0.69
8.75
M-SMoE
64
0.00
0.00
12.12
0.00
0.00
EASY-EP
X
64
72.81
55.10
38.02
42.51
67.47
26.63
33,90
27.26
45.22
DeepSeek
-RI
Frequency
Random
x
128
19.33
8.33
13.33
7.35
3.33
36,08
59.60
34.95
61.51
26,40
29.
31.59
Gating Score
x
63.78
66.50
31.94
M. SMoE
X
128
5.33
6.00
3.33
25.75
24.75
52.63
39.60
19:44
22.
79.17
70.12
78.80
Prequency
128
21.33
10.00
6.00
7.49
41.45
78.55
62.14
11.81
29.85
Gating Soore
128
29.33
21.33
18.00
22.75
41.69
62.06
27.29
30.56
31.67
M-SMoE
128
6.67
2,00
4.67
4.19
32.32
72:00
19.10
6.25
18.40
EASY-EP
128
75.94
61.98
42.50
57.63
70.36
91.20
57.95
34.17
61.47
Full
256
55.73
47.71
28.75
48.50
66.87
87.51
64.22
33.33
54.08
Random
x
64
0.00
0.00
0.00
0.00
26.87
0.39
0.00
0.69
3.49
Frequency
64
31.35
15.73
1.95
45.25
61.96
22.74
31.65
Galiig Score
X
64
43.96
25.10
23.12
14.97
51.52
78.68
64.20
0.00
37.69
M-SMoE
64
16.67
13.33
3.33
1,20
22.22
47,00
21.52
EASY-EP
X
64
53.12
41.56
28.85
27.99
57.35
84.57
72,50
27.55
49.19
DeepSeek -V3-0324
Random
x
1.33
Frequency
X
128
55,73
42,60
30.10
36.08
63.54
84.29
66.84
31.71
$1.36
Gating Score
x
M. SMoE
X
128
48.00
38.67
28.67
30,53
$5.82
86.72
66.60
33.33
48.54
EASY-EP
31.56
Frequency
128
51.35
37.60
24.27
17.07
55.90
83.47
66.80
36.25
46.59
Gating Soore
128
53.75
27.19
28.74
58.88
67.74
M. SMoF
128
43.33
30,00
20.00
7.19
52.53
82.13
62,20
29.17
40.84
EASY-EP
128
57.81
33.33
40.72
64.95
72.26
38.74
$4.92
Baselines. In our experiments, we employ three expert pruning methods and one expert merging method for comparison. For expert pruning, we employ different methods to assess the expert scores, frndue.requeecy. including random. frequency, and gating scores (as diseussed in Equation [ and only keep Top- M
experts with the highest expert scores 4 For expert merging, we select M-SMoF (25), which first employs neuron permutation alignment to mitigate neuron misalignment of experts and then merges experts into dominant ones with similarity of router logits.
5.2
Main Results
Table showcases our method's performance against baselines under various pruning configurations. First, our approach consistently outperforms all baseline methods across diverse benchmarks and pruning settings. Notably, in domain-specific pruning, our method matches and even surpasses full model performance on certain benchmarks with only half the experts. This may be attributed to the effective removal of irrelevant experts, enhancing the model's ability to utilize domain-specifte knowledge. Furthermore, our method demonstrates strong resilience high compression ratios (e.g., 75%), where most methods experience significant performance degradation, particularly on DeepSeek-RI. In contrast, our technique preserves substantial model capabilities, highlighting its effectiveness in identifying critical experts for specific tasks.
Second, non-reasoning models exhibit greater robustness compared to their reasoning-oriented oriented counterparts. DeepSSeek-V3-0322 for instance, retained more performance across most pruning methods than DeepSeek-R1. Under domain-specific pruning, De3322ek-V3-0324's performance on datasets like AgentBench-OS even improved notably after pruning some experts. However, this phenomenon is not observed with DeepSeek-R1. We hypothesize that while domain capabilities might be preserved in pruned reasoning models, their long-term generation abilities are compromised.
WWe do not include perturbation based pruning methods in our comparison 6 7 which is computationally prohibitive for MoE models with 256 experts (the detailed analysis is shown in Appendix B
7

---

Finally, our method also excels in preserving performance under mixed-domain pruning. It retains over 90% of the original performance, surpassing -emain-specific compression with other methods. Conversely, other expert pruning techniques struggle to maintain balanced performance across different domains. This underscores that the overlapped experts identified by our approach, which are linked to general reasoning abilities, effectively contribute to a broad array of downstream domains.
5.3 Detailed Analysis
In this section, we conduct further ablation studies and detailed analyses investigate the effective- ness of our approach and the few-shot expert localization phenomena of large MoE models. For more analysis experiments, we present them in Appendix
5.3.1 Ablation Study
We conduct ablation studies to analyze the impact each component in our method. Specifically, we evaluate two variants with 64 remaining experts: (1) removing the token-level contribution estimation and (2) replacing the product of gating values and L2 norm of expert outputs with only the gating scores. As shown in Table 3 both incorporating token-level contribution estimation and considering the L2 norm of expert outputs lead to improved performance compared to using only the gating score. Furthermore, combining both components results the best overall performance. These findings highlight the importance of both components in our method.
Table 3: Results of ablation study. norm denotes whether considering L2 norm of expert output and Token denotes whether considering token contribution scores.
Method
Metric
Experts
AIME-24
AIME-25
HMMT
LiveCode
GPQA
A-OS
Ours
stl.llel, 1
72.81
55.33
36.00
42.51
67.47
27.26
w/o Token
dl,llet,all 9i of
64
65.33
49.33
31.33
27.54
56.57
21.53
w/o norm
70.00
40.00
23.33
19.76
61.11
18.75
w/o both
64
2.67
1.33
2.67
0.00
20.20
0.69
5.3.2 Generalization Capacity
Beyond same-domain task evaluations, assessed the generalization capacities on unrelated do- mains after domain-specific pruning (e.g., pruning DeepSeek-RI with AIME2023 and evaluating on LiveCodeBench). As Table 4 shows, the model demonstrates certain generalization ability, especially in similar domains (e.g.. math/science, code/OS -agent, science/medicine). We hypothesize that while some comain-specifii experts are pruned, core reasoning experts preserved. However, the out-of-domain performances are typically poorer than mixed-domain pruning. Thus, we suggest using mixed-domain pruning when facing multiple downstream tasks.
Table 4: Results of generalization capacities of pruned models. Domain denotes the domain of pruning data. Bold denotes in-domain performances.
Domain
AIME24
LiveCudeBench
GPQA
Agent-OS
USMLE
FinlQ
Math 79.17
46.11
46.91
3.47
46.43
58.20
Coding
38.00
61.11
39.90
15.97
41.79
53.00
Science
64.64
53.59
70.12
4.17
75.88
57.50
5.3.3 Effect of Pruning Data
We also study the impact of pruning data. Instead of using the full model's input and output, we examine five types of data: (1) hnp: just input context data; (2) Out: only the model's generated data; (3) Inp+Ans: input context data and the correct generated answer; (4) PT: pre-training data from same domain; and (5) CC: data from CommonCrawl. Table 5 anddd the experimental results for DeekSeek-RI and DeepSeek-V3-0324 V3-0324. Compared to using the combination of input and model output, performance decreased under other settings for both math and coding tasks. This suggests that,

---

even within the same domain, there are notable differences in expert distributions among data types. Additionally, employing CommonCrawl data causes significant performance declines, demonstrating that pruning task-irrelevant model with pre-training data is not suitable for current models.
Table Comparison of performances of DeepSeek-RI with different data.
Table 6: Comparison of performances of
DeepSeek- V3-0324 with different data.
Data
AIME24
LiveCodebench
GPQA
Data
AIME24
LiveCodebench
GPQA
Inp+Out
79.17
61.11
70.12
Inp+Out
55.73
48.50
66.87
Inp+Ans
75.33
53.89
67.98
Inp+Ans
54.00
43.11
61.62
Inp
66.00
50.90
70.81
Inp
51.33
14.97
63.64
Out
77.33
59.28
70.10
Out
54.67
47.90
63.13
29.33
38.32
66.16
13.33
37.72
56.57
CC
0.00
0.00
28.92
CC
0.00
0.00
29.80
5.3.4 Effect Number of Pruning Demonstrations
In Section 3.2. we observe that domain-relevant ex- perts can be identified with only few demonstrations. Here, we further investigate the impact the number of demonstrations final performance. To do so, sample varying numbers of demonstrations from the same distribution and prune half of the experts in each layer. The performance variations on AIME24 and LiveCodeBench are presented in Figure When utilize only single sample for pruning. the selected experts are often influenced by the characteristics of that individual sample, thus resulting in lower perfor- mance. As we further increase the number of demon- strations, the performance rapidly rises, achieving comparable performances with the full model.
Purning Parformance with Diffireut Number Shots
70-
Perfomone
LiveCoda expera)
AIME 20040 experts)
2024/all
15
26
25
Figure 4: Comparison of performance with different numbers of shots for pruning.
5.3.5 Analysis of Throughput
To evaluate the throughput of pruned models with different numbers of experts, we use the SGLang (26] package and measure performance under maximum request concurrency of 32. We evaluate the settings with 1K input and IK output length. For configurations with more than 192 experts, two H800 GPUs are used. Figure [ shows scaling throughput for this setting. We observe that reducing the number of experts significantly improves throughput, particularly when the model can be deployed on single node. Compared the full DeepSeek-R1 model, configurations with 128 and 64 experts achieve 2.99x and 4.33 throughput, respectively. Compared to the full model, the pruned model can be deployed on single node, thereby avoiding inter-node communica- tion overhead. Moreover, using fewer experts further reduces communication between GPUs within the node, improving computational efficiency. We provide more experiments in Appendix F3
Related Work
MoE architectures improve computational efficiency by activating only a small subset of experts per input (27 28, 4]. However, the increased number of parameters introduced by these architectures leads to substantial memory overhead. To address this, existing efforts broadly fall into categories. The first line of work focuses architectural optimization to reduce computation parameter size. Representative methods include pyramid-shane
expert allocation [29] fine- mme-grained expert design with smaller per-expert modules [14), and the transformation of dense models into sparse MoE variants R2]. While effective, these approaches typically require modifying model architecture and retraining from scratch, which limits their applicability to already deployed pretrained models. The second line of research approaches the problem from a memory efficiency perspective by applying post-hoc compression techniques, primarily pruning and quantization 35]. These

---

methods typically estimate expert importance based on routing frequency [ gating scores [36]. or direct measurement of their contribution model outputs 6 7 Some approaches further reduce redundancy by merging similar experts g 9 though they may meet the problem of neuron misalignment and additional memory costs I0 11. contrast, our method enables efficient pruning with single forward pass and avoids storing additional model variants during compression.
7 Conclusion
In this work, we investigated the domain specialization of experts in large MoE models. Our observations indicate that domain-specifie experts play a crucial role in their respective domains and can effectively identified with few demonstrations. Building on these insights, we proposed pruning strategy that leverages demonstrations from tasks within the same domain. Specifically, we introduced EASY-EP. a simple yet effective pruning method that combines output-aware expert importance assessment with expert-level token contribution estimation. Experimental results showed that our approach maintained comparable performance while utilizing only half of the experts in domain-specific settings and retained over 90% of the original performance in mixed-domain pruning. We believe that our method can facilitate the deployment of large MoE models, particularly for efficiently handling a high volume of samples within the same domain.
8 Limitation
Our work investigates the phenomenon of few-shot expert localization in large MoE models and proposes simple yet effective method for domain-specific expert pruning. We leave the investigation of training to further enhance the pruned model's performance, particularly in balancing in-domain and out-of-domain capabilities, future work, given current focus evaluating pruning effectiveness under realistic resource constraints. We observed differing levels of robustness to expert pruning between reasoning-oriented (DeepSeek-RI) and non-reasoning models (DeepSeek-V3-0324)) While this trend is noteworthy, further validation on a broader range of architectures is needed to strengthen the generality of this observation.
Acknowledgments
This work was partially supported by National Natural Science Foundation of China under Grant No. 92470205 and 62506077. Beijing Natural Science Foundation under Grant No. L233008 and Beijing Municipal Science and Technology Project under Grant No. 7231 100010323009. Peiyu Liu and Xin Zhao are the corresponding authors.
References
[1] DeepSeek-Ar AI, Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin
Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu,
Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li. Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingehang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong. Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li. Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. CoRR, 005/2501.119493 2025.
[2] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand,
10

---

Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian. Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. CoRR, abs/2401.0405 2024.
[3] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303.18222 18223, 2023.
[4] Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang, A survey on mixture of experts. CoRR, abss24407.0620k 2024.
[5] Alexandre Muzio, Alex Sun, and Churan He. Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts. CoRR, abs/2404.05089, 2024.
[6] Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts equal: Efficient expert pruning and skipping for mixture-of- experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 6159-6172. Association for Computational Linguistics, 2024.
[7] Mingyu Cao, Gen Li, Jie Ji, Jiaqi Zhang, Xiaolong Ma, Shiwei Liu, and Lu Yin. Condense, don't just prune: Enhancing efficiency and performance in moe layer pruning. CoRR, abs/2412.00066) 2024.
[8] Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7, 2024. OpenReview.nnt, 2024.
[9] I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, and Chun- Yi Lee. Retraining- free merging of sparse mixture-of-experts via hierarchical clustering. CoRR, abs/2410.08589, 2024.
[10) Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25- 2022. OvenReview.net, 2022.
[11] Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 2022, 2022.
[12] Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck, Man Luo, Sungduk Yu, Chendi Xue, and Vasudev Lal. Semantic specialization moe appears with scale: A study of deepseek expert specialization. CoRR, abs/2502. 10928, 2025.
[13] Robert Dahlke, Henrik Klagges, Dan Zecha, Benjamin Merkel, Sven Rohr, and Fabian Klemm. Mixture of tunable experts-behavior modification of deepseek-rl at inference time. arXiv preprint arXiv:2502. 1096, 2025.
[14] Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng. Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-expertt language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-6, 2024, pages 1280-1297. Association for Computational Linguisties, 2024.
11

---

[15] DeepSeck-Al, AI, Aixin Liu, Bei Feng, Bing Xue. Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongje Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong. Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang. Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang. Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang. Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, and Wangding Zeng. Deepseek-v3 v3 technical report. CoRR, abs/2412. 194337, 2024.
[16] Dee,Seek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, Hao Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, L. Cai. Jian Liang. Jianzhong Guo. Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang. Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, Tao Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng. Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, and Xiaowen Sun. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, Aas/2405.044344 2024.
[17] Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanovíć, and Martin Vechev. Math- arena: Evaluating Iims on uncontaminated math competitions, February 2025.
[18] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. CoRR, abs/231 1. 12022, 2023.
[19] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. CoRR, abs/2403.07974, 2024.
[20] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. CoRR, abs/2403.03853, 2024.
[21] Zican Dong, Junyi Li, Jinhao Jiang, Mingyu Xu, Xin Zhao Bingning Wang, and Weipeng Chen. LongRel: Mitigating short-text degradation of long-context large language models via restoration distillation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10687-10707, Vienna, Austria, July 2025.
[22] Di Jin, Fileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. CoRR, abs/2009. 13081, 2020.
[23] Xuanyu Zhang and Qing Yang. Xuanyuan 2.0: large chinese financial chat model with hundreds of billions parameters. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023, pages
ACM, 2023.
12

---

[24] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai. Yu Gu, Hangliang Ding. Kaiwen Men, Kejuan Yang, Shudan Zhang. Xiang Deng, Aohan Zeng. Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agenthench: Evaluating llms as agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[25] Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
(26) Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 37:62557-62583, 2024.
[27] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. In ICLR (Poster). OpenReview.net, 2017.
[28] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23: 10::11199109 2022.
[29] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of- experts inference and training to power next-gemerre gexetrenenttion AI scale. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 18332-18346. PMLR, 2022.
[30] Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer feed-forward layers are mixtures of experts. In Findings of the Association for Computational Linguisties: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 877-890. Association for Computational Linguistics, 2022.
[31] Hongyu Wang, Shuming Ma, Ruiping Wang, and Furu Wei. Q-sparse: All large language models can be fully sparsely-activated activated. CoRR, abs/2407.10969, 2024.
[32] Peiyu Liu, Tianwen Wei, Bo Zhu, Xin Zhao, and Shuicheng Yan. Masks can be learned as an alternative to experts. In ACL (1), pages 15800-15811. Association for Computational Linguisties, 2025.
[33] Peiyu Liu, Zikang Liu, Ze-Feng Gao, Dawei Gao, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Do emergent abilities exist in quantized large language models: An empirical study. LREC/COLING, pages 5194-5190. ELRA and ICCL, 2024.
[34] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 71, 2024. OpenReview.net, 2024.
[35] Haokun Lin, Haoli Bai, Zhili Liu, Lu Hou, Muyi Sun, Linqi Song, Ying Wei, and Zhenan Sun. Mope-clip: Structured pruning for efficient vision-language models with module-wise pruning error metrie. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 27370-27380, 2024.
[36] Mingjie Sun, Zhuang Liu, Anna Bair. and J. Zico Kolter. A simple and effective pruning approach for large language models. In ICLR. OpenReview.net, 2024.
13

---

A Expert Overlap Across Domains
To further analyze the overlap of top experts across different domains, we select AIME-2023, LiveCodeBench- V3, and GPQA as calibration sets and select top experts with gating scores. Sub- sequently, we compute the overlap ratio of Top-M experts (M E {1.2. 4, 8, 16, 32, 64, 128)) and expert overlap across different layers, which are shown in Figure 5 We can observe that the experts with the highest gating scores differ significantly across different domains, and as the number of top experts increases, the overlap ratio increases. Additionally, in the lower layers, there is higher over- lap among experts with significant gating scores across different datasets, but this overlap becomes relatively lower in the middle and deeper layers. This indicates that experts in the lower layers tend to focus on general capacities, while as the network depth increases, they gradually specialize in handling knowledge from distinct domains.
Ovedap
H4-
111-
Nasher Of Lapur
60-
Aw
78-
3
64
20
40
Nacbet oFTip Sapects
Lager
Lager
Figure 5: Left: Overlap ratios of Top-M experts with different M. Middle and Right: Overlap experts in top- 16 and 128 experts different layers.
B Empirical Study with EASY-EP
To further illustrate the phenomenon of few-shot expert localization, analyze it with the expert scores (as discussed in Equation 3 from our proposed EASY-EP method in place of the gating scores.
B.1 Expert Specialization Across Domains
Distinct Expert Distribution Across Domains. We compute the overlap ratios of the top experts identified by method and visualize them in Figure 6 Similar to previous observations, large differences in expert distribution still exist across different domains. However, the expert overlap between domains is larger than that of experts identified by gating scores. This may be because our method identifies experts not only frequently activated within domain but also those who contribute more than other experts in residual connections. The latter may be more similar across different datasets, thus leading to higher overlap.
Impact of Removing Domain-Specific Experts. Similarly, we remove experts whose score, as calculated by our method, falls within the top-128 for a single domain but not for any other. As shown Table removing these in-domain experts leads larger performance degradation than out-of- domain experts. However, compared with pruning with gating scores, the performance degradation is less severe. We speculate this is due to the higher overlap among the top experts identified by our method, resulting in a smaller pruning ratio.
B.2 Expert Locality Within One Domain
Effect of Calibration Set Size. We present the overlap ratios of top-128 experts selected via our method with different numbers demonstrations. As shown in Figure 7 (Left,, as the number of demonstrations increases, the identified domain-specific experts gradually become stable. For 25-shot
14

---

Top Expert Number
Top Expert Number
Top Expert Number
128
1.0
0.9
math
0.8
077
0.6
0.33
1.00
0:13
0.47
1.00
0.35
0.77
1.00
0.4
0.22
9.13
1.00
0,48
11:35
1.00
0.77
0.74
1,00
0.3
math
coding
math
science
coding
math
science
coding
Figure 6: Overlap ratios of experts with top expert scores computed with methods different datasets.
Table 7: Results of removing domain-specific experts with EASY-EP.
Domain
AIME24
GPQA
LiveCodeBench
Full
77.08
70,91
63.32
Math
72.67 (-3.41)
73.17 (+2.26)
64.22 (+0.90)
Code
76.67 (-0.41)
72.93 (+2.02)
62.27 (-1.05)
Science
76.00 (-1.08)
58.23 (-12.68)
64.97 (+1.65)
pruning, can achieve the overlap ratio of 93% with 10-shot pruning, further demonstrating the feasibility of few-shot domain-specine expert pruning.
Consistency of Expert Activation Across Datasets. Figure (Right) illustrates that the top-128 experts identified by method exhibit consistent overlap ratios across different datasets within the math domain. Our method not only presents expert consistency over different datasets similar to the gating score metric, but also yields significantly higher overlap ratios. We hypothesize that our approach is more adept at identifying experts of true domain-wide significance, as opposed to those who only important in single dataset due the dataset-specific differences.
Overlp of Top 128 Expects with Defierent Number of Shots
Overlap of Top 128 Experts on Differzni Datasets
1.00
AIME23
0.95
IMMMT-25 AIME25 AIME24
0.90
0.79
1.00
1.00
Ditoset
0.35
0.80
25
0.79
0.39
1,00
0.93
0.89
0.90
1.00
0.38
100
0.82
0.88
0.99
1.00
0.91
0.88
0.88
1.00
0.70
25
100
ATME23
AIME24
AIME25
HMMT--25
0.65
Dataset
Figure 7: Left: Overlap ratio of top-122 experts identified by EASY-EP with different numbers of demonstrations, Right: Overlap ratio of rop-128 experts identified by EASY-EP pruned with different math datasets.
15

---

Relationship Retworn Different Metries of Exports
Similarity between Ispur and Output of Rontod Experts
60
1.00
0.99
53
Top- 128
(<(
0.97
0,000
0.94
0.00
0.84
35
0.77
0.010
Laaes 3823
0.89
0.59
0.48
0.96
0.000
Tayer
0.000
0.01D 0.015
30
Token
Figure 8: Left: Gating scores and averaged product of gating value and L.2 Norm of expert outputs. Blue/red dots indicate experts in top-128 gating scores Pt.s blue/yellow dots denote experts in top- 10--128 expert importance c², black dots indicate neither. Right: Consine similarity between representations before and after incorporating the outputs the routed expert. The red box and green box indicate high similarity and low similarity, respectively.
C Empirical Analysis of Components in EASY-EP
C.1 Output-Aware Expert Importance Assessment
To analyze the relationship between whether a large gating score g¹,, ensures large product gating scores and L2 norm of expert output df,.lle/.rll. we visualize the relationship of top- 128 experts selected by the two metrics. As shown in Figure (Left), (Left), despite a significant degree of overlap among experts, there still exist some experts who excel exclusively in single, focused metric. Although the gating scores of some experts are not large, the L2 norms of their outputs are larger than others. This proves the necessity of considering experts' outputs.
C.2 Expert-Level Token Contribution Estimation
Given one sample selected from AIME-2023, we first obtain representations before and after incor- porating the outputs of routed experts in each layer of DeepSeek-R and compute the similarities between them, as illustrated in Figure (Right). We can observe that the similarities differ signifi- cantly across tokens and layers. For some tokens at specific layers, skipping the expert module results in minimal changes to the hidden states, with over 99% similarity between the input and output representations (e.g., the 15th layer of the first tokens). In contrast, for certain tokens and layers (e.g., the 12th layer the first tokens), the hidden states show substantial differences after expert routing.
D Experiment Details
As shown in the previous analysis in Section 3.2 datasets within the same domain can identify the important experts on other datasets. Thus, for different domains and tasks, we select one dataset calibration set, which is shown in Table 8 For mixed-domaa pruning,
choose the scores calculated on each 25-shot calibration set and average the normalized scores as the final scores of experts. For the evaluation of FinancelQ, we randomly select 1000 samples with the seed of 42 since the test set is too large.
Additionally, all the experiments are conducted in one 8x H200 GPU. We set the maximum context length to 32K, the temperature to 0.6, the top-p sampling value 0.95 most benchmarks (temperature 0.2 for LiveCodeBench), To ensure statistical reliability, most benchmark is evaluated independently times (32 times for math benchmarks), and we report the average performance of pass @ 1.
16

---

Table 8: Calibration Set of Each Domain
Domain
Calibration Set
License
Math
AIME 2023
co-by-nc-sa-4.0
Coding
LiveCodeBench-V2 GPQA-Main
MIT
Science
MIT
Agent
Dev Set of AgentBench-OS
Apache-2.0 license
Finance
Dev Set of FinancelQ
MIT
Medical
Dev Set of USMLE
by se-by-nc-sa-4.0
E Analysis of Perturbation-based Pruning Method
Previous studies 6 [ have proposed methods that utilize representation perturbation after expert pruning determine which experts should be removed. In NAEE (6, identifying the optimal subset of experts requires CN evaluations, where and N' represent the original and target numbers of experts, respectively. In CD-MoE (7), greedy search algorithm selects the expert retain based on minimal representation perturbation through a rolling mechanism, requiring ((N + N2) evaluations. For DeepSeek-RI. which has 256 experts and target expert count of 128, these methods methods
require over 1075 and 24768 evaluations per layer, respectively. Thus, the perturbation-bast affordable for MoE models with a large number of experts. Conversely, our method only requires one forward operation to identify critical experts, which is cheaper and more suitable for large MoE models.
F
Further Experimental Analysis
F1 Experiments with Qwen3-30B-A3B
To further demonstrate the effectiveness of our method, evaluate the performance of Qwen3-30B- A3B with different expert pruning methods. Specifically, the expert number of the model pruned from 128 to 64. As shown in Table 9 our method can achieve significantly better performances than other methods, which demonstrates the effectiveness of our method, Additionally, compared with the DeepSeek series, it is harder to compress Owen3-30B-AA3 since it has shared experts.
Qwen-30B-A3B
AIME24
AIEM25
HMMT
GPQA
LiveCodeBench
Averge
FULL
80.42
70.83
50.00
68.59
62.80
66.53
Random
0.00
0.00
0.00
23.23
0.00
4.65
frequency
13.33
10.00
0.00
61.11
14,00
19.69
Gating Scores
13.33
6.67
6.67
35.35
14.00
15.20
EASY-EP
72.92
61.25
38.75
63.13
46.30
56.47
Table 9: Experiments on Qwen3-30B-AAB.
F.2 Expert Overlap with Different Metrics
We calculate the pairwise overlap of the top-128 experts chosen by various metrics across three datasets: AIME23, GPQA-mait -ain, and LiveCodeBench-Ve V3. The results are presented as heatmap in Figure Our analysis reveals that the experts selected based on gating scores and frequency exhibit a remarkably high overlap, approximately 90%. However, the experts identified by our method show significant divergence from these metrics, underscoring the influence of incorporating expert outputs and residual changes in our approach. We posit that the large discrepancy among experts results in huge performance gap between models.
17

---

Overlap Top- Exports for Math Domain
Overlap of 128 Science Domain
Overla of Top 128 for Coding Domain
* Conting Soors Fregurtig
Guslaa Store Preguctccy
Colhig Saare Fresuency
035
MeMii:
0.80
0.25
EASY-EP
EASY-EP
0.70
0.65
0.0
Freganncy
Pregancy
Guinng
Freqiency
Guting Seare
Metii
Metii
Mettte
Figure 9: Overlap of Top-128 Experts Selected via Different Metrics.
Model
AIME24
AIME25
FULL
80.42
70.83
EASYEP
72.92
61.25
EASYEP w.o. reroute
73.33
63.33
Table 10: Performances of removing experts without rerouting.
E33 Effect of Directly Removing Experts Without Reroute
In addition to our settings, where we remain the same number activated experts, we also experi- mented with computing the gating weights for each expert normal and selecting them accordingly. Then, for the pruned experts, we set their outputs to zero. Thus, to some extent, tokens that would have been routed to the pruned experts are processed by fewer experts. We employ Qwen3-30B-AL --3--333 for experiments and compare the performances with and without reroute, as shown in Table TO, We observe that the performances slightly improve compared to the original version. In addition, can save computation due to the reduced number of activated experts, which can further accelerate the inference speed.
F,4 Results of Pruning with Different Ratios at Different Layers
To further investigate pruning performance our method, we employed layer-wise dynamic pruning. Specifically, we first normalized the expert scores for each layer and ranked all experts across all layers. Subsequently, we pruned the top experts using different pruning ratios. As shown in Table [I] observe that employing layer-wise dynamic pruning does not always lead to better performance, and the performance on LiveCodeBench even drops significantly. In addition, the dynamic compression ratio leads deployment difficulties, requires different numbers of experts for each layer. Thus, we suggest just employing fixed pruning ratio across all layers.
Table 11: Performance comparison of EASY-EP with and without employing layer-wise compression ratios.
Model
Ratio
Layer
AIME2024
GPQA
LiveCodeBench
0.5
X
79.17
70.12
61.11
0.5
/
79.33
65.15
44.31
DeepSeek-RI
0.25
x
72.67
67.47
42.51
0.25
65.33
66.16
28.74
0.5
55.21
65.25
46.71
0.5
/
58.67
62.63
38.23
DeepSeek-V3-0324 V3-0324
0.25
53.12
57.35
27.99
0.25
58.67
60.10
21.56
18

---

E55 Throughput with Different Lengths
To further explore the throughput changes on pruning models, we consider three additional length settings: (1) 1K+4K: IK input length and 4K output length; (2) 4K+/K: 4K input length and IK output length; and (3) 4K+4K: 4K input length and 4K output length. Figure [0 (Right) presents results for the other settings. Compared with /K+/K, either increasing the length of input or output leads lower throughput and throughput acceleration after pruning. Additionally, output scenarios with an equívalent total sequence length, the 128-expert configuration achieves significantly higher acceleration ratio (2.91 under the KK4KK setting, compared 2.52 under the 4K+ IK setting).
long
Throughput with Different Number of Experts
128
4960
Thoughgut (Tokenss)
256
3000
2493
1830
1969
1526
841
IK,4K
4K.IK Length Settings
4K.4K
Figure 10: Total throughput across different numbers of experts.
19

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide the main claims in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include Limitation section in Appendix A.
20

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We have no aussmptions and proof of theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We have provided the information to reproduce the results in Section 5.1 and Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Our code is available https///hthabb.omm/UUUCCBBBEEEES
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (tttps://niis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pblic/guideeee http:/ooeeosmmiiiooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
22

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We give experiment settings in Section 5.1 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not provide this.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resources in Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
23

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/nnerrpps http:/uuwi......ppooooaaaaaaaaaa
Answer: [Yes]
Justification: The paper respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite existing papers and URLs. We also point out their license in Table 8.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We will release the code and documentation after our paper has been accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
25

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: We only employ LLMs for polishing writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.M policy /ccp:p//enearrrssscc/oofeeeeee/e for what should or should not be described.
26


Reviews:
[{"review_id": "0wByiuTceC", "reviewer": "Reviewer_kkGE", "summary": "This paper investigates domain specific pruning of the experts in large SOTA MoE models (e.g., DeepSeek-R1 (671B)). The paper confirms that for such models, different expert specializes to different domain. Based on this confirmation, the paper designs a method for identifying domain-irrelevant experts for pruning them in order to reduce inference cost.", "questions": "1. How do you use the tokens of the pruned experts? Do you re-route them? Or, do you avoid computing the tokens?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe empirical analysis provided in this paper to demonstrate the domain-specific specialization of experts in large-scale MoE models with many experts validates (probably for the first time in this paper) the hypothesis, and useful to MoE research community\n2.\tThe proposed method of combining routing statistics with some expert’s output dependent metrics (output L2 norm and cosine similarity) outperforms some baselines provided in this paper\n3.\tThe paper is well-written and easy to follow\n\nWeakness:\n\n1.\tAlthough, the specific combination of metrics used in this paper for evaluating the experts’ importance is novel, most of these metrics are used before in literature.\n2.\tIt’s not clear why the specific combination of metrics works better than the baselines provided in this paper, i.e., no theoretical justification has been provided. Specifically, for the case of using expert’s output L2 norm, equation (6) shows that larger the L2 norm of the output of an expert, larger the contribution of the expert in the overall L2 norm of the final output. But, why the contribution to final output’s L2 norm is a good measure?\n3.\tThe comparison with existing baselines is inadequate. Although it has been mentioned that evaluating some of these baselines are computationally expensive (e.g., [1], [2]), still a comparison should be provided with those baselines to check that whether the proposed method performs worse than these methods. Specifically, those baselines should be included which doesn’t share any metric with the proposed combination of metrics (e.g., router-norm based method in [3]).\n\n[1] Lu, Xudong, et al. \"Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models.\" Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.\n\n[2] Cao, Mingyu, et al. \"Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning.\" arXiv preprint arXiv:2412.00069 (2024).\n\n[3] Chowdhury, Mohammed Nowaz Rabbani, et al. \"A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts.\" Forty-first International Conference on Machine Learning.", "was_revised": true, "final_justification": "The authors clarified a major part of the weaknesses. Therefore, I updated the rating.", "rebuttal": "Thank you for your insightful suggestions!\n## W1: Most of these metrics have been used before in literature\n\nWe appreciate the reviewer’s observation that several of the metrics employed in our work have appeared in prior literature. We would like to clarify two key points regarding novelty and contribution.\n\n1. For the issue of simplicity being mistaken for lack of novelty， our method’s simplicity is not due to insufficient innovation but is a deliberate design decision. For ultra-large MoE models such as DeepSeek-R1 (671B), overly complex pruning schemes are impractical. A lightweight scoring mechanism that requires only a single forward pass is crucial to ensure efficiency and deployability. Thus, simplicity is a necessity for practical impact rather than a limitation.\n2. Regarding the reviewer’s request to clarify our core novelty beyond metric reuse, our primary contribution lies in uncovering the counterintuitive phenomenon of “few-shot expert localization”. We demonstrate, for the first time, that in ultra-large MoE models, identifying domain-specialized experts actually becomes *easier* with only a few demonstrations. This sheds light on the clear functional partitioning inside very large MoEs and enables effective domain-specific pruning.\n3. In addition, compared to prior metric-based approaches, our method provides a stronger theoretical grounding. Specifically, we propose using the L2 norm of the gated expert output and the similarity between the input and output of MoE modules as the importance score, which can be interpreted as an approximation of the gradient-based contribution to the loss. Under this theoretical framework, our method serves as a principled proxy rather than a heuristic, ensuring that our pruning strategy aligns with the optimization objective. A detailed derivation is provided in our response to W2.\n\n## W2: Why is the contribution to the final output’s L2 norm a good measure in theory?\n\nWe provide a principled derivation from our optimization objective and gradient analysis, showing how the L2 norm is a good measure.\n### Optimization Objective:\n\nOur pruning goal is to minimize the expected loss on a calibration dataset $D_{\\text{calib}}$, under a constraint that only $M$ experts are retained per layer. Formally, we solve:\n\n$$\n\\min_{m_l} \\mathbb{E}^{(x,y)}  (\\mathcal{L}(f(x; \\{m_l\\}), y)) , \\quad \\text{s.t. } \\sum_{i=1}^N m_{il} = M\n$$\n\nHere, $m_{il} \\in \\{0, 1\\}$ is a binary mask indicating whether expert $i$ in layer $l$ is retained.\n\nThis is a combinatorial problem and is NP-hard. To solve it approximately, we adopt a greedy strategy that relies on estimating the importance of each expert.\n\n### Gradient-Based Expert Importance:\n\nWe approximate the change in loss when removing expert $j$ via a first-order Taylor expansion:\n\n$$\n\\Delta \\mathcal{L} \\approx -\\frac{\\partial \\mathcal{L}}{\\partial m_{jl}}\n$$\n\nApplying the chain rule, we obtain:\n\n$$ \\frac{\\partial \\mathcal{L}}{\\partial m_{j,l}} = \\sum_{t=1}^T \\left( \\frac{\\partial \\mathcal{L}}{\\partial \\tilde{h}{t,l}} \\right)^T \\left( g_{j,t,l} e_{j,t,l} \\right) $$\n\nThis shows that the importance of expert $j$ is proportional to the projection of the gradient of the loss onto the expert’s output.\n\n### L2 Norm in Output-Aware Expert Importance Assessment:\n\nWhile we cannot compute the gradient $\\partial \\mathcal{L} / \\partial \\tilde{h}\\_{t,l}$ during a forward-only pass, we **can** compute the expert’s contribution term $g\\_{j,t,l} e\\_{j,t,l}$.\n\n$$\n( \\frac{\\partial \\mathcal{L}}{\\partial \\tilde{h}\\_{t,l}} )^\\top  g\\_{j,t,l} e\\_{j,t,l} ) \\leq \\| \\frac{\\partial \\mathcal{L}}{\\partial \\tilde{h}\\_{t,l}} \\|\\_2 \\cdot \\| g\\_{j,t,l} e\\_{j,t,l} \\|\\_2\n$$\n\nThis upper bound shows that **the L2 norm of $g_{j,t,l} e_{j,t,l}$** is a direct proxy for the expert’s contribution to the loss-sensitive direction in the model. Since the gradient component is unavailable during a forward pass, we approximate expert importance using this measurable term.\n\nThus, **using the expert’s output L2 norm as part of the importance score is not ad hoc—it follows directly from a first-order gradient-based analysis of our objective.**\n\n### Additional analysis of Expert-Level Token Contribution Estimation\nFor expert-token contribution estimation, we need to compute $\\partial \\mathcal{L} / \\partial \\tilde{h}_{t,l}$. However, it is very costly to conduct gradient computation. Thus, we emply $1-Sim(\\tilde h_t^l,h_t^l)$ as a approximation.\n\n\n## W3: Comparison with perturbation-based methods and router-norm-based methods\n\nDue to the enormous parameter size of DeepSeek-R1 and the high complexity required by this method, it is difficult for us to implement it directly. As analyzed in our paper, NAEE and CD-MOE require 10^75 and 24,768 evaluations per layer, respectively. On a single H200 GPU, performing one round of pruning would require over 10^75 evaluations and approximately 9,976 days to complete (as described in Appendix E), which is unacceptable for current frameworks. Therefore, this method is, in any case, impractical to apply to models as large as DeepSeek-R1.\n\nFor the router-norm-based method, we require the models before and after fine-tuning to evaluate the change in router. Thus, we select DeepSeek-V3-base and DeepSeek-R1 to select experts with the maximum L2 Norm of the difference of router weights and remove other experts. As shown in the following Table, this method can not achieve comparable performance to our method, which further demonstrates the effectiveness of our method.\n| Model | Method | Mix | \\#E | AIME-24 | GPQA | LiveCodeBench |\n| --- | --- | --- | --- | --- | --- | --- |\n| DeepSeek-R1 | EASYEP | $\\times$ | 128 | 79.17 | 70.12 | 61.11 |\n|  | EASYEP | $\\checkmark$ | 128 | 75.94 | 70.36 | 57.63 |\n|  | Router-Norm | - | 128 | 16.67 | 50.00 | 20.96 |\n## Q1: How do you use the tokens of the pruned experts?\n\nIn our framework, tokens originally routed to the pruned experts are **not discarded**. Instead, they are **re-routed to the remaining experts** based on the gating mechanism. Specifically, after pruning, the router operates over the retained experts only, and each token is dynamically assigned to the most suitable expert(s) among this reduced set. This ensures that every token continues to receive expert processing without additional overhead."}, {"review_id": "oYMBHQlczv", "reviewer": "Reviewer_376Z", "summary": "This paper investigates domain specialization in large Mixture-of-Experts (MoE) models. The authors observe that with just a few domain-specific examples, MoE models consistently activate a sparse and stable subset of experts (\"few-shot expert localization\").\n\nBased on this observation, they develop a pruning framework, EASY-EP, that combines output-aware expert importance assessment with expert-level token contribution estimation to identify and retain only the most relevant experts for specific domains.", "questions": "1. The token contribution metric, $s_{t}^{l} = 1 - Sim(h_{t}^{l}, \\tilde{h}_{t}^{l})$ , measures the representational change induced by the entire MoE module for a given token. Have the authors investigated how this change correlates with the properties of the activated experts? For example, does a larger change (lower similarity) correlate with higher variance in the gating scores for that token?\n\n2. How does the method perform on other MoE architectures beyond DeepSeek models?\n\n3. Can the method be extended to dynamic expert selection during inference?", "limitations": "The paper could benefit more by including a discussion of tasks that span multiple domains.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The discovery of \"few-shot expert localization\" is compelling and well-documented. The systematic analysis showing that only 5-25 demonstrations are sufficient to identify domain-specific experts (Figure 2C) is a valuable empirical finding that advances our understanding of MoE model behavior.\n\n2. EASY-EP is well-designed with two complementary components. The method is computationally cheap, requiring only a single forward pass to calculate scores, making it practical even for models with hundreds of experts. \n\n3. The paper is well-written\n\n## Weaknesses\n1. All experiments use DeepSeek variants. It's unclear whether the findings generalize to other MoE architectures (e.g., Moonlight-MoE, OLMoE, Mistral). This significantly limits the generalizability claims.\n\n2. The comparison is mainly against simple baselines (random, frequency, gating scores). More sophisticated expert/model pruning methods from recent literature(e.g., MC-SMoE, Angular) could provide stronger baselines.", "was_revised": true, "final_justification": "In the discussion period, the authors have addressed my concerns raised about baseline choices and performance on more MoE models. I raise my score from 3 to 4 as a result.", "rebuttal": "Thank you for your insightful suggestions!\n## W1&Q2: Experiments on more models\n\nWe provide the experimental results on Qwen3-30B-A3B, the results are shown in the following Table. We can observe that for the MoE models without shared experts (which appear in DeepSeekMoEs), expert pruning is much harder. For example, the performance of other methods on AIME is very poor. On the contrary, our method can achieve over 80% performance on AIME with only 50% experts, further demonstrating the effectiveness and universality.\n\n| Qwen-30B-A3B | AIME24 | AIEM25 | HMMT | GPQA | LiveCodeBench |\n| --- | --- | --- | --- | --- | --- |\n| FULL | 80.42 | 70.83 | 50 | 68.59 | 62.8 |\n| Random | 0 | 0 | 0 | 23.23 | 0 |\n| frequency | 13.33 | 10 | 0 | 61.11 | 14 |\n| Gating Scores | 13.33 | 6.67 | 6.67 | 35.35 | 14 |\n| EASYEP | 72.92 | 61.25 | 38.75 | 63.13 | 46.3 |\n\n## W2: Comparison with complex baselines\n\nAs shown in our paper, we present M-SMoE (MC-SMoE is a further pruning version of M-SMoE, and its performance is worse than it). As shown in Table 2 in our paper and the following table, although the method is more complex, the performance of M-SMoE is significantly worse than our method, which demonstrates the effectiveness of our method.\n\n| Model | Method | Mix | \\#E | AIME-24 | AIME-25 | FMMT | LiveCode | GPQA | USMLE | FinIQ | A-OS | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DeepSeek-R1 | Full | - | 256 | 77.08 | 66.67 | 44.38 | 63.32 | 70.91 | 92.66 | 82.1 | 40.51 | 67.20 |\n|  | M-SMoE | $\\times$ | 64 | 0.00 | 0.00 | 0.00 | 0.00 | 12.12 | 0.00 | 0.00 | 0.00 | 1.52 |\n|  | EASY-EP | $\\times$ | 64 | 72.81 | 55.10 | 38.02 | 42.51 | 67.47 | 26.63 | 33.90 | 27.26 | 45.22 |\n|  | M-SMoE | $\\times$ | 128 | 5.33 | 6.00 | 3.33 | 25.75 | 24.75 | 52.63 | 39.60 | 19.44 | 22.10 |\n|  | EASY-EP | $\\times$ | 128 | 79.17 | 68.33 | 45.31 | 61.11 | 70.12 | 91.67 | 78.80 | 37.92 | 66.55 |\n|  | M-SMoE | $\\checkmark$ | 128 | 6.67 | 2.00 | 4.67 | 4.19 | 32.32 | 72.00 | 19.10 | 6.25 | 18.40 |\n|  | EASY-EP | $\\checkmark$ | 128 | 75.94 | 61.98 | 42.50 | 57.63 | 70.36 | 91.20 | 57.95 | 34.17 | 61.47 |\n| DeepSeek-V3-0324 | Full | - | 256 | 55.73 | 47.71 | 28.75 | 48.50 | 66.87 | 87.51 | 64.22 | 33.33 | 54.08 |\n|  | M-SMoE | $\\times$ | 64 | 16.67 | 13.33 | 3.33 | 1.20 | 22.22 | 12.18 | 47.00 | 21.52 | 17.18 |\n|  | EASY-EP | $\\times$ | 64 | 53.12 | 41.56 | 28.85 | 27.99 | 57.35 | 84.57 | 72.50 | 27.55 | 49.19 |\n|  | M-SMoE | $\\times$ | 128 | 48.00 | 38.67 | 28.67 | 30.53 | 55.82 | 86.72 | 66.60 | 33.33 | 48.54 |\n|  | EASY-EP | $\\times$ | 128 | 55.21 | 46.88 | 31.56 | 46.71 | 65.25 | 86.72 | 63.58 | 37.08 | 54.12 |\n|  | M-SMoE | $\\checkmark$ | 128 | 43.33 | 30.00 | 20.00 | 7.19 | 52.53 | 82.33 | 62.20 | 29.17 | 40.84 |\n|  | EASY-EP | $\\checkmark$ | 128 | 57.81 | 46.56 | 33.33 | 40.72 | 64.95 | 85.00 | 72.26 | 38.74 | 54.92 |\n\n## Q1: Relationship between similarities and variance in gating scores\n\nTo investigate the relationship between the token-level representational change (measured by our similarity-based token contribution metric) and the properties of the activated experts, we computed the Pearson correlation between token similarity and the variance of the corresponding gating scores. The result was −0.3119, indicating a weak negative correlation. Though tokens with larger representational changes (i.e., lower similarity) tend to have slightly higher variance in their gating scores, this relationship is not strong. \n\n## Q3: Potential for employment on dynamic expert selection during inference\n\nYes, I think our method can also be employed in the task. We infer that the dynamic expert selection during inference aims to identify important experts in the MoE models via the initial tokens and to prune the MoE models for other tokens in the long generation. We analyze the top experts in the first 1000 tokens and all the tokens and observe that there are over 90% overlap ratios. Thus, we can dynamically prune experts according to the initial 1000 tokens for the long-generation tasks."}, {"review_id": "9hzK3vLbOc", "reviewer": "Reviewer_mHRa", "summary": "The paper investigates the challenges associated with the memory overhead of large Mixture-of-Experts (MoE) models, particularly focusing on the DeepSeek-R1 model, which has 671 billion parameters. The authors introduce a novel pruning framework called EASY-EP, which leverages a few domain-specific demonstrations to effectively identify and retain the most relevant experts while discarding less important ones. \n\nThe key contributions of the paper include:\n\n1. Observation of Few-Shot Expert Localization: The authors identify a consistent behavior where, given a small number of in-domain examples, the MoE model activates a stable and sparse subset of experts that are highly relevant to the task at hand. This phenomenon is termed \"few-shot expert localization.\"\n\n2. Development of EASY-EP Framework: The proposed framework consists of two main components: output-aware expert importance assessment and expert-level token contribution estimation. This allows for efficient pruning of experts based on their relevance to the current task, significantly reducing memory requirements without sacrificing performance.\n\n3. Empirical Validation: The authors conduct extensive experiments on the DeepSeek-R1 and DeepSeek-V3-0324 models across various benchmark datasets. They demonstrate that their pruning method can maintain comparable performance while achieving a 2.99 times increase in throughput under the same memory constraints by retaining only half of the experts.\n\n4. Domain Specialization Insights: The paper provides insights into the domain specialization of experts within large MoE models, showing that experts are often highly specialized and can be reliably identified with minimal demonstrations. This finding underscores the potential for domain-specific pruning strategies in large-scale models.\n\nOverall, the paper presents a significant advancement in the efficient deployment of large MoE models, addressing both memory constraints and performance optimization through innovative pruning techniques.", "questions": "1. Statistical Significance Reporting: The paper does not provide error bars or any information regarding the statistical significance of the experimental results. This is a critical aspect of scientific reporting, as it helps to understand the reliability and variability of the results. The authors should include this information to strengthen their claims and provide a clearer picture of the model's performance.\n2. Reproducibility of Results: While the paper claims to provide sufficient information for reproducing the results, it would benefit from a more detailed description of the experimental setup, including specific hyperparameters, training procedures, and the exact datasets used. This transparency is essential for other researchers to validate the findings and build upon the work.\n3. Impact of Expert Pruning on Long-term Generation Abilities: The paper mentions that while domain capabilities may be preserved in pruned models, there are concerns about the long-term generation abilities being compromised. The authors should provide more empirical evidence or analysis to support this claim. It would be helpful to include experiments that specifically assess the long-term performance of the pruned models compared to the full models.\n4. Generalization Across Domains: The concept of few-shot expert localization is intriguing, but the authors should elaborate on how well the identified experts generalize across different domains. It would be beneficial to include additional experiments that test the model's performance on unseen datasets within the same domain to validate the robustness of the few-shot approach.\n5. Clarity and Structure of the Paper: The paper could improve its clarity by ensuring that the key contributions and findings are clearly articulated in the abstract and introduction. A more structured presentation of the results, including visual aids or tables summarizing key findings, would enhance the reader's understanding and engagement with the material. \n\nThese suggestions aim to enhance the rigor and clarity of the paper, ultimately contributing to its impact in the field.", "limitations": "The paper presents a thorough investigation into the domain specialization of experts in large Mixture-of-Experts (MoE) models, particularly focusing on the DeepSeek-R1 model. However, there are several limitations that should be addressed:\n\n1. Generalizability of Findings: While the study demonstrates the effectiveness of the proposed EASY-EP pruning method across various domains, it primarily relies on specific datasets (e.g., AIME-2023, GPQA-main, LiveCodeBench-V3). The authors should discuss how well these findings generalize to other domains or tasks not covered in their experiments.\n2. Long-term Performance: The paper notes that while domain capabilities may be preserved in pruned models, there are concerns about the long-term generation abilities being compromised. This aspect requires further exploration, as it could significantly impact the practical deployment of the model in real-world applications.\n3. Statistical Significance: The paper does not report error bars or other statistical significance measures for the experimental results. Including this information would strengthen the validity of the claims made regarding the performance of the proposed method.\n4. Potential Negative Social Impacts: The authors state that there is no societal impact of the work performed. However, it is essential to consider potential negative implications, such as the misuse of the technology or biases that may arise from the model's training data. A more in-depth discussion on these aspects would be beneficial.\n5. Clarity and Detail in Methodology: While the methodology is described, there could be more clarity regarding the specific steps taken during the pruning process and how the calibration sets were selected. This would enhance reproducibility and understanding of the method.\n\nIn summary, while the paper makes significant contributions to the field, addressing these limitations and suggestions would enhance its robustness and applicability.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper presents a well-structured analysis of expert activations in large Mixture of Experts (MoE) models, particularly focusing on the DeepSeek-R1 model. The methodology is robust, employing a novel pruning framework (EASY-EP) that effectively leverages few-shot demonstrations to identify and retain relevant experts. The experimental results demonstrate that the proposed method achieves comparable performance while significantly increasing throughput, indicating a high-quality contribution to the field.\n2. The document is clearly organized, with a logical flow from the introduction to the methodology and results. Key concepts such as few-shot expert localization and the components of the EASY-EP framework are well-defined, making it accessible to readers with varying levels of expertise in the subject matter. The use of figures and tables to illustrate findings enhances understanding.\n3. The research addresses a critical challenge in deploying large-scale MoE models—memory efficiency. By demonstrating that domain-specific experts can be effectively identified with minimal demonstrations, the paper contributes valuable insights that could lead to more efficient model deployment in real-world applications. The findings have implications for various domains, including math, science, and coding, which adds to the paper's significance.\n4. The concept of few-shot expert localization is a novel contribution to the field of machine learning and MoE architectures. The integration of output-aware expert importance assessment and expert-level token contribution estimation into the pruning process showcases innovative thinking and a fresh approach to expert pruning strategies.\n\nWeaknesses:\n\n1.While the experimental results are promising, the paper lacks detailed statistical significance reporting, such as error bars or confidence intervals, which could strengthen the validity of the findings. This omission may raise concerns about the robustness of the results.\n2.  Although the paper is generally well-written, some sections may benefit from additional elaboration, particularly in the explanation of the underlying mechanisms of the proposed pruning method. More detailed descriptions of the experimental setup and parameters could enhance reproducibility.\n3. While the findings are significant, the paper could further explore the long-term implications of pruning on model performance, particularly regarding generation capabilities. A discussion on potential trade-offs between efficiency and performance in different contexts would add depth to the analysis.\n4.  Although the paper introduces a novel framework, it could benefit from a more comprehensive comparison with existing expert pruning methods. Highlighting the unique advantages of EASY-EP over other approaches would strengthen the argument for its originality and effectiveness.\n\nIn summary, the paper presents a valuable contribution to the field of MoE models, with strengths in quality, clarity, significance, and originality. However, addressing the identified weaknesses could enhance its overall impact and acceptance in the research community.", "was_revised": true, "final_justification": "The author addressed most of my concerns, however, after careful consideration and reference to the comments of other reviewers and the author's responses, such as whether the proposed combination of indicators was a suitable solution for the described scenario, etc., I decided to maintain my initial score", "rebuttal": "Thank you for your insightful suggestions!\n## W1/Q1/L1: Statistical Significance Reporting\n\nWe have provided the mean and standard deviation for computing error bars of DeepSeek-R1 on the AIME2024 dataset (32 samples), as shown in the table below. The results demonstrate that our method achieves better performance and comparable robustness compared to the full model baseline.\n\n| **Model** | **Mean Accuracy** | **Standard Deviation** |\n| --- | --- | --- |\n| Full Model | 0.7708 | 0.0232 |\n| Our Method | 0.7917 | 0.0289 |\n\n## W2/Q2/L2: Reproducibility of Results\n\nWe have provided all the experiment details in Appendix E and Section 5.1 to ensure the reproducibility of results. \n\n## W3/Q3/L3: Impact of Expert Pruning on Long-term Generation Abilities\n\nIn our paper, we analyze the gap in long-term generation abilities between the pruned reasoning and non-reasoning models. For reasoning models, the long-term generation abilities are easily harmed after pruning for baselines. However, our method can better preserve the long-term generation abilities with only half the experts in DeepSeek-R1 \n\n## W4: comprehensive comparison with existing expert pruning method\n\nIn our paper, we conduct a comparison with existing methods, including expert pruning and merging. Our method can achieve significantly better performance than these methods, demonstrating the effectiveness of our method. \n\nIn addition, there exist some perturbation-based methods that are not evaluated due to the huge computational costs. As analyzed in Appendix F, it will take over 10^7 for pruning one model, which is not useful for the large MoEs.\n\n## Q4: Generalization Across Domains\n\nWe analyze the generalization both in-domain and out-of-domain.\n\n**In-domain generalization:** To verify that our method is not only effective for a specific dataset, we conducted in-domain cross-dataset experiments in Section 3.2. As shown in Figure 2 (D), we found that within the mathematics domain, the overlap rate of important experts identified using different datasets (AIME-2023, AIME-2024, AIME-2025, HMMT-Feb 2025) exceeded 84%. This indicates that the experts identified by our method have strong in-domain generality and are not merely adapted to a specific calibration dataset.\n\n**Out-of-domain generalization:** We also evaluated the out-of-domain generalization capability. In Section 5.3.2 and Table 4, we show the performance on other unrelated domains after pruning with data from a specific domain. The experimental results show that the model exhibits a certain degree of generalization ability, especially between similar domains (e.g., mathematics/science, code/operating system agents). Meanwhile, we also point out that for multi-task scenarios, we recommend using a \"mixed-domain pruning\" strategy, as its cross-domain performance is superior to single-domain pruning.\n\n## Q5/L5: Clarity and Structure of the Paper and Method\n\nWe will improve the structure of our paper in the final version.\n\n## L4: Potential Negative Social Impacts\n\n Our work aims at the memory efficiency of large MoE models through a pruning technique. Thus, our method itself does not generate or exacerbate bias. Its impact is inherently linked to the societal impact of the base model being pruned, such as DeepSeek-R1.\n\nHowever, we fully agree that any technology that makes large models easier to deploy could indirectly amplify both the positive and negative societal impacts of these models themselves. In the final version, we will add a statement to the limitations section to clarify that while our method is technically neutral, its application should adhere to the responsible usage guidelines for the underlying large language models."}, {"review_id": "qWYEosqKS8", "reviewer": "Reviewer_3TmE", "summary": "This paper proposes EASY-EP, a training-free pruning framework that (i) scores each expert by the product of its gating value and output norm and (ii) weights scores by token-level representation change, then retains only the top-ranked experts per layer. Applied to DeepSeek-R1 and DeepSeek-V3-0324, pruning 50 % of experts maintains ≥90 % of full-model accuracy across math, coding, science, finance, medicine, and agent tasks, while boosting 1k-token throughput by 2.99× and allowing single-node deployment. EASY-EP needs just one forward pass, no retraining, and also supports mixed-domain pruning with balanced performance, enabling memory- and latency-efficient use of gigantic MoE models.", "questions": "1. The paper acknowledges a larger accuracy drop when pruning with a single domain and testing on a different one. Could you add qualitative or quantitative analyses (e.g., expert-activation overlap heat-maps) to pinpoint why EASY-EP underperforms here and outline mitigation strategies such as hybrid expert pools or dynamic gating?\n\n2. In line 121, could the authors properly define \"demonstrations\"?", "limitations": "No discussion on limitations. The authors could consider discussing whether the insights of this work can be extended to much boarder domains/modalities/MoE architectures.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The empirical study spans two large-scale SOTA LLMs with MoE architecture and six downstream domains. The comprehensive ablations verify the necessity of both scoring factors that the paper proposed.\n\n2. The method is straightforward: one forward pass suffices, avoiding the combinatorial evaluations required by perturbation-based baselines.\n\n3. This paper provides a lot of helpful insights, such as MoE routers naturally localize to a small, domain-consistent expert set with only a handful of in-domain examples. It further shows that a single-node deployment with minimal accuracy loss is practically valuable. \n\n**Weakness:**\n\n1.\tThe proposed method is largely incremental and offers limited structural novelty; its computations rely primarily on similarity or norm-based measures.\n\n2.\tTo enhance interpretability, the authors could analyze expert capacities and token-routing distributions across different domains, comparing these patterns between the baseline and the proposed approach.\n\n3.\tThe study’s statistical reliability is weak: error bars and significance tests are absent, leaving uncertainty about the robustness of the reported improvements.\n\n4. It would be better to validate the performance of the method on at least one additional MoE family other than DeepSeek, e.g. Qwen or Mixtral. This would be helpful to show that the method can generalize beyond DeepSeek’s router design", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful suggestions!\n\n## W1: limited structural novelty\n\nWe appreciate the reviewer’s concern about the structural novelty of our approach and the reliance on similarity- or norm-based measures.\n\n(1) First, our goal is not to introduce architectural changes but to provide a practical pruning framework tailored to ultra-large MoE models. In this context, structural changes in the sense of redesigning the MoE architecture would require retraining from scratch, which is infeasible at the scale of DeepSeek-R1 (671B). Instead, we focus on a post-hoc pruning strategy that is lightweight, training-free, and easily deployable, which we argue is a critical form of innovation for models of this magnitude.\n\n(2) Second, selecting an effective metric for expert pruning is non-trivial， especially for large MoE models. Through our analysis on DeepSeek-R1, we observed a counterintuitive phenomenon: with ultra-large MoEs, a small number of calibration samples is sufficient to consistently identify domain-specialized experts. We term this the *few-shot expert localization* phenomenon. Building on this, our integration of output-aware expert importance (gating value × output norm) with token-level contribution estimation enables us to capture both the strength and the influence of expert activations in a single forward pass, making accurate and efficient pruning feasible at this unprecedented scale.\n\n(3) Third, our scoring strategy is theoretically grounded. Beyond empirical observations, we show that using the L2 norm of the gated expert output and similarity of the input and output of MoE modules can be interpreted as an approximation of the gradient-based contribution to the loss. This ensures that our pruning strategy is not heuristic but aligned with the underlying optimization objective. We show the analysis below.\n### Optimization Objective:\n\nOur pruning goal is to minimize the expected loss on a calibration dataset $D_{\\text{calib}}$, under a constraint that only $M$ experts are retained per layer. Formally, we solve:\n\n$$\n\\min_{m_l} \\mathbb{E}^{(x,y)}  (\\mathcal{L}(f(x; \\{m_l\\}), y)) , \\quad \\text{s.t. } \\sum_{i=1}^N m_{il} = M\n$$\n\nHere, $m_{il} \\in \\{0, 1\\}$ is a binary mask indicating whether expert $i$ in layer $l$ is retained.\n\nThis is a combinatorial problem and is NP-hard. To solve it approximately, we adopt a greedy strategy that relies on estimating the importance of each expert.\n\n### Gradient-Based Expert Importance:\n\nWe approximate the change in loss when removing expert $j$ via a first-order Taylor expansion:\n\n$$\n\\Delta \\mathcal{L} \\approx -\\frac{\\partial \\mathcal{L}}{\\partial m_{jl}}\n$$\n\nApplying the chain rule, we obtain:\n\n$$ \\frac{\\partial \\mathcal{L}}{\\partial m_{j,l}} = \\sum_{t=1}^T \\left( \\frac{\\partial \\mathcal{L}}{\\partial \\tilde{h}{t,l}} \\right)^T \\left( g_{j,t,l} e_{j,t,l} \\right) $$\n\nThis shows that the importance of expert $j$ is proportional to the projection of the gradient of the loss onto the expert’s output.\n\n### L2 Norm in Output-Aware Expert Importance Assessment:\n\nWhile we cannot compute the gradient $\\partial \\mathcal{L} / \\partial \\tilde{h}\\_{t,l}$ during a forward-only pass, we **can** compute the expert’s contribution term $g\\_{j,t,l} e\\_{j,t,l}$.\n\n$$\n( \\frac{\\partial \\mathcal{L}}{\\partial \\tilde{h}\\_{t,l}} )^\\top  g\\_{j,t,l} e\\_{j,t,l} ) \\leq \\| \\frac{\\partial \\mathcal{L}}{\\partial \\tilde{h}\\_{t,l}} \\|\\_2 \\cdot \\| g\\_{j,t,l} e\\_{j,t,l} \\|\\_2\n$$\n\nThis upper bound shows that **the L2 norm of $g_{j,t,l} e_{j,t,l}$** is a direct proxy for the expert’s contribution to the loss-sensitive direction in the model. Since the gradient component is unavailable during a forward pass, we approximate expert importance using this measurable term.\n\nThus, **using the expert’s output L2 norm as part of the importance score is not ad hoc—it follows directly from a first-order gradient-based analysis of our objective.**\n\n### Additional analysis of Expert-Level Token Contribution Estimation\nFor expert-token contribution estimation, we need to compute $\\partial \\mathcal{L} / \\partial \\tilde{h}_{t,l}$. However, it is very costly to conduct gradient computation. Thus, we emply $1-Sim(\\tilde h_t^l,h_t^l)$ as a approximation.\n\n\n## W2: Expert capacities and token-routing distributions across different domains with different methods.\n\nIn Figures 2, 6, and 9, we present a comparison of the similarities between the gating scores and EASYEP experts across different domains, as well as the similarities among EASYEP, gating scores, and frequency. It can be observed that our method exhibits greater variability in expert importance compared to the others, allowing it to better capture the relative importance among experts. On the other hand, our method shows higher similarity across domains than the gating scores, which to some extent indicates that our approach captures expert importance more effectively, rather than merely domain-specific characteristics, thereby maintaining a stronger overall capability.\n\n## W3: Weak statistical reliability\n\nWe have provided the mean and standard deviation for computing error bars of DeepSeek-R1 on the AIME2024 dataset (32 samples), as shown in the table below. The results demonstrate that our method achieves better performances and comparable robustness  compared to the full model baseline.\n\n| **Model** | **Mean Accuracy** | **Standard Deviation** |\n| --- | --- | --- |\n| Full Model | 0.7708 | 0.0232 |\n| Our Method | 0.7917 | 0.0289 |\n\n## W4: Experiments with more models\n\nWe provide the experiments results on Qwen3-30B-A3B, the results are shown in the following Table. We can observe that for the MoE models without shared experts (which appears in DeepSeekMoEs), expert pruning is much more hard. For example, performances of other methods on AIME is very poor. On the contory, our method can achieve over 80% performances on AIME with only 50% experts, further demonstrating the effectiveness and universality.\n\n| Qwen-30B-A3B | AIME24 | AIEM25 | HMMT | GPQA | LiveCodeBench |\n| --- | --- | --- | --- | --- | --- |\n| FULL | 80.42 | 70.83 | 50 | 68.59 | 62.8 |\n| Random | 0 | 0 | 0 | 23.23 | 0 |\n| frequency | 13.33 | 10 | 0 | 61.11 | 14 |\n| Gating Scores | 13.33 | 6.67 | 6.67 | 35.35 | 14 |\n| EASYEP | 72.92 | 61.25 | 38.75 | 63.13 | 46.3 |\n\n## Q1: **Analysis of performance drops of task-specific pruning and methods to mitigate it.**\n\nWe agree that pruning with a single-domain focus can lead to performance degradation when testing on different domains, and we have performed both qualitative and quantitative analyses to better understand and address this limitation of EASY-EP.\n\n**Low Cross-Domain Expert Overlap**: Figure 2 (A) and (B) in our paper provide direct evidence. When examining the top 128 experts ranked by gating scores, the overlap between the Math and Code domains is only 74%. For the top 16 experts, the overlap is almost negligible, with nearly disjoint sets.\n\n**Essential Role of Domain Experts**: Removing Science-specific experts causes GPQA accuracy to drop by **11.82 points**. In contrast, the same pruning leads to only a **2.25-point** decrease on LiveCodeBench, a Code task. This asymmetry highlights that domain experts are critical within their respective domains but have limited transferability.\n\nWays to mitigate it:\n\n● **Mixed-Domain Pruning:** By pruning with mixed-domain data, we can preserve a broader range of important experts. As shown in our paper, it can achieve a great performance preservation on multiple tasks.\n\n● **Task-Aware Expert Activation**: using a lightweight classifier or predictor to identify the domain or task of the input, and then selectively activating the corresponding experts.\n\n## Q2: In line 121, could the authors properly define \"demonstrations\"?\n``Demonstrations'' is a common term in In-Context Learning (ICL). Specifically, it refers to examples or sample inputs that are provided to an LLM. These samples serve as references that guide the model's responses or predictions."}]

Author rebuttal:

Meta-review: This paper presents that with just a few domain-specific examples, MoE models consistently activate a sparse and stable subset of experts ("few-shot expert localization"). Based on this observation, they develop a pruning framework, EASY-EP, that combines output-aware expert importance assessment with expert-level token contribution estimation to identify and retain only the most relevant experts for specific domains.

Strengths:
1. interesting findings.
2. simple and effective methods.

Weaknesses:
1. limited novelty (almost using existing similarity or norm-based measures)
2. the authors should add more baselines as suggested by the reviewers
3. the authors should test their methods on a more model families besides deepseek.
Decision: Accept (poster)