MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
Authors: Geng Zhang
Han Yuxuan
Yuxuan Lou
Yiqi Zhang
Wangbo Zhao
Yang You
Venue: ICLR 2026 Poster (2026)
Keywords: Model Compression
Mixture-of-Experts
Structured Pruning
Expert Pruning
Primary area: foundation or frontier models, including LLMs

Abstract: Mixture-of-Experts (MoE) enables efficient scaling of large language models by activating only a subset of experts per input token.
However, deploying MoE-based models incurs significant memory overhead due to the need to retain all experts in memory. 
While structured pruning is promising to reduce memory costs, existing methods often show suboptimal performance and unstable degradation in three dimensions: model architectures, calibration data sources, and calibration sample sizes.
This paper proposes \textbf{M}ixture-\textbf{o}f-\textbf{N}ovices-and-\textbf{E}xperts (\textbf{MoNE}), a novel expert pruning method that replaces redundant experts with lightweight novices to achieve effective and robust model compression. 
MoNE evaluates expert redundancy based on two metrics: access frequency and output variance. 
Experts exhibiting low usage and stable outputs are pruned and replaced with lightweight novices—unbiased estimations of their original outputs—minimizing performance degradation. 
Extensive experiments demonstrate that MoNE consistently outperforms baseline methods with minimal accuracy degradation across the three dimensions, confirming its effectiveness and robustness. 
Notably, it outperforms baselines by up to 2.72 for the average zero shot accuracy across nine downstream tasks under 25\% pruning ratio, with only 0.14 performance drop for Qwen2-57B-A14B. 
The code is available at \url{https://github.com/zxgx/mode-pd}.

MONE: REPLACING REDUNDANT EXPERTS WITH LIGHTWEIGHT NOVICES FOR STRUCTURED PRUNING OF MOE
Geng Zhang', Yuxuan Han", Yuxuan Lou, Yiqi Zhang, Wangbo Zhao', Yang You National University of Singapore
(zhangg, youy) @comp. nus edu. sg wangbo. [han_ yuxuan, yuxuanlou, yigi zhang fu.nus. edu
com
ABSTRACT
Mixture-of-Experts (MoE) enables efficient scaling of large language models by activating only a subset of experts per input token. However, deploying MoE-bass based models incurs significant memory overhead due the need to retain all experts memory. While structured pruning is promising to reduce memory costs, existing methods often show suboptimal performance and unstable degradation in three dimensions: model architectures, calibration data sources, and calibration sample sizes. This paper proposes expert pruning method that replaces redundant experts with lightweight novices to achieve effective and robust model compression. MoNE evaluates expert re- dundancy based two metrics: access frequency and output variance. Experts exhibiting low usage and stable outputs are pruned and replaced with lightweight novices-unbiasee estimations of their original outputs- minimmiziiiiiiggggg performance degradation. Extensive experiments demonstrate that MoNE consistently out- performs bascline methods with minimal accuracy degradation across the three dimensions, confirming its effectiveness and robustness. Notably, it outperforms baselines by up to 2.72 for the average zero shot accuracy across nine downstream tasks under 25% pruning ratio, with only 0.14 performance drop for Qwen2-57B: A A14B. The code is available at https: / / thub. com/ zxzzzxxx/o/ooo pd.
Experts (MoNE), a novel
INTRODUCTION
Ketture-of-Epperet (MoE) has emerged as a powerful architecture for advancing the capabilities of large language models (LLMs) (Liu et al., 2024; 2025; Muennighoff et al., 2025). MoE-based LLMs achieve higher parameter efficiency than vanilla transformer-basee LLMs by replacing the MLP module with a set of smaller MLP modules (experts) and sparsely activating partial experts for each input token (Lepikhin al., 2021). Despite performance benefits, the deployment of MoEebased models often incur additional memory overhead to maintain the non-activated experts in memory, which is valuable but limited for existing accelerators such as GPU and TPU (Jouppi et al., 2023).
While diverse structured pruning methods have been proposed to reduce deployment memory costs by removing different model components while minimizing the performance degradation (Voita et al., 2019; He et al., 2024; Xia et al., 2024; Zhao al., 2026; 2025a;b), approaches often exhibit suboptimal performance and unstable degradation when applied to different MoE models. Specifically, we identify three critical dimensions where existing methods fall short: model architectures, calibration data sources and calibration sample sizes, as shown by experiments in Section 5.3. These limitations evident across two main categories of structured pruning approaches for MoE models: general structured pruning and expert pruning as shown in Figure (a). First, general structured pruning methods that remove model layers (Angular (Gromov al., 2025)) or weight matrix channels (FLAP (An et al., 2024)) fail to account for the sparse computation scheme of MoE models when evaluating model component importance, resulting in inconsistent performance drop across the aforementioned three dimensions. Second, existing expert pruning
observe that these
"Equal contribution. 'Corresponding author.

---

pruning
pruning
Layer
Weight
Anguar
FLAP
Tranaromme Layer
(a)
Par
Ar
Aar
Transfomerr
Weight
mabrix
Transfomert
Layer
Aroce Aroe
MMLU
Winsgrande
Frequency
Layers
06
(b)
Weltnne
Layo7s
02
Esports
Experts
Figure 1: (a) Different structured pruning methods. (b) Layer-wise normalized expert access frequency and output variance of Deepseck- Deepseck-V2-Lite V2-Lite for three downstream tasks. Experts with high access frequency or output variances are the same across downstream tasks. Expert in blue circles has both high frequency and variance. Expert in red circles only has high variance. Expert in green circles only has high frequency. Similar observations on other models and tasks are in Appendix D.
methods such as MC-SMoE (Li et al., 2024) and RS (He et al., 2024) remove experts from MoE models primarily based the expert access frequency. However, shown in Figure (b), this feature alone fails to fully capture the expert redundancy. Besides, these methods lack mechanisms to recover the performance loss caused by pruning.
To improve the effectiveness and robustness of structured pruning for MoE models, this paper proposes novel expert pruning method, -nxture-f--Novices--ndd Eupdd--xetts (MoNE) which replaces redundant experts with a lightweight structure, novice. Specifically, to prune an MoE model with MoNE, it first evaluates the expert redundancy by the access frequency and the output variance for each expert calibration dataset. Then, it identifies and prunes redundant experts that show low access frequency and stable output activations to reduce memory overhead from redundant experts. Finally, the unbiased estimation of the pruned expert output is employed as the lightweight novice to reclaim the performance loss caused by the pruned expert. The intuition behind MoNE that experts with low access frequency contribute less to the final outputs and experts whose outputs have low variance can be replaced with a constant but introduce less discrepancy. Moreover, Figure (b) reveals that experts with less redundancy identified by MoNE exhibit strong consistency across various downstream tasks.
The contribution of this paper is summarized follows:
We propose novel expert pruning method named MoNE which replaces redundant experts with lightweight novices to compress MoE models with minimal performance loss.
We exploit the expert access frequency and output variance to measure the expert redundancy and employ the unbiased estimation of the expert output to minimize the output discrepancy after pruning, thus achieving effective and robust pruning results.
Extensive experiment results demonstrate that MoNE consistently outperforms baseline methods under varying MoE architectures, calibration data sources and calibration sample sizes. Notably, it outperforms baselines by up to 2.72 for the average shot accuracy across nine downstream tasks under 25% pruning ratio, with only 0. performance drop for Qwen2-57B-A114. -1BB..
2

---

2
RELATED WORK
Model pruning compresses a model by removing certain redundant model parameters while preserving accuracy. Existing pruning methods generally fall into two categories: unstructured pruming and structured pruning. Unstructured pruning eliminates any model parameter that has minimal impact on model performance. Methods such as SparseGPT (Frantar & Alistarh, 2023), Wanda (Sun et al., 2024), SparseLLM (Bai et al., 2024) excel in maintaining accuracy while achieving high compression ratios. However, the resulting irregular sparsity patterns hinder efficient representation and execution on hardware accelerators.
In contrast, structured pruning removes certain modules of model, preserving hardware-friendly structures. Early researches prune redundant transformer layers of a LLM (Fan et al., 2020; Ling et al., 2024; Gromov et al., 2025). LLM-Pruner (Ma al., 2023), FLAP (An et al., 2024), MoE-Pruner (Xie et al., 2024) and SlimMoE (Li et al., 2025) remove rows or columns of individual weight matrices. Recent work also proposes to delete components such as attention, MLP or MoE modules within each transformer layer (Voita et al., 2019; He et al., 2024). Minitron (Muralidharan et al., 2024) and Sheared LLaMA (Xia et al., 2024) combine different granularity and automatically search for the optimal structures to prune. Despite their versatility, existing structured pruning methods often exhibit inconsistent performance across MoE architectures.
This work focuses on expert pruning, unique direction of structured pruning for MoE models (Lepikhin et al., 2021; Liu et al., 2024). Expert pruning targets on deleting individual experts for each layer to compress MoE model. Previous expert pruning methods either require exhaustive search to identify redundant experts (Lu et al., 2024), or heavily rely on retraining to recover accuracy due to the suboptimal pruning performance (Li et al., 2024; He et al., 2024). However, the exhaustive search is not applicable to modern MoE model architectures such as Deepseek (DeepSeek-Al 2024; Liu al., 2025), OLMoE (Muennighoff- et al., 2025) Qwen (Qwen, 2024; Yang et al., 2025), their MoF. layer contains 64 experts or even more, yielding a tremendous search space that is intractable. Retraining obscures the advantages of expert pruning over other structured pruning methods.
3 PRELIMINARIES
3.1 MIXTURE-OF-EXPERE (MoF)
MoE-based LLMs replace the traditional MLP module in the transformer layer with MoE module. Each MoE module consists of router network G and set of experts & = (F1, Fg..... FM J. where M is the number of experts and each expert is smaller MLP. Let E Rd be the hidden state of an input token, where d is the hidden size of the model, the output of an MoE module is computed as:
G,E)
Σ Gx(x). (x)
(1)
E,ESLA
The output of the router network G(x) E M represents the routing scores for all experts, and SR. C & denotes the top i experts with the highest routing scores for input The final output of the MoE module is the weighted sum of outputs from the top k experts. While Equation 1 captures the general MoE computation, implementations for G and E, may vary across model architectures (De, 2024; Liu et al., 2024; Muennighoff et al., 2025; Liu et al., 2025).
3.2 EXPERT PRUNING FORMULATION
Previous studies have revealed that not all experts contribute equally, and pruning less important ones can reduce memory overhead with marginal performance degradation (Lu et al., 2024; Li et al., 2024; Huang et al., 2025). However, searching for the target experts prune the global model perspective falls into tremendous search space, as the number of experts per transformer layer increases with the evolving of the MoE model architectures (Lepikhin al., 2021; Jiang et al., 2024; DeepSeek-AI, 2024; Liu et al., 2024). Following the layer-wise pruning scheme (Frantar & Alistarh, 2023; An et al., 2024; Lu et al., 2024; Ling et al., 2024), our goal of expert pruning is to identify a subset of redundant
experts P C m such that we can minimize the output difference after compressing their parameters:
min min||MoE(Xx, G,E PP) MoE(x, G,E)|l2 PCE
(2)
3

---

Ansragel cutpuz
Siitue of Newioas and Experts
M Enoula
Curput
Conntruding
Zmppe
6 0 .gt
Enpotrukrekeey
PG
Av
A
Gxi
Car
The overview of MoNE, Given MoE model, it first exploits calibration dataset to evaluate the expert access frequency and output variance. Then, the two metrics are fused to get the expert redundancy. Finally, the novices are derived from the averaged outputs for redundant experts.
Figure 2:
To achieve this goal, the core problem is twofold: (I) find metric evaluate the importance of the experts in each layer, so that we can identify the expert subset P. and (2) find an pruning method to compress the parameters of P, that removing redundant experts (He et al., 2024; Lu et al., 2024), existing methods have also explored expert merging to mitigate the expert redundancy (Li al., 2024).
reduce the model size. While 3 \ P implies directly
4 MIXTURE OF NOVICES AND EXPERTS
This section introduces Mixture-of-Ne wtaprroo--ooviiee-a-------e (MoNE), a novel expert pruning method designed to achieve effective and robust compression for MoE models while minimizing performance degradation. Section 4.1 presents the computational framework of MoNE, Section 4.2 defines the metric to evaluate the redundancy experts. Section 4.3 explains the pruning process that compresses the redundant experts to lightweight novices. The overview of MoNE is depicted in Figure 2.
4.1 MONE FRAMEWORK
MoE models are often trained with auxiliary losses to ensure load balance among experts in each layer, enabling each expert learn certain aspect of knowledge (Lepikhin et al., 2021; DeepSeek-AI, 2024; Muennighoff et al., 2025). However, most existing expert pruning methods directly remove experts (He al., 2024; Li al., 2024), often leading inconsistent performance drops across different model architectures or calibration data. MoNE addresses this issue by introducing lightweight structures called novices to replace the pruned experts. novice is designed capture the essential knowledge previously held by the removed experts. In contrast to simply removing redundant experts, MoNE compensates for knowledge loss by leveraging novices, thereby preserving the overall performance of the model while maintaining compression efficiency. Specifically, the output of the MoNE is computed as:
MoNE(x)
G2(x) E((x) + E.
Σ
Σ E
(x(x)-M)
(3)
where SE, P and Sa, n denote the preserved and pruned experts among the top & activated experts respectively. N₂ TRd is the novice that retains the essential knowledge of the pruned expert. Notably, N, is a compressed vector that does not involve any computation with the input token As result, the computation and memory overhead nearly identical to directly removing experts. Furthermore, replacing experts with novices introduces adaptive computation overhead for different tokens, leading to fewer activated parameters for tokens routed to novices. Nevertheless, empirical results in Section 5.2 demonstrate that MoE models pruned by MoNE maintain more zero shot performance on downstream tasks compared to existing expert pruning methods that only remove experts but keep the same activated parameters.
4

---

4.2 EXPERT REDUNDANCY EVALUATION
To identify the expert subset P, we introduce an expert redundancy score e to assess the redundancy of experts. To ensure the pruned experts contribute minimally to the model's overall performance, the expert redundancy score & takes two aspects into consideration: the variance in an expert's output across a calibration dataset C, and the frequency of an expert selected by the router network G.
Variance-based redundancy As the novices are constant vectors to ensure reduced computation and memory overhead, the outputs of the pruned experts are expected to have low variance across calibration dataset C. In other words, experts with high output variance should be retained to contribute more discriminative information during inference, whereas experts with low output variance could be compressed into more efficient representation. i.e., novice. The second row of Figure I (b) visualizes this motivation. Expert outputs exhibit diverse variances, but we can find experts in blue and red circles that maintain high variances across different downstream tasks. Therefore, we introduce a variance- based redundancy 6,"h to measure the output variance for expert E, Concretely, ofar is the L2 norm of the unbiased estimation for the output variance:
Σ (x) E1)² .(EE E Skx)
eaa
(4)
Exec I(E, E Sax) -1
Exec Ex(x) I(Ei E Sp,x)
E,
(5)
Exec I(E, E Sk,))
where I(E; E Sk.) is the indicator function to show whether E; is among top k experts for the input token x of the calibration dataset
Frequency-basee redundancy The routing scores and access frequencies of the router network G serve as strong indicators of the overall redundancy of an expert (He et al., 2024; Li et al.. 2024). Intuitively, experts which are rarely selected or consistently assigned lower routing scores are likely to have minimal impact on the model's output. As shown Figure/ (b), we can identify typical experts in blue and green circles that show consistent high frequency over the three downstream tasks. Notably, the expert in green circles only has high frequency. Therefore, the frequency and variance information can complement the discrepancy ignored by each other. Based on this observation, we define the frequency bbsedd redundancy of ofreq of the expert E, the average routing score across calibration dataset C of which F.; is among the top ls selected experts. Formally, the rreuuencybbase redundancy of freq is defined as:
of rreq Exec Gi(x) I(E, E Sux)
(6)
Exec E Skx)
Finally, the two redundancy metrics are fused to obtain the expert redundancy score Φ:
Ser drey
(7)
lower expert redundancy score d; indicates higher redundancy for expert E,, making a suitable candidate for pruning and replacement with novice N,.
4.3 EXPERT REPLACEMENT WITH NOVICE
After identifying the pruned expert subset P, we need to construct lightweight novices to replace them. According to Equation 2, the general objective for expert pruning is to minimize the discrepancy introduced by the removed expert outputs. Since the output after applying MoNE is formulated as Equation 3, the concrete objective for MoNE can be translated to:
min P.EP
> -Nll2)
(8)
XEC
Because N₂ is a constant vector, the optimal novice vector N, that best approximates the output of pruned expert E, can be obtained in a closed form, i.e., E, in Equation 5.
To sum up. MoNE uses the unbiased estimations of mean expert outputs to replace experts that have the minimum output variance. As a result, MoNE achieve the goal that effectively and robustly compresses the MoF experts while minimizing performance degradation.
5

---

5 EVALUATION
5.1 EXPERIMENT SETUP
Base MoE models To validate the effectiveness and robustness of MoNE, we conducted structured pruning on five open source MoE models with diverse architectures and model scales: OLMoE (Muennighoff et al., 2025), Moonlight (Liu et al., 2025), DeepSeek-V2-4 V2-Lite (DeepSeek-Al, 2024), Qwen2-57B-A14B (Qwen, 2024) and Qwen3-30B-A3B (Yang et al., 2025). OLMoE has 7B parameters with IB activated parameters per token. Both Moonlight and Deepseck- V2-Lite have 16B parameters with 3B activated pamaters per token. OLMoE and Moonlight represent SOTA MoE models at their respective scales. To demonstrate scalability to larger architectures, we also consider the Qwen series: Qwen3-30B-A3B with 30B parameters and 3B activated per token, and Qwen2-57B-A14B with 57B parameters and 14B activated per token. We chose the base version of the five models for experiments.
Baseline methods We selected structured pruning methods for different structures as baseline. Notably, unless explicitly stated, we did not apply any weight update to compare the effect of pruning methods. Specifically, for general structured pruning methods, we used Angular for layer pruning (Gromov al., 2025), which evaluates the layer importance by the angular distance between the input activations for different layers, and we used FLAP for weight pruning (An et al., 2024), which evaluates the channel importance by the fluctuation of the input activations and compensates the performance loss with the averaged output activations. For expert pruning methods, we adopted the expert merging method in MC-SMoE (Li et al., 2024) for one of the expert pruning baselines. Another expert pruning baseline is RS (He et al., 2024), which uses routing scores to evaluate the expert importance and discards less accessed ones.
Implementation details We tested two pruning ratios: 25% and 50%. To demonstrate the ro- bustness of MoNE to calibration data, we conducted experiments two calibration data sources: Zyda2 (Tokpanov et al., 2024) and C4 (Raffel et al., 2020). Both datasets are constructed for LLM pretraining and C4 is commonly used the calibration dataset for model compression (Ling et al., 2024; Frantar & Alistarh, 2023; Gromov et al., 2025; Xia et al., 2024). Besides, we also investigated the performance under three calibration sample sizes: 100, 500 and 1000 in Section 5.3.
Evaluation protocol Following previous researches (Ma al., 2023; Bai et al., 2024; Ling et al., 2024; Xia et al., 2024; An et al., 2024), we adopted im-evaluation-harness (Gao et al., 2024) to measure the zero shot accuracy and average results nine downstream tasks: Arc-c and Arc-e (Clark et al., 2018), BoolQ (Clark et al., 2019), COPA (Roemmele et al., 2011), MMLU (Hendrycks al., 2021), OBQA (Mihaylov et al., 2018). PIQA (Bisk et al., 2020), RTE (Wang et al., 2019) and Winogrande (Sakaguchi et al., 2021).
Though more complex downstream tasks such as coding (Zhang et al., 2025), math (Lightman et al., 2023) or reasoning (Lin et al., 2025) exist, these tasks are still challenging for full LLMs ( (Yang et al., 2025; Liu et al., 2024). Moreover, existing study shows that model compression for complex tasks often requires additional task-specific fine-tuning (Sarkar et al., 2024; Chen et al., 2022), which is beyond the scope of this work and we consider a promising direction of future work.
5.2 EFFECTIVENESS EVALUATION
This section validates the effectiveness of MoNE by comparing the zero shot performance of 25% pruned models with 100 calibration samples from the Zyda2 dataset. The results are presented in Table 1 and Table 2. Results under 50% pruning ratio are extended to Table 10 in Appendix C.
Table indicates that MoNE consistently outperforms baseline methods in terms of the average accuracy on the nine tasks. In particular, it shows average accuracy improvement as large as 2.72 for the pruned OLMoE compared to baseline methods, and it incurs accuracy drop as small as only 0.14 for the pruned Qwen2-5t 577--144 Furthermore, MoNE- --Nuned models can achieve either the best or the second best result for individual tasks under most settings.
'https:/ //gi cow.b/oom/b.....mmmmom// www.///////////./aaaa //1 m-eval uaa on-harness

---

Table Zero shot performance with 100 calibration samples from Zyda2 dataset. Best results are in bold, and the second best are underlined.
Green cells indicate results no less than original models.
(a) OLMoB
Pruning ratio
Model/Method
Aze-c
Are-e
BoolQ
COPA
MMLU
OBQA
PIQA
RTE
Winogrande
Avg.
0%
OLMoE
49.23
76.89
70.09
85.00
53.54
44,40
79.76
71.84
68.90
66.63
Angular
32.76
61.91
61.71
74,00
37.60
71.65
53.07
55.09
FLAP
40.53
67.55
62.69
78.00
41.16
37.80
74.81
61.37
60.93
58.32
25%
MC-SMa SMolE
35.67
54.92
63.49
73.00
29.04
30.60
67.19
55.23
65.75
52.77
RS
25.85
43.01
59.08
74.00
29.63
36,20
66.16
56.68
59.98
50.07
MoNE (Ours)
42.32
64.81
67.19
85.00
40.13
40.80
78.07
64.62
66.46
61.04
(b) Moonlight
Pruning ralio
Madel/Method
Arc
Arcie
BoolQ
COPA
MMLU
OBQA
PIQA
RTE
Winogrande
Avg.
0%
Moonlight
58.28
82.49
80,40
92.00
67,30
45.60
81.12
65.70
71.11
71,56
Angular
39.76
$2.69
38.90
79.00
42.57
32.20
68.50
61.01
62.04
52.96
48.55
77.97
64.26
25%
MC-SMoF
47.61
73.
78.72
89.00
46.11
43.60
80.36
56.32
71.43
65. 14
80.64
78.69
46.40
81.01
72.30
MoNE (Ours)
55.89
80.60
79.57
90.00
55.23
46.80
80.85
61.01
71.98
69.10
(c) Deepseek- V2-Lite
Pruning ratio
Model/Method
Are-c
Arc-e
BoolQ
COPA
MMLU
OBQA
PIQA
Winogrande
Avg.
Deepseck- i2-L
48.72
76.18
79.88
88.00
54.96
43.60
80.25
61,37
71.51
67.16
Angular
32,00
53.28
64.92
75.00
26.95
34.00
71.33
58.84
61.01
$3.04
FLAP
43.69
71.46
75.26
84.00
47.28
78.18
67.72
63.53
25%
MC-SMoF.
36.69
60.77
71.31
84.00
42.22
36.60
75.57
58.48
68.67
59.37
McNE (Ours)
46.67
74.62
78.47
90.00
49.05
43.00
79.76
62.09
71.43
66.12
(d) Qwen2-5785 Srrccn2-5B-AA1 17BBA14B
Pruning
Model/Method
BoolQ
MMLU
PIQA
Winogrande
Avg.
09%
Qwen2-57B-A14B
49.66
69.44
86.45
93.00
74.06
44.30
81.23
74.73
74.27
71.89
Angular
29.44
54.17
59.51
70.00
23.92
32.80
70.02
54.87
49.57
49.37
FLAP
72.85
91.00
65.02
45.40
81.12
77.62
76.09
71.78
25%
MC-SMoF
46.67
66.25
86.45
88.00
69:46
43.40
80.20
74.73
25.14
70.03
RS
49.15
69.78
84.77
87.00
70.99
44.80
81.34
74.37
74.19
70.71
MoNE (Ours)
49.66
68.73
86.88
94.00
71.64
45.20
81.07
75.45
73.09
21.75
(c) Qwen3-30B-Br
Pruning
Model/Method
Areie
Arc-e
BoolQ
COPA
MMLU
OBQA
PIQA
RTE
Winogrande
Avg.
0%
Owen3-30B-AA3
55.89
79.42
88.69
84.00
77.82
44,80
80,30
82.31
70.64
73.76
Angular
44.62
80.52
77.00
59.75
40.40
75.30
70.40
62.51
64.34
FLAP
50.85
76.68
85.72
85,00
69.43
42.80
77.31
81.95
70.17
71.10
25%
MC-SMeE
52.73
76.98
88.75
83.00
72.25
44.40
79.71
80.87
70.40
72.12
RS
53.75
78.32
88.53
85.00
74.60
43.00
79.92
79.78
09.61
72.50
MoNE (Ours)
43.00
Table 2: Maximum pruning ratios with 1 % accuracy loss after MoNE pruning using 100 calibration samples from Zyda-2 dataset.
Model
Max. pruning ratio
Avg. perf before pruning
Avg. perf after pruning
OLMoE
16%
66.63
66.00
Moonlight
16%
71.56
70.59
Deepseek- V2-Lite
20%
67.16
66.31
Qwen2-57B- 57B-A14B
25%
71.89
71.75
Owen3-30B-ANe
24%
73.76
73.61
An interesting observation is that all the three expert pruning methods, MC-SMoE, RS and MoNE achieve results par with even better than the original models certain tasks. The specific examples are shown in green background in Table 1. All these results indicate that there is indeed redundancy existing in the expert level for the examined MoE models, and expert pruning rule
7

---

Varying model architecture
Varing callbration data source
Varying callbration sample size
Arg scuuraty bendhmarks
65
MoNE
MONE
MoNE
RS
80-
RS
FLAP
RS
MC-SMoE
MC-SMcE
MC-SMGE
65-
MoNE (Dunx)
MoNE (Gura)
MoNE (Ours)
Angular
General pruniing
General
Geneal pruning
prining
Angular
Espaet peining
Angular
Eapert pouning
50
Pareto Frontier
Pareto Frontier
Fareto Fronter
3
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
accurary drop
socuracy drop
of acarracy drop
(a) 25% pruning ratio.
Varying model architecture
Varing calimation data source
Varying calibration sample size
Aug accuscy over 9 berch
55
MoNE
MoNE
MoNE
50-
FLAP
RS
FLAP
MC- NM0-SMkE,,
MC- M
MC-SMcE
MONE (Ours)
MoNE (Cu'st
MoNE (Curs)
jeuning
prating
praning
80-
Angular
Expert pruning
Parelo Frontar
Angular
Expart prusing
Panelo Fronter
Angular
Expert pouning
Parelo Frontier
2
3
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Std varfence accuracy drop
Std varlanee accuracy drop
Stdd varlanos of accuracy drop
(b) 50% pruning ratio.
Figure 3: Average accuracy versus accuracy drop variance. MoNE advances the Pareto frontier across varying model architectures, calibration data sources and calibration sample sizes.
out such redundancy to achieve even better results on these tasks. Among the three expert pruning methods, MoNE consistently surpasses the two baseline methods, demonstrating its strong capability,
Table 2 evaluates the maximum pruning ratios that maintain accuracy loss within 1% across different MoE models. As shown in Table 2, larger and more powerful models such as Qwen2-57B-A 14B and Qwen3-30B-AB tolerate more aggressive pruming (25% and 24%, respectively) compared to smaller models like OLMoE, Moonlight, and Veepseck-V2-LLig (16-20%). This finding highlights the scalability of proposed MoNE method for large model scales. In addition, combining the observations from Table 1 and Table 2, argue that larger MoE models may have stronger capability in language modeling but also contain increasing parameter redundancy at the expert level, and MoNE can efficiently eliminates such redundancy at minimal performance degradation.
5.3 ROBUSTNESS EVALUATION
This section evaluates the robustness of MoNE across three key dimensions: model architecture, calibration data source, and calibration sample size. Due to prohibitive compute for exhaustive experiments on large models, we tested three models: OLMoE, Moonlight and Deepseek- using 100, 500 and 1000 calibration samples on C4 and Zyda-2 dataset separately. For each dimension, vary one factor while averaging results over the other two, measuring both average accuracy and the standard deviation of accuracy drop. The results are visualized in Figure 3, with detailed scores provided in Appendix C. As shown in Figure 3a, MoNE advances the Pareto frontier across all three dimensions at the 25% % pruming ratio, demonstrating superior robustness and effectiveness compared to existing structured pruning methods. At the 50% pruning ratio (Figure 3b), MoNE exhibits slightly higher variance under varying model architectures and calibration sample sizes. Nevertheless, it remains the most effective method, outperforming baseline methods by significant margin of 2.85.
5.4 ABLATION STUDY
This section presents the ablation study to evaluate the effects of the two redundancy metrics and the impact of novice replacement across the downstream tasks. Figure 4 displays the average accuracy drop relative to our proposed methods, with lower values indicating greater degradation. Results averaged over the three evaluation dimensions provide robust assessment. We observe that integrating the fused expert redundancy score with novice replacement yiclds better performance, particularly under higher pruning ratios. This indicates that our approach is especially effective in
8

---

Pruning Mortic
Novice
Prunig
Replasement
100
1.30
0.45
0.43
0'07
133
9:87
0.67
1.41
1.87
1.23
7.5
Fabe
1.54
7:31
3.15
5.0
2.5
1.00
4.40
4.86 0..7
4.23
4.40
0.
18 -1.51 2.08 098 -.20 0.08 -144 -684
-1.50
5:04
-1.2
-2.00
-1.00
-25
-50
1157
-1.4 2.00
211 237 1.21 4.00
4.84
+148
4.11
4
80
-0.33
045
-1.3
-2.0
-1.00
-0.0
-76
AYE
Adc
100
Figure 4: Ablation study expert access fréquency, output variance and novice replacement. Numbers are the difference to the proposed MoNE. The detailed result is provided in Appendix E.
Table 3: Zero shot performance of the 25% pruned OLMoE after continued pretraining with 2B tokens from
Best results are bold, and the second best are underlined.
Model/Method
Arc-c
Arc-c
BoolQ
COPA
MMLU
OBQA
PIQA
RTE
Winogrande
Average
OLMoE
49.23
76.89
70.09
85.00
53.54
44.40
79.76
71.84
68.90
66.63
Angular
38.82
64.69
63.52
82.00
25.42
39.80
76.50
51.62
59.04
55.71
FLAP
42.24
69.07
69.51
80.00
45.56
40.40
77.42
50.18
63.54
59.77
MC-SMoE
42.75
70.41
69.76
80.00
44.13
37.60
75.79
66.43
64.96
61.31
RS
44.97
72.94
70.73
85.00
43.28
43.00
78.67
72.20
65.98
64.09
MoNE (Ours)
47.35
74.33
71.56
87.00
43.30
40.40
78.89
67.51
67.25
64.18
preserving model quality when pruning more aggressive. Notably, for tasks such BoolQ, COPA, and PIQA, our proposed method outperforms the ablation baselines by large accuracy gains of up 8.85. However, for MMLU, pruning based solely on frequency appears to offer a slight advantage, suggesting that frequently activated experts may play more critical role in domain-specific reasoning tasks.
5.5 ACCURACY RECOVERY WITH CONTINUED PRETRAINING
To evaluate performance recovery capabilities, we conducted continued pretraining on the 25% compressed OLMoE model pruned by 100 Zyda2 samples, as only this model releases its pretraining dataset, OLMoo--mi--0924 The sequence length was set to 4096 and the global token size per step was 4M. Each pruned model was trained with 2B tokens, i.e., 512 steps, and the peak and minimum learning rate (Ir) were 5e-5 and 5e-6, respectively. We employed the cosine Ir scheduler with 50 warm up steps. Other hyperparameters were the same as the original configuration for OLMoE (Muennighoff et al., 2025). All the experiments could run on single H20 GPU, but we accelerated the training with 16 H20 GPUs.
The results summarized in Table 3. This table shows that MoNE achieves the average accuracy closest to the original model with only 2B tokens from a pretraining dataset, demonstrating the promising capability of the MoNE computation framework. Besides, MC-SMoE and RS reclaim 8.54 and 14.02 average accuracy, indicating that expert pruning not only effective to eliminate redundancy, but also relatively easier to recover performance with continued pretraining.
CONCLUSION
In this paper, we propose MoNE, novel expert pruning method that replaces redundant experts with lightweight novices to compress MoE models. MoNE evaluates expert redundancy based on expert access frequency and output variance in each model layer, pruning experts with low usage and stable outputs while replacing them with novices that provide unbiased output estimates. Extensive experiments across different MoE architectures, calibration data sources, and sample sizes demonstrate that MoNE outperforms existing structured pruning methods by maintaining higher zero-shot performance across nine downstream tasks.
Phtrps:s/hugiiiggffa.ccom
ennai MOM.MO------ 0924

---

ACKNOWLEDGMENT
We would like to acknowledge that computational work involved in this research work is supported by NUS IT's Research Computing group using grant numbers CFP02-CF-004, Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDance grant. NUS ARTIC grant, Apple grant, Alibaba grant, Google Research and Google grant for TPU usage.
REFERENCES
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation- based adaptive structured pruning for large language models. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pp. 10865-10873, 2024.
Guangji Bai, Yijiang Li, Chen Ling, Kibaek Kim, and Liang Zhao. SparseLLM: Towards global pruning of pre-trained language models. In The Thirly-eighth Annual Conference on Neural Information Processing Systemx, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. Task-specific expert pruning for sparse mixture-of-expeee arXiv preprint arXiv:2206.00277, 2022.
Christopher Clark, Kenton Lee, Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq:
Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2934-2936, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: (rXXiv:1803.00 11/83.00557, 2018.
DeepSeek-AL, Deepseek-v2. A strong, economical, and efficient mixture-of-expprts language model, 2024.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. In International Conference on Learning Representations, 2020.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10333-10337. PMLR, 2023.
Elias Frantar, Roberto L Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. Marlin: Mixed-
precision auto-regressive parallel inference on large language models. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, pp- 239-251, 2025.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Dan Roberts. The unreasonable ineffectiveness of the deeper layers. In The Thirteenth International Conference on Learning Representations, 2025.
Shwai He, Daize Dong, Liang Ding, and Ang Li. Demystifying the compression of mixture-of-exppee through unified framework. arXiv preprint arXiv:2406.0223 025000 2024.
10

---

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++: Faster large language model inference on gpus. arXiv preprint arXiv:2311.01282, 2023.
Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and XIAOJUAN QI Mixture compressor for mixture-of-experts LLMs gains more. In The Thirteenth International Conference on Learning Representations, 2025.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.00 040088, 2024.
Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, et al. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In Proceedings of the 50th annual international symposium on computer architecture, pp. 1-14, 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. (GS] hardd: Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Representa- tions, 2021.
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient SMoe with hints from its routing policy, In The Twelfth International Conference on Learning Representations, 2024.
Zichong Li, Chen Liang, Zixuan Zhang, Ilgee Hong. Young Jin Kim, Weizhu Chen, and Tuo Zhao. Slimmoe: Structured compression of large moe models via expert slimming and distillation. arXiv preprint arXiv:2506. 18349, 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogic: On the scaling limits of llms for logical reasoning. arXiv preprint arXiv:2502.01 100, 2025.
Gui Ling, Ziyang Wang, and Qingwen Liu. Slimgpt: Layer-wise structured pruning for large language models. Advances in Neural Information Processing Systems, 17:107112-1077 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint MtXiv:2412.194377 2024.
Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for Il training. arXiv preprint ar arXiv:2502.16982. 2025.
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of offeurrr---exerrts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 6159-6172, 2024.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. 1lm-rrune: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720 2023.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2381-2397 2018.
11

---

Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Evan Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. OLMoe: Open mixture-of-experts language models. In The Thirteenth International Conference Learning Representations, 2025.
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact lan- guage models via pruning and knowledge distillation. Advances in Neural Information Processing Systems, 37:41076-41 102, 2024.
Team Qwen. Qwen2 technical report. arXiv preprint arXiv:2407.1067/ 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 202.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pp. 90-95, 2011.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106,
2021.
Soumajyoti Sarkar, Leonard Lausen, Volkan Cevher, Sheng Zha, Thomas Brox, and George Karypis. Revisiting smoe language models by evaluating inefficiencies with task specific expert pruning. arXiv preprint vrXiv:2409.01490 2024.
Mingjee Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference Learning Representations, 2024.
Yury Tokpanov, Paolo Glorioso, Ayush Dattagupta, Vibhu Jawa, Ryan Wolf, Vikranth Jeyakumar, Arham Mehta, Quentin Anthony, and Beren Millidge. Building Zyda-2, 5 Trillion Token High-Quality Dataset, with NVIDIA NeMo Curator, October 2024.
Flena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Anna Korhonen, David Traum, and Lluís Marquez (eds.), Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 5797-5808, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10.18653/v//19-111
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances neural information processing systems, 32, 2019.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference Learning Representations, 2024.
Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. Moe-pruner: Pruning mixture-of-expers large language model using the hints from its router. arXiv preprint arXiv:2410.12013, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu L.v, et al. Qwen3 technical report. arXiv preprint arXiv:2505.00 093888 2025.
Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang. Yufan Huang, Shengyu Fu, et al. Swe-bench goes live! arXiv preprint arXiv:2505.23419, 2025.
12

---

Wangbo Zhao. Yizeng Han, Jiasheng Tang, Kai Wang, Yibing Song, Gao Huang, Fan Wang, and Yang You. Dynamic diffusion transformer. 2025a.
Wangbo Zhao, Yizeng Han, Zhiwei Tang, Jiasheng Tang, Pengfei Zhou, Kai Wang, Bohan Zhuang, Zhangyang Wang, Fan Wang, and Yang You. Rapid 3: Tri-level reinforced acceleration policies for diffusion transformer. arXiv preprini urXiv:2509.22323 2025b.
Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Hao Luo, Yibing Song, Gao Huang, Fan Wang, and Yang You. Dydit++: Diffusion transformers with timestep and spatial dynamics for efficient visual generation. IEEE Transactions Puttern Analysis and Machine Intelligence, 2026.
13

---

THE USE OF LARGE LANGUAGE MODELS
All writing, visualizations, and experiments are completed by the authors. LLMs (e.g., Claude) are used solely to refine the writing.
We organize our appendix as follows:
Section shows the evaluation MoNE on domain-specific tasks.
Section presents ablation study on the pruning metric.
Section C provides detailed scores for various models, calibration datasets and calibration sample sizes.
Section visualizes expert access frequency and output variance for more tasks and models.
Section presents the detailed scores for ablation study.
Section F shows the inference latency and memory overhead benefits for the proposed MoNE.
14

---

Table 4: Base and Instruct OLMoE models on Math and GSM8K.
Model
GSM8K
Math
No calibration
OLMoE
52.77 67.63
15.78
OLMoE-Instruee
18.64
Calibration: Zyda2
OLMoE
3.34
1.94
GLMoE-IIntrune
7.96
2.54
Calibration: Task-specific
OLMoE-Instruet (Math)
65.28 67.48
18.22
OLMoF-IInstruct (GSM8K)
17.64
Table 5: Base and Instruct Moonlight models on Math and GSM8K.
Model
GSM8K
Math
No calibration
Moonlight
74.22 77.03
42.32 39.26
Moonlight-Instruct
Calibration: Zyda2
Moonlight
46.40
3.32
Moonlight-Instruet
51.86
6.64
Calibration: Task-specific
Moonlight-Instructt (Math)
75.28 76.72
38.56
Moonlight-Instruet (GSM8K)
35.84
Table 6: Base and Instruct DeepSeek-V2-H models on Math and GSM8K.
Model
GSM8K
Math
No calibration
VeepSeee--V-LLtee
38.82
16.54
66.49
18.56
Calibration: Zyda2
DeepSeck- CeapSeck--V-Litte
25.40
6.80
DeepSeek V2-Lite-Chat
37.07
4.76
Calibration: Task-specific
DeepSeek-V22 B--Bsckk--V--iee-ee V2-Lite-Cha (Math)
64.44
20.76
DeepSeek- Baieee-----LLL V2--Lte--LCiee (GSM8K)
63.76
19.70
A SPECIALIZED TASK EVALUATION
We extended MoNE to two specialized tasks: Math and GSM8K. We conducted experiments both base models and instruct models. To calibrate the instruct models, we adopted the first 100 samples from the training dataset of each task. The results are reported in Tables 4-8.
Our experiments on specialized tasks reveal two important findings regarding the effectiveness of different calibration strategies. After pruning with pretraining data (Zyda2), larger base models preserve significantly more accuracy, which consistent with our observations from general tasks in Section 5.2. However, pretraining data cannot accurately capture the distribution of specialized tasks, leading to accuracy drops of up to 49% for the smallest model, OLMoE. This demonstrates that while model scale provides some resilience pruning, the mismatch between pretraining data and
15

---

Table 7: Base and Instruct Qwen2-57B-A14B models Math and GSM8K.
Model
GSM8K
Math
No calibration
Qwen2-57BB 515-577BB1444
79.68 69.90
40.50
10n02-57B-A14B-BA
31.30
Calibration: Zyda2
Qwen2-576 10002--77-A14---
74.07
31.28
Qwen2- Qwen2-57B-A4 17B-A44B A14B-Instruct
64.90
22.84
Calibration: Task-specific
Qwen2-57B-A 4B-Innttruct (Math)
58.45
29.56
Qwen2-57B-A14BB IBB-tttt (GSM8K)
52.92
29.62
Table 8: Base and Instruct Owen3-30B-ANe models on Math and GSM8K.
Model
GSM8K
Math
No calibration
Qwen3-30B
85.37
50.48 46.94
Bwen3-30B-A3B-Insttte
90.90
Calibration: Zyda2
Owen3-30B-AM
71.72
9.44
78.01
9.88
Calibration: Task-specific
B-en3-30B-A3B-Imee (Math)
89.76 90.30
48.52
1ana---0--A-B-nstttee (GSM8K)
51.70
task-specific distributions poses substantial challenges for maintaining performance specialized tasks.
In contrast, MoNE can effectively preserve performance by pruning instruct models with training data from the specialized tasks themselves. For example, OLMoE-IInsrrce incurs only minimal accuracy drops of at most 1 1% for both tasks when calibrated with task- data. This result is not surprising, as state-of-the tuning and reinforcement learning with domain-specifee data to enhance their capability in these tasks (Muennighoff et al., 2025; Liu et al., 2025; Qwen, 2024; Yang et al., 2025). The effectiveness of task-specific calibration suggests that the distribution alignment between calibration data and target tasks plays a critical role in determining post-pruning performance.
models also rely on specialized fine-tuning approaches such as supervised fine-
In summary, acknowledge that calibration via pretraining data is insufficient preserve model capability for specialized tasks, and task -seciiicc calibration can effectively mitigate this issue. A promising future direction for model compression is to develop methods that bridge the gap between these two calibration approaches, potentially through hybrid calibration strategies or adaptive data selection techniques that better capture task-specific distributions while maintaining broad coverage.
B PRUNING METRIC ABLATION STUDY
We conducted extended ablation study the redundancy score evaluate the sensitivity of MoNE to different scoring formulations. Following the same evaluation configuration as Section 5.2, we compared MoNE against five variants: normalized scores, log-sum aggregation, and weighted sum with varying emphasis on output variance (25% 50%, and 75%). The results are reported in Table 9.
The results indicate that the pruning score exhibits minimal scale sensitivity across different for- mulations. In particular, the log-sum aggregation shows nearly identical results to MoNE, which is expected since log (ver) + log (og(0 log (fver (freg) does not affect the partial ordering
16

---

Table 9: Ablation study on redundancy scoring methods across different models. All values represent average performance scores.
Model
MoNE
Normalized
Log-sum
Weighted 25%
Weighted 50%
Weighted 75%
OLMoE
61.04
61.39
61.23
57.07
57.07
57.07
Moonlight
69.10
68.58
69.05
68.95
69.03
69.07
DeepSeck-V2- Boork---2--.iit Lite
66.12
65.70
66.02
65.66
65.66
66.07
Owen2-57B-A144
71.75
71.73
71.75
72.55
72.57
72.53
Qwen3-30B-A3B
72.69
73.62
73.66
71.96
72.29
72.48
Arcio Arcie
MMLU
Winogrande
Frequancy
Layers
0.8
0.6
Variance Layers
12162024252284044455555
116202123328444446525666 Experts
28323640 5-4555660
Experts
Experts
Figure 5: Layer-wise normalized expert access frequency and output variance of OLMoE for Are-C & Arc-E, MMLU and Winogrande.
during expert redundancy ranking. This mathematical equivalence ensures that both formulations produce the same pruning decisions despite their different representations.
We did not include learnable weights in this ablation study, as training five models with learnable parameters would be prohibitively expensive and time-consuming. However, the weighted sum results reveal model-dependent behavior, where only Qwen2-57B-A 14B achieves minor improvements of approximately 0.8% with weighted scoring. In contrast, smaller models like OLMoE incur severe performance drops of around 4% under weighted sum formulation. This suggests that while larger models may benefit from adjusting the relative importance of variance and frequency components, smaller models are more sensitive to such modifications and perform better with the balanced geometric mean approach used in MoNE.
C MORE DETAILED RESULTS
This section presents the experiment results on Zyda2 dataset with 100, 500 and 1000 calibration samples in Table 10, Table and Table 12. Table 13, Table 14 and Table 15 presents the experiment results on C4 dataset with 100, 500 and 1000 calibration samples. The observations are similar to those in Section 5.2 and Section 5.3. Nevertheless, all methods incurs more aggressive performance drop at higher (50%) pruning ratio. Such phenomenon is inevitable for all model compression approaches and inspires the development MoNE to push the performance limits of structured pruning.
D REDUNDANT EXPERT VISUALIZATION
This section complements the visualization of redundant experts for OLMoE, Moonlight and Deepseek-V2-Lite
V2-Lite across the nine downstream tasks. The results are depicted Figure Figure 12. As mentioned in Figure 1, for each figure, expert in blue circles has both high frequency and variance. Expert in red circles only has high variance. Expert in green circles only has high frequency. For each model across the nine downstream tasks, validating the effectiveness the redundancy metric, i.c., the expert access frequency and output variance.
always identify the same important experts,
17

---

Table 10: Zero shot performance with 100 calibration samples from Zyda2 dataset. Best results are in bold, and the second best are underlined.
(a) OLMoE
Pruning ratio
Model/Method
Are-e
BoolQ
COPA
MMLU
OBQA
PIQA
Winogrande Average
0%
OL.
49.23
76.89
70.09
85.00
53.54
44.40
79.76
71.84
68.90
66.63
Angular
27.22
37.50
53,91
62.00
23.96
26.60
58.27
52.35
51.85
43.74
HLAP
29.18
54.92
62.17
68.00
30.51
29.60
67.57
55.23
56.27
50.39
50%
MC-SMoE
24.49
31.44
59.35
67.00
23.01
26.00
53,92
53.43
53.12
43.53
RS
21.50
28.62
39:45
61.00
23.27
26.00
52.34
51.99
51.85
39.56
MoNE (Ours)
28.16
40.24
63.12
78.00
25.21
32.40
63.33
60.65
63,54
50.52
(b) Moonlight
Pruming ratio
Model/Methoo
Are-c
Arc-e
BoolQ
COPA
MML.U
OBQA
PIQA
RTE
Winogrande Average
0%
Moonlight
58.28
82.49
80.40
92.00
67.30
81.12
65.70
71.11
71.56
Angular
27.90
51.99
47.75
40.02
FLAP
33.87
61.36
63.30
75.00
36,80
36.00
69.37
57.04
62.12
54.98
50%
MC-SMoli
47.94
50.68
RS
37.80
58.42
70.86
89.00
23.27
38.00
78. 18
57.76
70.50
58.23
(Ours)
70.03
90.00
70.17
61.27
(c)
Sicepseek-V2-L.iee
Pruning ratio
Model/Method
Are-c
Are-c
BoolQ
COPA
MMLU
OBQA
PIQA
RTB
Winogrande Average
0%
Deepscok-V2-Lin V2-Lite
48.72
76.18
79.88
88.00
$4.96
43.60
80.25
61.37
71.51
67.16
Angular
56.42
41.28
TLAP
35.24
60.31
69.66
79.00
16.13
35.20
74.76
56.68
64.09
$6.79
MC-SMoE
35.82
28.29
RS
36.01
57.45
57.98
89.00
24.91
40.80
78.02
54.15
62.75
$5.67
(Ours)
(d) Qwen2-57B 2-n-2-2-BBB-- 55B-AA4B
Pruning ratio
Model/Method
Arc-c
Are-c
BoolQ
COPA
MMLU
OBQA
PIQA
KTE
Winogrando Avg.
Qwen2-57B-A14B
49.66
86.45
44.20
74.73
71.89
Angular
25.17
31.19
45.08
54.00
23.74
29.80
54.62
53.07
50.28
40.77
FLAP
36.35
61.95
73.43
80.00
46.50
38.20
74.59
68.95
68.59
60.95
50%
MC-SMoE
41.64
64.73
79.91
87.00
59.65
38.80
77.53
71.84
68.27
65.49
RS
20.90
29.38
50.83
57.00
23.39
25.40
52.88
51.26
49.25
40.03
MoNE (Ours)
45.43
65.22
80.99
85.00
62.00
42.00
78.41
70.90
69.51
66.61
(e)
Qwen3-30B-AN
Pruning ratio
Model/Method
Are-e
Arc-e
BoolQ
COPA
MMLU
OBQA
PIQA
RTE
Winogrande
Avg.
Qwea3-30B-ABB
55.89
73.76
Angular
26.37
39.48
50.86
59.00
24.65
28.80
60.17
55.23
48.46
43.67
FLAP
36.52
61.99
78.00
48.92
34.00
70.57
74.37
60.01
50%
MC-SMoE
39.76
63.09
85.84
83.00
56.30
39.40
74,54
78.34
69.14
65.49
48.04
73.06
83.00
61.37
79.43
70.40
MoNE (Ours)
44.62
67.05
85.41
85.00
54.58
41.60
79.00
71.84
67.80
66.32
18

---

Table 11: Zero shot performance with 500 calibration samples from Zyda2 dataset. Best results are in bold, and the second best are underlined.
(a) OLMoE
Pruning ratio
Model/Method
Are-c
Are-c
BoolQ
COPA
MMLU
OBQA
PIQA
Winogrande Average
0%
OLMoE
49.23
76.89
70.09
85.00
71.84
68.90
66.63
Angular
32.76
61.91
61.71
74.00
23.13
37.60
71.65
53.07
55.09
52.33
FLAP
37.03
63.43
64.28
81.00
41.12
38.80
72.63
54.51
63.54
$7.37
25%
MC-SMoF
33.36
54.46
70.03
81.00
37.05
33,80
68.17
64.62
65.19
56.41
RS
23.81
40.91
57.92
00
27.79
30.20
63.71
50.18
$7.38
46.77
MoNE (Ours)
41.04
65.66
70.24
87.00
41.21
40.00
76.61
64.98
66.61
61.48
Angular
27.22
37,50
53,91
62,00
23.96
26.60
58.27
52.35
51.85
43.74
FLAP
30.38
52.99
62.17
70.00
30.91
33,20
66.65
59.21
$7.06
51.40
50%
MC SMoE
25.43
32.28
54,80
66.00
22.95
25.40
55.22
54.51
54.14
43.41
RS
25.34
28.24
42.48
56.00
23.07
26.40
$2.34
52.35
51.78
39.78
MoNE (Ours)
26.28
36.03
65.17
75.00
26.02
29.00
61.64
57.40
62.67
48.80
(b) Moonlight
Pruming ratio
Model/MMthho
Are-c
Arc-e
BoolQ
COPA
MML.U
OBQA
PIQA
RTE
Winogrande Average
0%
Moonlight
58.28
82.49
80.40
92.00
67.30
45.60
81.12
65.70
71.11
71.56
Angular
39.76
52.69
38.90
61.01
$2.96
FLAP
48.55
76.05
77.49
89.00
55.12
42.40
76.66
65.34
68.59
66.58
25%
MC-SMoE
63.47
76.82
71.
70.
RS
55.63
79.46
78.93
91.00
46.60
45.80
80.90
59
72.14
67.82
(Ours)
55.03
78.96
79.36
90.00
71.74
68.23
Angular
27.90
28.54
48.01
51.99
47.75
40.02
FLAP
34.98
61.32
65.14
73.00
37.93
35.60
69.53
56.68
62.35
$5.17
50%
MC-SMoE
22.87
29.34
43.93
RS
37.71
59.97
71.65
89.00
25.48
38.00
76.22
$7.04
68.82
58.21
(Ours)
75.90
59.46
(c) Deepseek-V22 -e2-seee--V2LLite
Pruning ratio
Model/Method
Are-c
Are-c
BoolQ
COPA
MMLU
OBQA
PIQA
RTB
Winogrande Average
0%
Meepseek-V2-IA.
48.72
76.18
79.88
88.00
$4.96
43.60
80.25
61.37
71.51
67.16
Angular
53.28
26.95
53.04
FLAP
43.86
72.18
75.93
85.00
47.22
41.80
78.45
62.09
68.27
63.87
25%
MC-SMoE
52.95
41.68
32.20
52.35
56.03
RS
48.98
73.23
71.77
89.00
$2.68
44.60)
79.33
61.01
70.32
65.66
MoNE (Ours)
48.29
79.43
39.21
Angular
24.06
32.79
40.40
61.00
23.22
26.80
56.42
$7.76
49.09
41.28
67.77
31.87
72.91
55.59
50%
MC. SMoE.
25.34
32.07
47.98
59.00
26.28
25.60
56.20
$4.51
53.75
42,30
RS
MoNE (Ours)
36.69
66.04
73.36
86.00
41.07
35.60
75.73
$8.12
69.69
60.26
19

---

Table 12: Zero shot performance with 1000 calibration samples from Zyda2 dataset. Best results are in bold, and the second best are underlined.
(a) OLMoE
Pruning ratio
Model/Method
Are-c
Are-c
BoolQ
COPA
MMLU
OBQA
PIQA
Winogrande Average
0%
OLMoE
49.23
76.89
70.09
85.00
71.84
68.90
66.63
Angular
32.76
61.91
61.71
74.00
23.13
37.60
71.65
53.07
55.09
52.33
FLAP
38.91
66.20
64.65
79.00
40.05
37.60
74.65
62.82
63.77
58.63
25%
MC-SMoF
38.31
61.66
61.87
73.00
33.85
33.20
66.38 38
57.04
65.43
54.53
RS
26.62
43.35
59.76
70.00
27.50
30.00
65.02
49.46
56.67
47.60
MoNE (Ours)
42.32
64.52
66.45
88,00
41.55
40.80
78.02
64.26
67.01
61.44
Angular
27.22
37,50
53,91
62,00
23.96
26.60
58.27
52.35
51.85
43.74
FLAP
30.46
54.88
62.32
70.00
29.63
32,00
67.95
$7.04
57.54
51.31
50%
MC SMoE
25.43
35.31
55,11
67.00
22.92
25.40
54.52
51.99
51.14
43.20
RS
24.23
29.50
41.44
58.00
23 45
24.20
$1.25
50.54
51.07
39.30
MoNE (Ours)
26.71
37.21
65.44
75.00
26.50
31,80
63.22
55.96
63.46
49.48
(b) Moonlight
Pruming ratio
Model/MMthho
Are-c
Arc-e
BoolQ
COPA
MML.U
OBQA
PIQA
RTF
Winogrande Average
0%
Moonlight
58.28
82.49
80.40
92.00
67.30
45.60
81.12
65.70
71.11
71.56
Angular
39.76
52.69
61.01
$2.96
FLAP
49.49
76.52
76.91
90:00
54.47
42.00
77.26
65.70
69.06
66.82
25%
MC-SMoE
57.74
77.22
71.35
RS
56.23
79.25
79.02
91.00
46.76
45.20
80.63
59.57
72.14
67.76
(Ours)
55.55
79.29
79.66
90.00
59.21
72.14
68.55
Angular
27.90
28.54
48.01
49.00
$1.99
47.75
40.02
FLAP
36.95
63.09
64.56
71.00
37.14
34.80
69.86
$7.40
63.22
$5.33
50%
MC-SMoE
23.12
53.07
53.75
45.72
RS
39.42
64.56
70.98
90.00
24.76
40.60
77.58
58.12
68.
$9.45
(Ours)
(c)
Breepseek-V2-L..
Pruning ratio
Model/Method
Are-c
Are-c
BoolQ
COPA
MMLU
OBQA
PIQA
RTB.
Winogrande Average
0%
Meepseek-V2-IAi
48.72
76.18
79.88
88.00
$4.96
43.60
80.25
61.37
71.51
67.16
Angular
53.04
FLAP
44.28
72.39
76.12
85.00
47.16
41.20
78.29
62.82
67.40
63.85
25%
MC-SMoE
64.31
71.55
41.84
76.12
58.48
60.68
RS
49.23
73.32
71.38
89.00
52.44
45.00
79.71
60.29
71.19
65.73
MoNE (Ours)
73.99
78.35
39.93
Angular
24.06
32.79
40.40
61.00
23,22
26.80
56.42
57.76
49.09
41.28
50%
MC. SMoE.
33.19
47.81
55.35
81.00
24.91
33.80
67.52
51.99
64.333
$1.10
MoNE (Ours)
37.71
65.36
73.49
84.00
41.22
36.60
75.30
$7.40
69.53
60.07
20

---

Table 13: Zero shot performance with 100 calibration samples from dataset. Best results are in bold, and the second best are underlined.
(a) OLMoE
Pruning ratio
Model/Method
Are-c
Are-c
BoolQ
COPA
MMLU
OBQA
PIQA
Winogrande Average
0%
OLMoE
49.23
76.89
85.00
71.84
68.90
66.63
Angular
32.76
61.91
61.71
74.00
23.13
37.60
71.65
53.07
55.09
52.33
FLAP
36.01
62.67
64.83
75.00
36.31
37.00
75.73
58.84
62.75
56.57
25%
MC-SMoF
32.76
51.05
54.71
80.00
23.05
37.80
70.89
53.43
66.61
52.26
RS
34.56
50.38
63.64
85.00
26.51
39.60
76.01
55 36.96
64.01
55.07
MoNE (Ours)
36.69
55.18
67.03
86.00
25.53
40.20
77.80
56.68
67.64
56.97
Angular
27.22
37.50
53,91
62,00
23.96
26.60
58.27
52.35
51.85
43.74
FLAP
26.62
50.51
62.14
62.00
26.32
28.80
69.86
52.71
$7.06
48.45
50%
MC SMoE
25.68
32.24
57.65
60.00
22.83
27.40
$8.60
52.35
51.38
43.12
RS
23.12
31.99
45.02
59.00
80
24.80
$3.59
51.62
49.57
40.28
MoNE (Ours)
26.37
39.65
62.29
75.00
22.93
32,00
66.11
52.71
61.01
48.67
(b) Moonlight
Pruming ratio
Model/MMthho
Are-c
Arc-e
BoolQ
COPA
MML.U
OBQA
PIQA
RTF
Winogrande Average
0%
Moonlight
58.28
82.49
80.40
92.00
67.30
45.60
81.12
65.70
71.11
71.56
Angular
39.76
52.69
38.90
79.00
61.01
$2.96
FLAP
46.76
75.04
75.90
85.00
52.98
41.00
78.07
65.70
67.40
65.32
25%
MC-SMoE
48.38
74.20
92.00
71.19
RS
55.97
79.76
78.93
91.00
52.74
46.20
$1.39
59.21
72.38
68.62
(Ours)
54.27
79.25
78.07
90.00
57.76
72.14
68.02
Angular
27.90
28.54
48.01
51.99
47.75
40.02
FLAP
31.14
54.63
62.69
73.00
32.26
30.40
70.40
$8.12
60.93
$2.62
50%
MC-SMoE
29.10
52.23
57.22
52.71
52.31
RS
37.80
58.42
70.86
89 00
23.27
38.00
78. 18
57.76
70.80
58.23
(Ours)
57.07
72.75
90.00
70.17
(c) Deepseek-V22 --2-see
Pruning ratio
Model/Method
Are-c
Arc-c
BoolQ
COPA
MMLU
OBQA
PIQA
RTB
Winogrande Average
0%
Meepseek-V2-IA
48.72
76.18
79.88
88.00
$4.96
43.60
80.25
61.37
71.51
67.16
Angular
26.95
53.04
FLAP
39.68
66.58
76.45
$4.00
36.14
39.80
78.67
57.04
67.48
60.65
25%
MC-SMoE
59.30
36.80
75.30
54.87
$6.66
RS
49.32
74.41
69.39
90.00
50.35
43.50
80.14
62.09
70.24
65.53
MoNE (Ours)
80.25
66.06
Angular
24.06
32.79
40.40
61.00
23.22
26.80
56.42
$7.76
49.09
41
52.55
50%
MC. SMoE.
28.75
36.57
59.45
82.00
23.65
30.60
64.96
51.62
59:75
48,
$5.67
MoNE (Ours)
35.24
55.47
68.20
87.00
22.94
35.40
76.50
$3.43
68.35
55.84
21

---

Table 14: Zero shot performance with 500 calibration samples from dataset. Best results are in bold, and the second best are underlined.
(a) OLMoE
Pruning ratio
Model/Method
Are-c
Are-c
BoolQ
COPA
MMLU
OBQA
PIQA
Winogrande Average
0%
OLMoE
49.23
76.89
70.09
85.00
71.84
68.90
66.63
Angular
32.76
61.91
61.71
74.00
23.13
37.60
71.65
53.07
55,09
52.33
FLAP
36.26
63.80
63.49
72.00
35.09
36.00
75.35
53.07
61.96
55.23
25%
MC-SMoF
26.71
48.61
65,90
69.00
35.14
31,00
61.92
54.87
61.09
50.47
RS
34.56
49.41
63.85
86.00
27.23
39.80
75.57
56.32
64.40
55.24
MoNE (Ours)
37.88
55.89
65.23
86,00
25.03
41,20
77.42
57.40
67,80
57.10
Angular
27.22
37.50
53,91
62,00
23.96
26.60
58.27
52.35
51.85
43.74
FLAP
26.71
50.42
62.20
64.00
27.52
30,20
70.29
52.35
$5.09
48.75
50%
MC SMoE
23.98
32.07
61.77
63.00
23.01
25.40
$4.79
57.40
52.49
43.77
RS
22.95
31.78
49.11
59.00
23.75
23.00
$3.59
53.43
50.04
40.74
MoNE (Ours)
29.01
42:51
62.29
79.00
22.98
32,00
70.84
52.71
60.06
50.16
(b) Moonlight
Pruming ratio
Model/MMthho
Are-c
Arc-e
BoolQ
COPA
MML.U
OBQA
PIQA
RTF
Winogrande Average
0%
Moonlight
58.28
82.49
80.40
92.00
67.30
45.60
81.12
65.70
71.11
71.56
Angular
39.76
52.69
38.90
61.01
52.96
FLAP
45.22
73.40
72.29
84.00
53.81
40.40
78.13
64.98
67.32
64.40
25%
MC-SMoE
48.38
78.56
71.03
RS
55.72
79.80
79.20
91.00
52.93
44.60
81.23
59.57
71.74
68.42
(Ours)
54.52
78.87
78.56
90.00
57.04
71.74
67.87
Angular
27.90
28.54
48.01
51.99
47.75
40.02
FLAP
31.83
54.50
63.18
74.00
29.18
31.40
71.16
$5.96
61.88
$2.57
50%
MC-SMoE
33.45
54.59
52.71
$2.68
RS
39.59
58.67
70.49
86. 00
23.36
38.60
78.02
$4.15
70.24
$7.68
(Ours)
37.60
57.85
(c) Deepseek-V22 V22LLte
Pruning ratio
Model/Method
Are-c
Are-c
BoolQ
COPA
MMLU
OBQA
PIQA
RTB
Winogrande Average
0%
Meepseek-V2-IA.
48.72
76.18
79.88
88.00
$4.96
43.60
80.25
61.37
71.51
67.16
Angular
53.28
53.04
FLAP
40.53
64.81
77.43
$4.00
37.70
39.60
79.43
57.04
67.32
60.87
25%
MC-SMoE
41.37
77.75
58.28
RS
49.66
74.49
65.63
91.00
50.04
43.60)
79.92
61.01
69.77
65.01
MoNE (Ours)
62.09
65.68
Angular
24.06
32.79
40.40
61.00
23.22
26.80
56.42
57.76
49.09
41
52.71
50%
MC. SMoE.
23.55
35.73
57.43
71.00
23,86
32.20
61.10
51.62
51.30
45.31
$4.64
MoNE (Ours)
34.39
55.72
67.68
$4.00
22.90
35.60
76.28
$4.51
67.72
$5.42
22

---

Table 15: Zero shot performance with 1000 calibration samples from C4 dataset. Best results are in bold, and the second best are underlined.
(a) OLMoE
Pruning ratio
Model/Method
Are-c
Are-c
BoolQ
COPA
MMLU
OBQA
PIQA
Winogrande Average
0%
OLMoE
49.23
76.89
70.09
85.00
71.84
68.90
66.63
Angular
32.76
61.91
61.71
74.00
23.13
37.60
71.65
53.07
55.09
52.33
FLAP
36.35
61.24
64.50
72.00
37.08
36.00
75.41
57.40
62.59
55.84
25%
MC-SMoF
40.10
68.22
61.65
70.00
36.91
39.60
71.65
59.57
57.93
56.
RS
34.39
49.41
64.80 89
85.00
26.64
40.60
75.95
57.04
63.46
55.26
MoNE (Ours)
38.74
57.45
65.57
86.00
24.59
41,80
77.42
56.68
68.43
57.41
Angular
27.22
37.50
53,91
62,00
23.96
26.60
58.27
52.35
51.85
43.74
FLAP
26.79
49.92
62.17
63.00
27.34
31.60
70.24
53.43
56.35
48.9%
50%
MC SMoE
26.96
41.58
58.56
$6.00
23.00
28.60
60.23
52.35
50.83
44.23
RS
23.81
32.28
46.09
58.00
23.76
24.00
$3.59
51.99
48.86
40.26
MoNE (Ours)
29.78
42.80
62.32
80,00
22.99
33.60
71.49
52.71
60.06
50.64
(b) Moonlight
Pruming ratio
Model/MMthho
Are-c
Arc-e
BoolQ
COPA
MML.U
OBQA
PIQA
RTE
Winogrande Average
0%
Moonlight
58.28
82.49
80.40
92.00
67.30
45.60
81.12
65.70
71.11
71.56
Angular
39.76
52.69
38.90
61.01
$2.96
FLAP
45.22
74.20
72.78
86.00
53.40
41.20
78.35
65.34
68,90
65.04
25%
MC-SMoE
51.62
76.52
77.80
92.00
08
71.11
RS
55.97
79.59
78.75
90.00
53.01
45.20
81.34
59.57
71.90
68.37
(Ours)
53.75
79.38
78.53
57.76
71.59
Angular
27.90
28.54
48.01
51.99
47.75
40.02
FLAP
31.31
55.26
63.73
72.00
32.98
33.00
71.16
56.32
62.75
$3.17
50%
MC-SMoE
28.75
$8.90
49.87
RS
38.74
58.88
69.85
838 00
23.32
38.00
77.80
61.01
70.17
58.42
(Ours)
57.58
70.72
58.20
(c)
Soopssek-VV-LLiig
Pruning ratio
Model/Method
Are-c
Arc-c
BoolQ
COPA
MMLU:
OBQA
PIQA
RTB
Winogrande Average
0%
Meepseek-V2-IA.
48.72
76.18
79.88
88.00
$4.96
43.60
80.25
61.37
71.51
67.16
Angular
26.95
53.04
FLAP
39.33
66.50
76.33
$4.00
38.14
40.60
78.84
59.93
67.88
61.28
25%
MC-SMoE
58.42
65.72
37.91
57.65
RS
49.23
74.33
68.44
88 18.00
$1.59
44.40
80.25
62.45
69.69
65.3%
MoNE (Ours)
48.90
65.50
Angular
24.06
32.79
40.40
61.00
23.22
26.80
56.42
57.76
49.09
41.28
75.41
52.71
50%
MC. SMoE.
28.16
36.49
56.94
$1.00
25.76
31.80
64.69
52.71
58.09
48, (4.40
24.58
$4.51
MoNE (Ours)
34.04
55.51
68.26
85.00
22.92
36.40
76.44
34.15
66.61
55.48
23

---

Frequency
15 Layers
0.8
0.4
Varfenne Layers
22162222992844444422220
12162024283222222222 12180222222222444444222
Experts
Experts
Experts
Figure 6: Layer-wise normalized expert access frequency and output variance of OLMoE for OBQA,
PIQA and RTE.
BoolQ
COPA
Frequency
Layers
9
0.8
9
1512
15
0.6
0
0.4
3
Variance Layers
1512 6
0.0
1216002288236-444888226 8 1216202-22333344444555560
Experts
Experts
Figure 7: Layer-wise normalized expert access frequency and output variance of OLMoE for BoolQ
and COPA.
Winogrande
Frequenny Layers
0.8
0.4
Veriance Layers
0.2
1216202428322
12162024 48520442
Experts
Experts
Experts
Figure Layer-wise normalized expert access frequency and output variance of Moonlight for Are-C & Arc-E, MMLU and Winogrande.
E COMPREHENSIVE ABLATION STUDY RESULTS
Figure 13 reports the detailed ablation study on the impacts of the three factors: expert access frequency, output variance and novice replacement. The results this figure validates that the three factors play important role in maintaining the effectiveness of the pruned models different model architectures, calibration data sources and calibration sample sizes. Fusing the three factors ensures the robustness of the proposed MoNE.
24

---

Fiequency Layers
0.8
0.4
Variande Layers
121622222822504444266666
12162024 288226000
12188222222264444822222 Experts
Experts
Experts
Figure 9: Layer-wise normalized expert access frequency and output variance of Moonlight for
OBQA, PIQA and RTE.
BoolQ
COPA
1.0
Frequency
Layers
12
0.8
16
20
24
0.4
Variance
Layers
12
0.2
16
20
24
0.0
0
4 8 12162024233264444485222222
0 4 8 12 1226202228224444444562
Experts
Experts
Figure 10: Layer-wise normalized expert access frequency and output variance of Moonlight for BoolQ and COPA.
RTE
Fraquincy
2. Layers
0.8
Vartence Layers
0.2
122222222228404444222626
12162024 2220022222224444444566666
1212024222284444855333
Experts
Experts
Exports
Figure 11: Layer- normalized expert access frequency and output variance of Deepseek V2tLitee for OBQA, PIQA and RTE.
F INFERENCE LATENCY AND MEMORY FOOTPRINT
While this work mainly focuses on enhancing performance preserving capability of structured pruning, this section evaluates the inference latency and memory footprint for pruned models. We integrated the pruned Owen3-30B-AA3 with the popular inference framework, SGLang with transformers backend. We fixed the 512 input tokens and 256 output tokens, and profiled the latency and memory footprint with SGLang built-in utilities. We varied the random seed generate different input sequences. Experiments were conducted on a single H20 GPU. The results are listed in Table
According to Table 16, would like to clarify three points regarding the latency and memory usage. First, inference latency speedup MoNE sensitive to batch size. When batch size 1, there is minor speedup, as the decoding phase only executes tokens per step, leading memory bound
25

---

BoolQ
COPA
Frequency
Layers
12
0.8
16
20
0.6
24
0.4
Variance
Layers
12
16
20
24
0.0
8 021622428323604444425666
0
Experts
Experts
Figure 12: Layer-wise normalized expert access frequency and output variance of Deepseek-V22 -V2-Lee for BoolQ and COPA.
Table 16: Inference latency and memory footprint under different pruning ratio, input sequence and batch size.
Pruning ratio
Seed
Batch size
Novice hit ratios*
Total latency (s)
Speedup
Memory usage (GB)
62.72
25%
1020
0.08
27. 84
1.04
47.72
50%
1020
0.35
26.27
1.10
32.73
0%
1998
I
0
29.01
62.72
25%
1998
0.06
28.61
1.01
47.72
50%
1998
0.34
26.22
1.11
32.73
1020
128
183.12
66.00
25%
1020
128
0.08
174.60
1.05
$1.00
50%
1020
128
0.34
141.07
1.30
36.01
0%
1998
128
182.37
66.00
25%
1998
128
0.08
169.66
1.07
$1.00
50%
1998
128
0.33
142.38
1.28
36.01
1020
512
248.02
75.90
25%
1020
512
0.09
212.40
1.17
60.90
50%
1020
512
0.33
189.57
1,31
45.91
0%
1998
512
245.42
75.90
25%
1998
512
0.08
212.88
1.15
60.90
50%
1998
512
0.34
180.92
1.36
45.91
*Novice hit ratios: the portion of tokens routed to novices across the model.
situation where the GPU tensor core underutilized and the major bottleneck the GPU memory bandwidth (Hong et al., 2023; Frantar al., 2025). By increasing the batch size to 512. MoNE can achieve speedup of 36%. Second, inference latency speedup is sensitive to novice hit ratio instead of novice counts (pruning ratio). Qwen3-30B-4 routed to 8 experts per novices. The actual speedup is decided by the routing results for each input token at runtime. With higher novice hit ratio, MoNE expected attain more speedup. While higher pruning ratio may lead to more novices in each layer, it cannot directly translate to speedup. For the most extreme case where there no token routed novices, the computation overhead well as the accuracy) will be the same the original model. Finally, Table 16 indicates that MoNE achieves maximum memory reduction that consistently increases with the pruning ratio. For a given pruning ratio, the heavy expert parameters are directly replaced by light weight constant vector.
has 128 experts per layer, and each input token is
26

---

0..0
18
4.4
10
36
14
-.6
3.9
169
460
500
480
6.
-12
4.8
0639
111
103
138
13
12
436
1M
135
203
-130
146
6.20
0.24
-10
143
515
4.5
4.34
4.00
11000
1.5
133
038
5.8
542
832
615
1913
18
4.00
3.5
1200
140
401
-300
128
1.2
028
505
730
1.5
2000
6.2
030
500
13
19
1100
1000
4.99
9:1
2N
540
15
813
4.30
1:00
1.20
0438
11:1
13.1
0.80
11.0
1280
6:8
9.90
08.9
11.
110
471
4.0
1989
0902
12.0
0.8
-.20
1949
6.8
0.79
4.00
2.8
1.00
0.3
280
1.77
64"
19:0
7:0
1200
3.4
0.1
1100
001319
130
217
1.3
204339
031
113
1400
240
20:2323
1.85
1100
0.00
12.0
644
1929
1380
640
080
632
437
1350
130
111
5-
136
19
177L
3.21
1.54
1.41
0.6
6.36
0.10
4.8
843
-113
3.80
4.71
0110
172
1240
1225
7960
8.0
3.10
+1.0
0.38
1880
1.18
6H4
130
236
3.00
+1.00
12.9
634
616
515
8.2
1.6
1.91
1.90
610
280
4.00
3.10
1500
11.0
1.5
090
1.46
1.4
105
171
710
1.5
7000
18
330
3.77
0.3
136
0609
100
460
128
245
470
150
Figure 13: Ablation study on expert access frequency, output variance and novice replacement with detailed results. Numbers are the difference to the proposed MoNE
27


Reviews:
[{"review_id": "80QlE9PpyN", "reviewer": "Reviewer_qHPd", "summary": "This paper proposes MoNE, a structured pruning approach that replaces redundant experts with lightweight \"novices\" - essentially constant vectors representing averaged expert outputs. The key idea is identifying redundant experts using two metrics: access frequency (how often an expert is selected) and output variance (how stable an expert's outputs are). Experts with low frequency and low variance get replaced by their mean output. The authors test this on five MoE models (OLMoE, Moonlight, DeepSeek-V2-Lite, Qwen2-57B-A14B, Qwen3-30B-A3B) at 25% and 50% pruning ratios, showing better performance than existing methods like MC-SMoE, RS, Angular, and FLAP.", "questions": "1. How does MoNE perform on specialized benchmarks like Math or GSM8K? The current evaluation on general tasks may not be sufficient to prove that specialized experts are not being harmed.\n\n2. What is the justification for per-token routing to a static novice vector? Would a simpler approach, like adding the novice as a scaled, static bias, perform comparably while saving routing computation?\n\n3. Are novices trainable during continued pretraining?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core idea is intuitive, simple, and training-free. The fused metric (frequency + variance) is well-justified, and the \"novice\" replacement (the expert's mean output) is an effective closed-form solution to minimize output discrepancy.\n\n2. The experimental validation is a major strength. Testing on five different MoE architectures with varying sizes (7B to 57B parameters) demonstrates the method works across scales. The robustness evaluation across model architectures, calibration data sources (Zyda2 vs C4), and sample sizes (100, 500, 1000) is thorough.", "weaknesses": "1. There is a lack of specialized tasks (e.g., coding, math) in evaluation. It's unclear if the redundancy metric, calibrated on general text, might inadvertently prune experts that are critical for these specialized capabilities.\n\n2. The paper doesn't explain or ablate the benefit of computing a dynamic, per-token gate for a static, constant \"novice\" vector. This appears computationally redundant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eSxblhrnfU", "reviewer": "Reviewer_14Ca", "summary": "MoNE proposes to prune MoE models by replacing selected experts with constant novices (per-expert mean outputs) instead of simply deleting or merging them. Redundancy is estimated by two metrics computed on a small calibration set—access frequency and output variance—and the novice for a pruned expert is its unbiased mean output. The method is evaluated across several MoE architectures (OLMoE, Moonlight, DeepSeek-V2-Lite, Qwen2-57B-A14B, Qwen3-30B-A3B), calibration sources/sizes, and pruning ratios, with ablations on the metrics and novice replacement.", "questions": "* After pruning, do tokens get routed more often to the remaining real experts or to novices? Any trends per layer?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Simple, compute-friendly pruning primitive (constant novices) that retains router behavior and keeps overhead close to removal. \n\n* Consistent gains/robustness across models and calibration setups; headline numbers are competitive.", "weaknesses": "* The ablation in Figure 4 is intersting, seems like the variance metric can bring improvement without the novice. It would be helpful to include more comprehensive ablation, i.e. more combination (e.g. only frequency and only variance) to show the gain from each part.\n\n\n* The novice is the unbiased mean output of a pruned expert (a constant vector), similar to FLAP’s use of averaged activations for compensation but at a different granularity. The paper should more explicitly discuss the relation with FALP and isolate the contribution.\n\n \n* The redundancy score is the product of variance and frequency. This hard-coded fusion may be scale-sensitive; it would be helpful to include normalized scores, log-sum, or a learned weight λ and report stability.\n\n\n* Some related works worth mentioning [1,2]\n\n[1] MOE-PRUNER: PRUNING MIXTURE-OF-EXPERTS LARGE LANGUAGE MODEL USING THE HINTS FROM ITS ROUTER\n[2] SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q6FNyAaF4e", "reviewer": "Reviewer_cM6H", "summary": "The research focus of this paper is expert pruning in Mixture-of-Experts (MoE) models. To address this issue, the paper proposes Mixture-of-Novices-and-Experts (MoNE), a novel expert pruning method that replaces redundant experts with lightweight novices to achieve effective and robust model compression. Experiments demonstrate that MoNE consistently outperforms baseline methods across three dimensions—model architectures, calibration data sources, and calibration sample sizes—with minimal accuracy loss, validating its effectiveness and robustness.", "questions": "Refer to Weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper propose a novel expert pruning method named MoNE which replaces redundant experts with lightweight novices to compress MoE models with minimal performance loss\n2. This paper uses expert access frequency and output variance to measure redundancy, and unbiased output estimation to minimize post-pruning discrepancy, yielding effective and robust pruning.", "weaknesses": "1. The combinatorial forms of frequency and variance adjacency matrices require ablation, such as weighted summation.\n2. Replacing experts with constant vectors may reduce expressiveness; could learnable vectors or biases be used instead?\n3. Could comparative experiments on pruning strategies (without finetuning) be provided to demonstrate the superiority of the proposed frequency- and variance-based pruning strategy?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LIc6nuZvI5", "reviewer": "Reviewer_tzmx", "summary": "This paper introduces MoNE, a pruning method for Mixture-of-Experts models. It identifies experts with low routing frequency and low output variance, and replaces them with constant vectors called novices. This reduces memory cost while keeping model accuracy stable. Experiments across multiple MoE architectures show that MoNE performs better than existing pruning baselines and remains stable across different calibration settings.", "questions": "No further questions, see above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of replacing experts with constant novice vectors is simple, easy to implement, and maintains the computational benefits associated with pruning.\n\nThe redundancy score combining frequency and output variance is well motivated and avoids relying solely on routing frequency, which previous methods often do.\n\nThe experimental evaluation is broad, covering multiple MoE architectures, different pruning ratios, and variations in calibration data, showing consistent robustness.", "weaknesses": "Replacing experts with constant vectors may reduce the model's expressive power, and test cases, due to their limitations, may not be able to cover the negative impacts of the evaluation.\n\nPruning strategies depend on the distribution of the calibration dataset. Different application scenarios may have different dependencies on experts, and differences between the distribution of the calibration dataset and the distribution of the real-world scenario may lead to the erroneous removal of important experts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)