DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
Authors: Sikai Bai
Haoxi Li
Jie ZHANG
Zicong Hong
Song Guo
Venue: NeurIPS 2025 poster ()
Keywords: Expert Pruning
Differentiable Optimization
Language Model
Mixture-of-Experts
Primary area: deep_learning

Abstract: Despite the significant breakthrough of Mixture-of-Experts (MoE), the increasing scale of these MoE models presents huge memory and storage challenges. Existing MoE pruning methods, which involve reducing parameter size with a uniform sparsity across all layers, often lead to suboptimal outcomes and performance degradation due to varying expert redundancy in different MoE layers. To address this, we propose a non-uniform pruning strategy, dubbed Differentiable Expert Pruning (DiEP), which adaptively adjusts pruning rates at the layer level while jointly learning inter-layer importance, effectively capturing the varying redundancy across different MoE layers. By transforming the global discrete search space into a continuous one, our method handles exponentially growing non-uniform expert combinations, enabling adaptive gradient-based pruning. Extensive experiments on five advanced MoE models demonstrate the efficacy of our method across various NLP tasks. Notably, \textbf{DiEP} retains around 92\% of original performance on Mixtral 8$\times$7B with only half the experts, outperforming other pruning methods by up to 7.1% on the challenging MMLU dataset.

DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
Sikai Bai
Haoxi Li HKUST Hong Kong, China
HKUST
Hong Kong. China
sbai Dbaiia@@conectt.sse ust.
h1 illibbeconnect.cm ust.
Jie Zhang HKUST
Zicong Hong HKUST
Song Guo HKUST
Hong Kong, China csejzhang@ust.hr hk
Hong Kong, China congcong@ust. hk
Hong Kong, China songguo@cse ustes.eee........ hk
Abstract
Despite the significant breakthrough of Mixture-of-Experts (MoE), the increasing scale these MoE models presents huge memory and storage challenges. Existing MoE pruning methods, which involve reducing parameter size with a uniform sparsity across all layers, often lead suboptimal outcomes and performance degradation due to varying expert redundancy in different MoE layers. To address this, we propose non-uniform pruning strategy, dubbed Differentiable Expert Pruning (DiEP), which adaptively adjusts pruning rates at the layer level while jointly learning inter-layer importance, effectively capturing varying redundancy across different MoE. layers. By transforming the global discrete search space into continuous one, our method handles exponentially growing non-uniform expert combinations, enabling adaptive gradient-basse pruning. Extensive experiments five advanced MoE models demonstrate the efficacy of our method across various NLP tasks. Notably, DiEP retains around 92% of original performance on Mixtral 8x 7B with only half the experts, outperforming other pruning methods by up to 7.1 % on the challenging MMLU dataset.
Intorduction
Large Language Models (LLMss), such GPT4 (36] and Llama series 14 33
have demonstrated remarkable performance across diverse domains. However, real-world deployment poses significant challenges due ever-growing number of parameters, including high computational demands and storage costs. To address these issues, the ofxture-of--xpertt (MoE) architecture [10] 12 has emerged a promising solution, activating only subset of parameters during training and inference. Notable MoE-based models, such as Mixtral 8B [21], and DeepSeek V3
(27]. achieve faster
inference while maintaining competitive performance with dense models 14 3 of comparable scale.
Despite their computational efficiency. MoE models suffer from substantial memory and storage costs due larger model sizes, making their deployment in essoucce-onssrrinee environments challenging 199 2 For example, DeepSeck V3 has 256 experts per layer and 67 IB parameters.
Recent empirical analyses have shown that the routing policies learned by current MoE LLMs yield markedly unbalanced expert utilization 7 28). To mitigate the attendant waste of parameters, a grow- ing body of work aims to prune experts while preserving the task performance of the full MoE model.
Corresponning authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Most existing approaches impose uniform spar- sity budget on each layer: they either drop fixed number of experts in each layer, or ex- haustively search the combinatorial space of per-layer expert subsets. For instance, Zhang et al., [49] remove the same number of experts in
Intralbyer Expert Sinilarity Layer
Layer
Layer
Layer
1.
88
86
84
84
32345073 32345673
32345878
800
heuristics, whereas search-based methods such as EEP [29] and NAEE [32 enumerate all k-expert combina- tions inside each MoE layer. Unfortunately. con- sidering the discrepancy of expert redundancy across different MoF. layers (i.e., more number of experts required to be activated in shadow layers than deeper layers, as demonstrated in Sec. S.c. simply applying uniform pruning ratio for all layers may cause poor performance during inference. Worse still, such limitation cannot be solved through global layer-wise brute-force searches. For instance, in a 64-expert layer, pruning only 12.5% of the experts (k- = 8) already requires evaluating (64) # x 10 configurations, making exhaustive glohal optimization computationally infeasible.
each layer using srrivaationfrequeene
Inter-ayar Expert Siilarity
B
and
and
30
38
36
54
22
Figure I: Visualized analysis of the intra-layer and inter-layer similarity between expert pairs for dif- ferent MoE layers in Mixtral 8x through RBF kernel-based CKA criteria [22]. Darker colors rep- resent higher expert similarity.
Layer-aware strategies have begun to surface, but they still fall short of capturing the heterogencous relationships between layers. Among them, Li et al. [24), merge infrequently routed experts into their high-traffic counterparts after within-layer normalization
activation counts. Such normalization, however, erases cross-layer information and implicitly assumes that redundancy is independent across depth. Figure T contradicts this assumption: (i) the intra-layer similarity matrices for layers 8, 12, 20, and 30 exhibit distinct block structures and sparsity patterns, and (ii) the inter-layer CKA heatmaps reveal both strongly correlated and strongly divergent expert pairs across adjacent layers. These observations underscore the need for adaptive, depth-sensitive pruning framework that leverages both intra- and inter-layer statistics to decide how many and which experts to retain at each layer.
To tackle these obstacles, propose novel and efficient approach, Differentiable Expert Pruning (DiEP), which reformulates expert pruning as a continuous optimization problem. Specifically, instead of searching over global discrete search space with exponentially increasing choices, DiEP relaxes expert selection into a differentiable process. It performs joint optimization to determine the relative significance of experts within each layer and inter-layer importance scores that regulate the contribution of different layers. By incorporating layer-aware importance modulation, DiEP enables globally optimized selection of experts through gradient- based optimization, effectively capturing both expert-level and layer-level impacts on the pruning process. Beyond permanently eliminating unimportant experts, we further propose online expert skipping mechanism that assigns decayed expert weights to highly similar experts during inference. It bypasses redundant expert computations for each input token and accelerates inference speed.
While the idea of continuous search has been explored in dense neural architectures | 43 DiEP first introduces this principle into the sparsely activated Moï paradigm-a setting with funda- mentally different structural and computational constraints. Unlike traditional bilevel differentiable search methods [30]. DiEP jointly optimizes intra-layer expert logits and inter-layer importance scores in a single-stage training process, guided by a lightweight reconstruction regularizer and without reliance
validation set. By decoupling gradient updates for intra- and inter-layer impor- tance scores, DiEP mitigates optimization interference and enables a global ranking mechanism that produces precise, depth-aware sparsity patterns without manual heuristics. Extensive experiments show that DiEP outperforms other pruning methods in various NLP tasks and MoE architectures, while reducing model size and enhancing inference efficiency.
2 Related Work
2.1 Sparse Mixture-of-Experts Models (SMoE)
selectively activates a small subset of specialized networks (experts) for each input, enabling efficient model scaling [5, 20). In early research, Shazeer et. al. [4] introduced the Sparsely-Gated

---

MoE layer, demonstrating the effectiveness of selective expert activation. 23 advanced SMoE by implementing distributed architecture that enabled efficient scaling across multiple devices. Recent studies have further refined SMoF. architecture based on SOTA L.Mss 44 Mixtral models [21] demonstrated successful scaling with a balanced approach of using two experts per token; Qwen- MoE [4]] and DeepSeek-MoF [10 [6] explored larger expert pools with selective activation. They have attracted great attention from the Al community. Despite these advances, current SMoE-LLM architectures require huge memory to load trillion parameters and suffer from low expert utilization during inference.
2.2 Expert Pruning for SMoE
expert pruning has become a promising technique to reduce model complexity while maintaining performance for SMoE. Existing solutions can be divided into two branches: 1) Features statistics identifies unnecessary experts based on the activation frequency or feature similarity, but such methods either dramatically compromise performance [50 35 or rely post post-procsssigg [24] [5]. 2) Greedy search heuristically searches all possible choices for pruned experts within each layer, which becomes impractical for the latest SMoE models due to exhaustive search (32 or task-specific fine-tuning [29 45). To make matters worse, all the above
Inspired by recent advances in LLMs 31 25
methods either fail to account for the varying levels of expert redundancy in different MoE layers by applying identical pruning rates or incur heavy computation costs to implement non-uniform pruning. However, our DiEP uses parameter-efficient intra-layer and inter-layer differentiable optimization to adaptively search pruned experts, reducing redundancy based on each layer/expert characteristics while keeping the full model's performance.
2.3 Continous Optimization
The concept of architecture search and optimization within a continuous domain has been explored 43 Early research [ 40 focuses on fine- mee-ttning architectural components such as filter shapes or branching patterns in convolutional networks. After that, representative framework DARTS 30 and its variants 6 46 were introduced to learn high-performance architec- ture building blocks with complex graph topologies, but they employ memory-intensive operations in the architectural search process and require costly nested optimization and validation-set dependence. Moreover, DiffPruning (39) was proposed to remove redundant parallel processing units in dense transformer architectures through differentiable pruning. Although updates head importance scores and model parameters via monotonic gradient descent, there's a risk of gradient conflict between importance scores and weight matrices, and requires threshold tuning after continuous relaxation. In contrast, our DiEP method decouples the intra-layer and inter-layer gradient optimization paths and achieves exact sparsity through unified global ranking without threshold tuning. To the best of our knowledge, our DiEP is the first method to explore continuous expert search for Mixture of Experts (MoE) architectures in the context of Large Language Models.
before [ 30 39
40
Preliminary: Mixture-of-Experts (MoE) Language Model
Generally, a Mfxture-of-Expprt (MoE) model consists of L layers, where each layer / (1 I,.... L) contains N experts. The input to all experts in the 1-th layer is denoted as 2a()) E IRdd, where d is the input dimension. A router network produces routing logits ς! for each expert (i 1,..., N). which are normalized using softmax function to compute the routing weights w!!
exp(s)))
W
. En ())
where w(!h represents the contribution of expert i in layer I.
To enforce sparsity, the router network selects the top-k experts with the largest routing weights w!!
The output of the 1-th MoE layer is then computed as:
((+1)
Σ
(x.)
k(
3

---

Initial Model
Differential Search
Pruned Model
MoF loyer
MoE Layor
Mot Loyer
Stoge 2. Updde
1:
MoE Loyer
Loyer
MoE Loyer
100 GOLE
DB
Token Fectures
Expert
Expert
Computation
Backwardard Update
Expert Weight
p: Loyer Weight
Figure 2: The schematic illustration of the Differentiable Expert Pruning (DiEP) Framework. (a) Initial MoE model with substantial expert redundancy and memory cost. (b) Differentiable Pruning: transforming discrete expert search into a continuous optimization by jointly learning intra-layer expert scores (a:) and inter-layer importance (3) via alternating update strategy, enabling adaptive
non-uniform pruning. (c) Final pruned model: achieving streamlined MoE architecture that
maintains high performance while reducing the model's footprint.
where (-) denotes the feed-forward function of expert and Top-k(w(0) refers to the indices of the k-largest routing weights. The final output (1++1) is passed to the subsequent layer.
4 Method
4.1
Sparse Expert Search Space
Following the design principles of differentiable architecture search, first define sparse expert search space tailored for Mfxture-of-Expere Experts (MoE) architectures, as illustrated in Figure 2 this framework, MoE layer is modeled directed acyclic graph (DAG) consisting of only nodes: an input node representing the token representations entering the expert layer and an output node representing the sum of selected expert transformations. Instead of treating individual experts as independent computational units, we formulate the expert pruning process as discrete operation over single aggregated expert node.
Based on expert pruning principles, a subset of experts is retained according to their importance, governed by binary selection mask m, E {0, 1), where m! indicates that expert is retained, and m; = indicates pruning. The expert aggregation process in an MoE layer is then expressed as: (1)
y ((+1)
Σ (m) (z.))
where F-N,(-) denotes the feed-forward function of expert i.
This discrete selection process inherently results non-differentiable search space, making direct optimization intractable. To enable gradient-basee optimization and structured pruning within the MoE framework, we introduce continuous relaxation mechanism, allowing smooth updates the expert selection process while preserving the structured sparsity of the model.
4.2 Continuous Relaxation and Optimization
Specifically, we decompose the expert importance into two components: intra-layer importance scores that determine the relative significance of experts within each layer and inter-layer im- portance scores that regulate the contribution of different layers the selection process. This formulation allows to perform structured pruning in data-driven and globally optimized manner.

---

We define the intra-layer importance weights, (0) by normalizing the intra-layer importance scores using a softmax function:
(1) exp(a,)) ) 2j=1 exp(a,")
(4)
where a are learnable logits that determine the relative importance of experts within layer l. This normalization ensures a smooth and differentiable selection process. Similarly, the inter-layer importance score 3() is introduced as a trainable scalar that modulates the overall contribution of layer The output of MoE layer / is then computed
N
(x"))
(5)
To ensure that the pruned model retains fidelity to the original MoE model F(x) (before pruning), we introduce reconstruction regularization term (a. B), defined as:
a B) = =I=F a, F(
(6)
where || - || denotes the Frobenius norm. This regularization encourages the pruned model F to
maintain consistency with the original model.
The overall objective function is formulated as: min C(a, 3) == Lee (y. (x; a, B)) + (a, 8),
where À is a regularization coefficient, and Lce is the cross-entropy loss.
Alternating Update Strategy. To optimize the objective function, we adopt an alternating update strategy where the intra-layer importance scores a and inter-layer importance scores are updated iteratively:
a' t a' no V 8² - B 118 2 3²)).
(8)
(9)
Here, I denotes the iteration index, and are the learning rates for a and B, respectively, and L(a, (3) represents the overall objective function defined in Equation 7 From the theoretical perspec- tive, we summarize the optimization process in Algorithm Tand provide the detailed convergence analysis in Appendix B.2
Pruning Strategy. To derive a discrete architecture, we apply a structured pruning mechanism that eliminates the least significant experts based their global contribution across all layers. Instead of pruning experts layer-by-layer in isolation, we leverage the learned intra-layer importance scores og and inter-layer importance scores ((1) to determine expert significance in a unified manner.
Formally, the overall importance of expert in layer / is computed as the product of its intra-layer and layer- importance scores:
1 = (1) (())
(10)
Given the expert sparsity ratio the total number of experts to be pruned across the entire MoE model is NLr, where N is the number of experts per layer and is the number of layers. The pruning process is performed by globally sorting all experts based on their importance scores s; and removing the bottom-K least significant experts. The resulting pruning mask m is defined as:
(1)
0 if P, otherwise,
(
(11)
where the set of the bottom-K experts selected for pruning.
By jointly considering both intra-layer and inter-layer importance scores, this pruning strategy ensures a globally optimized selection of experts, effectively reducing computational redundancy while maintaining structural balance across layers.
5

---

4.3 Adaptive Skipping During Inference
During the inference process, processing each token with all selected top-k experts introduces unnecessary computational overhead, but researchers in [32] find that not every selected expert provides essential contributions for tokens. This observation motivates the need for adaptive ex- pert skipping, which selectively bypasses less significant experts during inference to enhance ef- ficiency. For each token a: an MoF layer, the top-k experts are chosen using routing weights w { Weg...... Wea-1 ). and their outputs are denoted as Veg .e....... :eee.... Following common practice, we assume
for simplicity. Unlike previous approaches [32) that rely solely on routing weights, our method incorporates expert similarity to dynamically skip less important experts during inference, thereby enhancing computational efficiency.
Assume experts with indices eo and are selected, with we, < Wea To improve inference speed, We, < ?y expert e1 is skipped, where y is hyperparameter specific to each MoE layer and generation step.
In our implementation, 7 is calculated the product of two factors. First, ? is determined the dian value of "*11 across sampled calibration data for each MoE layer. Second, "/2 is computed based
Who on the similarity between expert outputs, evaluated using Centered Kernel Alignment (CKA) [22 Specifically, "/2 is the ratio the CKA similarity p(ye, Ye,) ) to the mean CKA similarity p(ye, Ve,)
across all data samples in layer 1. The final value of y is given by:
X 72-
(12)
This method dynamically adjusts expert skipping based on both expert routing weights and similarity, significantly enhancing inference efficiency and maintaining model performance. In our experiments, we observe a speedup in inference 1.2x to 1.3x while retaining approximately 92% of the average performance with only half of the experts Mixtral 8x
5
Experiments
5.1 Experimental Settings
Model Settings. Our primary experiments are conducted using the widely adopted SMoE model, Mixtral 8B.B. To validate our method's generalizability across different models, we extend our experiments to an instruction-followii 8x22B, and other types of SMoE models such as Deepseek- Mone--MME-1100 16B and Owen2-57B-144 14A. In the Mixtral architecture, each token activates two experts in every MoE layer. Both Mixtral 8x and Mixtral 8x 8x22B contains 56 MoE layers with the same number of experts per layer. Deepseek-Mon -ooee- 16B employs a different architecture with 28 layers and 64 experts per layer, where each token passes through two shared experts and seleets six additional experts. Similarly, Qwen2-57B- 14A consists of 28 MoE layers with 64 experts in each layer but utilizes eight experts per token during inference.
model, Mixtral In-sstruut the larger model Mixtral
contain 32 sparse MoE layers with eight experts per layer while Mixtral
Dataset. We evaluate model performance using the Language Model Evaluation Harness library across four zero-shot tasks: MMLU [18] OpenBookQA 34, BoolQ [8) and RTE [4. MMLU
13
18 represents the most comprehensive and challenging benchmark, encompassing 57 subtasks distributed across four major domains: humanities, social sciences, STEM, and other. More results on other tasks are provided in the Appendix A.
Implementation Details.
During the expert pruning phase, construct small calibration subset with 128 samples from the C4 dataset for fine-tuning purposes. We implement parameter-efficient differential learning through alternating training cycles, with a 3:1 ratio between intra-layer scores and inter-layer scores B updates. Both training processes employ a learning rate of 5e-3 with a cosine learning rate scheduler. In addition, the complete training protocol consists of 10 epochs with a batch size of 16. For weight hyperparameter settings, we use 0.01 for all Mixtral architectures and À = 0.01 for other MoE models. All experimental evaluations are conducted using four NVIDIA GeForce A800 GPUs.
Baselines. We compare our method with the following pruning methods: M-SMoF (24, which merges experts based customized permutation alignment and routing strategies; Expert Trimming (17),

---

Table 1: Zero-shot performance comparison of different expert pruning methods Mixtral-8x 7B, Mixtral-8 x x7B In-ttuttut,, and Mixtral-8 x22B. Expert sparsity r indicates the proportion of pruned experts in the full model across all layers. The first and second columns represent results for expert sparsity r 25% and r 50%, respectively.
Model
Methed
25%
humamilies
science
MMLA
stem
other
agg
BaniQ
OpenBookQA
RTE
Average
Full
60.5
77.8
58.9
74,2
67.9
K5.3
35.4
71.5
65.1
51.8/24.8
60.5/26.5
46,9/24.7
60.5/25.0
54.9/25.3
82.03999
32.001.6
70.455099
60.0319
Mixiral
Expert Trimming
49.2/36.9
59.7/45.6
45.0/35.1
58.24334
54.1/4577
77.2/766
35.02664
56.455.9
55.2/512
NAEE
52.443.5
66.4152.7
49.0/40.4
63.7/43.5
58 734.3
84,0990
32,6/888
67.96614
60.8/54.6
S S-MO
$6.048.0
73.1/57.0
52.4/13.3
68.2/51.6
99.91508
66.4/83.3
314/2622
69.366.1
61,5155.9
DIEP(Ours)
58.15599
75.4/99.
56.8/49.1
72.0663.5
64.9/57.9
86.6844.0
33.1/29.6
70.7668.
63.8/59.
Full
61.2
79.7
39.6
15.8
68.1
88.5
36.6
722
66.4
M-SMOE
48.533388
62 62.3/37.5
44.0/33.8
55.1/35
52.03550
85.4/766
29.0266
67,5661.
$8,5/50
Mistral 8x78
Expent Trimming
52.945.0
74,361.1
50,5/39.2
64.8/50.8
58.647.3
86.383.0
37,0/32.
63.2/66.8
61,M57.3
55.944.77
69.5155.6
54.1/42.3
68.7/662
62.4/528
87.3/948
35,03044
70.075.5
63,89999
DIRP(Ours)
61.2/85.1
78.1773
59,4/53.4
73,866..8
67.386133
87.7/85.
35.9/310
72.2/74.0
65.863.0
Full
68.6
84.
67.1
78.7
72.6
87.9
35.8
71.5
67.0
27.3/22-7
2/25/25.8
24.4/24.0
27.9/334
264//339
62.8662.7
12./13.0
54.2/49.5
29.1/37.3
Mixtral
Expert Trimming
5K.045.7
74.9157.7
$4,1/2.00
70.2/45.7
64.3447.8
81.5/74.
35.2/27.0
69.355.4
62/6/5177
NAEE
604/53.9
78.0667.2
59.5/52.3
73.0541.
67,3159.1
67.4/90.5
35,0/111
70.1667.9
65.1/59.7
S-MOB
623457.8
78.5199.7
60,2/113
73.4/4.2
68.69008
87.683.1
35,8/322
71.168.1
65,76613
(IRP(Ourr))
65.0/589
81.8/332
63.2/54.2
76.0668.7
70.7662.4
87.7/84L5
35.8/444
71.370.4
66.4/62.
MMLU
Average
MMLU
Average
40.5
50.5
17.%
-4
Aecuunery
15.0
32.3
20.0
Expat Trmning
Eapan Tremi
Laper Eveniing
20.0
14
14 af
10
Number of experts
Number of secprts
Number of experts
Number eopuets
(a) Deepscck-Mofe MoE-
(b) Qwen2-57B-144
Figure 3: Zero-shot performance comparison on Deepseck-MoF- 16B and Qwen2-57B-14A.
which removes less important experts using activation frequency or removes structured modules through layer and block dropping; NAEE (32), which enumerates expert combinations and selects optimal remaining experts by minimizing reconstruction loss; S-SMOE [49], which identifies and addresses expert redundancy through similarity-basse pruning and merging operations.
5.2
Main results
To illustrate the efficacy of our proposed method, we report the performance comparison of DiEP and other state-of-the-a art pruning methods through comprehensive experiments on five SMoE architectures across various tasks.
5.2.1 Results on Mixtral Models
Table shows experimental results Mixtral 8BBB, Mixtral 8 Ix7s-Instruct, and Mixtral 8xB,B, where all three architectures have 8 experts in each MoE layer and we prune them under 25% and 50% expert sparsity, respectively. Mixtral 8 x 7B: Compared to other pruning strategies, our proposed DiEP significantly outperforms them on all tasks with a clear margin performance improvement, up to 7.1%. Specifically, when evaluated on MMLU, which is challenging dataset with numerous sub-tasks, other methods suffer from performance bottlenecks under 50% expert sparsity, but our DiEP effectively alleviates the negative influence of removing large number of experts.
These results demonstrate that DiEP effectively preserves the key expert knowledge by differential optimization and search on task-agnostic data using intra-layer scores and inter-layer scores. Mixtral 7n our DiEP significantly surpasses other pruning strategies by substantial margin. Specifically, DiEP achieves optimal performance with an average reduction of only 0.6% compared to the full model under 25% expert sparsity (i.e., removing 64 experts after pruning). These outcomes indicate that DiEP successfully mitigates the detrimental effects of expert pruning. Mixtral 9x2BB:
7

---

We further extend our pruning strategy to a larger model, Mixtral 8x22B which activates 39 billion parameters out of a total of 141 billion. Our proposed DiEP method continues to demonstrate substantial improvements across all tasks, retaining 94% of the full model's performance even after the removal of 50% of the experts. These results reveal the significant redundancy present the MoF layers and showcase the scalability of DiEP for large-scale SMoE models.
5.2.2 Results on Deepseek and Qwen Models
To demonstrate the generalizability of our proposed method across various models, we further apply it to the Deepseck-MoE-I66h and Qwen2-57B-14,, architectures, which differ significantly from those Mixtral models. In these architectures, each layer comprises 64 experts, with 8 experts activated for each token at every layer. Specifically, shown in Figures Ba and 3b] we averagely reduce the number of experts in each layers from 64 to 62, 60, 58, 56, 54, 52, 50 and 48 in both models.
Deepseek-MoE-II 16B: We observe that the frequency-based method (M-SMoF) suffers significant performance degradation on the MMLU dataset. In contrast, our DiEP consistently showcases superior performance across various pruning ratios, achieving average advantage of approximately 1.57% compared to second runner strategy (S-MoE), which relies on additional expert merging and large similarity matrix computations.Afies removing 244 experts from the full Qwen-MoE model, our DiEP retains a promising average performance of 68.7%, reflecting a mere 0.4% degradation compared to the full model.
Qwen2-57B-14A Actually, DiEP always achieves comparable performance to the full model and sur- passes all baseline methods across various tasks. This underscores the adaptability and effectiveness of our method for different SMoE models, grounded in general- purposs differential optimization.
5.3 Ablation Studies
5.3.1 Effectiveness of Components
To measure the importance of key components in our DiEP, we con- ducted ablation studies on Mixtral 8B with the following variants. As shown in Table 2 Row I serves as the baseline (NAEE), only performs a layer-wise search for all possible ex- pert combinations and has poor per- formance. Row 2 focuses Icarn- ing intra-layer expert importance a to measure the global contribution of each expert. Row 3 denotes the variant of eliminating a, and applies inter-layer scores /3 to reweight expert activation frequencies as global importance scoring of each expert. Compared with the baseline, the substantial performance gains demonstrate that the two components are both effective. To further investigate the efficiency of learnable inter-layer importance scoring, we replaced learnable 3 with fixed scores in Row 4-7, where ratio of 2:1 means = 2 for layers 1-16 and B = 1 for layers 17-32. The results show that assigning higher scores to lower layers yields better performance in Mixtral 8x7B, but this method of artificially fixing parameters cannot be generalized to other SMoE architectures and could result in a significant waste of computational resources identify the optimal B. Furthermore, our method leverages complemen- tary knowledge from intra-layer scores and inter-layer scores 3 for better expert selection, yielding superior performance.
Table 2: Performance analysis different components.
Method
MMLU
BoolQ
OpenBookOA
RTE
Avg.
Baseline
58.7497.3
81.0/30.8
32.6/28.8
67.9/61.4
60.8/51.6
W. 60,5/51.0
86.0/82.
32.2/27.8
67.5/65.3
61.6/56.7
Wa
61.051.4
85.18333
32/0/29.
67.36662
61.3/57.6
W. + Wa(rendone)
57.649.2
85.6183.4
32.3/27.2
664/62.
60447/55.5
W. Wa(1
55 1/46.2
81,57774
3066/6688
66.4/4422
$7.8/44.2
:1
63.3/54.2
68.2/67.5
62.4/58.8
W. :1)3:
64.6/55.2
85.9/84.2
32.8/29.6
69.7/67.8
63.3/59.2
DiEP(Ours)
643/5599
86.67540
5.3.2 Visualization Analysis for a and 3
To further validate the effectiveness of our proposed method, we visualized the variation of the updated lntra-layer scores and inter-layer scores 8 after the pruning stage. As shown in Figure 4a the distribution of intra-layer importance scores a reveals that experts in layers 1-15 tend to have higher average scores compared those in layers 16-32. This suggests that shallower layers generally play more significant role in the overall model. Figure 4b illustrates the inter-layer importance scores, which corroborate the intra-layer observations. The overall trend indicates that the alternating update strategy effectively captures both intra- and inter-layer dependencies, ensuring that the MoE
8

---

The variation of value
The variation of Value
25
MoE: Layers
25
Value
1.
1.0
18
16
20
24
28
32
Experts
Experts
(a) Intra-layer scores
(b) Inter-layer scores 8
Figure 4: Visualization of values distribution for intra-layer scores a and inter-layer scores / on
Mixtral 7B when r = 50%..
model retains critical information from shallower layers. Furthermore, a closer examination of layer 2 reveals that two experts. Specifically, the fourth and eighth experts exhibit markedly high values relative to their peers. This shows that these experts are consistently considered highly important by the model. Conversely, the remaining experts in layer generally have low importance scores, indicating that the pruning strategy in this layer is not governed by single importance criterion. Instead, it shows a clear preference for retaining these two experts through global selection. Overall, these empirical findings further confirm the efficacy of our proposed differentiable expert pruning approach and underscore synergistic relationship between and 3.
5.3.3 Computation Cost Analysis
We further analyze the efficiency of our DiEP during the pruning and in- ference stages. For pruning, as shown Table our baseline (NAEE), us-
Table 3: Pruning time comparison of our DiEP and NAEE on different models under 25% expert sparsity.
Method
Mixtral 8x7B
Mixtral 8x22B
DhepseckkMMB--16
Qwen2-57B-14n
ing an exhaustive heuristic search, be- comes computationally prohibitive for models with large expert pools like Deepseck-MoFF Moo--e and Qwen2-57B- 14A. In contrast, our DiEP, with only 0.01% parameter overhead, maintains consistent pruning time and achieves superior performance regardless of model architecture expert count. Furthermore, Table 4 shows DiEP's inference cost reductions Mixtral 8x in terms of latency and GPU
NAEE
1.31h
1.57h
to 94000d
t 113000
DMRP(Ours)
0.23h
0.31h
0.28h
0.34h
Table 4: Inference cost analysis on Mixtral 8 x 7B after expert pruning.
Pruning
Skipping
Avg. Acc
Speedup
GPU
0%
65.1
1.00x
1.00 x
0%
64.1
1.07 x
1.00 x
25%
63.8
1.18x
0.76 x
25%
v
/
63.3
1.21x
0.76 x
50%
59.9
1.26x
0.52
50%
59.6
1.28
0.52
memory. Our DiEP enhances inference efficiency via online expert skipping, which adjusts router weights according to expert similarity with negligible loss in performance. Using half the experts, DiEP retains nearly 92% performance on Mixtral 8x7B, achieving 1.28 token generation speedup and 48% memory savings. We provide more experimental analysis for ablation study in Appendix A
Conclusion
In this paper, we propose DiEP, novel differentiable expert pruning framework that reframes expert selection continuous optimization problem. By enabling gradient-basee based optimization and introducing an adaptive expert skipping mechanism, our DiEP significantly reduces memory usage and accelerates inference while maintaining high model performance. Extensive experiments show that our DiEP outperforms other MoE pruning methods across various language tasks, and sets new benchmark for efficient Sparse MoE deployment.

---

7 Acknowledge
This research was supported by fundings from the Hong Kong RGC General Research Fund (152169/22E, 152228/23E, 162161/24E, 162116/25E), Research Impact Fund (No. R5060-19, 19, No. R501 1-23), Collaborative Research Fund (No. C1042-23GF), NSFC/RGC Collaborative Re- search Scheme (Grant No. 62461 160332 & CRS [L__HSST660/22) Areas of Excellence Scheme (AoE/E-601/22-R), and the InnoHK (HKGAI).
References
[1] Karim Ahmed and Lorenzo Torresani. Connectivity learning in multi-branch networks. arXiv preprint arXiv: 1709.0952, 2017.
[2] Sikai Bai, Shuaicheng Li, Weiming Zhuang. Jie Zhang, Kunlin Yang, Jun Hou, Shuai Yi, Shuai Zhang, and Junyu Gao. Combating data imbalances in federated superviid learning with dual regulators. In P Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 10989-10997, 2024.
[3] Sikai Bai, Jie Zhang, Song Guo, Shuaicheng Li, Jingcai Guo, Jun Hou, Tao Han, and Xiaocheng Lu. Diprompt: Disentangled prompt tuning for multiple latent domain generalization in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27284- 27293, 2024.
[4] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009.
[5] Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang. Sunghun Kim, and Jayi Huang. survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, 2025.
[6] Xiangning Chen and Cho-Jui Hsich. Stabilizing differentiable architecture search via perturbation-ban -aaed regularization. In International conference on machine learning, pages 1554-1565. PMLR, 2020.
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song. Xian-Ling Mao, et al. On the representation collapse sparse mixture experts. Advances in Neural Information Processing Systems, 35:34600-34613. 2022.
[8] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint
arXiv: 1905. 10044, 2019.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafford. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.0 05457, 2018.
[10] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu. Huazuo Gao, Deli Chen. Jiashi Li, Wangding Zeng. Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-off-xxper language models. arXiv preprint wwXiv:2401.06066 2024.
[11] Omer Elkabetz and Nadav Cohen. Continuous vs. discrete optimization of deep neural networks. Advances in Neural Information Processing Systems, 34:4947 4960, 2021.
[12] William Fedus. Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(00)) 2022.
[13] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff. et al. A framework few-shot language model evaluation. Version vQ. O.I. Sept, 10:8-9, 2021.
[14] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:240..2177 2024.
[15] Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decompression for moc-based lims compression. arXiv preprint 25X2 2025.
[16] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint 250::25001.1244 2025.
10

---

[17] Shwai He, Daize Dong, Liang Ding, and Ang Li. Demystifying the compression of mixture-of-exxerts through unified framework. arXiv preprint arXiv:2406.02500, 2024.
[18] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0:3: 03300, 2020.
[19] Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[20] Robert A Jacobs, Michael I Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
[21] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXiv:2400I 04088, 2024.
[22] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519 3529, PMLR, 2019.
[23] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikum, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[24] Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy. The Twelfth Interna- tional Conference on Learning Representations, 2024.
[25] Zhu Liao, Victor Quétu, Van-Tam Nguyen, and Enzo Tartaglione. Can unstructured pruning reduce the depth in deep neural networks? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1402-1406, 2023.
[26] Bin Lin, Zhenyu Tang. Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang. Munan Ning, et al. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint 1055: 2024.
[27] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, al. Deepseek-v3 technical report. arXiv preprint 2xXv:2412.119437. 2024.
[28] Boan Liu, Liang Ding, Li Shen, Keqin Peng, Yu Cao. Dazhao Cheng, and Dacheng Tao. Diversifying the mixture-of -xpeeets representation for language models with orthogonal optimizer. arXiv preprint arXiv:2310.09762 2023.
[29] Enshu Liu. Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficientt expert pruning for sparse mixture-of-expprts language models; Enhancing performance and reducing inference costs. arXiv preprint 240X::2407.00945 2024.
[30] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint 1rXiv:1806.000 2018.
[31] Jing Liu, Bohan Zhuang, Zhuangwei Zhuang, Yong Guo, Junzhou Huang. Jinhui Zhu, and Mingkui Tan. Discrimination-awe aware network pruning for deep model compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4035-405 2021.
[32] Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 6159-6172, 2024.
[33] AI Meta. The llaaa 4 herd: The beginning of a new era of natively multimodal ai innovation. httpx:/aai. meta httpl////lmmmmammmmmmmmma-iiiiii
checked on, 4(7):2025, 2025.
[34] Todor Mihaylov. Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint 1rXiv:1809.02775 2018.
[35] Alexandre Muzio, Alex Sun, and Churan He. meer-moe: Sparse expert efficiency through regularization for cextture-of-experee arXiv preprint arXiv:2404.050 005899 2024.

---

[36] OpenAL Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[37]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[38] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communicattinss of the ACM, 64(9)99-106, 2021.
[39] Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in neural information processing systems, 33:20378-20389, 2020.
[40] Shreyas Saxena and Jakob Verbeek. Convolutional neural fabries. Advances in neural information processing systems, 29, 2016.
[41] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Lc, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.0653 2017.
[42] Paul Tseng. Convergence of a block conrdinate descent method for nondiffcrentiable minimization. Journal of Optimization Theory and Applications, 109(3):475-49)) 4944, 2001.
[43] Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, 2020.
pages
[44] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXi::2412.15135 2024.
[45] Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wengi Jia, Miao Yin, Yu Cheng, and Bo Yuan. Moe-i": Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. arXiv preprint vrXiv:2411.01000 2024.
[46] Peng Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. b-darts: tion for differentiable architecture search. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10874 10883, 2022.
Beta-decay regulariza-
[47] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 190.. 078300 2019.
[48] Zhi-Hui Zhan, Lin Shi. Kay Chen Tan, and Jun Zhang. A survey evolutionary computation for complex continuous optimization. Artificial Intelligence Review, 55(1):59-110 110, 2022.
[49] Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianseng Gao. Diversifying the expert knowl- edge for task-agnostic pruning in sparse mixture-of-exxpets. Findings of the Association for Computational Linguistics, 2025.
[50] Hao Zhao, Zihan Qiu, Huijia Wu, Zili Wang, Zhaofeng He, and Jie Fu. Hypermoe: Towards better mixture of experts via transferring among experts. In Proceedingy of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 16605-00115 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide discussion for the limitations the our work in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

We summarize the optimization process of our DiEP in Algorithm and provide the detailed convergence 700 analysis in Appendix B.2
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide implementation details Section 5.1 and demonstrate result
reproducibility using extensive experiments in Section 5.2 Section 5
and ppendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [NA]
Justification:
We will release codes after our paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cc
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips
www.eesuummssssoooolooooooo ssssssosicyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide experimental settings Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We follow experiments statistics from the previous work 332 24 and don't 't report error bars, confidence intervals, or statistical significance tests. But we chose the average results after three runs for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide experiments compute resources in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://///eeeroooooo////////ttta
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide discussion for broader impacts on Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, |ppper swi thcode. com/ cona/datsets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
There are no crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: We LLM for formatting purposes and does not impact the core methodol- ogy, and originality of the research.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http::///eeerrpssscco////reeeees for what should or should not be described.
19

---

A Experimental Appendices
Full Model
Layer-wise Pruning
DiEP (Ours)
0.4
Layor
SMoE
0.2
0.0
Expert
Expert
Expert
Figure 5: Distribution of expert activation frequencies in the Mixtral 8x7B. The full model (left) uses all experts across all 32 layers, resulting in substantial memory consumption. Layer-wise pruning (middle) enforces uniform expert sparsity per layer. Our DiEP (right) provides more flexible approach, performing cross-layer expert pruning based on their global contributions.
A.1 Visualized Analysis of Expert Activation Frequency
To demonstrate the efficacy of adaptive expert pruning in our DiEP, conducted comparative analysis of different methods terms of expert activation frequency. As shown in Figure while the full MoE utilizes almost all experts, there significant disparities in activation frequencies among different experts, leading to substantial resource waste. Previous methods (i.e., Layer-wise Pruning) apply uniform expert pruning ratios across all layers, overlooking the intra-layer and inter- layer variations and dependencies among experts different MoE layers. In contrast, our method obtains non-uniform and adaptive expert pruning that varies pruning ratios according expert- specific characteristics. On Mixtral 8x 7B, we observed an increasing trend in expert pruning rates from shallow higher layers. We attribute this phenomenon the fact that shallow layers primarily process diverse low-level linguistic features, such as part-of surt-of-speech tagging and local word ordering, necessitating a larger number experts capture detailed linguistic information. Meanwhile, higher layers primarily handle global contextual and semantic information, abstract away from fine- ine-grained details, and thus can operate effectively with fewer experts.
A.2 Efficiency Analysis for Inference on Deepseek-MoE-166.
We further verify the efficiency of our adaptive skipping strategy on Deepseek-MoE. 16B in Table S and it can be observed that our method maintains more than 95% performance of the full model while reducing model size and improving inference efficiency.
Table 5: Inference cost analysis on Deepseek-MoE-166.
Model
Pruning
Skipping
Avg. Ace
Speedup
GPU
0%
56.2
1.00
1.00 x
0%
55.7
1.04
1.00
6.25%
/
54.8
1.07
0.95
Deepseck-MofF 16B
6.25%
54.4
1.08x
0.95
12.5%
54.1
1.Ix
0.89
12.5%
53.6
1.13x
0.89
A.3 Efficiency Analysis for GPU Memory Pruning Cost
To investigate the efficiency of DiEP concerning its memory footprint during the pruning process, we conducted a detailed comparison of GPU memory costs T highlighting DiEP's advantages in both efficiency and scalability. Computational Efficiency (Time Cost); Compared to NAEE (1.31 hours), DiEP's pruning process 5.7 times faster, requiring only 0.23 hours. DiFP also demonstrates 25.8% faster execution (0.23 hours) compared MC-MoE (0.31 hours). This indicates significant
20

---

advantage for DiEP in terms of the time required for pruning. Memory Optimization (Peak Memory); DiEP utilizes 60% less peak memory (139.0GB) than MC-MoE (348,4GB), showcasing superior memory efficiency. While DiEP's peak memory (139.0GB) is 46% higher than NAHE's (95. IGB), this is offset by its dramatically faster pruning time, a factor reflected in its overall resource efficiency. Overall Resource Efficcency (Memory-Hour Cost): DiEP's memory-hour cost (31.97 GB-h) is 70% lower than that of MC-MoE (108.00 GB-h). Furthermore, DiEP's s memory-hour cost 74% lower than that of NAEE (124.58 GB-h). These results clearly demonstrate that DiEP maintains lightweight resource footprint while drastically reducing runtime, positioning as a more resource-efficient choice for MoE pruning.
Table GPU memory pruning cost on Mixtral 8x
Method
Peak Memory(GB)
Time (h)
Memory-houe hour Cost (GB-h)
NAEE
95.1
1.31
124.58
MC-MoE
348.4
0.31
108. 00
DiEP (Ours)
139.0
0.23
31.97
A.4 More calibration data validation adaptability
To further validate DiEP's adaptability, we evaluated its performance on the domain-speciffe GSM8K dataset using distinct calibration datasets: the general-purpose C4 dataset and the domain- relevant Math dataset, comparing DiEP against the NAEE method. As detailed in Table Z the experimental results systematically demonstrate DiEP's advantages across these varied calibration settings. Specifically, when employing the general-purpose C4 calibration data, DiEP achieved consistent improvements over NAEE, outperforming by +3.93 points at 50% pruning rate and by +4.96 points at a 25% pruning rate, indicating robust performance gains with common calibration data. Furthermore, when utilizing the domain-specific MATH calibration data, DiEP maintained its superior performance, securing a +1.10 point advantage at 50% pruning and extending this lead to +2.21 points at 25% pruning. These findings collectively underscore DiEP's enhanced generalization capabilities and adaptability across calibration datasets with different data distributions.
Table 7: Adaptability validation GSMSK using different calibration datasets (C4 and Math).
Method
Pruning Dataset
r-25%
r=50%
Random
36.39
0.68
NAEE
C4
41.02
24.87
DiEP (Ours)
45.98
28.80
NAEE DiEP (Ours)
MATH
51.25
37.07
MATH
53.46
38.17
A.5 Merging Strategy
Inspired by S-SMoE [49]. we introduce a merging strategy for DiEP to consolidate redundant experts while preserving their diversity. Specifically, pruned experts are grouped with their most similar retained counterparts based on normalized CKA similarity, which is then normalized by the softmax function as the merging weight. Table demonstrates that the merging strategy further enhances performance under 25% and 50% expert sparsity, which highlights the strong scalability of our DiEP. not only effectively maintains the performance of the full model but also further restores the diversity of pruned experts by incorporating other orthogonal strategies.
Table 8: Performance analysis when integrating merging strategy.
Samples
Strategy
MMLU
BoolQ
OpenBook
RTE
Avg.
DiEP
64.9
86.6
33.1
70.7
63.8
25%
DiEP+Merging
66.6
86.1
34.1
71.0
64.4
DiEP
57.9
84.0
29.6
68.2
59.9
50%
DiEP+Mcrging
58.2
84.0
29.8
68.8
60.2
21

---

A.6 Impact of Calibration Data Size
To analyze the impact calibration data size, we randomly sampled 32, 64, 128, 256, 512, and 1024 sequences from C4 dataset [37 to learn DiEP's intra-layer scores (a' and inter-layer scores B. As shown Table TO 128 sequences achieve optimal performance when pruning Mixtral 8x from to 6 experts. More importantly, DiEP avoids performance collapse with only 32 samples. We attribute it to KD regularization enforcing DiEP's s features aligned with the full model.
Table 9: Performances of expert pruning when changing the number of samples in the calibration dataset.
Samples
MMLU
BoolQ
OpenBookQA
RTE
Avg.
32
62.8
84.3
31.6
65.5
61.1
64
63.6
85.3
32.2
66.4
61.9
128
64.9
86.6
33.1
70.7
63.8
256
64.7
85.9
32.6
70.4
63.4
512
64.3
84.5
32.6
67.5
62.3
1,024
63.7
83.9
32.8
66.3
61.9
A.7 Impact of Intra-layer a and Inter-layer /3 Update Ratio
The update ratio empirical choice aimed at achieving optimization stability. The high- dimensional a scores learn fine-grained expert rankings, while the low-dimensional 3 scores make coarse-grained, systemic adjustments. Our hypothesis is that allowing a to update more frequently helps stabilize the local expert rankings before the more impactful (3 scores are adjusted.
To rigorously justify choice, conducted new ablation study the a: update ratio. The results, shown below, are for 25% and 50% pruning ratios (formatted as 25%/50%). The experimental results clearly validate design choice. The 3:1 update ratio consistently achieves the best or best performance across almost all tasks and pruning ratios, culminating in the highest average scores for both 25% (63.8) and 50% (59.9) pruning. The data shows that giving intra-layer scores (a) generally leads to better performance (e.g., 2:1 and 4:1 outperform 1:1 and 1:2). This supports our hypothesis that stabilizing the local expert rankings is crucial. The 3:1 ratio strikes an optimal balance, outperforming both less frequent (e.g.. 2:1) and more frequent (e.g.. 4:1) update schedules in terms of average performance. In conclusion, our a/3 decomposition provides a principled way to model multi-scale redundancy, and our chosen 3:1 update ratio is not arbitrary but empirically validated to be the most effective schedule for stable and high-performing optimization. We will incorporate these new results into our revised manuscript.
updates the
Table 10: Ablation study on the a;33 update ratio.
Qr
MMLU
BoolQ
OpenBookQA
RTE
Avg.
1:1
65. 1/54.3
85.2/81.8
31.8/29.8 8
67.2/65.4
62.3/57.7
1:2
64.5/54.8
85.5/79.5
31.8/27.6
69.0/64.5
62.7/56.6
2:1
66. 66.2/56.5
85.1/82.5
32,4/28.9
71.2/67.2
63.7/58.7
2:2
65.3/55.1
85.6/83.9
32,0/27.8
67.9/65.1
62.7/57.9
3:1
64.9/57.9
86.6/84.0
33.
70.7/68.2
63.8/59.9
4:1
65.2/56.7
85.3/83.7
31.4/29.4
70.8/66.1
63.1/58.9
A.8 Complete Visualized Analysis of Expert Similarity
To validate our motivation regarding the necessity of cross-layer pruning, we first visualized the intra-layer expert similarities in each layer using the CKA similarity metric [22] for Mixtral 7B in Figure The analysis reveals significant variations in expert similarities, particularly pronounced in layer 31. Moreover, substantial differences in expert similarities exist between different layers, with layers 28-29 showing higher similarity compared to layers 10. Furthermore, we investigate expert-pairs similarities in adjacent layers in Figure 7 which demonstrates varying degrees of expert relationships across layers, exemplified by the strong correlation between expert in layer 30 and expert 5 in layer 31. These cross-layer expert dependencies have been overlooked by previous
22

---

Liyer
Leyera4
Layer 21
Liger222
Layer 13
Liyer 14
Laper
Mapar 14
0.
44
SApt Layer
22
Loy
Lage
Myw
Figure 6: Visualization for feature similarity of expert-paii within each MoE layer.
Migar 2
Liyer 2an3
Layer 3.44
Ligerdard Gands
LigerSand
Layer Gand?
Japeraard
Layer' arE6
1.0
Lover ST and
Laver 53
Lave20
Lave 21 2x122
Lawer
Lewer en
Lover 24
0.2
6
6
Figure 7: Visualization for feature similarity of expert-pairs across adjacent MoE layers.
pruning methods. Our approach effectively captures both inter-layer and intra-layer variations through alternating differentiable optimization of expert weight a and layer weight 8. In addition, we observed that the learned intra-layer and inter-layer scores do not fully correspond the visualized inter-layer similarity between expert pairs. It is plausible because we only provide expert similarity across adjacent layers for visualized analysis. However, DiEP learn expert redundancy and dependency across all MoE layers.
A.9 Hyperparameters Analysis
There are two key hyperparameters. including the number of epochs during differentiable search and the value of the weight À between reconstruction regularization term and cross-entropy loss of the overall objective function in Eq. 7 We first analyze the impact of X by varying its value in (5. 10, 15, 20, 30). Fa demonstratess that optimal performance achieved with À = 0.01 for both 25% and 50% expert sparsity. Additionally, we investigate how the number of epochs affects
23

---

MMLU
Average
MMLU
Average
65
12
Accuracy
ACcuraty
15
25
550
18
45
25%
27%
-9-
22%
6.62
5.43
311
Value of
Valun of
Number Sports
Number of Epochs
(a) Hyperparameters analysis in of value of (h) Hyperparameters analysis terms of the number of epochs.
weight coefficient in Eq.
Figure 8: Hyperparameters analysis in terms of the number of clients and weight coefficient À on Mixtral8 x 7B under 25% and 50% expert sparsity.
model performance. As shown in Figure 8b DiEP achieves optimal results when trained for 10 epochs under both 25% and 50% expert sparsity settings.
A.10 Results More Datasets.
We provide more experimental results on more datasets including ARC-c, ARC-e HellaSwag 47 and WinoGrand 38 Mixtral 8B and our DiEP is much better than NAEE across all tasks. As shown in Figure 11, these results further demonstrate the effectiveness of our proposed method.
Table 11: Zero-shot evaluation result on more datasets, including ARC-e, ARC-e, HellaSwag, WinoGrande.
Model
Method
ARC-c
ARC-e
HellaSwag
WinoGrande
NAEE
51.62 /48.89
81 81.94/78.16 16 61.60/57.66 83.31/82.52
75.37/72.85 76.03/73.55
Mixtral 8x
DiEP(Ours)
52.54/49.26
63.22/58.96
B Theoretical Appendices
B.1 Algorithm Pipeline of DiEP
Algorithm 1: DiEP Differentiable Expert Pruning
Input:
Model inputs æ. targets y, initial intra-layer scores a., initial inter-layer scores 8 regularization coefficient A.
while not converged do
/* Update intra-layer
(fix ())
Update a by descending:
F( x0(a',
/* Update inter-layer scores (fix a)
x
Update ge+1 by descending:
(C ((; 141 ,1)))))))) + x*(a"'1 181)
Set t + ++1 Output: Optimized intra-layer importance scores a and inter layer importance scores 8.
B.2
Convergence Analysis of DiEP
Let θ {(a, 8)} be the parameter space, where N. and R... Denote 0, =a and 0g = 3. The overall objective of DiEP is
L(0,, 0z)
Fxx + (x) F(x)
24

---

Assumptions.
A1. Lower-Boundedness.
inf C(a,, B) > -0. (o,
A2. Lipschitz Smoothness. Ve,C C is Liiischhitzz continuous, i.c [Vo, Ve, ((v)] & L. o| for {1,2}.
A3. Stepsizes. Fixed learning rates satisfy 0 < 1 < L. for (1,2).
A4. Level-Set Boundedness. The set {(a, B) 9 [(a, 3) - C(a", B00 is compact.
Algorithmic update. Fort t 0, I,..
0,+1 0 13 Vo, [() 02), iE [1,2}.
(13)
Lemma (Descent). Under A2-A3, (( ++ 02+1) < [()), 05) Ei= (2-
1
0,
2n
Proof. Apply the standard descent lemma to each block update in 13
Corollary (Monotonicity and Bounded Iterates). Assumptions AI-A4 imply (E(0),05) monotonically non-increasing and convergent, and {01 (2) bounded.
$20 is
Lemma (Vanishing Updates).
lim ||0,+1 0 = 0, é E {1,2}.
(+00
Proof. Summing the non-negative terms Lemma over gives a telescoping series dominated by L(O): 02) inf L; hence the series of squared update norms is finite.
Theorem 1 (Subsequence Convergence to Critical Points), Under Al-A4, the sequence {(0), 05) )22 generated by (13) possesses at least one convergent subsequence, and every limit point (0i, 05) satisfies 0a (00 = =0 and V [O0 07) 0; i.e. it is critical point of L.
Proof. By Corollary and A4, {0f 0/)} lies in compact set; hence the Bolzano- Weierstrass
theorem guarantees convergent subsequence (0 6,th 13) by and taking k - oo yields V C(T 05) = 0 for i = 1,2.
(07, 05). Lemma 2 gives
0 fu+t1
0. Dividing
Discussion. Theorem aaligns with the classical results of block coordinate descent [42 while instantiating them for our two-block differentiable pruning objective. It guarantees that DIEP converges (up to subsequences) to first-order stationary points under standard smoothness and stepsize conditions.
25

---

C
Limitations
While our proposed DiEP method achieves strong performance in MoE pruning, several limitations remain. Due to computational constraints, our main experiments cannot be conducted on some larger- scale MoE models, like Deepseek V3 [27]. and Qwen2.5-Max 44 Furthermore, our study primarily focuses on language models, leaving the effectiveness of DiEP in multimodal MoE architectures unexplored. Investigating whether our approach can achieve competitive performance on Vision- Language tasks, such as MoE-LLaVA 26 remains important direction for future research.
D Broader impacts
The development of DiEP, a method for compressing large offturreof--xpertss (MoE) models, carries several potential societal impacts, both positive and negative. Positive impacts: In regions scenarios where access to high-perfornance computing infrastructure is limited, DiEP can enable the deployment of capable Al models that would otherwise be infeasible. This could support applications education, healthcare, and public services underserved communities. Negative impacts: As Al models become more capable and efficient, they may automate tasks currently performed by humans, potentially leading to job displacement in certain sectors. While DiEP aims at efficiency, the broader trend of Al advancement contributes this concern.
26


Reviews:
[{"review_id": "uAWTGPUjw4", "reviewer": "Reviewer_xJgk", "summary": "This paper proposes DiEP, a differentiable expert pruning framework that formulates expert selection as a continuous optimization problem. By leveraging gradient-based optimization and incorporating an adaptive expert skipping mechanism, DiEP aims to reduce memory consumption and accelerate inference while preserving the performance of sparse Mixture-of-Experts (MoE) models. The method is evaluated across multiple language tasks, showing superior performance over existing MoE pruning baselines and establishing a new benchmark for efficient MoE deployment.", "questions": "See Weakness.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The paper is well-motivated, addressing the important and practical challenge of reducing the computational cost of MoE models through expert path pruning.\n\n- The proposed use of differentiable NAS for expert path selection is intuitive and well-formulated. While not novel, it aligns with recent trends in neural architecture optimization and is a reasonable application to the MoE setting.\n\n- The writing is clear and the paper is easy to follow, with appropriate background and methodological descriptions.\n\n- Experiments are conducted on multiple strong open-source MoE language models, including Mistral and DeepSeek, demonstrating the practical utility of the method.\n\nWeaknesses\n\n- The core technical contribution is limited in novelty. The idea of applying differentiable NAS to select or prune expert paths has strong conceptual overlap with earlier work in CNN model search and compression. This paper primarily adapts these techniques to MoE models without introducing new architectural or algorithmic innovations tailored to the unique characteristics of MoEs.\n\n- A major concern lies in the lack of domain generalization. The routing paths are statically searched and fixed on a single dataset, which introduces data-specific biases. MoE models are designed to leverage dynamic expert selection for better generalization across diverse inputs (e.g., mathematical reasoning vs commonsense QA). Static pruning may significantly impair this flexibility, reducing the model’s robustness and transferability.\n\n- The paper does not sufficiently engage with recent MoE pruning or compression papers, e.g., [1,2].\n\n[1] MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts\n[2] Mixture Compressor for Mixture-of-Experts LLMs Gains More", "was_revised": true, "final_justification": "Most of my concerns have been addressed through the authors’ rebuttal. The revised version should include a detailed comparison with conventional MoE compression methods.", "rebuttal": "We thank reviewer xJgk for providing positive feedback and helpful suggestions. Below please find a response.\n\n**W1: Novelty Concerns**\n\nWe thank the reviewer for their insightful feedback. We agree that our DIEP builds upon the established concept of differentiable search. However, we respectfully argue that DiEP is not a straightforward adaptation. Instead, it introduces **significant and novel algorithmic and architectural innovations to address the unique challenges of MoE models**, which differ fundamentally from the dense CNN/RNN architectures targeted by early NAS methods.\n\nWe highlight two key areas of novelty:\n\n**1. A Fundamentally Different Problem Formulation and a Novel Optimization Framework**\n\nThere is a core difference in both the goal and the optimization mechanics between DARTS and DiEP.\n\n*   **DARTS targets operation search within a DAG-based cell.** It solves a bilevel optimization problem to find the best set of operations (e.g., convolution, pooling) and their connections:\n    $$\n    \\min_\\alpha \\mathcal{L}_{val}(w^*(\\alpha), \\alpha) \\quad \n    $$\n\n     $$\n    \\text{s.t. }  w^*(\\alpha) = \\arg\\min_w \\mathcal{L}_{train}(w,\\alpha)\n    $$\nThis formulation is memory-intensive, relies on a validation set, and is designed to search a discrete set of operations, not prune entire expert sub-networks.\n\n*   **DiEP targets global, non-uniform expert pruning across all MoE layers.** We formulate this as a single-level optimization problem with a novel reconstruction objectivet:\n    $$\n    \\min_{\\alpha, \\beta} \\mathcal{L}_{ce}(y, F'(x; \\alpha, \\beta)) + \\lambda \\cdot \\Phi(\\alpha, \\beta)\n    $$\nWhere:$F'$ is the pruned model, and $F$ is the original full model. $\\Phi(\\alpha, \\beta) = ||F'(x; \\alpha, \\beta) - F(x)||_F$ is a **reconstruction regularization term**. \n\nThis approach avoids the validation set. The architecture parameters ($\\alpha, \\beta$) are learned directly on the training data alongside the model weights (via an alternating strategy), making the process more efficient and robust.\n\nThis algorithmic shift from bilevel, validation-based search to single-level, reconstruction-guided optimization is a primary contribution of our work.\n\n**2. Architecture-Specific Innovations**\n\nMoE models present a unique challenge: managing redundancy across dozens large expert sub-networks, where redundancy varies significantly from layer to layer. Our method is explicitly designed for this context.\n\n*   **Targeting the Right Granularity:** Instead of searching for micro-level operations like DARTS, DiEP introduces learnable, continuous proxies designed for the macro-level task of expert selection: **Intra-layer scores ($\\alpha$)** learn the relative significance of experts *within* a single MoE layer. **Inter-layer scores ($\\beta$)** learn to modulate the pruning pressure across different layers, enabling a non-uniform pruning budget. This mechanism directly addresses the observed phenomenon that expert utility is not uniform across model depth in MoEs.\n\n*   **Efficient Design:** The combination of $(\\alpha, \\beta)$ allows us to compute a global importance $s_i^{(l)} = \\bar{\\alpha}_i^{(l)} \\cdot \\beta^{(l)}$ for every expert in the model. This enables a global sort to find the least important experts, regardless of their layer. This entire mechanism has a lightweight memory complexity of $O(L \\times N)$ (layers $\\times$ experts), making it highly scalable for modern MoEs with many layers and experts. A naive application of DARTS-style search to this problem would be computationally infeasible.\n\n**References**\nDARTS: Differentiable Architecture Search. ICLR 2019.\n\n\n**W2: Domain Generalization and Static Pruning**\n\nWe sincerely thank the reviewer for raising this critical point. Here our response is structured around two key points: a clarification of our methodology and direct empirical evidence from our new experiments.\n\n**1. Clarification: Static Pruning of Experts vs. Dynamic Routing of Tokens**\n\nA crucial distinction must be made: DiEP performs **static pruning of the expert pool**, but the resulting pruned model retains its **fully dynamic routing mechanism**. For any given input, the router still dynamically selects the top-k experts from the *remaining, optimized set of experts*. We are not fixing the routing paths for tokens; we are identifying and removing a subset of experts that are fundamentally redundant, thereby creating a more efficient expert library. The core strength of MoE — adapting to diverse inputs by selecting different experts — is preserved.\n\n**2. Empirical Evidence: Robust Generalization from General-Domain to Specific-Domain Tasks**\n\nTo test for domain generalization, we pruned our model using **general-domain (C4), in-domain (Math), and mixed (C4+Math) calibration sets**. We then evaluated performance on two specialized tasks: CommonseQA (commonsense reasoning) and MathQA (mathematical reasoning).\n\n**Table 1: Pruning on General-Domain Data (`C4`)**\n\n| Method          | Pruning Ratio | CommonseQA | MathQA    |\n| :-------------- | :------------ | :--------- | :-------- |\n| Full Model      | -             | 66.83      | 42.38     |\n| NAEE            | r=50%         | 42.01      | 34.86     |\n| **DiEP (Ours)** | **r=50%**     | **51.52**  | **35.02** |\n| NAEE            | r=25%         | 63.31      | 41.04     |\n| **DiEP (Ours)** | **r=25%**     | **65.52**  | **41.77** |\n\nWhen calibrating on the general C4 dataset, DiEP demonstrates strong robustness.\n\n**Table 2: Pruning on In-Domain data (`Math`)**\n\n| Method          | Pruning Ratio | CommonseQA | MathQA    |\n| :-------------- | :------------ | :--------- | :-------- |\n| Full Model      | -             | 66.83      | 42.38     |\n| NAEE            | r=50%         | 53.40      | 35.97     |\n| **DiEP (Ours)** | **r=50%**     | **56.27**  | **37.19** |\n| NAEE            | r=25%         | 60.21      | 42.20     |\n| **DiEP (Ours)** | **r=25%**     | **65.27**  | **42.34** |\n\nEven when calibrated on Math data, DiEP excels on the out-of-distribution (OOD) CommonseQA task.\n\n**Table 3: Pruning on Mixed-Domain Data (`C4+Math`)**\n\n| Method          | Pruning Ratio | CommonseQA | MathQA    |\n| :-------------- | :------------ | :--------- | :-------- |\n| Full Model      | -             | 66.83      | 42.38     |\n| NAEE            | r=50%         | 51.27      | 35.00     |\n| **DiEP (Ours)** | **r=50%**     | **58.67**  | **36.86** |\n| NAEE            | r=25%         | 61.79      | 41.43     |\n| **DiEP (Ours)** | **r=25%**     | **64.98**  | **42.69** |\n\nWe can see adding a small amount of in-domain data to the calibration set improves performance.\n\n**Conclusion:**\nThese experiments directly counter the concern about domain generalization. DiEP's strength lies in its ability to leverage a small, task-agnostic dataset to learn a globally optimized, non-uniform pruning mask that preserves the most critical experts for a wide range of downstream tasks. \n\n\n**W3: Engagement with Recent MoE Compression Literature**\n\nWe sincerely thank the reviewer for highlighting these important recent papers. Here we make the detailed comparison with such approaches.\n\n**Comparison with Mixture Compressor (MC-MoE) and Delta Decompression (D²-MoE)**\n\nThese two methods represent the state-of-the-art in compression of MoE models. They differ significantly from DiEP in both their paradigm and methodology.\n\n*   **Paradigm Difference:**\n    *   **MC-MoE & D²-MoE:** They apply techniques like mixed-precision quantization (MC-MoE) or weight decomposition into shared components (D²-MoE) directly to a trained model using heuristics.\n    *   **DiEP:** DiEP is a **lightweight, gradient-based fine-tuning** method. It leverages a small, task-agnostic dataset to learn expert importance scores via optimization, allowing for a more data-aware and principled pruning decision.\n\n*   **Methodology Difference:**\n    *   **MC-MoE & D²-MoE:** These are **hybrid compressors**. They modify and transform the weights of *all* experts through quantization, SVD, or decomposition. They do not entirely remove experts.\n    *   **DiEP:** DiEP is a pure **pruning method**. It identifies and completely removes the least important expert sub-networks, resulting in a model with fewer total experts. \n\n**Experimental Comparison**\n\nWe compare DiEP against MC-MoE[1] and D²-MoE [2]. The results on several key benchmarks are presented below.\n\n| Method          | Compression Setting | MMLU     | OpenBookQA | CommonseQA | MathQA   |\n| :-------------- | :------------------ | :------- | :--------- | :--------- | :------- |\n| MC-MoE          | 4 bits              | 60.3     | 31.8       | 63.2       | 40.6     |\n| D²-MoE          | r=25% (pruned)      | 60.6     | 32.8       | 63.6       | 37.1     |\n| **DiEP (Ours)** | **r=25%**           | **64.9** | **33.1**   | **65.5**   | **41.8** |\n| D²-MoE          | r=50%               | 58.6     | 29.5       | 41.4       | 32.9     |\n| **DiEP (Ours)** | **r=50%**           | **57.9** | **29.6**   | **51.5**   | **35.0** |\n\nThe results clearly show that DiEP significantly outperforms both D²-MoE and MC-MoE, especially at r=25%. On the challenging MMLU benchmark, DiEP achieves a score of **64.9**, which is **4.3 points higher** than D²-MoE and **4.6 points higher** than MC-MoE, despite MC-MoE operating at a higher bit-rate. \n\n**Comparison with MoE++**\n\nMoE++ and our DiEP method are orthogonal, as MoE++ focuses on accelerating inference by routing simple tokens to \"zero-computation experts\" to reduce FLOPs, without shrinking the model's static size. In contrast, DiEP is designed specifically to compress the static model footprint, reducing storage and memory requirements. Given their fundamentally different objectives, a direct compression performance comparison is not applicable.\n\n[1] Mixture Compressor for Mixture-of-Experts LLMs Gains More, ICLR2025.\n[2] Delta Decompression for MoE-based LLMs Compression, ICML2025."}, {"review_id": "0D7GnxGxRz", "reviewer": "Reviewer_3XFq", "summary": "This paper introduces DiEP, a differentiable pruning framework for Mixture-of-Experts (MoE) models that addresses expert redundancy by learning to prune both intra-layer and inter-layer experts adaptively. Rather than relying on heuristic-based or exhaustive search methods that use uniform pruning across all MoE layers, DiEP formulates pruning as a continuous optimization problem. It introduces two sets of learnable parameters: intra-layer importance scores and inter-layer importance scores, which together guide a globally optimized pruning decision. Additionally, the paper presents an adaptive expert skipping mechanism during inference based on routing weights and expert similarity.\nExperiments across five MoE-based LLMs, including Mixtral 8x7B, DeepSeek-MoE, and Qwen2, show that DiEP achieves better performance than prior pruning methods while reducing up to 50% of experts and improving inference efficiency.", "questions": "How sensitive is the method to the choice of calibration dataset? You mention results on C4 and Math datasets; however, can you clarify how performance varies when the calibration data is misaligned with downstream tasks?\n\nCan the method handle dynamic pruning ratios? Would it be possible to adaptively determine the pruning ratio during optimization rather than treating it as a fixed hyperparameter?\n\nLimitations in Multi-task or Continual Learning. How would DiEP behave in a continual learning or multi-task setting where expert utility might shift?", "limitations": "Yes, the authors have adequately discussed the limitations and trade-offs involved, such as reliance on calibration data and the inability of prior uniform pruning approaches to handle inter-layer heterogeneity. The discussion on societal impact is minimal, but the work primarily targets efficiency improvements without introducing major ethical concerns.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\nThis paper addressed an important scalability challenge in MoE-based LLMs: making these models deployable by reducing computational and memory overhead without significant accuracy loss.\n\nThe proposed method is technically sound and well-motivated, grounded in clear observations of expert redundancy variability across MoE layers.\n\nThe continuous relaxation and differentiable optimization approach is well-integrated and executed with practical efficiency.\n\nEmpirical evaluation is conducted across diverse model families and tasks, including ablation and visualization studies that substantiate the claims.\n\n**Weakness**\n\nThe method introduces additional training overhead (although minor) and requires calibration data, which might not be readily available in all deployment settings.\n\nThe paper could provide more theoretical guarantees or robustness analysis, especially under domain shift or data drift conditions.\n\nWhile the visualization analyses are compelling, deeper analysis on why certain layers or experts are deemed more important would strengthen the interpretability of the method.", "was_revised": true, "final_justification": "The authors' rebuttal addressed my concerns. I changed my score to 5 and suggest an acceptance.", "rebuttal": "We thank Reviewer 3XFq for the thoughtful feedback. Below, we address each concern.\n\n**W1: Training Overhead & Calibration Data**\n\nWe agree that practical compression methods should incur minimal cost. DiEP is designed as a **lightweight, one-shot pruning** approach.\n\n**1. Efficient Training**\n\nDiEP’s calibration is fast and cost-effective:\n\n**Cost Comparison (Appendix A.3)**: On Mixtral 8x7B, DiEP takes just **0.23 hours / 31.97 GB·h**, a **74% reduction** over NAEE (124.58 GB·h).\n\n**Quality vs. Cost**: For similar cost to S-MoE, DiEP achieves **+4.0 accuracy**. Compared to MC-MoE, it's **70% cheaper** with **+28 points** accuracy gain.\n\nThus, DiEP offers **better performance at lower overhead** than many training-free methods.\n\n**2. Minimal Calibration Data**\n\nAs shown in **Table 9 (Appendix A.6)**, DiEP achieves optimal performance on Mixtral 8x7B with just **128 samples**. This is a tiny, almost negligible amount of data that can be easily sourced. The table also shows that even with as few as **32 samples**, DiEP avoids performance collapse and still functions effectively.\n\n**Task-Agnostic Data**: Our domain generalization test (see W2) shows DiEP pruned on **C4** performs well on **MathQA/CommonseQA**, indicating no need for in-domain data.\n\n\n**Conclusion**: The overhead and data requirements of DiEP are intentionally designed to be minimal to ensure broad applicability. It uses minimal task-agnostic data, and delivers superior pruned models.\n\n---\n\n**W2&Q1: Robustness & Theory**\n\nWe thank the reviewer for raising the importance of robustness under domain shift. DiEP’s design encourages generalization both **theoretically and empirically**.\n\n**1. Theory: Stable & Knowledge-Preserving**\n\n**Convergence Proof** (Appendix B.2): Our alternating update converges to a stationary point under standard conditions.\n\n**Knowledge-Preserving Regularization:** The key to DiEP's robustness lies in our objective function: $$ \\min_{\\alpha, \\beta} \\mathcal{L}_{ce}(y, F'(x; \\alpha, \\beta)) + \\lambda \\cdot \\Phi(\\alpha, \\beta) $$The term $\\Phi(\\alpha, \\beta) = ||F'(x; \\alpha, \\beta) - F(x)||_F$ acts as a powerful **knowledge-preserving regularizer**. which transfers generalization to the pruned model $F′$, mitigating overfitting to limited calibration data.\n\nFuture plans include extending to **stochastic settings**, analyzing **non-convex landscapes**, and formalizing the role of λ.\n\n**2. Empirical: Robustness to Domain Shift**\n\nWe simulate domain shift by pruning on general (C4), in-domain (Math), or mixed (C4+Math) data, and testing on **CommonseQA & MathQA**.\n\n**Table 1: Pruning on General (`C4`)**\n\n|Method|Pruning|CommonseQA|MathQA|\n|---|---|---|---|\n|Full Model|-|66.83|42.38|\n|NAEE|50%|42.01|34.86|\n|**DiEP**|**50%**|**51.52 (+9.51)**|**35.02 (+0.16)**|\n|NAEE|25%|63.31|41.04|\n|**DiEP**|**25%**|**65.52 (+2.21)**|**41.77 (+0.73)**|\n\nDiEP on general data retains performance, beating NAEE by **+9.51** points under aggressive pruning.\n\n**Table 2: Pruning on In-Domain (`Math`)**\n\n|Method|Pruning|CommonseQA|MathQA|\n|---|---|---|---|\n|Full Model|-|66.83|42.38|\n|NAEE|50%|53.40|35.97|\n|**DiEP**|**50%**|**56.27 (+2.87)**|**37.19 (+1.22)**|\n|NAEE|25%|60.21|42.20|\n|**DiEP**|**25%**|**65.27 (+5.06)**|**42.34 (+0.73)**|\n\nEven when biased toward Math, DiEP generalizes well to CommonseQA.\n\n**Table 3: Pruning on Mixed (`C4+Math`)**\n\n|Method|Pruning|CommonseQA|MathQA|\n|---|---|---|---|\n|Full Model|-|66.83|42.38|\n|NAEE|50%|51.27|35.00|\n|**DiEP**|**50%**|**58.67 (+7.40)**|**36.86 (+1.86)**|\n|NAEE|25%|61.79|41.43|\n|**DiEP**|**25%**|**64.98 (+3.19)**|**42.69 (+1.26)**|\n\nDiEP improves further with in-domain data; in fact, **outperforms full model** on MathQA.\n\nWe will highlight these as **\"robustness to domain shift\"** results in the paper.\n\n---\n\n**W3: Deeper Analysis on the Importance of Layers and Experts**\n\nWe thank the reviewer for highlighting the importance of interpretability. Understanding _why_ certain experts and layers are retained by DiEP provides valuable insight into the method’s effectiveness. We offer the following analysis based on the learned α and β scores.\n\n**1. Why DiEP Prioritizes Shallow Layers (High β Scores)**\n\nAs shown in **Figure 4b**, DiEP assigns consistently higher β scores to earlier layers (roughly 1–15). This pattern aligns with well-established findings on the **hierarchical structure of Transformers**:\n\n**Shallow layers** are known to extract foundational syntactic and semantic features [1–3], forming the base for all subsequent reasoning. These layers handle diverse, low-level signals shared across tasks and domains. **Preserving experts** in these layers is essential, as early-stage degradation can propagate and amplify in deeper stages.\n\n**Deeper layers**, in contrast, operate on refined features and are responsible for more abstract contextualization. Their computations tend to be more task-specific and **less universally critical** [4], making them more suitable targets for pruning.\n\nThus, DiEP’s β scores reflect a principled, data-driven retention of layers based on their **functional necessity**, not just position.\n\n---\n\n**2. Why DiEP Prefers Certain Experts (High α Scores)**\n\n**Figure 4a** reveals notable variance in intra-layer α scores, with only a subset of experts per layer being deemed essential. This supports the MoE hypothesis of **functional specialization** [5, 6]:\n\n**Generalist experts** exhibit high α scores across data points, likely specializing in high-frequency, broadly useful linguistic patterns. **Specialist experts** may focus on niche phenomena and are invoked less frequently.\n\nCrucially, DiEP’s joint modeling of α (expert importance) and β (layer importance) allows **fine-grained pruning decisions**:\n- A layer may be important (high β), but only a few of its experts (high α) are truly impactful.\n- DiEP thus preserves these \"hub\" experts while pruning redundant or less frequently activated ones.\nThis **coupled, multi-scale selection** is a key strength of DiEP: it balances global layer utility with local expert relevance, leading to sparse yet highly functional subnetworks.\n\n---\n**Conclusion**\n\nThe α–β scores learned by DiEP reveal how MoE models encode hierarchical and distributed knowledge:\n- **β scores** prioritize depth-wise importance, aligning with representational hierarchy.\n- **α scores** capture intra-layer specialization, aligning with expert sparsity and task coverage.\n\nBy jointly modeling both, DiEP achieves **interpretable and structured pruning**. \n\n**References**  \n[1] BERT Rediscovers the Classical NLP Pipeline. ACL 2019.  \n[2] Analyzing Multi-Head Self-Attention. EMNLP 2020.  \n[3] A Primer in BERTology. TACL 2020.  \n[4] Sparsely-Gated MoE Layer. ICLR 2017.  \n[5] Ultimate Expert Specialization in MoE LMs. arXiv:2401.06066.  \n[6] Efficient Expert Pruning for MoE LLMs. ACL 2024.\n\n---\n\n**Q2: Adaptive Pruning Ratio**\n\nWe thank the reviewer for this insightful question. Below we elaborate on both current DIEP capabilities and potential extensions.\n\n**1. Dynamic Ratios via Threshold-Based Pruning (Already Supported)**\n\nDiEP's core output is a continuous importance score ($s_i^{(l)}$) for each expert. Instead of pruning a fixed percentage (top-K), one can simply set an **importance threshold, T**, and prune all experts with scores below it. This naturally leads to a dynamic pruning ratio that adapts to the model's specific redundancy profile, a flexibility already supported by our method.\n\n**2. Learning the Optimal Ratio via Differentiable Regularization**\n\nFurthermore, our differentiable framework can be extended to learn the pruning ratio directly. We propose a simple yet powerful extension:\n\n$$ \n\\min_{\\alpha, \\beta} \\mathcal{L}_{ce}(y, F'(x; \\alpha, \\beta)) + \\lambda \\cdot \\Phi(\\alpha, \\beta) + \\gamma \\cdot \\mathcal{L_s}(\\alpha, \\beta) \n$$\n\nThe added term is $\\mathcal{L_s}(\\alpha, \\beta) = \\sum_{l=1}^{L} \\sum_{i=1}^{N} |s_i^{(l)}| = \\sum_{l=1}^{L} \\sum_{i=1}^{N} |\\bar{\\alpha}_i^{(l)} \\cdot \\beta^{(l)}|$, which encourages the model to shrink unimportant scores. The final pruning ratio thus becomes an **emergent property** of the learned solution, shaped by the calibration data and task constraints.\n\nThis formulation allows the model to **\"justify\" the retention of each expert**, resulting in pruning strategies that are both principled and task-adaptive.\n\n**Conclusion**\nThis is a valuable direction for future work that our framework is well-positioned to explore. We will add this discussion to our revised manuscript.\n\n---\n\n**Q3: Continual Learning Evaluation**\n\nWe thank the reviewer for this critical question. To assess DiEP's robustness to shifting expert utility, we conducted a **continual learning experiment** on Hellaswag, BoolQ, and MathQA as the initial calibration dataset. We then evaluate the performance of the pruned models to measure how well they generalize and retain knowledge. A lower \"Forgetting\" score indicates better knowledge preservation. \n\n**Experimental Results:** The tables below show the performance (25%/50% ratio).\n\n**NAEE**\n\n|Calib.|Hellaswag|BoolQ|MathQA|\n|---|---|---|---|\n|hellaswag|58.8 / 55.9|-|-|\n|boolq|58.4 / 55.5|84.1 / 81.2|-|\n|mathqa|58.1 / 54.9|83.7 / 80.6|41.1 / 37.2|\n\n**DiEP**\n\n|Calib.|Hellaswag|BoolQ|MathQA|\n|---|---|---|---|\n|hellaswag|**60.0 / 57.0**|-|-|\n|boolq|**59.7 / 56.7**|**85.2 / 82.8**|-|\n|mathqa|**59.4 / 56.3**|**84.8 / 82.3**|**42.4 / 38.4**|\n\n**Summary**\n\n|Method|Avg. Acc. (25%/50%)|Forgetting (25%/50%)|\n|---|---|---|\n|NAEE|61.0 / 57.6|0.55 / 0.80|\n|**DiEP**|**62.2 / 59.0**|**0.50 / 0.60**|\n\nThe results clearly show that DiEP is more robust in a continual learning setting. By regularizing towards the original model's function, DiEP learns to preserve experts that are fundamentally important for general capabilities, rather than just for the specific calibration task. This makes our pruning method inherently more robust to the challenges of continual learning."}, {"review_id": "Hd0LwCpX6N", "reviewer": "Reviewer_sDD7", "summary": "This paper proposes DiEP, a differentiable expert pruning framework for compressing sparse Mixture-of-Experts (MoE) models. Unlike existing methods that impose uniform pruning across layers or rely on costly discrete search, DiEP introduces a two-level optimization strategy: intra-layer expert importance scores (α) and inter-layer importance modulation (β). These are optimized on a small calibration dataset using a lightweight reconstruction-based loss. A novel expert skipping mechanism is also proposed to further reduce inference latency by dynamically skipping experts based on routing weight ratios and precomputed similarity statistics. DiEP is shown to achieve competitive performance (e.g., 92% retention with 50% experts) across multiple large-scale MoE models such as Mixtral, DeepSeek-MoE, and Qwen2.", "questions": "- Could you elaborate on the γ computation, particularly the role of γ₂ derived from CKA similarity? My understanding is that γ₂ is computed offline using the calibration dataset and remains fixed during inference. If so, how can this static similarity-based threshold generalize to unseen prompts or tasks at inference time? Is there theoretical or empirical evidence supporting the transferability of γ₂ across different inputs?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The paper proposes a novel method. Reformulating expert selection as a differentiable global optimization problem is both novel and practically effective.\n- The method introduces comparatively minimal overhead while delivering notable acceleration across various models.\n- The paper is well-structured and supports the proposed method with comprehensive experiments.\n\n### Weaknesses\n\n- The finetuning cost of the proposed technique remains unclear. \n- Some design choices lack rigorous justification—for example, the use of α and β to quantify expert and layer importance, as well as the rationale behind the choice of the α:β update ratio.", "was_revised": true, "final_justification": "The author's response has addressed the majority of my concern. Therefore, I maintain the scoring and recommend acceptance.", "rebuttal": "We thank reviewer sDD7 for providing positive feedback and helpful suggestions. Below please find a response.\n\n**W1: Concerns on the Pruning Cost**\n\nWe appreciate the reviewer's feedback and the opportunity to provide a detailed analysis of the computational cost of our method. We totally agree that analyzing pruning cost is crucial. Our experiments show DiEP is highly efficient, providing a superior cost-accuracy balance. We use **Memory-hour Cost (GB·h)**—a metric capturing both peak GPU memory and time—to holistically measure the total computational effort on two distinct MoE models.\n\n**Table 1: Pruning Cost and Accuracy on Mixtral 8x7B**\n\n| Method | Peak Memory (GB) | Time (h) | Memory-hour Cost (GB·h) | Average Accuracy |\n| :--- | :--- | :--- | :--- | :--- |\n| NAEE | 95.1 | 1.31 | 124.58 | 54.6 |\n| MC-MoE | 348.4 | 0.31 | 108.00 | 31.9 |\n| S-MoE | 118.3 | 0.27 | 31.94 | 55.9 |\n| **DiEP (Ours)** | **139.0** | **0.23** | **31.97** | **59.9** |\n\n**Analysis:**\n*   **Superior Efficiency:** DiEP achieves a memory-hour cost of just **31.97 GB·h**. This is a **74% reduction** compared to NAEE and a **70% reduction** compared to MC-MoE.\n*   **Best-in-Class Performance-per-Cost:** While DiEP's cost is on par with S-MoE, it achieves a significantly higher accuracy (**59.9 vs. 55.9**). This demonstrates that DiEP offers the best trade-off, delivering the highest accuracy for a given, minimal computational budget.\n\n**Table 2: Pruning Cost and Accuracy on Deepseek-MoE16B**\n\n| Method | Peak Memory (GB) | Time (h) | Memory-hour Cost (GB·h) | Average Accuracy |\n| :--- | :--- | :--- | :--- | :--- |\n| MC-MoE | 210.42 | 0.42 | 88.38 | 48.8 |\n| S-MoE | 56.73 | 0.36 | 20.42 | 52.7 |\n| **DiEP (Ours)** | **55.43** | **0.28** | **15.42** | **54.1** |\n\n**Analysis:**\n*   **Scalablility:** The efficiency gains of DiEP are even more pronounced on models with a larger number of experts. DiEP's memory-hour cost is only **15.42 GB·h**, which is **24% lower** than the next most efficient method, S-MoE (20.42 GB·h).\n*   **Highest Accuracy at Lowest Cost:** Crucially, DiEP not only has the lowest pruning cost but also achieves the highest accuracy (**54.1**), improving upon S-MoE's accuracy by **+1.4** points while being more efficient.\n\n**Addressing the Hidden Costs of Other Methods:**\n\nIt is important to note that methods like S-MoE, while appearing fast, have hidden computational costs that scale poorly. S-MoE's multi-step expert merging process involves:\n1.  **Expert similarity analysis (CKA):** This requires $O(S \\cdot N^2)$ computations, where S is the number of samples and N is the number of experts. This quadratic scaling with the number of experts makes it increasingly expensive for modern MoE architectures.\n2.  **Learning merging coefficients:** This introduces an additional optimization overhead.\n\nIn contrast, DiEP's gradient-based approach avoids these expensive, quadratically scaling steps. Its computational cost is primarily determined by a few forward/backward passes on a small calibration set, making it highly scalable and efficient.\n\n**Conclusion:**\nThe experimental data clearly demonstrates that the \"finetuning\" or pruning cost of DiEP is not a concern; rather, it is a significant strength. It delivers state-of-the-art accuracy at a minimal and scalable computational cost, making it a practical solution for task-agnostic deployment.\n\n\n\n**W2: Justification for Design Choices**\n\nWe thank the reviewer for this insightful question regarding our core design principles. These choices were motivated by the multi-scale nature of expert redundancy in MoE models.\n\n**1. Rationale for `α` (Intra-layer) and `β` (Inter-layer) Scores**\n\nWe decompose expert importance into two components to model redundancy at different granularities:\n*   **`α` (Intra-layer scores):** Learns the *local, relative importance* of experts competing within the same layer.\n*   **`β` (Inter-layer scores):** Learns the *global, depth-aware importance* of entire layers, addressing the fact that shallower layers are often more critical than deeper ones (as shown in our Figure 4).\n\nBy combining them ($s_i^{(l)} = \\bar{\\alpha}_i^{(l)} \\cdot \\beta^{(l)}$), we create a unified score that enables a **principled, non-uniform pruning** across the entire model in a single, global sort. This factorized approach is designed to be more expressive and effective than using a single, monolithic importance score.\n\n**2. Rationale and Empirical Validation for the `α:β` Update Ratio**\n\nThe update ratio is an empirical choice aimed at achieving **optimization stability**. The high-dimensional `α` scores learn fine-grained expert rankings, while the low-dimensional `β` scores make coarse-grained, systemic adjustments. Our hypothesis is that allowing `α` to update more frequently helps stabilize the local expert rankings before the more impactful `β` scores are adjusted.\n\nTo rigorously justify our choice, we conducted a new **ablation study on the `α:β` update ratio**. The results, shown below, are for 25% and 50% pruning ratios (formatted as 25%/50%).\n\n| `α`:`β` Ratio | MMLU | BoolQ | OpenBookQA | RTE | Avg |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| 1:1 | 65.1 / 54.3 | 85.2 / 81.8 | 31.8 / **29.8** | 67.2 / 65.4 | 62.3 / 57.7 |\n| 1:2 | 64.5 / 54.8 | 85.5 / 79.5 | 31.8 / 27.6 | 69.0 / 64.5 | 62.7 / 56.6 |\n| 2:1 | **66.2** / 56.5 | 85.1 / 82.5 | 32.4 / 28.9 | **71.2** / 67.2 | 63.7 / 58.7 |\n| 2:2 | 65.3 / 55.1 | 85.6 / 83.9 | 32.0 / 27.8 | 67.9 / 65.1 | 62.7 / 57.9 |\n| **3:1** | 64.9 / **57.9** | **86.6** / **84.0** | **33.1** / 29.6 | 70.7 / **68.2** | **63.8** / **59.9** |\n| 4:1 | 65.2 / 56.7 | 85.3 / 83.7 | 31.4 / 29.4 | 70.8 / 66.1 | 63.1 / 58.9 |\n\n**Key Findings:**\n\n*   The experimental results clearly validate our design choice. The **3:1 update ratio consistently achieves the best or near-best performance across almost all tasks and pruning ratios**, culminating in the **highest average scores** for both 25% (**63.8**) and 50% (**59.9**) pruning.\n*   The data shows that giving more updates to the intra-layer scores (`α`) generally leads to better performance (e.g., 2:1 and 4:1 outperform 1:1 and 1:2). This supports our hypothesis that stabilizing the local expert rankings is crucial.\n*   The 3:1 ratio strikes an optimal balance, outperforming both less frequent (e.g., 2:1) and more frequent (e.g., 4:1) `α` update schedules in terms of average performance.\n\nIn conclusion, our `α`/`β` decomposition provides a principled way to model multi-scale redundancy, and our chosen 3:1 update ratio is not arbitrary but is empirically validated to be the most effective schedule for stable and high-performing optimization. We will incorporate these new results into our revised manuscript.\n\n\n\n\n**Q1: Elaboration on $γ_{2}$**\n\nWe sincerely thank the reviewer for this insightful question. It correctly identifies a crucial aspect of our adaptive skipping mechanism and allows us to elaborate on the design rationale and provide new empirical evidence for its effectiveness.\n\nIn fact, traditional routing weights effectively measure the *importance* of an expert for a given token. However, they do not capture the *redundancy* between the selected experts. A scenario can easily arise where the router assigns high scores to two experts (e.g., top-2) that are functionally very similar. In this case, activating both is computationally wasteful. The core motivation for our design is precisely this: **to prune not just unimportant experts, but also redundant ones.** This is where $γ_{2}$ comes in. It acts as a **redundancy penalty**. \n\n**Generalizability of$γ_{2}$ (CKA Similarity): A Task-Agnostic Structural Prior**\n\nYou are correct that $γ_{2}$ is computed offline on a calibration set and remains fixed. The key to its generalizability lies in what it represents: $γ_{2}$ is not a data-specific feature but rather a measure of the **inherent, task-agnostic structural similarity** between expert weight matrices.\n\nThe expert weights are the result of pre-training on trillions of tokens. The functional similarities and differences captured by CKA are therefore a deeply embedded, stable property of the model itself. While the *activation* of experts is dynamic, their fundamental functional relationship is static. Therefore, $γ_{2}$ acts as a robust and transferable structural prior that remains valid across unseen prompts and tasks.\n\nHere we conducted an ablation study as **empirical evidence** based on the NAEE common framework[1] . We started with the full model and then compared the performance of a skipping mechanism based solely on routing weights ($γ_{1}$) against our proposed method which combines both routing weights and similarity ($γ_{1} \\times γ_{2}$).\n\n| Method                    | Avg. Acc | Speedup   |\n| :------------------------ | :------- | :-------- |\n| Full Model (No Skip)      | 65.1     | 1.00X     |\n| $γ_{1}$                   | 63.7     | 1.06X     |\n| $γ_{1} \\times γ_{2}$  | **64.1** | **1.07X** |\n\n**Analysis of Results:**\n*   **$γ_{2}$ Improves Accuracy:** The results clearly show that incorporating the similarity-based $γ_{2}$ leads to better performance. Our method achieves an average accuracy of **64.1**, recovering **0.4 points** of the accuracy lost by the $γ_{1}$-only approach, while still providing a slightly better speedup.\n\n**Conclusion:**\nIn summary, our two-factor $γ$ is a principled design. $γ_{1}$ (from routing weights) provides a dynamic, per-token importance signal, while $γ_{2}$ (from CKA) provides a static, generalizable structural redundancy penalty. The empirical evidence confirms that this combination leads to a more robust and accurate adaptive skipping mechanism that successfully transfers to unseen inputs at inference time. We will add this detailed justification and ablation study to our revised paper.\n\n[1]. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. ACL 2024."}]

Author rebuttal:

Meta-review: This paper received high ratings of BA, A, and A. Additionally, concerns raised by the reviewers have been effectively addressed in the rebuttal. The decision is to accept.
Decision: Accept (poster)