Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
Authors: Mingyu Cao
Gen Li
Jie Ji
Jiaqi Zhang
Ajay Jaiswal
Li Shen
Xiaolong Ma
Shiwei Liu
Lu Yin
Venue: Accepted by TMLR ()

Abstract: Mixture-of-Experts (MoE) has garnered significant attention for its ability to scale up neural networks while utilizing the same or even fewer active parameters. However, MoE does not alleviate the massive memory requirements of networks, which limits their practicality in real-world applications, especially in the era of large language models (LLMs). While recent work explores the possibility of removing entire layers of MoE to reduce memory, the performance degradation is still notable. In this paper, we propose ConDense-MoE (CD-MoE), which, instead of dropping the entire MoE layer, condenses the large, sparse MoE layer into a smaller, denser layer with only a few experts activated for all tokens, while maintaining hardware friendliness. Our approach is specifically designed for fine-grained MoE with shared experts, where Feed-Forward Networks are split into many small experts, with certain experts isolated to serve as shared experts that are always activated, such as DeepSeekMoE and QwenMoE. We demonstrate the effectiveness of our method. Specifically, for the DeepSeekMoE-16B model, our approach maintains 90% of the average accuracy while reducing memory usage by 27.5% and increasing inference speed by 1.26 times. Moreover, we show that by applying lightweight expert fine-tuning—only to the condensed layers—and using 5 hours on a single 80G A100 GPU, we can successfully recover 98% of the original performance.

Published in Transactions on Machine Learning Research (10/2025)
Condense, Don't Just Prune: Enhancing Efficiency and Per- formance in MoE Layer Pruning
Mingyu Cao University of Surrey
m. m.cao
Gen Li Clemson University
gen@g.clemmso.. edu
Jie Ji Clemson University
edu
Jiaqi Zhang Meituan
ahougjaaq33@@mmciiaan.comm
Ajay Jaiswal University of Tezas at Austin
ajoggiasnual@nutems..dd
Shen
mathshendi@@mail.ccm
Sun Yat-sen University
Xiaolong Ma The University of Arizona
zicolongmooBarizonn...l
Shiwei Liu ELLIS Institute Tübingen, Max Planck Institute, for Intelligent Systems, Tübingee Al Center
sliu@tue. wlweeelllsse.... eu
Lu Yin
1. Lyin@surrey.aac.um uk
University of Surrey
Reviewed on OpenReview: https: // openreview. net/ formm? id- BQe6
Abstract
Mixture-of-Experts (MoF) has garnered significant attention for its ability scale up neural networks while utilizing the same even fewer active parameters. However, MoE does not relieve the massive memory requirements of networks, which limits their practicality in real-world applications, especially in the era of large language models (LLMs). While recent. work explores the possibility of removing entire layers of MoE to reduce memory, the performance degradation still notable. In this paper, we propose ConDense- MoE (CD-MoE) that, instead of dropping the entire MoE layer, condenses the large, sparse MoE layer into smaller, denser layer with only few experts activated for all tokens, while maintaining hardware friendlincss. Our approach is specifically designed for fino-grained MoE with shared experts, where Feed-Forward Networks are split into many small experts, with certain experts isolated to serve as shared experts that are always activated, such as DeepScckMoE and QwenMoF. We demonstrate the effectivencss of our method. Specifically, for the DeepSeekMoEE1611 model, our approach maintains 90% of the average accuracy while reducing memory usage by 27.5% and increasing inference speed to 1.26 times. Moreover, we show that by applying lightweight expert fine-tuning only the condensed layers and using 5 hours performance. Our code is available al: https: htpp:://githui com (omm//tteerrscccCC-Moo...
single 80G A100 GPU,
successfully recover 98% of the original
* *Corresponding author.

---

Published in Transactions on Machine Learning Research (10/2025)
1 Introduction
Large Language Models (LLMs) continue to grow in both size and capability (Brown 2020 Touvron et al.
2023b fueling demand for architectures that can scale with minimal increases in computational cost.
Mixture of Experts (MoE) architectures have emerged as a promising solution to this challenge (Shazeer et al.
2017 Jiang et al. 2024 Team et al. 2024]. Unlike standard dense networks, MoEs selectively activate only the most relovant subset of parameters (referred to as "cxperts") for a given input, enabling substantial growth in model capacity without a proportional escalation in compute overhead. Recent advances in fine-grained expert segmentation (Dai et al. 2024 Team 2024 push this concept further by splitting feed-forward layers into many small experts, while designating small set of shared experts that remain active across all tokens. Despite the clear benefits, MoE architectures still face major deployment barriers due to high memory costs. Storing a large number of experts, even if most remain dormant per token, can be prohibitively expensive in real-world systems. This reality highlights the urgent need for approaches that preserve the performance benefits of MoEs while lowering their memory footprint.
In this paper, present new framework called ConDense-MoB (CD-MoE), novel framework that significantly euhances the efficacy specifically for DeepSeek-style shared-expert-type MoE (Dai et al. 2024 Team 2024). Unlike prior MoE compression approaches that often remove entire layers or relain the routing mechanism with fewer experts (He et al. 2024b), our method eliminates the routing process at selected layers and condenses them into dense layers. Specifically, we devisc a greedy strategy to identify which layers and which experts within them can be condensed with minimal impact on model accuracy, leading to significant. reductions in memory usage and faster inference.
MoE Layer with Shared Experts
Condensed MoE Layer
expert
shared expert
shared expert
Wo
expert
expeit
W;
expert
W2
expprrt
router
selected export K
expert
topk important experts all route experts
W43
expert 63
input
tapk
output
input
output
Figure 1: Left: The structure of the Deepseek MoE layer. w/ represents the weights after normalization. Right:
The structure of the ConDense-Moli layer, where only the most important top-k experts retained. úú represents the fixed weights that are pre-computed during condensing using the average weight of all calibration Lokens.
CD-MoE stems from surprising yet critical insight: in shared-expert lype MoEs, simply removing the routing mechanism while retaining only the shared experts at given layer generally incurs minor output difference. Building on this, CD-MoE preserves these essential shared experts and a small subset of routing experts, thereby condensing originally sparse layers into compact, dense analogs. As these preserved experts account for only a small fraction of overall parameters, we substantially cut memory requirements without sacrificing much performance. Our experiments demonstrate that on DeepSeekMoB- reduces memory by 27.5% and maintains nearly 90% of the original accuracy suite of downstream tasks. Moreover, our method uncovers additional experts that are beneficial for fine-tuning, enabling up to 98% recovery of the original performance after lightweight supervised finc-tuning that updates only the condensed layers. In summary, our contributions are:
(Dai et al.) (2024), CD-MoE
A novel MoE condensation approach that selectively removes less critical experts and condenses large sparse MoF. layers into small, dense structures.
2

---

Published in Transactions on Machine Learning Research (10/2025)
An efficient selection algorithm that identifies which layers to condense and which experts to preserve, achieving a favorable trade-off between memory savings and accuracy.
Empirical validation on DeepSeekMoE-1633 demonstrating 27.5% memory reduction and 1.26x inference speed, while retaining up to 90% of the zero-shot accuracy. Moreover, lightweight fine-tuning can effectively reclaim 98% of the original performance using only single A100 GPU in few hours.
2 Related Work
Mixture-of-Experte (MoB) architectures have become a powerful framework for scaling neural networks without linearly increasing computational burden (Shazeer et al., 2017 Jiang et al. 2024 Dai et al.) 2024
Team 2024). Farly methods such as Switch Transformer &Fedus al. 2022) demonstrated that sparse activation can substantially boost model capacity while maintaining tractable compute. Further work has refined routing policies ęJiang et al.] 2024) and introduced fine-grained segmentation of feed-forward layers into many small experts, often augmented with "shared" experts that remain active for every token (Dai ct al. 2024 Team 2024). Those advances collectively enhance training stability and generalization, yet the growing pool of experts poses serious deployment challenges, chiefly due to large memory demands.
A range of pruning and compression strategies have emerged to address these overheads. Some works prune experts by identifying those that are minimally activated or less essential for downstrcam tasks (Chi et al. 2022: Muzio al.] 2024), with solutions ranging from exhaustive searches over all expert subsets 2024 to filtering experts by their activation frequency. While these methods typically preserve routing, more aggressive approaches prune entire MoE layers, achieving considerable memory reductions but at the risk of pronounced accuracy losses (He et al. 2024bb. In contrast, our technique selectively eliminates the routing mechanism in particular layers and retains only a small subset of experts (including the shared experts), thus reducing memory demands while sustaining most of the model's representational power.
Sun al.,
2024 [Lu et al.
3 ConDense-MoE (CD-MoE)
In this section, we present a comprehensive explanation of the CD-MoE process, which follows a sequential workflow comprising several critical steps. First, we delve into Expert Selection and Condensing (Section 3.20, where we detail the methods used identify and condense the most essential experts from all available experts within layer. Second, we explore Layer Selection and Condensing (Section 3.3), focusing selecting the most suitable layers for condensation to preserve as much of the LLM's inherent capabilities possible. Last but not least, we highlight that only performing lightweight fine-tuning (Section 4.3 on condensed layers,
can almost recover the original performance of MoE.
3.1 Preliminaries of MoE
Here we define the basic structure of an MoE' layer. All experts in an MoE layer are represented as {E1, E,,.., }. For most fine-grained MoE models, in addition to routing experts, the shared experts E, is used. Any token will activate the shared experts and several routing experts, as shown in Figure left. For the input data X, the output computed as follows:
h
Σ (gr. E. (x+)) + Eg (xi) + xt.
(1)
where
Sitt:
85, E TopK (($5, 1 S SN)A),
0, otherwise,
Si.t Softmax, (x7 e:).
Here, N denotes the total number of route experts, E₃(-) represents the i-th expert, E, is the shared expert, Iitt is the gate value for the i-th expert, 8L, denotes the token-to-expert similarity, TopK(:,k) represents the
3

---

Published in Transactions on Machine Learning Research (10/2025)
set consisting of the k highest similarity scores among those computed for the t-th token and all N experts, and is the centroid of the i-th expert.
3.2 Expert Selection and Condensing
Algorithm 1: Condense Experts Selection
Input: Calibration data input X, routing expert Eroute (Ei...., Enj, number of selected experts K
Output: Condense expert set Econdense (Eshared)
Initialize Econdense + Esharedy, Ozet + Layer route (X) // //Compute reference output with all routing experts for k to K do for each expert Es E Eroute do Eccoueasee Beondense U {Fi} //Temporarily add expert condense
Oump &-
Layerroonenne (X) b //Compule output with current condense experts loss; t JS(Oret, Oemp) > //Compute JS divergence as loss for E;
Econdenan Econdense {E.) > //Remove expert after computing loss
Ebeat <- aargmit E Fremea loss; D //Select the expert with minimum loss Econdease &- Eeondense U [Ebeat] b //Add best expert to condense set Eronte Eroute [Erea) b //Remove selected expert from routing set
Our CD-MoE framework removes the standard routing mochanism during inference. For cach expert E,, we use open-source C4 calibration data to compute the mean of gr. when that expert is activated, which is then used the fixed gate value da for that expert during the inference phase:
9: = (1/1t g6. # OO)
Σ
(2)
0/0...00
For the condense layer, we only retain K routing experts including the shared experts, which is consistent. with the number of active experts during the training phase. Additionally, benefiting from sufficient training during the training stage, the shared expert is retained, as shown in Figure I right. After condensation, the MoE layer represented as:
K h = Σ E, + E. (xi) +x
(3)
Next, we describe how to select the most crucial experts. Lu et al. 2024) explore all possible expert combinations and choose the combination of experts that makes the layer outputs before and after pruming as close as possible, while pruning the rest. However, for tne-grained MoE models like DeepSeekMoE- 16B (Dai et al. 2024) and Quenn.5-MoE-A2... 72 [Team] 2024), where the number of experts often exceeds 60, testing all permutations of experts requires significant computational resources and time. Therefore, we propose using a greedy search approach Algorithm I to select the most critical experts.
In which, is consistent with the number of routing experts(non-shared experts) activated in the original model, and JS(()) is the Jensen-Shannon divergence used to measure the difference between two outputs. Zhang et al. (2024) shown that in model pruning, this metric is better than angular distance and Euclidean distance. We also demonstrated its superiority over other metrics such Kullback Leibler (KL) divergence and perplexity in Table
3.3 Layer Selection and Condensing
Clark et al. (2019b) have confirmed that different layers in language models exhibit distinct functionalities. While our objective is to retain the most impactful experts during the condensing process of each layer, it is
4

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm 2: Condense Layers Selection
Iuput: Data imput X, routing layer set Lroute = [Li...., Ln); number condensed layers Output: Condense layer set Leondeuse = [}
Initialize Leandense + O. Ont t Model(X) //Compute reference output with all routing layers for 1 to N do for each layer La E Lroute do Leondenee Leondense U (Li} //Temporarily add layer condense set
Oump f- MMoee(X)) b //Compute output with current condense layers loss, < JS(O Oump) > //Compule JS divergence as loss for L; Leondense Leondenne (I.) // //Remove layer after computing loss L.beet f argmin Eroute loss; b //Select the layer with minimum loss Leonndense Leondensn U [Leeett //Add best layer to condense set //Remove selected layer from routing set
Troute Lroute \ [bbest]
inevitable that the outputs will deviate from their original values due to reduced capacity. Indiscriminately condensing all layers can thus lead to significant degradation in model performance and the loss of essential features. Therefore, akin to our approach in expert selection, we prioritize condensing those layers that exert minimal impact on model output changes post-pruning.
Divergence enne
Diverennca
17.5
0.125
1.1
Perp Paplexity
0.8
15
0.075
0.5
0.0025
0.2
10
15
20
25
10
15
20
25
10
20
25
Condense Layer Index
Condense Layer Index
Condense Layer Index
Figure 2: Left: Fluctuations in the JS divergence between the the outputs of the condensed model and the original dense model across different layers. Middle: fluctuations in the KL divergence. Right: fluctuations in the perplexity.
To systematically quantify the impact of condensing individual layers, we conducted preliminary experiments assessing the output changes using the JS divergence, KL divergence and perplexity between the pruned and unpruned model outputs as our evaluation metric. As illustrated in Figure 2 condensing layer 0 results in significant shift in the output distribution, indicating its critical role in the model's performance. While the middle layers exhibit some fluctuation, condensing the last three layers also causes noticeable deviations, suggosting that these layers are essential for maintaining output fidelity. These observations indicate that condensing different layers introduces varying degrees of change in the three metrics, reflecting their varying importance to the overall model performance.
Motivated by these findings, we adopt a greedy search strategy analogous to that used in expert selection to determine the optimal layers for condensing, Specifically, we compute the JS divergence before and after condensing based the output of the final layer, iteratively adding layers to the Layer.codenn set that minimize the distance between the model's final layer output and the output before condensing, as detailed in Algorithm 2 For layers that are not condensed, the standard token routing mechanisms are employed during forward propagation to maintain their full expressive power. For the layers being condensed, the routing mechanism is removed, and the MOE is compressed into a dense structure. The number of condensing layers N in can be chosen based on goals (c.g., memory budget vs. desired accuracy). This selective condensing approach allows us to effectively reduce computational complexity and memory usage while minimizing degradation in model performance.
5

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Results on DeepStckMoo- The underlined variables are tried to kept consistent for fair comparison, and the bolds represent the best results. '#L represents the number of pruned layers; 'ACT! represents the number of activated parameters: and 'MEM' the remaining memory ratio compared to the original model. The lightweight SFT results averaged over three runs with different random sceds, and the standard deviation is indicated below the accuracy.
method
#L ACT. MEM. Speedup ARC-C
BOOLQ
HELLA
MMLU
OBQA
PIQA
RTE
WINO
AVG.
GPT-NEO-2.7R
2.7B
29.8
61.7
55.2
25.0
34.6
72.9
53.4
57.8
48.8
OPT-2.7B
2.7B
31.2
59.9
60.6
25.4
35.2
74.7
54.2
60.5
50.2
BLOOM-3B
3B
30.4
61.4
54.5
25.9
32.4 4
70.7
56.0
58.6
48.7
OPENLLAMA-OE
36.1
64.4
23.9
38.6
75.1
54.2
62.3
52.7
UNPRUNED
2.8B
100%
1.0x
48.3
72.7
77.4
38.3
44.2
78.7
63.2
70.1
61.6
w/o
LRoP
8
2.4B
72.2%
1.34X
35.3
65.4
57.5
29.8
32.6
62.9
48.0
63.6
49.4
BROP
2.3B
71.3%
1.42x
36.2
62.2
57.2
29.0
34.6
70.2
63.2
64.7
52.2
M-SMoE
2.8B
70.0%
1.0x
37.1
64.0
28.0
36.4
59.9
67.2
CD-MoE-S
8
2.4B
73.1%
1.34x
37.8
70.6
65.9
28.8
38.6
75.0
58.8
65.0
55.1
CD-KoE-SR
9
2.8B
72.5%
1.26x
37.9
72.3
64.4
27.1
37.6
75.9
61.7
67.0
55.5
W/ LIGHTWEIGHT SET
LAveRDRop
2.4B
72.2%
1.34X
37.8
71.0
58.9
39.0
34.2
60.8
70.0
54.5
10.7
11.0
4:04
40.7
+0.9
#1.1
#0.4
40.3
10.3
CD-MoE-S
8
2.4B
73.1%
1.34x
41.5
72.0
66.0
39.5
40.3
75.9
67.7
70.0
59.1
+0.7
+0055
#0.55
+0.8
+0.5
#0077
#1.1
20.4
#0.2
CD-KoE-SR
2.8B
72.5%
1.26X
42.4
79.4
66.7
38.5
38.6
76.1
67.0
73.9
60.4
10.6
1:.3
#0.7
1:08
+0.7
#0.3
11.5
3.4 Time Complexity Analysis
The computational efficiency of our greedy search algorithms stems from their polynomial time complexity relative to key model parameters:
For a layer with E routing experts and target selection of K routing experts, each iteration evaluates all remaining candidates. The total complexity is O(K E), which improves upon exhaustive search's O(2B) complexity (Lu al. 2024).
Greedy Expert Selection:
Greedy Layer Selection: With I candidate layers and N target condensed layers, each step evaluates up to L layers. The complexity becomes O(N L).
Taking DeepScekMoE-169 (E 66, L 27) as an example, selecting K = 6 routing experts per condensed layer requires only 369 inferences, demonstrating practical scalability. The linear dependence on E and L makes our approach feasible for billion-paramete MoEs.
4 Experiments
4.1
Experiment Setup
We adopt the open-source models DeepSeckMoE-1ot 16B and Quent.5-MoE-42.77 TTam 2024
(with 16B and 14.3B parameters, respectively) as representative high-capacity examples, and a larger model Qwen2-MoF: 57B-A1/B. We randomly sample 100 examples from the open-source C4 dataset to serve as calibration data during the condensing phase of our model, due to its widespread adoption in the LLM community Sun et al.,
2024 Frantar & Alistarh 2023 Yin et al. 2024).
Evaluation: To provide comprehensive assessment of our model's performance, we follow the protocol established by He et al. (2024b). We report. zero-shot. accuracies eight. diverse tasks selected from

---

Published in Transactions on Machine Learning Research (10/2025)
60
55
55
50
45
40
40
1.0
0.8
0.7
1.0
1.3
14
Memory
SpeedUp
1.0
0.9
0.8
1.0
1.3
1.4
Memory
SpeedUp
Layer Drop
Block Drop
CD-McE-S
CD-MoE-SR
Baseine
Figure 3: CD-MoE against baselines on zero-shot tasks w/o fine-tuinn on DeepSeckMoo- 16B (top) and - 7B (bottom). Left: Average accuracy with varying Memory Ratio against the original model. Right: Average accuracy with varying SpeedUp against the original model. The gray dotted line is the original model result.
the EleutherAl Language Model Evaluation Harness (Gao et al. 2024) which includes the 7 open-source commonsense reasoning benchmark datasets: ARC- ChC-Chale (Clark et al. 2018), BoolQ (Clark et al. 2019a).
HellaSwag (Zellers et al. 2019), OpenBookQA (Mihaylov et al. 2018), PIQA Bisk et al. 2019), RTE Dagan
el al.
2005), WinoGrande (Sakaguchi el al. 2019), and more challenging datasel MMLU (Hendrycks el al.
2021). In addition, we evaluate the reasoning ability of the models on the mathematical task GSM8K (Cobbe
ict al.
2021 and the code generation task HumanEval (Chen et al. 2021
For quality assessment, we use accuracy as the evaluation metric for commonsensse reasoning, MMLU, and GSM8K. For HumanEval, we adopt Pass01 as the metric. To provide more detailed comparison across different pruming methods, employ the following two metrics:
SpeedUp: We perform inference 5,000 times fixed input sequence and measure the average Latency(Original Model)
latency. UpeedUp is computed as SpeedUp Latency(Pruncl) Model]
Memory Ratio:
We measure the peak GPU memory consumption during inference for the full model and each pruned variant using a fixed input. Memory Ratio computed as Memory Ratio = Peak Memory(Pruned Model)
Ponk Memory((riginaa Model)
Additionally, the Recovery Ratio reported in our abstract and experiment analysis is computed Average Accuracy(Pruncl Model)
Average Accuracy(Original Model)
Experimental Methods: We primarily compare our method against the Block Drop and Layer Drop He et al. 2024b) and M-SMoE (Li et al. 2024b). These methods serve as strong baselines for pruning MoE models and are directly relevant to our study. Block Drop and Layer Drop prunc all experts and the routing process in the layer, which is more aggressive than our method, while M-SMoB more conservalively retains the routing process and only merges the experts into a smaller number. We also compared the results with dense models that have similar activations, such as GPT-Neo-2.7B (Gao et al. 2020), OPT-2.7B (Zhang ct al. 2022 BLOOM-3B (Press et al. 2022). and OpenLLaMA-3B [Tovvoon et al. 2023a
For our proposed method, we introduce two variants::
CD-MoE-S: This variant retains only the Shared experts after condensation. The number of shared experts varies across different models, as detailed in Appendix A.1
7

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Results on QwenI.5-MoBBAA. AB. The underlined variables are tried to kept consistent for fair comparison, and the bolds represent the best results. '#L" represents the number of pruned layers; 'ACT! represents the number of activated parameters; and 'MEM' the remaining memory ratio compared to the original model.
method
#I ACT. MEM. Speedup ARC-O
BOoLQ
HEELA
MMLU
ORQA
PIQA
RTR
WINO
AVG.
UNPRUNED
2.7B
100%
1.0X
45.0
79.5
77.3
61.2
43.6
80.3
67.9
69.3
65.5
W/O SFT
LAYERDROP
2.3B
78.3%
1.23X
34.4
62.6
63.1
45.4
32.8
73.6
54.5
62.6
53.6
BLOCK
2.2B
76.1%
1.36x
31.3
62.2
57.4
38.4
32.0
70.7
52.7
58.2
50.4
MC-SMoE 2.7B
73.0%
1.0X
40.9
76.1
71.5
53.2
39.7
77.5
70.5
64.7
61.8
CD-MoE-S
2.5B
78.3%
1.22X
41.1
75.8
72.9
56.0
41.0
78.9
67.2
68.1
62.6
CD-MoE-SR
2.7B
79.7%
1.19x
40.2
77.2
72.1
54.9
39.4
78.0
71.1
66.4
62.4
Table 3: Results on Quen2-MoB-57B-A14 The notations (#L, ACT., MEM.) and formatting conventions
follow Table
method
#L ACT.
MEM.
Speedup ARC-C
nOoLQ
IELLA
OBQA
PIQA
RTE
AVG.
UNPRUNED
14.2B
100%
1.0X
55.6
88.2
82.3
45.6
82.0
76.9
71.8
W/ SFT
LAYERDROP
11.6B
79.3%
1.24x
42.9
77.3
69.0
37.6
76.0
62.1
BLOCKDROP
11.4B
79.0%
1.35x
28.3
61.3
51.6
32.6
62.7
59.9
49.7
MC-SMoE
14.2B
76.4%
1.0x
53.2
87.5
77.4
42.0
80.2
79.4
70.0
CD-MoE-S
12.9B
81.6%
1.22x
54.0
86.7
79.1
43.4
81.5
78.5
70.5
CD-McE-SR
14.2B
83.9%
1.20X
54.1
86.6
78.6
43.0
80.7
80.2
70.5
This variant selects not only the Shared expert(s) but also an additional Routed experts, preserving them as dense. Here, K consistent with the number routed experts activated in the original model, i.c., K Number of Activated Experts Number of Shared Exports (sce Appendix A.1
CD-MoE-SR:
4.2 Zero-Shot Evaluation
To assess the effectiveness of our condensation approach in purely zero-shot setting, we evaluate CD-MoE-S and CD-MoE-SR on eight downstream tasks without any additional fine-tuning. Table shows the results on DeepSeekkMoB-1611 we report each model's performance alongside its memory reduction ratio. Because pruning different layers can lead to varying memory footprints, we normalize the number of pruned layers so that all methods operate under comparable memory budgets.
Under similar memory constraints, both CD-MoE-S and CD- CD-MoE-SR consistently outperform the baselines- Block Drop, Layer Drop and M-SMoF- M-SMMF--highiihhtin the importance of preserving the most informative MoE experts. Moreover, Figure 3 plots the average accuracy against increasing pruning rates and memory savings, illustrating that the performance gap between CD-MoE and conventional pruning techniques widens as pruning becomes more aggressive. Comparison to Small Dense LLMs: Notably, even without fine-tuning, CD-MoE-S and CD-MoE-SR often surpass smaller dense LLMs of comparable size (e.g., BLOOM-3B, OpenLLaMA-3B). This suggests that training dense model from scratch at reduced scale does not match the performance of selectively condensed MoE architectures that retain high-value parameters.
Specifically, O Comparison to Baselines:
CD-MoE-S vs. CD-MoE-SR: In zero-shot mode, CD-MoE-S exhibits comparable or slightly superior performance to CD-MoE-SR. However, as detailed in Section 4.3 the routed experts in CD-MoE-SR confer a substantial advantage during fine-tuning: they enable nearly 98% of the original performance to recovered in just a few hours on a single 80GB A100 GPU.
Table and Table presents the results on the Qwen1.5-Me -- 7B and Quon2-MoE-57B-A14B. Under similar memory costs, the advantages of CD-MoE baseline methods more pronounced, and with
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Results on GSM8K and HumanBval. The underlined variables are tried to be kept consistent for fair comparison, and the bolds represent the best results. '#L' represents the number of pruned layers and 'MEM' is the remaining memory ratio compared to the original model.
DeepSeekMoE-167
Qwen1.5-MoB-A2.77
Method
#L
MEM.
SpeedUp
GSMSK
HumanEval
#L
MEM.
SpeedUp
GSM8K
Hvaan
UNPRUNED
100%
1.0x
63.7
100%
1.0x
66.1
6.1
LAYER DROP
9
72.20%
1.34X
48.1
2.4
6
78. 78.30%
1.23X
56.5
4.3
BLOCK DROP
71.30%
1.42X
43.5
2.4
6
76. 10%
1.36x
52.9
2.4
MC-SMoE
70.00%
1.0x
53.5
3.7
6
73.00%
1.0x
57.4
4.3
CD-MoE-S
73. 10%
1.34X
52.8
3.7
6
78, 78.30%
1.22X
58.1
4.9
CD-MOE-SR
72.50%
1.26x
54.0
3.7
79.70% 6
1.19x
58.9
4.3
more than 20% reduction in memory usage, the average accuracy approaches that of the original model. We attribute this to the fact that the Qwen series MoE models have larger proportion of shared experts, shown in Table 8 CD-MoE preserves these experts, cnabling the condensed model to retain more knowledge acquired during the pre-training phase.
Table presents the results on GSM8K and HumanEval. Similar to the observations on commonsense reasoning, CD-MoE maintains strong performance under comparable memory footprints. Overall, these results confirm that the condensation strategy preserves core model capacities more effectively than layer-level pruning alone. By selectively retaining expert parameters, CD-MoE consistently delivers strong zero-shot. accuracy while substantially reducing memory and computational overhead.
4.3 Lightweight Fine-tuning Evaluation
We also explored expert fine-tuning to enhance task-specific performance. Although condensation reduces language modeling capacity, fine-tuning can help recover it (Frantar & Alistarh 2023 Shi et al. 2023 Sanh et al. 2020). Our goal is to show that fine-tuning can restore condensed layers to near-original performance. To this end, we propose lightweight approach that updates only the condensed layers. For instance, with CD-MoE-SR, only 3.8% of parameters require gradients, and SFT on the MMLU training set takes just 2 hours single A100 GPU. Following Li et al. (2024a). finc-tuned using commonsensel70k. and MMLU training sets.
Table I (bottom panel) summarizes the performance gains attributable to lightweight SFT. CD-MoE-S improves from an average score of 55.1 1 to 59.1, while CD CD-MoE-SR jumps from 55.5 to 60.4-ncarly matching the original uncondensed performance 61.6. Notably, Commonsense Reasoning: Task like BoolQ shows marked improvements. The CD-MoE-SR SFT lifts performance by up to 7 points in compared to counterpart. before fine-tuning. @ Complex Knowledge Tasks: On MMLU, we observe significant gains (e.g., from 27.1 to 38.5 with CD-MoE-SR SFT), indicating that even after pruning, the model retains foundational knowledge that can be effectively recovered through selective fine-tuning. @ Efficiency: Despite the notable performance boost, the computational overhead remains low because only a fraction of parameters (~~ 3%) is updated. This design ensures training remains practical for resource-constrainee settings, aligning with real-world demands.
For fair comparison, we also apply SFT Layer Drop, updating only its self-attention sub-layers. While fine-tuning recovers some performance, Layer Drop's complete expert removal fundamentally limits recovery. In contrast, CD-MoE-SR retains condensed expert set, enabling significantly better recovery.
Figure 4 shows how the average accuracy changes after SFT as more layers are condensed. Compared to CD-MoE-S, CD-MoE-SR delivers consistently higher accuracy with lower memory usage, suggesting that selecting additional routing experts via greedy search preserves more the model's foundational knowledge. Morcover, CD-MoE-SR can restore and even surpass the original model's performance through lightweight SFT, reducing memory usage by up to 20%. These findings indicate that our condensation strategy retains recoverable knowledge despite significant pruning. With targeted lightweight. SFT, the condensed experts
9

---

Published in Transactions on Machine Learning Research (10/2025)
80
80
Accurscy
55
55-
50-
1.0
0.7
Memory Rati
SpeedUup
60
Acquray
50
50-
1.0
0.7
Momory Ratio
SpeedUu
CD-MoESS
CD-ME-SS SFT
CD-MoE-SR
CD-MoE-SR++ SFT
Baseline
Figure 4: CD-MoE with lightweight fine-tuning. Teft: SFT results on CD-MoE-S with increasing number of condensed layers. Right: SFT results CD-MoE-SR with increasing number of condensed layers. Bascline indicated performance of the dense model.
regain most of their original performance, and in certain tasks, CD-MoE-SR + SFT even exceeds the unpruned baseline.
4.4 Experiment Analysis
Expert selection algorithms. We evaluate several strategies for selecting the most critical experts at each layer. Our primary baselines rely on easily computable statistical properties of the expert parameters:
Random: Randomly select K experts.
L1: Calculate the sum of the L, norms each expert's parameters, sort them, and select the K experts with the lowest L1 norms.
PT_Alpha. Hil: Following heavy-tailed seffrregularization measure each layer's spectral densily via the Hill estimator
MMrrtin & Mahoney
2019
2020
2021
we
Hill 1975 X Xiao el. al.
2023
PL Alpha Haala,_Hllll_a1 = 1 & Σ In An- / 2h-x).
(4)
where } are sorted eigenvalues and & is chosen via the Fix-Finger method Yang et al. (2023). A lower PL_A1 Alpha Hill, indicates a more heavy-tailed layer, so it less likely to be pruned.
We also experimented with selecting experts having the highest L.-norms, but observed substantially worse performance, potentially because lower norms often indicate better-converged paramcters. In Table we compare these methods our Greedy Search, which iteratively selects experts that minimize the deviation between the post-condensation and pre-condeenstton outputs of the layer. Notably, none of the statistical baselines (Random, L,, or PL Alphap Hill) outperform Greedy Search. We hypothesize that the latter's layer-level output fidelity objective directly retains the most impactful experts.
We also measure the time cost of our Greedy Search on the DeepSeekMME- 16B model and compare with the exhaustive traversal in (Lu al. 2024), which cnumerates all expert combinations. Each experiment uses 100 C4 calibration samples with maximum sequence length 512 and batch size 256, running on an 80G A100 GPU. As shown in Table 0 the exhaustive approach requires intractably large number of inferences and is
10

---

Published in Transactions on Machine Learning Research (10/2025)
Table Comparison of expert selection methods. Ln denotes condensing n layers. Each entry is the average accuracy (%) on eight downstream tasks.
Methods
Le
L12
L15
Random
54.6
52.7
50.1
46.8
PL ___AAppa__H111
56.8
52.5
50.3
47.3
L;
56.5
53.2
50.4
47.5
Greedy Search
56.5
55.5
52.8
49.0
Table 6: Time cost comparison for expert and layer selection.
indicates that the method does not require
the step, ** denotes estimates.
Method
Expert Selection
Layer Selection
Inferences (Counts)
Time (Hours)
Time (Hours)
Lu et al. 2024
75,041,808
2500
CD-MBES
0.12
CD-MoF.SR
369
0.13
0.15
therefore infeasible. By contrast, Greedy Search requires only few hundred inferences per layer, striking favorable balance between accuracy preservation and runtime efficiency.
Layer selection algorithms. We compare our proposed Greedy Search (Layer) method against two baselines that rank layers by measuring the divergence between the layer outputs before and after condensation:
Layer Rank:
For each layer Li, we compute the JS divergence (S(()er(LL), Oooneense (Li) between its output before (Oret) and after condensation (Ocondeuse). Layers then sorted by this divergence in ascending order. The layer set Leondense is constructed by selecting the top-N layers with the smallest divergence, i.e,, Leouddesse = (L. Rank,s([.)) N).
Global Layer Rank: For each layer L,, we apply condensation and evaluate the JS divergence (Modee)(M Ocondense (Model, I,)) between the model's final outputs before condensation (Oret) and after condensing only that layer (Ocondeuse). Layers are sorted by this global divergence in ascending order. The layer set Leandense is constructed by selecting the top-N layers with the smallest global divergence, i.e., Leondense = {L (L)) < NN.
Table 7 shows a comparison of these methods in the context CD-MoE-SR. Notably, our Greedy Search consistently surpassos the basclines, underscoring its effectiveness identifying layers that minimally disrupt the overall model output. We also measure the time needed to condense 15 of the 27 MoE layers using the same setup as in expert selection (see Table G). This operation requires only 0.15 hours, further validating the computational efficiency of the CD-MoE approach.
Searching metrics. To quantify how well each condensed model approximates the original model's output, we also compare several metrics during the Greedy Search process. Following Zhang et al. (2024). we adopt JS divergence as our primary measure, as it has been shown to be more sensitive to output changes than angular distance or Euclidean distance. We compare JS against two alternative metrics:
Table 7: Comparison of layer selection methods. I, indicates the condensation of layers. Entries are average accuracies (%) eight test datasets.
Methods
Ls
Lg
L12
L15
Layer Rauk
45.5
44.2
42.8
42.2
Global Layer Rank
55.7
53.1
48.8
45.9
Greedy Search
56.5
55.5
52.8
49.0
11

---

Published in Transactions on Machine Learning Research (10/2025)
Table 8: Performance of different metrics for layer selection. Ln indicates the number of condensed layers. The values shown are average accuracies (%) on eight test datasets.
Metrics
LG
Lg
L12
L15
PPL
58.7
55.4
50.9
KL divergence
57.1
54.7
52.2
JS divergence
56.5
55.5
52.8
49.0
0.05
BlockDrop CD-MeE-SR
0.04
Divargance
0.1
2
10
12
16
19
20
Prune Layer Index
Figure 5: Layer-wise JS curves. The three curves represent the JS divergence between the outputs of pruned layer and the outputs of the original layer, using different pruning methods.
Perplerity (PPL): For each calibration sample, we compute the change in PPL and select layers that minimize the average difference PPL between the original and condensed models.
KL Divergence: We measure the shift between the two output distributions using KL divergence and select layers with minimal KL distance.
Table presents the results. While PPL achieves marginally better score for La, JS divergence excels more layers are condensed (Lg, L12, Lis). These findings substantiate the choice of JS divergence robust metric for maintaining the model's overall quality under heavier compression.
Explanation of shared expert dominance. Our analysis Table reveals a noteworthy finding: CD-MoE-S, while retaining only shared experts in condensed layers, achieves comparable zero-shot performance CD-MoE-SR, which preserves both shared and selected routed experts, significantly outperforming Block Drop. This suggests that shared experts alone may sufficiently preserve model capability.
To investigate this phenomenon, we quantitatively evaluated layer-wise output deviations between pruned and original models using JS divergence (Figure 5j. The results demonstrate that CD-MoE-S induces minimal output distortion, closely approximating CD-MoE-SR's behavior. In contrast, Block Drop's 's aggressive layer removal causes substantially greater output deviation, explaining its inferior performance.
5 Conclusion
We have presented CD-MoE, a novel framework for compressing large-scale ceaturr-of--xppert (MoE) models by selectively condensing them into denser layers. Our approach removes the token routing mechanism and uses greedy search algorithm prune the majority of less significant experts, allowing all tokens to activate only handful of highly impactful experts. Extensive experiments several fine-grained MoE models demonstrate that CD-MoE effectively retains most of the original model's accuracy while achieving significant memory reduction and inference speedup. Morcover, we demonstrate that lightweight expert finc-tuning, locused solely on the condensed layers, can further reclaim the original model's performance within a few hours on a single GPU. These results highlight the practical benefits CD-MoE for resource- essuurcecoonstraine sccnarios, where memory and computational efficiency are critical. Future directions include exploring the synergy of CD-MoE with quantization and knowledge distillation, and extending the method to more diverse MoE structures lacking shared experts.
12

---

Published in Transactions on Machine Learning Research (10/2025)
References
Alexci Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations, 2020. URL pttps///arrd....g//abb///o// 11477
Abhinav Bandari, Lu Yin, Cheng- Yu Hsieh, Ajay Kumar Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, and Shiwei Liu. Is c4 dataset optimal for pruning? an investigation of calibration data for Ilm pruming. arXiv preprint Xrx:2224 10. 07.174 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL https: wttp://wwwrrrabb/11.111111.
Tom B Brown. Language models are few-shot learners. arXiv preprint 2rXiv:2005.44166 2020.
Mark Chen, Jerry Tworck, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkín, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Tgor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL
http:/axivv.oooagboaa// wwt.:oog////aaa..e 03374
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representation collapse of sparse mixture of experts, 2022. URL httts::///aaett http::///xxvvvrr../.. org/ abs/ abaaabas////// 09199
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019a. URL https://arxiv.ors abbs/1955.n 10044
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does bert look at? an analysis of bert's attention, 2019b. URL httpss///arriv..rg//aa///// 04341
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL
https://arxivi http:///aaxxii..orrbb////100000 wot./oo//////////oooo/ 15855
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL /tttps://arsii.oorrabb/////
14168
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognizing textual entailment challenge. In Proceedings of the PASCAL Challenges Workshop on Recognizing Tertual Entailment, 2005.
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. CoRR, abs/2401.066
2024. URL httpa////aaxxiiv.ogaab//////////
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. CRL http:///aaaw..orr//////20000133
Elias Frantar and Dan Alistarh. Sparsogpt: Massive language models can be accurately pruned in one-shot, 2023. URT. http://rwwv..og//aa/2//.0000./
13

---

Published in Transactions on Machine Learning Research (10/2025)
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXie:2101.0002: 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoft, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo. org/records/1266662
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation, 2024a. URL https: 7/arxiv.org/ hatps//407005533
Shwai He, Daize Dong, Liang Ding, and Ang Li. Demystifying the compression of mixture-of-experts through a unified framework, 2024b. URL https: org/
02500
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv. wtg. 2009. 03300
Bruce M Hill. A simple general approach to inference about the tail of distribution. The annals of statistics, pp. 1163-1174, 1975.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaune Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Témothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https : 7/arxiv. org httgabbs/240..00000 0400.00088
Pengxiang Li, Lu Yin, Xiaowei Gao, and Shiwei Liu. Owlore: Oullie--wighed layerwise sampled low-rank projection for memory-effiiien Ilm fine-tuning, 2024a. URL https://arxiv..rrr ab/////b//400.... 18380
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoc with hints from its routing policy, 2024b. URL https wwt.xiv.org/abb/23111 01334
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experls large language models, 2024. URL https:///arat wwwpiiiv.or///bbaa/ orgg/bs///022222 14800
Charles Martin and Michael W Mahoney. Traditional and heavy-tailed self regularization in neural network models. arXiv preprint arXiv:1901.00 082766 2019.
Charles H Martin and Michael W Mahoney, Heavy-tailed universality predicts trends in test accuracies for very large pre-trained deep neural networks. In Proceedings of the 2020 SIAM International Conference on Data Mining, pp. 505-513. STAM, 2020.
Charles H Martin and Michael W Mahoney. Implicit reguaaauulii in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165): 1-73, 2021.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391. Brussels, Belgium, October 2018. Association for Computational URL httpp:///aclamthoooooooor///11-1
Linguistics. doi: 10. (L.1663/v1/D1s--1
14

---

Published in Transactions on Machine Learning Research (10/2025)
Alexandre Muzio, Alex Sun, and Churan He. Secr-moe: Sparse expert efficiency through regularization for mixture-of-experts, 2024. URL
05089
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022. URL
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https://azxiw.org/bbbll wttp:///.wwrr/b/////////// 10641
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning.
Advances in neurol information processing systems, 33:20378 0422378-203889, 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff
Dean. Outrageously large neural networks: The sparsely-gated mixture--o--xxprrs layer, 2017. URL
https://arxiv.. wwt.://////...../555003/0 http:://arrvvoorr/a////T.......
Enshong Shi, Yanlin Wang, Hongyu Zhang, Lun Du, Shi Han, Dongmei Zhang, and Hongbin Sun. Towards
efficient fine-tuning of pre-trained code models: An experimental study and beyond, 2023. URL https
//arxvv.org/ab///2044.000
Mingjic Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models, 2024. URL. https://arxiv. http:///axxiv.oo/aaa/////// wrg..bs////b/6/5/// 11695
Gemini Team, M Reid, N Savinov, D Teplyashin, Lepikhin Dmitry, T Lillicrap, JB Alayrac R Soricut, A Lazaridou, Firat, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, in arxiv Jes. cl]. arxiv, 2024.
Qwen Team. Qwen1 5-moe: Matching 7b model performance with 1/3 activated parameters', , February 2024.
URL https::/ http:///qwelm...hhuubb.. ht/bll////////o htw../////oo///e/mmmeeeeee
Hugo Touvron, Thibaut Lavril, Gautier Tzacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint Xix:2000 18971, 2023a.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Mario-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023b. URL
https::/arrae wrtpaaaiwworra//////022... 13971
Xuanzhe Xiao, Zeng Li, Chuanlong Xie, and Fengwei Zhou. Heavy-tailed regularization of weight matrices in deep ncural networks. In International Conference on Artificial Neural Networks, pp. 236-247. Springer, 2023.
Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E Gonzalez, Kannan Ramchandran, Charles H Martin, and Michael W Mahoney. Test accuracy vs. generalization gap: Model selection in nlp without accessing training or testing data. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3011-3021. 2023.
Lu Yin, You WU, Zhenyu Zhang, Cheng-Yi Yu Hsich, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning 1lms to high sparsity, 2024. URL https://arxiv.orz
abs/2310. 05175
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Vejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pro-trained transformer language models, 2022. URI https://arxiv http:///aaa...oog/aa/////0...0.0
15

---

Published in Transactions on Machine Learning Research (10/2025)
Yang Zhang, Yawei Li, Xinpeng Wang, Qianli Shen, Barbara Plank, Bernd Bischl, Mina Rezaci, and Kenji Kawaguchi. Finercut: Finer-grained interpretable layer pruning for large language models, 2024. URL wwwp.ii.or//bb///455...... 18218
A
Appendix
A.1
Implementation Details
We utilize Hugging Face and PyTTrch for the implementation of our work H All inference and fine-tuning operations are executed using bf16 (Brain Floating Point 16) precision on NVIDIA A100 GPU equipped with 80GB of memory. During the fine-tuning phase, we employ an initial maximum learning rate of 1 x 10-1, implement a warm-up ratio of 10% to gradually ramp up the learning rate, and utilize cosine anncaling as the learning rate scheduler ensure smooth convergence and prevent potential overfitting. Due to the large computational requirements and our limited resources, all experiments
conducted in single run.
The expert configurations of DeepSeekMoE-116
7B and Qwen2- Moem2-MoE-57B-A11 A14 repre-
sented at Table A.1
Table A.1: Expert configurations in MoE models
Model
DeepSeek MoE-16B
Doven.5--Mo-A2..
Parameters
11,3757228125
14,315,784.191
57,421,636 606
Activated Paramoters
2,828 650,496
2,680, 177,536
14,249,346,044
Number of Experts
66
64
72
Number of Activated Experts
8
8
16
Ratio of Activated Experts
12.1%
12.5%
22.2%
Number Shared Experts
2
4
8
Ratio Shared Experts Activated Experts
25%
30%
50%
A.2 Calibration Data Analysis
C4 dataset commonly serves the calibration benchmark. In our study, we aim to ensure that the performance of the condensed model aligns closely with specific task requirements. To this end, we experiment by utilizing questions extracted from multiple test sets as calibration data, thereby tailoring the language modeling capabilities of the condensed model to the tasks at hand. Specifically, we sampled 20 questions from each of cight distinct test sets, resulting a total of 160 questions.
Table A.2: Performance comparison across different calibration data across different layer pruning indexes. I. indicates the condensation of n layers. Entries represent evaluation metrics (%).
Methods
Lg
Lg
Lg
L12
L16
DownStream Data
59.3
56.1
55.3
51.9
48.7
Audio Thken
59.0
56.7
55.0
52.0
49.1
C4
59.5
56.5
55.5
52.8
49.0
As demonstrated in Fig A.2 it is noteworthy that across varying numbers of condensed layers, the C4 data consistently achieves superior performance compared to the downstream tasks data. This suggests that the diversity and comprehensiveenss of the C4 dataset provide a more robust foundation for calibrating the condensed model. Consequently, we adopt C4 as the calibration data in all experiments. This finding is consistent with the work by Bandari et al. (2024), where they perform unstructured pruning of LLM parameters, while we focus on layer condensation.
Additionally, we probed more extreme domain shifts by using small andio-domain samples from the open- source Emilia dataset He et al. 2024a), processed through wav2vec(Baevski et al. 2020) for calibration.
Ouu repository built on top Transformers: http:///tttnab.ooo//gggggggggne/ transsfrrrrrrsssserrsrr...
16

---

Published in Transactions on Machine Learning Research (10/2025)
The accuracy remained broadly comparable to the C4-based results, reported for the average accuracy of DeepSeckMoF-110 16B on commonsense reasoning and MMLUU. This demonstrates that CD-MoE is not highly dependent on specific calibration dataset.
17


Reviews:
[{"review_id": "iOwrK1d8GQ", "reviewer": "Reviewer_MByv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes ConDense-MoE (CD-MoE), a method for compressing fine-grained shared-expert MoE layers by removing routing and retaining a small set of always-active experts. The authors introduce greedy expert- and layer-selection strategies based on minimizing output JS divergence. Experiments on DeepSeekMoE, Qwen1.5, and Qwen2 models show ~27% memory reduction, ~1.26× speedup, ~90% zero-shot accuracy retention, and up to ~98% recovery after lightweight fine-tuning.\n\n**Strengths:**\n\n- Clear motivation and practical relevance for MoE compression.\n- Simple but effective greedy algorithms. Both expert-level and layer-level greedy selection strategies are conceptually simple and easy to implement.\n- Strong empirical results across multiple MoE architectures.\n- Low-cost calibration and parameter-efficient fine-tuning.\n\n**Weaknesses:**\n- The experiments rely on standard commonsense reasoning and MMLU tasks. It is unclear whether performance would degrade more on long-context, code and math tasks.\n\n- Baselines like LayerDrop, BlockDrop prune whole layers, which is generally more destructive than MoE-specific condensation. The advantage of CD-MoE over M-SMoE is not uniformly large and the evaluation lacks breadth, particularly in tasks requiring expert specialization.", "claims_and_evidence_yn": "No", "claims_explanation": "- Evaluations focus on commonsense reasoning and MMLU. Tasks that may rely more on specialized routed experts like math, coding, and long context are missing. The claim that condensation preserves “core capabilities” is not fully validated across task types.\n\n- Experiments were run “in a single run” (A.2). No standard deviation, random seed sweep, or statistical tests are reported. Claims like “98% recovery” and “90% retention” need error bars / multiple seeds to be convincing.\n\n- M-SMoE is included, but only one configuration appears tested and little hyperparameter tuning / variant coverage is shown. The current evidence shows CD-MoE outperforms that M-SMoE setting on the chosen tasks, but it is not decisive that CD-MoE is generally superior.", "audience_yn": "Yes", "audience_explanation": "Researchers working on LLM efficiency and model compression would be generally interested in this paper.", "requested_changes": "- Expand evaluation beyond commonsense/MMLU tasks. Add benchmarks that rely more heavily on expert specialization including math, code and long-context, or discuss limitations if this is not feasible. \n\n\n- Run key experiments with multiple random seeds and report mean/variance, or justify single-run reporting if resource limits preclude it.\n\n- Strengthen the comparison with M-SMoE by evaluating on a broader set of tasks beyond commonsense reasoning, e.g., math, coding, and by including additional MoE architectures.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a7tDe3BPS4", "reviewer": "Reviewer_JQZf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper’s main objective is to propose ConDense-MoE (CD-MoE), a method that condenses Mixture-of-Experts (MoE) feed-forward (FFN) layers into smaller dense layers to reduce memory usage and runtime during inference while maintaining performance.\n\n* The authors explicitly enumerate three contributions at the end of the Introduction: a novel condensation approach that replaces sparse MoE layers with small dense ones, an efficient greedy algorithm for selecting layers and experts, and empirical validation on DeepSeekMoE-16B demonstrating memory savings, faster inference, and preserved accuracy.\n\n* The proposed method, CD-MoE, removes routing at selected layers and replaces the MoE layer with a dense layer that retains shared experts and a small number of important routing experts; the target models are fine-grained MoEs with shared experts such as DeepSeekMoE and Qwen1.5-MoE.\n\n* Expert and layer selections, parts of the proposed method, are performed via a greedy search that minimizes Jensen–Shannon divergence between the reference (original) and condensed outputs; the procedures run in polynomial time.\n\n* This paper claims that, on DeepSeekMoE-16B, CD-MoE maintains about 90% of the original average accuracy while reducing memory by 27.5% and increasing inference speed by 1.26×; with lightweight SFT only on the condensed layers, performance recovers up to 98% on a single 80-GB A100.\n\n* In comparative experiments, under similar memory budgets, CD-MoE achieves higher zero-shot accuracy than baselines such as block or layer dropping and simple expert-count reduction.", "claims_and_evidence_yn": "No", "claims_explanation": "The current version of this paper has at least two major limitations.\nFirst, it appears to omit several substantial information in the proposed method and experiments.\nSecond, the experiments reported in the paper do not convincingly demonstrate the effectiveness of the proposed method, including the expert and layer selection components.\n\n\n\n\n* [C1] Targeting only a subcategory of MoE models: \nThe method seems to assume only on fine-grained MoE models with shared experts.\nThis limited subcategory of entire MoE models only includes very limited number of current MoEs among the wide variety of MoE models.\nMoreover, there is no clear discussions that the proposed method can be applied or extended to more standard MoE architectures (e.g., Mixtral, Qwen-3, GPT-oss, and OLMoE). \nIn addition, nowadays it seems that current trend is not using shared experts, so that the proposed method is not in the main stream.\n\n\n\n* [C2] Lack of important description:\n  - There are no clear definitions about CD-MoE-S and CD-MoE-SR in the main content.  \nThey only explained only in captions of several Tables nd Figures, and only by one sentence; \"CD-MoE-S refers to retaining only the Shared expert after condensing, while CD-MoE-SR means selecting additional 6 Routed experts and keeping as dense.\"\nThis sentence is too ambiguous, so that It is extremely difficult to understand the details of what exactly compute CD-MoE-S and CD-MoE-SR.\nSimilarly this statement \"6 route experts are consistent with the number of experts activated in the dense model.\" is also too ambiguous.\n\n  - In FIgure 3, several plots are obtained by changing \"Memory Ratio\" and \"SpeedUp.\" However, there is no clear description how authors control such \"Memory Ratio.\" My guess is changing the number of layers to prune, namely $N$, but not clearly described. \n\n   - Difference between Layer Rank and Global Layer Rank is not very clear. These methods are described only on the two or three sentences. They should be clearly defined (for example by using equations) to avoid misunderstanding by the ambiguous text definition.\n\n\n\n* [C3] Missing important and closely related studies:  \nThe method is not compared against several recent approaches in expert selection and structured pruning, for example as follows:  \n   [ACL'25 findings] Zhang+, Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse MoE  \n   [ACL'25] Lee+, STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning  \n   [EMNLP'24 findings] Yang+, MoE-I^2: Compressing Mixture of Experts Models through Pruning and Distillation  \n   [ICLM'24] Chowdhury+, A Provably Effective Method for Pruning Experts in Fine-tuned Sparse MoE  \n  This gap makes it hard to judge the empirical significance and competitiveness of the proposed method.  \n  Additionally,   \n  [NeurIPS'23] Ma+, LLM-Pruner: On the Structural Pruning of Large Language Models  \n  should also be cited. Although the method differs, the purpose is the same, including applying fine-tuning after the proposed procedure.\n\n\n\n* [C4] Risk of leading unreliable conclusions:  \nThe results are reported based on only a single run. They should be reported as averages over several runs, especially for lightweight SFT, since the selection of 100 samples from the C4 dataset and SFT initialization can be random and can be repeated easily.\nAdditionally, the results should be assessed using statistical tests over multiple runs to show statistical significance.\nFor example, the results of Table 5, comparing JS divergence, PPL, and KL divergence, the performance is nearly the same. \nThere may be no significant difference among them.\n\n* [C5] Marginal Effectiveness:  \nThe authors claim that the proposed method is effective. However, the experimental results show that it reduces memory usage by only 27.5% and increases inference speed by only 1.26× at most, while maintaining similar, albeit degraded, performance. These results do not seem particularly impressive from a practical perspective. Moreover, pruning only the experts, namely, as in CD-MoE-S, also achieves similar performance. Based on these results, it is difficult to conclude that the proposed method, i.e., condensing experts into a dense model, is essential.\n\n* [C6] No clear discussions:  \nThere are several detailed analysis given in Section 4.4. \nHowever, it only reports the experimental results. \nThere are no discussions why such results are obtained.\nFor example, why condensing to a dense model works and why JS divergence works well for other metrics.\nSuch evaluations and discussions are essentially the most important part of analysis.\nThe readers cannot obtain new findings without the discussion and only with the ablation studies.\n\n* [C7] Not appropriate use of Appendix:  \nI think there is no reason that the content in \"A.1 Fluctuations in Metrics across Condensing Different Layers\" is not discussed in the main body unlike the current location in Appendix.\nSimilarly, the experimental results of Qwen1.5-MoE-A2.7B and Qwen2-MoE-57B-A14B are also discussed in the main body.\nI have no idea why the authors decided to put these results on Appendix.\nAt the last part of the Introduction section, the authors claims that \"Empirical validation on DeepSeekMoE-16B, demonstrating a 27.5% memory reduction and a 1.26× inference speed, while retaining up to 90% of the zero-shot accuracy.\" \nI do not understand why the authors did not include the results of Qwen1.5-MoE-A2.7B and Qwen2-MoE-57B-A14B  as their contributions.\nIn addition, the appendix \"A.2 Implementation Details\" is no problem to be in Appendix.\n\n\n\n\n\nIn sum up, given the limitations and drawbacks explained above, the answer to the question, \"Are the claims made in the submission supported by accurate, convincing, and clear evidence?\" is that I cannot say yes without reservation; rather, I am leaning toward no.", "audience_yn": "No", "audience_explanation": "Developing methods that reduce memory and runtime at inference time while preserving an acceptable level of the original performance is an important topic for LLM applications. \nMoreover, the idea of condensing experts, meaning converting important experts into one dense model, is interesting.\nAccordingly, there is likely to be a broad audience interested in this area.\n\nThat said, as noted above, the paper has several serious issues:  \n(1) insufficient detail about the method and the experiments, and  \n(2) a limited experimental design that does not support strong conclusions.  \n\nTherefore, it is difficult to argue that readers will find the results compelling overall, and interest in the paper’s specific findings is likely to be limited.", "requested_changes": "I have already listed the points that need to be updated in the textbox above, as indicated by [C1] - [C7].\nPlease refer to them.\n\nAdditionally, the content of the paper should be self-contained.\nThe paper should be substantially revised to clearly explain both the proposed method and the experiments.\nMoreover, the paper should include additional comparisons and analyses to support the effectiveness of the proposed method.", "broader_impact_concerns": "I think there is no serious ethical implications of this study.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2xUvzbxsKW", "reviewer": "Reviewer_B32G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents ConDense-MoE (CD-MoE), a framework for improving the efficiency of MoE models by condensing sparse MoE layers into smaller, dense ones instead of pruning them. The method removes the routing mechanism in selected layers and retains only a subset of influential experts, determined through a greedy selection process based on JS divergence between original and condensed outputs. Experiments on large-scale MoE models such as DeepSeekMoE-16B and Qwen-MoE show that CD-MoE can achieve a 27.5% reduction in memory and 1.26× speedup, while maintaining around 90% of the original performance, and recovering 98% after lightweight fine-tuning of condensed layers. The approach is conceptually clear and empirically effective, providing a practical trade-off between compression and accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper presents pruning experiments on several MoE models, including DeepSeekMoE-16B and Qwen-MoE. As shown in Table 1, the proposed method achieves superior performance, reaching an average score of 55.5 even without supervised fine-tuning. Figure 3 further demonstrates a more favorable trade-off between efficiency and accuracy compared to prior pruning approaches. Moreover, the analysis in Figure 5 supports the paper’s claim that condensing experts yields better performance than directly removing entire layers.", "audience_yn": "Yes", "audience_explanation": "This work provides a comprehensive study on compressing MoE models, addressing a highly practical problem. Given that many industrial-scale systems rely on MoE architectures, improving their efficiency is of significant importance for real-world deployment and scalability.", "requested_changes": "1. In Table 1, baselines such as BlockDrop achieve a lower average performance (52.2) compared to the proposed method (55.5). However, it would be interesting to examine how these baselines perform after applying the same lightweight supervised fine-tuning (SFT). A possible outcome is that, after SFT, all methods might reach comparable levels of accuracy. Therefore, it is important to include and analyze these results to clarify whether the proposed condensation approach offers advantages during fine-tuning rather than merely before it.\n2. It is unclear how the reported “speedup” is measured. Specifically, the paper does not specify whether the improvement refers to prefill speed, decoding speed, or an overall end-to-end inference throughput.\n3. It is unclear how sensitive the proposed method is to the choice of calibration data. Since the condensation process relies on statistics computed from a specific dataset (e.g., C4), the resulting performance may vary across different data domains. It is recommended to evaluate the method using multiple calibration sources, such as C4, WikiText, or STEM-oriented datasets to assess its robustness and generalization across diverse data distributions.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)