Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
Authors: Shwai He
Daize Dong
Liang Ding
Ang Li
Venue: Accepted by TMLR ()

Abstract: Scaling large language models has driven remarkable advancements across various domains, yet the continual increase in model size presents significant challenges for real-world deployment. The Mixture of Experts (MoE) architecture offers a promising solution by dynamically selecting and activating only a subset of experts during inference, thus substantially reducing computational costs while preserving high performance. Despite these benefits, MoE introduces new inefficiencies, such as excessive parameters and communication overhead. In this work, we present a holistic study of compression techniques for Mixture of Experts to enhance both efficiency and scalability. While recent efforts have focused on Expert Trimming, which reduces the number of experts, these approaches still suffer from considerable communication and computational costs. To address this, we propose more aggressive strategies, such as Layer Drop, which removes entire MoE layers, and Block Drop, which eliminates transformer blocks. Surprisingly, these aggressive pruning techniques not only preserve model performance but also substantially improve computation and memory efficiency. Furthermore, beyond Expert Trimming, we also introduce Expert Slimming, which compresses individual experts to further boost performance and can be seamlessly integrated with Expert Trimming. Extensive experimental results demonstrate the effectiveness of our proposed methods—Layer Drop and Block Drop—along with the comprehensive recipe that integrates Expert Slimming and Expert Trimming, achieving a 6.05× speedup with 77.1% reduced memory usage while maintaining over 92% of performance on Mixtral-8×7B. Our code is released at https://github.com/CASE-Lab-UMD/Unified-MoE-Compression.

Published in Transactions on Machine Learning Research (03/2025)
Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
Shwai He*
shwoihe@ummd edu
University of Maryland, College Park
Daize Dong*
dzdong20198dmmail com
Rutgers University
Liang Ding University of Sydney
Blangdinglaam@aaee com
Ang Li
angliece@ummd.co edu
University of Maryland, College Park
Reviewed on OpenReview: https: openreview. net/ forum? id- 6aSSI
Abstract
Scaling large language models has driven remarkable advancements across various domains, yet the continual increase in model size presents significant challenges for real-world deploy- ment. The Mixture of Experts (MoE) architecture offers promising solution by dynamically selecting and activating only subset of experts during inference, thus substantially reducing computational costs while preserving high performance. Despite these benefits, MoE intro- duces new inefficiencies, such as excessive parameters and communication overhead. In this work, we present. a holistic study of compression techniques for Mixture of Experts to enhance both efficiency and scalability. While recent efforts have focused on Expert Trimming, which reduces the number of experts, these approaches still suffer from considerable communication and computational costs. To address this, propose more aggressive strategies, such as Layer Drop, which removes entire MoE layers, and Block Drop, which eliminates transformer blocks. Surprisingly, these aggressive pruning techniques not only preserve model perfor- mance but also substantially improve computation and memory efficiency. Furthermore, beyond Expert Trimming, we also introduce Expert Slimming, which compresses individual experts further boost performance and can be seamlessly integrated with Expert Trimming. Extensive experimental results demonstrate the effectiveness of our proposed methods Layer Drop and Block Drop along with the comprehensive recipe that integrates Expert Slimming and Expert Trimming, achieving 6.05x speedup with 77.1% reduced memory usage while maintaining over 92% of performance on Mixtral-8x7B. Our code is released at htttp:///ithuu: com/CASE- SMAAb-UDDUnnifii -wwp://nniiiieee-aaaaaaaAa Compressooooooonn
1
Introduction
While scaling large language models has shown exceptional performance across various domains (Ramesh
el al. 2021 OpenAT 2024 Team 2024a], the increasing model size poses significant challenges in real-world deployments (Sun et al. 2023: Frantar et al. 2022) due to excessive computational demands and associated costs. The Mixture Experts (MoE) (Shazcer et al. 2017). which selectively activates a subset of parameters during inference, offers a promising solution to reduce these computational burdens. Additionally, integrating
"Equal contribution
Work done during assistantship

---

Published in Transactions on Machine Learning Research (03/2025)
MoE with Large Language Models (LLMs) has been shown to further enhance performance (Jiang et al. Dai et al. 2024).
2024
Despite these advancements, MoE models still suffer from significant redundancies that increase deployment costs. Standard MoE implementations replicate feed-forward layers across multiple experts, resulting in heavily parameterized models. For instance, Mixtral-8x7ß: 7B (Jiang et al. 2024) contains 47B parameters, but only 13B parameters are activated per token, leading to substantial GPU memory consumption and limited scalability. In addition, replicating experts often introduces redundant experts. For example, He et al. 2023 observed that expert parameters could be compressed through parameter sharing. Similarly, Lu et al. 2024) noted that not all experts are essential, suggesting that some can be safely removed. These findings underscore the potential for compressing MoE models to improve efficiency without sacrificing effectiveness.
In this paper, we first investigate the Expert Trimming based compression techniques, which reduce the number of experts to enhance the efficiency of MoE (Cheng ct al. 2020 Liang et al., 2027]. The most prevalent approach for Expert Trimming is Expert Drop, which scores each expert and drops the less important ones (Lu et al., 2024 Muzio et al., 2024). While Expert Drop reduces model size, it does not eliminate costly computations within the MoE layer and complex communication among experts, leading to negligible improvements in the inference speed. To this end, we propose more aggressive Expert Trimming methods to enhance MoF. efficiency. Specifically, to mitigate communication and computation costs, we present. Layer Drop that removes the entire MoE layer. Additionally, given the cuuputation-intessii nature of the attention mechanism within transformer blocks, we further propose Block Drop, which removes the whole transformer blocks. We use siiilaitty-basel metrics to demonstrate the feasibility of Layer Drop and Block Drop. Surprisingly, these two coarse-grained methods outperform fine-grained Expert Drop by a large margin in balancing performance and efficiency. Additionally, with small-scale post-finetuning, the compressed models can be further optimized to achieve originaiginal performance.
Beyond removing experts, we also explore Expert Slimming, which focuses on compressing individual experts. Techniques such as network pruning (Han et al. 2016 Zhu & Gupta 2017) and quantization (Jacob et al.) 2017 Nagel et al. (021)) have proven effective for the model compression, with quantization being particularly suitable for hardware acceleration. By integrating Expert Slimming with Expert Trimming, we propose a unified framework for comprossing MoE models that further maximizes efficiency gains while maintaining strong performance.
Our experimental results on representative MoE models, Mixtral-8x7B (Jiang et al. 2024 and DeepSeek-Mol: 16B (Dai et al., 2024), demonstrate the effectiveness of our proposed mothods. For Export Trimming, Expert Drop significantly reduces the memory usage but provides only marginal improvements in inference speed. In contrast, Layer Drop and Block Drop signíficantly accelerate inference and reduce memory usage while maintaining comparable performance to the original models. The combined strategy of Expert Trimming and Expert Slimming results in 6.05x speedup with only 22.8% memory usage (20.0GB) while maintaining over 92% of the original performance on Mixtral-8x7B, The findings offer valuable insights for enhancing the efficiency of MoE models. Additionally, post-finctuning allows compressed models to recover most of their original performance, resulting in a minimal 0.6% performance gap compared to the uncompressed model.
DeepSeek-MotE
In summary, by conducting a holistic study on compressing Mixture of Exports, our key contributions are:
We extend Expert Trimming to higher architectural with Layer Drop and Block Drop, significantly enhancing computation and memory efficiency while preserving model performance.
We integrate Expert Trimming with Expert Slimming to further achieve efficiency gains without compromising performance.
Extensive experimental results demonstrate the effectiveness of our proposed methods, achicving a 6.05x speedup and reducing memory usage just 20.0 GB, all while maintaining over 92% of performance Mixtral-8x7I3
2

---

Published in Transactions on Machine Learning Research (03/2025)
2
Related Work
Mixture of Experts The Mixture of Experts (MoE) is kind of neural network architecture with an extended set of parameters (referred to as "experts") controlled by router, which is first introduced in the context of conditional computation (Jacobs et al. 1991 Jordan & Jacobs 1994). The potential of sparse activation in MoE is subsequently exploited by Shazeer ct al. (2017) for efficient training and inference on pretrained models with special designs, opening the door for MoE in various vision (Riquelme et al., 2021 and
language (Lepikhin et al. 2020; Du et al.) 2022 Fedus et al.) 2022) scenarios. Attributed to its exceptional efficiency, MoE has been adopted
foundational framework in the designs of large language models (LLMs)
Zhu ct al.) 2024 Team 2024b). achieving superior 2022 Further investigations emerge in developing Rajbhandari et al., 2022 Dai et al., 2024), router designs (Lewis et al. 2021: Roller et al., 2021 Zhou et al.) 2022), and training strategics (Shen ct al. 2023:
(Jiang et al. 2024 Dai et al.
2024 Xue et al. 2024a
scaling laws at low computational costs (Clark et al.
improved expert structures (Gururangan et al. 2022
Chen et al.) 2022), propelling the continuous evolution on the representation capability and computational efficiency of MoE models. Despite the success, MoE also suffers from efficiency issues. For instance, MoE replicates the experts, significantly increasing the parameter budget (He et al. 2023). On the other hand, adopting multiple experts to process input tokens introduces communccation costs and enhances latency [Song et al.
2023 Xue et al. 2024b).
Compression Methods The increasing size of large language models presents considerable challenges for their practical implementation. Consequently, range of efficient methods has emerged to address the implementation issues. Among them, model quantization (Frantar et al. 2022 Lin et al., 2024) and network pruning (Sun et al. 2023 Frantar & Alistarh 2023) are widely utilized. Model quantization reduces the precision of neural network weights to lower bits (Jacob et al. 2017), while network pruning (Han et al.) 2016) removes redundant paramcters or architcctures. Although these mothods have shown promising results dense models, they lack consideration for the inductive bias inherent in MoE. To bridge this gap, Expert Drop, as proposed in studies like (Muzio et al., 2024 Lu et al.) 2024). addresses the unique nature of MoE by removing unimportant experts. By climinating redundant experts, the MoE architecture becomes more compact and can be deployed at a lower cost. However, while Expert Drop leads to a more compact architecture, it may also lead to non-negligible performance drop and rely on post-training for recovery.
3 Preliminaries
3.1 Mixture of Experts
A Mixture of Experts (MoE) layer consists of a collection of n experts, {,i, E2.... En), each associated with weights (W1, W...., Wa); and a router G that dynamically selects the most relevant experts for a given input æ. The router computes selection scores, G(æ) E IR,, for all experts and selects the top k experts, resulting in a sparse activation pattern. The input æ is processed by the selected experts, and their outputs are combined into a weighted sum based on the router's scores. This process is mathematically expressed as:
K = (SpX(Sootmxxx(((2))
(1)
Σ KEK G(z); (x(,),)).
(2)
where IC denotes the indices of selected experts, G(zz;; represents the selection score for the i-th expert, and Ex(x) is the output from the i-th expert. In transformer models, the MoE layer usually replaces the ford-forward network (FFN). In this context, each expert functions as an independent FFN module, enhancing the model's capacity without proportional increase in the computational cost |Vaswani et al. 2017).
Challenges While MoE models have demonstrated strong performance across various tasks (Jiang et al., 20224 Dai et al. 2024), they also encounter significant. deployment challenges. On one hand, MoE models replicate multiple expert networks, inflating the model size and memory usage. For instance, Mixtral-8x77 has total of 47B parameters and requires 87.7GB of memory for deployment, though only 13B parameters
3

---

Published in Transactions on Machine Learning Research (03/2025)
are activated for each token. On the other hand, the communication required to manage multiple expert networks increases latency and slows down inference speed, especially in distributed environments (Song et al. 2023 Yu et al. 2024
3.2
Overview of Previous Compression Methods
To address the efficiency challenges, we first review several mainstream and state-of-the-art compression techniques for MoE models.
Pruning: Pruming reduces the number of active parameters by selectively disabling parts of the model's weights. In an MoE layer with n experts {E,) and corresponding weights (Wi) pruning introduces binary masks {Mil deactivate certain weights:
W, = M, 3 We
(3)
Pruning can be unstructured (Lee et al. 2021 Bai et al., 2022), semi-structured, or structured. Unstructured sparsity tends to yield the best performance, semi-structure sparsity strikes a balance between efficiency and performance, and structured sparsity, while prrdware-friendly often results in lower performance.
Quantization: Unlike pruning, which involves masking out unimportant parameters, quantization reduces memory usage by converting model weights to lower-bit representations. For MoE layers, quantization is applied in the following way:
W, = Quant
(4)
where "Quant" denotes the quantization function. Quantization decreases the computation and memory consumption without reducing FLOPs or the total number of parameters, making it particularly advanlageous for hardware acceleration.
Expert Drop: Different from fine-grained pruning and quantization, Expert Drop entails the removal of
expert networks, based on the observation that not all experts are equally important (Lu et al. 2024 Muzio
ct al.] 2024). Given expert-wise importance scores S (e.g., the routing scores, S(E,) G());;) Export Drop retains only the experts with the highest n' scores:
T'
(5)
E - (EdJiET' G GiET
(6)
Here, T' denotes the subset of the original expert indices (1,2, .n. Expert Drop reduces FLOPs conditionally: when T' contains more than or equal to indices, MoE still utilizes the top k experts for each input; otherwise, it uses all remaining experts. While this approach reduces communication between experts, the resulting speedup is usually insignificant when maintaining acceptable performance.
Other Compression Techniques: Other methods, such as low-rank decomposition (Li et al. 2024b a), aim to compress model weights into smaller matrices, further reducing memory and computational costs. In this work, we primarily focus on the widely-used methods (pruning, quantization, and Expert Drop), leaving a more dotailed exploration of these additional mothods for future rescarch.
4 Holistic Study of MoE Compression Techiniques
In this section, we propose a general framework that unifics various compression methods for MoE. This framework provides comprehensive understanding of MoE model efficiency issues and helps identify new design spaces for further performance improvements.
4.1 Overview
Existing MoE, compression methods primarily address two types of inefficiencies: structural redundancies in the overall architecture and internal redundancies within individual experts. To address both issues, we categorize these methods into two complementary perspectives: Expert Trimming that focuses removing
4

---

Published in Transactions on Machine Learning Research (03/2025)
(a) Transformer Block
(b) Mixture Experts
(c) Expert Slimming
w f(W)
Pruning ((W) Ad e W
Quantization
(W) quant(
Mixture of Experts
Bg
G2
G2
MoE Normalization
(d) Expert Trimming
{ (Eeeer
Expert Drop
Layer Drop
Drop
TT,
Ø2 MoRNoem
22 R
Multi-Head Allention
Rg
By
A2
A
G2
MoE Normalization
Normelization
O
o'o
Muli-Hiead Attention
Granularty Seales Up
MoF Normatzration
MHA Normaization
Figure 1: The Unified View of MoE Compression. The view integrates two complementary perspectives: Expert Slimming and Expert Trimming. Expert Slimming compresses individual experts, while Expert Trimming directly drops structured modules.
structured components (e.g., experts, layers, and blocks), and Expert Slimming that compresses individual experts through techniques like pruning or quantization. An overview of these compression perspectives is illustrated in Figure 1
Expert Trimming deals with compressing struct modules by selecting and relaining only subset of the experts, denoted as T'. This is represented by the transformation T + T', Methods like Expert Drop, which selectively drops unimportant experts, are examples of this approach. On the other hand, the compression of individual experts (Expert Slimuning) focuses on the transformation and reduction of expert weights, denoted as W. We utilize transformation function f(W) represent this process. The transformation function f(W) can be understood the model. For example, in pruning, f(W) could be function that sets subset of the weights to zeros. In quantization, f(W) might. reduce the precision of the weights from 32-bit. floats to 8-bit. integers. By integrating these two perspectives, we can derive a general form for efficient MoE models. The compression within and across experts can be expressed as follows:
general mapping that applies various compression techniques to the weights of
y= E IET G. (.(=)f(W.).
(7)
In the following sections, we will elaborate on Expert Trimming and Expert Slimming, respectively.
4.2 Expert Trimming
The core operation of Expert Trimming involves updating the set of remaining exports denoted as T T', where T' subset of the original expert. indices T. Specifically, Expert Drop updates the experts and their corresponding routing weights follows: E + {E, HeT, and G + GiET..
However, Expert Drop carries the risk of collapsing fcature transformation. The absence of certain experts can lead to incorrect selections for given inputs, thereby degrading model performance (Chen el al. 2022). Additionally, partially reducing experts can disrupt routing patterns, negatively impacting the model's overall efficiency and effectiveness. Despite its benefits, Expert Drop still retains the costly computation within each expert and the complex communication between experts. These limitations highlight the need for further optimization of Expert Trimming to promote the efficiency. By systematically analyzing the redundancies
5

---

Published in Transactions on Machine Learning Research (03/2025)
Table 1: Summary of Compression Methods. "V" means effective, indicating that the method performs well as intended. "X" means ineffective, where the method fails to optimize the specified metric. "O" represents conditionally effective, depending on specific settings and environments.
Method
Formulation
Parameter
Memory
FLOPs
Speedup
Expert Drop
T
O
Expert Trimming
Layer Drop
T T+
Block Drop
Pruning Quantization
M M W Quant(W)
/
O
/
O
Expert Slimming
X
/
/
and inefficiencies inherent MoE, models, we propose extending beyond expert-level optimizations to identify new design spaces for efficiency improvements.
We propose two novel techniques: Layer Drop and Block Drop. Layer Drop focuses on removing entire MoE layers, which significantly reduces both computation and communica- tion overhead. Block Drop extends this concept by eliminat- ing entire blocks, including attention layers and MoE layers, within transformer models. These advancod techniques aim to streamline the model architecture, improve performance, and enhance overall efficiency.
v/ a'
MoB(z)
S00)
Mixture of Experts
(NM)
Norm (z')
MoE Normalization
Layer Drop Inspired by Raposo et al. 20244) Elhoushi et al. (2024), we consider special scenario of Expert Drop where all experts are dropped (T + T' = Ø), effectively removing entire MoE layers. We refer to this approach as Layer Drop. To perform Layer Drop, we use a similarity-based metric where high similarity indicates high redundancy in transfor- mation. One straightforward metric is the cosine similarity between the input and the output y = MoF(w)):
Figure 2: Illustration of Similarity Mea- surements in Layer Drop. Features for calculating S(M) and S(NM) are colored with red and blue, respectively.
S
| 1 where y = MoE(x).
x y
However, this metric alone does not adequately capture the impact of the MoE layer within the context of transformer block, which includes layer normalization module ("Norm") (Ba et al.] 2016) and residual connections (He et al.] (2015). To address this, we propose concurrently removing both the MoE and Norm layers. This approach ensures that the similarity metric more accurately reflects the combined functionality of these layers, allowing for a more precise identification of redundancy and a streamlined model architecture, as illustrated Figure 2 By considering the similarity between the raw residual input and the aggrogated output, we can better evaluate the necessity of the MoE layer in the overall architecture:
x' y
((NM)
where y' = x' MoE(No(((zz));
22 1
Block Drop Within transformer block, Layer Drop removes the MoE layers but retains the computation- costly attention layers (Ribar et al. 2024 Zhang et al. 2023 To address this issue, we further utilize the same similarity-based metrics to investigate whether the attention layer can be dropped without a significant performance drop. If feasible, this allows us to drop the entire block within MoE models, thus enhancing efficiency. We introduce Block Drop as an extension Layer Drop, which also removes the attention layers. Specifically, for the i-th block, we assess importance score by evaluating the similarity between its inputs æ and outputs yı. Compared to Expert Drop, both Layer Drop and Block Drop focus on structures beyond expert level, with the ial to further enhance the efficiency of MoF models.

---

Published in Transactions on Machine Learning Research (03/2025)
4.3 Expert Slimming
Given that employing multiple experts in MoE significantly escalates parameters and inference costs, Ex- pert Slimming, stemming from single-model compression techniques, targets the compression of individual expert weights W exclusively. We denote any efficient transformation function as f(.), which encompasses pruning M @ W and quantization Quant(W), Through the application of such functions, we reduce the redundancy within each expert and create several light-weighted slim experts, thus improving their intrinsic efficiency. However, it is important note that Expert Slimming primarily focuses on compressing individual experts without addressing the redundancy across multiple experts. For maximum efficiency gains, Expert Slimming and Expert Trimming can be integrated to compress both individual experts and structured components. We summarize the efficiency contributions of all the discussed Expert Trimming and Expert Slimming methods in Table D highlighting the unique advantages of each approach.
5 Experiments on Expert Trimming
In this section, we evaluate the effectiveness of Expert Trimming techniques, starting with Expert Drop, and comparing it with our proposed methods, Layer Drop and Block Drop. Implementation details are provided in Appendix A
Performance Degradation with Limited Efficiency Gains While experts are specific structures in MoE, not all experts hold equal significance. Figure TT visualizes the distribution of expert-wise importance scores, highlighting this variability. To systematically drop experts at varying proportions, we conduct experiments using both layer-wise and global dropping approaches (see Appendix A.3 Given the Under
Expert Drop:
importance of shared experts (Appendix E we only dropped normal experts for PrepSeek--ME-BRe both settings, Expert Drop causes consistent performance degradation. For example, dropping 25% of experts in Mixtral-8x7B results in a 23% performance drop on the MMLU task. The efficiency improvement from Expert Drop is also marginal. For instance, dropping 12.5% of experts results in less than a 1% speedup, despite significant performance losses. More experimental results are available in Appendix
HellaSwag
WinoGrande
73
Mietral-3-75 DeepstekkMed--
Mictral-s-75 75 4oe-5 Rancoo Gureox
Mittrl 3x78
DrepSenk-MMt--1
Rantom Gurees
Ransto Guesc
Swntoon Guess
42-
12-
45
55-
33
19-
23
25% Dropped Percent
20%
75%
305%
0%
25
20%
30%%
35
25%
50%
15%
130%
85
25%
50%
70%
200
Dropped Percent
Drapped Percent
Dropped Percent
Figure 3: Evaluation of Expert Drop. We consider two strategies: layer-wise dropping (dotted lines) and global dropping (solid lines). "Random Guess" refers to randomly generating an output rather than using the model's predictions, serving as a baseline to assess the extent of performance degradation.
Layenwise Similarity
Layer Drop: Comparable Performance with Greater Ef- ficiency To verify the feasibility of Layer Drop, we visualize feature similarity across different modules in Figure 4 This vi- sualization shows a high level of similarity for features across the the MoE normalization module (Norm) and the MoE layer. In contrast, the low similarity for features across the MoF layer indicates the infeasibility of removing only MoE layers. Results from Figure show that Layer Drop preserves performance within a wide range of compression ratio, e.g. 1% performance drop on MMLU when dropping 8 layers for Mixtral-8x7B, revealing significant redundancy in the MoE layers.
1.0
6.8
Cosiiee Simllrrty
6.4
0.2
-0.2
15 Layer Index
32
Figure 4: Layer-Wal Similarity. We consider two scenarios, i.c., for "MoE" and "Norm + MoE".
7

---

Published in Transactions on Machine Learning Research (03/2025)
HellaSaag
MMLU
OBOA
WinoGrende
Mot 5-75
70-
Mixeal 24
Mazul 8-
DeecSeek MeC- 150
Rtancom Saaes
itandom Seance
Faando Guees
(Dancon Grasa
53-
20
Dropoed Lyurs
Dropoed Lyurs
Drasped Lyyar
Drosped Lyars
Figure 5: Evaluation of Layer Drop. We show results on Mixtral-8x7B and PrepSeek-MoE--T (solid lines), along with the bascline and random guess performances (dotted lines).
HellaSwag
MMLU
OBOA
WinoGrende
DeepS
Rarcom Gamee
Random Gumee
Fbrnto Gueas
Deencom Gumen
23
20 Dropped Blooks
15
16
20
Drasped Biockos
Drooped Buckks
Dropped Blooks
Figure 6: Evaluation of Block Drop. We show results on Mixtral-8x7B and DeepSeek-MoE-168 (solid lines,, along with the baseline and random guess performances (dotted lines).
Block Drop: Further Optimizing Efficiency by Pruning Entire Transformer Blocks While Layer Drop maintains the performance of the original models, it still preserves the computation-costt atten- tion layers. To address this, Block Drop extends Layer Drop by removing whole transformer blocks, includ- ing both MoE and attention layers, further reducing computational and memory costs. Figure 7 visualizes bisck-wise similarity, where both Mixtral-8x x 7B and DeepSeek-Mofe M6EE16B demonstrate high similarity be- tween specific blocks. Based on this observation, we conduct the empirical study by varying the number of dropped blocks.
Table 2: Comparison of Layer Drop and Block
Drop on dense and MoE models. "-Ln/m" "_ Bn/m" represents dropping n out of corresponding modules with Layer Drop and Block Drop, respec- lively.
Mistral-7B (Dense)
Method
ARC-C HellaSwag MMLU
OBQA
Average
Baseline
61.5
83.7
62.5
43.8
62.9
L4/32
53.2
77.7
61.7
40.0
58.2 (-4.7)
L8/32
36.7
33.6
53.3
30.6
38.6 (-213)
B4/52
66.1
77.5
61.6"
30.0
68.1 (448)
B8/32
40.0
63.9
60.0
30.6
48.6 (-14.3)
Mixtral- 7B (MoE)
Surprisingly, as shown in Figure 6 the Mixtral- 8x7B maintains over 90% of the original performance
Method
ARC-C
HellaSwag MMLU
OBQQ
Average
Baseline
59.4
84.0
67.9
46.8
64.6
even after removing 5 blocks (over 7 billion parame- ters). Similar observations are also found in DeepSeek- MoE-16B, where blocks can be removed when main- taining 90% performance. Since Block Drop removes
L4/32
56.2
81.3
67.6
44.6
62.4 (-2.2)
+ L8/32
17.7
75.2
67.3
40.0
57.6 (-7.0)
F BT/322
"3.
"50.2
67.9
13.0
61.2 (3.1)
B8/32
40.8
55.8
66.3
37.2
50.0 (-14.6)
computationally expensive attention layers, it outperforms Layer Drop by a large margin in terms of both memory and inference cost, as illustrated in Figure
On the other hand, Block Drop prunes attention layers along with their corresponding KV-Cache Pope et al. 2022). For instance, an input sequence with a batch size of 128 and a sequence length of 2048 results in 32GB of KV-Cache, which can be reduced by 5GB using Block Drop. Overall, by targeting higher- level structures, Layer Drop and Block Drop achieve substantial efficiency improvements while maintaining acceptable performance levels.
8

---

Published in Transactions on Machine Learning Research (03/2025)
Mixtral-*+78
Seek-SekMMoE- Me-e-M
Mitrra-8x7B6
DeepSeek- Moteeii--ME-166
Decoding Speed (TOS)
Drop
Drop
Block
Book Drp
5000
23-4
5
10
Block Index
20
30
13
Block Intes
20
25
5%
15%
28% Compression Ratio
35%
47%
50t
65
10%
Compression Rado
20%
60h
42%
50%
Figure 7: Normalized Block- We measure the cosine similarity among hidden features between blocks.
Wise Wise Similarity.
Figure 8: Speedup Scaling Curves Expert Trimming Methods. where we measure the aver- aged decoding speed during generation.
MoE Layers are More Redundant than Dense Counterparts Since Layer Drop and Block Drop can also be applied lo dense models, we take Mistral-7B, the corresponding dense model of Mixtral-8x77B for comparison. Both models have the same depth and differ only in the FFN implementation, so we remove the equal number of blocks, both MoE and dense
number of layers blocks from cach. When dropping models exhibit performance degradation. However, the MoE model suffers less performance drop under the same compression setting. For example, when dropping 8 MoF layers, the Mistral-7B receives a performance drop of 24.3, while Mixtral-8x7B only receives drop of 7.0. This interesting finding highlights the higher redundancy in MoE layers, and further validates the effectiveness of applying Layer Drop and Block Drop to MoF models.
6 Visualization Examples of Layer Drop and Block Drop
In this section, we visualize the layer-wise similarity and the corresponding dropping order of MoE layers and blocks to investigate the varying levels redundancy across different depths.
Since our similarity-based metrics depend on the hidden states cach block, the choice of data may influence feature similarity across layers. To investigate this, we conducted ablation studies Mixtral-8x7B. examining both the number of samples and the types of datasets used for feature extraction. This analysis helps understand how data selection affects decisions regarding the dropping of layers or blocks. The results are presented in Figure 9
Layer Similarity
Layer Similanity
Sample Number
Meso
Mach
Lima
10
20
25
30
16
Layer
Block Similarity
Block Similarity
Sample Number
$12
Mach
Lio
10
Block Index
Block Index
(a) Similarities under different number of samples.
(b) Similarities under different datasets.
Figure 9: Influence of Data Choices on Feature Similarity. We measure the similarity among layers and blocks Mixtral-8x7B. (a) The similarity calculated using different number of samples from C4 Raffe] et al. (2019). (b) The normalized similarity calculated using 1, 0224 samples from different. datasets, i.e., C4, Lima Zhou et al. 2023) and MetaMathQA u et al. (202)).

---

Published in Transactions on Machine Learning Research (03/2025)
Robustness to Calibration Datasets In Figure 9a we note that the feature similarity remains relatively stable across different layers the sample size increases, indicating that Layer Drop and Block Drop maintain consistency regardless of the sample quantity. This confirms that using 128 samples suffices for computing similarity, which adopted for all our experiments. Similarly, Figure 9b shows that varying the datasets, from preiraining with C4 to instruction luning with Lima and MetaMathQA, does not significantly alter the feature similarity. This demonstrates the resilience of Layer Drop and Block Drop to variations in data distribution.
78
DcepSeok
Mixtral-8x
DeepSeck
Dropppod Number
Droggoo Number
10
15
15
20
20
23
25
10
15
20
30
31
36
20
15
25
20
25
20
20
35
20
25
Layer Index
Layer Index
Block Index
Block Index
(a) Layer Drop
(b) Block Drop
Figure 10: Dropping Patterns for Layer Drop and Block Drop. We visualize of the remaining layers and blocks under different dropped numbers, where yellow areas represent the retained portions and red areas indicate the dropped
blocks.
Redundant Deeper Layers Figure visualizes the remaining and dropped layers/blocks as the number of dropped modules increases. Both MoE architectures exhibit similar patterns in Layer Drop and Block Drop: initially, both models tend to drop the deeper layers, followed by the shallower ones. These findings are consistent with Xu et al. Men et al. 2024). which suggests that deeper layers tend to be more redundant.
7 Integration of Expert Trimming and Expert Slimming
Beyond Expert Trimming, another avenue for MoE compression is Expert Slimming, which targets individual expert compression. Techniques such as quantization and network pruning are among the most commonly employed methods. We provide a detailed comparison of network pruning and quantization in Appendix B where the quantization outperforms both in performance and efficiency.
Since Expert Trimming and Expert Slimming focus different aspects of compression, we further explore their potential integration. Given the superior average performance and practical efficiency of quantization, we use it for Expert Slimming. For Expert Trimming, we include all three methods to provide a complete comparison. The discussion on the order of applying these two compression techniques is provided in the Appendix
Quantization Preserves the Performance of Expert Trimming As shown in Table 3 the integration of Expert Slimming and Expert Trimming significantly enhances overall efficiency. Quantization can be seamlessly combined with three different levels of dropping, achieving comparable performance. For instance, after quantization, the average performance of Layer Drop and Block Drop is nearly the same, maintaining more than 90% of the performance the original models.
The Integration Significantly Enhances Efficiency In Table 3 the integration of Expert Trimming and quantization promotes efficiency by large margin. Different Expert Trimming strategies showcase different advantages. Specifically, Expert Drop contributes to reducing memory usage, but its specdup is marginal. Layer Drop and Block Drop excel in speedup as illustrated in Figure S with Block Drop demonst rating both higher performance and greater speedup. Taking into account all settings, the combination of Block Drop and quantization offers the best efficiency with comparable performance: a 6.05x speedup with only 20.0GB memory usage, while maintaining over 92% of the performance on Mixtral-8x 7B, making it available to be deployed on NVIDIA RTX 3090 GPU.
10

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: Experimental Results of the Integration of Expert Trimming and Expert Slimming. --En/m" denotes dropping n out of experts per MoE layer on average. "-In/m", "-Bn/m" represents dropping n out of layers/blocks with Layer Drop and Block Drop, respectively. The speedup quantifies the relative reduction in processing time achieved by the compressed models compared to the bascline. Both the speedup and FLOPs are evaluated by running forward pass on input sequence of length 2, 2404
Mixtral-8x7B
Method
SpeedUp FLOPs
Memory
ARC-C BoolQ) HellaSwag MMLU OBQA
PIQA
RTE WinoGrande Avg.
Baseline
54.4T
87.7GB
59.4
84.2
84.0
67.9
46.8
83.8
70.4
75.6
71.5
w/AWQ
5.08x
54.41
24.4GB
58.4
84.2
83.3
66.6
45.8
83.0
69.0
76.3
70.8
E2/8
1.06x
54.4T
66.7GB
53.2
77.7
80.5
52.2
46.2
81.7
55.6
76.8
65.5
W/ w/WQQ
5.28x
54.TT"
20.1GB
50.7
79.1
78.9
52.4
44.2
"81.2"
55.6
75.9
61.8
L8/32
1.19x
42.9T
66.6GB
47.7
85.3
75.2
67.3
40.0
75.8
69.7
74.6
67.0
w/AQQ
6.05
42.9T
20.0GB
46.2
84.2
74.2
66.2
39.0
75.5
69.3
74.2
66.1
B5/32
1.17x
46.0T
71.1BB
51.3
85.3
78.7
67.9
42.0
79.3
69.7
74.3
68.6
w/AWQ"
5.94x
16.0T
21.9GB
50.6
85.1
77.5
66.9
11.1
76.1
71.8
74.5
68.0
DeepSeek MoE-16B
DeepSeek-MoE-16B
Method SpeedUp FLOPs
Memory
ARC-C BoolQ HellaSwag MMLU OBQA
PIQA
RTE WinoGrande Avg.
Baseline
11.7T
30.8GB
48.1
72.4
77.3
37.9
44.0
80.4
63.9
70.3
61.8
w/AWQ
3.16x
11.7T
9.8GB
46.8
71.2
76.6
36.4
43.6
80.1
62.1
70.1
60.9
E16/64
1.06x
11.7T
23.9GB
45.0
67.1
75.6
31.8
42.2
80.2
59.9
70.0
59.0
w/AWQ"
3.34x
II.FI
7.7GB"
44.0
66.0
74.5
27.9
42.6
788.3"
56.3
67.3
57.1
L4/28
1.14x
10.6T
26.6GB
39.5
70.2
67.6
35.2
40.4
75.8
48.4
65.7
55.3
w/AWO,
3.60x
10.6T
8.5GB
42.1
72.0
69.2
33.7
39.8
75.1
47.7
66.5
55.8
B4/28
1.16x
10.1T
26AGB
71.3
69.0
36.2
37.8
75.8
68.0
56.3
w/AWQ
3.67x
10.17
8.4GB
40.1
70.2
68.6
36.1
38.4
76.2
31.6
66.4
56.0
8
Post-Finetuning Recovers the Performance
While the discussed compression techniques maintains most of the performance of the original models, further conduct post-finetuning to recover the degraded performance. Specifically, for comparison, we full-finetune DeepSeek-MoE- 16B and corresponding compressed models on the Alpaca-GPT4 dataset (Peng et al. 2023 for 3 epochs using a learning rate of 8e-6 with 0.03 warmup ratio and cosine scheduling, where the global batch size is set to 32. As shown in Figure g the post-finctunng process significantly reduces the performance gap between the compressed models and the original models, e.g. narrowing it from 5.5% to 0.6% for the model following Block Drop.
Table 4: Performance of the Deep56ek-MoE-1133 models finetuned after Expert Trimming. We mark the relative average performance loss of the compressed models compared to baselines in brackets.
DeepSeek-MMo--1G.
Method SpeedUp FLOPs Memory ARC-C BoolQ HellaSwag MMLU OBQA
PIQA
RTE WinoCrande
Average
Baseline
48.1
72.4
77.3
37.9
44.0
80.4
63.9
70.3
61.8
11.7T
30.8GB
+SFT
44.6
75.3
79.0
40.3
44.6
80.3
70.4
71.7
63.3
+ E16/64
45.0
67.1
75.6
31.8
42.2
80.2
59.9
70.0
59.0
(-2.8)
1.06x
11.7T
23.9GB
+SPT
44.4
74.0
78.6
38.5
45.8
79.6
65.7
70.1
62.1
(-1.2)
L4/28
39.5
70.2
67.6
35.2
40.4
75.8
48.4
65.7
55.3
(-6.5)
1.14x
10.6T
26.6GB
+SFT
42.1
78.9
75.2
40.8
43.4
77.6
71.1
69.5
62.3
(-1.0)
B4/28
40.3
71.3
69.0
36.2
37.8
75.8
51.6
68.0
56.3
(-5.5)
1.16x
10.1T
26.4GB
+ST
48.2
78.2
75.0
40.4
43.8
76.8
74.0
70.2
62.7 (-0.6)
11

---

Published in Transactions on Machine Learning Research (03/2025)
9 Conclusion
In this paper, conducted holistic study of MoE compression techniques, facilitating systematic understanding of the efficiency issue of MoE and identifying the new design space to improve the performance further. Based on this study, we propose a comprehensive recipe that integrates Expert Slimming and Expert Trimming to further enhance efficiency. Our proposed methods and insights not only address current. challenges but also set the stage for future advancements the field of MoE.
References
Winogrande: An adversarial winograd schema challenge at scale. 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
Yue Bai, Huan Wang, ZHIQLANG TAO, Kunpeng Li, and Yun Fu. Dual lottery ticket hypothesis. In International Conference Learning Representations, 2022. URL http://///ppeeeeeeeeeeeettrrrree
f0aN52jn251
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.
Tianlong Chen, Zhenyu Zhang, AJAY KUMAR JAISWAL, Shiwei Liu, and Zhangyang Wang. Sparse moe the new dropout: Scaling dense and self-slimmable transformers. In The Eleventh International Conference Learning Representations, 2022.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang, A survey of model compression and acceleration for deep neural networks, 2020.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference machine learning, pp. 4057 4086. PMLR, 2022.
Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:240100600 06066, 2024.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, al. Glam: Effficient scaling of language models with mixture-of-experts. In International Conference Machine Learning, pp. 5547 5569. PMLR, 2022.
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. Tayerskip: Fnabling early exit inference and self-speculative decoding, 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1 39, 2022.
Elias Frantar and Dan Alistarh. SparseGPT:: Massive language models can be accurately pruned in one-shot. arXiu preprint srXiv:2301.00774, 2023.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint or Xiv:22000 17323, 2022.
12

---

Published in Transactions on Machine Learning Research (03/2025)
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace Нe, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL http:://aacoooorrrreee/////////0
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer. Demix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp- 5557 5576, 2022.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Shwai He, Liang Ding, Daize Dong, Boan Liu, Fuqiang Yu, and Dacheng Tao. PAD-net: An efficient. framework for dynamic networks. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14366, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: URL
10.18653 aww./...cloooo
wrg.:/oooo.rr//222 acl-long. 803
1803
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Malthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient tneggr-arithmeeic-nnk inference, 2017.
Robert A Jacobs, Michael I Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural compulation, 3(1):79 87, 1991.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of exports. arXiv preprint ar arXiv:2401.0400 2024.
Michael I Jordan and Robert A Jacobs. Hicrarchical mixtures of experts and the em algorithm. Neural compulation, 6(2):181 214, 1994.
Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaplive sparsity for the magnitude-based pruning. In International Conference on Learning Representations, 2021. URL https: //openreview.c www..n//oooum/ddi/A........
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint Xiv:2000 16668, 2020.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pp. 6265 6274. PMLR, 2021.
Guangyan Li, Yongqiang Tang, and Wensheng Zhang. Lorap: Transformer sub-layers deserve differentiated structured compression for large language models, 2024a.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Pingzhi Li, Zhenyu Zhang, Prateck Yadav, Yi-Lin Sung, Yu Cheng. Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystily efficient SMoe with hints from its routing policy. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openrevieer www. pett//oruuri.ddeeaae/3333
Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acceleration: A survey, 2021.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang. Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. In MySys, 2024.
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of ex-exps large language models, 2024.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct clectricity? new dataset for open book question answering, 2018.
Alexandre Muzio, Alex Sun, and Churan He. Seer-moe: Sparse expert efficiency through regularization for mixturee-f-exxperss 2024.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. white paper on neural network quantization, 2021.
OpenAL. Gpt-4 technical report, 2024.
Baolin Peng, Chumyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. Xrxiv preprint arxiv220040002 2023.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza. Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moer Advancing mixture-of-experts inference and training to power cext-generation ai scale. In International conference on machine learning, pp. 18332-18346. PMLR, 2022.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute transformer-based language models, 2024.
Luka Ribar, Ivan Chelombiev, Luke Hudlass- Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth--fffiient IIm inference, 2024.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Newral Information Processing Systems, 34:8583-8595, 2021.
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models, Advances in Neural Information Processing Systems, 34:17555-17566 2021.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Noam Shazeer, Azalia Mirhoscini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexxperts layer. or Xiv preprinl arXiv:1701.06338 2017.
Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, and Chuang Gan. Modulcformer: Learning modular large language models from uncurated data. ar preprinl arXie:2306.0400 2023.
Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen, Powerinfer: Fast large language model serving with a consumer-grade gpu, 2023.
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter, A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 11899066.16690 11695, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloviin llama model. https://github. com/ latts--labsttanford alpaca, 2023.
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024a.
Qwen Team. Qwenl.5-moc: Matching 7b model performance with 1/3 activated parameters", February 2024b.
URL https::/qqenimm.iiiae htt.://..iiiiubb.ooooooggggogggo qwweg////wemmmme/eme
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR.
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An carly effort on open mixture-of-experts langnage models. arXiv preprint arXiv:2402.01 01 789, 2024a.
Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. Moe-infinity: Activation-aware expert offloading for efficient moe serving, 2024b.
Dianhai Yu, Liang Shen, Hongxiang Hao, Weibao Gong, Huachao Wu, Jiang Bian, Lirong Dai, and Haoyi Xiong. Moesys: A distributed and efficient mixture--f-experts training and inference system for internet services, 2024. URL hnttps://aaatit wwarxivv.rg///e. wtw.//220...... 10034.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint XrXi::2309, 000009.222990 12284, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yojin Choi. Hellaswag: Can a machine really finish your sentence?, 2019.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc Le, James Laudon, et al. fixturr-of-experts with export choice routing. Advances in Neural Information Processing Systems, 35:7103-77 -114, 2022.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression, 2017.
Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moo moc: Building mixture-oofeexperts from llama with continual pre-training. oixie preprint ar Xrx:::2406 16554, 2024. URL http:://aax.x..oo/////////////// 16554
16

---

Published in Transactions on Machine Learning Research (03/2025)
A Implementation Details
A.1 Models and Datasets
Models For our experiments, we employed Mixtral-8x7B (Jiang et al. 2024 and DeepSeck-MoE-16B (Dai el al. 2024). Mixtral-8x7B utilizes 8 experts for MoE layers and activates the top two for each input token. In contrast, DeepSeek-MoEi MorpSeekkMME-aee 16BB-16B employs a dense FFN in the first block and utilizes two shared experts with an additional 64 experts within MoE layers in other blocks.
Datasets For compression experiments, we used the C4 dataset (Raffel et al. and an input sequence length of 2,048, following the setup in (Sun et al. 2023 Lu et al. 2024 Lin et al. Frantar et al. 2022). To evaluate model performance,
2019 with 128 samples
2024
report norma lized zero-shol accuracy on the LM- benchmark, which includes multiple tasks: ARC-C (Clark et al. 2018), BoolQ (Clark et al.
2019), HellaSwag (Zellers et al. 2019)) MMLU (Hendrycks et al. 2021 OBQA (Mihaylov et al. 2018 PIQA
Bisk et al. 2019) RTE [Wang et al. 2019 and WinoGrande
ai2 2019
The evaluation code based on
EleutherAl
LMM Harness
(Gao et al.
2023)
A.2
Implementation Details of Expert Slimming
Both Expert Slimming methods (i.e., pruning and quantization) require calibration data to estimate input statistics. To control this variable, we use 128 samples from the C4 dataset (Raffel et al. 2019) as the calibration dataset for pruming. For quantization, we follow the default settings of GPTQ and AWQ using 128 random samples from Alpaca (Taori et al. 2023) and Pile (Gao et al. 2020 respectively. We use the default group size 128 for Mixtral-8x x and 64 for DeepSoek-MoF-168.
A.3 Implementation Details of Expert Drop
The Expert Drop compresses MoE by preserving only important experts { (EiieT while removing others, where T' is determined by the importance scores (S(E,))eT) Following Muzio et al. 2024]. measure the importance scores through the averaged routing scores of a batched data X, ie., {S and consider two dropping strategies for Expert Drop: layer-wise dropping and global dropping.
কि ExEx Gx
Layer-wise dropping removes the same number of experts for each layer. Given the total number of experts n IT] and the preserved number of experts n' = IT'l < in layer 1, the preserved expert set (T()) is obtained by:
T
(E,"). where S(E,)) () (.....
(10)
Global dropping constrains the total number of preserved experts for the entire model. Given the total number of layers L in the model, the preserved expert set ( for layer I is obtained by:
m T (E("), where S(E!)) E Topk ((S(E 1=1
(11)
For the integration of Expert Slimming and Expert Trimming, we choose the global dropping as the strategy of Expert Drop, which shows competitive performance compared to the layer dropping for Mixtral-8x7B under low dropping ratios, well as consistent better performance for DoopSock-MoE-167 in Figure 1
17

---

Published in Transactions on Machine Learning Research (03/2025)
B
Expert Slimming
Comparable Performance with Deployment Challenges. In Table 5 we evaluate rep- resentative pruning algorithms (i.e., Wanda (Sun et al. 2023 SparseGPT (Frantar & Alistarh 2023 on Mixtral-8x7B and DeepSeek-MoF-16B. Since DeepSeek- MoF-166 utilizes both shared experts and normal experts, we conduct an ablation study on whether to prune shared experts, as discussed in Appendix We find that unstructured pruming preserves more than 95% of performance. However, is not compatible with existing hardware. Conversely, the hardware-friendll semi-structured pruning (i.e., 4:8 and 2:4 patterns) undergoes significant performance drop. Nevertheless, according to Lu et al. 2024 semi-structured sparsity is ineffective in speeding up MoE models.
Pruning:
Table 5: Performance of Pruning MoE. We consider two mainstream pruning methods (i.e., Wanda (Sun et al. 2023) and SparseGPT (Frantar & Aliisarr, 2023)) under 50% unstructured sparsity and 2:4 scmi-structured sparsity.
Mixtral-8x70
Method
Sparsity
ARC-C
BoolQ
HellaSwag
MMLU
OBQA
PIQA
RTE
WinoGrande
Avg.
Baseline
0%
59.4
84.2
84.0
67.9
46.8
83.8
70.4
75.6
71.5
Wanda
56.1
85.8
81.7
64.3
46.4
82.2
65.0
76.0
69.7
50%
SparseGPT
56.4
85.7
81.5
64.6
45.0
82.4
66.8
75.8
69.8
Wanda
51.4
79.4
77.8
60.3
44.0
80.7
65.3
74.1
66.6
SparseGPT
2:4
49.2
81.0
77.6
59.2
44.0
80.6
63.9
74.8
66.3
DoopSock-MoE-1GE
Method
Sparsity
ARC-C
BoolQ
HellaSwag
MMLU
OBQA
PIQA
RTE
WinoGrande
Avg.
Baseline
0%
48.1
72.4
77.3
37.9
63.9
70.3
Wanda
43.6
74.3
72.6
31.1
43.0
79.5
58.1
69.4
59.0
50%
SparseGPT
43.9
73.5
74.0
33.8
41.4
79.0
61.0
68.3
59.4
Wanda
38.2
66.1
67.5
27.6
39.77
777.0
53.8
66.7
54.5
SparsePPT
2:4
43.1
68.9
71.6
27.6
41.6
78.3
57.4
66.6
56.9
Quantization: Better Performance and Greater Efficiency In Table 6 we evaluate the impact of 4-bit quantization on MoE. The quantization offers two major benefits: it maintains the comparable performance of the original models and siguificantly reduces memory costs. Specifically, the quantized models achieve over 98% of the original performance while using less than 30% of the memory. When quantized with AWQ (Lin al. 2024), Mixtral-8x77 and DeepSock-MoE-16B achieve impressive speedups of x5.08 and x3.16, respectively. This demonstrates that 4-bit quantization is elfective for deploying MoE models in resource-constrained environments.
Table Performance of Quantization on MoE. We utilize GPTQ (Frnnaar et al.
2022 and AWQ Lin
al.
2024 as the quantization methods for 4-bit compression.
Mixtral-8 7B
Method
Bits
Memory
ARC-C
BoolQ
HellaSwag
MMLU
OBQA
PIQA
RTE
WinoGrande
Avg.
Baseline
16
87.7GB
59.4
84.2
84.0
67.9
46.8
83.8
70.4
75.6
71.5
GPTQ
59.0
84.4
83.4
67.1
45.2
83.1
70.1
75.2
70.9
4
24.4GB
AWQ
58.4
84.2
83.3
66.6
45.8
83.0
69.0
76.3
70.8
DeepSeek MoE-16B
Method Bits
Memory
ARC-C
BoolQ
HellaSwag
MMLU
OBQA
PIQA
RTE
WinoGrande
Avg.
Baseline
16
30.GGB
48.1
72.4
77.3
37.9
44.0
80.4
63.9
70.3
61.8
GPTQ
16.3
71.8
76.8
36.4
43.4
80.0
63.9
70.2
61.1
4
9.8GB
AWQ
46.8
71.2
76.6
36.4
43.6
80.1
62.1
70.1
60.9
18

---

Published in Transactions on Machine Learning Research (03/2025)
C Analysis on the Dropping Patterns of Expert Drop
Score Distribution Directs Expert Drop The
Score Distribution
distribution of importance scores is informative to determine the proportion dropped experts.
Mixtral-8x70 7B
DeepSeek Mot-168
Drop Percent
Drop Parcent
25%
25%
Figure I1 we visualize the score distribution Ex- pert Drop for Mixtral-8x7B 7B and DeepSocl-MoE-Ioft DeepSeek-MoF-1661, which allocates shows left-skewed distribution where most experts have low scores. In contrast, Mixtral-8x7B demon- strates a right-skewed distribution, with only few experts being deemed unimportant. This distribu- tion difference results in different resistance capa- bilities against Expert Drop, where DeepSeek-MoF- 16B can drop more experts than Mixttral-8x7B while maintaining competitive performance, as demon- strated in Table 3 and Figure 13
50%
50%
75%
experts,
Dorstty
0.0
0.2
0.4
0.6
5.8
1.0
0.0
5.2
0.4
0.5
0.8
Normalized Stores
Normalized Scones
Figure 11: Distribution of Normalized Impor- tance Scores S for Expert Drop. We highlight the density of scores under different drop ratios with different colors.
Global Expert Drop Removes Experts Finc- Grainedly We employed two different stralegies for Expert Drop, namely layer-wise and global. Layer-wise dropping treats cach layer equally by dropping the same number of experts, while global dropping results in different proportions of remaining experts across layers. We visualize the distribution of remaining experts after global dropping in Fig-
Mistralaxx7
Geep
10000
0136
Raseved Excurts
Roevad
ure 12 We find the global dropping shows a more fine-grain gne-grained pattern on dropping experts, where the bottom layers are more vulnerable under lower drop- ping ratios (yellow part).
Figure 12: Distribution of Dropped Experts for Expert Drop. We visualize the dropped experts under different drop ratios, where the dropped experts are colored from yellow to blue as the drop ratio increases.
19

---

Published in Transactions on Machine Learning Research (03/2025)
D Ablation Study on Compression Orders
In Section i wC discussed the combination of Expert Trimming and Expert Slimming. Here we ablate on the orders of compression when combining these two techniques. Results in Table show that the order of Expert Trimming and Expert Slimming doesn't have significant influence on the performance, where applying Expert Slimming then Expert Trimming ("S+T"") performs slightly better for Mixtral-8x7B (e.g. +0.5, +0.4 and +0.1 for Expert Drop, Layer Drop and Block Drop, respectively). To this end, we choose "S+T" the final implementation in experiments.
Table 7: Ablation results on different orders of Expert Slimming and Expert Trimming. "S+T" denotes first applying Expert Slimming then Expert Trimming, and "T+S" denotes the reversed order.
Mixtral-8x7B 7B
Method
ARC-C
BoolQ
HellaSwag
MMLU
OBQA
PIQA
RTE
WinoCrande
Avg.
Baseline
59.4
84.2
84.0
67.9
46.8
83.8
70.4
75.6
71.5
+ F2/8, AWQ (S+T)
50.7
79.1
78.9
52.4
44.2
81.2
55.6
75.9
64.8
I E2/8, AWQ (T+S)
50.8
79.9
78.7
49.2
44.4
80.9
55.2
75.4
64.3
+ L8/32, AWQ (S+T)
46.2
84.2
74.2
66.2
39.0
75.5
69.3
74.2
66.1
+ L8/32, AWQ (T+S)
46.8
84.4
74.0
65.3
39.8
75.0
66.8
73.2
65.7
+ B5/32, AWQ (S+T)
50.6
85.1
77.5
66.9
41.4
76.1
71.8
74.5
68.0
+ B5/32, AWQ (T-S)
50.3
84.7
77.4
65.8
42.0
78.8
70.4
74.0
67.9
DeepSeck-MoF-107
Method
ARC-C BoolQ HellaSwag MMLU OBQA
PIQA
RTE WinoGrande
Avg.
Baseline
48.1
72.4
77.3
37.9
44.0
80.4
63.9
70.3
61.8
+ E16/64, AWQ (S+T)
44.0
66.0
74.5
27.9
42.6
78.5
56.3
67.3
57.1
+ E16/64, AWQ (T+S)
44.7
64.1
29.0
42.6
79.9
54.2
68.4
57.1
+ L4/28, AWQ (S+T)
12.1
72.0
69.2
33.7
39.8
75.1
47.7
66.5
55.8
+ L4/28, AWQ (T+S)
42.4
71.7
69.1
33.4
40.1
74.8
66.2
55.7
+ B8/28, AWQ (S+T)
40.1
70.2
68.6
36.1
38.4
76.2
51.6
66.4
56.0
+ B4/28, AWQ (T S)
41.6
69.4
35.8
38.6
76.2
50.9
67.0
56.1
20

---

Published in Transactions on Machine Learning Research (03/2025)
E
Ablation Study on Shared Experts in DeepSeek PeepSeek-MMo-168 MoE-16B
Although most MoE models follow Equation implement the experts, models like DecpScck-MoE-168 adopt a residual (Rajbhandari et al. 2022) form of experts, which brings special scenario to discuss. In the residual MoF, an additional set of m shared experts {E1, Ez,... Æm is always selected by the router and activated for inputs. Given an input æ, the output can be represented as a degenerated form of Equation where the scores of shared experts are fixed to 1:
y Σ iEK G(x); E,(x)
Σ T Ex)
(12)
This special form of expert routing may bring difference in the redundancy distribution of MoE. Here discuss the influence of shared experts through pruning and present the results in Table & We find that pruning without the shared experts will boost the performance al considerable scale, i.e., +3.6% and +1.5% of the averaged accuracy for unstructured pruming with Wanda and SparseGPTT, respectively. This finding reveals a different pattern of inner redundancy which the shared experts are less compressible compared to the others in residual MoE models, which may inform future work.
Table 8: Ablation Study of Pruning Shared Experts on DeepSeek-MoB-166.. We consider two scenarios, i.e, pruning both shared experts and normal experts ("w/Pruning Shared Experts") and pruning normal experts only ("w/o Pruning Shared Experts"). We use two mainstreaam pruming methods (i.e., Wanda (Sun al. 2023) and SparseGPT (Frantar & Alistarh 2023 under both unstructured sparsity (50%) and semi-structured sparsity (2:4).
DeepSeck-MoE-16B
Method
Sparsity
ARC-C
BoolQ
HellaSwag
MMLU
OBQA
PIQA
RTE
WinoGrande
Avg.
Baseline
0%
48.1
72.4
77.3
37.9
44.0
80.4
63.9
70.3
61.8
w/ Pruning Shared Experts
Wanda
43.6
74.3
72.6
31.1
43.0
79.5
58.1
69.4
59.0
50%
SparseGPT
43.9
73.5
74.0
33.8
41.4
79.0
61.0
68.3
59.4
Wanda
38.2
66.1
67.5
27.6
39.4
77.0
53.8
66.7
54.5
2:4
GPT
43.1
68.9
71.6
27.6
41.6
78.3
57.4
66.6
56.9
w/o Pruming Shared Experts
Wanda
44.0
76.3
73.5
36.2
41.0
79.3
59.9
70.2
60.0
50%
GPPrseGPT
45.0
75.5
74.4
36.3
41.0
79.4
64.3
69.3
60.7
Wanda
40.1
75.7
69.9
33.5
40.0
77.9
58.8
68.6
58,1
2:4
SparseGPTT
40.7
75.7
69.9
33.3
39.0
77.7
61.4
69.4
58.4
21

---

Published in Transactions on Machine Learning Research (03/2025)
F
Full Experimental Results
We provide the full results of Expert Trimming, including Expert Drop, Layer Drop and Block Drop, in
Figure
13
14
and
15 respectively.
Mixtral-8x77 7B
ARC-C
Booi
Hell Hewla
MMLU
Glibal
Kandoni
Handam
Guess
Randam
Accuracy
56-
52-
28
Dropped Experts
Dropped Experts
Dropped Experts
Dropped Experts
OBDA
PIDA
RTE
WinoGranda
Layoreess
Layorese
Layoriase
Layerwbee
Glatul
Glital
dlcba
Glcba Random Goses
Random Guess
Random Gume
Random Guee
58-
Dropped Experts
Dropped Experts
Drapped Experts
Dropped Experts
DeepSeek-MoEs MoE-16B
ARC-C
BoolQ
Hell aSwag
MMLU
42-
75
28
Kandom
12
Random Guess
Handam Guess
Randam Guess
15-
65
Acourasy1S)
43
23
32
Drapped Experts
Dropped Exparts
Dropped
Drapped Exparts
OBQA
PIQA
RTE
WinoGrande
Deyersetee
610-
Layenaate
Glotal
Giobal
Giotal
Gett
Random Guess
Handom Guess
Handam Goess
Randum Goos
42-
011
03
$5.-
55-
53
510
Drapped Experts
Dropped Experts
Drapped Experts
Dropped Experts
Figure 13: Full Results for Expert Drop. We consider two strategies: layer-wise (dotted lines) and global (solid lines).
22

---

Published in Transactions on Machine Learning Research (03/2025)
HollaSwag
Midra-Bi28 DuccBuck MiE 188 Random Quess
Micini-tt T1
Maire-t-1
DuspSook M&E 166 Random Gusss
DoopBrok MuE
Random Ouess
Ra Randon Oueso
43
Dropoed Lyers
Dropped Lyars
Drogped Lyors
Drosped Lyors
OBDA
PIQA
RTE
WinoGmadd
Mixini 2x
Matra--aop
OuepBuckMMcE 188
OuepSeoi-AtE MoE- 108
DespSeci-MME 108
MoepBeckMMEE.oo 158
Random Quess
Random Dusss
Randam Guess
Random Oues
29
20
Dropped Lyars
Dropoud Lyors
Drooped Lyyar
Crooped Lyors
Figure 14: Full Results for Layer Drop. We show results on Mixtral-8x7D 7B and DeepSeek-MoE-110 (solid lines), along with the baseline and random guess performances (dotted lines).
HellaSwag
Midrai-Bi78 DeecSeok MrE 15
Micimal-- 3+ TI
Moire-t-TD
Viltost80-75 DespSeek MIE 15B
DeepSeei MNE 166
DoepSest MUE/160
Randem Guasss
Random Ceunes
Randon Guees
Random Greea
12
Dropped Blooks
Drosped Blocks
Drogped B Blocks
Dropped Blocks
OBQA
PIQA
RTE
WinoGrande
Mitt 5x78
Mixiaal 3:
Misril 2x78
Mittual 8-18
Buck
958
Ducppook NVE 168
DeepBook MrE.
DeopBook MaE 198
Random Guess
03
Random Ouess
Fandon Guess
Ra Ranitom Oues
75
70
00
55
Dropped
Dropped Blocks
Dropped Blocks
Dropped
Figure 15: Full Results for Block Drop. We show results on Mixtral-8x7B and Doe-S6ek-MMF-166 (solid lines), along with the baseline and random guess performances (dotted lines).
23


Reviews:
[{"review_id": "m4Cbg52sqT", "reviewer": "Reviewer_gQjV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This paper is well written, the preliminary section 3 and overview section 4.1 provide clear background to the readers.\n- The experiments are comprehensive, the pros and cons of expert trimming (expert drop, layer drop and block drop), expert slimming and their combinations are well studied, which can benefit the further research in this field.\n\nWeaknesses\n- The novelty of this paper may be modest.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores compression techniques for MoE models to improve efficiency and scalability. The authors introduce Expert Trimming and Expert Slimming to reduce computational and memory costs while preserving performance. Experiments on Mixtral-8×7B show a 6.05× speedup, 77.1% lower memory usage, while remaining over 92% initial performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Will dynamic pruning experts/layers/blocks be better than static pruning? This can be an interesting experiment to add.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0naQXe5f7Z", "reviewer": "Reviewer_Fcnp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n1. The paper explores various compression techniques including expert slimming (pruning and quantization) and expert trimming (expert/layer/block drop). Also, it explores the integration of expert slimming and expert trimming, showing that the integration approach yields good result.\n2. The paper conduct extensive analysis on compression techniques using Mixtral and DeepSeek models. \n\n**Weaknesses**\n1. Although the paper explores various compression techniques, it lacks novelty. For instance, expert slimming is already widely used technique, and expert trimming (expert/block drop) is already proposed in other papers ([1], [2]). It could explore what would happen if the model mixes different granularities of trimming. \n\n[1] Muzio, Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts, 2024.\\\n[2] Raposo, Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024.\n\n2. Graph legends can be more friendly. For instance, in Figure 3, what exactly is `Random Guess`? If it means randomly select which experts to drop, how it is flat as dropped percent changes?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates methods to improve MoE efficiency through compression techniques. It introduces layer/block drop expert trimming techniques. It integrates expert trimming with expert slimming (quantization and pruning) and validate the work on Mixtral-8B and DeepSeek-16B models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Refer to weaknesses section above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QnliFxD8Uh", "reviewer": "Reviewer_uKt4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper makes comprehensive experiments on different aspects of compression dimension. \n2. The post fine-tuning stage is proven to be effective for recovering initial model's performance.\n\nWeaknesses:\n1. While the paper's motivation is towards efficient MoE, most of the compression techniques are also available for dense models. MoE-specific compression methods are required to strengthen the motivation.\n2. Expert Slimming seems to be re-branding because only AWQ is used in the main paper. Even though pruning methods are compared in the Appendix, there are both existed techniques and irrelevant to MoE.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper makes a holistic study of compressing Mixture-of-Experts (MoE) language models. Two types of compressing techniques are introduced: expert trimming and expert slimming. \n\nBased on Expert Drop, this paper extend Expert Trimming to a higher architectural level by introducing Layer Drop and Block Drop, significantly improving the efficiency while maintaining the model performance. Besides,  this paper introduce Expert Slimming, leveraging orthogonal techniques like quantization to further compression.\n\nThe experiments show that Layer Drop and Block Drop can increase the efficiency while maintaining most of the capabilities, while Expert Drop leads to performance degradation. Quantization Preserves the Performance of Expert Trimming, showing the effectiveness of the combination of trimming and slimming. Finally, the paper further shows that post fine-tuning can recover the performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Overall, I suggest reconsidering the motivation of this paper:\n\n1. If the paper's motivation is on the MoE-specific compression techniques, methods like Expert Drop should be highlighted. Of course, Naive Expert Drop is proven bringing degradation. But experts are the main difference compared with vanilla dense models.\n\n2. If the paper's motivation can be changed to a holistic study of compression and post fine-tuning pipeline for language models, dense models' experiment should be added. Since applied methods are all available for dense models, a slight modification of the paper's direction is worth considering.\n\n3. Post fine-tuning is a valuable direction after compression. Maybe continue pre-training can bring better results rather than SFT. More detailed experiments are welcome.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a well-executed study on compressing MoE models, demonstrating significant efficiency improvements through Layer Drop and Block Drop while preserving performance. Extensive experiments validate these methods, making the findings relevant for researchers optimizing large-scale models. While some reviewers raised concerns about novelty, noting that expert slimming and trimming have been explored before, the authors argue that their systematic integration and evaluation provide new insights. The reviewers generally found the study valuable, with most leaning towards acceptance despite differing views on originality. Given the empirical support and relevance to the field, I recommend accepting the paper.

The authors are asked to revise the intro and abstract to better illustrate the contribution.
Decision: Accept (poster)