MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router

## Abstract

Abstract Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption and redundancy in experts. Pruning MoE can reduce network weights while maintaining model performance. Motivated by the recent observation of emergent large magnitude features in Large Language Models (LLM) and MoE routing policy, we propose MoE-Pruner, a method that prunes weights with the smallest magnitudes multiplied by the corresponding input activations and router weights, on each output neuron. Our pruning method is one-shot, requiring no retraining or weight updates. We evaluate our method on Mixtral-8x7B and Mixtral-8x22B across multiple language benchmarks. Experimental results show that our pruning method significantly outperforms state-of-the-art LLM pruning methods. Furthermore, our pruned MoE models can benefit from a pretrained teacher model through expert-wise knowledge distillation, improving performance post-pruning. Experimental results demonstrate that the Mixtral-8x7B model with 50% sparsity maintains 99% of the performance of the original model after the expert-wise knowledge distillation.

## 1 Introduction

Scaling neural network models is one of the main drivers of better performance in deep learning. From BERT (Devlin et al., ) to GPT-3 (Brown et al., ) to Llama 3.1 405B (Dubey et al., ) in natural language processing, or from ResNet (He et al., ) to ViT (Dosovitskiy et al., ) in computer vision, breakthroughs in performance have been obtained from larger models, datasets, and computational resources for training (Kaplan et al., ). However, the cost of training state-of-the-art models grows exponentially. For instance, BERT-Large (345M parameters, proposed in 2018) requires an estimated $5\times 10^{20}$ FLOPs (Devlin et al., ) to train, GPT-3 (175B parameters, from 2020) requires $3.14\times 10^{23}$ FLOPs (Brown et al., ), while Llama 3.1 (405B, released in 2024) requires $3.8\times 10^{25}$ FLOPs (Dubey et al., ) to train. This exponential growth motivates researchers to seek more efficient and effective training approaches.

Mixture-of-Experts (MoE) architectures (Jacobs et al., ; Shazeer et al., ) have been proposed to reduce the computing cost while enabling efficient scaling of network capacity. It has been successfully employed to scale both vision (Ruiz et al., ; Shen et al., ) and language (Lepikhin et al., ; Fedus et al., ) models. In addition, these models provide other advantages, including sparsity that can mitigate catastrophic forgetting in continual learning and an inductive bias that can enhance performance in multitask learning. Overall, MoE has proven to be a promising strategy for scaling deep learning models across various domains.

However, several crucial limitations persist in MoE for expanding its capacity. First of all, the static parameters, particularly those required for constructing the MoE architecture, introduce substantial memory overheads and constraints for deployment. For example, Mixtral-8x7B (Jiang et al., ) expert layers account for 96% of model parameters (45B out of 47B), which demands considerable memory and storage during inference. Moreover, MoE has a poor utilization of its experts. The conventional learning-based routing policy for MoE suffers from representation collapse issues since it encourages token embeddings to be clustered around expert centroids (Chi et al., ) and results in redundant experts (Mittal et al., ; Chen et al., ).

One possible solution to address those drawbacks and fully unleash the power of MoE is consolidating information from insignificant experts, aiming to establish a more compact MoE without hurting performance. Another solution is pruning experts that yield the lowest token reconstruction loss. Nevertheless, naively combining existing model merging mechanisms or expert pruning leads to performance degradation in the MoE architectures. We raise the following pivotal questions for MoE LLM pruning:
(i) How do we formulate and devise comprehensive pruning metrics that leverage existing methods?
(ii) How do we find the optimal pruning metric tailored for MoE Large Language Models?

In this paper, we systematically explore MoE LLM pruning and target a high-quality compressed MoE model in downstream fine-tuning scenarios. Specifically, we first analyze the open-source MoE model’s expert activation frequency and observe that different MoE expert initialization methods result in different expert activation frequencies and expert similarities. We leverage existing LLM pruning methods such as SparseGPT (Frantar & Alistarh, ) and Wanda (Sun et al., ), and design a novel pruning metric that incorporates MoE router weights information to identify and remove unimportant weights in expert layers. An overview of MoE-Pruner is shown in Figure 1. Since the pruning process is one-shot and only requires a small set of calibration data, the MoE model suffers from performance degradation. To recover MoE model performance, we further propose an expert-wise knowledge distillation method that utilizes the pretrained model as a teacher model, facilitating the recovery of the pruned model’s performance.

Figure: Figure 1: Overview of MoE-Pruner. For the MoE expert layer, the output is the weighted sum of the outputs from selected experts over inputs. $G_{i}$ denoted the routing logits and $\widetilde{G_{i}}$ denotes the normalized router weight of each selected expert. Our pruning metric is the multiplication of weight magnitude and the norm of input activations by the router weights.
Refer to caption: x1.png

Our main contributions can be summarized as follows:

- •
We propose a novel framework, MoE-Pruner, that is efficient and effective for pruning MoE models with minimal performance degradation.
- •
We design an innovative expert-wise knowledge distillation method that leverages the pretrained MoE model as a teacher model to recover pruned MoE student model performance.
- •
Experimental results on Mixtral MoE models across nine zero-shot evaluation benchmarks demonstrate the effectiveness of our MoE-Pruner algorithm. MoE-Pruner achieves minimal performance drop even at 50% sparsity with only a small set of calibration data compared with existing pruning methods. The pruned model maintains 99% of the performance of the original model after the expert-wise knowledge distillation.

## 2 Preliminaries

Mixture-of-Experts (MoE). Scaling model size increases learning capacity and enhances generalization (Kaplan et al., ; Brown et al., ; Hoffmann et al., ). MoE (Jacobs et al., ; Shazeer et al., ; Lepikhin et al., ; Fedus et al., ) is an efficient approach that enables significantly more compute-efficient pretraining and inference. It replaces the feed-forward network (FFN) layers in Transformers (Vaswani et al., ) with expert layers, where different experts are activated for different input tokens instead of utilizing the full network parameters. Sparse MoE architecture can dramatically scale the model with the same compute budget as a dense model.

Large Language Model Pruning. Magnitude pruning (Han et al., ) is a standard approach to induce sparsity in neural networks. It removes individual weights with magnitudes below a certain threshold. However, magnitude pruning fails dramatically on LLMs even with relatively low levels of sparsity (Frantar & Alistarh, ). SparseGPT (Frantar & Alistarh, ) proposes a one-shot, post-training pruning method that prunes LLM weights and uses Hessian matrix and calibration data to update the remaining weights without any retraining. Wanda (Sun et al., ) is a simple method that prunes LLM weights with the smallest magnitudes multiplied by the corresponding input activations without any additional weight update.

## 3 Methodology

### 3.1 The Mixture-of-Experts Architecture

Mixture-of-Experts (MoE) architecture. MoE architecture replaces the feed-forward networks (FFN) in Transformers with mixture-of-expert layers. A router or a gating network is trained to select a subset of experts for each input token based on its routing policy. Given $n$ experts in a layer, the output of the expert layer is given by:

$$ $y=\sum_{i=0}^{n-1}Gate(x)_{i}\cdot E_{i}(x),$ (1) $$

where the $Gate(x)_{i}$ is the router weights from the gating network assigned to the $i$-th expert, and $E_{i}(x)$ is the output of $i$-th expert. The router weights can be formulated as softmax over the Top-K logits:

$$ $Gate(x)=\mathrm{Softmax}(\mathrm{TopK}(x\cdot W_{g})),$ (2) $$

where $W_{g}$ is the weight of the router or gating network, and $\mathrm{TopK}(X)_{i}=l_{i}$ if $i$ is in the top-K coordinates of logits $l$ and $\mathrm{TopK}(X)_{i}=-\infty$ otherwise.

Since current LLMs mostly adopt SwiGLU (Shazeer, ) architecture for the FFN, and MoE LLM such as Mixtral-8x7B (Jiang et al., ) uses a top-2 to select experts, we can derive the output of an expert layer as:

$$ $y=\sum_{i=0}^{n-1}\mathrm{Softmax}(\mathrm{Top2}(x\cdot W_{g}))_{i}\cdot \mathrm{SwiGLU}_{i}(x).$ (3) $$

Some recent MoE LLMs, such as DeepSeekMoE (Dai et al., ), adopt shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.

MoE Expert Initialization. MoE expert initialization uses different strategies, which can be classified into two categories: sparse upcycling (Komatsuzaki et al., ) and training from scratch. Some open-source MoE models such as Mixtral (Jiang et al., ), Qwen1.5-MoE-A2.7B (Team, ), and MiniCPM-MoE (Hu et al., ) all employ the upcycling approach to reduce the total training costs. While some MoE models like DeepSeek-V2 (Liu et al., ), OLMoE (Muennighoff et al., ), and Yuan2.0-M32 (Wu et al., ) use the training from scratch approach to help expert diversification. We find that different MoE expert initialization methods result in different expert activation frequencies and expert similarities, which will impact the MoE pruning strategies. For instance, the MoE model initialized with upcycling can take advantage of the dense model and reduce training costs. The final MoE model exhibits higher expert similarity and more balanced expert activation frequency, which indicates that expert pruning will result in a performance drop, and weight pruning will be a better choice. MoE model trained from scratch might yield better performance as it avoids the limitations of starting with a group of identical experts, which can hinder diversification (Wei et al., ). It also shows imbalanced expert activation frequency, indicating that least-used expert pruning could help compress model size and not bring performance degradation.

MoE Expert Activation Frequency. We use a subset of the C4 (Raffel et al., ) dataset and collect the activation frequency of MoE experts. The expert activation frequency is task-agnostic since C4 pretraining datasets are comprehensive and not dominated by knowledge specific to any particular domain. Motivated by the load balancing loss (Shazeer et al., ; Lepikhin et al., ; Fedus et al., ), we propose to use the coefficient of variation of expert activation frequency in each layer to represent the load balancing score, where a lower score represents more balanced loads. Given $n$ experts and $l$ layers and a batch $\mathcal{B}$ with $T$ tokens, the load balancing score for one layer is:

$$ $\begin{split}&s=\frac{\sigma}{\mu}=\frac{\sqrt{\frac{1}{n}\sum_{i=0}^{n-1}(f_{ i}-\mu)^{2}}}{\mu},\\ &\mu=\frac{1}{n}\sum_{i=0}^{n-1}f_{i},\end{split}$ (4) $$

where $f_{i}$ is the number of tokens dispatched to expert $i$,

$$ $f_{i}=\sum_{x\in\mathcal{B}}\mathbbm{1}\{\text{argmax}\>p(x)=i\}.$ (5) $$

We can derive the load balancing score by calculating the mean of scores across all $l$ MoE layers, such that we can use this score to compare with various MoE models with different numbers of experts. Figure 2 shows the load balancing scores of Mixtral-8x7B (Jiang et al., ), Qwen-1.5-A2.7B (Team, ), DeepSeek-V2 and DeepSeeek-V2-Lite (Liu et al., ), MiniCPM-MoE-8x2B (Hu et al., ), and OLMoE (Muennighoff et al., ).

Figure: Figure 2: Load balancing score of MoE models. We collect the expert activation frequency of MoE models and calculate the load balancing score (lower is better). The circle area represents the model size. MoE model trained from scratch are marked with red, while MoE models that use upcycling are marked with blue. MoE models trained from scratch usually have more experts and imbalanced loads. MoE models initialized with upcycling tend to have more balanced loads and less number of experts. The only exception is Qwen-1.5-A2.7B, which is initialized with upcycling. But according to the report (Yang et al., ), its expert parameters are shuffled along the intermediate dimension to guarantee that each fine-grained expert exhibits unique characteristics and therefore exhibits more like trained from scratch MoE models.
Refer to caption: x2.png

### 3.2 Pruning Metric

Problem Formulation. Post-training pruning for LLMs can be decomposed into layer-wise subproblems (Lu et al., ; Frantar & Alistarh, ; Sun et al., ; Dong et al., ).
Given a sparsity ratio and a linear layer with weight $\mathbf{W}$, the pruning algorithm tries to find a sparsity mask $\mathbf{M}$ that minimizes reconstruction loss:

$$ $\underset{\mathbf{M}}{\text{argmin}}\lVert\mathbf{WX}-(\mathbf{M}\odot\mathbf{ W})\mathbf{X}\rVert.$ (6) $$

Optimal Brain Damage (OBD) (LeCun et al., ) first sets up a pioneering framework for neural network pruning. It uses second-order information without off-diagonal elements in the Hessian matrix for faster approximation. Optimal Brain Surgeon (OBS) (Hassibi et al., ) develops upon OBD partly by taking into account the off-diagonal elements. SparseGPT (Frantar & Alistarh, ) revisits the OBS, computes the inverse Hessian only once, and reuses to update weight in the remaining rows that are also in the mask to mitigate reconstruction loss. The pruning metric in SparseGPT is:

$$ $\mathcal{S}_{ij}=[|\mathbf{W}|^{2}/\mathrm{diag}(\mathbf{H}^{-1})]_{ij}.$ (7) $$

Wanda (Sun et al., ) further simplifies the pruning metric to the following form without the need to compute the inverse of the Hessian matrix $\mathbf{H}$:

$$ $\mathcal{S}_{ij}=[|\mathbf{W}|^{2}/\mathrm{diag}((\mathbf{X}^{T}\mathbf{X})^{- 1})]_{ij}\approx[|\mathbf{W}|^{2}/(\mathrm{diag}(\mathbf{X}^{T}\mathbf{X})^{-1 })]_{ij}=(|\mathbf{W}_{ij}|\cdot\|\mathbf{X}_{j}\|)^{2}.$ (8) $$

When it comes to pruning MoE, the expert layers constitute the majority of model parameters. For example, the Mixtral-8x7B (Jiang et al., ) has a total of 47B parameters where 1.3B belongs to attention modules and 45B is used for expert layers (2 out of 8 experts are activated, 12.5B active parameters during inference). Only a subset of experts are activated for different input tokens, so there is a large space of expert redundancy.

Router Tells It All. As shown in Equation 1, the router weights are assigned to each expert output. Motivated by the pruning metric in Wanda and the MoE routing policy, our approach, MoE-Pruner, prunes weights with the smallest magnitudes multiplied by the corresponding input activations and router weights, on each output neuron:

$$ $\mathcal{S}=|\mathbf{W}_{ij}|\cdot\lVert\mathbf{X}_{j}\cdot\mathbf{Gate}_{j}\rVert.$ (9) $$

**Table 1: Comparison of different pruning methods including magnitude pruning, SparseGPT, Wanda, and MoE-Pruner.**
| Method | Weight update | Calibration data | Pruning metric $\mathcal{S}$ | Complexity |
| --- | --- | --- | --- | --- |
| Magnitude | ✗ | ✗ | $|\mathbf{W}|$ | $\mathcal{O}(1)$ |
| SparseGPT | ✔ | ✔ | $[|\mathbf{W}|^{2}/\mathrm{diag}(\mathbf{H}^{-1})]_{ij}$ | $\mathcal{O}(d_{hidden}^{3})$ |
| Wanda | ✗ | ✔ | $|\mathbf{W}_{ij}|\cdot\|\mathbf{X}_{j}\|$ | $\mathcal{O}(d_{hidden}^{2})$ |
| MoE-Pruner | ✗ | ✔ | $|\mathbf{W}_{ij}|\cdot\lVert\mathbf{X}_{j}\cdot\mathbf{Gate}_{j}\rVert$ | $\mathcal{O}(d_{hidden}^{2})$ |

Table 1 summarizes pruning methods, including magnitude pruning, SparseGPT, Wanda, and MoE-Pruner and their corresponding pruning metric and complexity. Algorithm 1 presents the unstructured sparsity version of our MoE-Pruner algorithm. Our method is simple and efficient for MoE models and does not require a sophisticated weight update procedure.

Figure: Algorithm 1 The MoE-Pruner algorithm. We prune each expert layer weight matrix $\mathbf{W}$ to $p\%$ sparsity.

Structured N:M Sparsity. Structured N:M sparsity can leverage NVIDIA’s sparse tensor cores to accelerate matrix multiplication. While MoE-Pruner so far has been developed for unstructured sparsity, it can be easily extended to structured N:M sparsity (Mishra et al., ), where we compare weights using the same metric among every M consecutive weights, for all weights connected to an output.

Comparison Group. Generally, for a pruning method, each weight is first assigned an importance score, calculated by the pruning metric. These weights are then grouped into comparison groups where weights within each group are compared against one another, and weights with lower importance scores are pruned. Most previous pruning methods default to comparing weights locally within each layer or globally across the whole network. Our method compares weights using the comparison groups on each output neuron, which aligns with Wanda and can be easily extended to N:M semi-structured sparsity. Moreover, our method is not limited to pruning individual weights but can also group structured weights into a unit and compare those units among a larger comparison group, such that we can extend our method to structured sparsity.

### 3.3 Expert-Wise Knowledge Distillation

Expert-Wise Knowledge Distillation. MoE models can preserve most of their capacity after pruning but still suffer from performance degradation. To recover MoE LLM performance, we fine-tune the model by leveraging the unpruned pretrained model as a teacher model in an expert-wise knowledge distillation (KD) manner. The pretrained model is a natural teacher model for the pruned model since they share exactly the same number of layers, experts, and dimensions (Kurtic et al., ). The loss function for expert-wise knowledge distillation is formulated as follows:

$$ $\mathcal{L}_{KD}=\mathcal{L}_{CE}+\lambda\times\mathcal{L}_{expert}=\mathcal{L }_{CE}+\lambda\times\sum_{j=0}^{l-1}\sum_{i=0}^{n-1}\mathrm{MSE}(E_{it}^{j},E_ {is}^{j}),$ (10) $$

where $\mathcal{L}_{CE}$ is the cross entropy loss, $\mathrm{MSE}$ is the mean squared error calculated as $\mathrm{MSE}(X,Y)=\frac{1}{N}\sum_{i=0}^{N-1}(x_{i}-y_{i})^{2}$ for $N$-dimensional vectors $X$ and $Y$. $\lambda$ is a weighting coefficient and initialized based on the
strength of cross entropy loss and expert-wise knowledge distillation loss: $\frac{\mathcal{L}_{CE}}{\mathcal{L}_{expert}}$. We sum up all the differences between teacher experts and student experts. Figure 3 illustrates the expert-wise knowledge distillation for pruned models. The corresponding expert in the pretrained teacher model will be used to distill the expert in the pruned student model.

Figure: Figure 3: Expert-wise knowledge distillation for the pruned MoE model using the pretrained MoE model as the teacher to recover the performance of the pruned model.
Refer to caption: x3.png

## 4 Experiments

Models, Datasets, and Evaluation. We conduct pruning experiments on widely adopted open-source MoE models: the base and instruct version of Mixtral-8x7B and Mixtral-8x22B (Jiang et al., ). We use samples from the pretraining dataset C4 (Raffel et al., ) as calibration data for one-shot pruning since pretraining datasets are often more comprehensive and not dominated by knowledge specific to any particular domain. We use the exact same 128 sequences of calibration data for all one-shot pruning experiments to control this variable factor. We evaluate the perplexity on the WikiText (Merity et al., ) validation set. Our expert-wise knowledge distillation method uses a subset of the C4 (Raffel et al., ) as the training set. We measure the performance of pruned models on zero-shot tasks and language modeling. For zero-shot evaluation, we use nine popular tasks from EleutherAI LM Harness (Gao et al., ). The nine evaluated zero-shot tasks are: ARC-easy, ARC-challenge (Clark et al., ), Boolq (Clark et al., ), HellaSwag (Zellers et al., ), MMLU (Hendrycks et al., ), OpenBookQA (OBQA) (Mihaylov et al., ), PIQA (Bisk et al., ), RTE (Wang et al., ), and WinoGrande (Sakaguchi et al., ).

Baselines and Experiments Setup. We compare MoE-Pruner with prior pruning approaches, including SparseGPT (Frantar & Alistarh, ) and Wanda (Sun et al., ). Similarly, our pruning algorithm is implemented in a layer-wise reconstruction manner. All pruning experiments are conducted on a single NVIDIA H100-80GB GPU. The fine-tuning experiments use the pruned model as a starting point and perform full-parameter fine-tuning to preserve the sparsity mask. We implement the expert-wise knowledge distillation method in Llama-Factory (Zheng et al., ) and conduct experiments on 2 servers, each with 8 NVIDIA H100-80GB GPUs. We fine-tune the pruned student model for three epochs, using a learning rate of 2e-5 with the cosine learning rate scheduler.

### 4.1 One-Shot Pruning

**Table 2: Perplexity against other one-shot pruning methods, including SparseGPT and Wanda with 50% sparsity.**
| Model | Method | Sparsity | WikiText Perplexity $\downarrow$ |
| --- | --- | --- | --- |
| Mixtral-8x7B | Pretrained | 0% | 3.84 |
| SparseGPT | 50% | 4.99 |  |
| Wanda | 50% | 4.97 |  |
| MoE-Pruner (Ours) | 50% | 4.68 |  |
| Mixtral-8x7B | Pretrained | 0% | 3.84 |
| SparseGPT | 2:4 | 7.09 |  |
| Wanda | 2:4 | 6.98 |  |
| MoE-Pruner (Ours) | 2:4 | 5.88 |  |
| Mixtral-8x7B-Instruct | Pretrained | 0% | 4.14 |
| SparseGPT | 50% | 5.20 |  |
| Wanda | 50% | 5.16 |  |
| MoE-Pruner (Ours) | 50% | 4.94 |  |
| Mixtral-8x7B-Instruct | Pretrained | 0% | 4.14 |
| SparseGPT | 2:4 | 7.19 |  |
| Wanda | 2:4 | 6.92 |  |
| MoE-Pruner (Ours) | 2:4 | 6.11 |  |
| Mixtral-8x22B | Pretrained | 0% | 2.83 |
| SparseGPT | 50% | 4.19 |  |
| Wanda | 50% | 3.97 |  |
| MoE-Pruner (Ours) | 50% | 3.64 |  |
| Mixtral-8x22B-Instruct | Pretrained | 0% | 2.89 |
| SparseGPT | 50% | 4.27 |  |
| Wanda | 50% | 4.06 |  |
| MoE-Pruner (Ours) | 50% | 3.72 |  |

Table 2 shows the one-shot pruning model perplexity on WikiText with 50% sparsity. There is a clear difference between MoE-Pruner and other pruning methods, including SparseGPT (Frantar & Alistarh, ) and Wanda (Sun et al., ). For Mixtral-8x7B (Jiang et al., ) models, MoE-Pruner achieves 0.22-0.31 better perplexity over SparseGPT and Wanda. This improvement expands when the MoE model scales to the Mixtral-8x22B model. For the larger Mixtral-8x22B model, MoE-Pruner achieves 0.55 better perplexity over SparseGPT and 0.31-0.34 better perplexity over Wanda. MoE-Pruner further expands the improvement to 1.21 better perplexity over SparseGPT and 1.10 better perplexity over Wanda when we prune the MoE models with the 2:4 semi-structured sparsity.

Table 3 shows the average zero-shot accuracies on nine zero-shot tasks of the pruned Mixtral-8x7B MoE models with 50% unstructured sparsity. The average performance of pretrained models, SparseGPT, Wanda, and our pruned models are 69.16, 66.27, 65.90, and 67.23, respectively. MoE-Pruner outperforms the state-of-the-art pruning approaches, SparseGPT and Wanda, by a large margin. Given that no fine-tuning takes place at this time, there is a noticeable gap between the sparse pruned MoE model and the original pretrained MoE model.

**Table 3: Average zero-shot performance on 9 evaluation tasks of pruned models using SparseGPT, Wanda, and MoE-Pruner.**
| Model | Method | ARC-c | ARC-e | Boolq | HellaSwag | MMLU | OBQA | PIQA | RTE | WinoGrande | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mixtral<br>-8x7B | Pretrained | 56.91 | 84.47 | 85.29 | 64.78 | 67.03 | 35.0 | 82.43 | 70.4 | 76.16 | 69.16 |
| SparseGPT | 50.43 | 80.68 | 84.62 | 60.20 | 61.79 | 32.8 | 81.12 | 68.59 | 76.16 | 66.27 |  |
| Wanda | 51.02 | 80.89 | 85.08 | 60.45 | 62.73 | 32.6 | 80.90 | 64.64 | 74.82 | 65.90 |  |
| MoE-Pruner | 53.33 | 81.86 | 86.02 | 62.29 | 64.76 | 33.6 | 81.61 | 66.06 | 75.53 | 67.23 |  |

### 4.2 Expert-Wise Knowledge Distillation Performance

The gap between the pruned MoE model and the pretrained MoE model can be largely mitigated via expert-wise knowledge distillation. We only need 1000 training samples from C4 (Raffel et al., ), and training can be done in 1 hour. Table 4 shows the average zero-shot accuracy of the pruned and fine-tuned Mixtral-8x7B MoE models with 50% unstructured sparsity. The fine-tuned model could achieve a 68.40 average performance on nine zero-shot tasks. The performance is very close to the pretrained Mixtral-8x7B MoE model, which demonstrates a 69.16 average performance.

**Table 4: Average zero-shot performance after pruning and expert-wise knowledge distillation.**
| Model | Method | ARC-c | ARC-e | Boolq | HellaSwag | MMLU | OBQA | PIQA | RTE | WinoGrande | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mixtral<br>-8x7B | Pretrained | 56.91 | 84.47 | 85.29 | 64.78 | 67.03 | 35.0 | 82.43 | 70.4 | 76.16 | 69.16 |
| MoE-Pruned | 53.33 | 81.86 | 86.02 | 62.29 | 64.76 | 33.6 | 81.61 | 66.06 | 75.53 | 67.23 |  |
| MoE-Distilled | 54.35 | 81.19 | 85.26 | 68.77 | 65.59 | 36.0 | 82.48 | 68.23 | 75.72 | 68.40 |  |

### 4.3 Ablation Studies

Figure: Figure 4: Perplexity with different number of calibration samples at 50% sparsity.
Refer to caption: x4.png

Ablation on Different Number of Calibration Samples. We change the number of calibration samples by selecting different sample sizes ranging from 2 to 256. Results are summarized in Figure 4. We see a clear difference in trend as the number of calibration samples changes. MoE-Pruner is much more robust than SparseGPT when there are few calibration samples and performs the same trend but better perplexity over Wanda. Notably, even with just two calibration samples, pruned networks obtained by MoE-Pruner have a perplexity of just 4.95. This may be because input norm statistics could be much easier to estimate than the full inverse Hessian of the local layer-wise reconstruction problem.

Ablation on Different Sparsity Ratio. We also change the pruning ratio using the same 128 calibration samples. Figure 5 shows that at lower pruning ratios, such as 10% to 40%, all pruning methods result in almost the same perplexity. When the pruning ratio increases, the Wanda pruned model perplexity changes dramatically and fails at 70%. MoE-Pruner shows better and more stable pruning results than SparseGPT and Wanda, especially at higher pruning ratios. This demonstrates that router weights preserve important information when selecting experts and provide a clear hint for pruning unimportant weights.

## 5 Related Works

Pruning and Sparsity. Pruning (LeCun et al., ; Hassibi et al., ; Han et al., ) is an important approach for compressing neural networks through eliminating weights (Han et al., ) or activations (Rao et al., ), yielding sparse networks. It can be mainly classified into two categories based on the granularity: unstructured and structured pruning.

Unstructured pruning such as magnitude pruning (Han et al., ; ) removes individual weights to introduce sparsity while preserving accuracy even at high sparsity. Existing methods either require retraining or fine-tuning the pruned models (Liu et al., ) or the whole iterative retraining process (Frankle & Carbin, ). However, in the era of LLMs, these methods fail as retraining LLMs demands substantial computational resources. SparseGPT (Frantar & Alistarh, ) and Wanda (Sun et al., ) propose efficient post-training pruning method that prunes LLM weights in a layer-wise manner without retraining the model.

Structured pruning eliminates weights as a group, such as channel pruning (He et al., ), kernel pruning (Zhong et al., ), attention head pruning (Wang et al., ), token pruning (Rao et al., ), and layer pruning (Elhoushi et al., ). Unlike unstructured pruning, it leads to more hardware-friendly, dense blocks of computation, which facilitates acceleration on modern hardware platforms. Some methods explore structured pruning based on sparsity on the structural components of LLMs, such as attention heads (Wang et al., ) and FFN channels (Ma et al., ).  Muralidharan et al. () uses both structured pruning and knowledge distillation to compress LLM models and shows improvement over models trained from scratch. Due to the constraint of removing regular components, structured pruning usually has low sparsity ratios and high accuracy loss. NVIDIA proposes N:M semi-structured sparsity (Mishra et al., ), which can preserve model performance by retraining and leverage GPU tensor core acceleration.

Pruning for MoE Models.
Most of the works for MoE pruning focus on structured expert pruning.  Chen et al. () and  Koishekenov et al. () prune experts based on their utilization to save memory. However, this usually leads to degraded performance.  Lu et al. () enumerates expert combinations based on the required expert number and uses calibration data to find a set of remaining experts that has the minimum reconstruction loss.  Chowdhury et al. () prunes experts based on the change in the router’s norm and proves that the generalization accuracy can be preserved. However, expert pruning sometimes removes experts with certain knowledge and results in the loss of model performance. Therefore,  Li et al. () and Zhang et al. () both leverage expert merging techniques to compress the expert layer while also preserving expert knowledge.  He et al. () proposes a unified framework to compress MoE models. The framework consists of two perspectives: (i) expert slimming that compresses individual experts by weight pruning and quantization, and (ii) expert trimming that removes whole structured modules by layer drop and block drop.

Efficiency for MoE and Existing Solutions. MoE models require huge memory to host expert layers, while many experts have low utilization during inference. To address this,  Gao et al. () uses a tensor decomposition method to share the central tensor’s parameters across experts and keep different auxiliary tensors for each expert. MoQE (Kim et al., ) and QMoE (Frantar & Alistarh, ) both study extreme low-bit quantization for compressing MoE model size. Moreover, some works employ knowledge distillation (Fedus et al., ; Artetxe et al., ) to create either a smaller dense model or a MoE model with fewer layers. However, they also overlook the existing redundancy within MoE expert layers.  Yadav et al. () shows that experts can be compressed to a huge degree without any performance loss.

## 6 Conclusion

We propose a simple and effective pruning method for MoE models, MoE-Pruner. We prune weights with the smallest magnitudes multiplied by the corresponding input activations and router weights, on each output neuron. Our pruning method is one-shot and fast, without the need for any retraining or weight update procedures. Pruning MoE LLM with high sparsity will incur performance degradation, so we also propose a fine-tuning method that leverages the unpruned pretrained MoE model as a teacher to guide the pruned student model through expert-wise knowledge distillation. The fine-tuned MoE models could maintain 99% of the performance of the original model after the expert-wise
knowledge distillation, using only a small set of training data and low GPU hours.
In the future, MoE-Pruner could also be extended to structured pruning of MoE LLMs, such as channel pruning and expert pruning, for better hardware acceleration.

## Appendix

## Appendix A MoE Expert Activation Frequency Results

### A.1 Mixtral-8x7B

Figure: Figure 6: Mixtral-8x7B expert activation frequency on C4 datasets.
Refer to caption: x6.png

### A.2 Qwen-1.5-A2.7B

Figure: Figure 7: Qwen-1.5-A2.7B expert activation frequency on C4 datasets.
Refer to caption: x7.png

## Appendix B Open-Source MoE Models

**Table 5: Open-Source MoE Models List (Released after Jan. 2024).**
| Name | Active<br>Parameters | Total<br>Parameters | # Experts | Routing<br>Policy | Initialized<br>Method | MMLU^∗ |
| --- | --- | --- | --- | --- | --- | --- |
| OLMoE | 1B | 7B | 64 | top-8 | train from scratch | 54.1 |
| MiniCPM-MoE-8x2B | 4B | 13.6B | 8 | top-2 | upcycling | 58.9 |
| Qwen1.5-MoE-A2.7B | 2.7B | 14.3B | 4(shared)+60 | 4+top-4 | upcycling | 62.5 |
| Deepseek-V2-Lite | 2.4B | 16B | 2(shared)+64 | 2+top-6 | train from scratch | 58.3 |
| Yuan2.0-M32 | 3.7B | 40B | 32 | top-2 | train from scratch | 72.2 |
| GRIN-MoE | 6.6B | 41.9B | 16 | top-2 | upcycling | 79.4 |
| Mixtral-8x7B | 12.5B | 47B | 8 | top-2 | upcycling | 70.4 |
| Jamba | 12B | 52B | 16 | top-2 | unknown | 67.4 |
| Qwen2-57B-A14B | 14B | 57.4B | 8(shared)+64 | 8+top-8 | upcycling | 76.5 |
| DBRX | 36B | 132B | 16 | top-4 | unknown | 73.7 |
| Mixtral-8x22B | 39B | 141B | 8 | top-2 | upcycling | 77.8 |
| Skywork-MoE | 22B | 146B | 16 | top-2 | upcycling | 77.4 |
| Deepseek-V2 | 21B | 236B | 2(shared)+160 | 2+top-6 | train from scratch | 78.5 |
| grok-1 | 80B | 314B | 8 | top-2 | unknown | 73.0 |
| Snowflake Arctic | 17B | 480B | 128 | top-2 | unknown | 67.3 |

^∗Note: This table presents a subset of open-source MoE models and is not exhaustive. The list is sorted by total parameters. MMLU scores are extracted from original papers or reports and may not reflect model real performance.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. What is the runtime efficiency of the proposed method?

  2. What is the maximum sparsity that the proposed method can achieve without incurring significant perplexity loss? Could you provide performance comparisons under moderate losses, such as a 10% perplexity increase compared to the dense model? The perplexity loss at 50% density appears a bit high for both the proposed method and baselines.
soundness: 2
strengths: 1. New Insights: The paper provides valuable insights into how expert initialization methods influence final load balance and similarities between experts.

  2. Perplexity Performance Improvements: The proposed method leverages gate values in MoE to enhance the performance of pruned models, achieving perplexity improvements over previous methods designed for general transformer-based LLMs.
summary: The paper proposes a pruning method specifically designed for Mixture-of-Expert (MoE) models. It uses the Gate values from the MoE router to enhance weight importance evaluation during pruning. Additionally, it introduces an expert-wise approach for knowledge distillation to improve the performance of the pruned model.
weaknesses: 1. Efficiency Evaluation: The paper lacks an evaluation of the proposed method’s efficiency improvements, such as wall-clock speedup or memory reduction.

  2. Limited Technical Contribution: Pruning weights in the least-activated expert appears intuitive, yet it’s unclear how the method addresses potential drawbacks. For instance, which specific capabilities are impacted by this pruning? Does the improved performance on tested tasks come at the cost of decreased performance on other rare but crucial tasks?

  3. Unclear Contribution of Gate Value Insights: While the paper discusses gate value differences between two MoE initialization methods, it doesn’t clarify how these insights inform the pruning method’s design. Additionally, a brief definition of sparse upcycling initialization in related work would be helpful, as it is repeatedly referenced in later discussions.

  4. Writing and Support for Claims: Some technical claims lack proper references or justification. For example, the statement “MoE mitigates catastrophic forgetting in continual learning” (Line 44, Introduction) is not supported by references. Similarly, the claim that “expert activation frequency is task-agnostic” (Line 161, Methodology) seems conclusive, yet the paper does not provide corresponding experimental or logical evidence.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In line 154, the authors mentioned that upcycling initialization will lead to higher expert similarity in MoE models. Since the similarity between experts is relatively high, pruning experts should theoretically not result in a significant performance drop. Why do the authors reach the opposite conclusion?
  2. In line 158, the authors mentioned that train from stratch will show imbalanced expert activation frequency, indicating that least-used expert pruning could help compress model size and not bring performance degradation. However, I believe this may be task-dependent; these seemingly least-used experts could be very useful for specific tasks such as math or coding. Did the authors conduct any related validation?
soundness: 2
strengths: 1. The authors propose a simple yet effective MoE pruning method that can be easily applied to various existing MoE models, achieving notable performance improvements.
  2. The authors provide a detailed ablation study of each component and hyperparameter, clearly demonstrating the role of each component and the sensitivity to hyperparameters.
summary: In this paper, the authors propose a simple and effective pruning method for MoE models. Specifically, (1) the authors prune model weights based on the sensitivity criterion derived from the MoE gate outputs in each transformer block; (2) they further use a distillation method to recover task performance of the pruned model. Experimental results show that on zero-shot and language modeling datasets, the proposed method outperforms existing weight pruning methods in terms of algorithm performance.
weaknesses: 1. Equations lack detailed explanation. 

    - For equations 7, 8, and 9, what do i and j stand for? Please describe in detail which dimension in each tensor corresponds to i and j. By the way, I also notice that in equation 9, the authors use S instead of S_{i,j} to represent the sensitivity metric. Is this a typo?

    - For equation 10, is cross entropy the global next token prediction loss or the local cross entropy loss for the MoE gate?

2. Algorithm 1 needs detailed explanation. In line 251, the authors mentioned that "Algorithm 1 presents the unstructured sparsity version of our MoE-Pruner algorithm". I don’t understand where "unstructured" is reflected. Is it that the weights of the experts are unstructured? Does the weight WWW represent all experts or just one expert? Also, which linear layer in SwiGLU does the weight WWW correspond to?

3. The experimental comparisons are not comprehensive. In the Introduction, the authors mention various issues with expert merging and expert pruning. However, in the experimental section, they do not compare their method with any expert merging or pruning methods [1,2,3]. The authors need to provide some comparative data and analysis to demonstrate whether their proposed method is truly SOTA. 

[1] Liu et al., Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs

[2] Lu et al., Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

[3] Muzio et al., SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: - What is pruning and inference speedup of MoE-Pruner compared to other baselines?
- How did the authors extend Wanda to MoE models, is it equivalent to setting $Gate_j = 1$ in MoE-Pruner?
- The method is poorly motivated (only from L232-237). A deeper analysis (for example, Section 3, Wanda) is preferred.
soundness: 2
strengths: - Pruning MoE is an important and impactful research problem
- The proposed method is conceptually simple yet can achieve encouraging results
summary: This work proposes MoE-Pruner, a pruning strategy for MoE based LLMs. MoE-Pruner uses the scores from the routers as signals to prune the experts and then performs an expert-wise distillation training to recover the performance. Experiments on Mixtral-8x7B and Mixtral-8x22B show that MoE-Pruner can outperform other pruning strategies for dense LLMs.
weaknesses: - **Limited technical novelty and justification** - MoE-Pruner is to be a simple extension of Wanda's formulation without extending any of its analyses to MoE. Similarly, the expert-wise knowledge distillation seems to be quite straightforward to extend from the standard knowledge distillation. Therefore, I consider the technical contribution of this work to be limited.
- **Limited evaluation** - MoE-Pruner is only evaluated on Mixtral models, it would be helpful to test its robustness by considering other models such as DeepSeek-MoE or MiniCPM-MoE.
- **Complexity analysis** - The paper does not provide a complexity analysis showing the pruning and inference speedup.
- **Poor presentation** - The paper is poorly organized. Section 2 (Preliminaries) is too short while Sections 3.1 and 3.2 are essentially a literature review, serving the same purpose. Figure 1 does not clearly explain the method as the score S does not interact with other components in the figure.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Some previous studies have focused on pruning entire experts, as seen in [1, 2, 3]. Could this method potentially be integrated with these approaches to enhance overall pruning effectiveness?    
[1] Lu, et al. "Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models." ACL 2024.   
[2] Zhang, et al. "Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts." arXiv preprint arXiv:2407.09590 (2024).   
[3] Lee, Jaeseong, et al. "STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning." arXiv preprint arXiv:2409.06211 (2024).
soundness: 3
strengths: (1) The method is clearly explained.

(2) The proposed method is efficient and practical for real-world deployment.
summary: This paper focuses on the pruning of sparse mixture-of-experts (MoE) models. It introduces a method called MoE-Pruner, which selectively prunes weights based on the smallest magnitudes, factoring in both the corresponding input activations and router weights for each output neuron. Experiments are conducted on the Mixtral family models to evaluate the approach's effectiveness.
weaknesses: (1) The evaluation is restricted to the Mixtral models, raising questions about the method's scalability to other architectures like Qwen and DeepSeek.  

(2) The distilled step shows only minor improvements, as seen in Table 4. What could be the underlying reason for this limited gain?

(3) This method builds incrementally on the baseline, Wanda, by incorporating gate weights within the MoE pruning framework.