Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs

## Abstract

Abstract The rapid advancement of large language models (LLMs) has led to architectures with billions to trillions of parameters, posing significant deployment challenges due to their substantial demands on memory, processing power, and energy consumption. Sparse Mixture-of-Experts (SMoE) architectures have emerged as a solution, activating only a subset of parameters per token, thereby achieving faster inference while maintaining performance. However, SMoE models still face limitations in broader deployment due to their large parameter counts and significant GPU memory requirements.
In this work, we introduce a gradient-free evolutionary strategy named E fficient E xpert P runing (EEP) to enhance the pruning of experts in SMoE models. Specifically, EEP searches the pruning pattern and use expert merging as an memory-efficient way of fine-tuning the pruned model. EEP relies solely on model inference (i.e., no gradient computation) and achieves greater sparsity while maintaining or even improving performance on downstream tasks. EEP can be used to reduce both the total number of experts (thus saving GPU memory) and the number of active experts (thus accelerating inference).
For example, we demonstrate that pruning up to 75% of experts in Mixtral 8 × 7 8 7 8\times 7 8 × 7 B-Instruct results in a substantial reduction in parameters with minimal performance loss.
Remarkably, we observe improved performance on certain tasks, such as a significant increase in accuracy on the SQuAD dataset (from 53.4% to 75.4%), when pruning half of the experts. With these results, EEP not only
lowers the barrier to deploying SMoE models,
but also challenges the conventional understanding of model pruning by showing that fewer experts can lead to better task-specific performance without any fine-tuning. Code is available at https://github.com/imagination-research/EEP . † † footnotetext: Correspondence to Yu Wang <yu-wang@mail.tsinghua.edu.cn>, Zinan Lin <zinanlin@microsoft.com>, Xuefei Ning <foxdoraame@gmail.com>.

## 1 Introduction

Large language models have significantly advanced, evolving into highly versatile tools [, , , , , ]. As these models grow in accordance with scaling laws [], the norm has shifted towards architectures with billions to trillions of parameters. However, the larger scale brings considerable deployment challenges due to increased demands on memory, processing power, and energy consumption [, ]. In response to these challenges, there is a notable trend towards adopting sparse Mixture-of-Experts (SMoE) architectures [, , , ], as seen in models such as Mixtral $8\times 7$B and $8\times 22$B [], Qwen1.5-MoE-A2.7B [], Qwen 2-57B-A14B [], DBRX [], and Grok-1 []. SMoE models activate only a subset of parameters for each token, resulting in faster inference while maintaining competitive performance compared to dense models of the same scale. For example, Mixtral $8\times 7$B outperforms or matches Llama-2 70B [] and GPT-3.5 on many benchmarks, while it only activates 13B parameters to process each token. Although SMoE models have less computation per token, they remain parameter-heavy, e.g. Mixtral $8\times 7$B has 47B parameters in total while Grok-1 reaches 314B (see Tab. 6 for other models). This limits their broader deployment due to the substantial GPU memory requirements. Additionally, their throughput may not be ideal as the batch size needs to be restricted to fit the model within the available device memory. Therefore, it is vital to innovate methods that can reduce the size of SMoE models without compromising their performance.

Many studies have shown that only a subset of parameters significantly contributes to performance when applying LLMs to downstream tasks [, , , ]. Pruning is a crucial technique for eliminating redundancy in neural networks. It can be unstructured, achieving high sparsity while maintaining performance [, , ], or structured, removing entire channels or layers to provide computational efficiency and reduced latency [, , , , , ]. One particularly efficient way is expert pruning in SMoE LLMs, a type of structured pruning with coarse granularity, which enhances overall efficiency. Recent expert pruning methods achieve 25%-50% sparsity and accelerate inference, but struggle to maintain performance [] or need fine-tuning, requiring substantial GPU memory and resources [, ]. Thus, there is a pressing need for efficient pruning methods that operate within the constraints of inference resources for SMoE LLMs.

Figure: (a) A SMoE block before pruning.
Refer to caption: extracted/5697370/Figures/illustration_baseline.png

In this work, we propose a gradient-free evolutionary strategy that achieves high sparsity while maintaining performance given a small train set on the downstream tasks. Our method is divided into two phases: expert pruning and expert merging. To facilitate the search for optimal pruning configurations, we design a parameter space for router mapping and expert merging, represented by two weight matrices, $W_{RM}$ and $W_{EM}$. These matrices are applied to the router weighting and expert modules, as illustrated in Fig. 1. In the first phase, expert pruning, we search through the weight matrices to retain the most prominent experts without updating any network parameters. In the second phase, expert merging, we retrieve knowledge from the pruned experts and consolidate it into the retained experts. To these ends, $W_{RM}$ and $W_{EM}$ are set to one-hot rows in the first phase and to real numbers in the second phase. Since our method is gradient-free, it can be conducted on devices capable of inference. Our contributions can be summarized as follows:

- •
Pruning the total number of experts: smaller memory consumption and better performance. Our approach enables more aggressive pruning of experts compared to current methods [, ]. In experiments on Mixtral $8\times 7$B-Instruct, we reduce the number of experts in each SMoE block from 8 to 2, a 72% reduction in parameters, while maintaining comparable performance across various downstream tasks. Surprisingly, we observe that fewer experts can lead to better performance. For instance, on the SQuAD dataset, pruning 4 out of 8 experts result in a performance increase from 53.4% to 75.4% without updating the remaining experts.
- •
Pruning the number of active experts: better inference efficiency. We explore the pruning of active experts and find that effective expert merging compensates for the loss of active experts across downstream tasks. This process significantly improves efficiency without compromising the model’s utility on these tasks. For instance, by reducing the active experts in Mixtral 8 $\times$ 7B from two to one, we observe a prefill acceleration of up to 1.63$\times$.
- •
Generalization ability. We test the performance of our method on datasets with higher diversity and out-of-distribution tasks using MMLU []. Specifically, we take 50 of the 57 datasets included in MMLU and conduct EEP using data from a small subset of each of the 50 datasets. We then evaluate the pruned model on the test data of i) the 50 datasets and ii) the 7 unseen datasets. In both evaluation tasks, we observe that EEP consistently outperforms other pruning methods, demonstrating the strong generalization ability of our method.
- •
A novel and efficient pruning paradigm.
Common pruning paradigm usually conducts two steps. In the first step, parameters are pruned using empirical criteria. This operation often lowers performance. In the second step, retained parameters are fine-tuned through stochastic gradient descent to recover performance. This operation often requires substantial GPU memory and computation time, making it prohibitive for most users of LLMs. EEP adopts a gradient-free evolutionary strategy for both pruning and fine-tuning. As a result, our pruned model significantly outperforms the pruned models of other methods, while our pruning and fine-tuning processes can run on devices affordable for inference, making EEP more widely applicable.
In addition, to inherit knowledge from the unpruned model, existing methods either select a subset of weights based on predefined importance criteria [, , ], or rely on distillation techniques [, , ]. In contrast, EEP introduces a novel approach as a third paradigm, employing weight merging [] to transfer knowledge during the model compression process.

## 2 Related work

Sparse Mixture-of-Experts LLMs.
Shazeer et al. [] introduced the sparse MoE layer, which consists of multiple experts, each being a simple feed-forward network (FFN), and a trainable router network that selects a sparse combination of the experts to process each input. Such SMoE models can significantly increase model capacity while maintaining computational efficiency. However, this utility is ideally achieved when the router accurately and evenly assigns experts to each token during training and inference. Many works address these challenges [, , , ]. Recently, many SOTA LLMs adopt the SMoE structure to achieve high performance and computational efficiency simultaneously [, , , ]. Additionally, Zhang et al. [] propose transforming non-MoE models into SMoE models to accelerate inference, and Komatsuzaki et al. [] upcycle pretrained models by reusing the parameters to initialize SMoE models, where all experts are replicates of the original FFNs, and then fine-tune the SMoE models.

Pruning for LLMs.
Pruning techniques have emerged as a crucial strategy for optimizing LLMs by reducing model size and computational costs while maintaining performance. Unstructured pruning [, , , ] entails the removal of individual weights according to specific criteria, creating sparse networks that demand specialized hardware for efficient execution. In contrast, structured pruning [, , , , , , , , ] eliminates entire structures, such as neurons or attention heads, facilitating more straightforward implementation on standard hardware. Within structured pruning, specific focus areas include attention mechanisms, where redundant heads are pruned to streamline the self-attention layers, and FFNs where unnecessary neurons are removed to enhance computational efficiency. Additionally, expert pruning for SMoE models selectively prunes the expert networks [, , , ].

Evolutionary Strategy for Optimization.
Evolutionary Strategies (ES) have been increasingly recognized for their robustness and flexibility in various optimization tasks, particularly where gradient-based methods fall short []. Notably, ES is highly effective for optimizing non-differentiable objective functions, offering a powerful alternative in scenarios where gradients are unavailable or unreliable [, , , , ]. Furthermore, ES excels in discrete optimization spaces, making it suitable for a wide range of combinatorial problems [, , ]. Recent advancements have extended the application of ES to the domain of LLMs, enabling memory-efficient fine-tuning without the need for backpropagation [].

Figure: Figure 2: We leverage EEP for two purposes: reducing the total number of experts, which lowers the memory footprint (use case 1), and reducing the number of active experts, thereby accelerating inference (use case 2).
Refer to caption: extracted/5697370/Figures/use_case.png

## 3 Background of sparse Mixture-of-Expert language model

In this section, we discuss the general concept of sparse Mixture-of-Experts (SMoE) implementation in modern decoder-only models, using the Mixtral family [] as a specific focus. A schematic illustration is provided in Fig. 1(a).

Notations.
Let $\bm{X}\in\mathbb{R}^{n\times d}$ represent the input to a SMoE block, where $n$ is the sequence length and $d$ is the hidden dimension. The output of the attention block is denoted by $\bm{Z}\in\mathbb{R}^{n\times d}$. The main parameters in the attention block are the weight matrices for computing query, key, and value: $\bm{W}_{Q},\bm{W}_{K},\bm{W}_{V}$. In the SMoE structure, there are $E$ experts, each represented by a feed-forward network (FFN) with parameters $\bm{\theta}_{i}$ for the $i$-th expert. The router network, denoted by $\bm{W}_{R}$, produces routing weights $\bm{G}\in\mathbb{R}^{n\times E}$ for the sparse activation of the experts. For clarity, we omit the normalization layers and biases.

Self-Attention Mechanism.
The self-attention mechanism computes the query, key, and value matrices as follows: $\bm{Q}=\bm{X}\bm{W}_{Q},\,\bm{K}=\bm{X}\bm{W}_{K},\,\bm{V}=\bm{X}\bm{W}_{V}$. The attention scores and the output $\bm{Z}$ are then computed as:

$$ $\displaystyle\text{Attention}(\bm{Q},\bm{K},\bm{V})=\text{softmax}\left(\frac{ \bm{Q}\bm{K}^{\top}}{\sqrt{d_{k}}}\right)\bm{V},\quad\bm{Z}=\text{Attention}( \bm{Q},\bm{K},\bm{V})\bm{W}_{O},$ (1) $$

where $\text{softmax}(\cdot)$ denotes a row-wise softmax function.
The attention mechanism produces a weighted sum of the values $\bm{V}$, where the weights are derived from the dot product of the queries $\bm{Q}$ and keys $\bm{K}$, scaled by the square root of key/query dimension $\sqrt{d_{k}}$. Then the weighted averaged values are mapped by the output matrix $\bm{W}_{O}$ to $\bm{Z}$.

#### Router Network in SMoE Structure.

The router network determines which experts to activate and how to scale their outputs. The routing weights $\bm{G}\in\mathbb{R}^{n\times E}$ are computed as:

$$ $\displaystyle\bm{G}$ $\displaystyle=\text{softmax}(\bm{Z}\bm{W}_{G}).$ (2) $$

Sparse activation of the experts is achieved by selecting the top-$k$ routing weights for each input token. The output of the activated experts is scaled by the routing weights and aggregated to form the output of the SMoE layer $\bm{H}$:(^*^**The top-$k$ routing weights may be further normalized to sum to 1; this nuance is omitted here.)

$$ $\displaystyle\forall j=1\dots n,\quad\bm{H}_{j}=\sum_{i\in\text{TopK}(\bm{G}_{ j})}\bm{G}_{ji}\cdot\text{FFN}_{i}(\bm{Z}_{j}),$ (3) $$

where $\text{TopK}(\bm{G}_{j})$ denotes the indices of the top-$k$ routing weights for the $j$-th input token, and $\text{FFN}_{i}$ denotes the function of the $i$-th expert, as defined below.

FFN as Expert.
Each expert in the SMoE structure is an independent FFN with two fully-connected layers, denoted by $\bm{W}_{1i}$ and $\bm{W}_{2i}$. When applying SwiGLU [], an additional weight matrix $\bm{W}_{3i}$ is introduced for the activation function. The $i$-th expert processes the input as follows:

$$ $\displaystyle\text{FFN}_{i}(\bm{Z}_{sub})$ $\displaystyle=\text{SwiGLU}(\bm{Z}_{sub},\bm{W}_{1i},\bm{W}_{3i})\bm{W}_{2i},$ (4) $$

where $\bm{Z}_{sub}$ denotes the a subset of rows in $\bm{Z}$ that activates the $i$-th expert.
Depending on the activation function, the parameters of the $i$-th expert are either $\bm{\theta}_{i}=\{\bm{W}_{1i},\bm{W}_{2i}\}$ or $\bm{\theta}_{i}=\{\bm{W}_{1i},\bm{W}_{2i},\bm{W}_{3i}\}$.

## 4 Method

Figure: Figure 3: Performance from a single expert to an ensemble of experts.
Refer to caption: extracted/5697370/Figures/motivation.png

In this section, we introduce our proposed approach for optimizing SMoE LLMs through expert pruning and merging. We aim to enhance the efficiency and performance of SMoE architectures by leveraging evolutionary strategies. Our method addresses the challenges of large and complex search spaces without incurring the prohibitive computational costs associated with gradient-based optimization. The subsequent subsections elaborate on our motivation (Sec. 4.1), the configuration of the parameter space (Sec. 4.2), the evolutionary optimization strategy employed to achieve our objectives (Sec. 4.3), and the use cases we apply EEP (Sec. 4.4).

### 4.1 Motivation

LLMs based on the SMoE architecture have shown remarkable performance across various natural language processing tasks [, , ]. These models leverage multiple experts, activating only a subset for any given input, thus balancing computational efficiency and model capacity. Typically, top-2 experts are activated, striking a balance between performance and computational cost.

Fig. 3 presents our investigation into the activation of different numbers of experts on Mixtral $8\times 7$B-Instruct, revealing the following observations: i) Activating only a single expert does not lead to model collapse and may result in only a minimal performance drop compared to the default setting of using two experts. This suggests that individual experts possess redundant knowledge, enabling them to maintain reasonable performance independently. This redundancy indicates potential for expert pruning. ii) Conversely, activating all 8 experts leads to a noticeable performance gain, highlighting the benefits of expert ensemble. However, the computational cost of such an ensemble is substantially higher. Wortsman et al. [] have shown that merging differently fine-tuned models can efficiently substitute their ensemble, achieving similar performance with reduced computational overhead.

Building on these insights, we propose a two-step approach involving expert pruning followed by expert merging. Initially, we search for the optimal subset of experts given a fixed size. Subsequently, we employ expert merging to consolidate the knowledge from the pruned experts into the remaining ones. This approach not only restores the knowledge of the pruned experts but also updates the surviving experts to incorporate the collective expertise of the entire SMoE block.

### 4.2 Parameter space for expert pruning and merging

Expert Pruning and Merging Matrices.
To efficiently prune and merge experts in each SMoE block ($l=1\dots L$), we introduce two key matrices: the Router Mapping matrix ($\bm{W}_{\text{RM}}^{l}$) and the Expert Merging matrix ($\bm{W}_{\text{EM}}^{l}$). For clarity, we omit the block index $l$ in this section. A schematic illustration is provided in Fig. 1(b). The router mapping matrix $\bm{W}_{\text{RM}}\in\mathbb{R}^{E^{\prime}\times E}$, where $E^{\prime}$ is the reduced number of experts (i.e., $E$ > $E^{\prime}$), is applied to the routing weights $\bm{G}$ to reduce the dimensionality and handle fewer experts:

$$ $\displaystyle\bm{G}^{\prime}=\bm{W}_{\text{RM}}\text{softmax}(\bm{Z}\bm{W}_{G}),$ (5) $$

The expert merging matrix $\bm{W}_{\text{EM}}\in\mathbb{R}^{E^{\prime}\times E}$ is applied to the expert weights $\{\bm{\theta}_{i}\}_{i=1}^{E}$ to merge $E$ experts into $E^{\prime}$ experts. Each element in $\bm{W}_{\text{EM}}$ operates blockwise on the parameters of the experts. Denote $\{\omega_{j1},\omega_{j2},\dots,\omega_{jE}\}$ as the $j$-th row of $\bm{W}_{\text{EM}}$ that maps the original $E$ experts to the $j$-th new expert $\bm{\theta}^{\prime}_{j}$. We define merging as follows:

$$ $\displaystyle\bm{\theta}^{\prime}_{j}=\{\sum_{i=1}^{E}\omega_{ji}\bm{W}_{1i}, \sum_{i=1}^{E}\omega_{ji}\bm{W}_{2i},\sum_{i=1}^{E}\omega_{ji}\bm{W}_{3i}\},$ (6) $$

where the parameters of the experts are defined in Eq. 4.

Expert Pruning Phase.
During the expert pruning phase, the low-rank matrices $\bm{W}_{\text{RM}}$ and $\bm{W}_{\text{EM}}$ are initialized with each row as a one-hot vector to ensure that only pruning occurs. Additionally, $\bm{W}_{\text{RM}}$ and $\bm{W}_{\text{EM}}$ are set as to be identical $\bm{W}_{\text{RM}}=\bm{W}_{\text{EM}}$. Consequently, these matrices only retain the selected expert weights and their corresponding routing weights. During evolutionary search, EEP also maintains the one-hot format of $\bm{W}_{\text{RM}}$ and $\bm{W}_{\text{EM}}$.

Expert Merging Phase.
In the expert merging phase, $\bm{W}_{\text{RM}}$ and $\bm{W}_{\text{EM}}$ are decoupled and initialized from their optimal values obtained during the pruning phase. This decoupling allows for a more flexible transformation where multiple experts can be merged, and the router weights can be updated independently. During this phase, the elements of $\bm{W}_{\text{RM}}$ and $\bm{W}_{\text{EM}}$ transition from discrete $0/1$ values to continuous values. This allows the matrices to perform more nuanced transformations.

### 4.3 Evolutionary search for the router mapping and expert merging matrices

The search space of the router mapping and expert merging matrices is large and complex, making it difficult to design heuristics for determining a solution, as is done in other expert pruning studies [, , ]. Therefore, an efficient optimization strategy is necessary. Given the substantial size of SMoE LLMs, computing gradients for optimization is computationally prohibitive for most users. As a solution, we employ a gradient-free evolutionary strategy, similar to approaches found in previous works [, ]. Our algorithm is detailed in Alg. 1.

Initially, we populate the search space using random initialization. During the evolutionary search, each set of router mapping and expert merging matrices is treated as an individual. In each iteration, only the top-performing individuals are selected as parents to produce the next generation through crossover and mutation. Specifically, during crossover, we randomly combine the entries of the matrices from two parents or select one parent’s matrices entirely. For mutation, we introduce random Gaussian noise to the matrices, ensuring stochastic variations. This process conserves beneficial adaptations while discarding detrimental modifications, enhancing the optimization process.

This evolutionary reproduction process is repeated for a predetermined number of iterations within each search phase, updating the population with newly generated individuals. Upon completion of the search process, the best individual is selected as the output of our search algorithm.

### 4.4 Use Cases

We explore two applications of EEP: expert pruning and expert activation pruning. In expert pruning, EEP searches for optimal router mapping ($\bm{W}_{\text{RM}}$) and expert merging matrices ($\bm{W}_{\text{EM}}$) to minimize the total number of experts while maintaining high performance. For expert activation pruning, the goal is to achieve strong performance with only one active expert per token. Here, we use the same EEP search algorithm to conduct expert and router networks optimization by updating the $\bm{W}_{\text{RM}}$ and $\bm{W}_{\text{EM}}$ matrices, while only activates one expert during inference. Fig. 2 illustrates these two use cases. Additionally, we investigate the combination of these two approaches, reducing both the total number of experts and the number of active experts simultaneously (see Sec. 5.3).

## 5 Experiments

In this section, we validate the effectiveness of our method by considering two use cases: expert pruning and expert activation pruning. In Sec. 5.1, we introduce the experimental settings. In Sec. 5.2, we investigate the first use case, expert pruning, by applying EEP to reduce the total number of experts. In Sec. 5.3, we further explore expert activation pruning, applying EEP to maintain performance while reducing the number of active experts by changing the top-2 routing weights to top-1. We also examine a composite case where both the total number of experts and the number of active experts are reduced. In Sec. 5.4, we present the experimental results on larger and more diverse datasets, as well as performance on out-of-distribution datasets, to validate the generalization ability of EEP. In Sec. 5.5, we profile memory usage and inference speed to demonstrate that our method achieves significant improvements compared to the full SMoE models. In Sec. 5.6 we provide insights on the observation of fewer experts but higher performance. More results, including experiments on larger datasets and other models, can be found in App. D.

### 5.1 Experimental settings

Our main results are based on the popular SMoE models Mixtral 8$\times$7B []. We also include a larger model, Mixtral 8$\times$22B [], to demonstrate the generalization of our methods. We use the "Instruct" version of these models for the generation tasks. We select tasks from the SuperGLUE dataset, as well as several other generation tasks, including SQuAD [] and DROP [].
For each individual dataset, we randomly sample a subset from the training set to conduct evolutionary search and use the test set for evaluation. Additional details can be found in App. A.

Evaluation. We adopt a generation-based evaluation approach for all datasets. Specifically, we use the instruction fine-tuned model to generate answers directly in response to the given questions and apply template matching to determine the correctness of the answers. Our evaluation protocol primarily follows the implementation of OpenCompass [] for the design of question prompts, types of templates, and matching criteria, with a few modifications to better suit the Mixtral family of models. All experiments use the same evaluation settings. Examples of prompts and model outputs can be found in App. E and App. F.

Baselines. Since our method aims to compress the instruction fine-tuned SMoE models on downstream tasks, we consider the zero-shot performance as our main baseline to show that EEP can achieve a significant decrease on the memory footprint and/or computation overhead during the inference time while maintain or even achieve better performance. For the use case of decreasing the total number of experts, we additionally compare EEP with four other types of baseline to demonstrate the effectiveness of the designed search space and the evolutionary-search-based tuning method: (1) Random selection of pruned experts, (2&3) Pruning the experts with the lowest frequency of being activated or the lowest soft activation values [], and (4) NAEE [], which exhaustively evaluates the loss between the full model and all pruning choices for each layer and select the one with the lowest loss. For the use case of decreasing the active number of experts, we select the dynamic skipping method proposed by NAEE [] as an additional baseline. More details are given in App. A.

### 5.2 Reducing the total number of experts

We apply EEP to search for the optimal pruning configuration, parameterized by the router mapping matrix $\bm{W}_{\text{RM}}$ and the expert merging matrix $\bm{W}_{\text{EM}}$, for maintaining 4 experts and 2 experts. EEP (Prune Only) indicates the results from solely conducting the expert pruning phase as described in Sec. 4.2. In contrast, EEP (Prune + Merge) shows the results after the complete evolutionary search process. The results are shown in Tab. 1, and we discuss them below. Random is conducted 30 times, and we present the mean results here, deferring the complete results to Sec. D.4.

**Table 1: Results of expert pruning on Mixtral 8$\times$7B-Instruct. Bold values indicate the best across all methods; underlined values show the best without parameter updates (i.e., excluding EEP (Prune+Merge)).**
| Expert | Method | COPA | MultiRC | WIC | WSC | RTE | BoolQ | CB | ReCoRD | DROP | SQuAD | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Num=8 | Full Model | 89.0 | 83.0 | 51.8 | 63.5 | 73.2 | 77.4 | 51.7 | 50.3 | 30.6 | 53.4 | 62.4 |
| Num=4 | Random | 63.8 | 49.4 | 37.6 | 43.3 | 45.1 | 50.2 | 38.7 | 35.1 | 27.4 | 58.3 | 44.9 |
| Frequency [] | 63.0 | 74.8 | 36.0 | 34.6 | 18.1 | 71.0 | 30.4 | 41.6 | 29.9 | 58.2 | 45.8 |  |
| Soft Activation [] | 73.0 | 30.6 | 51.4 | 37.5 | 41.9 | 40.4 | 17.9 | 36.8 | 33.3 | 10.2 | 37.3 |  |
| NAEE [] | 87.0 | 76.0 | 52.6 | 64.5 | 61.7 | 77.2 | 51.7 | 50.4 | 30.6 | 53.0 | 60.5 |  |
| EEP (Prune Only) | 95.0 | 81.2 | 57.8 | 67.3 | 74.0 | 82.8 | 69.6 | 60.0 | 37.3 | 75.2 | 70.3 |  |
| EEP (Prune+Merge) | 99.0 | 84.6 | 65.0 | 73.1 | 76.9 | 84.8 | 75.0 | 63.6 | 39.7 | 80.6 | 74.2 |  |
| Num=2 | Random | 36.8 | 22.3 | 13.6 | 15.0 | 28.4 | 15.5 | 38.6 | 16.9 | 18.3 | 36.9 | 24.2 |
| Frequency [] | 51.0 | 17.6 | 8.8 | 1.9 | 48.4 | 30.6 | 35.7 | 10.4 | 14.9 | 9.2 | 24.9 |  |
| Soft Activation [] | 33.0 | 18.2 | 49.4 | 18.5 | 15.2 | 1.8 | 32.1 | 4.4 | 11.7 | 50.0 | 23.4 |  |
| NAEE [] | 75.0 | 42.4 | 48.4 | 49.0 | 54.5 | 49.8 | 19.6 | 42.0 | 31.2 | 58.2 | 47.0 |  |
| EEP (Prune Only) | 76.0 | 63.8 | 51.8 | 63.5 | 64.3 | 70.6 | 58.9 | 47.2 | 37.1 | 64.0 | 59.7 |  |
| EEP (Prune+Merge) | 93.0 | 71.6 | 58.6 | 65.4 | 69.0 | 75.6 | 66.1 | 47.2 | 38.4 | 70.2 | 65.6 |  |

EEP fully exploits expert-wise redundancy on downstream tasks. Based on the results obtained from the pruning phase of EEP, retaining only 4 experts allows the model to achieve better performance and lower computational costs simultaneously on most datasets, except for MultiRC. Even with a particularly low budget of retaining only 2 experts, EEP can still achieve comparable or even better performance than the full model on five datasets, with some datasets showing significant improvements over the best baseline (e.g., 58.9 vs. 51.7 on CB and 64.0 vs. 53.4 on SQuAD). For the remaining datasets, model collapse is avoided.

EEP is more effective than other baseline methods for selecting pruned experts. Comparing the results of other methods, we find that EEP is more effective for identifying the optimal pruning pattern. Random sampling of experts results in low mean accuracy and high variance. Pruning experts based on selection frequency also performs poorly on most datasets and has a high probability of collapse under high sparsity. NAEE can nearly maintain the performance of the full model when retaining four experts. However, EEP surpasses all methods by a large margin across all datasets.

Expert merging brings significant improvements after pruning. As shown in the last row for each pruning rate in Tab. 1, the results after expert merging exceed those obtained through the expert pruning phase alone. Specifically, expert merging achieves a general improvement on almost all datasets. On WIC, CB, and SQuAD under both pruning rates, and on WSC when four experts are retained, the accuracy improvement reaches 5%$\sim$7%, demonstrating its effectiveness in restoring the knowledge of pruned experts and enhancing individual experts. Additionally, we find expert merging to be an effective method for fine-tuning SMoE LLMs (i.e., keeping the number of total and active experts); the results of this are presented in Tab. 9.

Generality across models. With the promising results of Mixtral 8$\times$7B-Instruct model, we further apply EEP to a larger model: Mixtral 8$\times$22B-Instruct [], Qwen1.5-MoE-A2.7B-Chat [], and Qwen2-MoE-A14B-Chat []. We conduct experiments on fewer datasets due to the constraint of computational resource. Results are shown at Tab. 2, Tab. 7, and Tab. 8, respectively. EEP also achieves a strong improvement and above observations are still held, which indicates the scaling-up ability of EEP towards large SMoE models.

**Table 2: Results of expert pruning on Mixtral $8\times 22$B-Instruct. Bold values indicate the best across all methods; underlined values show the best without parameter updates (i.e., excluding EEP (Prune+Merge)).**
| Budget | Method | WIC | WSC | BoolQ | CB | SQuAD | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Num=8 | Full Model | 68.2 | 81.7 | 90.2 | 46.5 | 45.8 | 66.5 |
| Num=4 | Random | 27.0 | 30.2 | 37.8 | 34.6 | 37.2 | 33.4 |
| Frequency [] | 0.0 | 38.5 | 76.6 | 57.1 | 50.6 | 30.6 |  |
| Soft Activation [] | 25.2 | 60.6 | 6.4 | 60.7 | 54.2 | 41.4 |  |
| NAEE [] | 64.0 | 68.3 | 78.4 | 33.9 | 52.4 | 59.4 |  |
| EEP (Prune Only) | 70.2 | 84.2 | 89.6 | 75.0 | 71.4 | 78.1 |  |
| EEP (Prune+Merge) | 72.2 | 87.5 | 89.6 | 78.6 | 74.0 | 80.4 |  |
| Num=2 | Random | 13.9 | 10.1 | 11.0 | 24.9 | 15.6 | 15.1 |
| Frequency [] | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |  |
| Soft Activation [] | 2.4 | 1.9 | 3.6 | 19.6 | 52.6 | 16.0 |  |
| NAEE [] | 34.0 | 32.7 | 45.0 | 16.1 | 50.0 | 30.6 |  |
| EEP (Prune Only) | 57.8 | 63.5 | 76.0 | 50.0 | 71.0 | 63.7 |  |
| EEP (Prune+Merge) | 59.6 | 65.4 | 76.4 | 58.9 | 75.0 | 67.1 |  |

### 5.3 Reducing the number of active experts

Next, we present the experimental results for the second use case: decreasing the number of active experts. We modify the number of active experts by changing the top-k from $k=2$ to $1$ while applying EEP to restore model performance. We evaluate our method with two different total numbers of experts (8 and 4). The results are presented in Tab. 3. We summarize the observations below.

EEP can improve individual experts through expert merging, allowing a single expert to handle the inference. Keeping the total number of experts at 8 and reducing the number of active experts to 1 consistently leads to a decline in baseline performance. However, by optimizing the model with EEP, we introduce a reliable improvement that mitigates this gap, resulting in comparable or even better performance than the full model. It is important to note that when the total number of experts is maintained, there is no expert pruning phase; only expert merging is applied for EEP.

The two use cases can be combined through EEP. By retaining fewer experts and simultaneously reducing the number of active experts, we achieve significant savings *in both GPU memory and inference time* (see Sec. 5.5). EEP can be directly applied in this scenario. Results show that with 4 total experts and 1 active expert, EEP achieves performance comparable to or even better than the full model.

**Table 3: Results of active expert pruning on Mixtral $8\times 7$B. Bold values show the best performance. “Active” indicates the average number of experts active per token. Avg. stands for average.**
| Total | Active | Method | WIC | WSC | BoolQ | CB | SQuAD | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 8 | 2 | Full Model | 51.8 | 63.5 | 77.4 | 51.7 | 53.4 | 59.6 |
| 1 | Full Model | 50.8 | 48.1 | 66.0 | 48.2 | 43.8 | 51.4 |  |
| 1.4$\sim$1.5 | Dyn [] | 50.0 | 59.6 | 72.8 | 46.4 | 44.8 | 54.7 |  |
|  | 1 | EEP | 59.2 | 70.2 | 79.0 | 66.1 | 51.8 | 65.3 |
| 4 | 1 | NAEE [] | 48.6 | 20.2 | 56.2 | 33.9 | 51.8 | 42.1 |
| 1.4$\sim$1.5 | NAEE+Dyn [] | 43.4 | 61.5 | 36.2 | 53.6 | 53.4 | 49.6 |  |
| 1 | EEP | 55.8 | 70.2 | 74.4 | 64.3 | 72.0 | 67.3 |  |

### 5.4 In-distribution and out-of-distribution generalization on diverse datasets

In this section, we further test EEP on a larger dataset, MMLU, to validate the generalization ability of EEP. We randomly split all 57 datasets in MMLU into two subsets containing 50 and 7 datasets, as the base dataset and the out-of-distribution (OOD) test dataset, respectively. We further divide each dataset in the larger subset into training and validation sets. We conduct our EEP on the training sets and use both the validation sets and the OOD test dataset to evaluate the performance of the searched patterns. Results are shown in Tab. 4. We find that EEP outperforms baseline methods on both the base dataset and the OOD test dataset. This indicates that EEP possesses the ability to handle large and diverse datasets and exhibits a certain level of generalization capability.

**Table 4: Results of expert pruning on Mixtral $8\times$7B-Instruct on MMLU dataset. Bold values indicate the best performance; underlined values show the best without updating remaining parameters.**
| Budget | Method | IID (50 val. sets) | OOD (7 unseen datasets) |
| --- | --- | --- | --- |
| Num=8 | Full Model | 60.7 | 72.6 |
| Num=6 | Random | 53.0$\pm$9.6 | 64.6$\pm$10.0 |
| Frequency [] | 35.2 | 35.0 |  |
| Soft Activation [] | 54.3 | 65.6 |  |
| NAEE [] | 57.5 | 69.4 |  |
|  | EEP (Prune Only) | 59.6 | 71.4 |
|  | EEP (Prune+Merge) | 61.8 | 71.3 |
| Num=4 | Random | 45.1$\pm$6.1 | 50.3$\pm$10.7 |
| Frequency [] | 26.6 | 25.2 |  |
| Soft Activation [] | 46.7 | 53.1 |  |
| NAEE [] | 53.5 | 63.6 |  |
|  | EEP (Prune Only) | 55.4 | 62.4 |
|  | EEP (Prune+Merge) | 56.9 | 64.6 |

### 5.5 Improvements in memory usage and inference speed

We profile the memory overhead and inference speed of Mixtral $8\times 7$B model for the two use cases. We conduct tests on SQuAD with a batch size of 256 using two NVIDIA A100 GPU cards. We report the peak memory usage and the wall-time acceleration ratio in Tab. 5. As shown in Tab. 5, retaining only 4 and 2 experts from the whole model decreases the memory overhead by 47% and 71%, respectively. Additionally, reducing the total number of experts improves inference speed due to higher parallelism, achieving a speedup of 1.11$\times$ and 1.18$\times$ with 4 and 2 experts, respectively. In the use case of reducing active experts, an acceleration ratio of 1.24$\times$ is achieved. Finally, when combining the two use cases with 4 total experts and 1 active expert per token, EEP saves 47% of GPU memory and achieves a 1.41$\times$ increase in inference speed. The profiling results indicate that EEP can significantly reduce the computational cost and memory consumption of SMoE LLMs.

**Table 5: Profiling the memory footprint and inference speedup of Mixtral $8\times 7$B.**
| Total | Active | Method | Speedup | GPU Mem(GB) |
| --- | --- | --- | --- | --- |
| 8 | 2 | Full Model | 1.0$\times$ | 88.6 |
| 1 | EEP | 1.24$\times$ |  |  |
| 4 | 2 | EEP | 1.11$\times$ | 46.6 |
| 1 | EEP | 1.41$\times$ |  |  |
| 2 | 2 | EEP | 1.18$\times$ | 25.6 |

Figure: (a) Activation (1/0 means activated/not activated) correlation before and after pruning.
Refer to caption: extracted/5697370/Figures/squad_corr_layer0.png

### 5.6 Why fewer experts leads to better performance

At first glance, it may seem counterintuitive that reducing the number of experts can improve performance as shown in Tabs. 1 and 2, especially when the remaining parameters are not retrained. Our hypothesis is that the router network operates differently after expert pruning, leading to this improvement. Typically, the router network is implemented as a smaller network, such as a one-layer perceptron. This makes it challenging to accurately partition the high-dimensional hidden space among experts. The issue of imbalanced activation has been identified in several works [, ]. If the router network does not function optimally before pruning, there may be potential for improvement by enabling the router to focus on a smaller subset of experts.

Although it is difficult to directly evaluate the router network’s performance, we have observed that its behavior changes significantly after pruning. This change occurs because the pruning process eliminates some experts, and the routing weights for the remaining experts are normalized to sum to one. In Fig. 4, we observe distinct patterns in the accumulated activation times of the experts, their accumulated routing weights, and the activation correlation across experts. More demonstration of the expert activation pattern can be found in Sec. D.6.

## 6 Conclusion

In this work, we present EEP, a gradient-free evolutionary search method optimized for pruning within an efficienct parameter space. Through extensive experiments on various downstream datasets, we demonstrate that EEP achieves superior performance and greater sparsity compared to baseline methods. Additionally, we make a novel observation that the performance of SMoE models on downstream tasks can be enhanced through pruning, even without updating the remaining parameters. We discuss the potential reasons for this phenomenon, suggesting that pruning may lead to a more effective routing mechanism by reducing the complexity the router network needs to manage.

Limitations. Although we demonstrated promising results, our approach still requires a potentially costly search process. We leave the optimization of search cost to future work.

## Acknowledgement

This work was supported by National Natural Science Foundation of China (No. 62325405, 62104128, U19B2019, U21B2031, 61832007, 62204164), Flemish Government (AI Research Program) and the Research Foundation - Flanders (FWO) through project number G0G2921N, Tsinghua EE Xilinx AI Research Fund, and Beijing National Research Center for Information Science and Technology (BNRist). We thank for all the support from Infinigence-AI.

## Appendix A Additional Details on Experimental Settings

### A.1 Ours setting

Search Space. As mentioned in Sec. 5, to avoid optimizing too many parameters, we split the weights of all experts into several groups. The merging coefficients $\bm{W}_{\text{EM}}$ and $\bm{W}_{\text{RM}}$ within the same group are shared. Most of our main results are obtained by uniformly splitting all weights into four groups based on their depth, except for the experiments on the RTE, ReCoR, and DROP datasets in Tab. 1. We find that for these datasets, setting each layer as an independent group performs significantly better than using only four groups during the pruning phase. More detailed results can be found in Sec. D.5. For other datasets, we maintain the current setting without exploring other configurations, as it consistently yields good performance.

Search Process. We apply a two-stage search method as discussed in Sec. 4.2. The pruning phase consists of 40 iterations, followed by 160 iterations for the expert merging phase. At each iteration, we evaluate the accuracy on the training set and use this metric as the score for all individuals of merging coefficients in the population. Examples of the performance curve over the search iterations are provided in Sec. D.5.

Selected Datasets for OOD Evaluation. In Sec. 5.4, we randomly select 7 datasets for OOD test. These datasets are: (1) lukaemon_mmlu_electrical_engineering, (2) lukaemon_mmlu_professional_accounting, (3) lukaemon_mmlu_high_school_macroeconomics, (4) lukaemon_mmlu_high_school_computer_science, (5) lukaemon_mmlu_business_ethics, (6)
lukaemon_mmlu_miscellaneous, and (7) lukaemon_mmlu_high_school_psychology.

### A.2 Baselines

To evaluate the effectiveness of reducing the total number of experts, we compare our method against four baseline approaches: (1) Random selection of pruned experts, (2) pruning experts with the lowest frequency of activation, (3) pruning experts with the lowest soft activation values, and (4) NAEE [], which exhaustively evaluates the discrepancy between the full model and all pruning choices for each layer and selects the one with the lowest discrepancy. For reducing the number of active experts, we adopt the dynamic skipping scheme from NAEE as a baseline approach.

For random selection, we uniformly sample a corresponding number of experts from all 8 experts in each layer. The full results with error margins for random selection are presented in Tab. 11.

For the frequency-based method, we run the model on the training set and count the number of times each expert is activated. We then prune the experts with the lowest frequency in each layer.

For the soft activation method, we run the model on the training set and accumulate the router weighting (soft activation value) for each expert. We then prune the experts with the lowest accumulated values in each layer.

For NAEE, we enumerate all pruning choices for each layer and select the one with the smallest output discrepancy compared to the full model. We use a batch of calibration data with a size of 64 to calculate the discrepancy. For the dynamic skipping scheme, we run the model on the entire training set to determine the median value of the ratio between the two largest routing weights for each layer. During validation, we dynamically skip the expert with the second-largest routing weight if the ratio between its weight and the largest weight is below the threshold. This results in an average of approximately 1.5 active experts.

## Appendix B Size of current SMoE LLMs

Tab. 6 shows the basic parameter information of modern SMoE Large LLMs.

**Table 6: Active Parameters, Total Parameters, and Parameters of the Experts for Various Models**
| Model | Active Parameters | Total Parameters | Parameters of Experts |
| --- | --- | --- | --- |
| Mixtral 8x7B | 13B | 47B | 45B |
| Mixtral 8x22B | 39B | 141B | 136B |
| Grok-1 | 79B | 314B | 313B |
| DBRX | 36B | 132B | 128B |
| Qwen 1.5-MoE-A2.7B | 2.7B | 14.3B | 13.2B |
| Qwen 2-57B-A14B | 14B | 57B | 49B |

## Appendix C Algorithm Details

Alg. 1 presents the details of EEP. The notations are consistent with those in Sec. 4.2. For the $\mathrm{Crossover}$ operation, we combine the merging coefficients of the parent models along the dimension of the retained experts. For the $\mathrm{Mutate}$ operation, we perturb the merging coefficients. Specifically, during the pruning phase, we randomly replace the pruned experts with other experts and set the router weights accordingly. In the expert merging phase, we perturb the merging coefficients element-wise by adding Gaussian noise.

Figure: Algorithm 1 Evolutionary Search of EEP

## Appendix D Additional Results

### D.1 Results with other models

In this section, we further apply EEP to the Qwen 1.5 [] and Qwen 2 [] SMoE models. Results can be found in Tab. 7 and Tab. 8. The same observations in Sec. 5 hold for these models: (1) EEP selects better pruning patterns than other baseline methods without updating the remaining parameters, and (2) expert merging brings improvements in most cases.

For the Qwen1.5-MoE-A2.7B-Chat [], we notice that other methods are prone to collapse. Conversely, the situation is the opposite for the Qwen2-MoE-A14B-Chat model []. Most baseline methods can maintain the performance of the full model with an extremely low number of experts retained. In face, we observe that the experts in the Qwen2-MoE-A14B-Chat model are specifically homogeneous, as the model’s performance is largely maintained even when only one random expert is activated per token. However, according to the information provided in their technical report, both Qwen1.5-MoE-A2.7B andQwen2-MoE-A14B employ upcycling and 64 experts per layer. We thus speculate that other training configurations, such as sizes and optimizer hyperparameters, lead to different final statuses. Nevertheless, EEP always achieves comparable or better performance than the full model and outperforms all baseline methods across settings, demonstrating its adaptability to different SMoE models.

**Table 7: Results of expert pruning on Qwen1.5-MoE-A2.7B-Chat. Bold values indicate the best performance; underlined values show the best without updating remaining parameters. For NAEE, due to the excessive number of combinatorial possibilities, we only randomly select 5k of them for each layer.**
| Budget | Method | WIC | WSC | BoolQ | CB | SQuAD | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Num=60 | Full Model | 51.4 | 46.2 | 73.6 | 32.1 | 68.6 | 54.4 |
| Num=30 | Random | 3.7$\pm$12.1 | 7.6$\pm$14.3 | 8.1$\pm$12.9 | 5.6$\pm$8.4 | 19.5$\pm$23.0 | 8.9 |
| Frequency [] | 55.6 | 9.6 | 2.4 | 0.0 | 17.9 | 21.7 |  |
| Soft Activation [] | 51.4 | 30.8 | 0.4 | 44.6 | 28.0 | 31.0 |  |
| NAEE [] | 0.0 | 0.0 | 1.6 | 0.0 | 34.6 | 7.2 |  |
| EEP (Prune Only) | 59.8 | 59.6 | 78.0 | 71.4 | 70.6 | 67.9 |  |
| EEP (Prune+Merge) | 62.6 | 66.3 | 81.4 | 76.9 | 71.4 | 71.7 |  |
| Num=15 | Random | 1.4$\pm$5.9 | 0.5$\pm$1.3 | 2.0$\pm$4.1 | 4.3$\pm$10.6 | 1.1$\pm$3.4 | 1.9 |
| Frequency [] | 0.0 | 0.0 | 7.8 | 16.1 | 0.0 | 4.9 |  |
| Soft Activation [] | 26.2 | 3.9 | 0.0 | 0.0 | 25.4 | 11.1 |  |
| NAEE [] | 0.0 | 1.0 | 5.2 | 0.0 | 0.0 | 1.2 |  |
| EEP (Prune Only) | 51.0 | 36.5 | 45.4 | 60.7 | 57.6 | 50.2 |  |
| EEP (Prune+Merge) | 54.4 | 63.5 | 58.2 | 58.9 | 76.9 | 62.7 |  |

**Table 8: Results of expert pruning on Qwen2-MoE-A14B-Chat. Bold values indicate the best performance; underlined values show the best without updating remaining parameters. For NAEE, due to the excessive number of pruning patterns, we only randomly select 2k of them for each layer.**
| Budget | Method | WIC | WSC | BoolQ | CB | SQuAD | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Num=64 | Full Model | 60.2 | 68.3 | 88.8 | 67.9 | 74.4 | 71.9 |
| Num=8 | Random | 55.3$\pm$7.1 | 61.6$\pm$5.6 | 78.7$\pm$7.3 | 35.4$\pm$17.6 | 79.7$\pm$2.4 | 62.1 |
| Frequency [] | 58.8 | 59.6 | 79.4 | 46.4 | 78.2 | 64.5 |  |
| Soft Activation [] | 60.8 | 64.4 | 82.6 | 14.3 | 75.2 | 59.5 |  |
| NAEE [] | 56.6 | 60.6 | 82.6 | 41.1 | 81.2 | 64.4 |  |
| EEP (Prune Only) | 61.8 | 72.1 | 85.8 | 76.8 | 85.6 | 76.4 |  |
| EEP (Prune+Merge) | 63.4 | 75.0 | 85.8 | 85.7 | 87.0 | 79.4 |  |
| Num=4 | Random | 56.5$\pm$1.9 | 59.8$\pm$5.2 | 79.1$\pm$4.0 | 32.1$\pm$15.0 | 78.0$\pm$2.4 | 61.1 |
| Frequency [] | 56.8 | 60.6 | 83.2 | 17.9 | 80.0 | 59.7 |  |
| Soft Activation [] | 59.2 | 61.5 | 81.6 | 17.9 | 77.6 | 59.6 |  |
| NAEE [] | 55.0 | 61.5 | 75.8 | 21.4 | 79.6 | 58.7 |  |
| EEP (Prune Only) | 62.0 | 65.4 | 84.6 | 69.6 | 80.6 | 72.4 |  |
| EEP (Prune+Merge) | 63.8 | 72.1 | 85.8 | 80.4 | 84.2 | 77.3 |  |
| Num=2 | Random | 56.4$\pm$1.4 | 58.2$\pm$3.7 | 77.8$\pm$4.5 | 26.5$\pm$9.6 | 76.4$\pm$1.9 | 59.1 |
| Frequency [] | 58.0 | 60.6 | 79.6 | 42.9 | 72.4 | 62.7 |  |
| Soft Activation [] | 57.4 | 65.4 | 71.4 | 62.5 | 76.8 | 66.7 |  |
| NAEE [] | 55.6 | 56.7 | 73.4 | 16.1 | 75.0 | 55.4 |  |
| EEP (Prune Only) | 59.2 | 68.3 | 83.4 | 67.9 | 82.0 | 72.2 |  |
| EEP (Prune+Merge) | 61.0 | 70.2 | 84.4 | 76.8 | 83.8 | 75.2 |  |
| Num=1 | Random | 56.6$\pm$1.3 | 56.3$\pm$2.7 | 78.7$\pm$1.5 | 23.5$\pm$5.9 | 75.2$\pm$1.6 | 58.1 |
| Frequency [] | 52.2 | 62.5 | 78.6 | 35.7 | 77.0 | 61/ |  |
| Soft Activation [] | 57.8 | 63.5 | 77.4 | 42.9 | 76.0 | 63.5 |  |
| NAEE [] | 57.6 | 56.7 | 78.6 | 16.1 | 73.6 | 56.5 |  |
| EEP (Prune Only) | 57.8 | 65.4 | 82.6 | 57.1 | 81.4 | 68.5 |  |
| EEP (Prune+Merge) | 59.4 | 69.2 | 84.0 | 82.1 | 82.8 | 75.5 |  |

### D.2 Fine-tuning using EEP

EEP can also be applied to fine-tune the model without pruning. As shown in Tab. 9, the effectiveness of EEP in fine-tuning demonstrates the efficiency of expert merging. Notably, EEP does not compute gradients and can therefore be executed on devices capable of inference.

**Table 9: Results of fine-tuning on Mixtral $8\times 7$B using EEP.**
| Method | WSC | WIC | RTE | BoolQ | CB | Record | SQuAD | DROP | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 63.5 | 51.8 | 73.2 | 77.4 | 51.7 | 50.3 | 53.4 | 30.6 | 56.5 |
| EEP | 78.8 | 69.2 | 78.7 | 86.2 | 80.4 | 63.0 | 78.4 | 51.5 | 73.2 |

### D.3 Profiling Results

We notice that the speedup ratio brought by pruning experts is influenced by the batch size. Additionally, in different stages of the generation process, the speedup ratio is also different. Therefore, we report more detailed profiling results of Mixtral $8\times 7$B model in Tab. 10.

**Table 10: Profiling the inference speedup of Mixtral $8\times 7$B.**
| Total | Active | Method | Prefill Speedup | Decode Speedup |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BS=1 | BS=32 | BS=256 | BS=1 | BS=32 | BS=256 |  |  |  |
| 8 | 2 | Full Model | 1.0$\times$ | 1.0$\times$ | 1.0$\times$ | 1.0$\times$ | 1.0$\times$ | 1.0$\times$ |
| 1 | EEP | 1.05$\times$ | 1.58$\times$ | 1.63$\times$ | 1.34$\times$ | 1.06$\times$ | 1.02$\times$ |  |
| 4 | 2 | EEP | 1.47$\times$ | 1.02$\times$ | 1.03$\times$ | 1.05$\times$ | 1.60$\times$ | 1.29$\times$ |
| 1 | EEP | 1.75$\times$ | 1.77$\times$ | 1.72$\times$ | 1.37$\times$ | 1.60$\times$ | 1.33$\times$ |  |
| 2 | 2 | EEP | 2.00$\times$ | 1.20$\times$ | 1.03$\times$ | 1.15$\times$ | 2.43$\times$ | 1.53$\times$ |

### D.4 Random search

We demonstrate the full results of the random pruning baseline with error margin in Tab. 11 and Tab. 12. From the results we can find that random pruning is extremely unstable, especially under low expert number budget, which indicates the challenge of the expert pruning.

**Table 11: Error margin of ramdom pruning on Mixtral $8\times 7$B.**
| Expert | Method | COPA | MultiRC | WIC | WSC | RTE | BoolQ | CB | ReCoRD | DROP | SQuAD |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Num=8 | Full Model | 89.0 | 83.0 | 51.8 | 63.5 | 73.2 | 77.4 | 51.7 | 50.3 | 30.6 | 53.4 |
| Num=4 | Random | 63.8$\pm$17.5 | 49.4$\pm$18.0 | 37.6$\pm$17.9 | 43.3$\pm$20.8 | 45.1$\pm$11.9 | 50.2$\pm$21.3 | 38.7$\pm$13.8 | 35.1$\pm$12.7 | 27.4$\pm$4.6 | 58.3$\pm$11.6 |
| Num=2 | Random | 36.8$\pm$14.6 | 22.3$\pm$8.4 | 13.6$\pm$14.8 | 15.0$\pm$18.1 | 28.4$\pm$13.4 | 15.5$\pm$17.1 | 38.6$\pm$10.8 | 16.9$\pm$7.4 | 18.3$\pm$3.2 | 36.9$\pm$12.6 |

**Table 12: Results of random pruning on Mixtral $8\times 22$B.**
| Budget | Method | WIC | WSC | BoolQ | CB | SQuAD |
| --- | --- | --- | --- | --- | --- | --- |
| Num=8 | Full Model | 68.2 | 81.7 | 90.2 | 46.5 | 45.8 |
| Num=4 | Random | 27.0$\pm$24.7 | 30.2$\pm$23.7 | 37.8$\pm$32.7 | 34.6$\pm$14.1 | 37.2$\pm$26.2 |
| Num=2 | Random | 13.9$\pm$15.1 | 10.1$\pm$13.2 | 11.0$\pm$12.9 | 24.9$\pm$15.6 | 15.6$\pm$20.3 |

### D.5 Ablation study

The hyperparameters of EEP include the number of groups that share the same coefficients, and the number of search iterations.

Number of Groups. We uniformly split all expert weights into a number of groups. We evaluate the results when there are 4 groups (the merging coefficients are shared across layers within the group) and 32 groups (i.e., the merging coefficients of each layer are effectively independent) on RTE, ReCoRD, and DROP. Results are shown in Tab. 13. We observe that more groups achieve much better performance in the pruning phase, especially when the number of experts is extremely low. However, dividing weights into more groups introduces more parameters to optimize, which may be detrimental to the expert merging phase. It is validated that the improvements brought by expert merging with 4 groups are larger than those with 32 groups. Taking all these factors into account, we use 32 groups for these three datasets and keep 4 groups for the rest of the experiments.

**Table 13: Results with different number of coefficient groups.**
| Group Number | Expert | Method | RTE | DROP | ReCoRD |
| --- | --- | --- | --- | --- | --- |
| 4 | Num=4 | Prune Only | 62.8 | 35.5 | 59.2 |
| Prune+Merge | 71.5 | 38.9 | 63.2 |  |  |
| Num=2 | Prune Only | 53.8 | 25.3 | 36.0 |  |
| Prune+Merge | 61.7 | 27.5 | 38.8 |  |  |
| 32 | Num=4 | Prune Only | 74.0 | 37.3 | 60.0 |
| Prune+Merge | 76.9 | 39.7 | 63.6 |  |  |
| Num=2 | Prune Only | 64.3 | 37.1 | 47.2 |  |
| Prune+Merge | 69.0 | 38.4 | 47.2 |  |  |

Search Iterations. We plot the Accuracy-Iteration curve in Fig. 5. We report the best accuracy among all evaluated merging coefficients at each iteration. From the figure, we can see that the evolutionary search in the pruning phase is effective and efficient, finding good pruning configurations from poor initialization within only 40 iterations. The expert merging phase can further improve performance based on the pruning results.

Figure: (a) Accuracy-Iteration curve on CB dataset.
Refer to caption: x1.png

### D.6 Router Pattern

In Sec. 5.6, we demonstrate the changes in expert activation patterns using the statistics from the first transformer block in a Mixtral 8 $\times$ 7B-Instruct model. Additionally, in this section, we provide the statistics for the 15^th transformer block Fig. 6 and the 31^st transformer block Fig. 7.

Figure: (a) Activation correlation before and after pruning.
Refer to caption: extracted/5697370/Figures/squad_corr_layer15.png

Figure: (a) Activation correlation before and after pruning.
Refer to caption: extracted/5697370/Figures/squad_corr_layer31.png

### D.7 Demonstration of Searched Patterns

We demonstrate the final searched patterns (pruning + merging) in Fig. 8. There is always one highlighted block in each row, which corresponds to the primarily retained experts in the pruning phase, while other values are close to zero. This shows that the merging matrix does not deviate significantly from the discrete matrix obtained in the pruning phase. However, these slight changes bring significant improvements. Additionally, we observe negative coefficients in some positions, indicating that the knowledge from certain experts may not benefit the downstream task.

Figure: (a) Visualization of the searched expert merging matrix.
Refer to caption: x3.png

## Appendix E Prompt

We list the prompt we used for each dataset in Tab. 14. We follow the default prompt in the Opencompass codebase [].

**Table 14: Prompts for all datasets.**
| Dataset | Prompt |
| --- | --- |
| WIC | Sentence 1: <sentence1>$\backslash$nSentence 2: <sentence2> |
|  | Are ’<word>’ in the above two sentences the same?$\backslash$nA. Yes$\backslash$nB. No$\backslash$nAnswer: |
|  | [A/B](run:yesno) |
| WSC | Passage: <text>$\backslash$n |
|  | Does the pronoun # <span2> # refer to * <span1> *?$\backslash$nA. Yes$\backslash$nB. No$\backslash$nAnswer: |
|  | [A/B](run:yesno) |
| RTE | <premise>$\backslash$n<hypothesis>$\backslash$n |
|  | Is the sentence below entailed by the sentence above?$\backslash$nA. Yes$\backslash$nB. No$\backslash$nAnswer: |
|  | [A/B](run:yesno) |
| BoolQ | <passage>$\backslash$n |
|  | Question: question$\backslash$nA. Yes$\backslash$nB. No$\backslash$nAnswer: |
|  | [A/B](run:yesno) |
| CB | <premise>$\backslash$n<hypothesis>$\backslash$n |
|  | What is the relation between the two sentences?$\backslash$nA. Contradiction$\backslash$nB. Entailment$\backslash$nC. Neutral$\backslash$nAnswer: |
|  | [A/B/C](run:yesno) |
| ReCoRD | Passage: <text>$\backslash$nResult: <question>$\backslash$n |
|  | Question: What entity does ____ refer to in the result? Give me the entity name: |
| DROP | $\backslash$n$\backslash$nText: <prompt>$\backslash$n |
|  | Question: <question>$\backslash$nAnswer: |
| SQuAD | <context>$\backslash$nAccording to the above passage, answer the following question. |
|  | If it is impossible to answer according to the passage, answer ‘impossible to answer‘:$\backslash$n |
|  | Question: <question> |

## Appendix F Examples of model outputs, and metric evaluations

In this section, we provide examples of different approaches’ output in Fig. 9, Fig. 10 and Fig. 11.

Figure: Figure 9: Example of Mixtral 8 $\times$ 7B-Instruct on SQuAD.

Figure: Figure 10: Example of Mixtral 8 $\times$ 7B-Instruct on SQuAD. * means the answer is actually right but was marked as wrong due to the mismatch with the template.

Figure: Figure 11: Example of Mixtral 8 $\times$ 7B-Instruct on SQuAD. * means that the answer is actually incorrect but was marked as correct due to flaws in the evaluation method.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Why is the full model performance significantly higher in this paper compared to the NAEE paper? For example, in Table 2, the full model (Mixtral 8×7B-Instruct) achieves 85.8 on ARC-c with 8 experts, while in the NAEE paper's Table 1, the performance is only 62.20.

2. Can the proposed method still work for out-of-distribution cases when there is only one dataset available for validation?

3. What does the performance vs. dataset size curve look like during the search process? What is the rationale for choosing specific dataset sizes for validation?

4. How is the performance of the proposed method compared to dynamic MoE methods?
soundness: 2
strengths: SMoE can achieve faster inference while maintaining performance. To search for the subset of experts, this paper proposes a gradient-free approach based on evolutionary strategy. By using two phases expert pruning and expert merging, the method achieves high sparsity.
summary: This paper works on the topic of pruning experts in SMoE models. They propose a gradient-free evolutionary strategy named EEP. In EEP, there are two matrices named expert merging matrix and experts mapping matrix, used to search for pruning configurations. The goal is to achieve sparsity while reaching good performance on downstream tasks. The paper shows the performance of the proposed method on four different model sizes and uses more than 10 datasets.
weaknesses: 1. While the method is gradient-free, it does not necessarily imply a low search cost. The evolutionary strategy used requires multiple iterations and epochs, and the search cost is highly dependent on the settings for iterations, epochs, and the data size used for calculating the validation metric. To better illustrate the efficiency of the method, I suggest the authors provide concrete runtime numbers for different dataset sizes, comparing them to gradient-based alternatives. This would help clarify whether the method can be effectively extended to more complex datasets.

2. The method heavily relies on evaluation results from selected data to conduct the search process. While Table 15 shows in-distribution results for two dataset sizes, it is unclear whether similar trends hold for out-of-distribution cases. A comprehensive study on the influence of dataset size is recommended. Specifically, it would be beneficial to include a plot showing the performance vs. dataset size across a range of sizes for both in-distribution and out-of-distribution settings. Additionally, the authors should explain the counterintuitive result where a smaller dataset size provides better performance than a larger dataset size.

3. For out-of-distribution evaluation, 7 datasets from MMLU are used for validation. It would be helpful to know whether the number of validation datasets affects performance. To address this, I recommend conducting an ablation study that varies the number of validation datasets (e.g., from 1 to 7) and reporting how this impacts performance. This would offer insight into the method’s robustness in extreme cases where only a single dataset is available for validation.

4. The writing of the paper could be improved, as the key ideas are not easy to follow. A significant portion of the text is devoted to background and preliminaries, such as the description of the self-attention mechanism in Section 3, which is well-known and could be condensed or moved to the appendix. This would create more space to emphasize the main algorithm and core contributions in the main paper. I suggest reorganizing the structure to focus on the core algorithm earlier in the text for better clarity and accessibility.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Could the authors elaborate on how the choice of group numbers and search iterations affects the final performance? A more systematic analysis of these hyperparameters would be valuable for implementation.
soundness: 3
strengths: 1. The paper is exceptionally well-written and organized, with comprehensive experimental validation across multiple model scales and diverse datasets. The methodology is clearly presented with thorough ablation studies and reproducible code.
2. The proposed two-phase EEP framework presents a novel and effective paradigm for model compression. The authors provide insightful analysis of router network behavior that justifies their approach.
3. The method achieves remarkable results while only requiring inference capabilities, with pruned models sometimes even outperforming the original uncompressed models.
summary: This paper introduces Efficient Expert Pruning (EEP), a gradient-free evolutionary strategy for optimizing Sparse Mixture-of-Experts (SMoE) language models. The work addresses the challenge of reducing model size and computation costs while maintaining performance. The authors propose a two-phase approach combining expert pruning and expert merging, which can be executed using only model inference capabilities without requiring gradient computation or extensive GPU resources. The method is implemented through router mapping and expert merging matrices, enabling evolutionary search for optimal pruning configurations.
weaknesses: 1. The analysis of why performance sometimes improves after pruning in Sec. 5.6. remains somewhat speculative, lacking rigorous theoretical foundations and comprehensive empirical validation.
2. The performance gap between global setting and task-specific setting in OOD experiments suggests that EEP's impressive results might be more dependent on task-specific optimization than claimed, potentially limiting its general applicability across different tasks.
3. The search process of EEP is computationally expensive and time-consuming, particularly for large datasets. The paper does not address this efficiency bottleneck.

---

confidence: 2
contribution: 3
initial_rating: 3
presentation: 3
questions: see weakness
soundness: 2
strengths: (1) The writing is clear and easy to follow.  

(2) The proposed method is interesting, suggesting that evolutionary strategies can effectively optimize parameter combinations for model pruning.   

(3) The insights presented are valuable, demonstrating that MoE models contain redundancy and hold significant potential for pruning while maintaining performance and improving efficiency.
summary: This paper addresses the challenge of pruning in sparse mixture of experts (SMoE) models. The authors propose a gradient-free evolutionary strategy called Efficient Expert Pruning (EEP) to improve the pruning process for experts within these models. Experimental results on diverse models and datasets demonstrate the effectiveness of this approach.
weaknesses: (1) Lack of Reliable Evaluations: I find it puzzling that Mixtral-8x22B's average performance improves despite being pruned from 8 experts down to 2. This raises questions about the robustness of the evaluation methods, especially given the relatively small datasets used in this study. In contrast, NAEE—a key baseline—evaluates on the comprehensive MMLU dataset, which could serve as a more reliable benchmark.

(2) In Table 6, even after pruning from 8 experts down to 4 and then 2, the speedup is small. What factors might explain this limited increase in speed?

(3) The method’s efficiency is largely limited by its computational demands and the dataset used. The pruning dataset is relatively small, yet the process still requires significant time—up to several hours.

(4) Are the training and evaluation samples drawn from the same distribution? This could make sense as a task-agnostic pruning approach. However, given the long pruning time even with limited samples, why not fine-tune a roughly pruned model instead?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Section 5.1, the authors state that the dataset is randomly divided into training and test set, with the training set used for evolutionary search and the test set used for evaluation. How exactly is this division performed? Table 14 seems to contain some data on the trainset size, but it is not comprehensive enough. 
2. The authors used 50 MMLU datasets as the training set and 7 as the test set, utilizing a large number of datasets as the training set to test accuracy on out-of-distribution datasets. I would like to know the total search time EEP took in this case, and whether using fewer training sets would impact the out-of-distribution accuracy.
3. How does the accuracy of the EEP pruned model obtained from one dataset perform on other datasets? For example, if BoolQ is used as the training set, what is the accuracy of the pruned model on other validation sets?
soundness: 3
strengths: 1. This paper is well-written, well-organized including problem formulation, methodological exposition, and experimental setup, making it accessible to readers with a background in the field. Besides, the implementation details and code are provided for reproducing.
2. The EEP method employs a gradient-free evolutionary search strategy for expert pruning and merging in the SMoE model, which is a relatively novel method.
3. This paper conducts extensive experiments on multiple datasets and models to validate the effectiveness of the method, demonstrating significant improvements over existing baselines, which greatly enhances the practicality of SMoE pruning.
summary: This paper proposes Efficient Expert Pruning (EEP), a gradient-free evolutionary strategy to optimize the pruning and merging of experts within Sparse Mixture-of-Experts (SMoE) models. EEP aims to enhance performance while saving GPU memory and reducing inference costs by minimizing both the total number of experts and the number of active experts during inference. This is achieved through an evolutionary search process that optimizes two matrices: a router mapping matrix for expert selection and an expert merging matrix to retain pruned experts' knowledge. EEP is evaluated across various downstream tasks and shows considerable improvement over baselines in terms of memory efficiency, inference speed, and overall performance.
weaknesses: 1. In the NAEE [1] method, the authors use the C4 dataset as the calibration dataset and evaluate the zero-shot accuracy of the pruned model, which is a common experimental setup in the field of LLM compression. The authors should conduct similar experiments and provide data in this regard to allow for a more comprehensive assessment of the effectiveness of the EEP method.
2. I am concerned about the search cost of EEP. The total time spent on EEP for each dataset is several hours, and this is just for the smallest Mixtral 8x7B Instruct model. For larger datasets and larger models, EEP will take even longer, which is often unacceptable.
3. The out-of-distribution accuracy of EEP is much lower than that of NAEE [1] on many datasets (Tables 5, 12 and 13), so I am concerned about the generalization ability of EEP.
4. Some notations are confusing.
    - The authors use bold uppercase symbols to represent matrices. In line 91, $W_{RM}$ and $W_{EM}$ should be bolded to ensure consistency with the notation used in Section 4.2.
    - The symbol $\boldsymbol{W}_G$ in Equation 2 is not explained. I suspect it refers to the router network $\boldsymbol{W}_R$ in line 179.

[1] Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. ACL 2024 Main.

Meta-review: The deployment of Sparse Mixture-of-Experts models remains challenging due to their large parameter counts and GPU memory demands. To address this, this paper introduces Efficient Expert Pruning, a gradient-free evolutionary strategy that prunes and merges experts in SMoE models based solely on model inference, reducing parameters and memory usage while maintaining or improving performance. Experiments on multiple datasets and models, including Mixtral and Qwen, show that EEP achieves sparsity and inference speedups, outperforming baselines and enabling more efficient SMoE deployment.

Three reviewers have slightly positive feedback and most reviewers acknowledge the paper presentation & methods' efficacy, especially impressed by the high sparsity of compressing Mixtral MoE. However, it is well-known that Mixtral MoE is highly redundant, also for Qwen MoE. My major concern is the generalization ability of the proposed approach to other types of MoEs which have less redundancy due to improved optimization and designs. For example, deepseek and modularformer. Also, the discussions with related MoE merging papers are encouraged to be included. Overall, we recommend a rejection.

Decision: Reject