Ban&Pick: Ehancing Performance and Efficiency of MoE-LLMs via Smarter Routing

## Abstract

Abstract Sparse Mixture-of-Experts (MoE) has become a key architecture for scaling large language models (LLMs) efficiently.
Recent fine‑grained MoE designs introduce hundreds of experts per layer, with multiple experts activated per token, enabling stronger specialization.
However, during pre‑training, routers are optimized mainly for stability and robustness: they converge prematurely and enforce balanced usage, limiting the full potential of model performance and efficiency at inference.
In this work, we uncover two overlooked issues: (i) a few highly influential experts are underutilized due to premature and balanced routing decisions; and (ii) enforcing a fixed number of active experts per token introduces substantial redundancy.
Instead of retraining models or redesigning MoE architectures, we introduce Ban&Pick , a post-training, plug-and-play strategy for smarter routing. Pick discovers and reinforces key experts—a small group with outsized impact on performance—leading to notable accuracy gains across domains. Ban further dynamically prunes redundant experts based on layer and token sensitivity, delivering faster inference with minimal accuracy loss.
Experiments on fine-grained MoE-LLMs (DeepSeek, Qwen3) across math, code, and general reasoning benchmarks demonstrate that Ban&Pick delivers free performance gains and inference acceleration without retraining or architectural changes. For instance, on Qwen3-30B-A3B, it improves accuracy from 80.67 to 84.66 on AIME2024 and from 65.66 to 68.18 on GPQA-Diamond, while accelerating inference by 1.25× under the vLLM .

## 1 Introduction

Large language models (LLMs) have achieved remarkable progress (DBLP:conf/iclr/ZhouWLSLQLJSZ024,), yet the ever-growing demand for higher capacity presents significant challenges for efficiency. As a promising solution, Mixture-of-Experts (MoE) architectures scale parameter count without proportionally increasing computation, by activating a small subset of experts for each input (artetxe-etal-2022-efficient,).

Recent advances have pushed MoE towards a fine-grained design (dai-etal-2024-deepseekmoe,), where each layer contains a large number of relatively small experts. For example, DeepSeek-v2.5 (deepseekai2024deepseekv2strongeconomicalefficient,) employs 160 experts per layer, while Qwen3-MoE (yang2025qwen3technicalreport,) adopts 128 experts per layer. Unlike coarse-grained MoE (jiang2024mixtral,), which typically consists of a few large experts derived from the same base model, fine-grained MoE initializes each expert independently and randomly.
This design promotes specialization: experts gradually develop distinct skills, excelling in areas such as math, code, or general reasoning. Such strong differentiation suggests that fine-grained MoE has the potential to leverage expert diversity more effectively.

However, we argue that a critical but overlooked phenomenon exists: the routing strategies learned during pre-training cannot fully exploit this specialization. Specifically, we empirically discover that current routing strategies inadvertently limit the potential to leverage the most impactful experts while simultaneously introducing redundancy where many selected experts contribute minimally to the final output.
Instead of attempting to redesign training objectives or expert architectures, we take a post-training perspective, aiming to unlock this untapped potential efficiently and effectively.

Delving into this question, we surprisingly find that within the most frequently selected experts of fine‑grained MoE models, a small fraction plays an outsized role: simply forcing these experts to be chosen for every input can noticeably boost accuracy on certain datasets. Motivated by this striking observation, we conduct an empirical analysis of expert usage across domains and discover that each domain consistently relies on its own set of frequently selected experts, which we refer to as domain‑specialized experts.
Yet, a closer examination further shows that only a tight subset of these experts has a decisive impact on the final logits distribution, while the influence of the others remains marginal. We term these most critical ones key experts. Our Pick module is designed to reinforce their influence during routing,
getting performance gains by emphasizing these key experts.

In parallel, we aim to alleviate redundancy in expert activation. Under the constraint that a fixed number of experts must be selected for each token, many activated experts contribute little to the final output, reflecting the redundancy introduced by less relevant experts. Our Ban module jointly considers layer sensitivity and token sensitivity to assess the redundancy in expert selection, and applies dynamic pruning of experts during inference. In this way, Ban reduces unnecessary computation and achieves significant inference acceleration with only minor accuracy loss.

Attempting to bridge the gap between routing strategies learned during pre-training—which prioritizes stability and balance—and optimal inference-time expert utilization, Pick and Ban empower truly impactful experts to contribute more while reducing participation from low-contribution experts:

- •
Pick amplifies excellence by identifying experts that demonstrate superior performance on specific domains and strategically increasing their influence during inference, maximizing effectiveness.
- •
Ban eliminates inefficiency by dynamically excluding experts that contribute minimally during inference, reducing computational costs while preserving essential capabilities.

Together, we introduce Ban&Pick, a unified framework that optimizes expert utilization in inference to improve both the accuracy and efficiency of fine-grained MoE without retraining or architectural changes. Evaluated on five widely used datasets spanning math, code, and general reasoning, the results demonstrate that on the Qwen3 series models, applying Pick alone brings an average 2.83% performance improvement. By further incorporating Ban, the joint Ban&Pick strategy achieves an average 1.41% improvement while simultaneously accelerating inference by 1.25× under the vLLM.

## 2 Related Works

#### Enhancing Expert Utilization in MoE.

Most existing work attempts to improve expert utilization efficiency in MoE models by redesigning training objectives or expert architectures during pre-training (yan2025tcmoe,; huang-etal-2024-harder,; jin2024moeacceleratingmixtureofexpertsmethods,). However, while these approaches are insightful, they often sacrifice training stability for flexibility. Recently, (wang2025expertsneedsteeringthinking,) proposes a post-training method designed for MoE models that employ the <think> token, enhancing reasoning by amplifying the weights of Cognitive Experts.

#### Dynamic Expert Pruning of MoE.

Dynamic expert pruning has emerged as a key technique for accelerating MoE inference. (lu-etal-2024-experts,) proposed a pruning strategy that simply skips experts with less weights.
Building on this, (huang2024mc,) introduced a critical token protection mechanism.
However, the substantial sensitivity differences across layers of MoE models and across tokens in long chain‑of‑thought reasoning are often overlooked, leaving room for further optimization.

## 3 Empirical Findings on Routing Limitations in Fine-grained MoE

Figure: Figure 1: The intuition and empirical findings on expert utilization. Left (Pick): forcibly activating key experts (e.g., E43) improves accuracy on Winogrande (sakaguchi2021winogrande,). Right (Ban): reducing experts from 6 to 5 cuts 16.7% computation without accuracy loss.
Refer to caption: x1.png

### 3.1 Insights from Pre-training Patterns

Recent evidence from OLMoE (muennighoff2025olmoeopenmixtureofexpertslanguage,) and OpenMoE (xue2024openmoe,) suggests that during pre-training, routers converge much earlier than the experts. In particular, OLMoE reports that after only 1% of pre-training, around 60% of the routing decisions have already saturated, while OpenMoE finds that by 40% of pre-training, about 80% of the routing has stabilized. Such early fixation implies that routing decisions are largely determined before the experts have sufficiently matured. Moreover, the widely used balancing loss (lepikhin2020gshardscalinggiantmodels,), while effective at preventing collapse to just a few experts, may also discourage tokens from concentrating on the most impactful experts. This implies that the current pre-training paradigm for MoE makes a trade-off: in exchange for training stability, it locks routers into decisions before experts have fully matured. Combined with balancing constraints, this underutilizes high-potential experts and inflats the role of those that contribute little at inference. This presents an opportunity from a post-training perspective:

Can we recover this lost potential—amplifying the influence of the most impactful experts while trimming away redundancy—to both boost performance and accelerate inference?

### 3.2 Expert Utilization Inefficiencies and Optimization Potential

To validate the inefficiencies in expert untilization and better understand the optimization potential, we conducted two simple yet revealing experiments with the Deepseek‑V2‑Lite‑Chat model (activates 6 experts out of 64). Figure [1](https://arxiv.org/html/2509.06346v2#S3.F1) demonstrates how we adjust expert utilization and its effects.

(1) Underutilization of the most impactful experts: we examined expert selection frequency on the C4 (dodge2021documentinglargewebtextcorpora,) calibration set, identified the top-3 most frequently selected experts in each layer, and then conducted a grid search: each of these experts was manually added in addition to the original routing (E43 in Figure [1](https://arxiv.org/html/2509.06346v2#S3.F1)), and the resulting accuracy on Winogrande benchmark was recorded. The left part highlights a few experts contribute disproportionately: enhancing Layer‑1 Expert‑43 improved accuracy by +1.81%, L10E33 by +1.65%, and L25E11 by +1.26%. When these three were jointly enhanced, accuracy increased by +2.72% (71.67% $\rightarrow$ 74.39%), while most other high-frequency experts had negligible or noisy effects, indicating that certain impactful experts are insufficiently leveraged in the original routing strategy. We refer to these three experts as key experts.

(2) Computational waste from low-contribution experts: we directly reduced the number of selected experts per token from 6 to 5 and measured the resulting accuracy. Remarkably, as shown in the right side of Figure [1](https://arxiv.org/html/2509.06346v2#S3.F1), this pruning reduced computation by 16.7% and surprisingly preserved full accuracy on Winogrande (71.67%), demonstrating redundancy in the original expert utilization.

Together, these experiments validate the inefficiencies in expert utilization predicted by pre-training analysis, and support the intuition of our proposed methods—Pick and Ban, respectively, which aim to optimize expert selection beyond pre-training constraints for better inference-time performance. This empirical study motivates us to design a unified framework that can pick key experts while banning redundant ones, aiming at improving both accuracy and efficiency.

## 4 Pick

In this section, we begin with an analysis of expert specialization in fine‑grained MoE models to verify the existence of domain-specialized experts, which forms the foundation for key experts to effectively function. Building on this analysis, we then introduce our Pick method in two steps: (i) identifying key experts from the set of domain‑specialized experts, and (ii) designing strategies to enhance these key experts so that their impact on model performance can be effectively amplified.

### 4.1 Expert Specialization in Fine-Grained MoE

Figure: Figure 2: Expert specialization in fine-grained MoE (Qwen3‑30B‑A3B). Left: expert usage frequency across three tasks (math, general, code). Right: token word clouds for 3 high-frequency experts.
Refer to caption: x2.png

First, we take a closer look at the degree of expert specialization in fine‑grained MoE models. Specifically, we analyze the Qwen3‑30B‑A3B model and record expert usage frequencies on task-related benchmarks: math tasks (GSM8K + Math-500), code tasks (MBPP (austin2021program,) + Livecodebench), and a general task (GPQA-Diamond). The statistics cover all tokens in both the prefill and decode stages. For each type of task, we plot the per‑layer expert usage frequency and show 3 representative layers in Figure [2](https://arxiv.org/html/2509.06346v2#S4.F2) (complete results in [Appendix A](https://arxiv.org/html/2509.06346v2#A1)).

Our analysis reveals that in each layer there exist experts whose usage frequency is significantly higher than average–often up to 6 times higher. Moreover, in most layers these high‑frequency experts vary across tasks.
For instance, L16E95 is frequently activated only on math tasks, whereas L29E91 is almost exclusively activated on code tasks with an extremely high activation rate of 66.9%. In contrast, a small number of experts such as L4E48 remain highly active across all categories of tasks.

These task‑dependent high‑frequency experts suggest the existence of domain‑specialized experts. To validate this hypothesis, we further examine the functional roles of these experts by analyzing which tokens are most often routed to them. Taking the three aforementioned experts as examples, we visualize the tokens they frequently process using word clouds (after removing empty or semantically meaningless tokens for readability). The results show that the math‑specialized expert L16E95 is strongly associated with digits, operators, and math terminology; the code‑specialized expert L29E91 is associated with common programming tokens; and the general expert L4E48 is closely tied to branch tokens (zhong2025hybrimoehybridcpugpuscheduling,), which play a critical role in chain‑of‑thought across diverse tasks.

These observations provide empirical evidence that expert specialization in fine‑grained MoE models is highly pronounced. Different tasks rely heavily on different domain‑specialized experts, while a small subset of general experts remain consistently important across domains.

### 4.2 Picking Key Experts from Domain‑Specialized Experts

As shown previously, almost every layer contains domain‑specialized experts that are activated with notably high frequency. The key issue, however, is whether all of these experts contribute equally to model’s performance, or only a small subset is truly essential. To investigate this, we evaluate the importance of each expert by measuring the shift in the logits distribution when it is removed.

Concretely, taking math tasks as an example, we use the domain‑specialized experts identified in the previous analysis as candidate experts in each layer. We then run the model on 1000 randomly sampled instances from the MathQA (amini-etal-2019-mathqa,) dataset. For each sample, we record the output logits of the original model, and then repeat the inference while pruning one candidate expert at a time. The perturbation of the output distribution is quantified using the KL divergence between the probability distributions of the Top‑1000 tokens from the original and pruned models (see [Appendix B](https://arxiv.org/html/2509.06346v2#A2) for implementation details).

The results are summarized in Figure [4](https://arxiv.org/html/2509.06346v2#S4.F4). Surprisingly, despite their high activation frequency, most domain‑specialized experts cause only negligible shifts in the output distribution when pruned. In contrast, a small number of experts stand out, with three of them inducing particularly large changes. Among these, expert L9E18 is the most striking, leading to a divergence of about $0.16$. These findings suggest that only a few domain‑specialized experts truly have a decisive influence on the model’s behavior, and we hypothesize that they are the key experts we aim to identify.

To test this hypothesis, we construct a failure set consisting of the Math‑500 (lightman2023lets,) problems that the base model originally answers incorrectly (24 in total). For each candidate expert, we forcibly include it during routing and then re‑evaluate accuracy on this failure set. As shown in Figure [4](https://arxiv.org/html/2509.06346v2#S4.F4) (right vertical axis), only those experts that induce substantial distribution shifts also yield meaningful accuracy gains when enhanced, whereas others contribute little to no improvement. In particular, expert L9E18, which caused the largest divergence, also increases the number of correct answers by 5 out of 24 when included. This provides strong evidence of the close correspondence between distribution shifts and functional importance, and offers a principled way to distinguish key experts from the many domain‑specialized experts.

### 4.3 Enhancing Key Experts

Figure: Figure 3: Impact of pruning (measured by KL divergence, left axis) and enhancing (by accuracy gain, right axis) for domain‑specialized experts.
Refer to caption: x3.png

After identifying key experts, the remaining challenge is to determine how to leverage them most effectively. We take math tasks as a running example and design five enhancement strategies:

- A.
Forced Addition: Whenever a key expert is not selected, it is added alongside the chosen experts.
- B.
Forced Replacement: Similar to forced addition, but the key expert replaces the selected expert with the lowest routing weight, maintaining the total number of selected experts.
- C.
Range-based Addition: If a key expert is not selected (not in the top-$k$ candidates) but appears within the router’s top-$2k$ candidates, it is forcibly added.
- D.
Range-based Replacement: Under the same condition as range-based addition, the key expert replaces the selected expert with the lowest routing weight.
- E.
Logit Bias: Before routing, a fixed bias is added to the routing logits of key experts. In our implementation, the bias is set to $20\%$ of the average weight among the top-$k$ selected experts.

We evaluate these methods on AIME2024, AIME2025, and Math‑500. For the two AIME datasets, results are averaged over five runs to mitigate small‑sample variance. Figure [4](https://arxiv.org/html/2509.06346v2#S4.F4) summarizes the outcomes. Overall, strategy A brings consistent but modest gains. Strategy B, which forces replacement regardless of probability, is unstable and even reduces accuracy on AIME2024 and Math‑500, indicating that overly aggressive intervention can be harmful. By contrast, strategies C and D—enhancing only when the key expert is already close to the router’s top candidates—deliver the strongest improvements across all benchmarks. The difference between C and D is minor, but since D avoids expanding the number of active experts, it is preferable in practice and used in our experiments. Strategy E shows less stable performance, as the bias magnitude is sensitive and difficult to tune.

## 5 Ban

Our dynamic expert pruning method is motivated by two key observations:
1) Different layers in an MoE model exhibit highly diverse sensitivity to expert pruning and thus should be treated differently.
2) During reasoning, the token-wise routing distributions also vary significantly: when most of the routing weight is concentrated on a few top experts, expert pruning causes minimal loss; in contrast, when the weights are more evenly distributed, expert pruning becomes more harmful.

In the following, we first use Qwen3-30B-A3B as an example to illustrate the differences in layer sensitivity and token sensitivity, and then introduce our proposed dynamic pruning method.

#### Layer Sensitivity.

We measure layer-wise pruning sensitivity by reducing the number of selected experts from $k=8$ to $k=3$ in each layer, and computing the Top‑1000 KL divergence between the output logits of the original and pruned models on MathQA calibration set. The results (Figure [5](https://arxiv.org/html/2509.06346v2#S5.F5)a) reveal substantial differences across layers: while some layers, especially at the begin and end of the model, are highly sensitive to pruning, many middle layers are much more robust. This large variance highlights the necessity of treating different layers differently when designing pruning strategies.

#### Token Sensitivity.

We further analyze token‑level pruning sensitivity in long reasoning processes by examining how routing weights are distributed across experts. To quantify this, we compute the ratio between the cumulative routing weights of the top‑3 and top‑8 experts, averaged across all layers. Figure [5](https://arxiv.org/html/2509.06346v2#S5.F5)b plots these ratios for the first 200 generated tokens in a reasoning process from one GPQA‑Diamond sample. The results reveal substantial variability across tokens: some tokens concentrate most of their weights on a few experts and are thus robust to expert pruning, whereas others distribute weights more evenly and are therefore more sensitive.

Figure: (a) Layer-wise pruning sensitivity.
Refer to caption: x5.png

#### Dynamic pruning.

Before combining layer‑ and token‑level information, we normalize their sensitivity scores to the range $[0,1]$.
Here, $W_{l}$ denotes the KL divergence of layer $l$ when reducing expert selection from $k=8$ to $k=3$.
$W_{\min}$ and $W_{\max}$ are defined as the divergences of the least‑sensitive and most‑sensitive layers, respectively.
For tokens, $R_{i}$ represents the top‑3/top‑8 routing weight ratio of token $i$ at a specific layer during inference, with $R_{\min}$ and $R_{\max}$ similarly estimated from the calibration set.
The normalized sensitivity scores are then defined as:

$$ $L^{\prime}_{l}=(W_{l}-W_{\min})/(W_{\max}-W_{\min}),\qquad T^{\prime}_{i}=(R_{\max}-R_{i})/(R_{\max}-R_{\min}).$ $$

The combined sensitivity score for token $i$ at layer $l$ is then:

$$ $S_{i,l}=\lambda\cdot\tfrac{1}{2}\big(L^{\prime}_{l}+T^{\prime}_{i}\big),\quad\lambda\in(0,1)$ $$

where we simply average layer- and token-level sensitivities to balance their contributions, and use $\lambda$ to control the aggressiveness of pruning.
The dynamic number of selected experts is determined as:

$$ $K_{i,l}=\left\lfloor K_{\min}+(K_{\text{base}}-K_{\min})\cdot S_{i,l}\right\rceil.$ $$

Here $K_{\text{base}}=8$ is the default number of experts, and $K_{\min}=3$ serves as a safeguard to prevent selecting too few experts: from empirical inspection, we found that reducing the number of selected experts below this threshold leads to a complete collapse of the model. Details of the experimental validation for the setting of $K_{\min}$ and the choice of $\lambda$ are provided in the [Appendix C](https://arxiv.org/html/2509.06346v2#A3).

## 6 Experiments

In this section, we first introduce the experimental setup, including models, datasets, and implementation details. We then separately compare our proposed Pick and Ban approaches with existing methods, highlighting their respective strengths. After that, we demonstrate how combining Ban&Pick achieves both improved performance and faster inference. Finally, we examine the influence of key experts on tasks from other domains, offering insights for future research directions.

### 6.1 Experiment Setup

#### Models and Datasets

We conduct experiments on four fine-grained MoE models from two different model families, covering a wide range of parameter scales: DeepSeek-V2-Lite-Chat (16B), DeepSeek-V2.5-1210 (236B), Qwen3-30B-A3B, and Qwen3-235B-A22B. Our evaluation spans three categories of tasks using five commonly adopted datasets: AIME2024 (patel2024aimeaioptimizationmultiple,) and Math-500 for mathematical reasoning, GPQA-Diamond (rein2023gpqagraduatelevelgoogleproofqa,) for general knowledge reasoning, and LiveCodeBench (V1) (jain2024livecodebench,) with HumanEval+ (evalplus,) for code generation. For DeepSeek-V2-Lite-Chat, we replace AIME2024 with GSM8K, as AIME2024 is overly difficult for this smaller model. We use OpenCompass (2023opencompass,) as evaluation toolkit. The detailed generation hyperparameters are provided in the [Appendix D](https://arxiv.org/html/2509.06346v2#A4).

**Table 1: Comparison of Pick against three baselines across four models. Reported accuracies cover multiple benchmarks, with GSM8K only for DeepSeek-V2-Lite-Chat.**
| Models | Methods | GSM8K/AIME24 | Math500 | GPQA | Livecodebench | HumanEval+ |
| --- | --- | --- | --- | --- | --- | --- |
| Deepseek-V2-Lite-Chat | Baseline | 66.34 | 26.40 | 24.75 | 9.25 | 50.61 |
| Dynamic | 66.56 | 26.80 | 24.75 | 9.00 | 50.61 |  |
| Tips | 66.87 | 26.60 | 25.25 | 9.50 | 51.22 |  |
| Pick | 67.48 | 28.00 | 25.76 | 10.50 | 51.83 |  |
| Deepseek-v2.5-1210 | Baseline | 23.33 | 81.00 | 47.47 | 60.75 | 83.54 |
| Dynamic | 24.00 | 81.20 | 46.97 | 61.00 | 83.54 |  |
| Tips | 24.67 | 80.60 | 47.98 | 61.25 | 84.15 |  |
| Pick | 27.34 | 82.80 | 49.49 | 62.75 | 84.76 |  |
| Qwen3-30B-A3B | Baseline | 80.67 | 95.20 | 65.66 | 91.75 | 71.34 |
| Dynamic | 82.66 | 94.80 | 66.16 | 92.00 | 71.95 |  |
| Rice | 83.33 | 95.60 | 67.68 | 93.50 | 72.56 |  |
| Pick | 85.33 | 96.60 | 69.19 | 94.25 | 75.61 |  |
| Qwen3-235B-A22B | Baseline | 84.67 | 96.00 | 71.21 | 92.75 | 78.05 |
| Dynamic | 85.33 | 96.00 | 71.72 | 93.00 | 78.66 |  |
| Rice | 86.67 | 96.20 | 72.22 | 93.00 | 79.88 |  |
| Pick | 88.00 | 96.80 | 74.24 | 94.50 | 81.10 |  |

#### Implementation Details

For the Pick method, we follow the procedures described in
[Section 4](https://arxiv.org/html/2509.06346v2#S4) to identify key experts for each of the three task categories–math, code and general reasoning–and apply targeted enhancements to them. We then evaluate the resulting models on their corresponding task types.
For the Ban method, we set $K_{\min}=3$ and $\lambda=0.7$ for all four models, aiming to achieve a balanced trade-off between accuracy and speedup (further discussed in  [Appendix C](https://arxiv.org/html/2509.06346v2#A3)). When combining Pick and Ban, we note that in layers containing key experts, Ban has already pruned redundant experts, so directly applying the range-based replacement strategy (strategy D) leads to noticeable performance degradation. To address this, we instead adopt the range-based addition strategy (strategy C).

### 6.2 Results of Pick

We compare our proposed Pick method against three representative baselines, covering both routing-strategy adjustments and test-time scaling approaches (see [Appendix E](https://arxiv.org/html/2509.06346v2#A5) for reproducibility details):
(1) Dynamic Routing.
(huang-etal-2024-harder,) adaptively allocates experts according to routing weights, assigning more experts to harder tokens and fewer to easier ones, realized by accumulating experts until a threshold is reached. We set thresholds $\{0.7,0.8,0.9\}$ and report the best result.
(2) Tip.
(wang2025thoughtsplaceunderthinkingo1like,) addresses the issue of *underthinking*, where models abandon reasoning paths too early. It penalizes tokens that typically indicate switching paths, thereby encouraging the model to continue deepening its current line of reasoning.
(3) Rice.
(wang2025expertsneedsteeringthinking,) leverages explicit <think> tokens to identify *cognitive experts* that specialize in reasoning. During inference, the routing weights of these experts are multiplied by a constant factor to amplify their influence. Since RICE relies on explicit <think> tokens and is therefore inapplicable to the two DeepSeek models, and it has reported better results than Tip on Qwen3 series, we adopt Tip as the baseline for DeepSeek and RICE as the baseline for Qwen3 in our experiments.

As shown in Table [1](https://arxiv.org/html/2509.06346v2#S6.T1), Pick achieves consistent and notable improvements across four models with different parameter scales on five datasets, outperforming all three baselines. The gains are especially pronounced on the Qwen3 series, which we attribute to their stronger expert specialization. Remarkably, on the large-scale Qwen3-235B-A22B, whose performance is already comparable to leading closed-source models, Pick further boosts accuracy by 3.33%, 3.03%, and 3.05% on AIME2024, GPQA, and HumanEval+, respectively. These results highlight the potential of leveraging domain-key experts to further improve MoE model accuracy on diverse tasks.
Meanwhile, on DeepSeek-v2.5-1210, Pick still achieves an average improvement of 1.97% across the five datasets, demonstrating strong generalization capability.

### 6.3 Results of Ban

**Table 2: Comparison of Ban against DES and ODP across four models. We report average experts per token (Avg-Topk), accuracy, and inference speedup under the vLLM (0.9.1) backend.**
| Models | Methods | Avg-Topk | GSM8K/AIME24 | Math500 | GPQA | Livecodebench | HumanEval+ | Speedup |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Deepseek-V2-Lite-Chat | Baseline | 6.00 | 66.34 | 26.40 | 24.75 | 9.25 | 50.61 | 1.00 |
| DES | 4.18 | 63.76 | 21.40 | 22.22 | 7.25 | 45.73 | 1.20 |  |
| ODP | 4.22 | 64.76 | 23.20 | 23.23 | 7.75 | 46.95 | 1.19 |  |
| Ban | 4.02 | 66.11 | 25.00 | 24.24 | 8.50 | 48.17 | 1.22 |  |
| Deepseek-v2.5-1210 | Baseline | 6.00 | 23.33 | 81.00 | 47.47 | 60.75 | 83.54 | 1.00 |
| DES | 4.21 | 22.66 | 80.20 | 39.90 | 58.00 | 79.27 | 1.21 |  |
| ODP | 4.29 | 23.33 | 80.20 | 43.43 | 59.25 | 81.71 | 1.20 |  |
| Ban | 3.97 | 23.33 | 80.60 | 45.96 | 60.25 | 82.93 | 1.24 |  |
| Qwen3-30B-A3B | Baseline | 8.00 | 80.67 | 95.20 | 65.66 | 91.75 | 71.34 | 1.00 |
| DES | 4.94 | 73.33 | 92.60 | 56.06 | 84.50 | 65.24 | 1.25 |  |
| ODP | 4.98 | 74.67 | 93.20 | 60.10 | 86.00 | 67.07 | 1.25 |  |
| Ban | 4.82 | 80.00 | 94.60 | 64.65 | 90.00 | 70.12 | 1.25 |  |
| Qwen3-235B-A22B | Baseline | 8.00 | 84.67 | 96.00 | 71.21 | 92.75 | 78.05 | 1.00 |
| DES | 4.89 | 68.00 | 94.80 | 62.12 | 90.75 | 70.12 | 1.26 |  |
| ODP | 4.91 | 73.33 | 95.20 | 66.16 | 91.00 | 73.78 | 1.26 |  |
| Ban | 4.77 | 83.33 | 95.40 | 69.70 | 91.25 | 75.61 | 1.27 |  |

We compare our proposed Ban method against two mainstream dynamic expert-pruning approaches:
(1) DES.
(lu-etal-2024-experts,) (Dynamic Expert Skip) dynamically prunes low-weight experts
by observing that,
for easy tokens, the contribution of lower-ranked experts is often negligible. Concretely, it decides whether to drop the secondary expert based on the ratio between expert routing weights.
(2) ODP.
(huang2024mc,) (Online Dynamic Pruning) extends DES by adding a key-token protection mechanism. Tokens that receive disproportionately high attention scores are treated as key tokens, and their experts are exempted from pruning to preserve accuracy.

In terms of accuracy, as shown in Table [2](https://arxiv.org/html/2509.06346v2#S6.T2), both DES and ODP suffer from severe degradation on these challenging tasks, with drops exceeding 5% on certain datasets.
In contrast, Ban limits the accuracy loss to within 1.5% on most models and datasets, except HumanEval+, which we found sensitive to pruning, and even within 1% on more than half of the results.
Notably, on Qwen3-30B-A3B, Ban reduces accuracy on AIME2024 by only 0.67%–equivalent to misanswering just one additional problem out of five evaluations–and incurs merely 1.01% degradation on GPQA.

In addition to accuracy, we record the average number of selected experts per token and the relative speedup—computed from the total inference time across all datasets—under the vLLM backend to comprehensively evaluate inference efficiency. Ban restricts the average number of selected experts to around 4 for DeepSeek models, leading to more than 1.2$\times$ actual speedup. On Qwen3 models, Ban keeps the average expert count below 5, achieving about 1.25$\times$ speedup. These results demonstrate the strong ability of Ban to preserve model performance while delivering substantial efficiency gains.

### 6.4 Results of Ban&Pick

**Table 3: Results of combining Ban&Pick across four models. We report average experts per token, accuracy, and inference speedup under vLLM. Relative changes from baseline are annotated in gray.**
| Models | Methods | Avg-Topk | GSM8K/AIME24 | Math500 | GPQA | Livecodebench | HumanEval+ | Speedup |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Deepseek-V2-Lite-Chat | Baseline | 6.00 | 66.34 | 26.40 | 24.75 | 9.25 | 50.61 | 1.00 |
| Ban&Pick | 4.03 | 66.94 ↑0.60 | 26.80 ↑0.40 | 25.25 ↑0.50 | 9.50 ↑0.25 | 50.00 ↓0.61 | 1.22 |  |
| Deepseek-v2.5-1210 | Baseline | 6.00 | 23.33 | 81.00 | 47.47 | 60.75 | 83.54 | 1.00 |
| Ban&Pick | 3.99 | 26.00 ↑2.67 | 81.60 ↑0.60 | 47.98 ↑0.51 | 61.25 ↑0.50 | 84.15 ↑0.61 | 1.23 |  |
| Qwen3-30B-A3B | Baseline | 8.00 | 80.67 | 95.20 | 65.66 | 91.75 | 71.34 | 1.00 |
| Ban&Pick | 4.83 | 84.66 ↑3.99 | 95.80 ↑0.60 | 68.18 ↑2.52 | 92.85 ↑1.10 | 72.56 ↑1.22 | 1.25 |  |
| Qwen3-235B-A22B | Baseline | 8.00 | 84.67 | 96.00 | 71.21 | 92.75 | 78.05 | 1.00 |
| Ban&Pick | 4.79 | 86.67 ↑2.00 | 96.00 ↑0.00 | 72.73 ↑1.52 | 93.25 ↑0.50 | 78.66 ↑0.61 | 1.26 |  |

By combining Ban and Pick, our unified framework leverages smarter routing strategies to simultaneously deliver *free* accuracy gains and inference acceleration on the majority of models and datasets, demonstrating strong practicality. As shown in Table [3](https://arxiv.org/html/2509.06346v2#S6.T3), on Qwen3-30B-A3B, Ban&Pick achieves an average accuracy improvement of 1.99% together with a 1.25$\times$ speedup, while on the large-scale Qwen3-235B-A22B, it yields an average accuracy gain of 1.33% and a 1.26$\times$ speedup. As two plug-and-play modules, Ban and Pick share the common goal of more effective and efficient expert routing, while offering flexible choices: higher performance (Pick), faster inference (Ban), or a balanced combination of both improvements (Ban&Pick).

## 7 Interaction Among Key Experts Across Tasks

We further investigate the scenario where all identified key experts across math, general reasoning, and code are enhanced simultaneously. Using Qwen3-30B-A3B as a case study, we report in Table [4](https://arxiv.org/html/2509.06346v2#S7.T4) the performance when enhancing key experts for a single domain, for domain pairs (Math+General, Math+Code, General+Code), and for all three domains together. This setting allows us to examine whether experts from different domains complement or interfere with each other when activated jointly, providing a more comprehensive understanding of cross-domain interactions.

**Table 4: Task accuracies (%) on Qwen3-30B-A3B under different combinations of enhanced experts. A ✓ indicates that the corresponding domain experts are enhanced, while × means not enhanced.**
| Math | General | Code | AIME2024 | MATH-500 | GPQA-Diamond | LiveCodeBench | HumanEval+ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $\times$ | $\times$ | $\times$ | 80.67 | 95.20 | 65.66 | 91.75 | 71.34 |
| ✓ | $\times$ | $\times$ | 85.33 ↑4.66 | 96.60 ↑1.40 | 65.15 ↓0.51 | 91.50 ↓0.25 | 71.95 ↑0.61 |
| $\times$ | ✓ | $\times$ | 82.00 ↑1.33 | 95.20 ↑0.00 | 69.19 ↑3.53 | 91.75 ↑0.00 | 72.56 ↑1.22 |
| $\times$ | $\times$ | ✓ | 81.33 ↑0.66 | 95.00 ↓0.20 | 65.66 ↑0.00 | 94.25 ↑2.50 | 75.61 ↑4.27 |
| ✓ | ✓ | $\times$ | 84.67 ↑4.00 | 96.20 ↑1.00 | 67.68 ↑2.02 | 92.00 ↑0.25 | 71.34 ↑0.00 |
| ✓ | $\times$ | ✓ | 84.67 ↑4.00 | 96.00 ↑0.80 | 66.16 ↑0.50 | 93.50 ↑1.75 | 74.39 ↑3.05 |
| $\times$ | ✓ | ✓ | 80.67 ↑0.00 | 94.80 ↓0.40 | 68.18 ↑2.52 | 92.75 ↑1.00 | 75.00 ↑3.66 |
| ✓ | ✓ | ✓ | 84.00 ↑3.33 | 96.00 ↑0.80 | 67.68 ↑2.02 | 93.00 ↑1.25 | 74.39 ↑3.05 |

From Table [4](https://arxiv.org/html/2509.06346v2#S7.T4), we observe that enhancing multiple domains simultaneously still brings clear improvements. For instance, when both Math and General experts are enhanced, AIME2024 increases by +4.0% and GPQA by +2.02%. However, these gains are slightly smaller than enhancing each domain individually ($-0.66\%$ and $-1.51\%$, respectively). When all three domains are enhanced together, the model continues to outperform the baseline—+3.33% on AIME2024, +2.02% on GPQA, and +3.05% on HumanEval+—though the gains are somewhat reduced compared to the best single-domain setting.
These results indicate that while domain-specific key experts do interfere to some extent, their combined enhancement remains clearly beneficial,
confirming their crucial role.

## 8 Conclusion and Implications for Future Work.

In this study, we reveal that in fine-grained MoE models, relatively balanced routing strategy underutilizes a small set of key experts which have a disproportionately large impact on task performance, and fixed expert activation introduces substantial redundancy. By selectively increasing the utilization of key experts and dynamically pruning redundant ones, we release the potential of routing and provide a novel and effective framework that improves both accuracy and efficiency. For future work, we suggest exploring the underlying mechanisms of how key experts enhance model performance, and investigating more sophisticated enhancement strategies beyond uniform boosting.

## References

- [1]
Chufan Shi, Cheng Yang, Xinyu Zhu, Jiahao Wang, Taiqiang Wu, Siheng Li, Deng Cai, Yujiu Yang, and Yu Meng.
Unchosen experts can contribute too: Unleashing moe models’ power by self-contrast.
Advances in Neural Information Processing Systems, 37:136897–136921, 2024.
- [2]
Keisuke Kamahori, Yile Gu, Kan Zhu, and Baris Kasikci.
Fiddler: Cpu-gpu orchestration for fast inference of mixture-of-experts models, 2024.
- [3]
Shuzhang Zhong, Yanfan Sun, Ling Liang, Runsheng Wang, Ru Huang, and Meng Li.
Hybrimoe: Hybrid cpu-gpu scheduling and cache management for efficient moe inference, 2025.
- [4]
Mengru Wang, Xingyu Chen, Yue Wang, Zhiwei He, Jiahao Xu, Tian Liang, Qiuzhi Liu, Yunzhi Yao, Wenxuan Wang, Ruotian Ma, Haitao Mi, Ningyu Zhang, Zhaopeng Tu, Xiaolong Li, and Dong Yu.
Two experts are all you need for steering thinking: Reinforcing cognitive effort in moe reasoning models without additional training, 2025.
- [5]
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton.
Adaptive mixtures of local experts.
Neural Computation, 3(1):79–87, 1991.
- [6]
Michael I. Jordan and Robert A. Jacobs.
Hierarchical mixtures of experts and the em algorithm.
Neural Computation, 6(2):181–214, 1994.
- [7]
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.
Gshard: Scaling giant models with conditional computation and automatic sharding, 2020.
- [8]
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi.
Olmoe: Open mixture-of-experts language models, 2025.
- [9]
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You.
Openmoe: An early effort on open mixture-of-experts language models.
arXiv preprint arXiv:2402.01739, 2024.
- [10]
Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, and Sebastian Jaszczur.
Scaling laws for fine-grained mixture of experts.
In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 33270–33288. PMLR, 21–27 Jul 2024.
- [11]
Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan.
Moe++: Accelerating mixture-of-experts methods with zero-computation experts, 2024.
- [12]
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li.
Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification.
In ICLR, 2024.
- [13]
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.
arXiv preprint arXiv:1701.06538, 2017.
- [14]
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov.
Efficient large scale language modeling with mixtures of experts.
In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11699–11732, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
- [15]
William Fedus, Barret Zoph, and Noam Shazeer.
Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.
J. Mach. Learn. Res., 23(1), January 2022.
- [16]
Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P Xing, and Hao Zhang.
Toward inference-optimal mixture-of-expert large language models.
arXiv preprint arXiv:2404.02852, 2024.
- [17]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
- [18]
Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng.
Harder task needs more experts: Dynamic routing in MoE models.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12883–12895, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
- [19]
Shen Yan, Xingyan Bin, Sijun Zhang, Yisen Wang, and Zhouchen Lin.
Tc-moe: Augmenting mixture of experts with ternary expert choice.
In The Thirteenth International Conference on Learning Representations, 2025.
- [20]
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.
Finetuned language models are zero-shot learners, 2022.
- [21]
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto.
s1: Simple test-time scaling, 2025.
- [22]
Kuang-Huei Lee, Ian Fischer, Yueh-Hua Wu, Dave Marwood, Shumeet Baluja, Dale Schuurmans, and Xinyun Chen.
Evolving deeper llm thinking, 2025.
- [23]
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.
Tree of thoughts: Deliberate problem solving with large language models, 2023.
- [24]
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler.
Graph of thoughts: Solving elaborate problems with large language models.
Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17682–17690, March 2024.
- [25]
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu.
Thoughts are all over the place: On the underthinking of o1-like llms, 2025.
- [26]
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li.
Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6159–6172, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
- [27]
Yuanteng Chen, Yuantian Shao, Peisong Wang, and Jian Cheng.
EAC-MoE: Expert-selection aware compressor for mixture-of-experts large language models.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12942–12963, Vienna, Austria, July 2025. Association for Computational Linguistics.
- [28]
Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and Xiaojuan Qi.
Mc-moe: Mixture compressor for mixture-of-experts llms gains more.
arXiv preprint arXiv:2410.06270, 2024.
- [29]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report, 2025.
- [30]
DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen
Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie.
Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.
- [31]
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.
Mixtral of experts.
arXiv preprint arXiv:2401.04088, 2024.
- [32]
Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang.
DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
- [33]
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report, 2025.
- [34]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention, 2023.
- [35]
Bhrij Patel, Souradip Chakraborty, Wesley A. Suttle, Mengdi Wang, Amrit Singh Bedi, and Dinesh Manocha.
Aime: Ai system optimization via multiple llm evaluators, 2024.
- [36]
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step.
arXiv preprint arXiv:2305.20050, 2023.
- [37]
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi.
MathQA: Towards interpretable math word problem solving with operation-based formalisms.
In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2357–2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
- [38]
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman.
Gpqa: A graduate-level google-proof q&a benchmark, 2023.
- [39]
King Han Alex Gu Wen-Ding Li Fanjia Yan Tianjun Zhang Sida Wang Armando Solar-Lezama Koushik Sen Naman Jain and Ion Stoica.
Livecodebench: Holistic and contamination free evaluation of large language models for code.
arXiv preprint, 2024.
- [40]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.
Evaluating large language models trained on code.
2021.
- [41]
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.
Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation.
In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- [42]
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.
Winogrande: An adversarial winograd schema challenge at scale.
Communications of the ACM, 64(9):99–106, 2021.
- [43]
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner.
Documenting large webtext corpora: A case study on the colossal clean crawled corpus, 2021.
- [44]
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
Training verifiers to solve math word problems.
arXiv preprint arXiv:2110.14168, 2021.
- [45]
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He.
C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.
In Advances in Neural Information Processing Systems, 2023.
- [46]
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.
Program synthesis with large language models.
arXiv preprint arXiv:2108.07732, 2021.
- [47]
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin.
Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning, 2025.
- [48]
OpenCompass Contributors.
Opencompass: A universal evaluation platform for foundation models.
[https://github.com/open-compass/opencompass](https://github.com/open-compass/opencompass), 2023.
- [49]
Xing Hu Yuan Cheng Dawei Yang Zhixuan Chen Zukang Xu Jiangyong Yu Chen Xu Zhihang Yuan Zhe jiang and Sifan Zhou.
OSTQuant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting.
In The Thirteenth International Conference on Learning Representations, 2025.
- [50]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
Proceedings of the International Conference on Learning Representations (ICLR), 2021.
- [51]
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
Think you have solved question answering? try arc, the ai2 reasoning challenge.
arXiv:1803.05457v1, 2018.

## Appendix A Complete Results on Expert Specialization

In Section [4.1](https://arxiv.org/html/2509.06346v2#S4.SS1), we analyzed the specialization of experts in fine-grained MoE models using Qwen3-30B-A3B as a case study, and presented representative figures for a subset of layers. Here, we provide the complete visualization of expert usage frequencies across all layers of the model. Results are shown for three representative task types—math, code, and general. For each layer, the bar charts display the relative usage frequency of each expert, aggregated over all tokens in both the prefill and decode stages. Figures are grouped every 12 layers for readability. These results (Figures [6](https://arxiv.org/html/2509.06346v2#A1.F6)–[9](https://arxiv.org/html/2509.06346v2#A1.F9)) confirm that different task types consistently activate distinct subsets of experts, highlighting clear patterns of domain specialization.

Figure: Figure 6: Expert usage frequency for layers 0–11 across math, code, and general tasks.
Refer to caption: x7.png

Figure: Figure 7: Expert usage frequency for layers 12–23 across math, code, and general tasks.
Refer to caption: x8.png

Figure: Figure 8: Expert usage frequency for layers 24–35 across math, code, and general tasks.
Refer to caption: x9.png

Figure: Figure 9: Expert usage frequency for layers 36–47 across math, code, and general tasks.
Refer to caption: x10.png

## Appendix B Implementation details of Top-1000 KL divergence

Following prior work [49], we note that the model’s output logits typically follow a long-tail distribution, where only the top-ranked tokens carry substantial probability mass. Therefore, when computing KL divergence, we restrict attention to the Top-1000 tokens. Concretely, let $p$ denote the probability distribution obtained from the original model, and $q$ the distribution when a candidate expert is pruned. Denote by $\mathcal{I}_{1000}$ the index set of the top-1000 tokens under $p$. The restricted distributions are defined as

$$ $p^{\prime}(i)=\frac{p(i)}{\sum_{j\in\mathcal{I}_{1000}}p(j)},\quad q^{\prime}(i)=\frac{q(i)}{\sum_{j\in\mathcal{I}_{1000}}q(j)},\quad i\in\mathcal{I}_{1000}.$ $$

The KL divergence used in our analysis is then

$$ $D_{\mathrm{KL}}(p^{\prime}\parallel q^{\prime})=\sum_{i\in\mathcal{I}_{1000}}p^{\prime}(i)\log\frac{p^{\prime}(i)}{q^{\prime}(i)}.$ $$

This definition ensures that the comparison focuses on the most influential tokens, while ignoring the long tail of near-zero probabilities.

## Appendix C Hyperparameter Choices in Dynamic Pruning

As introduced in Section [5](https://arxiv.org/html/2509.06346v2#S5), our dynamic pruning strategy involves three hyperparameters: $\beta$, $K_{\min}$, and $\lambda$. The parameter $\beta$ balances the contributions of layer-level and token-level sensitivities. Since our motivation is to jointly consider both factors in a balanced way, we simply set $\beta=0.5$ empirically and do not explore it further. In this appendix, we therefore focus on discussing the selection of $K_{\min}$ and $\lambda$ in detail.

#### Choice of K min K_{\min} .

In the main text, we noted that reducing the number of selected experts below a certain threshold causes the model to collapse entirely. Therefore, $K_{\min}$ must guarantee that even when both layer- and token-level sensitivities are low, the number of selected experts never falls below this safe lower bound.

To study this effect, we take Qwen3-30B-A3B as an example and directly enforce the number of experts per token to be fixed between 2 and 7 (with $K_{\text{base}}=8$ as the default). Table [5](https://arxiv.org/html/2509.06346v2#A3.T5) reports the performance on five benchmarks. For efficiency, results on AIME2024 (except the baseline) are measured over a single run.

**Table 5: Performance of Qwen3-30B-A3B under different fixed numbers of selected experts.**
| TopK | AIME2024 | Math500 | GPQA | Livecodebench | HumanEval+ |
| --- | --- | --- | --- | --- | --- |
| 8 | 80.67 | 95.2 | 65.66 | 91.75 | 71.34 |
| 7 | 80.00 | 95.0 | 65.66 | 91.25 | 71.95 |
| 6 | 76.67 | 93.2 | 63.64 | 89.75 | 69.51 |
| 5 | 76.67 | 88.2 | 59.60 | 87.00 | 66.46 |
| 4 | 63.33 | 78.4 | 54.55 | 74.75 | 59.76 |
| 3 | 40.00 | 55.8 | 44.95 | 57.50 | 43.29 |
| 2 | 16.67 | 19.6 | 21.72 | 25.50 | 17.07 |

We observe that removing just one expert (TopK=7) leads to almost no performance degradation, consistent with the expert pruning observations reported in Section [3](https://arxiv.org/html/2509.06346v2#S3) in the main text. However, once two or more experts are pruned, noticeable drops appear across all benchmarks. The most critical boundary occurs at TopK=3: performance collapses when fewer than three experts are selected. Therefore, we set $K_{\min}=3$ to prevent the model from falling below this threshold. On the DeepSeek series, we observed a similar phenomenon, further reinforcing this choice.

#### Choice of λ \lambda .

The hyperparameter $\lambda$ controls the trade-off between accuracy preservation and inference speedup: larger values of $\lambda$ bias the pruning strategy toward retaining more experts (thus preserving accuracy), while smaller values encourage more aggressive pruning (thus improving throughput). To study this trade-off, we take Qwen3-30B-A3B as an example and vary $\lambda$ from $0.5$ to $0.9$ in increments of $0.1$. For each setting, we evaluate accuracy on the same five benchmarks as in the main experiments and measure the inference speedup using vLLM as the serving backend. The speedup is calculated as the ratio between the total inference time across all datasets and the baseline time without pruning. All results are reported from a single run.

**Table 6: Accuracy and inference speedup of Qwen3-30B-A3B under different values of $\lambda$. The first row corresponds to the baseline.**
| $\lambda$ | Avg TopK | AIME2024 | Math500 | GPQA | Livecodebench | HumanEval+ | Speedup |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | 8 | 80.67 | 95.2 | 65.66 | 91.75 | 71.34 | 1.00 |
| 0.5 | 4.33 | 73.33 | 93.8 | 59.60 | 86.50 | 65.85 | 1.30 |
| 0.6 | 4.61 | 76.67 | 94.0 | 63.64 | 89.25 | 68.90 | 1.26 |
| 0.7 | 4.82 | 80.00 | 94.6 | 64.65 | 90.00 | 70.12 | 1.25 |
| 0.8 | 5.09 | 80.00 | 94.0 | 64.65 | 90.75 | 70.12 | 1.22 |
| 0.9 | 5.32 | 80.00 | 94.8 | 64.14 | 90.50 | 70.73 | 1.20 |

From Table [6](https://arxiv.org/html/2509.06346v2#A3.T6), we observe that when $\lambda$ is below $0.7$, the model achieves higher speedups but suffers noticeable accuracy degradation compared with $\lambda=0.7$. When $\lambda$ exceeds $0.7$, accuracy improves slightly, but the gains are marginal relative to the loss in efficiency. Considering this trade-off, we set $\lambda=0.7$ in our main experiments and report results under this configuration. Nevertheless, $\lambda$ remains a tunable knob that allows users to flexibly balance accuracy preservation and inference acceleration according to their own application needs.

## Appendix D Evaluation Details

### D.1 Dataset Evaluation.

For AIME2024, due to the small sample size, we follow the common practice in technical reports by evaluating each method five times and reporting the average. For other datasets, constrained by time, we conduct only a single evaluation. On the GPQA-Diamond dataset, to further reduce errors caused by misaligned answer formats, we adopt an LLM-judge in addition to the base evaluator: specifically, we deploy a Qwen2.5-72B-Instruct model [33] with vLLM [34] as the judge model. For all other datasets, we rely on the default base evaluators provided by OpenCompass to compute accuracy.

### D.2 Generation Hyperparameters.

During generation, for all four models used in our experiments, we adopt the recommended hyperparameters provided in their original papers or Model Cards. The configuration is summarized in Table [7](https://arxiv.org/html/2509.06346v2#A4.T7).

**Table 7: Generation hyperparameters for the evaluated models.**
| Model | Top-k | Top-p | Temperature | Max Output Length |
| --- | --- | --- | --- | --- |
| DeepSeek-V2-Lite-Chat | None | 0.95 | 0.3 | 8192 |
| DeepSeek-V2.5-1210 | 1 | 0.90 | 0.3 | 8192 |
| Qwen3-30B-A3B | 20 | 0.95 | 0.6 | 32768 |
| Qwen3-235B-A22B | 20 | 0.95 | 0.6 | 32768 |

## Appendix E Details of Compared Methods and Reproduction Settings

In the main experiments, we compared our proposed Ban and Pick strategies against several existing baselines. For brevity, the main text only provided a concise overview of these methods. In this appendix, we present a more detailed description of each compared method, together with the specific reproduction settings used in our experiments. We organize the discussion into two parts: methods compared with Pick, and methods compared with Ban.

### E.1 Compared Methods for Pick

#### Dynamic Routing.

[18] first proposed the idea of dynamically allocating experts, arguing that assigning a fixed number of experts to each token is overly rigid. Instead, the number of experts should be adapted according to the difficulty of the underlying task: tokens from more challenging tasks should be allocated more experts, while tokens from easier tasks can be served with fewer. Concretely, experts are sequentially selected in descending order of routing weight until the cumulative weight exceeds a predefined threshold $\tau$. Formally, let $r_{(1)},r_{(2)},\dots,r_{(K)}$ denote the routing weights of experts sorted in descending order. The number of selected experts $k$ is determined as

$$ $k=\min\Bigl\{m\,\big|\,\sum_{j=1}^{m}r_{(j)}\geq\tau\Bigr\}.$ $$

In our experiments, since the optimal threshold $\tau$ cannot be known in advance, we evaluate three candidate values $\tau\in\{0.7,0.8,0.9\}$ and report in the main text the result corresponding to the setting that achieves the best average accuracy across benchmarks.

#### TiP.

The core idea of TiP is motivated by the observation that, during chain-of-thought (CoT) reasoning, models often switch reasoning paths too frequently in the early exploration stage. Such premature abandonment of a correct path is referred to as *underthinking*. To mitigate this issue, TiP introduces a penalty on tokens that typically indicate a switch of reasoning path (e.g., “alternatively”). Concretely, after the softmax operation, the probability mass of these tokens is reduced by subtracting a fixed penalty value from their logits, thereby encouraging the model to continue deepening its current reasoning path rather than switching too early.

This method involves two hyperparameters: $\alpha$ (penalty strength) and $\beta$ (penalty duration). The original paper did not provide an automatic tuning strategy but instead conducted a grid search to identify relatively effective values. Following their reported settings, we set $\alpha=3$ and $\beta=600$ in our reproduction.

#### RICE.

RICE builds on the observation that recent MoE-LLMs often employ an explicit <think> token to train chain-of-thought (CoT) reasoning ability. For example, the Qwen3 series can switch between deep-thinking and non-thinking modes by toggling the use of the <think> token. RICE leverages this property by comparing expert activation frequencies with and without the <think> token, thereby identifying *cognitive experts* that are closely associated with deep reasoning. Once identified, the routing weight of a cognitive expert is multiplied by a constant factor greater than $1$ whenever it is selected, amplifying its influence and enhancing the model’s reasoning ability.

Since this method relies on explicit <think> tokens, it is only applicable to models such as the Qwen3 series, but not to other models used in our experiments. For reproduction, we followed the original paper’s settings: for Qwen3-235B-A22B, we directly used the cognitive experts reported therein; for Qwen3-30B-A3B, as the original paper did not list them, we applied the ICE (Identify Cognitive Experts) procedure proposed in the paper to locate the corresponding cognitive experts before applying the enhancement.

### E.2 Compared Methods for Ban

#### DES.

DES (Dynamic Expert Skip) was proposed to dynamically adjust the number of active experts by exploiting the observation that, for tokens that are relatively easy to process, the routing weights of the selected experts often differ significantly. In such cases, the contribution of the lower-weighted expert to the final output is negligible. Based on this, DES introduces a dynamic pruning strategy: whether to keep the lower-weighted expert is decided according to the ratio between expert weights. For example, in Mixtral-8$\times$7B [31] where each token selects two experts, DES first computes the median ratio of top-1 to top-2 expert weights on a calibration set. During inference, if the actual ratio exceeds this median, the top-2 expert is skipped.

In our reproduction, we aimed to align the pruning degree and acceleration ratio with our proposed Ban method, so that the comparison mainly reflects differences in accuracy preservation. For Qwen3 models, when $\lambda=0.7$, the average number of selected experts under Ban is slightly below 5. Therefore, we set the maximum number of experts to 6 and the minimum to 4. Specifically, we computed the medians of the ratios $\tfrac{r_{(4)}}{r_{(5)}}$ and $\tfrac{r_{(5)}}{r_{(6)}}$ on a calibration set, where $r_{(j)}$ denotes the $j$-th largest routing weight. At inference time, if the first ratio exceeds its median, only the top-4 experts are kept; otherwise, if the first ratio does not exceed its median but the second does, the top-5 experts are kept; otherwise, all top-6 experts are retained. For the DeepSeek models, we followed an analogous procedure to ensure comparable pruning degree and speedup.

#### ODP.

ODP(Online Dynamic Pruning) is an extension of DES that incorporates an additional *key-token protection* mechanism. While DES prunes experts dynamically based on weight ratios, ODP further observes that certain tokens carry disproportionately high importance in the attention map. If a token receives an attention score significantly larger than the others, pruning its experts may harm model performance. To address this, ODP protects such key tokens: even if their expert weights satisfy the pruning condition, they are exempted from pruning.

In our reproduction, we followed exactly the same procedure as DES for the pruning part. On top of that, we adopted the evaluation metric for token importance proposed in the ODP paper to identify key tokens. Whenever a token is classified as key, we enforced the selection of all 6 experts for it, regardless of the pruning condition.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: How would the "Pick" module function on a task that intrinsically blends domains, such as writing a legal analysis (general reasoning) of a software patent (code)?
soundness: 3
strengths: - The proposed Ban&Pick is a post-training strategy. It does not require expensive retraining of the foundation model.
- The paper provides a thorough evaluation (Section 6) across multiple modern MoE models (DeepSeek-V2-Lite, DeepSeek-v2.5, Qwen3-30B, Qwen3-235B) and a diverse set of challenging benchmarks (AIME2024, GPQA, LiveCodeBench)
summary: This paper claims that the routing strategies learned during pre-training cannot fully exploit this specialization, and thus proposes Pick and Ban strategies to optimizes expert utilization in inference. It improves both the accuracy and efficiency of fine-grained MoE without retraining or architectural changes.
weaknesses: - Concern on generalization capability. The "Pick" module relies on identifying key experts for specific domains (math, code, general). This introduces a calibration step and raises questions about scalability. It's unclear how this would perform on tasks that are highly interdisciplinary or fall outside the tested domains.

- Hyperparameter Sensitivity. The method introduces new hyperparameters that may require tuning.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: Please refer to weaknesses.
soundness: 3
strengths: 1. The findings on domain-specialized experts and the redundant expert activation is critical to the design of the Ban&Pick. The overall design of the method is very straightforward and easy to implement.

2. The analysis of key expert is very thorough. The explanation is easy to understand. Same writing style is appreciated for redundant expert selection. The whole structure is clear and easy to follow.

3. I believe the discovery is novel and inspiring. Although the method seems to be straightforward, the discovery can be utilized in many future research.
summary: The paper proposes Ban&Pick, which is a post-training, plug-and-play strategy for smarter routing for selecting expert in the modern MoE models. The Pick discovers and reinforces key experts that is a small group with outsized impact on performance, which leading to notable accuracy gains across domains. Ban further dynamically prunes redundant experts based on layer and token sensitivity, delivering faster inference with minimal accuracy loss. The author of the paper performs experiments on fine-grained MoELLMs (DeepSeek, Qwen3) across math, code, and general reasoning benchmarks demonstrate that Ban&Pick delivers free performance gains and inference acceleration without retraining or architectural changes.
weaknesses: 1. It is not clear to me what is the key reason of failing to include of some key expert in specific domain tasks. According to the earlier discussion in the paper, the reason might come from the premature of the routing strategy during pretraining. Some of the related works also proves that. But in this paper, the author does not show the direct connection between this reason (premature) to the issue (expert selecting failure). The experiments in figure 1, 2, 3, is showing the fact that there are key experts and they are important, but the whole paper fails at showing why. The author of the paper should articulate on this and it will make the paper better by a lot.

2. In order to enhancing the key experts, the author of the paper proposes 5 strategy and the author of the paper uses strategy D. But the reader would be more convincing of why the author of the paper choose such design. 

3. Comparing to Pick, the Ban sometimes degrade the accuracy, which is understandable. But combine with Pick, the goal of this paper would become improve the performance of the MoE model. However, the Ban&Pick results in Table 3 shows varying improve. Although the difficulty of the tasks are different, some of the tasks like Math500 has limited improvement. And within the same task like GPQA, different models adapt to the Ban&Pick differently, which shows unstable performance of the method.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Can the proposed Ban&Pick framework be combined with adaptive routing methods or reinforcement-learning-based expert selection?

2. How sensitive is the method to hyperparameters (λ, Kmin) across different model scales?

3. Could online or task-specific adaptation further enhance the framework’s performance?
soundness: 3
strengths: 1. Demonstrates a clever engineering insight—leveraging untapped routing inefficiencies to boost performance and efficiency simultaneously.

2. Provides a lightweight, generalizable, and plug-and-play approach applicable to existing fine-grained MoE architectures.

3. The empirical validation is comprehensive, spanning multiple large models and diverse domains.

4. Shows clear, measurable engineering gains (up to +3% accuracy and 1.25× speedup) without any additional training cost.

5. Strong potential for real-world adoption in industrial MoE inference systems (e.g., vLLM, DeepSeek).
summary: This paper proposes Ban&Pick, a post-training, plug-and-play framework that improves both the performance and efficiency of fine-grained Mixture-of-Experts (MoE) large language models. The method introduces two complementary modules: Pick, which identifies and reinforces key experts with major impact on task performance, and Ban, which dynamically prunes redundant experts based on layer and token sensitivity.

Without retraining or architectural modification, the framework achieves consistent accuracy improvements and inference acceleration across multiple MoE-LLMs (DeepSeek, Qwen3) on math, code, and reasoning benchmarks. The work is technically sound and demonstrates strong engineering novelty and practical value.
weaknesses: 1. Limited theoretical depth. The paper focuses on empirical effectiveness but does not deeply analyze the underlying mechanism of key expert influence or convergence behavior.

2. Lack of comparison with more recent MoE optimization baselines, such as MoE++, adaptive gating, or mixed routing approaches (e.g., Mixtral, Soft-MoE).

3. Evaluation scope is somewhat narrow, emphasizing accuracy and speedup but lacking broader system-level metrics like throughput, latency variance, or memory usage.

4. Minor stylistic and formatting issues, including small typos and missing pseudocode for algorithmic clarity.

5. Noticeable typographical issue: the paper title itself contains a serious typo (“EHANCING” should be “ENHANCING”), which reflects carelessness in proofreading.


Decision: Reject