SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models
Authors: Juntong Wu
Jialiang Cheng
Fuyu Lv
OU Dan
Li Yuan
Venue: ICLR 2026 Poster (2026)
Keywords: MoE Model
Inference Acceleration
Batch Decoding
Expert Re-routing
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: We propose SERE, a Similarity-based Expert Re-routing method for Efficient batch decoding in MoE models.

Abstract: Mixture-of-Experts (MoE) architectures employ sparse activation to deliver faster training and inference with higher accuracy than dense LLMs. However, in production serving, MoE models require batch inference to optimize hardware efficiency, which may cause excessive expert activation and thus slow the memory-bound decoding stage. To address the fundamental tension between batch decoding and expert sparsity, we present **SERE**, a **S**imilarity-based **E**xpert **R**e-routing method for **E**fficient batch decoding in MoE models. SERE dynamically reduces the number of active experts in an input-aware manner by re-routing tokens from secondary experts to their most similar primary counterparts. It also leverages similarity patterns to identify and preserve critical experts, thereby preventing capability loss. Notably, SERE avoids static expert pruning or merging, instead enabling dynamic expert skipping based on batch-level expert redundancy. Additionally, we provide an efficient custom CUDA kernel for SERE, enabling plug-and-play use in vLLM with only a single-line code change. Extensive experiments on various complex reasoning benchmarks demonstrate that SERE achieves up to $2.0\times$ speedup with minimal quality loss, providing a practical solution for cost-efficient and latency-sensitive large-scale MoE deployment.

Published as a conference paper at ICLR 2026
SERE: SIMILARITY-BASER EXPERT RE-ROUTING FOR EFFICIENT BATCH DECODING IN MoE MODELS
Juntong Wu Jialiang Cheng!Stt Fuyu Lv Ou Dan', Li Yuan 2,50
Taobao & Tmall Group of Alibaba
2 Shenzhen Graduate School, Peking University
Correspondence: fichen.cjl alibaba-inc.com quali-ece@pku.eduuu
ABSTRACT
Mixture-of-Expera (MoE) architectures employ sparse activation to deliver faster training and inference with higher accuracy than dense LLMs. However, in pro- duction serving, MoE models require batch inference to optimize hardware effi- ciency, which may cause excessive expert activation and thus slow the memory- bound decoding stage. To address the fundamental tension between batch decod- ing and expert sparsity, we present SERE, Similarity-bbsed Expert Re-routing method for Efficient batch decoding in MoE models. SERE dynamically reduces the number of active experts in input-awar manner by re-routing tokens from secondary experts to their most similar primary counterparts. It also leverages sim- ilarity patterns identify and preserve critical experts. thereby preventing capabil- ity loss. Notably, SERE avoids static expert pruning or merging, instead enabling dynamic expert skipping based batch-level expert redundancy. Additionally, we provide an efficient custom CUDA kernel for SERE, enabling plug-and-pla and-play use vLLM with only single-line code change Extensive experiments on var- ious complex reasoning benchmarks demonstrate that SERE achieves up to 2.0x speedup with minimal quality loss, providing practical solution for cost-efficient and latency-sensitive large-scale MoE deployment.
INTRODUCTION
Large Language Models (LLMs) have shown remarkable performance across various applications. Recently, the Mixture-of-Expertt (MoE) paradigm has emerged as lead- ing framework for scaling LLMs Yang et al. 2025a Liu
Average ArerageActtvonn Activated Experts
100
84.
97.3
13.4
18.9
et al.) (2024b] Touvron et al.] [023] Jiang et al. 2024). Un- like dense L.MS that activate the entire forward net- work (FFN) for every token, MoE layer consists of mul- tiple lightweight FFN experts, where a learnable router as- signs each token small subset. By maintaining low per-token computation, sparse activation enables the model incorporate numerous specialized experts, scaling its ca- pacity while preserving training and inference efficiency.
353
Averape Activation
Avengg Mol: Time (14)
Despite the theoretical efficiency of MoE architectures, their practical gains are often limited by mismatch be- tween selective activation and batched inference (Kwon
al., 20233 Agrawal et al. [2224] Gupta al. 2024). In real-world services, multiple user requests are batched to improve hardware utilization (Kwon et al.) 2023). How- ever, tokens within a batch often require different experts, leading to total number of activated experts far above the per-token budget (Agrawal et al. 2024 Yun et al. 2024). As depicted in Figure] | even with striet limits (e.g., out
120-4
120-16
24
56
72
104
120
Active Experts
Figure 1: Larger batches activate
more experts. With a fixed batch size, more experts increase decoding time.
Equal contribution Project Lead sa Corresponding author
Code implementation of SERE can be found in Cumnddnitttpp:/ggthubbb..c///// http:::///ggtbbbbccbteee

---

Published as a conference paper at ICLR 2026
Quen3
QwentS
DeepSeek
MATH
Beal
Qwad-336-A33
30B-A3B
A2.78
V2-Lite
Qvad-I(B-A3B (K-2
44.4
36,8%
39,3
50.2%
26.4
29.0%
RC-SMOT
(40 Paps)
GEMBIC
CANNLE
20.6
18.5
19.6
MATHOHI
SERE (Ours)
(K-1)
Oraia LYNX
Orign
Oraie
(a) Accuracy Maintained
(b) Time Per Output Token (ms)
Figure 2: Visualizations of SERE's Performance. (a) Across all tasks, SERE (K=2) exhibits negli- gible performance loss, while SERE (K=1) still outperforms all baselines. (b) SERE significantly reduces batch decoding time, achieving up to 2x accelcration.
of 128 in Qwen3-30B-A3B Yang et al. 2025a)), a moderately diverse batch can still activate a ma- jority of the experts simultaneously. Moreover, the training-time timm load-balancing objectives further increase the expert diversity within batch (Lepikhin et al., 2021 Liu et al. 2024b). This issue is particularly acute during decoding (Yun et 2024). where sequential token generation makes the process seemory-badwwidth-boon -oouth.. As also can seen from Figure [ activating excessive experts during decoding raises communication and memory-access overhead and thus increases latency. Addressing the conflict between batched inference and sparse expert activation is therefore crucial
for unlocking the practical scalability of MoE architectures (Zoph et al. 2022 Liu et al. 2024b).
To address the problem mentioned above, various expert-reduction methods are proposed, which can generally be classified into static model compression and dynamic expert skipping. Static meth-
ods typically remove or merge experts in a fixed, pre-defined manner (Yang et al.) 2024; Liu et al. 2024c; Chen et al. 2025 Ai al. 2025). While these methods can efficiently reduce the memory footprint, they typically involve significant computational costs, rely on task-specific insights, and might reduce the model's capacity and ability generalize. Dynamic methods modify expert tivation at runtime based on token-level signals (Zhong et al. [2024] Huang et al. 2024 Lu et al.) 2024 Gupta al. 2024) Yang al., 2025b). These methods depend solely on router scores, over- look intrinsic expert characteristics, and often require extra training or threshold tuning. Moreover, their complex token-by- by-teen operations or modification of the decoding process hinder integration with high-performang inference frameworks, such as vLLM (Kwon et al. 2023), which limits their practicality in large-scale deployment.
Starting with these observations, we propose SERE, a Similarity-basee Expert Re-routing method for Efficient batch decoding in MoE models. SERE motivated by three key observations. First, many experts within an MoE layer exhibit high functional similarity. Therefore, SERE re-routes tokens from a subset of experts to their most similar counterparts, reducing the number active experts with minimal capacity loss. Second, small set of high-ranked primary experts dominate gating weights and output contributions, whereas secondary experts contribute little. SERE retains all primary experts and only re-routes secondary ones, thereby preserving dominant contributors while minimizing redundancy. Third, certain critical experts are highly dissimilar to others and specialize in unique input patterns. SERE preserves these experts prevent capability degradation during re-routing. In summary, SERE employs dynamic, input-aware strategy that jointly consid- ers token characteristics and inter-expert similarity, skipping more experts when redundancy is high and fewer when diversity is essential for accuracy. The expert similarity matrix is pre-computed once from a general calibration set, requiring no retraining or task specifie tuning. For deployment, efficient custom CUDA kernel for SERE that can be seamlessly integrated into
we implement the widely used vLLM framework (K et al., 2023), enabling plug-and-play use with only a single- -ine code change.
The contributions of our work are summarized as follows:
1. We propose SERE, similarity-based expert re-routing method for accelerating batch decoding MoEs. SERE significantly reduces the number of active experts while maintaining model performance, enabling faster decoding.
2. We develop an efficient, plug-and-ples CUDA kernel for SERE that works with various MoE models and can be easily integrated into the vL.LM framework (Kwon al. 2023
2

---

Published as a conference paper at ICLR 2026
3. We perform extensive experiments multiple state-of-the-art MoE models. (Bai et al. 2023
Liu et al. 2024a; Yang et al. 2025a). As shown in Figure 2 SERE achieves up to 2.0x speedup with minimal impact output quality.
RELATED WORK
Recent work on expert reduction can be mainly divided into two categories: static model compres- sion and dynamic expert skipping.
Static Model Compression methods leverage redundancy among experts to perform pruning or merging operations. For example, MoE-1² (Yang et al. 2024) reduces the size of MoE models via a two-stage process of inter-expert pruning and intra-expert low-rank decomposition. EEP (Liu al.) 2024c) employs evolutionary search that prunes experts and merges their knowledge into the remaining subsets. HC-SMoE (Chen et al. (025)) applies hierarchical clustering based on expert similarity iteratively merge similar experts. Other approaches, such as DeRS (Zhang
(025a), D²-MoE (Gu et al., 2025). and ResMoE (Ai et al. 2025), represent experts with shared weights augmented by low-rank residuals. While effective in reducing model size, these methods often incur high computation costs, rely heavily on calibration data and task-specife priors, and risk reducing the model's capacity and generalization ability due to decreased expert diversity.
Dynamic Expert Skipping aims to reduce the number of activated experts during inference dynam- ically. For instance, Top-p routing (Huang et al., (024) selects experts dynamically based on the confidence scores for each input. AdaMoE (Zhong al. (024) and MoE++ (Jin et al. (2225) enable token-adaptive routing via introducing null experts. Yang et al. proposes a layer-wise and fine-
grained top-k reduction strategy to improve inference efficiency. NAEE (Lu et al., 2024 skips less critical experts via token-wise analysis of router weights, and LYNX (Gupta et al. 2024 employs batch-aware confidence estimation filter out less relevant experts for unimportant tokens. While effective in reducing computation, these methods often require extra training, operate at coarse granularity, and overlook intrinsic expert characteristics by relying solely router scores. Their per-token operations also incur overhead and are challenging to integrate with high-performance inference frameworks, limiting their practical benefits in large-scale deployment.
3
METHOD
Original Route
SERE (Top-1)
Imput
Output
Reute
Esport
Pxpert
Top-2
Teken
Eeglts
Eapet
Prinuary Expert
Argmax
Kspert 1
Expert
Expert
Telon
Rouler
E
2
Input
Output
Expert
0
Expert
Stefleriy Matric
Expert
0
Legits
Expert33
Sccenalary Experi
Argmax
Kepert 3
Imput
Output
Xx
0
Telen
Legiss
Expot
Primary Expert
Rutain
Expert 4
Figure 3: Illustration of SERE with 4 tokens and experts as example. Tokens first routed to top-2 experts. SERE preserves the primary experts (1 and 4) and re-routes the secondary experts (2 and 3). As result, Expert 2 is replaced by Expert while Expert remains active as its similarity to all active experts falls below the threshold.
To accelerate batched decoding in MoE models, we propose SERE, a dynamic, input-aware expert skipping method. As illustrated in Figure SERE preserves the primary experts for all tokens as well as the critical experts within each layer, and re-routes tokens from secondary experts to their most similar retained counterparts. This dynamic strategy achieves substantial decoding speedups while maintaining model performance. the remainder of this section, we introduce the design motivations and technical components of SERE, We begin with expert similarity estimation (Sec.
3

---

Published as a conference paper at ICLR 2026
3.1
then describe the similarity- dynamic re-routing mechanism (Sec. 3.2 and finally present the implementation of a high-performance CUDA kernel for integration into large-scale
inference frameworks (Sec. 3.3
3.1
EXPERT SIMILARITY ESTIMATION
3.1.1 SIMILARITY MATRIX COMPUTATION
We adopt data-driven approach to measure expert similarity in MoE models. Consider an MoE model with L layers, where each layer contains M experts (E)" ..... EW) Using a calibration dataset Dealib, we process N batches and aggregate the results to obtain robust similarity estimates. For each batch tions obtained as A i1) i. E (x(-")). after which pairwise similarities computed via predefined similarity function Sim(...)
NJ, let X!" denote the input embeddings. In each layer expert activa-
P.g += Sim (A A
P
Common choices of Sim (:m-) (:..) include Cosine Similarity, Frobenius norm, and centered kernel align- ment (CKA) (Krrblith et 2019). More details can be found Appendix A.2
After all N iterations, the accumulated similarity matrices are normalized to obtain the average layer-wise similarity: (()) = /N The resulting set { provides a quantitative view of
the similarity relationships between experts within the same layer. High similarity values indicate potentially redundant experts, while low values reflect diverse expert specialization. The pseudocode provided in Algorithm [ in Appendix A.4
3.1.2 SIMILARITY MATRIX INSIGHTS
Layer
Layer
Similarity
Expert Indes
30
30
16
04
10
127
Expert Endoex
Experi Indes
Expert Indox
Layer Inder
Figure 4: Visualization of the expert similarity matrices and the average expert similarity across all layers in Qwen3-30B-ANe -AB (Yang et al. 2025a).
We computed the expert similarity matrices for all layers of the Qwen3-30B-A3B model (Yang et al., 2025a), with representative heatmaps and layer-wise average statistics shown in Fig. 4 The results reveal three notable patterns. First, within each layer, groups of experts exhibit consistently high pairwise similarity, indicating functional redundancy. Second, similarity patterns vary substantially across layers - Layer-1 has the highest average similarity, with nearly all pairs above 0.9, while Layer-6 has the lowest average similarity, with most pairs below 0.4. Third, every layer contains crit- ical experts whose similarity to all others is exceptionally low, as indicated by heatmaps that display distinct horizontal and vertical stripes. Even in Layer 1, Expert 92 stands out critical expert, with a similarity less than 0.1 to all others. These observations illustrate the balance between redun- dancy and specialization in MoE architectures, highlighting that certain experts contribute uniquely to model capacity while others may provide overlapping functionality. More visualization results of expert similarity matrices for different MoE models provided in Appendix C. These results demonstrate that high expert similarity is common in MoE models, regardless whether upcycling initialization (Komatsuzaki al. 2023) is employed.
4

---

Published as a conference paper at ICLR 2026
Key Insights:
Layer-wise redundaney: Within each layer, groups of experts exhibit high pairwise similarity.
Cross-layer variation: Average expert similarity varies substantially across layers.
Critical experts: Each layer contains critical experts with uniformly low similarity to others.
3.2
SIMILARITY-BASED EXPERT RE-ROUTING MECTIANISM
3.2.1 DESIGN MOTIVATION
To accelerate batch decoding in MoE models by reduc- ing the number of active experts, two key questions arise: (1) Which active experts should be skipped? and (2) How should they handled?
Top-k Expert Probability
For the first question, analysis of router weights distribution ( reveals that top-ranked (primary) experts dominate output activations and should therefore be retained, whereas low-ranked (secondary) experts contribute less and are nat- ural skip candidates. To address the second question, leverage insights from Sec. [.1.2] Because layers contain groups of highly similar experts. tokens from skipped sec- ondary expert can be re-routed to its most similar retained primary expert, thus mitigating disruption output activations. However, the analysis also identifies critical experts whose removal would de- grade performance. We therefore introduce similarity threshold that ensures such critical experts are always retained,
0.258
Percentage:
0.2
0.182
0.142
0.0
Top1
Top2
Top3
Top4
Figure 5: Weights Distribution
3.2.2 RE-ROUTING PROCESS
Building upon the observations and motivation, we now present our SERE method in detail. Let (-) denote the router function in layer of the MoE model. For a token E T. (t) (E!)) E. Ery.... E(I the ordered list of K experts selected for by descending router weight, and rk E (1,..,,,MM .M denotes the index of the k-th ranked expert.
Step 1: Primary expert selection. We identify the primary expert set in layer the union of the Top-S experts over all tokens in the current batch:
U (E lickss]
(2)
Here, S E K) is hyperparameter controlling the size of the primary expert set. Smaller S leads to fewer activated experts and higher acceleration, but may degrade quality. Experts in e/! are considered important and are always retained.
Step 2: Similarity-based re-routing for secondary experts. For each secondary expert E!!
(UleT (t)) E(!), we use the similarity matrix ((!) to find most similar primary expert:
sim max S()
" = arg max !!!)
u,
E.D "(
If sim, 2 where [0. is similarity threshold. we re-route all tokens originally assigned to E!!) to most similar primary expert Eet and preserved avoid unsafe substitutions. should be noted that the re-routing process does not modify the router weights. The formulaic expression is follows:
(()). If sim < p. E!!') determined as a critical expert
VG, E(" R(0(()) sim; 2 p
E
(4)
Step 3: Final execution. After re-routing, the final active expert set in layer / is: ( (()) U {D() sim; p). Ennal
(5)
which contains all primary experts and any preserved critical secondary experts. The MoE layer then utilizes this updated token-to-expert mapping produce the output activations.
5

---

Published as a conference paper at ICLR 2026
3.3 PICH-PEEFORMANCE KERNEL IMPLEMENTATION
We further develop perforeffrr,nc,, wardware-friendly and plug-and-play CUDA kemel for SERE. The implementation is model-agnostic, compatible with a wide range of MoE architectures, and can be integrated seamlessly into the vLLM framework (Kwon et al. 2023) without requiring modifications to its core execution pipeline. The pseudocode is outlined Algorithm@ Appendix.
In practice, this CUDA-accelerated SERE achieves substantial speedups in batch decoding while preserving model accuracy, making readily deployable both research and production environ- ments. Besides, enabling SERE requires only single additional line of code, ensuring effortless adoption in existing MoE inference pipelines.
4 EXPERIMENTS
4.1 EXPERIMENT SETTINGS
Models We evaluate SERE three representative MoE models: Owenl.5-MoE-A2.77 -aahh (Bai
et al. 2023), DeepSeek V2-LLte (Liu et al., 2024b), and Qwen3-30B-A3B YYng et al. 2025a).
Baselines We compare SERE against several SOTA methods, including HC-SMoE (Chen et al., 2025). Top-K reduction (Yang et al.) (025b), and LYNX (Gupta et al., 2024). All baselines are implemented using official code reproduced in strict accordance with the original papers to ensure a fair comparison.
Benchmarks For accuracy evaluation, we reasoning tasks from OpenCompass (Contribu- tors 2023) across three domains: Exam (CMMLU (Li et al.] (024). BoolQ (Clark et al., 2019). BBH (Suzgun et al. 2023)). ). Math (Math (Hendrycks et al. (202)) l) GSM8K (Cobbe et 2021
Math 401 Yuan et al.J () and Code (vumanEval (Chen et al. 202T), MBPP (Austin et al.,
2021
CoT mode is used for CMMLU and BoolQ. For acceleration evaluation, we measure Time
per Output Token (TPOT) under varying Queries per Second (QPS) using vLLM (Kwon et al.)
2023 with each model deployed on a single GPU. Input/output lengths are fixed at 128/32 tokens.
Hyper-Parameters We use the Frobenius norm the similarity metric and Web-FFe (Lozhkov al. (024) (400 sequencesx x 128 tokens) as the calibration dataset. For expert merging methods, pruning rates are chosen to match the TPOT of expert skipping methods for fair comparison. All experiments are conducted on NVIDIA H20 GPUs.
For more detailed settings, please refer to Appendix B
4.2 ACCURACY COMPARISON
We comprehensively evaluate SERE and competitive baselines on the aforementioned models and benchmarks, Table 1 and present both accuracy and per-token decoding latency (TPOT).
Methods \Tasks
Exam
Math
Code
Avg. (Ace. f)
TPOT
cmmlu
boolg
bbh
math
gsm8k
math401
heval
mbpp
(ms. {)
Qwen1.5 15-.2.7 togal
69.58
80.46
34.97
14.38
$1.86
60.60
45.73
30.60
48.52
17.29
Qwenl.5.. ..A.7BBBB
66.69
75.87
32.26
12.92
44.28
51.36
32.02
27.40
42.85
13.53
HC- HC-SMoE experis
45.11
74.95
29.01
4.26
27.67
42.64
4.88
1.80
28.79
14.20
LYNX Eup2
42.57
78.62
23.59
9.56
29.57
34.41
8.54
7.40
29.28
14.49
SERE 2op2; p=0.0
68.
80.15
33.16
14.06
50.19
58.35
46.95
26.20
47.15
13.83
SERE top2: p=0.3
68.49
79.97
34.61
14.66
51.63
58.35
42.68
27.60
47.25
13.93
Qwen1 Swen1.5-A2.72
A5A272BB top1
45.12
48.35
29.47
5.24
26.16
46.13
15.85
14,80
28.89
11.47
HC-SMoE 30 experts
7.93
34.19
29.14
1.72
8.95
29.18
0.61
0.00
13.97
13.30
LYNX topl
16.60
77.68
15.10
0.68
2.12
10.22
0.00
0.20
15.33
12.95
SERE Top1; p=0.0
60.09
79.85
32.71
7.58
33.36
52.12
17.07
20.20
37.87
12.13
SERE top3: p=0.3
65.83
78.69
33.62
9.74
39.88
53.12
17.07
20.60
39.82
12.95
Table 1: OpenCompass and TPOT (QPS=16) results Owenl.55-MBB-A2.7B. Bold for the best.

---

Published as a conference paper at ICLR 2026
Methods
\ (Tasks
Exam
Math
Code
Avg.
TPOT
cmmlu
boolg
bbh
math
gsm8k
math
heval
mbpp
(Acc.
(ms. 4)
DeepSeekV2 V2-Lite cops
53.34
82.39
49.37
23.82
59.14
70.32
$4.27
45.40
54.76
26.35
DeepSeckV24 V2-Litc copa
36.91
73.67
42.51
15.90
52.39
65.84
40.85
34.80
45,36
19,51
HC-SMoB 48 experts
39.74
80.70
41.97
9.16
47.92
45.14
10.98
7.00
35.33
22,36
LYNX top2
16.32
68.62
19.68
9.06
31.92
33.67
10.37
2.40
24.01
22.07
SERE 2ap2: pm0.0
53.13
82.11
48.67
23.04
61.03
71.07
56.10
45.80
55.12
21.60
SERE 2ny2: pm0.3
53.04
82.02
49.11
23.80
60.50
69.83
58.54
47.00
55.48
23.12
DeepSeekV2-Lite top1
19.41
58.90
33.81
2.56
17.82
48.88
7.93
7.60
24.61
18.02
HC-SMoE 32 exporss
26.51
63.06
33.48
0.94
6.29
13.97
0.00
0.80
18.13
20.28
LYNX Topl
2.16
49.91
3.96
0.14
1.29
2,00
0.00
0.00
7.43
20.00
SERB topl; p-0.0
53.81
82.11
48.69
23.74
58.53
72.32
57.93
45.40
55.32
18.54
SERB Eogl; p-0.3
53.49
82.63
48.90
22.94
59.36
71.32
59.15
47.20
55.62
20.59
Table 2: OpenCompass and TPOT (QPS=16) results on DeepSeekV2-Lite. Bold for the best.
Methods Tasss
Exam
Math
Code
Avg. (Ace. f)
TPOT
cmmlu
boolg
bbh
math
gsm8k
math.03
heval
mbpp
(ms. 4)
Qwen3-30BL -3BB top8
84.88
90.21
76.70
72.28
89.23
79.05
87.20
78.40
82.24
44.40
Qwen3-30B-A3B top2
10.01
60.52
10.48
3.38
6.97
16.96
3.66
2.40
14.30
30.97
HC-SMoB 80 experts
45.62
83.94
65.11
59.86
79.23
64,84
86.59
70.20
69 ,242
39.14
LYNX top2
81.36
90.12
72.27
69.10
80.44
76.81
84.15
73.40
78 .6
38.21
SERB top2; p.0.0
81.24
89.79
71.33
70.22
82.41
80.80
82.93
63.80
77.82
32.12
SERB top2; D.5.
81.51
90.37
74.15
72.06
85.97
81.55
85.37
72.00
80.37
32.82
Owen3-30B-A39 top1
0.00
61.68
4.89
0.08
0.91
1.25
0.00
0.00
8.60
27.28
HC-SMoB 48 experts
32.78
64.53
51.66
34.36
40.79
54.86
49.39
44.20
46.57
33.45
LYNX top]
70.76
88.26
59.08
44.28
48.37
47.88
55.49
46.00
57.52
33.38
SERE Eopel:
60.53
85.08
57,64
46.98
52.08
52.12
32.32
31.40
52,27
28.04
SERE Eogels
77.89
89.76
65.45
53.40
54.28
54.86
64.02
53.20
64.11
33.10
Table 3: OpenCompass and TPOT (QPS=16) results Qwen3-30B Bold for the best.
SERE consistently achieves the best trade-off between accuracy and inference efficiency. With aggressive expert skipping (c.g.. Top-2), SERE maintains over 97% of the original model's accuracy across all tasks, while reducing decoding latency by up to 1.6x on Qwen3 and 1.4x on Qwen1.5 and V2. In contrast, direct Top-K reduction yields the lowest latency but causes severe performance degradation (up to 90% accuracy drop), indicating significant loss of model capacity.
HC-SMoE and LYNX achieve competitive performance on Qwen3 but show significant accuracy drops on Qwen1.5 and DeepSeek V2, particularly for math and code tasks. This may stem from ar- chitectural differences: Qwen3 contains more fine-grained and redundant experts, allowing greater tolerance to merging or skipping, whereas Qwen and DeepSeekV2 have fewer, more special- ized experts and are thus more sensitive to expert selection. Methodologically, HC-SMoE's static merging reduces expert diversity, while LYNX ignores expert characteristics, thereby both impairing reasoning capability. In contrast, SERE incorporates both inter-expert similarity and the preserva- tion of critical experts into its dynamic skipping strategy, removing redundancy while safeguarding essential capacity, thereby delivering consistently superior performance across all models and tasks.
Furthermore, we can observe that SERE performs well even without preserving critical experts (p = 0), while preservation (p > 0) brings further accuracy gains with negligible latency. The similarity threshold provides fine-grained control over the trade-off between capability and speed.
4.3 ACCELERATION COMPARISON
In this section, we compare the acceleration performance of different methods across multiple mod- els and QPS settings. As shown in Figure 6 SERE consistently achieves substantial reductions in decoding latency under all evaluated QPS conditions. For Qwen3 and DeepSeck SERE yields
1.2x to 1.6x speedup, while for 1.e the acceleration ratio reaches up to 2.0 when QPS= 24, with almost performance loss (See Section 4.2 Moreover, the CUDA-implemented SERE de-

---

Published as a conference paper at ICLR 2026
Queal 5-A2.78
0P%
OPS 16
OPS
OPS
(1.3-1)
11.
132 MACEARTS
24., (RB01))
245
6-4
W-1
Rr1
K-
Kr
K-4
Kr1
DespSeekV2-Lite
-
OPS-
QPS-
QPS
132-0.2--
U1.
21.gg
(:311
Qu -000-00-
Opt
0P%
OPS
QPS
15.5
(1:3-1)
237
11.4
72
L3+1
112-11
L64
A0n6 X-1
XH
Te+
1e-
X-4
X-1
X00
Dasclino
LYNX
SIRE (PyTorch)
SERE (CUDA)
Figure 6: Batched Inference Latency between different methods in different QPS and Top-K.
livers approximately 1.5x speedup over the PyTorch version. Besides, shown in Figure 7c the additional re-routing overhead is negligible relative to expert computation and remains stable across batch sizes. These results confirm the efficiency of the custom CUDA kernel.
We further analyze the variation in the average activated expert count under different Top-K settings. As shown in Fig. 7a the count grows logarithmically with batch size for all Top-K values, and larger K consistently leads to more activations. The results indicate that the primary activated experts for different tokens are highly concentrated, which explains why SERE achieves significant acceleration. Furthermore, Fig. 7b shows that the inter-layer differences in activated expert count become more pronounced as K increases, highlighting the importance of dynamic expert skipping, where more aggressive skipping is applied to layers with higher activations.
We also examine how SERE behaves in the prefill stage, with details presented in Appendix
Batch
Attn
SERE
MI.P
16
115
6
137
24
117
6
186
32
119
6
227
Tip
119
233
(a) Activated No. vs. Batch Size.
(b) Activated No. vs. Layer No.
(c) Computation cost breakdown.
Figure 7: (b)&(b) Average activated expert count of Qwen3-30B-A3B under different Top-K: vari- ation with batch size and across layers (batch size=32). (c) Computational cost breakdown (J8) of key MoE operations for Qwen3-30B-A3B at varying batch sizes.
4.4 ABLATION STUDY
Ablation on Similarity Threshold We conduct experiments under both Top-1 and Top-2 settings, varying the threshold from 0.0 to 1.0, where p = 1.0 corresponds to the original model without any expert skipping. The resulting speedup and average accuracy for the three models are shown in Figure 3 Up to a point, increasing the threshold improves accuracy while adding only negligible decoding latency. Beyond that point, accuracy continues to rise, but the speedup drops sharply, in- dicating that too many active experts are being retained. In practice, the threshold at this inflection point offers good balance between accuracy and latency. For example, in the case of Qwen3-30B A3B, a threshold of 0.5 achieves this balance. We also notice that DeepSeek V2 maintains relatively stable performance across different settings, whereas Qwen3 and Qwenl 5 exhibit notable perfor-
8

---

Published as a conference paper at ICLR 2026
mance fluctuations. This finding highlights the substantial architectural and functional differences among different MoE models. More analysis on threshold can be found in Appendix C.1
Queca1_5.A2.78
DeepSeokV2-Litee
Qmcn3-308-AA38
Spadi
Spodup Krl
55
1.2
Scan
Soetla Erl
60
Threshuld
Threshld
Throbold
Figure 8: Speedup and Performance (Acc) under different similarity thresholds across models.
Ablation Similarity Matrix Computation We investigate how different similarity metries, parameter-based similarity measures, calibration datasets, and calibration data volumes used to com- pute the expert similarity matrix can potentially affect the overall performance and stability of SERE across downstream tasks. For similarity metrics, we compare Frobenius similarity, cosine similarity, and CKA-based similarity (Kornblith et al. 2011). For the data-free, parameter-based similarity computation methods, we follow Zhang et al. (2025b) and adopt two strategies for combining expert parameters: (1) Concat method that directly concatenates the three weight matrices {01, 022 0z), and (2) Logic method that constructs composite weight as 0g (0, 02). For calibration datasets, we use general datasets including FineWeb-Edu (Lozhkov et al. 2024).
(Raffel et al. 2020), and WIKI (Merity et al.] 2017), together with domain-specific datasets derived from specific domains (Exam, Math, Code) and the mixed domains (OpenCompass), For calibration data volume, we ex- periment with three configurations: 200 x 64, 400 128, and 800 x 256. Additional experimental details can be found in Appendix A.3 and Appendix B.4
Table that SERE is highly robust different similarity metries, and Frobenius provides the fastest calibration. By comparing Table 4 with Table [S] we observe that the parameter-basee sim- ilarity computation methods perform significantly worse than the activation-based methods. This suggests that capturing functional similarity through dynamic activations is more effective than com- puting similarity based on static expert parameters.
Table reports the performance of SERE under different calibration datasets and calibration data volumes. When K 2, the performance remains highly consistent across different calibration datasets and data volumes, indicating that SERE is robust with respect to both the type and the scale of calibration data. We also find that even when calibrated with Nooainn-specifi data, SERE maintains strong performance on other domains, which suggests that the similarity matrix captures transferable expert relationships rather than overfitting to particular domain. Furthermore, when K = 1, domain-speciff calibration provides slightly better results than general calibration, indicat- ing that in high skipping rate settings, using domain-speciff calibration data can further improve the performance of SERE.
Balancing calibration efficiency, effectiveness, and universality, we choose Frobenius Similarity and the Fine Web-Edu calibration dataset final implementation.
K=1
K=2
Time
Method
Exam
Math
Code
AVG
Exam
Math
Code
AVG
Cost (s.)
Frobenius
$7.55
31.02
18.64
37.87
60.48
40.87
36.58
47.15
28
Cosine
$7.90
26.57
17.16
35.96
60.49
38.57
33.62
45.55
75
CKA-RBF
58.39
31.29
19.98
38.62
60.94
40.84
34.74
46.85
16064
CKA-Poly
57.50
29.59
20.26
37.72
60.63
40.68
32.59
46.13
13459
-nn-rincar
58.12
29.77
19.46
37.83
60.57
40.31
35.18
46.62
541
Mcan-iStd
57.89 20.36 29.65
19.10-
37.60=99dz
60.62
40.25
34.54=1.54
46.464a.0
Table 4: Comparisons across different similarity metrics on Qwen O0wenl.5-MoE-A2.78B
Ablation on Re-Routing Methods We further evaluate model performance under three re-routing strategics: to the most similar expert, to random expert, and to the least similar expert. As shown in Table 7 re-routing the most similar expert consistently outperforms random expert selection,

---

Published as a conference paper at ICLR 2026
K=l
K=2
Combine
Metric
Exam
Math
Code
AVG
Exam
Math
Code
AVG
Concat
58.41
24.09
19.54
34.01
60.69
39.01
35.72
45.14
Concat
Cosine
25.38
13.70
32.61
60.76
39.88
34.32
44,99
Concut
CKA-L
30.20
18.54
35.66
60.76
40.06
33.82
44.88
Concat
CKA-R
58.34
30.53
19.77
36.21
60.67
39.55
32,60
44.27
Concat
CKA-P
58.73
30.52
20.26
36.50
39.60
30.37
43,59
Mean=
58.49 10.21
28x14:2.2
18.36
35.00
60.74 0.05
39.62 0.36
33.37 1.80
44.57 99.57
Logic
Frob
58.94
23.52
17.03
33.16
61.00
38.36
33,41
44.26
Logic
Cosine
58.89
29.19
17.74
35.27
60.61
40.43
31.69
44,24
Logic
CKA-I
58.02
28.91
18.55
35.16
60.72
39.52
31.77
44.00
Logic
CKA-R
57.76
28.55
17.13
34.48
60.57
38.24
35,34
44.72
Logic
CKA-P
58.42
28.49
16.92
34.61
60.89
40.29
32.60
44.59
Mean-Std Std
58,41
27.73+2213
17.47=0061
34.54+0.75
60.76-
39.37=003
32.96-1.34
44.36=0026
Table 5: Comparisons across different data-free similarity measures on Qwen 1.MMo.----
Calibration
K=l
K=2
Volume
Dataset
Exam
Math
Code
AVG
Exam
Math
Code
AVG
Fineweh
400 128
57.55
31.02
18.64
37.87
60.48
40.87
36.58
47.15
C4
400 128
57.42
30.82
17.54
37.85
60.87
41.21
35,24
47.09
WIKI
x00 128
57.64
30.93
18.34
37.90
60.70
40.92
35.65
47.02
Std
57,54
30.92
18.17 29.57
37.87 2.00
60.68 =..0
41.00=a.s
35.82:
47.09
Exam
x00x 128
58.20
33.34
22.88
38.14
60.80
40.26
35.34
45.47
Math
x00 128
58.15
32.48
23.50
38.04
60.88
40.25
36.43
45.85
Code
400 128
58.58
33.06
23.70
38.45
60.71
40.63
36.75
46.03
OpenCornpass
400 128
57.67
32.07
24.60
38.
61.16
41.65
37.78
46.86
Lea
98.15
32.74::0.0
23.67 :0.63
38.19
00.15
60.89 =a.e
40.70
36.58_.142
46.05
Fineweb
200
$7.54
30.93
17.54
37.56
60,82
40,20
35.12
46.66
Fineweh
400x 128
57.55
31.02
18.64
37.87
60.48
40.87
36.58
47.15
Fineweb
$7.95
31.88
17.53
60.44
40.87
34.34
46.58
Std
57.68_2023
31.28
17.90.2064
37.83
60.58
40.65
35.35_1.12
46.80_a.31
Table 6: Comparisons across different calibration datasets on Owenl.5-MMBEA2..
whereas choosing the least similar expert severely degrades performance. We also provide a theoret- ical analysis showing that similarity-base based re-routing method yields a tighter upper bound on output perturbation (Appendix A.5). These results demonstrate the critical role of the similarity matrix in guiding effective expert selection.
K=1
K=2
Method
Exam
Math
Code
AVG
Exam
Math
Code
AVG
Most Sim
57.55
31.02
18.64
37.87
60.66
40.87
36.58
47.15
Random
45.18
21.41
11.09
27.74
$7.12
34.91
28.66
41.68
Dis Sim
11.03
1.55
0.00
4.72
38.77
28.05
9.65
28.74
Table 7: Comparisons across different re-routing methods on Qwenl 1.eeo..5-Mo--2270
CONCLUSION
In this work, we investigate the challenges faced by MoE models during batched inference. We an- alyze the expert similarity patterns and activation weight distributions in MoE models. Building on the insights, we propose SERE, a novel method for accelerating batched decoding in MoE models. SERE dynamically re-routes tokens assigned to secondary experts toward their most similar primary experts, thereby reducing the number of active experts, while preserving critical experts to safeguard model capability. We further develop customized, efficient CUDA kernel for SERE. Extensive ex- periments demonstrate that SERE achieves up to 2x speedup with only a slight impact on model quality. Our study provides new insights into MoE inference optimization, highlighting re-routing as promising direction beyond traditional approaches such as pruning or quantization, and sets the stage for future work dynamic expert selection and efficient MoF: deployment.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported in part by the Natural Science Foundation of China (No. 62332002, 62425101), The Guangdong Grants (Grant No.20232T10X0755, and Shenzhen Science and Tech- nology Program (KQTD0024072100051666)
REPRODUCIBILITT STATEMENT
We have made significant efforts to ensure the reproducibility of our work. The full implemen- tation of SERE, including the efficient CUDA kernel, is available in the supplementary mate- rial and will be released upon publication. All experimental details, including model configu- rations, hyperparameters, and evaluation benchmarks, and Appendix B The expert similarity matrices were computed using standard metrics (Frobe- nius, Cosine, CKA), as described in Appendix [A.2] The calibration datasets Web, C4, WIKI, OpenCompass) are publicly available. Pscudocode for both similarity estimation (Algorithm I and the re-routing mechanism (Algorithm 2 is provided facilitate replication. Our results can be reproduced using the described setup, and all relevant code and scripts can be found in https://g www.:/tt...o.oo..o.ooooooooo www.eI- SERE
thoroughly documented in Section 4.1
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-1206 120b & gpt-oss-20b model card. arXiv preprint 2508 10925, 2025.
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Irrouuhput-Latengee tradeoff in [LLM} inference with "Sarathi-Serve) Implementation (OSDI 24), pp. 117-134, 2024.
18th USENIX Symposium on Operating Systems Design and
Mengting Ai, Tianxin Wei, Yifan Chen, Zhichen Zeng, Ritchie Zhao, Girish Varatkar, Bita Darvish Rouhani, Xianfeng Tang, Hanghang Tong, and Jingrui He. Resmoe: Space-efficient com- pression of mixture of experts Iims via residual restoration. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.I. KDD '25, pp. 1-12, New York, NY, USA, 2025, Association for Computing Machinery. ISBN 9798400712456. doi: 10. 041144//66002433701111 URL htt.ttsss///o.oorr/ert/t/t www.oogg1100.
3709199
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Temy, Quoc Le, al. Program synthesis with large language models. arXiv preprint urXiv:2108.07733: 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint wxXiv:2309.66600 2023.
I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, and Chun-Yi Lee. Retraining-free merging of sparse moe via hierarchical clustering. In Forty-second International Conference on Machine Learning, 2025. URL https: httpp::/opeereviiiee http////ppp//iii/ee//e/ee net fform?idd
hss02zRzzXL
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Mathhis Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021.

---

Published as a conference paper at ICLR 2026
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Papers), pp. 2924-2936 Minneapolis, Min- nesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18653/v URL https://aclanttol.ooo wwt.attoooooooo/....1.... 1300/
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint (t.::2100.14100 2021.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. http:://///thhabb..ooooooppeae
htwps/opppcccccccceeeee//s/ 2023.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseckmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint 0tff::2001000666 2024.
Hao Gu, Wei Li, Lujun Li, Zhu Qiyuan, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decompression for moe-based LLMs compression. In Forty-second International Conference on Machine Learning, 2025. URL
net
Vima Gupta, Kartik Sinha, Ada Gavrilovska, and Anand Padmanabha Iyer. Lynx: Enabling efficient moe inference through dynamic batch-aware expert selection. arXiv preprint arXiv:241 1.08982, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder task needs more experts: Dynamic routing in MoE models. In Lun-Wet Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 22883-12895, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 1/.188653///222..alll lo9...6666. 696. URI.https: / a/cll antho loyy . org/2024. acl vel-long.69e
Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04m 004088, 2024.
Peng Jin, Bo Zhu, Li Yuan, and Shuicheng YAN. Moe++: Accelerating mixture-of-experts methods with corpppoopttttion experts. In The Thirteenth International Conference on Learning Repre- sentations, 2025. URL
Dongwan Kim and Bohyung Han. On the stability-llastiiity dilemma of class-incremental learning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp- 20196-20204, 2023.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upeycling: Training mixture-of-experts from dense checkpoints. In The Eleventh International Conference on Learn ing Representations, 2023. URL http://penrrreeeeeeeeeferrrrreep
net
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMIR, 2019.
12

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang. Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {Grd: Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Represen- tations, 2021. httttttt/////ereeee wtt.rrrrrreeeeeeteooouudoddddddd www.::idddwwwwwwwwwdwwe
net forum?idui
Ang Li, Ben Liu, Binbin Hu, Bing Li, Bingwei Zeng, Borui Ye, Caizhi Tang, Changxin Tian, Chao Huang, Chao Zhang, et al. Every activation boosted: Scaling general reasoner to trillion open language foundation. arXiv preprint arXiv:2510.2211: 2025.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang. Hai Zhao, Yeyun Gong, Nan Duan, and Tim- othy Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 11260-11288 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18533/112024.11 wttp:///.......nnannno..... findings-acl.671. URL
findings-acl.66
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model. arXiv preprini wrxXvv220550443:: 2024a.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprini arXiv:2412.19437, 2024b.
Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs. arXiv preprint arXiv:2407.00945 2024c.
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL
BluginnFacceF.. fineweb eddu edul
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hong- sheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-expprts large language models. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11792 Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 10. ael-long. 334/
acl-oog..34 URI /tthtts:::/ //ac antho ogy. org/2///
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mix- ture models. In International Conference on Learning Representations, 2017. URL https:
//openzeview. B/t
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Evan Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk. David Wadden, Alexander Wettig, Binyuan Hui, Tim Deutmers Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. OLMoe: Open mixture-of-expprts language models. In The Thirteenth International Conference on Learn- ing Representations, 2025. URLhttps: http::////weeeeeeeeeememme. net/ htttt//oormmmmmddmxdm. 2id=xxTkbxxbxb TBmqq
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
7/jml wttp://ppppp//////pppa w21pe////w//////-/0 -74 html
13

---

Published as a conference paper at ICLR 2026
Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hin- ton, and Jeff Dean. Outrageously large neural networks: The sparsely-gad gatedd mixture-of- experts layer. In International Conference on Learning Representations, 2017. https::
//openreview. www.//frtommmee d=B1 ckMDqlg
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery. Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big- big-bench tasks and whether chain-of of of-thhhgh can solve them. In Findings of the Association for Computational Lin- guistics: ACI. 2023, pp. 13003-3351,, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397 20233.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
Cheng Yang. Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Cheng, and Bo Yuan. MoB-iP: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.). Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 104466-14466. Miami, Florida, USA, November 2024. Association for Computational Lin- guistics. doi: 10. 11w51/11/2022....../ hin.024..inniingnnnm......... URI http::htts///acattttat 2024. find ngs emn p. 612/
org/
Haoqi Yang, Luohe Shi, Qiwei Li, Zuchao Li, Ping Wang, Bo Du, Mengjia Shen, and Hai Zhao. Faster moe Ilm inference for extremely large models, 2025b. URL atttps://arx/l org/
abs/2505.0355
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. How well do large lan- guage models perform in arithmetic tasks?, 2023. URL htt.:/tps///ax/..oo/r//sss///
02015
Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P Xing, and Hao Zhang. Toward inference-optimal mixture-of-expert large language models, 2024. URL https: http//aawv....g/aaeea/aee
02852
Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for agnosagi pruning in sparse mixture-of-experts In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 86-102, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 979-8 89 76--256-5. doi: 20.18653/v1/2022 1/t.:/..............-... findings-acll... acl.4. URL Atttss:/aclaatthooogg..o org/ 2025. www.inniinnsa......m
Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experss In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 86-102, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8- 79176555- doi: 10.18653/v1/2025 12n.://i//....nndiinng...... acl.4. URL hwt.htpsss/accaanchllogg/ www.aanthhhooooooooo2eoo2ee 2025. findings-acl.
Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, Ru Huang, and Meng Li. Adapmoe: Adaptive sensitivity- bbasd expert gating and management for efficient moe inference. In Pro- ceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, pp. 1-9, 2024.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL
https://arxs owt/ 2ttp//aaaa/222....33 08900
14

---

Published as a conference paper at ICLR 2026
A APPENDIX ON METHOD
A. PRELIMINARIES ON MoEs
The MoE architecture enhances model capacity and computational efficiency by conditionally acti- vating only subset of parameters for each token (Shazeer et al. 2017). An MoE layer consists of set of expert networks {E1, ....... EM and a router network R. Given an input token embedding x, the router produces routing logits R(xx, which are transformed into probability distribution over experts. The output activation y of an MoE layer can be expressed as:
y Px()) Ex(x);
E(x) = (x gate) @ (x u) W down
(7)
where W gate, W up E Rd Xdm. W Woown E RRdmxdd a(.) denotes the activation function, @ denotes element-wise multiplication, and (z denotes the normalized routing weight assigned to expert
E. In practice, top-k gating strategy is often adopted to reduce computation. Specifically, only
the k experts with the largest routing logits are selected:
(x(x) (ootmax(Top-h((,(55)
(8)
MoE architectures achieve efficient scaling while maintaining strong performance, making them widely adopted paradigm in modern large-scale Transformer-batt models (Bai et al. 2023 Liu
al. 2024b Yang et al.) 2025a; Agarwal et al., (2255). Our proposed SERE method builds on
standard MoE architectures and changes only the decoding-timg routing targets, preserving all pa- rameters and layer structures.
A.2
EXPERT SIMILARITY METRICS
A.2.1 COSINE SIMILARITY
Given activation matrices Xg, XF E Raxd from experts E and F, the cosine similarity is computed by averaging the instance-wise cosine similarities between their outputs. For each input let XE and xp denote the i-th vectors of X and XF. The overall cosine similarity is computed by
(t) (8)
1
n2
(3)
Meos F,
72
12
A.2.2 FROBENIUS SIMILARITY
We measure Frobenius similarity between two experts E and F by first calculating the Frobenius norm of the difference between their activation matrices, and then normalizing this value by the maximum norm across all expert pairs. Let
FEFF = XE X ||r,
(10)
and let max(z) denote the maximum ZE.F among all pairs F). The normalized Frobenius similarity then given by
MIr
(E
JF.F
(11)
max(z)
This formulation ensures that the most similar expert pair achieves a score close to 1, while the least similar pair approaches
A.2.3 CENTERED KERNEL ALIGNMENT
Centered Kernel Alignment (CKA) Kornblith al. 2019) is widely used metric for quantifying the similarity between neural representations, as it is invariant to isotropic scaling and orthogo- nal transformations. CKA computes the similarity between two sets of expert representations by comparing their Gram matrices constructed with a chosen kernel function. In our experiments, we consider three types of kernels: linear, RBF (Gaussian), and polynomial.
15

---

Published as a conference paper at ICLR 2026
Given Xg. X5 E Raxd the CKA similarity is defined by
HSIC(Ke. KF) HSIC KB) HSIC KF)
McKa (E,
(12)
where KE and Kr are Gram matrices computed by kernel k(:,)):
Linear Kernel:
Kg XgXE
KF XxXF.
(13)
RBF (Gaussian) Kernel:
21
||x (4)
K exp
E
X
[KF exp
(14)
2x²
2x
where the bandwidth parameter.
Polynomial Kernel:
KEly = X
KF = (x(0)
(Q) ((1)
+
d
(15)
where is constant and d is the degree of the polynomial.
Here, HSIC denotes the Hilbert-Schmine Independence Criterion, which measures the dependence between two Gram matrices. For practical implementation, introduced by Kim & Han (003)). which provides (nn²) computational complexity.
the unbiased HSIC estimator as
A.3 PARAMETER BASED SIMILARITY COMPUTATION METHODS
data-free, parameter-based methods for computing expert similarities com- pare against the activation-based methods. Considering each expert consists of three weight matri- ces, namely 0, Wup- 02 W gate and 03 W down. we follow Zhang al. (2025b) and apply two parameter combination strategies to merge these weights:
We implement
Concat: The three weight matrices are directly concatenated: (01, 02, 0z). This method treats all weights equally without considering their functional roles in expert computation.
Logic: The three weight matrices are combined according to the computational structure of an MoE expert, expressed 0g (0, 0z). This approach reflects the structural dependency among the three components.
After obtaining the combined expert weights, compute the similarity matrices using the similarity metrics described in Appendix A.2 As discussed in Scction 4.4 the parameter-bes based methods perform noticeably worse than the activation-based methods. Although parameter-basse approaches are data-free, they are less effective at capturing the functional redundancy among experts.
A.4 PSEUDOCODE
We provide the pseudocode for expert similarity estimation (Algorithm and the CUDA-accelerated implementation of SERE (Algorithm 2 to facilitate readers' understanding of our approach. The pseudocode presents the key computational steps, helping bridge the gap between the conceptual description and its practical realization.
A.5 THEORETICAL ANALYSIS
In this section, we provide the theoretical justification that similarity-basedr expert re-routing can better preserve model capabilities.
Definition (MoE Layer Structure). Consider MoE model composed of k MoE layers, where the
i-th layer consists of M experts (E()) E ...., }. For an input E Rd, the layer output is convex combination:
(5)
M
N((2) = Σ
w (2)
( (z),
(16)
m=l
16

---

Published as a conference paper at ICLR 2026
where (1) (z) O and M Wm (z)
routing weights determined by router function. Let Do denote the input data distribution, and 'D; be the induced distribution of inputs to layer i, obtained by propagating samples a Do through the preceding layers.
Definition 2 (Expert Similarity). For two experts Ea. and Ea position in layer i, their simi- larity under the input distribution De is defined as:
5 5(E D EE-D, | E. (+)1l)
(17)
Theorem (Expert Substitution Error Bound). We consider replacing a single expert En in layer i with another expert E(i) while keeping all other experts and rouling weights unchanged, yielding modified layer No. Let F = NN o...oN1 11 be the original network, and F = N o...o ......,ooo...................... ... be the network with expert E replaced by E Assume each downstream module N2 for i + I....,k k is Lipschitz continuous with constant Lýj and define A 11 Lij Let wa the routing weight assigned to expert a. Then the substitution error satisfies
(1) (z) be
A.S.BE EAWD E ()
(x) (2) (1) A A(E. D..). ).)))
(18)
where the substitution error is
E(E EDWD [1[((x) F())]la]
(19)
Proof.
For any Do. let oN.)io ~ D. The layer output difference is
N((z;) - Ni(zz) =w! (z)) E (z.) (3.))
(20)
Let g = N o ... N1 which is A-Lipschitz. Then,
[[F(x) - F(x))l2 = (()()(2.)) - 9(V((2)))l
< A [[V((z)) -N,(=)ll
= A . ! (2)) [I -6. (x))l
(21)
Taking expectation over x ~ Do gives
E E
1) S A EN,DD,
w (o)) [B (2.) E
| (zi) ( (2))l2
< A D.
= A E (1) ((.))
(22)
where the second inequality follows from w:! (2;) < 1. This completes the proof.
This analysis shows that the error bound of expert substitution is jointly determined by the structural stability of downstream layers (A) and the similarity between experts (6(:.))). Therefore, under fixed model architecture, re-routing tokens to more similar expert yields a tighter upper bound on output perturbation. The above analysis provides theoretical support for the SERE method.
17

---

Published as a conference paper at ICLR 2026
Algorithm Expert Similarity Estimation
Input: Calibration dataset Pealib:
Number of iterations N;
Mixture-of-Expere (MoE) model with L layers, each containing M experts E ..... E Similarity function Sim(..)
Output: Layer-wise similarity matrices {8() E MMM1L li=1
for < 1 to do
! < O
I/ Initialize similarity matrix for layer /
end
for i 4- to . do
B +- the i-th batch from Dealib:
I/ Load calibration dataset I/ Input the first layer
(()) B;
for 1- 1 to L do
for j t M do
A + ; X (1 (1=
// Calculate activation for all experts.
end
for p 4- M do
for q t to M do
$ Sim AAp (!)
A (
I/ Accumulate pairwise similarities
5()) p. += 8;
S [g.p] += 88
I/ Ensure symmetry
end
end
( + MoE ( (x(-:));
// Standard MoE forward to get next layer input
end
end
for 1 <- 1 to do
5 (() /N;
// Normalize by number of iterations
end
return
(S
B APPENDIX ON EXPERIMENT SETTINGS
B. MODELS
We evaluate SERE on three representative MoE models: Qwenl.5 -.ool.--Mo-- Chat (Bai et al, 2023 VeepSeekV2-Lite (Liu et 2024b), and Qwen3-30B-A333 (Yang al. 2025a
1.5-MMo--AA277-hha. Each token activates shared experts and routed experts (out of 60) in each layer.
DeepSeek V2-Lite: Each token activates 2 shared experts and 6 routed experts (out 64) in each layer.
Qwen3-30B-A3B: Each token activates routed experts (out 128) each layer.
More details can be found in Table g
B.2 HYYPER-PARAMTTERS
For expert skipping, we evaluate two configurations that retain the Top-1 and Top-2 experts as the primary experts. For expert merging, select pruning rates that yield TPOT comparable to that of expert skipping methods, ensuring a fair comparison. For SERE, similarity matrices are computed using the Frobenius norm calibration subset of FineWeb-Fad (Lozhkov al. 2024) (400
18

---

Published as a conference paper at ICLR 2026
Algorithm 2 CUDA SERE
Input:
Top-K expert weights W E ERTXK;
Top-K expert indices ( E gT*k; Expert similarity matrix S E RRMMM
Retain count S E [1, K): Similarity threshold E []
Output: Re-routed expert indices P()) E g
I'O) - 11; H < Oar:
I/ Initialization
for f <- to T and
to S do
1 + 1;
I/ Mark current (primary) expert as retained
end
Reotal <- T X (K S); S);
// All secondary experts to be re-routed.
for each CUDA thread tid E [0, Riotal) in parallel do t tid/(K -S)]; k - S + (lid mod (K- - :
I/ Current token index
if 2 T or k K then return;
Corig f 4,ki
// Original expert
if H([orll =
then
10
1
t Corig:
No change if already retained // Init maximum similarity and best matched expert
continue:
end
Sbest -00, Ebest f- 0; for e < 0 to M - do
if HH(c] 1 then
Sourr t [eorig: e] if Scurr > Sbest then
// Pairwise similarity with retained experts
Sbest <- Scurrs Chest f- e
// Update best similarity
end end end
if p
> 0 and Sbest < then
r0 1 t Corig
I/ Keep original if below threshold
else
I T t,k Ebest ;
// Re-route the best matched retained expert
end
end
return [())
quences 128 tokens). The similarity matrices are normalized to [0, 1], where larger values indicate higher similarity between experts.
Tables summarize the main inference configurations for all MoE models studied in this work. For the SERE method, the parameters select_top.k and threshold are tuned according to ablation and experimental requirements. All calibration and experiments are performed on NVIDIA H20 GPUs
B.3 BENCHMARKS
For accuracy comparison, we select diverse set of complex reasoning tasks from the
benchmark (Contributors 2023), covering multiple domains: Exam (CMMLU (Li al. 2024).
BoolQ (Clark et al. 2019). and BBH (Suzgun et al. (023)); Math (Math (Hendrycks et al. 202
GSM8K (Cobee et al. 2021). ). and Math 401 (Yuan et al. 2023)): )); and Code (HumanEval (Chen
al. 2021), MBPP Austin al. 20211). Because CMMLU and BoolQ multiple-choice tasks, we adopt the CoT mode to evaluate the models' decoding capabilities. Details and examples of these
tasks provided in Table
19

---

Published as a conference paper at ICLR 2026
Model Config
Sweenl.5AA.7B--hrr.
PeepSeekV2-L.it
Qwen3-30B-A3B
Total Params (B)
14.3
16
30
Activated Params (B)
2.7
2.4
MoE Layers / Total Layers
24/24
26/27
48/48
Experts per MoE Layer
60
64
128
Activated Experts per Token
4 (selected) (shared)
6 (selected) (shared)
8
hidden size
2560
2048
2048
intermediate size
10944
6144
Vocahulary Size
151936
102400
151936
Inference Setting
Owenn.5-A2.7B-CCl..
DeepSeek V2-Lite
Qwen3-30B-A3B
Temperature
0.7
0.3
0.7
Top-p
0.8
0.95
0.8
Top-k
20
50
20
Repetition Penalty
1.05
1.00
1.00
Max Output Tokens
1024
1024
2048
Batch Size
16
Table 8: Main inference hyperparameters for each model.
For acceleration comparison, we measure the online inference speed of different models under var- ious methods using vLLM (Kwon et al., 2023). Each model is deployed record the Time per Outpul Token (TPOT, in ms) across different Queries per Second (QPS) settings emulate real-world service scenarios. The input and output sequence lengths are fixed at 128 and 32 tokens, respectively, and each test processes total of 5,000 requests.
single GPU, and
B.4 CALIBRATION DATASET
In this work, employ several calibration datasets estimate expert similarity within MoE mod- els, including three general datasets: FineWeh-Edu (Lozhkov et al.] 2024), WIKI (Merity et al.] 2017). C4 (Raffel al. 2020), and four Domain-Speeific datasets: Math, Code, Exam, and Open- Compass These calibration sets are used to perform forward passes through the model, collecting activation values for each expert every layer. The resulting activations are then utilized compute inter-expert similarity metrics, which guide subsequent rerouting strategies.
FineWeb-Edu (Lozhkov al., 2024) large-scale high-quality English web corpus designed for pre-training and evaluation of language models. It contains diverse and well-filtered content, calibration.
making it a representative resource for
WIKI Merity et al. 2017) refers to the English Wikipedia dump, widely adopted dataset in NLP research. Its encyclopedic coverage and high linguistic quality make suitable for calibrating models on general knowledge and formal text.
C4 (Colossal Clean Crawled Corpus) (Raffel al. 2020) massive web-crawled dataset filtered for high-quality English text. It is commonly used in large-scale language model pre-training and serves as robust calibration set for open-domain language understanding.
Math is domain-specine dataset constructed from Math (Hendrycks al. 2021), GSM8K (Cobbe) ct al. 2021), and Math401 (Yuan et al., 2023) within OpenCompass. We randomly sample prompts and answers from these benchmarks and shuffle them to form the calibration set.
Code is a domain-speciffe dataset constructed from HumanEval (Chen et al. 2021 and MBPP (Austin al.) 12021) within OpenCompass. We randomly sample prompts and answers from these benchmarks and shuffle them to form the calibration set.
Exam is domain-speciff dataset constructed from CMMLU at. 2024). BoolQ (Clark al, 2019) and BBH (Suzgun et al. 2023) within OpenCompass. We randomly sample prompts and answers from these benchmarks and shuffle them to form the calibration set.
OpenCompass combines the three domain- sppecifc calibration datasets above and generates the calibration data through uniform sampling.
20

---

Published as a conference paper at ICLR 2026
Task
Domain/Format
Description / Example
CMMLU
Exam / Multiple- Choice
comprehensive Chinese multi-subject exam benchmark with 57 subjects.
2024
(A) :H [5g (B) M (C) EXCEN (D) BAR
Example:
QoolQ
Clark 2019
Exam Multiple-
Reading comprehension questions with yes/no based on a passage.
Choice (Yes/No)
Example: Property tax Property tax or 'house tax" is local tax Is house tax and property tax are same?
BBH
Suzgun
Exam / Diverse
Big-Bew Big-Bench Hard, collection of challenging tasks covering logical, symbolic, and commonsense reasoning.
2023
Reasoning
Example: Which sentence the correct adjective or- der: \n(A) medium-size archaic prismlike purple American car) archaic purple prismlike American medium-size
Math (endryykks 2021)
Math / Open-Ended
A dataset of high school-level mathematical problems quiring step-by-step solutions.
Example: A positive multiple of 45 less than 1000 is ran- domly selected. What is the probability that it is two-digit integer? Express your answer common fraction.
GSMSK
Math / Open- Ended
Grade school math word problems with focus multi- step reasoning.
Cobbe
al
2021)
Example: Shiloh is years old today. In years, will be three times as old as his nephew. How old is his nephew today?
Math 401
Math / Open- Open-Ended
MATH 401 is a benchmark datasct specifically designed to evaluate the arithmetic capabilitics of large language mod- els through a variety arithmetic expressions and detailed performance analysis.
Yuan
al.
2023)
Example: 7.3542=*
Human Eval
Code / Code Gener- ation
Python programming problems requiring function imple- mentation based natural language description.
Chen
al.
2021
Example: Write function that returns the sum two num- bers.
MBPP (usstin et al.
Code / Code Gener- ation
Mostly Basic Python Problems: Short Python programming tasks with input-output examples.
202
Example:
Write function check if string is palin- drome.
Table 9: Overview of OpenCompass tasks used for evaluation.
For each calibration dataset, we randomly sample N sequences and select a fixed number of tokens (Length) from each sequence. FebeWee,Edu, WIKI, and C4 are used as general-purpose calibra- tion sets to evaluate SERE's performance under broad, diverse language phenomena, while Math, Code, Exam, and OpenCompass serve as task-specific calibration sets, aimed at testing whether downstream-orieme orienttd calibration data can further enhance SERE's capabilities, as well as the gener- alization stability across different domains.
21

---

Published as a conference paper at ICLR 2026
C APPENDIX ON EXPERIMENTS
C.I. DETAILED ANALYSIS ON SIMILARITY THRESHOLD
To better understand the relationship between the similarity threshold p and model performance, conduct a fine-grained empirical study on Qwen3-30B-A38 under K = 1 setting. Table T0 sum- marizes the performance across range of p values. The experimental results show that reasoning intensive tasks, such as mathematical problem solving and code generation, require higher similarity threshold compared with knowledge oriented tasks such exam. For example, when p reaches 0.5, the performance on Exam benchmarks is already close to the baseline, while the performance on Math and Code benchmarks still exhibits a noticeable gap. This suggests that complex reasoning relies more critically on high-fidelity expert routing than factual recall.
Threshold
emmlu
boolg
bbh
math gsm8k
math401
heval
mbpp
avg
TPOT
0.0
60.53
85.08
57.64
46.98
52.08
52.12
32.32
31.40
52.27
28.04
0.1
60.79
85.20
56.55
46.54
51.10
54.11
34.15
34.20
52.83
29.19
0.2
62.83
85,90
58.46
47.28
52.08
51.62
35.37
32.60
53.27
30.72
0,3
65. 65.24
85,60
59.17
47,90
53,37
54.11
39.63
32.40
54.68
29.21
0.4
72.11
87.61
61.78
48,56
53,30
54.61
45.12
34.20
57.16
29.81
0.5
77.89
89.76
65.45
53.40
54.28
54.86
64.02
53,20
64.11
33.10
0.6
80.77
89.91
71.33
59.56
63.00
63.34
83.54
68.80
72.53
34.28
0.7
80.92
90.31
74.87
70.24
88.40
82.04
86.59
74,20
80.95
35.37
0.8
84.08
89.94
76.10
70.92
89.39
81.30
86.59
76.60
81.86
38.92
0.9
84.33
89.82
76.66
72.42
89.61
79.05
86.59
75.60
81.76
46.02
84.92
89.82
76.62
72.46
88.93
81.30
88.41
78.00
82.56
44.54
Table 10: Performance of Qwen3-30B-NN 33B-A3B under K setting across different thresholds.
In summary, the similarity threshold serves as principled mechanism to balance efficiency and model performance. The empirical results suggest that setting p to moderate or high values signif- icantly improves performance on challenging tasks, primarily by eliminating part of detrimental set of low-similarity rerouting decisions.
C.2 DETAILED ANALYSIS ON PREFILLING STAGE
SERE is primarily designed to accelerate the batched decoding phase of MoE models. By reducing the number of activated experts, lowers the the memory-bound decoding process. Because it does not reduce the computation FLOPss, it is not expected to provide noticeable speedups in the compute-bound prefill stage. Nevertheless, to give more comprehensive understanding of SERE, we additionally conduct experiments evaluating its impact on the prefill stage, including its effect on prefill latency and the quality of the KV cache.
overhead and thus speeds up
We first evaluated the Time To First Token (TTFT) of three MoE models: Awen1.5-A2.7B. C--hat, Owen3-30B-A3B and DeepSeckV2 V--LLie,, under different QPS settings. As shown in Table []] SERE achieves slightly lower TTFT than the baseline, but the improvement is marginal. The results are consistent with our expectations and also indicate that our CUDA-based re-routing implemen- tation is highly efficient, introducing no additional overhead even when processing large number of tokens during the prefill stage. In a typical generation scenario (e.g., 128 input tokens followed by 256 output tokens), prefill accounts for less than 1% of the total latency, and this proportion will be even smaller when the outputs become longer. Therefore, we consider acceleration during the decoding stage to be substantially more impactful than acceleration during prefill.
We further examined whether SERE affects the KV cache generated during the prefill stage, since this could influence the quality of subsequent decoding. We first analyzed the proportion of primary experts among all activated experts under some typical batch settings. As shown in Table [2] for MoE models with fewer experts, such as Qwen 1.wel.55-2..77 5-A2.7B-Chh and VeepSeekV2-Lite, all activated experts are primary experts (100%). Even for Qwen3-30B- more than 80% of the activated experts are retained as primary experts. These results indicate that nearly all activated experts are preserved as primary experts during prefill. Besides, the small number of secondary experts that require re-routing can also find similar substitutes more easily because the pool of primary experts is large. As result, the impact KV cache quality is minimal.
that has larger number of experts,
22

---

Published as a conference paper at ICLR 2026
Model QPS
8
16
32
33.64
40.62
45.33
$1.43
SERE (K" 2)
33.57
38.53
45.24
50.24
SERE (K 1)
32.48
37.64
42.58
48.10
Qwen3-30BB-A3B
66.72
81.08
96.02
114.03
SERE (K 2)
65.09
78.52
92.69
104.36
SERE (K = 1)
64.94
78.62
92.25
107 44
DeepSeek V22-Lite
67.06
82.57
93.12
106.44
SERE (K 2)
66.03
79.99
91.10
108,23
SERE (K 1)
66.04
79.60
92.67
107.61
Table 11: TTFT(ms) under varying QPS settings.
Model / Batch Config
32x 128
16x64
4x256
Qwenl Swenl.5-AA 5-A2.7B-Chai
100%
100%
100%
Qwen3-30B-A.BB 94.53%
86.71%
81.65%
DeepSeekV2-LLit
100%
100%
100%
Table 12: Percentage of primary experts retained during prefill.
Methods \Tasks
Exam
Math
Code
Avg.
emmlu
boolg
bbh
math
gsm8k
mathso:
heval
mbpp
(Ace.
Twen3-330-A-e topis
84.88
90.21
76.70
72.28
89.23
79.05
87.20
78.40
82.24
Qwen3-30B-A3B top2
10.01
60.52
10.48
3.38
6.97
16.96
3.66
2.40
14.30
SERE p=ddlo
81.24
89.79
71.33
70.22
82.41
80.80
82,93
63.80
77.82
SERE (decode-only) tog2: p=0.0
80.31
89.42
71.81
69.60
82.41
80,80
84.15
63.60
77.33
SERE top2: pm0.5
81.51
90.37
74.15
72.06
85.97
81.55
85.37
72.00
80.37
SERE (decode-only) top2; am0.5
81.65
90.12
73.50
71.22
84.38
81.05
87.20
70.20
79.78
Sween--30--o0 topt
0.00
61. 68
4.89
0.08
0.91
1.25
0.00
0.00
8.60
SERE copil am0.0
60.53
85.08
57.64
46.98
52.08
52.12
32.32
31.40
52.27
SERE (decode-only) topl; p.0.0
62.96
85.02
57.56
46.32
50.95
$2.37
37.80
32.60
$1,20
SERE pm0.5
77.89
89.76
65.45
53.40
54.28
54,86
64.02
53.20
64.11
SERE (decode-only) topll p-0.5
78.68
89.82
65.60
52.48
53.68
53.62
66.46
51.00
63.34
Table 13: SERE vs. decode-only variant on Owen3-30BB-AB across OpenCompass benchmarks.
To directly understand how SERE affects the KV cache produced during the prefill stage, we im- plemented and evaluated decode-only variant in which all activated experts preserved during prefill and re-routing is applicd only during decoding. We tested this setting on the Qwen3-30B- A3B model across OpenCompass benchmarks, and the results shown in Table 13 Surprisingly, across different skipping rates and thresholds, the decode-only variant consistently underperforms the original SERE method. We consider this may be because inconsistent expert selection between prefill and decoding stages introduces a distribution shift that particularly affects reasoning tasks that rely stable internal representations.
In summary, although SERE does not provide significant acceleration during the prefill stage, it can be applied safely without degrading KV cache quality or overall performance.
C.3 SIMILARITY MATRICES VISUALIZZTION
In this section, we present detailed visualization of the expert similarity matrices for Qwen 1. 1.5-2.7B (Bai et al.) 2023). DeepSeek V2-Lite (Liu et al.) 2024a). Qwen3-30B-A3B (Yang) ct al. 2025a), DeepSeekMoE (Dai et al. 2024). Ling-mini-2.00 LLi et al. 2025). and OLMoE-IB. 7B-0125- (Muennighoff et al. 2025). as shown in Figure to Figure 14 respectively. These visualizations reveal that different MoE architectures exhibit distinct similarity patterns across lay- ers, i..., some layers display highly clustered experts with strong intra-gron group similarity, whereas others show more uniform or dispersed similarity distributions. Such layer-specific variation in- dicates that the functional roles and redundancy levels of experts vary not only between models
23

---

Published as a conference paper at ICLR 2026
but also across different layers within the model, highlighting the importance of layer-wise wise analysis when designing expert routing or pruning strategies.
D LLM USAGE STATEMENT
In preparing this manuscript, we used LLMs solely to aid in polishing the writing, such as improv- ing grammar, clarity, and readability. All substantive contributions to the research, including the conception of ideas, experimental design, data analysis, and so on, were made exclusively by the authors. The authors have thoroughly reviewed and taken responsibility for all content in the paper.
Layer
Layer 2
Layer 3
Layer 4
Layer
Layer
Layer
Layer
Layer
Layer16
Layer
Layer
Luyer:20
Layer
Layer
Layer
Figure 9: Visualization of expert similarity matrices DeepSeekV2 V2-Lite model.
24

---

Published as a conference paper at ICLR 2026
Layer
Layer 2
Layer3
Layer
Layer
80
Figure 10: Visualization of expert similarity matrices of 15-221.- model.
Layer
Layer 1
Layer 2
Layer 3
Layer
Layer
Layer
Layer 7
Layer
Layer
13
Layer
Layer15
Figure 1I: Visualization of expert similarity matrices of DA-MOE--B-77B-122--niiiii model.
25

---

Published as a conference paper at ICLR 2026
Layer
Layer 2
Layer 3
Layer
Layer
Layers
Layer 19
Layer 14
Layer 15
Layer
Layer
Layer 18
Layer21
Layer
Layer
Layer
Figure 12: Visualization of expert similarity matrices of DeepSeekMoF model.
Layer
Layer
Layer
Layer?
Layer
Layer
Layer10
Layer
Layer
Layer
Layer
Layer1s
Layer
Layer
Figure 13: Visualization of expert similarity matrices of Ling-mini-22 model.
26

---

Published as a conference paper at ICLR 2026
Layer
Layer1 1
Layer 2
Layer
Layer
Layer
Layer
Layer
Layert9
Layer
Layer 12
Layer 13
Layer
Layer
Layer16
Layer 17
Layer
Layer
Layer
Layer
Layer
Layer
Layer 24
Layer 25
Layer 26
Layer
Layer
Layer 29
Layer
Layer
Layer
Layer 34
Layer 37
Layer 31
Layer 30
Layer
Layer 41
Layer
Layer
Layer
Layer
Layer
Layer
Figure 14: Visualization of expert similarity matrices of Owen3-33B-AA3 model.
27


Reviews:
[{"review_id": "rv3HZEzBrM", "reviewer": "Reviewer_gTSr", "summary": "This work explores expert re-routing during the inference stage to reduce the number of active experts and thereby improve efficiency. The authors demonstrate a good trade-off between performance and efficiency with their simple yet effective approach.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This work tackles a critical and interesting challenge in MoE serving by employing a simple and intuitive mitigation method. \n- The authors' implementation, which features custom CUDA kernels, is good for its easy adaptation to vLLM. \n- Backed by several insightful observational experiments, the method achieves surprisingly good results.", "weaknesses": "- The method builds on the observation that experts exhibit high similarity scores. Does this phenomenon occur across all MoE models, or is it specifically a byproduct of using upcycling [1] as the initialization strategy?\n\nMinor problem:\n- It would be helpful to provide more details about the specific problem this work aims to address. At first glance, readers might assume the paper focuses on improving load balancing. Including key terms such as “expert eviction” could better clarify the intended scope and contribution.\n\n[1] SPARSE UPCYCLING: TRAINING MIXTURE-OF-EXPERTS FROM DENSE CHECKPOINTS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zMXJP0faQJ", "reviewer": "Reviewer_FvHj", "summary": "The paper introduces SERE (Similarity-based Expert Re-routing), an inference-time method to accelerate batched decoding in MoE models. The key motivation is that the many experts activated across a batch in the decoding stage create a memory-bandwidth bottleneck. SERE addresses this by observing that router scores are often concentrated on a few \"primary\" experts. It proposes to re-route tokens assigned to lower-ranked \"secondary\" experts to their most functionally similar primary counterparts. This similarity is pre-computed offline using a data-driven approach on a calibration set. The method is designed to preserve \"critical\" experts that are dissimilar to others, preventing catastrophic capability loss. The authors demonstrate SERE's effectiveness on several modern MoE models, showing up to a 2x speedup in decoding with minimal accuracy degradation on reasoning benchmarks. One major contribution is the release of an efficient, plug-and-play CUDA kernel compatible with the vLLM inference server.", "questions": "- In some MoE models, including Qwen3, there is a top-k weight normalization to adjust the total expert weights to 1. Do you keep this operation?\n- Would this method work for the prefilling stage?\n- It would be helpful to include standard deviation values to Tables 4 and 5 (AVG column).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors address a relevant problem that limits the efficiency and scalability of MoEs.\n- The idea of the method is simple yet meaningful. The results indicate speedups up to 2x with acceptable accuracy drop.\n- The authors provide an efficient kernel pluggable into vLLM.", "weaknesses": "**Reliance on and Sensitivity to Calibration Data:** The method's foundation is a similarity matrix computed on a calibration dataset. This introduces an offline computation step and a data dependency. While the ablations in Tables 4 and 5 show robustness to the choice of a general-domain dataset, the results also hint at a potential domain mismatch problem. For instance, the average similarity scores for the 'Code' domain are consistently lower than for others, suggesting that re-routing decisions might be less effective or even harmful for specialized domains not well-represented in the calibration data. The authors should (1) discuss this limitation and the potential need for domain-specific calibration, and (2) compare their data-driven similarity approach to a static, data-free alternative (e.g., using the cosine similarity of expert FFN weights) as a baseline. This would clarify the benefit of using activation-based similarity.\n\n**Focus on Decoding:** The paper's focus and evaluation are exclusively on the batched decoding stage. The authors do not discuss or evaluate SERE's impact on the initial prefill stage, which is typically compute-bound. It is not clear if the method would provide any speedup in this stage, and it could potentially degrade the quality of the initial KV-cache generation. The authors need to discuss this in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zLdUySvo5H", "reviewer": "Reviewer_2FF4", "summary": "The paper proposes SERE (Similarity-based Expert Re-Routing), a method to accelerate batch decoding in Mixture-of-Experts (MoE) large language models. The key idea is to identify and exploit functional similarity among experts to reduce redundant expert activation during inference. SERE precomputes an expert similarity matrix using a calibration dataset, then dynamically re-routes tokens from low-contribution (“secondary”) experts to their most similar “primary” experts while preserving dissimilar, critical ones. This reduces the number of active experts per batch without retraining or architectural changes.\n\nThe authors implement an efficient custom CUDA kernel compatible with vLLM, allowing seamless deployment with minimal code modification. Extensive experiments on several state-of-the-art MoE models (Qwen, DeepSeek) and reasoning, math, and coding benchmarks show up to 2× speedup in decoding with negligible accuracy loss.", "questions": "The similarity matrix is computed using a fixed calibration dataset (e.g., FineWeb-Edu).\n\nHow sensitive is SERE to domain shift or unseen data distributions? Would reusing the same similarity matrix across different domains (e.g., code vs. math) still maintain accuracy?\n\nAs I understand it, SERE first constructs a similarity matrix and then applies re-routing for the secondary experts. This is an interesting and promising idea. However, it would be valuable for the authors to discuss whether the dataset used to build the similarity matrix is representative enough and whether its distribution supports generalization. I do not consider this a weakness, but rather an opportunity for clarification and discussion.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The analysis in this paper is well-reasoned. It begins with a similarity analysis that reveals important insights into the relationships among MoE experts, which then guide subsequent research and design decisions. I like this style.\n\n\nThe kernel-level optimization and re-routing mechanism are well-aligned with system-level efficiency goals. This is an impressive aspect of the study. \n\nExtensive experiments on multiple MoE architectures (Qwen, DeepSeek) and benchmarks (OpenCompass, MATH, HumanEval) show up to 2× decoding speedup with minimal accuracy loss (<3%).", "weaknesses": "The work is mostly empirical and engineering-oriented; it lacks theoretical analysis of why similarity-based re-routing preserves model behavior or formal guarantees on capacity preservation.\n\nNo analytical insight is given into how similarity thresholds interact with model generalization or stability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)