Title: SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: MoE Model, Inference Acceleration, Batch Decoding, Expert Re-routing

Abstract: Mixture-of-Experts (MoE) architectures employ sparse activation to deliver faster training and inference with higher accuracy than dense LLMs. However, in production serving, MoE models require batch inference to optimize hardware efficiency, which may cause excessive expert activation and thus slow the memory-bound decoding stage. To tackle this issue, we present **SERE**, a **S**imilarity-based **E**xpert **R**e-routing method for **E**fficient batch decoding in MoE models. SERE dynamically reduces active experts in an input‑aware manner by re-routing tokens from secondary experts to their most similar primary counterparts. It also leverages similarity patterns to identify and preserve critical experts, thereby preventing capability loss. Additionally, we provide an efficient custom CUDA kernel for SERE, enabling plug-and-play use in vLLM with only a single‑line code change. Experiments on various complex reasoning benchmarks demonstrate that SERE achieves up to $2.0\times$ speedup with minimal quality loss, providing a practical solution for cost-efficient and latency-sensitive large-scale MoE deployment.

TL;DR: We propose SERE, a Similarity-based Expert Re-routing method for Efficient batch decoding in MoE models.