Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling
Authors: Yan Li
Zhenyu Zhang
Zhengang Wang
Pengfei chen
Pengfei Zheng
Venue: ICLR 2026 Poster (2026)
Keywords: Mixture of Experts
All-to-All Communication
Distributed Inference
Primary area: infrastructure, software libraries, hardware, systems, etc.

Abstract: Prevailing LLM (Large Language Model) serving engines employ expert parallelism (EP) to implement multi-device inference of massive Mixture-of-Experts (MoE) models. However, the efficiency of expert parallel inference is largely bounded by inter-device communication, as EP embraces expensive all-to-all collectives to route tokens to the remote experts if not collocating on the same GPU/NPU device. Nevertheless, state-of-the-art schemes treat expert device-placement and request (or token) device-scheduling as separate concerns, triggering excessive communication between them and compromising inference efficiency

This paper proposes Semantic Parallelism, a novel parallelism paradigm that minimizes the steep communication costs in EP-centric MoE serving via model-data collaborative scheduling. We implement Semantic Parallelism in a framework called Sem-MoE. Sem-MoE maximally collocates experts and their activating tokens onto the same device using proactively modeled activation likelihood between them and introduces three key techniques: (1) Offline model scheduling, which preliminarily clusters and collocates experts onto devices based on their co-activation tendencies for certain classes of input. (2) Online inter-request data scheduling for Attention-DP setups, which proactively rebatches incoming requests onto the device that hosts experts most likely and frequently activated by the corresponding requests. (3) Online intra-request data scheduling for Attention-TP setups, which seamlessly fuses a token reshuffling procedure into the original inference pipeline and proactively reschedules tokens to devices to reduce dispersed remote routing. We build Sem-MoE into a prevailing LLM serving engine SGLANG. Experiments show our collaborative scheduling approach can effectively reduce the all-to-all communication volume in EP and achieve superior inference throughput compared to existing solutions.

Published as a conference paper at ICLR 2026
SEMANTIC PARALLELISM: REDEFINING EFFICIENT MoE INFERENCE VIA MODEL-DATA CO-SCHEDULING
Yan Li"
Zhenyu Zhang"
Huawei Technologies 1iyan412ghuawei.
Sun Yat-Sen University
zhangzhy23 2mam i.1.1.2 12. sysu edu.
Zhengang Wang
Pengfei Chen
Huawei Technologies
Sun Yat-Sen University
vargz.eengangguuvei.coom com
chenpf7 @mail. s sysu, edu. en.
Pengfei Zheng!
Huawei Technologies
thennpengfeei@@@aaee..ccmm
ABSTRACT
Prevailing LLM (Large Language Model) serving engines employ expert paral- lelism (EP) to implement multi-device inference of massive Mixture-of-Experts (MoE) models. However, the efficiency of expert parallel inference largely bounded by inter-device communication, as EP embraces expensive all-to-all collec- tives to route tokens the remote experts if not collocating on the same GPU/NPU device. Nevertheless, faate-of-the-ar schemes treat expert device-placemen and request (or token) device-scheduling as separate concerns, triggering excessive communication between them and compromising inference efficiency
This paper proposes Semantic Parallelism, a novel parallelism paradigm that mini- mizes the steep communication costs in EP-centric MoE serving via model-data collaborative scheduling. We implement Semantic Parallelism a framework called Sem-MoE. Sem-MoE maximally collocates experts and their activating kens onto the same device using proactively modeled activation likelihood between them and introduces three key techniques: (1) Offline model scheduling, which pre- liminarily clusters and collocates experts onto devices based on their co-activation tendencies for certain classes of input. (2) Online inter-request data scheduling for Attention-DP setups, which proactively rebatches incoming requests onto the device that hosts experts most likely and frequently activated by the corresponding requests. (3) Online intra-request data scheduling for Attention-TP setups, which seamlessly fuses a token reshuffling procedure into the original inference pipeline and proactively reschedules tokens to devices to reduce dispersed remote routing. We build Sem-MoE into prevailing LLM serving engine SGLANG. Experiments show our collaborative scheduling approach effectively reduce the all-to-all communication volume in EP and achieve superior inference throughput compared existing solutions.
INTRODUCTION
The democratization of large language models (LLMs) has been largely driven by continuous model scaling. Over the past five years, the parameter count of the largest trained LLMs has increased by three orders of magnitude, posing significant challenges to the scalability and economic viability of both training and inference under modern AI hardware constraints.
"Equal contribution Corresponding author

---

Published as a conference paper at ICLR 2026
MOE Layer Breakotoon (mre)
(ali2all)
Comp, (grouped_gemee
10
16K
sequence length
Figure 1: Latency breakdown for DeepSeek-V2-Litt inference single MoE layer. Hardware:
8-GPU server with fast inter-GPU network (specialized connection with over 400GB/s bandwidth).
To mitigate these challenges, the coxture-of-Experee (MoE) architecture Fedus et al. (2022) Artetxe
iet al. (2022); Jiang et al. (2024) has been introduced. Unlike dense models, MoË models sparsely activate one or more expert sub-networks per input, enabling training of trillion-parameter models without compromising accuracy, while maintaining a sub-linear increase in computational cost. This approach has gained widespread adoption recent industriallstrrenthh LLMs, including DeepSeek- (T GPT-OSS GPT-OSS[OpenAL et al. 20255) the
V3 DeepSeckkAA (2024)) Deee/DDeeppeeR.
Qwen3-Series Yang et al. (2025). and Kimi-K2 Team et al. (2025).
However, at inference time, massive MoE models still require substantial GPU/NPU resources to compute, store, and load both expert and attention parameters. To achieve scalability and meet latency requirements, existing inference frameworks deploy multi-dimensional parallelism strategies that distribute experts and attention blocks across interconnected devices. An efficient parallelization scheme must effectively partition input tokens and model parameters, maximize resource utilization, and minimize communication overhead.
To address the memory demands of large-scale scale MoE deployment and leverage aggregate memory bandwidth, modern inference engines such as SGLang Zheng et al. (2024) and VLLM|Kwon et al. (2023) employ expert parallelism (EP), whereby experts distributed across devices. Attention layers are typically parallelized via data parallelism (DP) or tensor parallelism (TP). While EP enables parallel computation of experts across GPUs, it introduces significant communication overhead: intermediate activations must be dispatched from the gating module on a source GPU to the destination GPUs hosting the routed experts, and later combined back after expert computation. These operations often result in cluster-wide any-to-any token shuffling, typically implemented via two al12all collective operations (e.g., NCCL/HCCL's all2all).
Our analysis reveals that the inference performance of MoE models remains severely constrained by these costly all2all operations. For instance, a preliminary experiment running SGLang on the DeepSeek-V2-Litt model with 8 GPUs shows that EP communication accounts for up to 59.2% of the forward-pass latency the MoE layers, respectively-eeee even on high-speed interconnects (see Figure []. This bottleneck is further exacerbated on slower interconnects such as PCI-e or Ethernet. Therefore, systematically reducing EP communication has become a critical task for improving the efficiency and scalability of MoE inference.
In this paper, we demonstrate that the communication overhead of EP can be substantially reduced through novel semantic-aware dotel-data collaborative scheduling approach, namely Semantic Parallelism. This method forecasts expert routing paths for both requests and individual tokens, and proactively co-schedules tokens and experts to eliminate redundant communication. We implement the above idea in a system Sem-MoF, including two key techniques:
First, Sem-MoE performs offline model scheduling reduce expert dispersion. Experts that frequently activated together are clustered and placed on the same device or server based on predicted token-expert affinities. This grouping performed periodically offline to avoid runtime overhead.
Second, Sem-MoE employs online data scheduling to align input tokens with their corresponding expert groups. This includes: (1) Inter-request scheduling for based attention: dynamically batching requests to maximize expert affinity and minimize cross-device transfers. (2) Intra-request scheduling for Ta-based attention: proactively shuffling token activations during the TP commu- nication phase. Specifically, Sem-MoE replaces the standard post-attention a redduce ce with
We use GPU and NPU interchangeably in this paper.
2

---

Published as a conference paper at ICLR 2026
shuffled-reduce--eeeee and deferred shuffled-allgathee effectively merging proactive token routing with necessary data transformation.
Through collaborative model-data scheduling, semantic parallelism significantly reduces communi- cation volume and improves inference throughput, as demonstrated through extensive experiments implemented top of SGLang.
We list the contributions of this paper as follows.
1. We conduct comprehensive data analysis and reveal significant context-independet correlation between tokens and experts in large-scale MoE models, which provides foundational insight for optimizing expert placement and token routing.
2. We design and implement an efficient model-date collaborative scheduling algorithm that leverages the observed token-expert affinity. Our scheduler improves local activation rate by 15.4% compared baseline methods, substantially reducing unnecessary cross-device communication.
3. We implement semantic parallelism in Sem-MoE on top of the state-of-thee inference engine SGLang and perform extensive evaluations. The results demonstrate that Sem-MoF achieves a throughput improvement of up to 2.78x under specific SLOs in Attention-DP scenarios and up to 24.9% latency reduction under Attention-TP setups, validating the practical effectiveness of our approach.
2 BACKGROUND
Mixture-of- Exxeeerts The Mixture-of-Exppee (MoE) architecture is conditional computation
paradigm designed to scale model capacity without a proportional increase computational cost [1]. Unlike dense models, where all parameters are activated for every input, MoE model consists of multitude of expert sub-networks (typically Feed-Forward Networks, FFNs) and a gating network (or router). For each input token, the gating network predicts a sparse combination of experts (c.g.. the top-k experts) to which the token is dispatched. Only the selected experts are activated for computation. The most common gating function is the Top-K Gating, which selects the experts with the highest scores. This design enables models to possess vast number of parameters (e.g., trillions) while keeping the FLOPs per token roughly constant, as only a small, fixed number of experts (e.g.. k = 2) are active token. This has made MoE the de facto standard for building state-of-the-art large language models, such as the DeepSeek series DeepSeek- AT (2024a b) 2025)) the GPT-OSS series OpenAl et al. (2025), and the Qwen series Qwen-Team (0024) Yang al. (2025).
MoE Training Systems. There has been extensive research on optimizing systems of MoE training systems, including BastMoElae He et al. (202I), PasterMoEHHe et al. (2022). Then et al. (2022 SmartMoE Zhai et al. (203)) ,, and FlexMME Nie et al. (2023). However such optimizations can not directly translate to inference scenarios inference Workload-sensiteee and strongly emphasizes
latency over throughput.
MoE Inference Systems. Integrated serving engines such DeepSpeed-MIl Holmes al. (2024). TensorKT-LLM NVIDIA] vLLM/Kwon et al. (202)), and SGL.agg ZZeng et al. (2024) have holistic optimization for LLM inference that spans serving schedulers (e.g., continuous batching), dedicated high-perffrmmance kernels, efficient parallelization, quantization, and elaborate compiler passes for graph-level optimizations. Built upon these general holistic optimizations for LLM inference, DeepSpeed-Min -MoE Rajjbaandar et al. (2022); Singh et al. (2023) and Tutel Hwang et al. (2022) specifically optimize MoE models" computation and communication. Following the design paradigm of DeepSpeed-Moo popular industry and open-sourced inference engines like vL..M and SGL also adopted expert parallelism deployment. Sem-MoF specializes in optimizing MoE parallelization (particularly EP) and inherits holistic optimizations from prior work.
MoE Load- balanein and Experts Re-grouping. Lina Li et al. (023)) probes the variation of expert hotness and allots non-uniform expert replicas to achieve load-balanced expert computation. Similar studies Huang et al. (2023) exist to pursue expert load balancing and mitigate other sources of MoE computing inefficiencies. EPS EPS-MoE Qian al. (2025) optimizes the computation of MoE FeedForward Network (FFN) modules by dynamically selecting the best backend implementation of GroupGemm and DenseGemm. DeepSeek also adopts EPLB pexpert-paaaleelism load balancing)
3

---

Published as a conference paper at ICLR 2026
in its real-world deployment DeepSSek-A (2024b). ExFlow Yao al. (2024) exploits the affinity between experts across adjacent layers to reduce remote routing and collocate closely related experts. Exflow only considers the model scheduling for MoE models, and requires heavy a sllgather before the execution of each model layer, which significantly incurs memory pressure and extra communication overhead. MoFTuner & Mahajan (025) optimizes the MoE model serving by finding an optimal expert placement strategy to minimize inter-device communication.
MoE Offloading and Prefetching. Existing prediction-based work on MoE inference primarily focuses on prefetching offloaded experts and strategically saving GPU memories Yi et al. (2023); Xue et al. (004); Zhong al. (2024), though offloading can extend inference latency and is rarely used in latency-critica serving scenarios. In contrast, Sem-Mer Mom-MoE focuses on the speculative reduction of communication overheads and exposes no risks to compromise latency. The work also constructs probabilistic models to predict the token-expert routing paths, while Sem-MoE's features modelling more comprehensive MoE information, i.e., intra-layer and inter- inter-layer expert affinity and token- expert affinity, compared to prior work. Pre-gated MoE Hwang et al. (2024) modifies the MoE model architecture to predict the experts to route at the next layer. Sem-MoE requires no modification to MoE architecture.
@ Offfine Token-Expert Activation Modeling
Requests:
Scheduling (Inter-requasl)
Offine Collected Profile
Offinne Aftirt Modeling
Request Schoduler DP)
Taken IDs Taken
Activation Map
T20 Table J' 820 Table Jl g
Dey 0
Dey
peer Artn
Adin
Scheduling
Si Sruffid--Racooiee Scalter
@
Data Scheduling (Intr-rrquest)
F2D Twekk & Bisena
Shuffled-A Allgthe
TP)
Figure 2: The workflow of semantic parallelism in Sem-MoE.
3 METHODOLOGY
3.1 SEMANTIC PARALLELISM: OVERVIEW
Figure 2 iillustrates the overall workflow of semantic parallelism in Sem-MoE. Solid and dashed lines in the figure represent online and offline operations, respectively. The process begins with collecting token activation profiles, which include token identifiers and token-expert activation frequencies (Step ! in Figure Based on these profiles, Sem-MoB probabilistically model the token-expert routing likelihood and formulates balanced token-expert co-clustering problem to generate scheduling hints. These hints are materialized as lightweight lookup tables: a ceaenttttexpee--eee cerenneett-expeeee co--xpper-groonn group table A. and an expert grouping table E.
table 7 an
These scheduling tables drive the subsequent collaborative model-data scheduling. In the model scheduling phase (Step 2), Sem-MoE utilizes the expert-to-device table to reconfigure the place- ment of experts across all layers. In the data scheduling phase (Step ),, different policies are applied depending on the parallelism strategy of the attention layers:
For attention layers deployed with Data Parallelism (DP), Sem-MoE employs inter-request data scheduling. This policy reorders incoming requests according to the token-to-device table T to maximize saquest-exper-ggnee nication overhead across DP domains.
affinity, thereby reducing the all2all commu-
For attention layers partitioned via Tensor Parallelism (TP), Sem-MoE adopts intra-request data scheduling. This technique proactively shuffles tokens during the post-attention reduce scatter operation, directing them to devices predicted to host their target experts in advance, thus minimizing potential token redistribution in subsequent MoE layers.
We use expert group and expert cluster interchangeably.
4

---

Published as a conference paper at ICLR 2026
Case #1: Data Parallelism for Attention
Tensor Parallelism Attention
Dov
EIGBLO
Dev
INU
Dev
Dev
ROod
Comtonee
Dev
AAR
ARR
ETOOK
Figure 3: An illustrating example of Sem-MoE. In Case #1 (Attention with DP), Sem- Gem-MoE resched- ules requests and adjusts expert placement in Layer 1, reducing the number of remotely activated tokens (Remote activated tokens refer to the total number of tokens dispatched and combined from remote devices) from 16 to 4. In Case #2 (Attention with TP). token rescheduling via hefffledrreducee preduce-scatter and expert repositioning in Layer I reduce remote token acti- vations from 12 to 2.
Figure 3 provides a concrete example of Sem-MoE's operation. In the baseline of Case 1 (top row), requests are distributed across DP ranks for independent attention computation. After expert assignment, tokens dispatched to their respective expert devices via all2all operations. With Sem-MoE's inter-request scheduling, requests are intelligently mapped to DP ranks to enhance data locality, reducing a 12all volume. This effect is further amplified by complementary model scheduling that optimizes expert placement. In Case 2, which involves TP for attention, tokens require reduction before dispatch and gathering after combination. Sem-MoE's intra-request scheduling predicts expert routes prior to the gating module, allowing tokens to be shuffled and scattered via (SRS) operator. Combined with model scheduling, this approach achieves a higher local activation rate, significantly cutting down a all2all traffic.
customized shuffled- weweefeedrrdduueeseeeeeeee
The inference acceleration achieved by Sem-MoF. stems from the increased local activation rate enabled by collaborative model-data scheduling. Let G denote the number of devices, B the global batch size, S the sequence length, and k the number of experts activated per token, The communication volume of an all2all operation is given by akBS G where represents the fraction of non-local activations. By maximizing the local activation rate (i.e., minimizing a), Sem-MoE effectively trims communication overhead. The subsequent sections detail the offline modeling and online scheduling algorithms.
3.2 PREDICTING EXPERT ROUTING PATH
The routing choice of each MoE layer is given by the gating function: G. top-k(softmax/ ( Light. + bl.g)). Accurate prediction of token-expert routing patterns in advance is fundamental to Sem-MoE's scheduling optimization.
Lndeex--nneeenneent Token Activation Prediction. We observe that despite the theoretical depen- dence of expert routing on contextual semantics (as expressed by the gating function G2 (hz.g)), )), in practice, tokens exhibit strong context- inddtddddddddnn affinities to specific experts. Table Aabseeessssntss three metrics illustrating the activation concentration within DeepSeek- DeepSeek-V2-Lite. V2-Lite. By profiling the model on
the ShareGPT Datasers (2023) dataset, we record the intermediate top-k activation statistics. We then calculate the F1-score for predicting the activated experts in each layer by relying solely on the most frequently activated (hottest) top-k experts. Additionally, we compute the cumulative hotness of these top-k experts alongside the maximum hotness observed among the remaining non-top-k experts. For all evaluated metrics, the P25, P50, and P75 quantiles are reported.
For any specific token, the distribution of expert activation across diverse contexts is remarkably skewed yet stable: a given token consistently routes narrow, static subset of top-k experts, leaving the majority of experts largely dormant. Throughout all layers of DeepSeek- V2-Lite, the median (p50) cumulative hotness of the top-k experts ranges from 0.833 0.976. In contrast, the maximum
5

---

Published as a conference paper at ICLR 2026
Table 1: Empirical study of token activate patterns based DeepSeek- V2-I.ite.
F1-score
Cum. Hotness (Top-k)
Max. Hotness (Non-top-k)
Layer ID
p25
p50
p75
p25
P50
p75
p25
p50
p75
1.000
1.000
1,000
0.833
1.000
1.000
0.000
0.042
0.071
0.833
1.000
1.000
0.767
0.917
1.000
0.000
0.056
0.083
0.833
1.000
1.000
0.722
0,889
1.000
0.000
0.056
0.083
0.667
0.833
1,000
0.708
0.889
0.000
0.053
0.083
0.667
0.833
1,000
0.667
0.833
1.000
0.000
0.050
0.074
26
0.667
0.833
1.000
0.667
0.833
1.000
0.000
0.051
0.083
hotness recorded among the cold experts remains -ooerinn near 0.05 at p50-which highlights robust and pronounced disparity in activation. Finally, $(A.2 in the Appendix expands upon this phenomenon across more scenarios, confirming its broad generalizability.
The above phenomenon enables effective prediction based solely token identity. Through offline profiling on datasets such as Sharegpt using models including DeepSeek-V22 V2-lite and Qwen3-30B-
A3B, we construct token-to-expert activation table T(L) E NUxN(*) for each MoE layer L, where T(LL) j.k counts how frequently token z; activates expert E The corresponding routing probability is: (z)) Ei=1 v(44) T(L) -=
For efficient online inference, these probabilities are tabulated in a token-to-expert confidence table Cp R Out-of-vocabulary tokens are handled via nearest-neighhor matching in the embedding space.
The token-level predictions form the basis for scheduling in both Attention-DD and Attention- TP scenarios. For Attention-DP, the affinity of an entire request to expert group is derived by aggregating the predictions of its constituent tokens, enabling request-level scheduling. For Attention-TP, the fine-grained token-level predictions are directly utilized, and are further refined by modeling inter-layer dependencies, as discussed in Section 3.3 This predictive framework provides the essential guidance for Sem-MoE's collaborative scheduling optimization detailed next.
3.3 MODEL-DATA COLLABORATIVE SCHEDULING
We illustrate how such expert-routing forecasting models can guide the co-dispatching of tokens and experts. Sem-MoE formulates the model-data co-scheduling problem as 0-1 integer programming (ILP) -based co-clustering problem.
From the offline profiling, we obtain the number of deduplicated (un-deduplicated) tokens / (S), the number of experts per layer N, the number of clusters E' (also EP degree), the token frequency a,, and the activation probability Cp.sk that token j activates expert kt The decision integer variables are set as the routing R, E {0, 1} of token j cluster i, and the placement Cij {0. 1) of expert j to cluster i.
We aim to minimize an objective function & = o E E/ (Rija,)
+
0) Engtio EE (R.,, Ci CighCnsaaa) where the left part is to ensure that the token fre- quencies of different clusters as even as possible to promote load balancing among EP ranks, and the right part is to minimize the all2all communication overhead caused by remote activation (i.e., summation of all the activations of tokens and experts belonging to different clusters), a factor 0 E (0, 1) controlling the percentage of two sub-objectives. We further require that each token be- longs to only one class, each expert belongs to only one class, and the number of experts in each class
is equal by adding hard constraints D Rig = 1, for = = 1...t, D C₁ = 1, for j = I.
and Σ C.j E: 1
The above ILP problem is difficult to solve directly using LP solvers, given a large number of intermediate variables introduced in the linearization process. Sem-MoE provides an alternating optimization algorithm. It can quickly obtain feasible solution while ensuring load balancing. The detailed co-clustering algorithm can be referred to in $ B the Appendix. The solution can then be applied to offline model scheduling and online inter-/intra-request data scheduling.

---

Published as a conference paper at ICLR 2026
Model scheduling. Before deployment, Sem-MoE adjusts the expert placement layout according to the solved C, placing expert j to device l 5 = 1. Accordingly, Sem-MoE shuffles the column of the gate matrix, thereby realizing a transparent expert re- re-distribution.
Data scheduling: Attention-DP Scenarios. In Attention-DD setups, where requests are processed independently across DP ranks, scheduling operates at the request granularity. We use the variable S, E [ to denote the cluster to which request r needs to be scheduled. Once the scheduling of tokens is determined, the scheduling of the request can be determined by aggregating the routing result of its tokens, Sr = arg [E]] Eier Ray. Meanwhile, to achieve runtime load balance. Sem-MoE realizes a workload- aware balanced request scheduling algorithm. For continual E' requests, Sem-MoE guarantees these requests are distributed to all E ranks, such that the loads of all ranks would not skew decoding stage. Detailed algorithm could referred to Algorithm 2 in $ [ This request-level scheduling minimizes cross-device communication by collocating entire requests with their most likely expert group (i.e., DP rank).
Attention-TP Scenarios. In Attention-TP setups, the attention computation itself is distributed, requiring fine-grained, token-level scheduling. Here, we enhance the basic token- expert prediction with inter-layer expert-expert affinity. We observe that expert selections exhibit Markovian dependencies across layers: the experts chosen at layer L depend on selections at previous layers. We model this using an n-gram device transition model: Pr(D). ( ..... D(L-") where D {L..... Q}. where D( 1..... Q denotes the device index of the expert selected at layer These transitions are stored in an wttrggrrrppseeeuee-oooeeeeeeeeee Ap (we 2-gram in practice). Together with the token routing matrix R, can achieve more accurate proactive scheduling during the TP communication phase. The detailed algorithm can be found in Algorithm
Data Scheduling:
confidence table
3.4 IMPLEMENTATION AND SYSTEM OPTIMIZATION
Sem-MoE is implemented as a plug-in module for the SOTA LLM inference engine SGLang. Our system comprises approximately 5,000 lines of Python code, along with several custom Triton OpenAl kernels for high-performanee communication operations.
To support lffinity-awarr scheduling in the Attention-DD scenario, we extend SGLang's request scheduler to incorporate token-expert affinity information derived from our prediction models. This enables the runtime to batch requests with similar expert activation patterns onto the same device, minimizing cross-device communication.
For the Attention-TP scenario, we implement two fused communication primitives: wew.leedrrduuc--cctttt
(SRS), and shuffled-allgee nels integrate speculative token shuffling-bar on predicted expert routes-into standard reduce-scatter and allgather collectives. The shuffling logic relies on optimized argsort kernel, which outperforms the native PyTorch implementation by 25%. The overall overhead of embedding shuffling into the ring-hased communication schedule is negligible, measured at approximately 1 %. Furthermore, for efficient a 12all,, Sem-MoF. integrates frontier MoE communication libraries DeepEP Zhao al. (0225).
(SAG). These ker-
By combining offline expert reorganization with online token- and request-level scheduling, Sem- MoE achieves significant reductions in al [ll2aal] communication volume, leading improved end-to-end inference throughput in both DP and TP configurations.
4 EXPERIMENT
4.1 EXPERIMENTAL ENVIRONMENTS
We evaluate Sem-MoE 8-GPU server, representing commercial GPU servers unified for both training and inference, which are configured with 96GB-HBB -BM per GPU and fast homogeneous interconnects. GPUs inside a server can communicate with each other at a premium bandwidth (specialized network with over 400GB/s bandwidth). The server is equiped with two 44-core Intel CPUs and 2TB DDR5 memory.
7

---

Published as a conference paper at ICLR 2026
SeLang Throughput
McEtuner
Sem NoE
35Larg
Hoatuner
Som MaE
EZE
300
3.25
3.50
403
5L0
3.25
200
510
250
4.0
200
MW
300
20
150
200
Grande
300
12
Shawarr
300
14.
200
10
200
500
1000
1500
2000 2500
500 1000
1500
2000
2500
200
400
600
800
1000
200
400
600
800
1000
Thrughhut
Thougghuu
Through
Throughput
(tokens / second)
(tosens second)
(tokers) second)
stokkes// second)
(a) DeepSeek-V2-L V2-LLie.
(b) Qwen3-30B-A3B
Figure Attention-DP Scenario: Inference throughput under TTFT and E2E latency SLOs.
4.2 MODELS, DATASETS AND WORKLOAD TRACES
Models. We choose two types of typical MoE models for evaluation, i.e., Qwen3-30B-A3B with 128 experts per layer and DeepSeek-V2-1 V2-Lite with 64 routed experts per layer. Both Qwen3-30B-A3B and DeepSeck-V2 V2-Lite are prevailing open-sourced MoE models.
Datasets. We use the following three representative datasets MMLU Hendrycks et al. 202 lb a), Imsys-chut-Im224 ZZenng et al. (2023). only focus the prompt parts of these datasets. These datasets contain data from different domains, representing real-world user request patterns, and can effectively evaluate the affinity of requests and tokens from different domains for experts.
Datasets (2023). the experiments,
4.3 BASELINES AND PERFORMANCE METRICS
We select SGLang and MoETuner as the baselines compare, representing the SOTA LLM inference engine and SOTA MoE model scheduling technique.
SGLangg SGLang Zheng et al. (2024) a prevailing open-source LLM inference framework, incorporating numerous optimizations, including but not limited to continuous batching, paged attention, flash attention, radix-attention. advanced quantization, etc. SGLang declares optimizations for MoE models with nigh-performance, fused triton kernels OpenAl, supporting DP and TP for attention layers and EP for MoE layers. SGLang is the SOTA open-sourced LLM inference engine, which we set strong baseline to compare.
MoETuner: MoETuner Go & Mahajan (2025) is an optimization framework that enhances MoE model serving performance by finding optimal expert placement strategy. It addresses critical bottlenecks in expert parallelism, namely imbalanced token processing loads across GPUs and skewed inter-GPU communication. which lead significant tail latency. The core of MoETuner is an Integer Linear Programming (ILP) formulation that leverages predictable token routing dependencies across layers. It jointly optimizes expert-to-GPU assignments to balance computational workloads and minimize communication costs, thereby reducing end-to-end execution time. We embed MoETuner into SGLang as a comparable baseline.
Metries: To mitigate performance fluctuating, we set the input length and output length fixed. Then we vary the request rate from 10 175 req/s, and observe the following metrics. Throughput the number of tokens (tokens/s) that an inference system can process per unit time. TTFT (Time to First Token) measures the duration between the request's 's arrival and the first token's 's generation time. E2E (End-to-End) Latency measures the duration between the request's 's arrival and the last token's 's generation time. Following prior work /W et al. (2024b)a a). we set the latency SLO as 5x of 20% of the data to train the token activation prediction model and generate the expert placement table, and the remaining 80% is used to sample experimental requests.
the latency under the lightest input load (minimal request rate). For each dataset,
8

---

Published as a conference paper at ICLR 2026
p99 TTFT (ms)
Median E2B Latency (ms)
Models
Input Length
SGLang
MoETuner
Sem-MoFE
SGL.ang
MoETuner
Sem-MoB
256
84.87
80.96
75.63
617.87
604.46
603.10
DeepSeek-V2-Lane
512
98.10
92.17
88.69
609.81
602.02
599.89
1024
111.19
119.76
100.74
608.96
606.54
604,45
256
85.72
87.24
74.46
758.60
763.97
750.26
Qwen3-30B-Ah
512
104.76
101.86
83.87
766.80
759.41
759.16
1024
107.73
107.36
103.69
769.83
764.16
762.30
Table 2: Attention- Scenario: TTFT and E2E latency under different input lengths.
4.4 END-TO-END INFERENCE PERFORMANCE
Figure shows the end-to-end performance evaluation of Sem-MoE and two baselines.
Attention-DP Scenario. For the attention-DP scenario, requests are scheduled to different DP ranks for attention and synchronized MoE layer. The latency of each layer is determined by the slowest DP(EP) rank. Thus we use token throughput to measure the overall performance in attention- We draw sitency-throughie pre-defined SLOs, shown in Figure Data points near the bottom-right corner are better. For Deepseek-V2-Lite. Sem-MoE achieves throughput improvements of % and 221% against SGLang with DeepEP under TTFT and end-to-end latency SL. constraints, and 32% and 278% against MoFTuner, respectively. For Qwen3-30B-AS. Sem-MoF's throughput improvement peaks at 98% and 11% against SGLang under SLO constraints, while also achieving gains of 35% and 32% against MoETuner. As the request rate increases continually, baselines stock a bunch of unprocessed requests, yielding steeper curve, resulting in the above high throughput improvement (221% % and 278%). The results demonstrate that Sem-MoE can obtain certain performance gains by scheduling the requests across different DP ranks and co-placing the experts in appropriate devices.
curve measure the highest throughput a system can achieve under
Attention- TP Scenario. For the attention- scenario, there no scheduler for a single inference instance, different TP ranks receive the same input. Meanwhile, latency becomes main concern in TP settings. Therefore, we set the request rate to req/s and vary the input sequence length to observe the TTFT and end-to-end latency directly as shown in Table 2 For Drepseck-V2-Ling Sem-MoE outperforms the best baseline in TTFT by 12.21%. 10.60%, and 18.89% under input lengths of 256, 512, and 1024. For Swen3-30B-A3E. the performance optimization ratio 17.16%, 24.90% 3.80%. Thanks to the load-balance and inter-layer communication optimizing effect, MoETuner gains performance improvement in some cases, yet may slow down in the other cases. Model-data collaborative scheduling can bring holistic performance boosting in all tested scenarios. The speedup of TTFT also translates to the shrinking of end-to-end inference latency, just as shown in Table We would delve into the execution of MoE layers analyze the rationale behind the breakdown of the inference speedup.
Extensive Experiments. We also conduct performance evaluation using one additional model (Moonlight-Ile 16B Liu et al. 2025 ), and the results are consistent with the above observations, which provided in
C.2
4.5 A DETAILED LOOK AT EP COMMUNICATION REDUCTION
In Figure 5(a) we show the local activation rate and the resulting latency of a single MoE layer under the attention- TP scenario. Local activation means the tokens' activation is routed to an expert collocated on the same GPU device, and thus, remote routing and its associated EP communication can be skipped. Results show that, compared with the vanilla placement, Sem-MoF. can increase LAR by 37% and 43% for DeepSeek- V2--Lit V2-Lite and Qwen3-30B-A33 latency reduction of the belonging expert layer. Besides Vanilla and Sem-MoE, other bars in the figure are measured by mocking the routing module of SGLang and skipping the delays in communication to fabricate hypothetical baselines just for reference. Note that 100% LAR may not be achieved in theory, as different tokens can contradict each other to group their own hot experts, but GPU memory limited.
which translates to 41.8%/44.66%

---

Published as a conference paper at ICLR 2026
singis-ayen pall--byerrizzza aizaal labency tmg
disaall-ispatch
alZall-combine
DeepSeek Lite Layer #1
Qwen3: 30B A38 Layer #1
41.0%
0.
Weille
Seme Mel 10099
10
102
Local Activation Rate
Local Activatio Rate
(a) Local activation rate against overall EP overhead.
Sem- MoE
Vanilla
MoElu
Local Activatton Roto
2.0
Inbalancce
0.4
1.5
EP Size
EP Size
(b) Local activation rate and load-imbalance rate of Sem-MoE and baselines.
Figure 5: Breakdown Evaluation
4.6 ALGORITIIM EVALUATION
The model-data collaborative scheduling algorithm needs to find balanced co-clusters tokens and experts, with experts having maximal likelihood of being gated (routed) from tokens within the same cluster, and minimal likelihood across clusters. An additional regularizer is load balancing that ensures hot and cold experts relatively evenly distributed. Sem-MoF: adopts Algorithm [ to approximately solve the problem and is evaluated against two baselines, the vanilla scheduling policy (original expert placement policy and round-robin scheduling) and MoETuner. Figure (5))e)((sss))))) shows the averaged local activation rate (ratio of tokens computed at the local device) and load imbalance rate (maximum load divided by the median load) all the MoE layers in SeepSec--V2-Lite Sem-MoE achieves the best local activation rate with balanced expert clusters outperforming the best baseline by 15.4% and 36.7% under EPS setting.
We also evaluate the cross-dataset zero-shot transfer performance of the predictor model, proving the robustness of the model-data affinity and the generalizability of the scheduling policy, which can be referred to C.I
CONCLUSION
The communication overhead of expert parallelism renders a significant bottleneck in serving large- scale MoE models. We present semantic parallelism and Sem-MoE, which can proactively and losslessly trim EP's all2all communication volume via model-data collaborative scheduling, leveraging the intrinsic affinity between model experts and input tokens. Experiments show that Sem-MoE can significantly reduce communication overhead and boost inference throughput under differently specified SLO constraints, both in attention-De DD and attention-TP scenarios.
REFERENCES
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O' Horo, Jeff Wang, Luke Zettlemoyer. Mona Diab, Zornitsa Kozareva, and Ves Stoyanov. Efficient large scale language modeling with mixtures of experts, 2022. URL https: arriv. org. crg/abs/////eeoo.coe//./////.e.r abs 2212
10684
10

---

Published as a conference paper at ICLR 2026
Chang Chen, Min Li, Zhihua Wu, Dianhai Yu, and Chao Yang. Ta-moe: Topology- aware large scale mixture-of offuure-of--perrrt training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 22173-22186. Curran Associates, Inc., 2022.
Hugging Face Datasets, Sharegpt vicuma unfiltered dataset., 2023. URL / soo/aaaaasttssaamm22111111111111 htt.:/99113///aaaraeeeee
ShareGPT Vii___iccua___ ceccune_unfilttree
DeepSeek-AL. Deepseck-v2: A strong, economical, and efficient mixture-of-experts language model, 2024a.
DeepSeek-AI. Deepseek-vž technical report, 2024b. URL httppttt////rc...rrrrabb////
19437
DeepSeek-Al. Deepseck-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Jourmal of Machine Learning Research, 23( 1-39, 2022. URL httt:://mmr wttgg/appes//2///11112 0998 html
Scokjin Go and Divya Mahajan. Moetuner: Optimized mixture of expert serving with balanced 0550/
expert placement and token routing, 2025.
Jiaao He, Jiezhong Qiu, Aohan Zeng. Zhilin Yang, Jidong Zhai, and Jie Tang. Fastmoe: fast mixture-of-expert training system, 2021.
13362
Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium Principles and Practice Parallel Programming, PPoPP '22, pp. 120-134, New York, NY, USA, 2022. Association for Computing Machinery, ISBN 40044939204, doi: 10.1145/553221.3506444 URL https:://ooi. org/
10.1145/350322111 25004321123503411
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. Deepspeed-faste fastgge:: High-throughput text generation for llms via mii and deepspeed-inferent 2024.
Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S. Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee. Towards moe deployment: Mitigating inefficiencies in mixture-ofeexpert (moe) inference, 2023. TRLhtttps: //ariiv/////vvvia/ orgg abs abtpabbb/.33... 22303 0618.66188
Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang. Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang. and Yongqiang Xiong. Tutel: Adaptive mixture-of-experts scale. CoRR, fbs/2200.00388.. June 2022. URL pdf
aittps://arxiv: http://aaaaa...rr/aa/2222..000.
Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of -e-perrtt inference. In 2024 ACM/IEEE 51st Anmual International Symposium on Computer Architecture (ISCA), pp. 1018-1031, 2024. doi: 10.1 19100///A55977..224000000

---

Published as a conference paper at ICLR 2026
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel. Guillaume Bour, Guillaume Lample. Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URLhttps: http/arxiivv..i.//ee//a/a
http://///..aai...//a/////////// http://.wrrgbb/////////// 2401 04088
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP '23, pp. 61 611-626, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702297.
doi: 00.1445/3600006666111666 URL BtLLhttps://// //tp:::://do.... org/ IO.1145/360006. 361360000063311665
Jiamin Li, Yimin Jiang. Yibo Zhu, Cong Wang, and Hong Xu. Accelerating distributed MoE training and inference with lina. In 2023 USENIX Anmual Technical Conference (USENIX ATC 23), pp. 945 959, Boston, MA, July 2023. USENIX Association. ISBN https:/ www. usenissniii. X. www.eenii..rrrooeoooooooooo http:///oeeeeeeeeea//aaeeeeeeeee
URL
on /1n jiamin
Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang. Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for IIm training, 2025. URL https: /ttp::i:///axvv....// org/ abs 2502 16982
Xiaonan Nie, Xupeng Miao, Zilong Wang, Zichao Yang, Jilong Xue, Lingxiao Ma, Gang Cao, and Bin Cui. Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement. Proc. ACM Manag. Data, 1(1), May 2023. doi: 10. 1.114455/88994. URI. https
7/doi wrg/10...00000..............1... 1144/
NVIDIA. Telsort--lm. [Online]. Accessed 23 Oct 2024, https://gtttuu..mmm//// TensorRT-
OpenAl. Applied ai experiments and examples for pytorch. [Online]. Accessed 23 Oct 2024, http:////oyootcccaaaaaaaapllllee
OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclere, James Park Lennon, Scott Lessans, Mario Sazcano-Casadoo Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney. Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Prochl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang. Miles Wang. Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss- 120 & gpt-oss-20b model card, 2025. URL attps: : arxiv org/ abs 2508 2000./..22252002225
Yulei Qian, Fengcun Li, Xiangyang Ji, Xiaoyu Zhao, Jianchao Tan, Kefeng Zhang, and Xunliang Cai. Eps-moe: Expert pipeline scheduler for cost-efficient moe inference, 2025. URL https; 12247
7/ warxi..org/aaaaa....o.
12

---

Published as a conference paper at ICLR 2026
Qwen-Team. Qwen1.5-moc: Matching 7b model performance with 1/3 activated parameters". February 2024. URL httph::t/////.tt /wwp:///////mmmmmmmmmmmmm github. itcii///llli hwtpl////eee//neeeeeeeeeeeeeeeee moe/
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed-MoE: Advancing mixture-of- experts inference and training to power next-generation Al scale. In Kamalika Chaudhuri, Stefanie Jegelka. Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceed- ings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18332-183444 PMIR, 17-23 Jul 2022. URI. htttst /proceedings mir.smmmm....oom
mlr www.mmt....rrsa..mm prw.s /vlsss/vvs.. v1l./v1662222 62 ra waoohandari22.. html
Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, and Abhinav Bhatele. A hybrid tensor-expert-datt parallelism approach to optimize mixture-of- experts training. In Proceedings of the 37th International Conference on Supercomputing. ICS '23, pp. 203-214, New York. NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700569. doi: 10.1145/3577198.399704
URL https::/aol.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang, Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai. Cheng Li. Fang Li, Haoyang Li, Ming Li. Wentao Li, Yanhao Li, Yiwei Li. Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang, Dinglu Wang, Feng Wang. Haiming Wang, Jianzhou Wang, Jiaxing Wang, Jinhong Wang. Shengjie Wang, Shuyi Wang, Yao Wang. Yejie Wang, Yiqin Wang. Yuxin Wang, Yuzhi Wang, Zhaoji Wang. Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang. Ying Yang, Zhen Yang, Zhilin Yang. Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi k2: Open agentic intelligence, 2025. URL
https: / /axivv.ivvvv/./////iv/aaa/a/aviv www.or/g//aa... abs 2507 20534
Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism, 2024a.
URL BttLtttss//aaxiiaa org/ abs abs/2404 09526
Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models, 2024b.
URL
abs/ /2305. 05920
Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. Moe-infinity: Offfoading-efficiin moe model serving, 2024. URL httpttttt//sxii....a orq. abs /00..////2..1111
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan I Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang
13

---

Published as a conference paper at ICLR 2026
Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang. Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL httph:tttttptttttt org/
09388
Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K., and Panda. Exploiting inter-layer expert affinity for accelerating mixture-of-expert model inference, 2024.
Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. Edgemoe: Fast on-device inference of moc-based large language models, 2023. URL
abs/2308 10..14430 14352
Mingshu Zhai, Jiaao He, Zixuan Ma, Zan Zong, Runqing Zhang, and Jidong Zhai. SmartMoE: Efficiently training Sparsely-Actie Activated models through combining offline and online parallelization. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pp. 961 995, Boston, MA, July 2023. USENIX Association. ISBN 978-1-939133-35-99 URL bttps:::: www. masenix.orgi
co..ference/aatc2///reesstattton zhai
Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, Kuai Yu, Jiashi Li, and Liang Zhao. Deepep: efficient expert-parallel communication library. ai/DeepEP)
https://git wtw.tttww...oceooeeeeeeeeeeeeeee com wtt.:/oo//ccceeeee a 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang Lmsys-chat-Im: A large-scale real-world 1lm conversation dataset, 2023.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024. URL
abs/2312.0713 07104
Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, Ru Huang, and Meng Li. Adapmoe: Adaptive sensitivity based expert gating and management for efficient moe inference, 2024. URL https://axxiv. org/ abs/2408 10284
A EMPIRICAL STUDY OF TOKEN-EXPERS AFFINITY
A.1 IMPACT OF REQUEST SEMANTICS ON EXPERT ACTIVATION
To better illustrate the impact of request semantics on expert activation, we selected requests from several different types of topies in the MMLU dataset and profiled the expert activations in the 24th layer of Owen3-30B-A3BB as shown Figure 6(a) After performing t-SNE dimensionality reduction, it can be observed that requests from similar topies exhibit similarity in activated experts. Requests from the math-related topics of abstract algebra and college mathematics activate similar experts, whereas requests from humanities topics, such as philosophy and professional law, relatively distant from the math-related ones in the this semantic affinity between requests to perform co-scheduling of data and models for requests under the Attention-DP scenario.
space, Sem-MoE leverages
A.2 TOKEN-EXPERT CONJUGACY
Intra-layer token-expert affinity Within each MoE layer, each expert module in an LLM layer is trained to process a particular semantic domain of tokens. Tokens and experts exhibit high affinity in different dimensions. We profile the intermediate activation of the gating module in each MoE layer in the DeepSeek- V2-Lite. One important observation shows that strong bi-clustered conjugacy between tokens and experts, as shown in Figure 6 That is, experts are likely to be activated by certain mub-group of tokens with high semantic affnity, while they are not likely to be activated by other general tokens in the vocabulary. And from the tokens' perspective, it is true that semantically similar tokens are likely to activate a certain sub-group of experts. This is the preliminary motivation for motel-data collaborative scheduling.
14

---

Published as a conference paper at ICLR 2026
Inter-layer expert-expert affinity The left picture of Figure shows the activation correlation of the 4th and 5th layer of the Mixtral-8x7B model. The x-/y-axis represents the expert groups of a layer. For Mixtral-8x7B,, each token is routed to 2 out of experts at each layer. Thus, the number of expert groups at each layer is (:) 28. When tokens choose some concrete experts at the fourth layer, they tend to choose a rather fixed set of experts at the next layer with high probability. We name this phenomenon inter-layer expert-expert affinity.
Simple conditional probability model for token activation path The above examples illustrate the tabularized relationship between tokens and experts. We argue that, simple conditional probability model can work to predict the activation path of tokens by combining the above intra-layer conjugacy and/inter-layer affinity. We first calculate the kurtosi of each token's 's activation map. The right picture of Figure ((b) shows most of the kurtosis values are higher than 8, indicating that the to-route experts for each token concentrate on narrowed set, regardless of the context. We further use partial (25%) of the profile dataset to calculate each tokens' most routed top-k experts. Then the static top-k experts are used to predict the tokens activation using the left part (75%) of the profile dataset, achieving 96.3% precision and a 78.8% Fl score. The right part of Figure [[() predicts the coxt-layer-acttvee
via looking back the prior layers. As we know more about the previous activation sequence at layer L, we can predict the activation at layer L + 1 with higher confidence (about 70% when looking back 5 layers).
Expert pre-grouping and token re- re-batching Leveraging the intrinsic conjugacy between tokens and experts in MoE models to achieve token-expert co-dispatching, may help reduce communication volume and boost distributed parallel inference. First, similar experts can be pre-grouped together at deployment time based on pre- pre-profiled and modeled affinity. Second, on the fly, individual tokens can be re-shuffled and re-batched to the GPU devices that host the expert groups whose member experts have largest modeled conjugacy with the token. Such co-scheduling aims to avoid scattered, cross-device token-expert activations, or equivalently, maximize the probability of intra-device, local activations. Figure ((d) shows an micro- benchmark experiment, testing the all2all latency under different local activation rate using the neel all2 all2allv v API. With the local activation rate (a) varying from 0.2 to 0.9, the latency decreases gradually, showing the performance gains with improved expert-token co-scheduling.
B ALGORITHM FOR THE MODEL-DATA CO-SCHEDULING SOLVER
Algorithm describes the model-data co-scheduling alternating optimization algorithm in Sem-MoE. The algorithm achieves optimal performance by alternating between optimizing the scheduling of requests and the placement of experts. Initially, requests are clustered based on their affinity to experts to determine their scheduling (line 44). In each iteration, the algorithm alternates between optimizing expert placement with fixed requests and request scheduling with fixed experts. For optimizing expert placement, experts are first sorted by their hotness in descending order. Given the current request scheduling and expert placement, the affinities between experts and the cluster's s experts/requests computed and aggregated via weights ae and Be, which is adjusted by the cluster's current load to derive a final affinity score (lines 11-13). The expert is assigned to the highest-scoring cluster, with saturated clusters masked (line 14). The algorithm then performs f't steps fine-tuning rounds, randomly selects two clusters and swaps their experts if it improves the affinity score (lines 20-25). Request scheduling optimization is similar to expert placement. The req-req affinity and req-expert affinity for each cluster are calculated, aggregated to obtain an affinity score, and the request is scheduled to the cluster with the highest score (lines 28-42).
By now, the token-device scheduling table T, token-device scheduling confidence table Tp. and expert-device scheduling table& & are generated. After the scheduling table E is constructed, the experts at each layer need to be rearranged according to the scheduling table during online inference service deployment. In addition, the Sem-MoB rearranges the gating module by column to implement transparent expert shuffle. The semantics of other layers are not affected. The rearranged experts are highly boxed, so that the token activation at each layer is de-cohesive, and the redundant network communication overhead caused by dispersive activation is reduced.
Kurtosis is measure of the tailedness of distribution. High Kurtosis indicates a token favors several fixed experts during multiple occurrences.
15

---

Published as a conference paper at ICLR 2026
5800
(a) Example of expert activations for requests different topics (24th layer of Qwen3-30B-A3B profiled using the MMLU dataset), with t-SNE dimensionality reduction.
Accunalane lold frequency
1.0
0.8
0.6
0.4
0.0
Experts in depe 16B layer 1 (shulfled)
Activation kurtosis
(b) Example intra-layer token-expert activation map (1st MoE layer of DeepScck-V244 V2-Lite profiled using the Sharegpt datasct). Darker color in the map indicates higher activation frequency or stronger correlation.
Prodication sccuracy (8)
70.0
Expert groups layer
87.5
65.0
62.5
60.0
15
20
Expert groups at layer
MLayens to lookup
Example of inter-layer expert-expert correlation map (4th/5th MoE layer of Mixtral-8x77 profiled using the LongBench datasct). Darker color in the map indicates stronger correlation.
1.0
API: torch to all single buffer 8192X 1962
Norm Coenn
0.5
albreuuee
all2all 0-0.2
alllall 0-0.3
all
a122
a112
all2al
a1l2al
x112el
0.0.4
0-0.5
0-0.6
8-0.
8
2-0.9
(d) Example of perfromance of all allruduee and a112al1 under different local activation ratio (a)).
Figure 6: Conjugacy illustration and collective communication micro- benchmark.
16

---

Published as a conference paper at ICLR 2026
Algorithm 1: Alternating- based model co-scheduling algorithm
input: stspeps: number of iteration steps;
C,: the token-2-expert confidence table; a: the token frequency:
v: requests list; K: number of requests;
N: number of experts per layer, E: number of co-clusters;
number of tokens;
output: E: expert labels; T: token labels;
Tp: confidence tokens choosing specific experts
p_matria_e _epporttepp-pppteeep &- zeros (N, F) /
p matriz req opt <- zeros (K, E) / E
Function expert p1 al (C p matria req. de, Be):
loads compute per expert load by (Cp)
sort.by_Load loads)
mask, enler <- ones (E), zeros (E)
EAfE, EAfR zeroa (N, E), zeros (N, E)
p. p.matriz ep f zeros (N, E)
loads cls + zeros (E)
10
for in e do
EAfE +- compute expert-expert affinity by (mask, p.matrii. .pp. Cp) EAfR[( compute req-expert affinity (mask, p.matrin. Cy) aff-score t a, [le]
13
Blf %ee loods cls
14
cls. t arg maX,ta assoe p-matrraz.cpel[csss]] enter(cls,] inter(cla.] +1 if enter[cla,]] >= N/F: then
15
makss makss(cs.]] +
19
loads cls(sscs < loads [oodescls[elss] + loads[e]
20
repeat
21
cls1, cls. + randomly select a cluster in [6] e1, + randomly select experts in p=s] and p.matrir.eplijee.. if ifff-ggain (e1, e2, clsı, clæ2) > 0 then
22
swap (e1, 62. p matria ep)
25
until iterating for ft_steps steps return p. p.matria.ce
26
27
Function requesttschediie (Cp, p.matriz.epp. ar Br)
sort.by-len (r)
30
mask, enter f- ones (E), zeros (E)
31
RAFR. RAfE + zeros (K, E), zeros (K, E))
32
p_matria_re _rer + zeros K, E) E
for in do
RAfR(r compute req-req affinity by (mask, p matria req, Cp) RA/E(r] + compute req-expert affinity by (mask, p matrix ep, Cp) aff_score t a, RAARR(] + 3. RAfE
35
36
37
clst arg max.ts asfor
p.matris.reqir]er
39
entericls cls,] + cnter(cls, | entericls. >= K/B then maks [cl..] +
40
41
42
return p. p.malrirree reg
43
p.matrix.rre. + cluster based expert affinity
45
repeat
46
-matriicep--oxppt.pplaae (Cp, p.matriæ.re, Oc. 3c) (Cp, p.matrix_ep. or, 3r)
47
p_matriz_req resustsshhheeee
scores + summation the max load and communication cost given p.matris.ee and pmmatrio_ree samples with
better scheduling update p matria ep opt and p malria reg opl
50
until iterating for steps steps
53 argmax (p matria ep opl, axis=l)
54 p.matriæ_kkaa <- count the tokens per req in p.maariaareq.m
55 T. argmax_wis with vulus (p.matrir Ik-ol,, axis=1)
17

---

Published as a conference paper at ICLR 2026
B.1 MODELING INTER-LAYER ACTIVATION CONJUGACY
Leveraging the conditional probability model described in $ A.2 we use a simple probability-base first-order Marcov chain model the inter-layer activation conjugacy. To reduce the combination space, we model the activation device sequence rather than the activation expert sequence, because we only care about the device-level token rebatching. When looking back layers, we construct table shaped like [E', .E], where the row of the table indicates the sequence of devices selected at the previous / layers and the column indicates the probability of activating the E devices in the current layer. Like the $ shows, we also calculate the activation sequence to device table A and the confidence table Ap. In practice, we set the number of looking-back layers as 2.
Algorithm 2: Online request scheduling based on fast lookup
input: R E N Input requests; T: -oteett-- Schedule Table; E: number of DP size dev_mask + ones (E)
Function get_dp.rank (R, T):
dev_score 4- sum (T dim 0)
dev_scorel dev_mask] dev.id <- argmax (der_score) dev hoy_maskklerr id] + False if dev mask all are False then dev_mask +- ones (E)
inf
return dev_id
dev_id <- get_dp_rank (R. T) n chedule (R, dev
B.2 SPECULATIVE TOKEN SHUFFLING ON THE FLY BASED ON FAST LOOKUP
To reduce the combination space, we model the activation device sequence rather than the activation expert sequence, because we only care about the device-level token rebatching. We implement fast online token re-batching mechanism based on fast looking-up tables in both Attention-DP and Attention-TP (Algorithm & Algorithm
Data Scheduling: Attention-DP Scenarios. The algorithm 2 Qqeeries the titen-to-exper--lluee scheduling table based on the token IDs appearing in the request R, and aggregates the results to obtain score for each device for that request (line 3). Then R is scheduled to the device with the max valid score (line 5). To prevent requests biased toward a subset of experts, which could skew the load during the decoding phase, introduce dev mask. The device is masked after it is allocated (line 4-5). Once a round of allocation is completed and all devices are masked, the dec mask is reset and enters new round (line 7-8). This ensures that Sem-Meeaschiees expert affinity while maintaining load balance across devices.
Data Scheduling: Attention-TP Scenarios. The algorithm B queries the token-to-expert-cluser cttpuusttesssuuuuee--oeeerrrrrre
scheduling table T and dences first. Then, the table with higher confidence is adopted to obtain the device ID list which the current batch token needs to be shuffle (line 2). The algorithm performs the argsort operation to obtain the shuffle indicators (line 3) of the token. Then, the final shuffle indicators are obtained by grouping, aligning, and concatenation, and the token is shuffled (line 4 to line 7). After rebatching is complete, Sem-MoF calls the reduce-scatter sccateer operation. After MoE computing is complete, Sem-MoE runs the allgather operation to collect tokens. Finally, the order of tokens shuffled back based on the previously calculated shuffle indicators (lines 14-18).
table S, together with their confi-
The both algorithm do not involve complex load calculation and decision-making They are directly completed by querying tables. The runtime overhead mainly involves large token matrix shuffling. which we optimize via nigh-performance kemnels. The memory occupation of the scheduling tables negligible. For example, for DeepSeek-V2 V2, the memory space that the token-to-devine table T occupies is x02400x60x2 z 11.72M B (assuming the data format is in 16). 1024"
18

---

Published as a conference paper at ICLR 2026
Algorithm 3: Online token re-batching based on fast lookup
input: B E N": Input token IDs; T: likken-to-xxper--lustee Schedule Table; Schedule Table
A: httpt/ccccttcrrqqoooooooooeeeeee
Function rebatch.tooarnt (B, T):
dev_ids <- cend (Tp [B] > Ap (8), T [8), (())
shfiindccss argsort (dev_ids)
g g-sh. findice- group by key (sh indices) g-shffindicss + align indicee))
f -indices + concat (g_shf lindices)
B 4 B f indicees
return shf sin_ine
Function resume.tokens (B, sh.f indicee)
r-she-indices argsort (shf Lindices)
12
B B _indcciiiices inndices]
13
f Lindices + rebatch_token (B,
Blocal + reducesscatter (B)
16 executing MoF. layer
17 < aligather (Bioent)
resume.tokens (B. (indices)
C EXTENSIVE EXPERIMENTS
C.I CROSS- DASAST VALIDATION OF EXPERT ROUTING PREDICTION
To assess the generalization of our method, conducted cross-domain experiments using two large language models: DeepSeek-Ve V2-LLiee and Qwen3-30B. Specifically, we trained Sem-MoE's activation predictor on single source dataset and evaluated the quality of its token-expert co-scheduling decisions, measured by the Local Activation Rate (LAR), on the other two unseen, outoof-distribution target datasets. Note that LAR measures the proportion of tokens routed local experts, reflecting reduced communication volume to experts on remote devices. Therefore, higher LAR indicates lower cross-device communication overhead and better.
The results reveal robust zero-shot transfer performance for our method.
Table 3: Cross-datasee evaluation of zero-shot transfer for DeepSeek-V2-Ie Vi-Liee
LAR (p50)
ShareGPT
Lmsys-Chat-IM
MMLU
Method
Gem-MoB trained on ShareGPT
46.49
41.25% 47.19%
41.55%
Sem-MoE trained Lmsys-Chate Im
40.68%
41.40%
Baseline: SGLang
24.98%
25.01% %
24.97%
Table 4: Cross-dataset evaluation of zero-shot transfer for Qwen3-30B
LAR (p50)
ShareGPT
Lmsys-Chat-IM
MMLU
Method
Sem-MoB trained on ShareGPT
46.88 %
39.28% 43.61 %
35.30%
Sem-MoE: trained Lmsys-Chat-Imm
36.57%
33.37%
Baseline: SGLang
25.00%
25.01% %
25.00%
As shown in Table 3 for DeepSeck-v2-Ling the predictor trained ShareGPT achieves in- distribution LAR of 46.49%. When applied without retraining or fine-tuning to Lmsys-Chat 1M still attains 41.25% LAR-one only modest drop from the best in-domain result that dataset (47.19%. achieved when training directly on Lmsys-Chatt -h), More importantly, this out-of-distributin performance is 1.65x higher than the SGLang's 's default scheduling setting (25.01%) %).. Similarly,
19

---

Published as a conference paper at ICLR 2026
MMLU domain with very different content and structure the same ShareGPT-ttrainee predictor yields 41.55% LAR, far surpassing SGLANG defaults (24.97%) and remaining close to in-distribution levels.
As shown in Table the trend consistent for Qwen3-30B: the ShareGPT-tramned predictor achieves 39.28% LAR Lmsys-Chat- MM and 35.30% MMLU, compared to peak in-domain scores of 43.61 % and 36.57% (when trained on Lmsys), and both are substantially above the SGLang baseline (55%). A similar trend holds when transferring reservedly from Lmsys-Chat-IN ShareGPT and MMLU.
These results demonstrate that when fed with real-world, representative datasets, Sem-MoF tends to capture the invariability in token-routing patterns that remain effective across different linguistic styles, task types, and knowledge domains. While training on the target distribution offers marginal improvements, it is not required except for extreme performance pursuits.
C.2 THROUGHPUT IMPROVEMENT OF MOONLIGHT- 16B
To address the issuc of model diversity, we evaluate the Moonlight- 16B model. The results align closely with the performance trends observed for DeepSeek-V2-4 and Qwen3-A30B, reported the $ In summary, our approach Sem-MoE, consistently outperforms SGLANG (MoETuner): it achieves average performance improvement of 1.12x (1.22x) for prefill and 1.10x (1.14x) for end-to-end inference, respectively. Detailed results are provided in Table and Table 6
Table 5: Throughput Optimization Effect under TTFT SLO
Throughput (Req/s)
Speedup v.s. SGLang
Speedup v.8. MoETuner
SGLang
MoETuner
Sem-MoF
Dataset
Lmsys-Chat-Ine Chat-IM
32.7
28.6
38.6
1.18x
1.35x
MMLU
32.2
30.1
38.6
1.20x
1 1.28x
ShareGPT
2.9
2.8
2.9
1.00x
1.02x
Average
22.6
20.5
26.7
1.12x
1.22x
Table 6: Throughput Optimization Effect under E2E latency SLO
Throughput (Rey/s)
Speedup SGLang
Speedup
SGLang
MoETuner
Sem-MoF
v.s. MoETuner
Dataset
Lmsys-Chat-IN
Chat-1M
37.2
35.5
38.6
1.04x
1.09x
MMLU
37.9
35.3
38.6
1.02x
1.10x
SharcGPT
1.9
1.9
2.4
1.23x
1.23x
Average
25.7
24.2
26.5
1.10x
1.14x
20


Reviews:
[{"review_id": "EaLxExbCe5", "reviewer": "Reviewer_Nc2B", "summary": "The Sem-MoE paper introduces an efficient method to enhance the performance of Mixture-of-Experts (MoE) models by reducing heavy communication between GPUs during inference. The authors do not treat model placement and data routing separately. Instead, they design a semantic-aware scheduler. This scheduler predicts which experts each token will likely activate. It aims to keep those experts and tokens on the same GPU. The method combines offline expert clustering with online request routing. It also uses customized communication kernels to minimize overhead. Experiments on models like DeepSeek-V2-Lite and Qwen3-30B-A3B show significant gains in throughput and latency, demonstrating the effectiveness of the approach. However, the method relies on static token–expert correlations, requires costly optimization, and may struggle to adapt to real-world, changing workloads. Overall, it is a smart and well-engineered step toward more efficient MoE inference.", "questions": "Summarized in Weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The semantic-guided approach is intuitive and elegant. It leverages predictable token–expert correlations that were mostly ignored by other MoE schedulers.\n2. Extensive experiments on DeepSeek-V2-Lite and Qwen3-30B-A3B show consistent and significant improvements in both throughput and latency.", "weaknesses": "1. The semantic prediction is based on historical activation statistics. In real-world, open-domain inference, input distributions can shift significantly. Such shifts may invalidate prior correlations and reduce performance.\n2. The optimization process is largely offline and static. When the model or workload changes, Sem-MoE requires re-optimization, which can be computationally expensive.\n3. Although the method is non-intrusive, Sem-MoE modifies the runtime scheduler and introduces custom collective operations. This makes integration into production-level inference frameworks more complex than suggested.\n4. There is no analysis of energy consumption or communication bandwidth efficiency in the paper.\n6.  The author should consider more types of MoE LLMs, such as Qwen3, Moonlight-A3B, gpt-oss-120b, and gpt-oss-20b, should be considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iL2mxov0Bw", "reviewer": "Reviewer_Rf5L", "summary": "The paper proposes Sem-MoE, a semantic-aware model–data collaborative scheduling framework that improves inference efficiency for large-scale MoE models. It jointly optimizes expert placement (offline model scheduling) and token/request routing (online data scheduling) to reduce all-to-all communication in expert parallelism.", "questions": "1. Beyond SGLang and MoETuner, were any additional baselines included for comparison? The current comparison seems limited.\n2. Do tasks or queries from different knowledge domains require extra training to maintain accurate token–expert mappings? \n3. How often must Algorithm 1 be executed in practice, and does this introduce notable or impractical computational overhead?\n4. The paper mentions online inter-request data scheduling to proactively rebatch incoming requests. In the experiments, what stochastic process assumptions or models are used to generate the query arrivals?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses a key bottleneck in MoE inference—the high communication overhead associated with expert parallelism.\n2. It introduces a novel idea that aligns expert placement with token–expert affinities through semantic-aware co-scheduling.\n3. The proposed framework demonstrates a comprehensive design, effectively supporting both DP and TP configurations.", "weaknesses": "1. The experimental setup is relatively limited, with evaluations conducted only on DeepSeek and Qwen; I recommend validating the conclusions across a broader range of models and datasets.\n\n2. The reported performance gains in E2E latency appear limited, which limits the practical significance of the proposed scheme.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZSgKHrHq97", "reviewer": "Reviewer_4HBM", "summary": "This paper introduces Sem-MoE, a mixture-of-experts (MoE) scheduling framework that leverages offline profiling of token–expert activations to optimize communication and load balance. The method first computes a token–expert activation matrix $$C_{p,jk}$$ from model logs, then formulates an integer linear program (ILP) minimizing cross-expert communication and balancing expert loads. The ILP solution is later applied at runtime (§3.4) to pre-group experts and improve all-to-all efficiency. Experiments on an 8×A100 NVLink setup report a 2.78× throughput gain and 25% latency reduction compared with FasterMoE.\n\n The overall mathematical formulation and implementation are coherent and elegant, yet the theoretical and empirical justifications for using offline token–expert frequencies as stable, context-independent features are insufficient. Experimental validation focuses mainly on efficiency and omits semantic and multi-node evaluation.", "questions": "(1) The paper defines “context-independent routing” (§3.2) based on the observed stability of token–expert activation frequencies computed from full-sentence attention representations. However, the gating function $$G_L(h_{L,j})$$ inherently depends on contextual hidden states. How do the authors justify that the offline probability $$P(E_k|x_j)$$ estimated under contextualized inputs approximates the true context-independent term $$P(E_k|x_j,h_{L,j})$$? In particular, what does “semantic independence” mean in this formulation—is it statistical stability of frequent tokens, or an actual invariance of routing under different semantic contexts?\n\n(2) The authors report stability (96.3%) and F1 (78.8%) on ShareGPT, but the analysis seems limited to a single layer and dataset. Could they clarify whether similar stability holds across different layers, seeds, or corpora?\n\n(3) Figures 6(b)(c) analyze only common function tokens. The text briefly mentions that rare tokens are “more uniformly distributed,” but no quantitative results are given. Could the authors elaborate on the behavior of low-frequency or semantic tokens?\n\n(4) The author mentioned “the benefit will be even larger under slower interconnects (PCIe or InfiniBand)” in §4 and suggests that greater improvements would be observed under slower interconnects (PCIe or InfiniBand), yet this claim is untested. Have the authors evaluated or simulated multi-node scenarios to confirm that the ILP-based grouping remains effective when communication latency and bandwidth vary?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(1) The ILP modeling in §3.3 is mathematically elegant and well aligned with the MoE scheduling problem. It directly encodes both communication minimization and load balancing as discrete constraints. The objective function depends explicitly on $$C_{p,jk}$$, linking statistical observations to optimization variables. The derivation is concise and solvable, and the implementation (§3.4) confirms that this optimization is executed in practice.\n\n(2) The pipeline from offline profiling to runtime integration is clear and systematic. The ILP-generated token–expert and expert–expert tables are concretely used for expert grouping, confirming that the method bridges theory and system.\n\n(3) The experiments (§4) show consistent and interpretable performance gains, demonstrating real engineering improvement, not mere theoretical potential.", "weaknesses": "(1) The ILP depends on the token–expert frequency matrix $$C_{p,jk}$$, which the authors assume to be stable and context-independent (§3.2). However, this assumption lacks theoretical grounding or rigorous validation. Appendix A reports stability (96.3%) and F1 (78.8%) computed on a single corpus (ShareGPT) and layer, but provides no variance analysis across datasets, seeds, or layers. Since the gating probability $$P(E_k|x_j,h_{L,j})$$ inherently depends on contextual hidden states $$h_{L,j}$$, averaging over contexts removes critical variance information. Moreover, although the authors describe this as an “offline distribution,” the collection of $$C_{p,jk}$$ itself is derived from attention representations obtained during **normal Transformer model inference on full sentences**. Thus, each token’s recorded expert activation already embeds contextual dependencies from its neighboring tokens. This makes the so-called offline mapping not truly context-independent but an average over context-conditioned activations. Consequently, the statistical input to the ILP may not reflect real routing behavior when context distributions shift.\n\n(2) Figures 6(b)(c) mainly show high-frequency function tokens such as “the”, “and”, “you”. Their activation stability is trivial and does **not** generalize to some specific semantic or rare tokens, which dominate real workloads. Furthermore, Figure 6 shows that activation entropy increases in deeper layers, yet the paper interprets this as “uniform expert reuse.” Uniformity here indicates uncertainty, not stability, thus contradicting the claimed “context independence.” The analysis therefore captures lexical regularity rather than semantic invariance.\n\n(3) The communication cost term in the ILP objective aggregates token-level co-activation frequencies into expert–expert communication weights (§3.3), but no strong empirical correlation between these weights and actual communication traffic is shown. The paper lacks ablation studies testing ILP sensitivity to noise or sampling bias in $$C_{p,jk}$$.\n\n(4) The reported results measure only throughput, latency, and load variance, **lacking evaluation** of model quality (perplexity, accuracy, downstream metrics). So it is unclear whether the offline routing preserves semantic fidelity or harms inference correctness.\n\n(5) All experiments are performed on a single 8×A100 NVLink node (§4). The paper suggests that the method could achieve larger gains under slower interconnects (PCIe or InfiniBand), but this claim remains untested. Including cross-node or heterogeneous-network experiments would strengthen the scalability argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)