Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models
Authors: Jingcong Liang
Siyuan Wang
Miren Tian
Yitong Li
Duyu Tang
zhongyu wei
Venue: ICLR 2026 Poster (2026)
Keywords: local routing consistency
MoE analysis
expert offloading
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce *local routing consistency* as a critical property for efficient expert offloading, conduct empirical analysis across various MoE LLMs, and provide practical insights for MoE architecture and cache system design.

Abstract: Mixture-of-Experts (MoE) enables efficient scaling of large language models (LLMs) with sparsely activated experts during inference.
To effectively deploy large MoE models on memory-constrained devices, many systems introduce expert offloading which caches a subset of experts in fast memory, leaving others on slow memory to run on CPU or load on demand.
While some research has exploited the locality of expert activations, where consecutive tokens activate similar experts, the degree of this **local routing consistency** varies across models and remains understudied.
In this paper, we propose two metrics to measure local routing consistency of MoE models:
(1) **Segment Routing best Performance (SRP)**, which evaluates how well a fixed group of experts can cover the needs of a segment of tokens, and
(2) **Segment Cache best Hit rate (SCH)**, which measures the hit rate of an expert cache utilizing a length of future information under a cache limit.
We analyze 20 MoE LLMs with diverse sizes and architectures and use toy models to verify key factors related to local routing consistency.
We find a strong trade-off between local routing consistency and *local* load balance, while showing that *global* load balance can coexist with local routing consistency.
Meanwhile, settings like shared experts that decrease expert combination space can lead to low local routing consistency.
We further reveal that domain-specialized experts contribute more to routing consistency than vocabulary-specialized ones, and that most models balance between cache effectiveness and efficiency with cache sizes approximately twice the active experts.
These findings pave the way for memory-efficient MoE design and deployment without compromising inference speed.

Published as a conference paper at ICLR 2026
NOT ALL MODELS SUIT EXPERT OFFLOADING: ON Lo- CAL ROUTING CONSISTENCY OF MIXTURE-OF-ERE MODELS
Jingcong Liang Fudan University
Siyuan Wang" University of Southern California sw_641@usc.edu
jcliang22@m. fudan. cn
Miren Tian & Yitong Li Duyu Tang
Zhongyu Wei"
Huawei Technologies Ltd.
Fudan University & Shanghai Innovation Institute zywei@fudan. edu. cn
tianmirent .1ll tong3, tangduyu@huawei.com
ABSTRACT
Mixture-of-Experts (MoE) enables efficient scaling of large language models (LLMs) with sparsely activated experts during inference. To effectively deploy large MoE models on memory-constrainet devices, many systems introduce expert offloading which caches a subset of experts in fast memory, leaving others on slow memory rum on CPU or load on demand. While some research has exploited the locality of expert activations, where consecutive tokens activate similar experts, the degree of this local routing consistency varies across models and remains understudied. In this paper, we propose two metrics to measure local routing consistency of MoE models: (1) Segment Routing Best Performance (SRP), which evaluates how well fixed group of experts
cover the needs of segment tokens, and (2) Segment Cache Best Hit Rate (SCH), which measures the hit rate of expert cache utilizing length of future information under cache limit. We analyze 20 MoE LLMs with diverse sizes and architectures and use toy models verify key factors related to local routing consistency. We find strong trade-off' between local routing consistency and local load balance, while showing that global load balance can coexist with local routing consistency. Meanwhile, settings like shared experts that decrease expert combination space can lead to low local routing consistency. We further reveal that domain- specialized experts contribute more to routing consistency than weeabullrr-specialiic balance between cache effectiveness and efficiency with cache sizes approximately twice the active experts. These findings pave the way for memory-efficient MoE design and deployment without compromising inference speed. We publish the code for replicating experiments at httpt:tt//// /tts::///ihht thub. com
ones, and that most models
INTRODUCTION
-ixture-of-Experts (MoE) is a widely adopted model architecture in many large language models (LLMs) that enables efficient model size scaling through sparse activation(Fedus al. 2022 Tiang et al. 2024 Dai et al. 2024 Abdin et al. (2024). MoE models replace dense feed-forw forward networks (FFNs) with multiple expert modules, with only subset activated during inference. However, the vanilla implementation requires all experts to be loaded into memory, restricting its application on memory-constrainee devices such as mobile phones. To address this limitation, the expert offloading technique has been proposed to allow partial loading of expert modules during inference (Eliseev & Mazur 2023 Hwang et al. 2024 Yi et al. 2025). Specifically, expert offloading caches a subset of experts in fast memory (e.g., GPU memory) based predefined heuristics, while storing remaining experts in slower but larger-capacity storage (e.g., CPU memory or disk). During inference, especially
Corrresonniigg authors.

---

Published as a conference paper at ICLR 2026
in the decoding stage, if a token activates expert not cached in fast memory. the system either computes the expert forward results with CPU and slow memory (Kamahori et al. 2024 Tang et al.) 2024), or unloads cached expert according to specific rules such as Least Recently Used (LRU), and replaces it with the demanded expert (Kong et al. 2024 Zhong et al. 2025).
However, frequent CPU offloads on-demand loading within short period can significantly degrade the efficiency of the expert offloading system and slow down inference, particularly when processing lengthy contexts with inevitable topic shifts. Prior research has focused on optimizing the design of the expert offloading system, aiming to strategically select which experts to cache in a given context to maximize cache hit rates. Among them, some observed and exploited the locality of expert activations, where similar routing choices appear within consecutive segment of tokens, thereby minimizing the need for CPU offloads and on-demand loading (Fliseev & Mazur, 2023 Xue et al.. 2024b; ZZhan et al., 2025). This is especially beneficial during the decoding phase, where tokens are generated one after another.
Nevertheless, not all MoE models exhibit such continuous routing patterns uniformly, and the degree or frequency of this phenomenon varies across models. Understanding this variance may help design MoE architectures that are friendly to expert offloading systems and vice versa. In this work, we investigate the degree of this inherent consecutive routing property, which we term local routing consistency, of different MoE-based LLMs to explore their potential effectiveness in segment-based illlutratts how different levels of local routing consistency reflect different routing patterns. Specifically, we propose two metrics that quantitatively reflect the local routing consistency of a specific model. (1) Segment Routing Best Performance (SRP) measures how effectively segment router that selects a fixed group of experts for all tokens in a segment can approximate the original router's decisions. SRP not only reflects local routing consistency without parameters other than segment length. but also enables analyzing activation patterns of individual experts. (2) Segment Cache Best Hit Rate (SCH) represents the hit rate of an oracle expert cache that evicts unused experts based on the activation frequency within a specific length of future, under a cache size limit related to the number of active experts. SCH measures local routing consistency on model- and router-levels, yet is more related to the performance of real expert offloading systems as accounts for the true cache limit.
expert routing or caching. Figure T
SRP 65.35 Consistent!
McE
SRP 38.48 What mess!
MMin
Figure 1: Routing results by GRIN-MoE Liu et al.
2024 layer 21 and Jamba-Mini-L 1.6 (Lenz
al. 2025) layer 25 on the same input (Java code). Despite having similar model sizes and the same number of experts, GRIN-MoE exhibits more consistent routing patterns than Jamba-Mini- 1.6. activating certain experts continuously, so expert caching with it will be more feasible and effective.
We conduct experiments on 20 MoE-based LLMs, covering parameter scales ranging from 3 billion to 54 billion and encompassing diverse architectures. While most models exhibit similar local routing consistency within a few tokens, the variance enlarges when the segment length increases. Further verification on toy models reveals a trade-off relation between local routing consistency and local load balance: however models can achieve good global load balance with high local routing consistency. Meanwhile, Shared experts harms local routing consistency, as well as expert configurations that limit expert combination space. Additionally, we investigate local routing consistency across different context domains and assess its relationship with experts' domain and vocabulary preferences. The findings reveal that domain-specialized experts, if they exist,
2

---

Published as a conference paper at ICLR 2026
contribute more local routing consistency, whereas vocabulary specialization has less impact. Finally, we verify the strong correlation between SCH and hit rates of common cache algorithms, confirming the bond between local routing consisteney and expert offloading efficiency. We conclude that most MoE models can achieve optimal balance between segment caching effectiveness and deployment efficiency with a cache size approximately 2x the number of active experts
Overall, our contributions are threefold:
1. We propose local routing consistency, property of MoE models that reflects the potential efficiency of expert offloading for the model. We design two metrics to quantify local routing consistency: segment routing best performance (SRP), which provides parameter-free fine-grained analysis, and segment cache best hit rate (SCH), aligned with practical expert offloading.
2. We conduct empirical analysis across 20 MoE-based LLMs, identifying and verifying (with toy models) key factors that affect local routing consistency, such as local load balance, shared experts, and expert combination space. We also reveal that domain-specialized experts contribute more to local routing consistency than vocabulary-specilll
ones, well global load balance. 3. We analyze SCH under different cache sizes relative to the number of active experts. Alongside the optimal cache size derived from SRP results, we conclude that cache sizes 2x the size of active parameters achieve the best segment caching results on most models.
DEFINITIONS
2.1 PRELIMINARY: MIXTURE OF EXPERTS
Transformer- baasee based language models have most parameters on feed-forward network (FFN) layers. As the model size scales up, LLMs often replace them with sparse MoE layers to reduce computational cost during inference. typical MoE layer with E experts can parameterized by E smaller FFNs Fi ( (; (g), where each F;: Rd - Rd defines a single expert. There is also router R: Rdd + R.E in the MoE layer to choose experts and give weights. For each token æ with hidden representation h2 E gd, its output is given by
E
[11.
SE = (oottmax(R(hz))) [wi,.... wE)] = (s,.. (s)) Oz = Σ w. F. 0.) (1)
where Top; preserves the k largest scores and sets others to 0. Experts with null score can be effectively deactivated without calculating, thus saving computational resources. Other components in the transformer architecture may also be replaced by their MoE variant, such as Mixture-of- Attention (Zhang et al. 2022) for self-attention and MixLoRA A (Li et al. 2024a) for LoRA adapters. Nevertheless, we focus on MoE layers that replace FFNs, as this the most prominent and effective design (in terms of the number of parameters), More discussions about MoE LLMs can be found in Appendix B where we also briefly review expert offloading for MoE models.
The above routing procedure is done token by token, which does not guarantee consecutive routing. Since the consistency of consecutive routing decisions can benefit expert offloading systems, is necessary to investigate the degree of consecutive routing of different MoF-based L.MMs. In the following sections, we propose two metrics to measure this local routing consistency and compare them across MoE models with different structure parameters.
2.2 SEGMENT ROUTING BEST PERFORMANCE (SRP)
An intuitive way to measure local routing consistency to compare the distribution of routing choices between tokens in a continuous segment. However, typical metrics for distribution comparison, such as the Kullback-Leibler divergence, are designed for two distributions and are therefore unsuitable for purpose. Instead, we propose Segment Routing best Performance (SRP) (Figure 2 which measures how well a simplified, segment-based router can mimic the behavior of the original token-based router. Here give brief introduction of SRP, with formal definition in Appendix C.1
'We propose this insight from the aspect of model design, especially when building models for devices with known memory constraints (e.g., mobile phones).
3

---

Published as a conference paper at ICLR 2026
Expert Activation Sequence
(Best) Segment Estimator
Router Decision Sequence
(Best) Segment Router
T23
SRP=F1
SRP=F1
TP2 FR2
..7144
TP2
TS FRB TNd
Tha
An illustration of segment routing best performance (SRP). Left: SRP on a single expert, where a segment estimator gives segmented predictions on every segment, and SRP is the best possible Fi score. Right: SRP on a group of experts, where a segment router predicts which experts are activated in segmented manner, and SRP is the upper bound of Fi.
Figure 2:
On single expert Given input sequence, the activation pattern of an expert in an MoE model on this sequence can be seen as a series of binary classification tasks. A segment estimator with segment length m aims to predict this activation sequence, but in segmented manner: At position i, it predicts the same result (active or not) for positions from i to i + m 1. We select the F1 score to measure the performance of the estimator, because missing major activations is worse than activating on minor activations (Kong et al., 2024 Skliar et al. 2025); recall (hit-rate) is also inappropriate as we do not set up an upper bound on how frequently the expert can be activated. Moreover, to diminish positional effects, we consider all possible positions and treat each position as an individual binary classification task with m samples. Based on all the settings above, the SRP on this expert is defined as the upper bound Fi of such estimators.
Despite modelling using a virtual segment estimator, SRP itself solely depends on the expert and the segment length: The Fi score is maximized if and only if the estimator gives active predictions for all segments that had the expert activated more times than a specific threshold; a formal proof is given in Appendix C Therefore, SRP is an intrinsic property of the expert that reflects its local routing consistency, unrelated to any specific routing methods.
On a group of experts For a group of experts (in the same layer or in the same model), we can combine single-expert SRPs for expert-choice routing (Zhou et al. 2022), since experts independent of each other. However, in the more traditional and prevalent token-choice routing, experts are not activated independently of one another. Therefore, we use a segment router that decides which experts should be activated, also in a segmented manner. Similar to the single expert case, define SRP the upper bound of Fj between the routing decisions of the original router and the segment router, respectively. For a group of experts, SRP measures how well the original router coordinates the experts achieve layer-level or model-level routing consistency.
Like SRP on a single expert, we do not limit the number of activated experts the segment router can choose each position. Nevertheless, we do not want it activate too many experts to achieve high Fi. Therefore, we define the segment routing size ratio I as the ratio between the average number of activated experts of the segment router and the original router (which is usually a fixed number). A small ratio indicates that the local routing consistency of the experts is high enough, so that segment routing does not need to select too many experts to cover real demands under average cases. We use it as supplementary metric to distinguish between cases where groups of experts have similar segment routing best performances.
2.3 SEGMENT CACHE BEST HIT RATE (SCH)
The advantage of SRP as a local routing consistency metric that it solely relies on the expert (group) and the segment length, making it suitable for analyzing individual experts. However, real expert offloading scenarios usually have a hard cache size limit, so the best global Fi score may not be achievable. Moreover, when considering caching performance, F, score is not as straightforward as hit rate (recall). Therefore, we propose Segment Cache best Hit rate (SCH) (Figure 3) which is more related to expert offloading.
Following the expert group case, instead of a segment router with unlimited activation settings, we now consider segment cache with hard cache size limit. More specifically, the cache has specific
4

---

Published as a conference paper at ICLR 2026
cache ratio p. which is the ratio between the cache size and the number of activated experts. This cache works like the other caches in expert offloading systems: for each token, it loads (or preloads) the demanded experts not in the cache while evicting the same number of unused experts The difference is that it evicts experts that are activated the least times in the next m tokens. Under this setting, SCH is defined the hit rate of the segment cache. Due their similar segment-bane -hased behavior, SCH can act as a bridge between SRP and the efficiency of real expert offloading systems.
Note that the upper bound hit-rate of any cache is given by the clairvoyant replacement algo- rithm that evicts the expert whose next activa- tion will occur farthest in the future. Although SCH also modeled on a cache algorithm that relies on some oracle information, the required information is easier learn and predict (future activation frequency vs. precise time of next ac- tivation). Thus, practical cache algorithms are more likely to reach cache hit rate close to SCH. Furthermore, in Section [5.2 we show that SCH is already close to the optimal cache hit rate under certain conditions. Therefore, we stick to SCH to align with SRP-based results.
Router Sequence (8 choose 2)
Segment Coche (2x, m-4)
SCH:HinRate 133318 728
Figure 3: An illustration of segment cache best hit rate (SCH). An oracle expert cache evicts experts (red shade) that are least activated in the next m tokens (grey dash box); SCH is its hit rate.
3 SRP-BASED CONSISTENCY ANALYSIS
3.1 EXPERIMENT SETUP
Models We first conduct experiments on 20 MoE-based LLMs with model sizes ranging from 3B to 57B, covering both popular TansfoeTr,ssormmesss, Mixtral) and recent (DeepSeek- V2, Qwen3) models. We list their architecture and configuration details in Appendix D.I We may also use shorter names (e.g., OLMoE and Qwen3) when there is no ambiguity. Given that many models also have post-trained (e.g., SFT, RL) versions, we compare the local routing consistency between base and post-trained versions of several models in Appendix E.T where we find no significant difference. Therefore, stick to the base version in main experiments.
To validate our observations on existing MoE models concerning architecture design, we further pretrain series of OLMoE-like toy models from seratch, and conduct the same observational experiments on them. The baseline model has 1.43B parameters, with shrunk depth and hidden dimensions compared to the original OLMoE. Each other model modifies one key architectural or training parameter, such as expert granularity, number of shared experts, and load balance loss coefficient, while maintaining the same total number of parameters (I.43B). Appendix D.2 provides more details about the model and training configurations. In the following sections, we refer to the above two groups of models as REAL and TOY, respectively.
Dataset We construct our sample corpus S from two sources: (1) Generic Corpora: We include all 7 categories from RedPajama (Together Computer 2023), including C4, CommonCrawl, Books, Wikipedia, ArXiv, StackExchange, and GitHub. (2) Downstream Application: We append several datasets with cases aligned with modern LLM applications. including arena-human wataaaaumaa--pefeeeeeee www.--rreeeeeeee4-11 140k (LMArena; [MArena 2025), Open (OpenMath: /Toshniwal et al.) (025). OpenCode- Instruct CodenCode; Ahmad et al. 2025), and SeeenScieneeRasoninng (OpenScience; NVIDIA Corporation] 2025). We treat each RedPajama category and downstream application dataset as a distinct domain, and refer to its subset corpus using the data source (Books, GitHub. etc.). The full corpus contains 22,528 input samples, each sample having 512 tokens. Appendix D.3 gives more details on the data processing and input generation process.
Method and configuration We collect every MoE layer's routing decisions for every inpu and for each expert, count the number of activated tokens if in every segment. Although modern LLMs utilize position encodings to distinguish tokens from different positions, demonstrate in Appendix E.2
We only describe the decoding stage; prefilling is similar except that one expert may handle multiple tokens. II encoder-decoder models, encoder layers only consider encoder input, and decoder layers likewise.
5

---

Published as a conference paper at ICLR 2026
that segments from different positions have nearly identical SRP, except for the very first ones that may contain special head tokens. Therefore, we perform calculations on all segments and do not care about their positions. To obtain SRP, count the number of segments with the same f for each expert (group), then compute the Fj score for every a candidate, choose the a that achieves the highest Fj and finally obtain the segment routing best performance and size ratio.
3.2 OVERALL RESULTS
Figure 4 illustrates the SRP of each REAL model under various segment lengths m. While most models have similar SRP and أ when m = 4, the difference between models becomes significant as the segment length increases, yet the relative order remains after 16. There is gap between short-term (m = 4) and long-term (m 2 16) local routing consistency, where many models exhibit the short-term but only few demonstrate the long-term one.
2.46
XF
2.44
2.43
06
8 2.42
SH2
FE
2.41
2.5
SH1
2.39
0.25
0.35
0.4
0.45
SRP
Baseine Pewerfap
ActMore
ActFower
15hrfsp
25hrfap
0.5 11
SRP
5R2
DenseFst
Denselit
KoLB
OverLB
Paveerec LLaMAAarDDl OUME SyttiTanrremmnn LLAW-MOD eehict + OpenPet
SoiiE PICE Queed 34122
Figure 5: SRP and log perplexity of TOY models on the full corpus. We only show m 16 results for simplicity.
PHISSRIC GRIVIGE PibsaAdl
HLD-AE Oward
Figure 4: SRP and e of REAL models on the full corpus.
We roughly divide REAL into four groups that have similar SRP characteristies, whose SRP when m 16 are demonstrated in [able
Group 1 (LLaMA-MoE-22-LLMOO has the highest SRP (> 0.5 when 16) and X (~ 1.25) across all segment lengths, showing strong long-term local routing consistency.
Group 2 (Mixtral-8x7) Laaaa--x----LLL--M-M-
have the second highest SRP (~> 0.48 when m 16),
but their forg-term ý becomes high (~~ 2.5).
Group 3
has a significantly lower SRP than group 2, especially the long-term one (~ 0.36 when m 16), but their ý is bit lower ~ 2.0.
Group LLLLB-MME--SwiihTrrannnte
have the lowest SRP (< 0.31 when m 16): contrasting other models, their short-term is already high, but their long-term becomes lower.
3.3
WHAT AFFECTS LOCAL ROUTING CONSISTENCY?
Table Jaready provides clues about the relationship between local routing consistency (SRP) and model architecture. Nevertheless, due the vast heterogeneity among REAL models, we further validate possible factors with TOY models, results illustrated in Figure Significant points include:
Load balance This is one of the most intuitive factors related to local routing consistency, which is another key feature for efficient MoE inference (Lepikhin al.) (202)). For instance, both DeepSSekk Al et al.] (004b) and [Sklaar et al.) (2025) suggest adding bias to router outputs. Still, the former promotes little activated experts for load balance and the latter promotes recently cached experts for

---

Published as a conference paper at ICLR 2026
Table 1: REAL model SRP in descending order compared with several architecture parameters.
"A:T"; ratio between active and all experts; "S:A": ratio between shared and active experts; "every x": apply MoE every layers; "after Ist": apply MoE after the first layer.
Model
SRP
MoE
A:T
S:A
Model
SRP
MoE
A:T
S:A
LLaMA-MoE-v2
78.16
all
1:4
0
XVERSE-MoE
38.58 all
3:32
1:3
Yuan2.0
63.48
all
1:16
0
Jamba-Mini
38.08 every 2
1:8
0
PowerMoE
55.17
all
1:5
0
SeepScck-V2-Liin
37.92 after Ist
3:32
1:3
Qwen3
54.14
1:16
0
DeepScekMoB
36.94 after
3:32
1:3
Phi-3.5-MoE
51.98
all
1:8
0
Qwen2
36.74
all
1:8
1:1
OLMoE
50.91
all
1:8
0
NLLB MLLB-MoBB (encoder)
25.24
every 1:64
GRIN-MoE
50.39
all
1:8
0
(decoder)
31.35
Mixtral-8x7B
49.36
all
1:4
0
Qwen I. ..-MoE
30.71
all
1:15
1:1
MiniCPM-MoF
48.85
all
1:4
0
OpenMoB
28.77
every 6
1:16
1:2
JetMoF
47.45
all
1:4
0
SwitchTF (encoder)
19.33
every
1:128
LLaMA-MoEE
45.29
1:4
(decoder)
19.27
effective caching. To investigate their relation, we compute the standard deviation (SD) of all experts" activation frequencies model and compare it with SRP in Table 2 Many REAL models with high SRP exhibit imbalanced routing, which largely contributes their local routing consistency (see Appendix E.7). We also verify this on TOY models, where NoLB reaches a high SRP but has very poor load balance, while OverLB, which further prioritizes load balance during training, has very low SRP. Although load balance is also important for efficient training, from the aspect expert offloading applications (e.g., edge computing), it is still worth trading some load balance for local routing consistency if expert offloading will be involved.
Table 2: SRP (m 16) and load balance (LB) measured by activation frequency SD of experts.
Model
SRP
LB
Model
SRP
LB
Model
SRP
LB
LLAMA-MoB-ve
78.16
29.04
XVERSE-ME MoE
38.58
2.71
NoLB
56.42
13.21
Yuan2.0
63.48
13.86
Jamba-Mini
38.08
3.05
ActMore
55.69
6.54
PowerMoF.
55.17
12.90
DeepSeek- DeepScek--2-L.itt
V2-Lite
37.92
2.34
DenseFst
44.87
4.25
Qwen3
54.14
3.19
DeepSeckMoF:
36.94
2.03
DenseHIf
43.67
3.71
Phi-3.5-MoF
51.98
4.90
Qwen2
36.74
6.74
Baseline
43.56
4.02
OLMoE
50.91
6.79
NLLB-Mol (en)
25.24
1.75
FewerExp 41.62
3.63
GRIN-MoE
50.39
3.89
(de)
31.35
2.13
1ShrExp 41.38
3.43
Mixtral-8x BB
49.36
2.70
Qwen1.5-MoE
30.71
0.58
2ShrExp 38.79
3.06
MiniCPM-MM EMo
48.85
2.59
OpenMoE
28.77
2.56
OverLB 36.42
1.79
JetMoE
47.45
1.12
SwitchTF (en)
19.33
0.58
ActFewer 27.13
1.14
ALLMMA-MEE-V
45.29
2.66
(de)
19.27
0.66
Note that models like Qwen3 and GRIN-MoE show high SRP and moderate load balance simultane- ously. Since high local routing consistency almost always means low local load balance. we conclude that these models exhibit good global load balance: While a single query may only activate portion of experts, queries from different topics are likely to activate different sets of experts, eventually covering all experts. Section 4 further reveals that these models possess strong domain-specializer experts that help increase both local routing consistency and global load balance.
Shared experts and expert combination space Besides training objectives, model architecture can also play an important role in forming local routing consistency, the most significant of which we found is the existence of shared experts: Among REAL models, all models in groups and do not use shared models; Among TOY models, Sharel I and Share2, having similar perplexity levels with Baseline, show significantly lower SRP than Baseline. We suggest two potential reasons for why shared experts harm local routing consistency: One reason could be the bypass effect, where more information is processed by shared experts, making the real MoE part less important. Another reason, which is linked multiple MoE design factors, is the decreased size of expert combination space, which is also mentioned by Muennighoff et al." (2025). More specifically, the existence of shared experts decreases the number of both available and activated experts, resulting in fewer possible expert
7

---

Published as a conference paper at ICLR 2026
combinations for routing. This may prevent the router from making local adjustments routing decisions between consecutive tokens, resulting in low local routing consistency. In fact, among the TOY models, 32Exp (use fewer experts) and Top2 (activate fewer experts) do exhibit lower SRP than Baseline, while 16p16 (activate more experts but less than half) show higher SRP. This further demonstrates that more expert combinations benefit local routing consistency. Nevertheless, this is a less significant factor compared to load balance and shared experts, as some REAL models (c.g.. Phi and GRIN) do not strictly follow this rule.
Interleaved MoE layers Different from introducing dense components (shared experts) inside MoE layers, interleaving or concatenating MoE layers with dense ones seems to have less significant impact. Although Skipl (first layer dense) and Sparse2 (MoE every 2 layers) possess higher SRP than Baseline, the lead is minor, and both models suffer from high perplexity. Meanwhile, all REAL models that involve dense layers fall into groups 3 and 4. However, the low SRP of these models may not be due to dense layers: they either use shared experts (e.g., DeepSeekMoE) or activate experts sparsely (e.g.. Switch (ransforranss both of which can contribute low local routing consistency.
4 LOCAL ROUTING CONSISTENCY AND EXPERT SPECIALIZATION
4.1 DOMAIN-WISE LOCAL ROUTING CONSISTENCY
In Section 3 we analyze models on the full corpus, which consists of text data from 11 different domains. However, each domain has its token distributions, which may affect the router decision's distribution.
the relative difference between domain-wise and global SRP of each model when 16, where we observe three different patterns among all models: (1) Models like Phi-3.5-MoE, GRIN-MoE, and OLMoE have significantly higher SRP on ArXiv, StackExchange and GitHub, whose SRP can be more than 10% higher than global SRP. They also exhibit higher SRP on OpenMath and OpenCode, indicating specialized experts for math and coding tasks. (2) Models like LLAMA-MMF-v2, Yuan2.0, and Qwen3 have significant higher SRP Wikipedia and other generic domains but lower on OpenMath and OpenCode. They seem to have specialized experts generic text (e.g., multilingual experts) instead of math or coding. (3) Models like Mixtral-8x7B, MiniCPM-MoE and JetMoE have similar SRP across all domains with insignificant differences. All of them have mediocre to low SRP, Interestingly, we found that all TOY models show the first pattern (like the original OLMoE) regardless of architectural or training configuration tweaks, indicating that the formation such patterns may stem from the pretraining data distribution.
LLMMAA-Mo x2
Yusn2.
Powerftn
Owen3
Ph--355Mof
DI MaF
CRIN Mof
Mistral Bt77
MinCPMM-M
MrM
42%
SORTERSEERS XVERSEMAE
2AR3E RUSIRR
R DERRFERR LAREB-SSRRA NLLB
LLE
Jamba Mini
DcesSook
Desp5colMNE
Owan2
Owen11 MUE
OsenMot Switch andformrss
*g-
SRRgS4CEgRD CAREEMDERSS CASERMEERRA CRREE D2REE
DAREEWDEERR
Figure 6: SRP (m = 16) on each domain corpus, relative to SRP on the full corpus. In-group bars (from left to right): C4, CommonCrawl, Books, Wikipedia, ArXiv, StackExchange, Hub, LMArena, OpenMath, OpenCode, and OpenSeience. For encoder-decoder models, light colors represent the encoder and dark colors represent the decoder.
4.2 EXPERT SPECIALIZATION
We argue that domain-wise local routing consistency patterns appear across models due to specialized experts in each model. To clarify this, we consider two types of expert specialization, first introduced by Muennighoff et al. (2025): (1) Domain specialization: the normalized frequency of an expert being activated tokens from specific domain. We compute the coefficient of variation (CV) of activation frequency across all domains as a domain-free metric. (2) Vocabulary specialization: the normalized frequency of an expert being activated specific token ID. We follow Muennighoff
8

---

Published as a conference paper at ICLR 2026
et al. (2025) to obtain the vocabulary specialization of each expert. We compare each model's SRP, average expert specialization, and the correlation between its experts' specialization and SRP in Figure 7 expert distribution between specialization and SRP is also demonstrated in Appendix E.7
Domain specialization Many REAL models show a positive correlation between their experts" domain specialization and SRP. Exceptions include LLaMA-MoE-v2. which constantly activates a group of ex- perts, resulting in very high SRP: Qwen2 and LLaMA-MoE, meanwhile, hardly have any domain specialization. In contrast, Qwen3, Phi-3.5-MoE, GRIN-MoE, and OLMoE exhibit high SRP, high average expert domain specialization, and strong correlation between them simultancously. As mentioned in Section 3.3 these models also demonstrate global load balance (see Table with domain specialized experts this is explanable: A domain-specializer expert tends to be active when the context comes from certain domains or topics, but not others. Therefore, when the context matches, the expert is likely to be consis- tently activated (local routing consistency), but as long the context becomes unre- lated, the expert becomes inactive (global load balance).
Damain Spac.
42
Comsttitn
455
02
0.4
3.5
0.5
3.7
0.5
03
244
35
0.5
07
00
SAP
SAP
Oweech
Jarka
Qwoe
Figure 7: SRP, expert specialization, and their correla- tion. Correlation between SRP and expert specialization is per model, based on single-expert results.
Vocabulary specialization We consider
three kinds of vocabulary specialization on the input, the model's predicted output, and the ground- truth, respectively (Muennighoff et al., (025). Most models demonstrate negative or insignificant correlation between input vocabulary specialization and SRP, except for LLaMA-MoE- due to its constantly activated experts. On the other hand, SRP is slightly positively correlated to prediction or ground truth vocabulary specialization. We conjecture that such specialization happens more in later layers (Muennighoff et al. (025) that process high-level information related the context topic.
Above all, we can see that domain specialization plays a more important role in local routing consistency than vocabulary specialization, especially on models with both high local routing consistency and global load balance.
5
SCH-BASED CONSISTENCY ANALYSIS
5.1
OVERALL RESULTS
As mentioned in Section SRP has several flaws that hinder its application in expert offloading. This section focuses on the segment cache best hit rate (SCH), which works with a size limit, to obtain more straightforward insight into expert offloading and cache management. We calculate each model and each layer's SCH on every possible cache size by simulating the oracle cache described in Section and recording the hit rate.
Figure llsttraess SCH of REAL models under different ms and We can easily identify the four groups of models mentioned Section 3.2 starting from m 16: Group models have the fastest growing SCH with respect to rho when p is small, well turning points near 2, after which they share similar SCH with group 2 models. Meanwhile, models from groups 3 and have relatively low SCH, growing nearly linearly as p increases. Since only group I models (with the highest local routing consistency) have turning points on SCH, we claim that in general. = can balance cache effectiveness and efficiency.

---

Published as a conference paper at ICLR 2026
0.0
OUNGE Solichiun mers (trecder)
Mod-i
jusMec
Opestiut
Doand 5-Hoz DescSedk-r2uui
DeSSSEcAPUE EVERSS Huz
Owor3
G4374 Motrrll tx jantto
RLLS (encoder Oween
... RLLD Mot (Decocer)
Figure SCH of REAL models on the full corpus under different segment length m and cache ratio Solid line: group 1; dashed: group 2; dash-dotted: group 3; dotted: group 4.
5.2 SCH AND COMMON CACHE ALGORITHM
Since SCH based on an ideal cache system that relies on oracle information, it crucial to understand how well it is correlated with real-world implementations. Table BaaleBllsss Bists the correlation between SCH and the cache hit rate of two widely adopted cache algorithmss: least recently used (LRU) and least frequently used (LFU). All compared cache algorithms have hit rates highly correlated to SCH, even for short segments (m = 4). Furthermore, Appendix [E.]arevslls that SRP and SCH are also highly correlated with each other. This suggests that models with higher local routing consistency tend to achieve higher expert cache hit rates, hence greater performance gain with expert offloading.
Furthermore,
the optimal cache hit rate given by the clairvoyant replacement algorithm as a baseline, and compared the relative hit rate of the mentioned cache algorithms in Figure (using Baseline as an example). When the cache size is moderate, SCH (m 16) is very close to the optimal hit rate, while LRU and LFU consistently have hit rates lower than SCH. From this point, SCH can act as "practical" ideal case for analytical analysis on expert offloading.
Table 3: Correlation between SCH and hit rate of cache algorithms across REAL models. LRU: least recently used; LFU: least frequently used.
Table 4: Baseline's SCH (m 16) and hit rates of LRU and LFU relative to the optimal hit rate under different cache ratios p. The optimal hit
LRU
LFU
Fixed
rate is normalized to 100.
LRU
LFU
SCH
4
81.20
77.39
76.26
90.43
88.70
89.79
1.0
56.49
61.92
80.97
64
93.10
92.82
95.50
2.0
67.04
70.87
90.55
256
97.52
99.20
97.91
3.0
75.26
78.35
96.23
6 CONCLUSION
In this paper, we investigate the property MoE LLMs where similar experts can be continuously activated, namely local routing consistency. We propose two metrics to measure this property: segment routing best performance (SRP) and segment cache best hit rate (SCH). We compare SRP and SCH between multiple models and identify several key designs that may help improve local routing consistency of MoE. L.Mss, including local load balance trade-off, (no) shared experts and (enlarging) expert combination space. We also identify that domain semain--peciialzzed experts contribute more to local routing consistency and help achieving global load balance. We further suggest that cache size around 2x the number of active experts can balance cache effectiveness and efficiency.
The work is supported by Al for Science Program, Shanghai Municipal Commission of Economy and Informatization 22025-ZZL-RG22 ------BBBX------ The project's computational resources partially supported by CFFF platform of Fudan University.
Ackmowledgements
10

---

Published as a conference paper at ICLR 2026
Ethics statement Our analytic methods and results still help design new MoE LLMs friendly to expert offloading and enable deployment on resource- conuurccoontrrieedd edge devices. While our study will likely have indirect social impact, developers implementing local routing consistency to build more powerful LLMS must take responsibility for their products' societal implications.
Reproducibility statement We constructed our sample corpus with deterministic algorithms, and we will publish the sampled corpus to ensure reproducibility. We also conducted all experiments with a deterministic configuration and will release relevant source code.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffiman, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy. Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang. Lijuan Wang, Shuohang Wang, Xin Wang. Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt. Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang. Jianwei Yang. Yifan Yang. Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint, April 2024. doi: 008.550/ARXII.24441144444
Wasi Uddin Ahmad, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Vahid Noroozi, Somshubra Majumdar, and Boris Ginsburg. Opencodeinstruec:: A large-scale instruction tuning dataset for code Ilms. arXiv preprint, April 2025. doi: 10.48550/ARXIV 2504.04030
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering (TKDE) 2025, pp. 1-20, June 2024. ISSN 2326-3865, doi: 10. 2109/kkde.005555540222
Shengzhuang Chen, Ying Wei, and Jonathan Richard Schwarz. Automatic expert discovery in LLM upeyeling via sparse interpolated mixture--ffexpperts In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 16703-1677 Austria, July 2025. Association for Computational Linguistics. ISBN 979 8 89176-2971111 doi: 202/
Vienna,
10.18653/v1/200 acl-long. 8 816.
2025 lon-long..0. 816/
Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. Task-specific expert pruning for sparse mixture-of arXiv preprint, June 2022. doi: 10.48550/ARXIV 22006.00277
Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In Lun-Wer Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1280, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 00188653/11200 h/tp://.-o2..alloooggggg lclg
acl long. 70/

---

Published as a conference paper at ICLR 2026
DeepSeek-A1, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao. Chengqi Dengr, Chong Ruan, Damai Dai. Daya Guo, Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin. Fuli Luo. Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song. Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang. Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen. R. L. Jin, Ruiqi Ge, Ruizhe Pan, Rumxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma. Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang. Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang. Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang. Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma. Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang. Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen. Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2 A strong, economical, and efficient mixture-of-eepprrs language model. arXiv preprint, May 2024a. doi: 10.48550/ARXIV WAAS5///AAII..2.......M 2444.04434.
DeepSeek-Al, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu. Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang. Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang, Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. Jin, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye. Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun. Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xubeng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang. Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang. Zhenda Xie, Zhengyan Zhang. Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report. arXiv preprint, December 2024b. doi: 10.48550/ARXIV 221.
Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai Helen Li, and Yiran Chen. Sida: Sparsity data-aware serving for efficient and scalable large mixture-of-expers models. In P. Gibbons, G. Pekhimenko, and C. De Sa (eds.). Proceedings of Machine Learning and Systems, volume 6, pp. 224 238, 2024. URL https: https::/pproceedingg... http://gw....aapeeeeeee/eaa/aaaa
12

---

Published as a conference paper at ICLR 2026
Artyom Eliseev and Denis Mazur. Fast inference of mixture-of-expert language models with offloading. arXiv preprint, December 2023. doi: 10.48550ARXX1.331227755
Zhiyuan Fang, Zicong Hong, Yuegui Huang, Yufeng Lyu, Wuhui Chen, Yue Yu, Fan Yu, and Zibin Zheng. Accurate expert predictions in moe inference via cross-layer gate. arXiv preprint, February 2025. doi: 10.48550/ARXIV 1222.12224.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-35 2022. URL http::/mtmm org/ http/pppes////2//222 0998. html
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixtureoof-LoRA: An efficient multitask tuning method for large language models. In Nicoletta Calzolari, Min-Yer Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (IREEC- COLING 2024), pp. 11371 1380, Torino, Italia, May 2024. ELRA and ICCI. www.ooooooooooooooooooooooooooo org/ 2024. 1rec- main. m94/
URL
Hongcan Guo. Haolang Lu, Guoshun Nan, Bolun Chu, Jialin Zhuang, Yuan Yang. Wenhao Che, Xinye Cao. Sicong Leng, Qimei Cui, and Xudong Jiang. Advancing expert specialization for better moe. In The Thirty- Annual Conference on Neural Information Processing Systems, 2025. URL http:://pppereeiieeeeeeffrrrrrre
Xin He, Shunkang Zhang. Yuxin Wang, Haiyan Yin, Zihao Zeng, Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, Ivor Tsang, and Ong Yew Soon. Expertflow: Optimized expert activation and token allocation for efficient mixture-of-experts inference. arXiv preprint, October 2024. doi: 10.48550/ARXIV. 1790.19954.
Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang. Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, da- hai li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling, 2024. URL https://openreview.. https://ooeemrriii.eeeeffuuuuum wtwp:eewwwwwwweeeounneee.ee net:/oormmmd==32222rF.
Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S. Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee. Towards moe deployment: Mitigating inefficiencies in mixture-of-exper (moe) inference. arXiv preprint, March 2023. doi: 00448500ARXIV.230000 182.
Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and Xiaojuan Qi. Mixture compressor for mixture-of-expprts LLMs gains more. In The Thirteenth International Conference on Learning Representations, 2025. URL httphss///ppeeeeiiwweeeee -hmnidddhheeYJosW
Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang. Ting Cao, and Mao Yang. Pre- Pre-gated moe: An algorithm-syster co-design for fast and scalable mixture-of-expert inference. In 2024 ACM/IEEE s Annual International Symposiun on Computer Architecture (ISCA), pp. 1018-1031, June 2024. doi: 10. D0.119991SCA59997.222400000
Albert Q. Jiang, Alexandre Sablayrolles. Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mixtral of experts. arXiv preprint, January 2024. doi: 10.48550/ARXIV. 2401.04088.
Keisuke Kamahori, Yile Gu, Kan Zhu, and Baris Kasikci. Fiddler: CPU-GPU orchestration for fast inference mniturre-of-expeee models. In 5th Workshop on practical for limited/low resource settings, 2024. eeeei
13

---

Published as a conference paper at ICLR 2026
Rui Kong, Yuanchun Li, Qingtian Feng, Weijun Wang, Xiaozhou Ye, Ye Ouyang, Linghe Kong, and Yunxin Liu. SwapMoE: Serving off-the-sshe shelf MoE-based large language models with tunable memory budget. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties ( (Volume I: Long Papers), pp. 6710-6720, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: URL
Lun-Wei
orwpe//.r//222222 attp:/222.acc111oom0 acl-long. 363/
Barak Lenz, Opher Lieber. Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M. Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar, Itay Dalmedigos, Jhonathan Osin, Julie Fadlon, Maria Rozman, Matan Danos, Michael Gokhman, Mor Zusman, Naama Gidron, Nir Ratner, Noam Gat, Noam Rozen, Oded Fried, Ohad Leshno, Omer Antverg, Omri Abend, Or Dagan, Orit Cohavi, Raz Alon, Ro'i Belson, Roi Cohen, Rom Gilad, Roman Glozman, Shahar Lev, Shai Shalev-Shwarts Shaked Haim Meirom, Tal Delbari, Tal Ness, Tomer Asida, Tom Ben Gal, Tom Braude, Uriya Pumerantz, Josh Cohen, Yonatan Belinkov, Yuval Globerson, Yuval Peleg Levy, and Yoav Shoham. Jamba: Hybrid transformer-mamba language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevieww rdtt://oorommddddFa1
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL /ttpps://operreeiii wtw.r/perrreeeeeeeeeeoorrrroiooo www.tneeeeetteeneeeeeee (ott//oorrmn1ddr/reeeettt
Dengehun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts. arXiv preprint, April 2024a. doi: 10.48550/ARX IV IV.2404.15159.
Jiamin Li. Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. Accelerating distributed MoE training and inference with lina. In 2023 USENIX Anmual Technical Conference (USENIX ATC 23), pp. 945-959, Boston, MA, July 2023. USENIX Association. ISBN 978-1-939133-35-9 URL https://www usenix.ee.... org/ /ttofrrreeeeacc3///eeeettttttttt jiamin
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient SMoe with hints from its routing policy. In The Twelfth International Conference Learning Representations, 2024b. URL https
//openreview.net//oormit www.teet/ooonnee mett/foorummiddee//ssyy...
Jiacheng Liu, Peng Tang, Wenfeng Wang, Yuhang Ren, Xiaofeng Hou, Pheng Ann Heng, Minyi Guo, and Chao Li. A survey on inference optimization techniques for mixture of experts models. ACM Comput. Surv., February 2026. ISSN 0360-0300. doi: 10. 10.1145/3794845. URL hhtps Just Accepted.
Liyuan Liu, Young Jin Kim, Shuohang Wang, Chen Liang, Yelong Shen, Hao Cheng, Xiaodong Liu, Masahiro Tanaka, Xiaoxia Wu, Wenxiang Hu, Vishrav Chaudhary, Zeqi Lin, Chenruidong Zhang, Jilong Xue, Hany Awadalla, Jianfeng Gao, and Weizhu Chen. Grin: Gradient-informed moc. arXiv preprint, September 2024. doi: 10.48550/ARXIV 1409.12136.
LMArena. arena--humaan-preeeenee--- s/lmarena- w/Lmarena-ai/arenaaaa
August 2025. URL htttss://uuggingaae..com coodaaataacceeeooataaaeeeeeeee
140k
Ka Man Lo. Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. A closer look into mixture-of-experts in large language models. In Luis Chiruzzo. Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACI 2025, pp. 4427-4447, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-8911 19176-11771117 195-7. doi: 10.11655//122255501 http:///22.....inss....aaa 2aaedinngsnaall255... naa. httltttss///ccanttiii h c1.2517
www.o/////22/ findings naa
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-expert large
14

---

Published as a conference paper at ICLR 2026
language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6159-6172 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1/663//112024..aa-oooo
http://ooyyoooo/22...o. ooog./o..../..2.......o...g. 334/
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Jacob Morrison, Sewon Min, Weijia Shi, Evan Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig. Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. OLMoe: Open mixture-of-expeeee experts language models. In The Thirteenth International Conference Learning Representations, 2025. URL htttps://oppereete
net net/forum? fnrforrmmmmxxxx TXTkb Bmqq
Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. htttp::////ppnnree Featured Certification.
ew. eww.nt//ooumm1dd711999002.
NLLB Team, Marta R. Costa-jussa, James Cross, Onur Çelebi. Maha Elbayad, Kenneth Heafield, Kevin Hefferman, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human- centere machine translation. arXiv preprint, July 2022. doi: 10.48550/ARXIV.
NVIDIA Corporation. OpenSciienceReassonnngg2 June 2025. URL http:://///ugggngfacccec dat betts//wwidaa//pppmsiieennesssss
Quang Pham, Giang Do, Huy Nguyen, TrungTin Nguyen, Chenghao Liu, Mina Sartipi, Binh T. Nguyen, Savitha Ramasamy, Xiaoli Li, Steven Hoi, and Nhat Ho. Competesmoe effective training of sparse mixture of experts via competition. arXiv preprint, February 2024. doi: 10.48550/ARXIV 0A.S50AARRIXV2222022222222 2402.0226...
Xiaoye Qu, Daize Dong, Xuyang Hu, Tong Zhu, Weigao Sun, and Yu Cheng. Llama-moe v2: Exploring sparsity of llama from perspective of mixture-of-experts with post-training. arXiv preprint, November 2024. doi: 10.48550/ARXIV.224 22111..50...........
Qwen Team. Qwenl.5-man Matching 7b model performance with 1/3 activated parameters", February 2024. htttts:////:ttln.
io/blog/qwen-nonn
Samyam Rajbhandari, Conglong Li, Zhewei Yao. Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed-MoE: Advancing mixture-of- experts inference and training to power next-generation Al scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceed- ings of the 39th International Conference on Machine Learning, volume 162 of Proceed- ings of Machine Learning Research, pp. 18332-18346. PMLR, July 2022. URL https //proceedings. mlr. htt..rpeeee////////aaaaaarrrrrr
html
Jie Ren, Dong Xu, Shuangyan Yang. Jiacheng Zhao, Zhicheng Li, Christian Navasca, Chenxi Wang, Harry Xu, and Dong Li. Enabling large dynamic neural network training with learning-based memory management. In 2024 IEEE International Symposium on Perfo-Perfommnnce Computer Architecture (HPCA), pp. 788 802, March 2024. doi: 10. 10.1199HPCA5774..224..0000
Noam Shazeer, * Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gater mixture-of-exper layer. In International Conference on Learning Representations, 2017. URL httptht::t//oeeeviiiieeee et/forum?id-B1 ckMMgg
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Y. Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou.
15

---

Published as a conference paper at ICLR 2026
Mixture-of offture-of-expertt meets instruction tuning: A winning combination for large language models. In The Tweifth International Conference on Learning Representations, 2024a. URL https //openneview.net/foooe wttp://wweettooomm/dooooooooo//o
Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, and Chuang Gan. Mod- uleformer: Modularity emerges from mixture-of-expert.. arXiv preprint, June 2023, doi: 10.48550/ARXIV .336.004640.
Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. Jetmoe: Reaching Ilama2 performance with 0.1m dollars. arXiv preprint, April 2024b. doi: 1044850/ARXXIV.2444.7711
Yikang Shen, Matthew Stallone, Mayank Mishra, Gaoyuan Zhang, Shawn Tan, Aditya Prasad, Adriana Meza Soria, David D. Cox, and Rameswar Panda. Power scheduler: batch size and token number agnostic learning rate scheduler. arXiv preprint, August 2024c. doi: 10.48550/ARX WV.2408.13599
Andri Skliar, Ties van Rozendaal, Romain Lepert, Todor Boinovski, Mart Van Baalen, Markus Nagel, Paul N. Whatmough, and Babak Ehteshami Bejnordi. Mixture of cache-conditional experts for efficient mobile device inference. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL httt:s///pperreeiie..teeffrrrree 2W22K
Xiaoniu Song, Zihang Zhong, Rong Chen, and Haibo Chen. Promoe: Fast moe-based Ilm serving using proactive caching. arXiv preprint, October 2024. doi: 10.48550/ARXIV.22 241.0X2222211.222144
Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, Pheng-Am Ann Heng, Chao Li. and Minyi Guo. Hobbit: A mixed precision expert offloading system for fast moe inference. arXiv preprint, November 2024. doi: 10.48550/ARXIV.220 2211 .01433.
Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, April 2023. URL /ttp::ss: thg.p:b.///gt.bbb........ com http:/tw...ooggggggeeeeooooeeeee computer/RedPpppapeeeeee
-ata
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstructt-2 Accelerating AI for math with massive open-source instruction data. In The Thirteenth International Conference on Learning Representations, 2025. URL
cctps://openreviiee wttp://ewwwweeeeeeoooooooooooooo
Shaohua Wu, Jiangang Luo, Xi Chen, Lingjun Li, Xudong Zhao, Tong Yu, Chao Wang, Yue Wang, Fei Wang, Weixu Qiao, Houbo He, Zeru Zhang, Zeyu Sun, Junxiong Mao, and Chong Shen. Yuan 2.0-m32: Mixture of experts with attention router. arXiv preprint, May 2024. doi: 10.48550/ARXIV. 240..11977..
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models. Forty-firsi International Conference on Machine Learning, 2024a. URL
8Lt5
Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. Moe-infinity: Efficient moe inference personal machines with sparsity-aware expert cache. arXiv preprint, January 2024b. doi: 10.48550/ARXIV. 240.V...011..131111.
XVERSE Technology Inc. XVERSE-MOE-A4225 April 2024. URL httphttsss//hugiiggfaee co/x verse/XVERSE-Mot-Nn
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. arXiv preprint, July 2024a. doi: N0.48550/ARXII 24077766.6.
16

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint, May 2025. doi: 10. 10.48550/ARXIV 2505.09388.
Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. MoE-i": Compressing mixture experts models through inter-expert pruning and intra-expert low-rank decomposition. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. (0456-10466, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/v http://///22...f// 2i0.224.inniggs-emnnp.. gs es.emn1p.6122 612/
URL /tttpp::/aalannthooogs// org/ findin
Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. Edgemoe: Empowering sparse large language models on mobile devices. IEEE Transactions on Mobile Computing, pp. 1-16, 2025. ISSN 358-0660. doi: 10.1 101/0T0C..225555446444
Hanfei Yu, Xingqi Cui, Hong Zhang, Hao Wang. and Hao Wang. fmooe: Fine- graieer expert offloading for large mixture-of-experts serving. arXiv preprint, February 2025. doi: 10.48550/ARXIV 25002. 05370.
Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, and Zhang Xiong. Mixture of attention heads: Selecting attention heads per token. In Yoav Goldberg, Zormitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4150-4162, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational L Linguistics. doi: 10. 111633/11/22222mmmmpp gy. www.22222m.mmpppm...m.mmmmmm 2787
URL https:hts////clatttii
Yujie Zhang, Shivam Aggarwal, and Tulika Mitra. Daop: Data-aware offloading and predictive pre- calculation for efficient moe inference. In 2025 Design, Automation & Test in Europe Conference (DATE), pp. 1-7, March 2025. doi: 00.23919/DATH64A0 2022 202...9922.......4...
Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, Ru Huang, and Meng Li. Adapmoe: Adaptive sensitivity-baat based expert gating and management for efficient moe inference. In Proceed- Computer-Aided Design, ICCAD '24,
ings of the 43rd IEEE/ACM International Conference Machinery. ISBN 9798400710773. doi: New York, NY, USA, 2025. Association for Computing 10.1145/367655663666667 URL httLhhttss:///ii... http://oo.orrgg///// org/ 01411.11111113356666.11 3676741
Zexuan Zhong, Mengzhou Xia, Danqi Chen, and Mike Lewis. Lory: Fully differentiable mixture-of- experts for autoregressive language model pre-training In First Conference on Language Modeling. 2024. http://pes..ggeeeeeeeeeeee//r/ra
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao Andrew M. Dai, zhifeng Chen, Quoc V. Le, and James Laudon. Mixture-of-experts with expert choice routing. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems, volume 35, pp. 7103-7114, Curran Associates, Inc., 2022. URL neur https///nnsssttteeeeeee/eeeeeeee 3613de9880772ebb 3667 Pan
Tong Zhu, Xiaoye Qu, Daize Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. LLaMA- MoE: Building mixture-of-experts from LLaMA with continual pre-training. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 15913-18 10113311923,,, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v 1026533/1/222..mmllm wtt.:/02...mmmmmmamme canlp--min.890 m90. URL
http:://eelanthooogyy.oo///// http:///e2222emmmmmaammmmmm 890/
17

---

Published as a conference paper at ICLR 2026
A USE OF LLMS
We primarily employ LLMs to polish the writing of this paper, utilizing tools such as Grammarly and Writeful. In all other cases, LLMs are the object of our experiments and analyses, and we do not use them for other purposes.
B RELATED WORK
B.1 MOE-BAN HOE-BASED LLM AND EXPERT ANALYSIS
Since its introduction into large neural networks, MoE has become critical strategy to build large
language models up to trillions of parameters (Shazeer et al. 2017 Fedus et al. 2022 Rajbhandari
al. 2022). While some early models like
(Fedus al.
(022) and NLLB
(NLLB Team et al. 2022) employ encoder-decoder structures as their backbone, due to the success
GPT-3, the most recent popular MoE-based LLMs use decoder-only structures (Jiang et al. 2024 Yang et al. 2024a DeepSeek-A AI et al. 2024b; Abdin et al.) 2024), replacing their original FFN layers with MoE layers containing multiple experts (other components may be replaced too, e-g.. self-attention (Shen et al.) 2023] 2024b) and LoRA (Li et al., 2024a Feng et al. 20244). Cai et al. (2024) systematically introduces MoE architectures and implementation LLMs.
The popularity of MoE LLMs has triggered interest in understanding how experts are activated in such models. Many model reports and individual studies focused on the relation between expert selection and the input context. For example, Muennighoff et al. (2025) reported that OLMoE shows a significant difference in expert activity across different domains. Contrastively, Xue et al. (2024a) found that the routing choice of OpenMoE is highly related to the input token rather than the input context. Other works investigated the similarity among expert activation patterns (Li et al. 2024b Lu
et al.
2024), well the relation between expert output and routing choice (Pham et al. 2024
et al. 2025
2025 Some further proposed methods to reinforce such patterns (Guo et al. 2025 Chen et al.
However, few of them have focused on the local activation pattern of experts. For example,
Jiang et al. (2024) reported that in Mixtral, experts are more likely to be activated consecutively, compared the random case. While their results provide fundamental support for many efficient MoE inference systems Liu et al. (2026), they only examined the case of 2 consecutive tokens, which may be insufficient to ensure the consistency of expert activation in longer segments.
B.2 EFFICIENT MOE INFERENCE AND EXPERT OFFLOADING
The discrete nature of routers and redundant parameters has caused MoE models to infer more slowly and consume more memory than dense models with the same number of activated parameters. Many techniques have been proposed to boost the inference of MoE models, ranging from model
modifications like model compression (Chen et al.] 2022 Huang et al. 2025 Yang et al. 2024b Rajbhandari et al., 2022) and soft routing (Muqeeth al. 2024| /hong al. 2024) system
implementations like baal-balanced expert parallel (Lepikhin et al. 2022 Huang et al. 2023 Li al. 2023) and hardware adaptation (DeepSeck-AL et al. 2024b Yi ct al. 2025). Liu et (2026) provides an in-depth summary of various inference optimization strategies of MoE models.
In this paper, we mainly focus on the potential performance of expert offloading, which enables lossless inference of MoE models on memory-comeeeem conntrri-d devices by caching only some experts on (fast) memory while leaving others on slow memory or disk storage. Many such systems use pretrained external models and/or information from previous layers to prefetch experts for upcoming layers (Ren et al. 2024| Du et al. 2024. He al., 2024 [ong et al., 2024). Various expert offloading systems propose curated heuristies manage the expert cache (Skliar et al. 2025 Xue et al.) 2024bb]
Yu et al. 2025 Fang et al. 2025). Among them, some examine the locality of expert activations as empirical support for expert caching efficiency:
Jiang et al. (2024) first observed that Mixtral-8x77 7B is likely to choose the same expert for the next token, with probabilities higher than the random expectation. [Eliseev & Mazur (2023) extended the argument to 2-4 consecutive tokens through a case study, and boosted inference performance the same model with LRU caching (plus other techniques such as prefetching and quantization). These
18

---

Published as a conference paper at ICLR 2026
works align with settings, but their analyses are limited to single model (Mixtral-8x7B) -x7rr, short token spans, and lack a systematic and quantitative study.
Xue al. (2024b) reported frequent expert reuse during decoding, and observed that the reused experts depend on the prefilled input. Zhang et al. (025)) found similar routing choices between prefilling and decoding stages. Both observations are utilized back the effectiveness of expert caching, yet are too coarse in terms of locality (at the input level instead of the token span level).
As more MoE LLMs emerge, understanding what models are more friendly expert offloading becomes important for the development of both MoE architectures and expert offloading methods.
C FORMAL DEFINITIONS AND PROOFS
C.I FORMAL DEFINITION OF SRP
Single expert For any input sequence T [fi...., 4r7l we denote the activation sequence of expert e on T as A(e, T) = (i....., (.r)) where 0g E {0, 1] indicates whether e is activated on 4 (a, 1) or not (a, 0). segment-based router R with segment length m > 0 for expert e will
try to mimic [ap..... 4p+m- -1] for any I' and p. Let R (T.p) = [b_...., 6m] be its prediction, with the segment- segment-prediction constraint:
b, 0, vi=1,...
or b, =1 1, vi
(2)
In other words, 1s decides that is either always active or always inactive on [t,..... -- For simplicity, we write (T.
O and 1ett (T,p) = 1 for the two cases respectively. By treating each segment routing attempt as a binary classification task with m samples, and considering all possible segments of all possible inputs, we can calculate the F1 score of RT:
2x E R (T,P) (e,TTppm
Fi RR =
(3)
Σ
ITI 11--++1 p=1 (m Tn- (T, + f(e, p,
where [e, T.p.m) = Di-p Dptm-1 [(t] is the active frequency of the segment of T with length m starting at position p. We demonstrate the detailed process to obtain this equation in Appendix C.2. Based on Equation 3 we define the segment routing best performance of e under segment length m as the maximum Fi score any R can achieve: SRP(e, m) - max Fi (R")). Furthermore, in Appendix C.3 we prove that Fi (R is maximized if and only if 1 gives active predictions for all segments that activates at least oe times, where a [0, m] is only related to and m:
②In ExterTpmmaap [e, T.p.p.pm m)
SRP m)
(4)
Σx ST +1 [m (e,,, T,p.m) 2 a.) + (e,TT,p,
Therefore, SRP(e, m) is intrinsic property of the expert that reflects its local routing consistency, unrelated to any specific segment routing methods.
Expert group For group of experts E, let RT be a segment- based router that decides whether each expert E E should be activated in a segment of some input T' with length m; more specificallyy Re T, p) is prediction sequence similar to 1 (T,p) p) that also follows the segment--rediicton constraint (Equation?)
2). Following the same procedure in Appendix C.2 we have
2E ETT mm EEEE Rm (e. T, T,p) f(e.T.p.m)
F
(5)
Σx ΣS' pp1
- E.EE (m (e, (,)) + (e,T,,p,)) m))
Again, (R maximized and only if R' gives active predictions for all expert-segment pairs where the expert is activated at least a times in the segment, where a is decided by E and m. Therefore we have
② T
Σ (l,TTppm
Fle.T.
SRP(E, m) A max F, (RE)
(6)
T m
Σ
Σ
Σ
|m [[fe, T,p,m) 2 ar) + (e,TT,p,
e
SRP(E, m) measures how well a group of experts is coordinated by the original router(s) to achieve layer-level or model-level local routing consistency.
19

---

Published as a conference paper at ICLR 2026
C.2 PROOF OF EQUATION
In Appendix C.I we consider each routing decision of Re for segment of length m as binary classification task with m samples. If we merge all samples from all segments of all possible inputs
into one global binary classification task, and define f(e, T, p, m) Limp E A(e, []] as in
Appendix C.1
we will have the following prediction statistics:
71-m+1
TP(R.)
=
Σ
Σ
[[(e, T)p +i-ptii10 1] 1A R (T,p) = 1]
T
==1
m--++1
Σ
Σ
(T.
S(e,T,p,
77 M+++1
FP(R..) Σ
p)] =
71-m+1
Σ
Σ
(m f(c,T.p.m)
(8)
m+
m
FN(R.) Σ
Σ
Σ
R =
1=1
17|-++1
Σ
Σ
R (T, (pp.)
Therefore we have
Fi (R()
1/Precision( (1T) + 1/ Recall
(TP(RT) + FP Rm ) //TPP(RR) (R + (TP(RT) + FN(R
(
(P((RT) FP Pm
(TP(RM)) FN (N.)
(10)
2 ②Σn E 2p=1
M1-m+1 Rs (T.P) f(e, m.
T p=1 m R (T,p) + Er ΣpA
m++
17| 177 f(e,T,p.m)
2 ΣT E|r -
m R (T,p) f(e, T.p.m)
Σr Σ 2p=1
71-m+1 in (T.)) (e,
which gives Equation
C.3 PROOF OF EQUATION
Assume that RT (To, Po) for some e, m > 0, R To and Po, then we have
(R = Er p=1
m1-m+1 R T)pp: fle, Tp
Σr r
m+11 (m Rume (T,p) [e, m)y p=1
2Σ > +1 R (T,p) f(e, T, m)
m Σr
1=
m- Rm + ΣT 2p=1 mm+
(11)
2 T TTTAppp
R (T, p)
m ETsTArorm R (T, + Σ- -|T m+1 fle.T.p.m)
2X
mY +Z
20

---

Published as a conference paper at ICLR 2026
where
Σ 1
x=
Σ
Re (T,p) T.p.))
Y= E (T.P).
Z=
Σ
(e,,T
T+To pspo
T+To
pspp
Let 1h be a copy of te except that (T, p) = 1; all other routing decisions remain the same.
Then the Fi score of the new segment router will be
2Σ IT| 1+
Re [e. T. m)
Fi
Pm
Er 2 T m++ m R (T, + fe, T, p. m)
p=1
2(X + [e, To,Po,m)
m(Y 1) Z
(12)
(mY + *)) Fi( + 2f To.Po. m) m((Y + +1)++++++ 11-22 2
(mY + -) - Fi( + m [2f(e, To, Po, Po, mm/ml
(my + 7) + m
which is a weighted mean of (i( and 2e To, po, m)/mmmm m/m with weights my + 2 and m. Note
that Z Er Ep= - T,p, m) 2 0, and Z 0 if and only if fe, p,m) 0 for all T and p. If Z = O, then is inactive everywhere and Fi Fi(R..) = 0 for any R thus SRP m) = and we can simply let a² = 0. Therefore, assume that Z 0, then both and mY + Z are positive. Hence, Fi
P 2 (RTT) if and only if (e, To. po, m)/m 2 Fi (R."). Equality is achieved when and only when all equalities hold.
The above result indicates that, in order to increase F1 (RRT", for any segment satisfying 1 (T, pT 0 and f(e, T, m) 2 (m/2) F1 (R ). we should change the routing decision to R (T: p) = and for any segment satisfying R (T, p)
and f(e. T, p, m) (m/2) Fi (R we should change the routing decision to RT (T: p) = O. Under the case where the number of possible inputs is finite (which is the case for most LLMs due their limited context windows), this will eventually result in
Rm that activates and only activates all segments with fe, T, m) 2 (m/2) Fi Rm whose Fi
cannot increase further. Such R
must be unique and maximizing Fi (RTT) Otherwise, there exists then the only segments where R and Rm disagree are Rug where R (T.
another Rm with Fi Rm
Fi
R
the ones satisfying (m/2)
Rm
[ee T.p, < (m/2) Fi
and Ru (T.p) = 0; however, changing R on these segments to 0 should not increase Fi Rm
thus Fi Rm
< Fi
Rm
a contradiction. Therefore, we can let Q..m
F
R
which
yields Equation D
D EXPERIMENT SETUP DETAILS
D. REAL MODEL ARCHITECTURE LIST
Table lists the detailed architecture and configuration of all REAL models where we conduct our experiments.
A few notes:
SwiichTrannsormers-BBBee 128 and NI.LB-MoF-54B are encoder-decoder models that use the
T5 architecture. Switch Transformers--Baii... Base-128 has 12 encoder layers and 12 decoder layers. NLLB-MOE-54B has 24 encoder layers and 24 decoder layers.
$IFY = Rs (T, p) 0 for all T and p. then Fi (R.") is undefined, which we do not concern.
Whee f(e, T.p, m) (m/2) Fi (R."), changing R. (T,p) does not affect F (R.)).
21

---

Published as a conference paper at ICLR 2026
Table 5: REAL Model architecture and configuration, sorted by model size. Experts: T: total; A:
active: S: shared (not included in total).
Params (B)
Experts
Model
# Layers
MoE Layer
Total
Active
T
A
S
PowerMoE--3B Shen et al. 2024c
3.30
0.88
32
all
40
8
LLaMA-MMO-w>133311
Zhu al.
2024
6.74
3.50
32
all
16
4
OLMoB--BB-BB-0125
6.92
1.28
all
64
8
(Muennighoff
2025
Swithhiranssoomerss Base-
7.42
0.22
24
every
128
1
(Fedus et al. 2022)
I. 3.88
Qu et al.
2024
8.03
3.80
32
all
8
2
JetMoF-8B (Shen al.
2024b)
8.52
2.33
24
all
8
2
OpenMoF-88 [uue et al.
2024a
11.86
3.80
every
32
2
MiniCPM-MoE--X225
Hu et al.
2024
13.87
4.32
40
all
8
2
QwenI.5-MoE--n -5-MoE-A2.7B
+ Qwen leam
2024
14.32
2.69
24
all
60
4
4
Veep-eeee
15.71
2.66
27
after 1st
64
6
(DeepScek-Al et al.
2024a
DeepSecKMoE (Dai et al
2024
16,38
2.83
28
after 1st
64
6
"VERSE-MoE-A4.21)
25.78
4.23
28
all
64
6
2
(XVERSE Technology Inc.
2024
Qwen3-30BBAA3B (Yang
2025
30.53
3.35
48
all
128
8
Yuan2.0-M32 (Wu ct al. 2024
39.94
3.70
all
2
Phi-3.5-MoE (Abdin ct all 2014
41.87
6.64
32
all
16
2
GRIN-MoE (Liu et al. 2024)
41.87
6.64
32
all
16
2
Mixtral-87BB-v..1 ( Oiang al. 2024
46.70
12.88
32
all
8
2
Lumba-Mini-L.. (Lenz
2025
51.57
12.11
every
2
NLLB-MoE-54PP (NLLB Team ct
2022)
54.50
3.75
every
128
2
Qwen2-57B-A14B Yang et al. 2024a
57.41
14.25
28
all
64
8
JetMoF-8B employs mixture-of-attentioo (hhen et al. 2024a). which we keep intact in our experiments.
GRIN-MoE shares the same architecture with Phi-3.5-MofE but is trained using different methods. Jamba-Mini- 1.6 employs hybrid SSM-Transformer structure, yet the MoE part is identical to vanilla transformer-bas based MoE models.
D.2 TOY MODEL CONFIGURATIONS
To validate the potential factors that affects local routing consistency, modify the configuration of OLMoE (Muennighoff et al., 2025) and create a series of toy MoE models which we pretrain from scratch. The baseline model, Baseline, has only layers and hidden dimension of 1,280, compared to the original 16 layers and a hidden dimension of 2,048. Other architectural hyperparameters, such the number of experts (activate 8 out of 64 experts) and the hidden dimension ratio between attention and expert (2: 1, so Baseline is 640 and the original OLMoE is 1,024), are left intact. We sample 20B tokens from OLMoE's pretraining data, and pretrained the model on for 10,000 steps; other training configurations such as sequence length (4,096), global batch size (1,024) and learning rate (cosine decay from 4e-4 5e-5) all follow the original pretraining stage settings.
Starting from the configuration of Baseline, we tweak one single setting once to create the following TOY models (including Baseline); all models have around 1.43B parameters, although they may activate different number of parameters:
FewerExp: Use 32 experts instead of 64, with doubled expert hidden dimension (1,280) and halved
activated experts (4).
ActMore: Activate 16 experts instead 8, under the same total number experts;
ActFewer: Activate 2 experts instead of 8, under the same total number of experts;
1ShrExp: Replace an expert with a shared expert, so the router selects out of 63 experts;
2ShrExp: Replace 2 experts with shared experts, so the router selects out of 62 experts;
DenseFst: Replace the first layer with a dense MLP layer, whose hidden dimension is the sum of all experts' hidden dimensions (40,960);
22

---

Published as a conference paper at ICLR 2026
DenseHlf: Replace the 1st, 3rd, 5th and 7th layers with dense MLP layers of DenseFst;
NoLB: Adjust the load balance auxiliary loss coefficient from 0.01 to 0 (no regularization);
OverLB: Adjust the load balance auxiliary loss coefficient from 0.01 to 0.1 (over regularization).
D.3 DATA PROCESSING AND INPUT GENERATION
We first extract samples from RedPajama and downstream application datasets in plain text format. For RedPajama, this is already done. For LMAren each of the original instances consists of Two human-LLM conversations and a preference vote. We keep the instances where of the conversations is preferred and concatenate all rounds from the preferred conversation (each round with its role and content) into one document. For OpenMath, OpenCode, and OpenScience, simply concatenate the input and output of each instance.
After collecting samples from each RedPajama category and the downstream application dataset, concatenate the samples within the domain, cutting them into input sequences of 512 tokens (the context window size of SwitchTraie Transforrrss, We sample 2,048 input sequences for each domain, resulting in 22,528 input samples in total.
since the model is trained for masked language modeling, we randomly select 64 tokens from each input sequence, masking them in the original sequence as the encoder input and constructing the corresponding label sequence as the decoder input. For NLLB-MoE, as model is trained for machine translation, use the same sequence (with the English language token prepended) as both the encoder input and the decoder input. All other models do not need further data preprocessing, they are decoder-only and trained for next token prediction.
For
E ADDITIONAL RESULTS
E. BASE POST-TRAINED
We selected three Loaells-LLMM-MMoo- OLMoE, and that have both base and post- trained versions released, and calculated SRP and و of each version. 6 lists the results, from which
that the differences of both SRP and ý between models before and after post- training are not significant enough to change the degree of local routing consistency, regardless of what type of post-training (SFT, DPO, etc.) is applied. Another related fact is that Phi-MoE-3.5 and GRIN-MoE, which share the same model architecture but are trained differently, have similar local routing consistency. Both indicate that the training method may be less important than the model architecture concerning local routing consistency.
Table 6: SRP between models before and after post-training.
Model
m = 4
m
16
m
64
m = 256
SRP
SRP
SRP
SRP
LLaMA-MoE-v
55.78
1.03
45.29
2.39
41.61
2.92
40.62
3.52
+SFT
+0.01
-0.00
-0.01
-0.00
-0.01
+0.00
-0.00
-0.00
OLMoE
64.69
1.00
50.91
1.06
45.53
1.21
42.64
1.19
+SFT
+0.40
+0.00
+0.47
+0.01
+0.50
+0.02
+0.60
-0.02
+DPO
+0.37
+0.00
+0.43
+0.01
+0.47
+0.02
+0.57
-0.02
+Instruct
+0.45
+0.00
+0.56
+0.01
+0.62
+0.02
+0.74
-0.02
JetMoE
60.22
1.09
47.45
2.26
42.78
2.69
41.09
3.15
+SFT -0.20
-0.00
-0.14
+0.01
-0.12
+0.02
-0.09
+0.03
+Chat
-0.20
-0.00
-0.15
+0.01
-0.13
+0.02
-0.10
+0.03
E.2 SRP PER SEGMENT POSITION
To determine whether the segment position can affect the segment routing best performance, we calculate SRP each segment position by summarizing statistics of all segments that share the same
23

---

Published as a conference paper at ICLR 2026
position. Figure illustrates this position-wise SRP each possible segment position. Most models have nearly constant SRP at every position except = O, where many models activate specialized experts to handle the beginming of the imput sequence. This stability of local routing consistency across input positions allows us to use segments from all positions to calculate SRP, and apply conclusions based SRP any segment of the input (except the very first one).
Ewenz 100
KELL
Owini.s
OBarMo
SaBoh
Fostion
on
Rration
200y o0)
196)
Figure 9: Position-wise SRP of each model on the full corpus. For encoder-decoder models, dotted lines show the encoder SRP and solid lines show the decoder ones.
E.3 SRP ACROSS DOMAINS
To verify whether local routing consistency is transitive across different domains, we calculate the correlation of expert segment routing best performance between pair-wise domains and demonstrate it in Figure We also compute correlation of expert activation frequency between pair-wise domains, results illustrated in Figure 11 By comparing corresponding heapmaps, we can see that local routing consistency nearly always positively correlated, even between distant domains on which the experts' activation frequencies consistency is transitive; domain- menain--peciilized experts with high local routing consistency in one domain tend to exhibit it in any other domain. We also found that some models (e.g.. LLaMA-MoE- v2 and Qwen2) do not show a significant difference between domains, which is aligned with the results in Section 42
negatively correlated. This means that local routing
ULINA McE-22
Ya2n2.0
Powe MoE
Owinn
Phi-3.5 MaD
OLMUE
GRIN Roz
Mintral 5x78
Min CPVS-MoE
Owent S-Mer
Suttchranntommee
VA 2 270
215E 13 205 053 100
Figure 10: Correlation between domain-wise expert SRP of each model. C4: C4; CC: Common- Crawl: BK: Books; WK: Wikipedia; AX: ArXiv; SE: StackExchange; GH: GitHub; LM: LMArena; OM: OpenMath; OC: OpenCode; OS: OpenScience.
E.4 SRP VS. SCH
To clarify the relation between SRP and SCH, Table 7 lists the correlation between them across all models. The two metrics are always highly positively correlated regardless of the values of m and p. This ensures that SCH shares the same property of SRP under reasonable segment length and cache size. Furthermore, when is around 1.5, the two metrics perfectly linear, aligned with Figure where most models have f E [1,3] when m 2 16, as well as previous claim that p 2 balances cache effectiveness and efficiency.
most closely related, nearly
24

---

Published as a conference paper at ICLR 2026
Mistral
JotMeE
ULINA MAEVV VA
XVERSE AGE
Jamba Min OcepSool SoooSpoRMMe
Owen2
NLLS
Owonl MEE
Oponiice Switch Trasfommss
Figure 11: Correlation between the domain-wis wise expert activation frequency of each model. C4: C4; CC: CommonCrawl; BK: Books; WK: Wikipedia; AX: ArXiv; SE: StackExchange; GH: GitHub; LM: LMArena; OM: OpenMath; OC: OpenCode; OS: OpenScience.
Table 7:
Correlation between SRP and SCH across all models. Bold font indicates the highest correlation across p for each m.
m
p
0.5
P
1.0
p
1.5
p
2.0
p = 2.5
p = 3.0
67.28
94.17
97.46
91.91
87.47
81.49
16
81.16
97.09
97.37
94.87
91.02
84.87
64
86.77
97.76
98.03
96.15
92.22
87.31
256
88.28
97.35
97.89
96.24
92.68
87.69
E.5 STATISTICAL SIGNIFICANCE
Due to the very high correlation between SRP and SCH, we choose SCH to represent local routing consistency, and report the 95% confidence intervals when 2 in Table The confidence intervals are obtained by bootstrapping 1,000 times with samples from the full corpus.
Table 95% confidence interval of SCH (p= 2) of REAL models. The decoder of SwitchTrans-
former does not have valid data when 256.
Model
4
16
64
256
LLaMA-MoE-v2
(96.88, 97.17)
(97.82,98.03)
(97.28, 97.58)
(96.64, 97.01)
Yuan2.0
(77.45, 78.17)
81.333
(78.69, 79.44)
(76.98, 77.83)
PowerMoB
(79.77,80.30)
(80.62,81.24)
75.77
(72.47, 73.44)
Qwen3
(22.86,7.778
776.49,77.46)
08..44
(62.91, 64.55)
Phi-3.5-MoE
74.03)
77.957,75.91)
69..37,69.13)
(63.65, 65.71)
OLMoE
71.27,72.46
(73.39, 74.71)
(65.74.67.42
(61.94, 63.87)
GRIN-MoE
(7..02,72.24)
(72.75, 74.09)
(65.35, 67.10)
(61.53, 63.57)
Mixtral-8x7B
(76.67,77.M)) 77.03)
(73.92, 74.42)
(66.59, 67 20)
(63166. 63.86)
MiniCPM-MoE
76.44,
(73.29, 73.67)
(65.57, 65.96)
(61.99, 62.44)
JetMoE
74.37,74.60
(70.60,
(63.60, 64.04)
(60.30, 60.80)
LLaMA-MoE-vl
70.50,70.80
(66.07,66.44)
(60.64,
(58.16,58.57)
XVERSE-MoE
56.04)
(68.38,59.14
(47.53,48.40)
42.72,43.64)
Jamba-Mini-l.6
58.00)
58..51,58.24)
(47.49,48.37
(42.82, 43.84)
DeepSeek- V2-Lite
-22Lite
(55.16, 56.00)
58.65,58.44)
U7.00447.96
(41.98, 42.99)
DeepSeckMoE
(53.77,54.63
(56.24,57.13
(45.41,46.52)
(40.25, 41.28)
Qwen2
55.099
55.088
(46.52, 46.74)
(42.91, 43.14)
NLLB-MoB (en)
(28.36, 29.55)
38.92,36.96)
(28.94,30.06)
(24.62, 25.79)
(de)
36.39,36.887
43.24,43.54)
(37.31,38661)
(32.51, 33.83)
Qwen Iiven.5-Mool 15-MoE
41.47,
45.79,46.500
(34.73,35.50)
(29.45, 30.32)
OpenMoE
33.521,66555)
48..42,40.68)
(32.35.33.80)
(29.02, 30.56)
SwitchTF (en)
(12.39, 13.58)
(20.74.21.86)
(16.92, 18.06)
(14.50, 15.66)
(de)
(16.43, 17.81)
03.89,25.116
(21.39, 22.61)
(21.39, 22.61)
25

---

Published as a conference paper at ICLR 2026
E.6 LAYER LEVEL RESULTS
Figure 12 illustrates each model's layer-wise SRP. Most models have peak SRPs among middle layers, while some (e.g,, Yuan2.0 and MiniCPM) have another peak at the last layer. We conjecture that middle layers are less tied to input/output tokens and thus more sensitive to the general topic. and the final layers process highly abstract information that is also more related to the overall topic. Both encourage routers to select similar experts within local segment that share the same topic across tokens. PowerMoE and Qwen2 have another peak on layer 2 due to expert imbalance. Appendix E.7 gives clear view on this.
LL Purevl
Xverlb Moe
KLLLSAM
24 Owerd Sale
Opervite
12 Layer
Layar
Layer
550 A1m 14]
2000
68)
Layar
J0x
60
J0H 150
Lager
Layer
Figure 12: Laser-wise SRP on the full corpus of each REAL model. Solid lines show SRP while dotted lines show corresponding p.
We also calculated layer-wise SCH, results demonstrated in Figure SRP, indicating a high correlation between the two metrics.
The patterns are the same as
"Lase
Mri
deeptellor..ar
Deep
opritet
0010
Sttlle
Stat 336,
(010)
205
p
Figure 13: Layer-wise SCH on the full corpus of each REAL model. Solid lines show SCH when 1 while dotted lines show SCH when p 2.
E.7 EXPERT LEVEL RESULTT
We demonstrate expert-wise segment routing best performance against activation frequency in Figure 14 LLaMA-MoE-v2, Yuan2.0, and PowerMoE have experts with very high activation frequency. These experts naturally have very high local routing consistency and contribute to these models' high model-level local routing consistency. The imbalanced experts of PowerMoE mainly belong to layer 2, which also explains the observation in Section E.6
Furthermore, Figures 15 16 17 and 18 compares SRP with domain and vocabulary specialzations. The plots are aligned with the conclusion of Section 4.2 that when the model exhibits domain specialization, domain -peeian-peeeializedd experts contribute more to overall local routing consistency than vocabulary-specialinn experts.
F FURTHER DISCUSSIONS ON THROUGHPUT
The purpose of SRP and SCH is to provide a general metric that is agnostic to concrete implemen- tations, where SRP solely depends on the model and SCH also considers hard limit cached experts. On the other hand, the performance of a true expert offloading system, usually measured through throughput, is not only decided by the deployed model, but also the implementation of the
26

---

Published as a conference paper at ICLR 2026
92
AACE SHONG
ASC. FEYA
Figure 14: Per-expert activate frequency vs, SRP. The x-axis stretched to show experts with very low or high activation frequency. Gray dashed lines indicate the theoretical lower bound of SRP at different activation frequencies. Green dashed lines show the expected activation frequency of experts from each model.
Preeetie
Ower13
DuMett
Hiatral EXIT
HENCCILL
VYAUE
580
Dumman Spet.
Comwin Spec
Doma Spec
Domann Spec.
Comenn Speci
Domai Spec
Domaih Soor.
Doman Spec.
Doman Sonr.
Commin
Figure 15: Per-expert domain specialization vs. SRP.
Yaand.3
Poweetitt
Owerd
DUPet
GLI-PPtt
De
DeepSesiiis
Gwin
Oper Mai
Spec.
Spec.
Spec.
Spec
Spec.
Figure 16: Per-expert input vocabulary specialization vs. SRP. Encoder-decoder models are not involved due to different input formats from other decoder-only models.
Yaand.3
Powerist
Onand
DUPit
GLINPPEE
340
Vic
NSC. Spac
spec
F-O. Voc Spec
8.0 Wec Spec
Figure 17: Per-expert predicted output vocabulary specialization vs. SRP. Encoder-decoder models are not involved due different input formats from other decoder-only models.
27

---

Published as a conference paper at ICLR 2026
Excese
jantes siea
Deoplcee Like
Deny seniina
Owiez
Operita
65
05
0.
Per-expert ground-truth output vocabulary specialization vs. SRP. Encoder-decoder models are not involved due to different input formats from other decoder-only models.
Figure 18:
system, including but not limited cache management, overlap exploitation, hardware coordinate, etc. Nevertheless, local routing consistency still plays an important role on the model side. Below is an informal, theoretical analysis about how local routing consistency (we use SCH as an example) may affect the actual throughput.
For simplicity, we assume that there is only one GPU with limited GPU memory (insufficient for the whole model but enough for activated parameters and calculation), and group of CPU with sufficient flash memory. (This a common configuration on edge devices.) Consider an expert offloading system that offloads whole experts only. During the decoding stage (the more time-consuming stage), compared to full GPU inference, it may introduce the following overhead:
1. During the calculation of the last layer, the system (if capable) may predict what experts the upcoming layer (or the next forward run) will use, and prefetch these experts to GPU. The overhead of prefetching one expert can be relatively small because the prefetch process can overlap with the current calculation.
2. After the router decides what experts to use, if demanded expert is not the GPU, the system will need to either (1) load the expert to GPU on-the-fly, adding a communication overhead, or (2) run the expert CPU directly, adding a calculation overhead. Both overheads more significant than the prefetch overhead because no overlap can be utilized.
Based on the above analysis, during forward run in the decoding phase, an ideal expert offloading system will always prefetch the correct group of experts for the upcoming layer or the next forward run, so the only overhead occurs during prefetching. This overhead also consists of two parts: (1) predicting the prefetched experts, whose overhead can be treated as constant as the system is ideal; (2) loading the selected experts, whose overhead is proportional to the number of cache misses between forward runs. As long as the GPU memory can hold more experts than the activated ones, the system will have to decide what extra experts keep on GPU. When the expert activation sequence is known, the optimal eviction list is given by the Beladi algorithm; however, this algorithm relies on the precise time each expert will be activated in the future, which very difficult to predict in practice. To this end, SCH with a specific segment length can be used as approximation that considers the frequency of close-future expert activations, which is easier to predict. Therefore, SCH (more precisely 1 SCH) be seen an upper bound of the minimum number of cache misses, which is approximately proportional to the minimum overhead any expert caching system under the same single-expert prefetching overhead.
To further verify the relation between local routing consistency and the actual throughput or overhead of typical expert offloading systems, we implemented naive version that utilizes LRU cache and always load missed experts on-demand. We use it to deploy all TOY models and measure their throughput on the full corpus under different cache sizes. The benchmark results show that the relative overhead w.r.t. full GPU inference has different correlations to SCH at the two inference stages: positive during prefilling (r Rs 0.2), and negative during decoding (r -0.3). The relation holds under various cache size ratio p. Note that the directions of the correlations align with the relation between local routing consistency and local load balance: During prefilling, multiple consecutive tokens are processed in one run, where tokens belong to the same expert will be dispatched to that expert together, so the bottleneck is the largest number of tokens an expert needs to process, which is related to local load balance. During decoding, however, only one token is processed per run, making the activated expert distribution between consecutive runs more important, which is the concern of local routing consistency. Since decoding is almost always more time-consuming on single queries,
28

---

Published as a conference paper at ICLR 2026
conclude that local routing consistency is more important and local load balance may be sacrificed to some extent. Nevertheless, the correlation coefficient not far from O indicating that there are also other significant factors that affects the system throughput, the model (as well as local routing consistency) should not be the sole decider.
29


Reviews:
[{"review_id": "rshlfCmBan", "reviewer": "Reviewer_jCkR", "summary": "Mixture-of-expert (MoE) models typically have high parameter count and not all parameters are always present on the GPUs. Efficiency and throughput of the model will be higher if active experts are present in the GPUs and there is minimal computation on CPU / communication between CPU and GPU. This work proposes two metrics to analyze how correlated expert activations are within a segment of tokens (termed local routing consistency). The authors empirically analyze various MoE models and suggest architecture design choices that might impact the local consistency. They also propose a way to empirically select the ideal cache size for a given model.", "questions": "1. Questions are primarily based on the weaknesses above. Provide answers to weaknesses (1) and (2). If possible provide a function connecting the proposed metrics to throughput / inference time and empirical results for the same.\n2. Clearly explain why the dense modules might decrease local consistency (SRP).\n3. Is it useful to have the marker size representing model size in Figure 2? Similarly, can expert specialization vs SRP be used in the plot in Figure 4 instead of correlation? Why do LL2 and Y2 have a very low correlation while they have high SRP and relatively high expert domain specialization? Why do STe have high correlation but low domain specialization?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The authors tackle a specific and important problem in LLM deployment. The work proposes two metrics (SRP and SCH) to analyze the local routing consistency. The empirical analysis is quite thorough with 20 different MoE models and several datasets from both general training corpora and downstream tasks. The paper considers several reasons that could be responsible for the consistency - model architecture, expert specialization to specific domains or vocabulary subsets and load balancing. The findings are interesting - shared experts have lower local consistency (SRP), load-balanced models can have high SRP and cache size of $2\\times$ number of experts results in a high SCH for most models.", "weaknesses": "1. There is no explicit connection of proposed metrics to throughput. For instance, how would SRP/SCH affect throughput given cache size, communication time and LLM forward propagation time? There are no measurements of throughput in any of the experiments either. \n2. What advantage does SCH have compared to common cache algorithm hit rate? Can we not determine required cache size by analyzing LRU hit rate vs $\\rho$? \n3. In analysis in Section 3.3, the authors suggest that applying MoE on every layer and not sharing experts results in higher routing consistency. While there might be a correlation between the two, it does not imply causation. Particularly in the case of ‘MoE on every layer’, there are just 4 out of 20 models that apply at every ‘k’ layer instead and of those four, two of them have extremely high ratio of active to total experts (1:64 and 1:128). It is difficult to come to such conclusions without decoupling the factors and with such limited data. The conjecture on why these might cause inconsistency (L:334-336) is also unclear. \n4. Writing needs significant improvement: \\\ni. The proposed metrics in Section 2 require a lot of time to understand. Providing intuitive / textual explanations for the equations, removing repeated mentioning of variables (for e.g., saying expert $e \\in E$ at all locations instead of just saying expert) and moving unnecessary equations to appendix and replacing them with text in the main section would vastly improve the reader’s experience. \\\nii. The term $\\rho$ is overloaded - it is used in both SRP for segment routing size ratio and in SCH for segment cache size ratio - causing a lot of confusion. \\\niii. Some of the figures (e.g. Figures 2 and 4) are hard to parse. \n5. Since the proposed metric requires access to expert activation, it is not possible to measure them for black box models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qT8hoCfi21", "reviewer": "Reviewer_9czm", "summary": "This paper investigates a key property of Mixture-of-Experts (MoE) models, termed \"local routing consistency\", which determines a model's suitability for expert offloading systems. To quantify this property, the paper proposes two novel metrics: Segment Routing Best Performance (SRP) and Segment Cache Best Hit Rate (SCH). Through an extensive empirical study on 20 MoE large language models with diverse architectures, the authors obtain several findings, for example, models applying MoE on every layer and without shared experts exhibit the highest routing consistency.", "questions": "1. Could you design an experiment to more directly verify that dense components indeed \"interfere\" with sparse routing, thereby decreasing consistency?\n\n2. Is there an inherent trade-off between maximizing local routing consistency (for inference efficiency) and maintaining the model's predictive performance (e.g., accuracy)? Could designs that pursue high SRP potentially limit the model's expressive capacity?\n\n3. Does your analysis and conclusions apply equally to the prefill stage for long context inputs? \n\n4. How well do the findings in this paper guide model or system design? Specifically, is there any action item (with validated performance) for the MoE model/system developers to optimize their design?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper offers a novel analytical perspective on the important and practical problem of efficient MoE model deployment. The concept of \"local routing consistency\" is insightful. It provides a clear and quantifiable framework for evaluating and comparing the deployment potential of different MoE models in resource-constrained environments. \n\nThe experimental evaluation is thorough and comprehensive, standing out as a primary strength of this work. The authors analyze up to 20 representative MoE models, covering a wide range of parameter scales and architectural variants, which lends strong credibility and generalizability to their conclusions.", "weaknesses": "Although the analysis is insightful, its conclusions are primarily based on correlation rather than causation. For example, the study observes that architectures with \"MoE on every layer\" and \"no shared experts\" correlate with high routing consistency and conjectures that dense modules might \"interfere with or weaken routing signals\". Ablation studies, such as modifying these architectural features on the same backbone (even a small one is ok) and observing the change in consistency, would greatly strengthen the reliability of these claims and uncover the underlying mechanisms.\n\nThe work did not explore the potential trade-off between this routing consistency and the model's core performance (e.g., accuracy). It is plausible that forcing or guiding a model to produce highly consistent routing could limit its expressive power, making it harder to capture fine-grained, token-level semantic shifts and thus hurting its prediction accuracy. The paper lacks a discussion on this \"efficiency-accuracy\" trade-off, which makes it difficult for readers to assess whether the design choices made to pursue high SRP/SCH might come at an unacceptable performance cost.\n\nThe paper's definition of \"local routing consistency\" primarily focuses on the processing of contiguous text segments, which corresponds mainly to the autoregressive decoding phase. However, in practical applications, the prefilling phase for long prompts is also a significant performance bottleneck. The routing patterns during prefilling may differ from those during decoding, but the current analytical framework does not explicitly distinguish between or investigate consistency in these two modes. \n\nThe paper's core claim is to guide memory-efficient model design and deployment, yet this claim is not well demonstrated. The study focuses primarily on analyzing proxy metrics like cache hit rates but does not provide implementation of a real model or a real expert offloading system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7MXP1rNNXn", "reviewer": "Reviewer_3aLj", "summary": "MoE architectures have recently enabled scaling of LLMs to even larger sizes without introducing the same amount of computation. To reduce the memory footprint, expert offloading has become a major strategy for inference under memory-constrained environments. However, limited studies have examined the potential cache hit rate of different MoE LLMs in recent progress. The authors performed experiments across a wide range of MoE LLMs and datasets and identified that routing consistency can vary across models. Based on this observation, the authors proposed several metrics to estimate the capability of a particular MoE model to perform expert offloading.", "questions": "I would like to see some of the discussion regarding the actual performance of offloading that takes the architectural differences into the consideration.\n\nIt would also be interesting to see why particular training strategy or model architecture design leads to the split between the domain specialization / vocabulary specialization in the MoE models.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- The idea is intuitive and based on an interesting observation that the activation consistency could change due to MoE architectures. Early studies in this field were unable to cover the later architectural changes in MoEs, so the proposed study addresses an area that has not been well studied.\n\n- The selected models and datasets are comprehensive, covering both classic and recent variants of MoE LLMs (within hardware constraints). Based on these empirical results, the authors have provided some great observations.", "weaknesses": "- The abstract is poorly written. It does not clearly mention the different MoE architectures (shared experts, small/large experts) as the potential reasons for the local consistency discrepancy, which I believe is the most important contribution of this paper. \n\n- While the authors have shown that the different MoE models could have vastly different expert activation consistency, the authors did not perform an end-to-end empirical evaluation on the expert offloading performance with the proposed strategy. As the MoE model architectures themselves could be vastly different, whether a certain model suit expert offloading would not only depends on the cache hit rate.\n\n- I personally believe that it would be better to describe the concepts of SRP and SCH using visualizations. While the listed formulas are helpful, they do not effectively convey the core idea. For SCH in particular, since it describes the cache hit rate, the authors should consider including a figure to illustrate the concept, as is common in other papers that discuss caching.\n\n- Figure 2-6 are way too small. It's really difficult for me to identify the relevant colors even with a large screen. These figures give me the sense that they were copy pasted from some research reporting slides, instead of being curated for this manuscript.\nThe author should consider ways to improve the rendering of these figures, especially for figure 2 and 3, given that these figure consist of the most important observation of this paper.\n\n- Conclusion is badly written, as it does not make any suggestion on the MoE architecture as promised by the abstract.\n\n- Minor\n  - L150 -- single expert is a weird paragraph title here. \"Single expert case\" might be better as it's mostly related to SRP, instead of any particular expert. Similar comment could be made to line 180 and 256 (\"Dataset\").", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)