Expert Divergence Learning for MoE-based Language Models
Authors: Jiaang Li
Haibin Chen
Langming Liu
Yujin Yuan
Yadao Wang
Yizhen Zhang
Chengting Yu
Xin Tong
Weidong Zhang
Shilei Liu
Wenbo Su
Bo Zheng
Venue: ICLR 2026 Poster (2026)
Keywords: Mixture of Experts
Large Language Model
Pretraining
Primary area: foundation or frontier models, including LLMs

Abstract: The Mixture-of-Experts (MoE) architecture is a powerful technique for scaling language models, yet it often suffers from expert homogenization, where experts learn redundant functionalities, thereby limiting MoE's full potential. To address this, we introduce Expert Divergence Learning, a novel pre-training strategy that explicitly encourages functional specialization among experts. Our method incorporates a label-driven auxiliary loss that leverages domain labels inherent in pre-training corpora to maximize the Jensen-Shannon Divergence between the expert routing distributions of different data domains. This optimization objective guides the model to develop diverged routing policies for varied domains and closer routing policies for the same domain, which leads to emergent and organized expert specialization. We validate our approach by pre-training MoE models of up to 15 billion parameters from scratch. Experimental results demonstrate that models trained with Expert Divergence Learning not only achieve a lower language modeling loss but also exhibit significant performance improvements across a diverse range of downstream benchmarks. Further analysis confirms that our method effectively mitigates expert homogenization and brings greater functional specialization, all with negligible computational overhead during training.

Published as a conference paper at ICLR 2026
EXPERT DIVERGENCE LEARNING FOR MOE-BASED LANGUAGE MODELS
Jiaang Li, Haibin Chen, Langming Liu, Yujin Yuan, Yadao Wang, Yizhen Zhang, Chengting Yu, Xin Tong, Weidong Zhang, Shilei Liu, Wenbo Su, Bo Zheng
Alibaba Group
ABSTRACT
The Mixture-o--Experts (MoE) architecture is powerful technique for scaling language models, yet often suffers from expert homogenization, where experts learn redundant functionalities, thereby limiting MoE's full potential. To address this, we introduce Expert Divergence Learning, a novel pre-training strategy that explicitly encourages functional specialization among experts. Our method incor- porates a label-driven auxiliary loss that leverages domain labels inherent pre- Divergence between
training corpora to maximize the pairwise the expert routing distributions of different data domains. This optimization ob- jective allocates the total routing diversity develop diverged routing policies for varied domains, which leads to well-coordinated expert specialization. We val- idate our approach by pre-training MoE models of up 15 billion parameters from scratch. Experimental results demonstrate that models trained with Expert Divergence Learning not only achieve a lower language modeling loss but also ex- hibit significant performance improvements across diverse range of downstream benchmarks. Further analysis confirms that our method effectively mitigates ex- pert homogenization and brings greater functional specialization, all with negligi- ble computational overhead during training.
INTRODUCTION
The Mixture-of -oxture-of-Experts (MoF) architecture (Shazeer et al. 2017 Fedus et al. 2022) has become the de facto standard for scaling Large Language Models (LLMs) (Wei et 20.2
al.) 2023
Clark et al.] 2022). ). By sparsely activating a fraction of its feed-forward network (FFN) parameters for each input, an MoE model can match the performance of much larger dense counterpart at lower computational cost (Artetxe et al.) 2022 Jiang et al. 2024). This paradigm of high capability with reduced FLOPs has cemented MoE's role many recent state-of-the models (Comanicil
ct al.
2025
Team
2025b
DeepSeek-Al Al 2024 OpenAi 2025).
However, the full potential of MoE. is hindered by its standard training paradigm. MoE is naturally designed to divide and conquer, allowing experts to develop specialized representations and form di- verse combinations fit heterogeneous real-world data distributions (Kudugunta al. 2021| Wang et al. 2024a Qiu et al. (025).. Yet, the current training approach provides no explicit objective to guide this crucial specialization. The sole constraint is typically
bala-cinniing loss (Lepikhin et al.)), which promotes uniform expert usage to ensure routing diversity but lacks mechanism to guide what each expert should learn. Consequently, experts are often trained on largely overlap- ping data distributions [ leading to expert homogenization phenomenon where they develop redundant functionalities and specialize insufficiently (DeepSeck-A] 2024 Baidu-ERNIE Team 2025). This redundancy causes the intended ensemble of specialists to collapse into group of similar generalists, which is fundamentally at odds with the diverse, multi-doman nature of real- world data that demands expert diverging. This insufficient specialization ultimately diminishes the MoE's effective capacity (Chi et al. 2022 Krishnamurthy et al.) 2023 Qiu et al. 2025).
To address this limitation, argue that effective specialization should not be an emergent prop- erty left to chance, but rather a goal that is explicitly guided by an external signal. Building on
Correspondence Jiaang Li <forgol denbi te@gma com>

---

Published as a conference paper at ICLR 2026
Layar
LalanBai
Lag miln dateball
Expert Activation Heatmap
N
(b)
Top-K Gating
Clofuf-erevrge Expert
Sievkar Eaperto
Expert Distributions
Experts
Diffcam Demure
Expert-Divergance
Softmax
Tep d,
Expert Activation Heatrop
Routing Network
4
Global Batch with Domain Divisions
Dowalls cverage Experi
Deerged Experts Bifferer Domuirt
Exports
(a)
Figure I: A conceptual comparison of the standard Load-Balancing Loss and our proposed Expert- Divergence Loss. (a) For a batch of tokens partitioned by domain, an MoE layer generates expert routing distributions. (b) The standard Load-Balancins Loss (CLB) promotes uniformity by oper- ating on the global average of expert distributions. This can lead to homogenization, as experts are trained on indistinct data mixtures. (c) contrast, our Expert-Divergence Loss (Led) provides a targeted signal by maximizing the divergence between domain-specific average distributions. This guides distinct experts to train on differentiated data subsets and develop functional specialization.
this insight, we introduce Expert Divergence Learning, novel training strategy that incorpo- rates a label-driven auxiliary loss to enforce divergent routing policies across domains. Our method leverages the domain labels, such as source or topie, inherently available in large-scale pre-training corpora to maximize the pairwise Jensen-Shannon Divergence (Lin 1991) between the average ex- pert routing distributions of different data domains. By maximizing this divergence, our auxiliary loss provides targeted gradient that complements the load-balancing constraint. While load bal- ancing encourages overall routing diversity, our objective channels this diversity to manifest as clear distinctions between domains, thereby promoting functional specialization and cultivating more effective and well-coordinated ensemble of specialists.
To validate approach, pre-train MoE language models of up to 15 billion parameters from scratch. Results demonstrate that Expert Divergence Learning yields substantial benefits, with a lower language modeling loss and stronger downstream performance across multiple domains. Fur- ther analysis indicates enhanced expert specialization, revealing that experts develop distinct func- tionalities. This work proves the importance of explicitly guiding expert roles during pre-training to unlock more capable MoE-based L.Mss.
2 RELATED WORKS
MoE-based Language Models. Sparsely-gated MoE (Shazeer et al., (007)) has emerged as a key strategy for efficiently scaling Transformer models, demonstrating the power of conditional com- putation (Lepikhin et al. Fedus et al. 2022). This paradigm now underpins many state-of-the-an
LLMMs OpenAl (2025); DeepSeek-Al (2024): Team (2025b a); Comaniei al. (2025). A central component of MoE training is the load-balancing mechanism. The standard approach involves an auxiliary loss (Cai et al. 2025). Alternative strategies include expert-choice routing (Zhou et al. 2022). where each expert selects tokens based on its capacity. and the use of router bias term to achieve balance without an explicit loss Wang et al.] (2024a). These methods focus on ensuring uniform expert utilization, lacking explicit guidance of what each expert learns.
Expert Specialization. While MoE models naturally tend towards expert specialization, a lack of explicit guidance can lead to redundant and overlapping functionalities (Chaudhari et al.) 2025 Chen et al. 2022). Some attribute this phenomenon to representational collapse, where tokens clus- ter around expert centroids (Chaudhari et al., 2025: Chen et al. 2022). To counteract this expert homogenization issue, prior work has focused on architectural modifications. prevailing strategy involyes fixed FNN into MoE, referred to shared experts, with the goal of capturing commonali-
2

---

Published as a conference paper at ICLR 2026
ties and alleviating parameter redundancy among other routed experts (DeepSeek-Al Al 2024). Besides, Zhong et al. calculate expert merge scores for each sequence and merging all experts into single expert before computing the corresponding sequence. Although this shows better im- provements and specialization, its complex mechanism has not been verified by large-scale training. These structural changes often retain the standard MoE training objectives, and fostering organized collaboration and specialization among these experts is an ongoing challenge.
2024 Team
More recently, attention has shifted towards modifying the training objective. Qiu et al. (2025) has shown that computing the load-balancing loss in the global batch rather than micro-batch can enhance specialization. A concurrent study, ERNIE 4.5 unsupervised objective that encourages orthogonality in the router's weight matrix, without con- sidering the domain prior of data structure. Our work fills this gap by introducing a supervised, label-driven loss that explicitly uses domain information to guide specialization.
EENIEENNIE-ee Team, 2025), proposes an
3 METHODOLOGY
In this section, we first review the standard Mixture-of-Experts (MoE) architecture and its training objectives. We then introduce proposed strategy, Expert Divergence Learning, and detail the auxiliary loss designed to foster expert specialization.
3.1 BACKGROUND: MOE STRUCTURE AND STANDARD TRAINING OBJECTIVES
A standard MoE layer replaces the feed- forward network (FFN) in a Transformer block with a set of N parallel expert networks (Fi...., Ex } and a routing network. As shown Figure I (a), for each input token the router computes logits h(x) W,x and applies a softmax to produce probability distribution p(x) over the experts. Top-K gating mechanism then selects the set of experts T = TopK(p(x), K) with the highest probabilities. The final output y is weighted sum of the selected experts' outputs:
Px(x) ΣjeT (x
Σ (x),, where (x iET
y
The standard pre-training objective, Catandard, combines the primary language modeling loss, LLM, with an auxiliary load-balancing loss, LLB. The language modeling loss is the average negative log-likellng log-likeldoodd for sequence of tokens (I1, .T,
1 CLM = - Σ log Po Pa (r)),
where 0 represents the model parameters and Pa (x 0) is the predicted probability of token 2y given the preceding context Fet.
The balaningg loss is designed to encourage uniform expert usage, as shown in Figure (b). is implemented as:
LLB N Σ P.
For batch B containing T tokens, Sa + ExEE liET, is the fraction of tokens routed expert where Tx is the set of selected experts for token x, and P. 1 ExeB P: is the average router probability for expert i. While load-balancing loss, LLB, encourages global diversity of expert selection across tokens in a batch, this objective is indiscriminate and does not guide how this diversity should be structured.
The final standard objective is weighted sum:
Lstandand LLM aLLB-
3

---

Published as a conference paper at ICLR 2026
3.2 EXPERT DIVERGENCE LEARNING
The standard training objective is agnostic to the semantic or topical domain of the input data. To address this, we introduce more targeted, fine-grained constraint: we explicitly encourage greater routing diversity between different domains. To explicitly guide experts toward specializing in dis- tinct domains, we introduce Expert Divergence Learning. This strategy incorporates a novel train- ing loss, the Expert Divergence Loss (CEd), which leverages the domain labels readily available in large-scale pre-training corpora. This process is demonstrated in Figure
Let training batch B consist of sequences, where each sequence (x1, Z2, ... FT) is associated with a domain label from set of unique domains D = (1.2..., M). Our method computes LED in three-step process for each MoE layer:
1. Token-to- -esequee-ee Aggregation: For each token Ie in sequence 8, the router outputs full probability distribution p(xx) over the N experts. We first determine the average expert distribution for the entire sequence, denoted as P., by averaging the distributions of all its constituent tokens:
T
P. T
Σ p(x)).
(5)
2. Sequence-to-Domain Aggregation: Next, we group these sequence-level distributions by their domain labels. For each domain j E 'D present in the batch, we define its subset of sequences as B, (s) d)) E B,d = f}. We then compute the average expert probability distribution for domain j. Pj. by averaging over all sequences within that group:
Pj =
Σ P. BBB
(6)
B
3. Pairwise Divergence Computation: The Expert Divergence Loss is designed to maximize the dissimilarity between these average domain-level distributions. We employ the Jensen- Shannon (JS) Divergence for this purpose due to its symmetry and bounded nature. For all unique pairs of domains {j, k} present in the batch, we compute their JS Divergence:
1
Dis(P,llmn) =
( (Px( DKL DKI(Pellm)
where m = (p, + PP) and (xI(- is the Kullback-Lebller (KL) Divergence.
Our final LED defined the average negative logarithm of the pairwise Divergence over all unique pairs of domains present in the batch, which encourages the routing patterns of different domains diverge from each other:
CED
E G.A)
-log(DDs(7,,1p)) +c),
(8)
MD
where Da is the set of domains represented in batch B, MB DBl. and is small constant (c.g., 10-8) for numerical stability. The negative logarithm in the loss function amplifies the gradient signal when divergence values small, preventing the vanishing gradient problem and leading to more robust optimization.
This auxiliary loss is integrated into the final training objective with hyperparameter 3:
Cunal LLM + aLLB + BLED-
This combined objective encourages the model to actively learn distinct routing patterns for data from different domains. As we will demonstrate theoretically in Section 3.3 this approach channels the model's 's overall routing diversity, which is induced by LLM and LLB, towards creating diver- gence between domains rather than within them. This reallocation diversity gives rise to expert specialization.
4

---

Published as a conference paper at ICLR 2026
3.3 THEORETICAL MOTIVATION: FINER-GRAINE GRAINED DIVERSITY ALLOCATION
The theoretical basis of our method lies in its ability to allocate routing diversity. While LL indiscriminately promotes the global diversity in expert selection across all tokens, LED provides finer-grained allocation by explicitly encouraging each domain using diverged experts, as shown in Figure I We formalize this concept by decomposing the total routing diversity into two parts.
First, we define the Total Divergence (Wotta) as measure of the global routing diversity across all tokens. calculated as the average KL Divergence between each token's routing distributions p(z;) and the average distribution for the entire batch, Pelobali
Duoal
E
1
DKL (pL(((())))) (ilobal) H (Pglobal)
Σ H(p(x)))
(10)
T all tokens
all tokens
This total divergence be decomposed into two parts. The first is the Inter-Domain Divergence (Dinter), which quantifies the routing diversity between different domains. It measures how much the average routing policy for each domain, Py- deviates from the global average:
())) Mn Ta DEL (Potbal)) H (ggloba) Σ
Mn T
Dinter Σ
(1I)
T
The second component the Intra-Domain Divergence (Dintra), which represents the routing di- versity within each single domain. It measures the average divergence between a token's routing path and the average distribution of its domain:
Me
Ma T =Σ
intra
Σ
T,
PKL(p(a))ll
=Σ H(T,) j=1
1
Σ H(p(x:)). (12)
j=1
2e j
all tokens
Full derivations are in Appendix A Eqs. 10 12 directly lead to the following key proposition.
Proposition 1 (Divergence Decomposition). The total routing divergence in batch is the sum of its inter-domain and intra-domain components: Dionat = Dinter + Dintra
This decomposition reveals the distinct roles of the loss functions. The standard load- balaneinnn loss LLB remains agnostic to how the overall diversity Diaal is allocated between its inter- and intra- domain components. contrast, our proposed expert divergence loss LED is explicitly designed to mitigate this issue. As shown in Appendix [B] LED directly targets and increases the inter-domain divergence Dineer by maximizing pairwise divergence between domains, which leads to our second proposition.
Proposition 2 (Synergistic Optimization). Optimizing the Expert Divergence Loss, CED. promotes an increase in the theoretical inter-domain divergence, Dmaer-
Therefore, the two losses operate in synergy. LED acts as a finer-grained guiding signal that directs the flow of global routing diversity, which promoted by the load- baad-balancing loss LLB- Specifically, LED channels a larger fraction of the total diversity (Dwaa) to allocated between domains to create meaningful distinctions (Diner, thereby leading to the emergence of specialized experts each domain.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Model and Implementation Details. We adopt the Qwen3-MoE (Team) 2025b) model architecture and the tokenizer. Due to computational constraints, we train scaled-down versions of the Qwen3- MoE 30B-A3B model, which has 30 billion non-embedding parameters and activates 3 billion per input. To thoroughly validate approach models with different scales, train models with three varying sizes: 15B-A1.5B, 8B-A0.8B, and 3B-A0.3B. The detailed architectural parameters for each presented in Table All experiments 64 GPUs with 80 GB of memory each.
5

---

Published as a conference paper at ICLR 2026
Table Architectural parameters of selected MoE models.
#layees
hidden-size
intermediate-size
#Experts
Top-K
15B-A1.5B
16
2048
1560
96
8
8B-A0.8B
16
2048
1344
60
4
3B-A0.3B
16
1536
640
64
All models are trained using the AdamW optimizer (Loshchilov & Hutter 2017) with $1=0.9. $2=0.95 and a weight decay 0.1. We employ constant learning rate of 5 x 10 with a 100-step linear warm-up. For data processing, we use global batch size of million tokens and pack all inputs maximum sequence length of 8192. For the loss components, the weight for the load- balancing loss a is fixed at 1 10-5 The expert-divergence loss weight /3 is set to 5 10-4, a value selected empirically based on training loss stability and performance.
Training Data. We pre-train our models from scratch on 100B text tokens. To ensure reproducibil- ity, the training corpus is entirely composed of open-source data. The corpus is mixture from three sources: 40% from the English content Nemotron-ce (Su et al. 2024). 40% from the Chinese content Fineweb-euu-chinesee Yuu al. 2025), and 20% from the mathematics content FineMath (Allal
et al. 2025).
To guide expert specialization, we unpack monolithic pre-training corpora by employing two distinet domain labeling schemes:
3-Class Diverging: This scheme consists of domains, where each sequence is directly labeled with its data source (i.e., English, Chinese, or Math).
49-Class Diverging: To create a more granular set of domains, we categorized our data into 49 distinct topics. We used separate classifiers to label the English and Chinese corpora with 24 topic domains each, following the methodology proposed by /Wtttig et al. (0025). Mathematics
treated single, separate domain, resulting in total of 24(EN) 24(ZH) 1(Mahh)
49 domain labels. A full description of each topic is available in Appendix and details on the domain classifiers are provided in Appendix D
Evaluation. We use OpenCompase] to evaluate our base models on a wide range of capabili- ties, including English language capability, Chinese language capability, and mathematical rea- soning capability. The evaluation is conducted using 7 benchmarks: Ceval (Huang et al., (023)) MMLU (Hendrycks al., (02T), CMMLU (Li et al., (2024). ARC-easy, ARC-chullenging (Clark et al. 2018), Race-midalle, and Race-high (Lai et al. 2017). The former three are evaluated with five-shot while the others are evaluated zero-shot.
4.2 MAIN RESULTS
The experimental results, presented in Table 2 confirm the effectiveness of our proposed Expert Divergence Learning strategy. Across all model scales, integrating the label-driven auxiliary loss, CED. consistently improves performance over the standard MoE training paradigm, validating our hypothesis that explicitly guiding expert specialization is beneficial. Even the coarse- coarse-grained, 3- class divergence scheme was sufficient to achieve benchmark gains over the baseline on the larger 8B and 15B model scales.
key observation is that the performance gains from Expert Divergence Learning scale positively with model size. This trend is most pronounced in our largest model, the 15B5A1.5B, which achieves a final average score of 36.65 under the 49-class scheme-a significant improvement over the base- line's 35.59. While smaller models also show improved average scores, the greater capacity of the 15B model allows it to more effectively translate the benefits of guided specialization into robust per- formance gains across the evaluation suite. This suggests that models become larger, they exhibit an emergent ability to convert the structured specialization into substantial performance gains.
Furthermore, analysis of the training dynamics reveals that our approach improves the primary train- ing objective regardless model size. As illustrated for the 3B-A0.3B model in Figure 2 all

---

Published as a conference paper at ICLR 2026
Table 2: Downstream task performance of baseline MoE models versus models trained with Expert Divergence Learning. The best result for each model size is bolded.
Model Configurations
CEVAL
MML.U
CMMLU
ARC.
ARC.
RACE
RACE,
Avg.
MoB + 3-class Div.
32.80
33.24
34.64
56.61
29.83
33.36
28.64
35.59
15B-A1.5B
32.53
32.98
35.16
59.08
33,56
32.94
28.10
36.34
+ 49-class Div.
33.45
33.21
35.17
57.85
33.56
34.54
28.76
36.65
MoE
32.88
32.54
34.22
53.62
31.53
32.52
28,42
35,10
8B-A088
3-class Div.
33.27
31.69
33.40
56.79
32.20
32.80
28.62
35.54
+ 49-class Div.
33.81
32.05
34.50
58.55
30.85
31,75
28.87
35.77
MoE
32.87
31.16
32.73
50.79
31.19
31.82
28.36
34.13
3B-A0.38
+ 3-class Div.
32.08
31.47
33.07
51.15
30.85
30.78
28.44
33.98
49-class Div.
33.75
31.41
32.91
54.50
31.53
32.03
28.27
34.91
configurations trained with Expert Divergence Learning consistently converge to a lower language modeling loss LLM than the standard MoF baseline. Altering /3 leads to loss variance, but still robustly better than the standard MoE training. This suggests that our auxiliary loss, LED, suc- cessfully steers the model toward a better optimization landscape for the primary task. The scaling trend observed in the downstream benchmarks suggests that greater model capacity crucial for converting this fundamental training advantage into superior performance.
Finally, our results suggest a link be- tween the granularity of domain super- vision and model performance. Across all three model sizes, the fine-grained 49-class scheme, based on semantic top- ies, consistently outperforms the coarse- grained 3-class scheme, an advantage supported by both superior downstream task performance and lower final train- ing loss. This consistent advantage leads us to hypothesize that providing more specific, semantically meaningful domain signals is key factor in culti- vating a more effective ensemble spe- cialized experts. This finding, in turn, points to a promising direction for future MoE optimizations: curating web-scale corpora with fine-grained topical labels appears be powerful strategy for un- locking the full potential of sparse models.
2.4
class cosfrie
Coargend cashCee
class Deerge beergenne
Scleve Diargamnen peeeeeeeeded
Soleus Divargmmnn ceeddddd
841
2:00
Trannig Fog Aialay
Figure 2: Comparative analysis of the primary language modeling loss LLM for the 3B-A0.3B models. This fig- ure contrasts the training performance of the baseline MoE with models trained using our Expert Divergence Loss under various configurations, including different do- main schemes (3-class and 49-class) and divergence co- efficients (3).
4.3 ANALYSIS ON EXPERT SPECIALIZATION
To investigate the mechanism by which Expert Divergence Learning influences expert specializa- tion, we first construct a dedicated validation set. We sample 1,000 instances from each of the three pre-training domains (English, Chinese, and Math), ensuring no overlap with the training data. This resulted in three distinct validation sets: en_val_1000, vvlvll1000. and math_val_1000 Using these sets, we analyze the internal routing behavior of our largest 15B-A1.55 model, comparing the baseline against the version trained with the expert divergence loss.
We conduct a routing perturbation analysis quantify the functional specialization of experts in each MoE layer by measuring the performance degradation after disrupting the learned routing pol- icy. For each layer in pre-trained MoE language models, we ablate its learned routing strategy by randomly permuting the row vectors of its router weight matrix, W, E g Let the matrix be composed of row vectors, W, = [Wi, W2...., wN]" where each row w; corresponds to the weights for expert i. We then create
matrix W/f by applying random permutation и to the indices of these row vectors: W/ = [w)())= -..):..... (x))) This operation shuffles the expert assignments for that layer, forcing tokens to be routed to experts randomly. We then measure the

---

Published as a conference paper at ICLR 2026
Validation Dotaset Chirese
Validation Dataset Engltsh
Perpodly
Avragg
Portubed Layer Index
Figure 3: Increase in perplexity (APPL) after randomly permuting router weights for each layer of the pre-trained 15B-A1.5B models. Higher values indicate greater expert specialization.
resulting increase in perplexity (APPI' PP shufffed fled Poriiigial) on our validation sets. A small APPL' indicates expert homogenization, where experts develop overlapping and redundant functionalities, thereby randomly swapping them would have minimal impact on performance. On the other hand, large increase signifies that experts have unique, non-interchangeable roles.
Layer
Bareine
3-ClbE DK:
40-Clt0n ON
Layer
Esselin
3-CINe Div.
49 Cp-ccoos
Layer 14
Besctre
Ciclo DIV.
69 Clas Di:.
Figure 4: Expert activation heatmaps of different domains for representative layers (0, 4, 14) of the 15B-A1.5B models. Each row shows the average expert activation probabilities for given
validation domain. Darker colors indicate more frequent activation.
The results of APPL across layers, shown in Figure 3 provide several key insights. First, for all models, random routing in any layer would degrade model performance with different extents,
8

---

Published as a conference paper at ICLR 2026
Table 3: Training throughput (k tokens/s/gpu) across different configurations.
15B-A1.5B
8B-A0.SB
3B-A0.3B
MoF + 3-class Div. 49--class Div. MoE + 3-class Div.
49-class Div. MoE + 3-class Div.
499-class Div.
TPs 5.07
5.03
4.96
7.97
7.94
7.91
9.99
9.88
9.85
Table 4: Inference throughput (tokens/s) of 15B-A1.5B models on different domains.
en0val_1000
1000
math
MoE + 3-cllass Div. + 49-claas Div. MoE
3-class Div. + 49-class Div. MoE
3-class Dix. + 49-ccaas Div.
TPs 14.45
14.36
14.39
19.18
19.08
19.27
19.19
19.13
19.14
indicating the existence of expert specialization. The impact varies significantly across layers, with prominent peaks at specific layers (c.g.. 0, 4, and 14). This suggests that expert specialization is concentrated in critical layers rather than being uniformly distributed, consistent with observations that different layers in L.L have different functions (Geva et al. 2020). This natural LLM behavior learned from massive data is not overridden, but rather enhanced by Expert divergence Learning, which steers the optimization process towards a better landscape. Second, models trained with both the 3-class and 49-class divergence mostly exhibit a greater PPL increase than the baseline, particularly at these specialization peaks. This provides direct evidence that method successfully reduces expert homogenization and makes the expert become more diverse. Moreover, the model trained with the fine-grained 49-class scheme produces dramatically higher APPL Layer 4, far surpassing all other configurations. This result strongly suggests that a more granular supervisory signal induces a higher degree functional, subsiiuusieeeeele specialization, compelling the model to develop highly specific roles for its experts.
To more detailed visualize the differences of the routing behaviors, we present expert activation heatmaps for representative layers in Figure 4 (the complete set of heatmaps is available in Ap- pendix[F) The average expert activation probabilities for each domain validation set are computed to generate these heatmaps. These resulting vectors are then normalized to sum to for visualiza- tion and plotted to reveal the expert utilization patterns. These heatmaps provide a clear qualitative confirmation of our findings from the perturbation experiment. For instance, in layers with a high APPL such as Layer 4, the model trained with 49-class divergence exhibits markedly more distinct domain-specific activation patterns, revealing less-overlapping set of preferred experts for each domain. In contrast, the baseline model's activations for different domains are more overlapping.
In summary, this analysis demonstrates that Expert Divergence Learning effectively amplifies the natural specialization tendencies within MoE models. By encouraging divergent routing policies, method guides the model to develop experts with stronger functional specificity in key layers, which in turn drives the performance improvements reported in Section 4.2
4.4 EFFICIENCY ANALYSIS
We evaluated the computational overhead of Expert Divergence Learning by measuring both training and inference throughput, targeting potential inefficiencies from the LED loss calculation and poten- tial latency from expert specialization during domain-specine inference. As shown in Tables 3 a and our method introduces negligible overhead in either setting. The training throughput remains com- parable to the baseline because the LED calculation is lightweight, operating on small-dimensional router outputs. Similarly, inference speed on our largest 15B-A1.55 1.5B model is maintained even on domain- specific data, indicating that expert specialization does not create significant expert imbal- ance that leads to latency. These results confirm that the performance and specialization benefits of our method are achieved without compromising computational efficiency.
5 CONCLUSION
In this paper, introduced Expert Divergence Learning, novel learning objective that mitigates expert homogenization in MoE models by maximizing routing divergence between domains. Our approach is grounded in the theoretical insight that total routing diversity can be decomposed, allow-

---

Published as a conference paper at ICLR 2026
ing method channel this diversity specifically toward creating inter-domain distinctions. Ex- periments demonstrate that this strategy yields a lower language modeling loss and superior down- stream performance, particularly for larger models, all with negligible computational cost. Further analysis confirms that these performance gains are driven by enhanced expert specialization, as the guided routing trains experts more differentiated data mixtures to develop distinct functionalities.
This work validates the importance of explicitly guiding expert roles during the pre-training of sparse models and demonstrates that leveraging the inherent domain structure of web-scale corpora is powerful and efficient strategy for advancing MoE capabilities. Future directions include scaling LLMs with more sophisticated properties and connections curated from massive data.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, et al. Smollm2: When smol goes big data-centric training of a small language model, 2025. URL ltttp:://axxii org/ abs/ 2502. 02737
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, et al. Efficient large scale language modeling with mixtures of experts. In Proceedings of the 2022 Conference on Empirical Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/w1/2022.cmm main. 804. URL httpt:tt///om http:::://aallaatholoooooo wrgp:hhooloooy.../2222222222 wwtp/////20022 emn 1p ma 804/
Methods in Natural Language Processing, pp. 1 11699 1132,,
Baadu-ERNNE-Teaae Ernie 4.5 technical report, 2025.
Weilin Cai, Juyong Jiang, Fan Wang. Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mix- ture of experts in large language models. IEEE Transactions on Knowledge & Data Engineering, (01):1-20, 2025.
Marmik Chaudhari, Idhant Gulati, Nishkal Hundia, Pranav Karra, and Shivam Raval. an expert is all you need. In Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference, 2025.
Moe lens
/ttp:////ppeeeeeeeeeee net forum?
i 1d=GS4wXnewSF
Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. Task-specific expert pruning for sparse mixture-of-experts arXiv preprint arXiv:2206.0033 00277, 2022.
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 35:34600-346133 2022.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference machine learning, pp. 4057- 4086. PMLR, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1rXiv:1803.003. 05457, 2018.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06200 2025.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. Pre-training with whole word leee. ort/documet/////////oommo
masking for chinese bert, 2021. URL
9599397
10

---

Published as a conference paper at ICLR 2026
DeepSeek-AL Deepseek-v3 technical report, 2024. BRLL:ttss///rrvv.orr//bb//// 19437
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter
models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1 2022.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXhv:2012.14913 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang. Yao Fu, et al. C-eval: multi-level multi-discipline chinese eval- uation suite for foundation models. Advances in Neural Information Processing Systems, 36: 62991-63010, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04000 004088, 2024.
Yamuna Krishnamurthy, Chris Watkins, and Thomas Gaertner. Improving expert specialization in mixture of experts. arXiv preprint urXiv:2302.14703 2023.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh- Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of-experts for efficient inference. arXiv preprint arXiv:2110.03742, 2021.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, 2017.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, In Findings of the Association for Computational Linguistics ACL 2024, pp. 11260-11288 11285, 2024.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:238..03281, 2023.
J. Lin. Divergence measures based the shannon entropy. IEEE Transactions Information Theory, -551 1991. doi: 10.1 105
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2017.
OpenAl gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.orr/eee 25C8. 10925
Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Demons in the detail: On implementing load balancing loss for training specialized mixture-of-exper models. In Proceedings of the 63rd Annual Meet-
ing of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 5005-5018 Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10. 10.18653/v 1/2025.22eaa acl lcl-long 249. URL https::// htttt::://acaathelloyyeo htw.eaannnnoooooooo///22222.
ac wow.on..2.00 249/

---

Published as a conference paper at ICLR 2026
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The harsely-gated mixture-of-expere layer. In International Conference Learning Representations, 2017.
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings. Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-ce: Transforming common crawl into a refined hong-horizon pretraining dataset. arXiv preprint arXhv:2412.02599, 2024.
Kimi Team. Kimi k2: Open agentic intelligence, 2025a.
URL https://arxiv.org/aaat.
2507. 20534
Qwen Team. Qwent.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024. URL httLhtttt//:/mennn wtt.g/u...o/bll////eeaaaaa...
Qwen Team. Qwen3 technical report,
Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-oos--fre load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408. /5664, 2024a.
Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-free load balancing strategy for mixture-of-expers 2024b. URL htt.tttts///rrd..orr//b///////// 15664
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0766 2022.
Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation. In Forty-second International Conference Machine Learning, 2025.
Yijiong Yu, Ziyun Dai, Zekun Wang. Wei Wang. Ran Chen, and Ji Pei. Openesg chinese corpus: A series of high-quality chinese datasets for Ilm training. arXiv preprint arXiv:2501.08197, 2025.
Zexuan Zhong, Mengzhou Xia. Danqi Chen, and Mike Lewis. Lory: Fully differentiable mixture- of-experts for autoregressive language model pre-training. In First Conference on Language Mod- eling.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. ofxture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114 2022.
12

---

Published as a conference paper at ICLR 2026
A DERIVATIONS FOR DIVERGENCE DECOMPOSITION
This appendix provides detailed, step-by-step derivations for the three key components of our di- vergence decomposition identity, Dinter + Dintr Diotal- We begin by deriving the entropy-based formula for the generalized nents.
Divergence, which is fundamental to all three compo-
A.I DERIVATION OF THE GENERALIZED JSD FORMULA
The generalized JSD for a set of distributions } with weights {Ty} is defined as the weighted average of the Kullback-Leiller (KL) divergence from each distribution to their collective mean, p Σx T&Pk- The derivation from its KL-bbased definition to the more intuitive entropy-based formula follows:
JSD ((ph)) = Σ Th xxDxL(PxlF))
(1. Fundamental Definition)
k
Σ
Σ pkue log Pk.
(2. Expand KL Divergence)
Pi
k
Σ
Σ Phis (log [l.s ph.s - log
(3. Use log property)
k
Σ
Σ Pk.i log Σ pai log
(4. Distribute the term pk.))
k
Σ TR
-H(Pk)) Σ Paii pa log
(5. Use Entropy definition)
k
Σ (pH))
-
Σ T Σ pk.t log Pr
(6. Distribute the outer weight TT))
k
k
Σ (pk)
-
Σ
Σ
TkPki
logp:
(7. Swap the order of summation)
k
Σ
Th (pw)) - Σ Di log Pr
(8. Substitute def. of p,)
k
Σ (pw) - (-H(p)
(9. Use Entropy definition again)
k
(p)
Σ (px)
(10. Final Form)
k
This final expression, H (mean distribution) mean forms the basis for the sub- sequent derivations.
A.2 APPLICATION TO DIOTAL DINTER, AND DINTRA
Using the general formula derived above, we can now specify it for the three components in our main argument. We assume the lengths of all inputs are equal. Then the sequence-level average distribution is equal to token-level average distribution, which is directly used in our derivations.
A.2.1 TOTAL DIVERGENCE (DrOTAL.
The total divergence is the generalized JSD applied to the set of all individual token-level distri- butions (p(x:)) in the batch, each with uniform weight of 1/T. Their collective mean is the global
13

---

Published as a conference paper at ICLR 2026
average distribution, Pelobal-
Dwaa JSD / (T(((()))
T
H (Pglobal
H(p(z)))
T
T
H (Pgtoba))
H(p(x,))
T
A.2.2 INTER-DOMAIN DIVERGENCE (DINteR)
The dote-ddomain divergence is the generalized JSD applied the of My domain mean distri- butions {p, J. where each distribution is weighted by its relative size in the batch, "j = Tj/T. The collective mean of these also Pglobal-
Dinter = JSDT, /T (rr,((1,,)))
Ma
T3 H(P))
(Pgloball
Σ
T
==1
A.2.3 INTRA-DOMAIN DIVERGENCE (DINTRa)
The intra-domain divergence is defined as the weighted average of the JSD within each domain. For a single domain j, the intra-domain JSD is computed over its Tj token distributions.
JSD ISD(domain j) H(p,) T E H(p(z.))
Edomain
Dintra the average of these values, weighted by my = Tj/T.
Mp T,
Dintra Σ
(JSD(domain, j))
(1. Definition)
T
1=1
MB
T
1
Σ 1=1
H(),)
Σ Hp((z))
(2. Substitute JSD formula)
T
MB
Tj
H(p)))
1 Σ
H(p(xt))
(3. Distribute the weight)
T
m.Ej
Tj H(p,)
1
Σ
H(p()))
(4. Split into summations)
T
T
j=
Mn T
1
Σ H(p(x))
(5. Combine inner summations)
all tokens t
With these three expanded forms, one can directly verify the decomposition identity: Dinter Dinra
D
D
B PROOF OF PROPOSITION 2 (SYNERGISTIC OPTIMIZATION)
This section formally proves that minimizing our proposed Expert Divergence I Loss (LED) synergis- tically promotes the increase the theoretical inter-domain divergence (Diner. We first provide an intuitive geometric explanation. Then, rigorously establish this by using second-order Taylor expansion. We show that the sum of pairwise divergences (Spuir Σjer DJs(P,, P).) and Dinke are locally proportional (Spair & Dmner). A direct proportionality guarantees a monotonic relation- ship, meaning they increase or decrease together. In practice, observe that the stronger optimiza- tion signal on individual pairs LED brings more training gains while directly using Dinter falls behind.
14

---

Published as a conference paper at ICLR 2026
B.1 INTUITIVE EXPLANATION.
We begin with an intuitive explanation. we view each domain's expert distribution as point in high-dimensional space, the theoretical objective (Dimer) measures the spread of these points from their collective center of mass. Our practical objective, which maximizes the sum of all pairwise distances (Spai),, imposes stricter condition: pushes every point away from every other point. Maximizing all pairwise distances inherently increases the distance from the center, thus providing strong optimization pressure towards increasing D
B.2 PROOF STRATEGY.
The goal is prove that increasing Spair promotes increase in Dinter- We will establish this by showing that the two quantities are approximately proportional the local region (Spair O D Damerr- We will achieve this by showing that:
1. The second-order Taylor expansions of both measures are directly proportional.
2. The perturbation term in this expansion is small enough in our practical setting to render higher- order remainder terms negligible.
This establishes monotonic relationship, confirming that maximizing the sum of pairwise diver- gences (the objective of CED) serves as an effective proxy for maximizing the overall inter-domain divergence.
B.3 DERIVATION.
For notational simplicity, we assume the weights of all MD domains within the batch are equal, i.e., T = 1/MB- Let the shift of each domain distribution from the mean be 85 = P, Patobal which are centered such that E, = O.
B.3.1 1. APPROXIMATING GLOBAL DISPERSION (DINTER):
The global dispersion is defined as Dinter = H (Pglobul) first expand the average entropy term Ma E, H (D,)):
Me H(py). To approximate this, we
1 Σ H(),)) MB 3
MB Σ H (D Pagaaall + 65)
H H + VH 2
MB
H (T globa
THT
Σ d3
+
Σ
62
M
2M
Since the mean shift E5 = 0, the linear term vanishes. Substituting this back into the definition of D Dimer:
1 EoN
Dinter
H (Pglabaa)=
H (Pglotaat) +
2M
Σ 2 Ho₂)
2M
15

---

Published as a conference paper at ICLR 2026
B.3.2 2. APPROXIMATING PAIRWISE DISPERSION (SPARR:
The sum of pairwise divergences is Spair Ejen DJs(P,,PP)) where DJs(D
H
Pk 2
3 H)()) - (Ta). We expand each of the three entropy terms around Petohal
Pj + PP
89 + 6k
(a) H
H Petohal +
2
2
1
H (Pelobal)
VH (6, + dz) +
8 g + (2)+6,) (v))" + 6x)
2
1
(b)
H(),))
H (PPltoha)
VH 89 + 0 v2 H6
2
2
(c) 2
H(p)) ~
- H(Pgloha)
V 5x + 0/ 42 H6:
Summing these three parts, the constant and linear terms cancel out, leaving only the quadratic (Hessian) terms:
1 (6, 5t) ++
(H 5k) 1 16 o2 Hor)
Dis( 8
(5, 5n H - -6))
Summing over all pairs, can get Spair. Since E, 6, O, we have the identity Ejcn(O; 5².²² A(5, 6x) Mn E, 6z A6, for a symmetric matrix Applying this identity, we have:
Spair ~
(6,
- v² - 6x)
8
jek
MB
Σ
8
8
B.3.3 CONCLUSION.
By comparing the two approximations, we find direct proportionality between their dominant second-order terms:
M
Spuir Ri
D Dinter
B.4 JUSTIFICATION OF THE PROPORTIONAL APPROXIMATION
The above derivation shows that the principal components of the Taylor expansions for Spair and Dinter are positively proportional. We further show that the higher-order remainder terms of the expansion are relatively negligible and don't affect this proportionality.
In our MoE training context, the perturbation vectors, dj. are indeed kept small. This is conse- quence of two key factors: the architectural design, which distributes probability mass across large number of experts (N > 60), and the load- bad-balanncin regularization (LLB), which explic- itly penalizes routing concentration. Together, these ensure that the components of the perturbation vector, 16 significantly less than 1.
Because the perturbations are small, the higher-order terms of the Taylor series (e.g., O(8?)) dimin- ish at a super-linear rate and have negligible impact relative to the second-order term. Therefore, we can confidently approximate the relationship between the two functions by the proportionality of their dominant second-order terms. This establishes local monotonic relationship. confirming that optimization step increasing Spair will also increase D inter- This validates the theoretical foundation our Expert Divergence Learning method as claimed in Proposition 2.
C TRAINING PROCEDURE OF EXPERT DIVERGENCE LEARNING
The pseudo code of one training step can be found in Algorithm 1.
16

---

Published as a conference paper at ICLR 2026
Algorithm Training Step with Expert Divergence Learning
Require: Model parameters 0, hyperparameters a, .3.
Sample a batch of sequence-domain pairs B {sk,, d d
2: Unpack all sequences into flat list of tokens {
1. Forward Pass
3: Logits, (p(x))) + Modelo
Get LM logits and router probabilities
4: LIM < CrossEEtroppy
5: LL f- Lottssyoosooysss/ (ogiis)
Loaialannng Loss( (p(x;)))=
2. Compute Expert Divergence Loss
6: //Aggregaee token-level probabilities to sequence-level
7: For each sequence 8& E B: Pa, < Ty E p(xt)
8: //ggregate sequence-level probabilities to domain-level
9: Let Di be the set of unique domains in B.
10: For each domain j E DD: Pd, + [8, Ences, P...
11: /Nectorized computation of pairwise divergence
12: Let PD E H. Pal X be the matrix of stacked domain distributions [Pa, Pdl.....
JSD Matrix -log( VectorizeedParrise////PPPP
Results in [Dgl x (Dal matrix
14: LED &
Swapppppperrrrlleeeee 3. Update Parameters
15: Linal &- LIM + LLL + -BLED
VoCiinl -- backward (Lana))
17: 0 t OptimizerStep(( 0, V Vaoal)))
D DOMAIN CLASSIFIER DETAILS
To create granular of domains for experiments, we employed two separate classifiers to categorize the English and Chinese content of our pre-training corpus into distinct topics.
English Classifier. We directly utilize the classifier developed by Wettig al. 2025 for English content. This model is initialized from gt gte-base- srase-en-vl.. (Li et al. 2023 and fine-tuned in
two stages: first with one million annotations generated by Llama-3.1 --ana--.--B-Instrueec. followed by second stage of fine-tuning on 80,000 high-quality annotations from Llama-3 1aama-3.1140555 1--015---nsntiie
Chinese Classifier. We trained the Chinese classifier in-house. To generate training dataset, we translated the annotation prompt from Wettig et al. (2025) into Chinese and used it to col- lect 500,000 annotations with DeepSeek-rl. We then used these annotations to fine-tune the chinese roberta wwn ext large model (Cui et al. 2021), which serves our final Chi- nese domain classifier.
DOMAIN DESCRIPTIONS
This section details the topic schema used for our 49-class divergence experiments. create fine- grained, semantically meaningful domain labels, adopted the 24-topic classification system pro- posed by Wettig et al. 2025). The description and scope each topic are provided the table
F EXPERT ACTIVATION HEATMAPS
In figure we provide the complete expert activation heatmaps for all MoE layers in our 15B- A1.5B models, offering comprehensive view of their routing behaviors. Across all layers. models trained with Expert Divergence Learning, particularly the 49-class variant, exhibit more distinct and diversified expert utilization patterns for different domains compared to the baseline.
17

---

Published as a conference paper at ICLR 2026
Figure 5: Expert activation heatmap of layers 15B-A1.5B models.
18

---

Published as a conference paper at ICLR 2026
Table 5: Domain schema for the 49-class divergence experiments. The 24 topics listed below were applied independently to both the English and Chinese corpora. Mathematics was treated as a dis- tinct, 25th topic, resulting total of 24(EN) + 24(ZH) + (MMahh) 49 unique domain labels.
Topic
Description
Adult
General adult-oriented content.
Art & Design
Includes: architecture.
Crime Law
Includes: law enforcement.
Education & Jobs
Includes: pedagogy.
training, certification, and academic topics.
Entertainment
Includes: music, movies, TV shows, videos, celebrities, humor, and nightlife.
Fashion & Beauty Finance Business
Includes: clothing, accessories, and cosmetics.
Includes:
taxes, regulations, investments, insurance, credit, personal finance, and corporate matters.
Food & Dining
Includes: recipes, groceries, beverages, and restaurants.
Games
Includes: video games, board games, and gambling.
Hardware
computer hardware, phones, televisions, and other consumer electronics.
Includes:
Health
Includes: medicine, wellness, mental health, and veterinary science. geography and archacology.
History Home & Hobbies
Includes:
Includes: real estate, renting, furniture, home improvement, DIY, gar- dening, pets, toys, and collecting. Includes:
Industrial Literature
topics related to manufacturing, utilities, and construction. Includes: literary criticism, linguistics, philosophy, and related subjects in the humanities.
Politics
Includes: social issues, political campaigns, the legislative process, geopolitics, and activism. Includes: spirituality.
Religion Science & Technology
Includes: physics, chemistry, biology, environmental science, mathe- matics, statisties, and biotech.
Social Life Software
Includes: family, friends, relationships, and community. Includes: topics related to the use of software and the internet. Includes: algorithms, coding, and web development. Includes: martial arts, motor sports, outdoor activities, and sports equipment.
Software Development
Sports & Fitness
Transportation
Includes: cars and other vehicles, taxis, public transportation, traffic, commuting, aviation, rail, shipping, logistics. Includes: hospitality. hotels, cruises and sight-seeing.
Travel
G USE OF LARGE LANGUAGE MODELS DURING PAPER WRITING
During the preparation of this manuscript, we used large language model to polish our writing. Its primary role was refine the text that we had already written, including the descriptions of our methodology and the presentation of mathematical derivations. This was done to improve the clarity and readability of the paper and hopefully make it easier for the readers to understand. We've 've care- fully checked that the LLM- -LM-generated texts make sure there are contents that are inconsistent with our initial intents.
H ADDITIONAL TRAINING DYNAMICS AND SPECIALIZATION ANALYSIS
H.1 TRAINING CURVES TO 100B TOKENS
We present the full training loss trajectories extending to 100B tokens of all model scales in Figure The curves demonstrate that our Expert Divergence Learning method consistently achieves lower language- moddeingg loss compared to the standard MoE baseline throughout the entire pre-training process.
19

---

Published as a conference paper at ICLR 2026
3B-A0.38 Models
Standard
2.44
Bsclass Dio.
49-class Div.
Modgigg Loss
2.42
Language Modeling
2.40
2.38
2.36
40000
50000
60000
70000
B0000
90000
Training
(a) 3B-A0.3B Model
BB A0.88 MoE Models
2.36
49-class Div..
Language-Modellngg Loss Cun
2.34
2.32
2.28
40000
50000
60000
70000
80000
90000
Training Step Dlog scale)
(b) 8B-A0.8B Model
150 A1.58 5B MoE Models
Standard McE
Language-Modelngg Loss
60000
70000
Training Step llog stale)
(c) 15B51..BB Model
Figure 6: Training Loss Curves to 100B Tokens. Comparison of the language modeling loss (LLM) curves for 3B, 8B, and 15B MoE models.
H.2 INVERSE EXPERT ACTIVATION HEATMAP AND TERNARY SIMPLEX ANALYSIS
To investigate expert specialization from an expert-centric perspective, we visualize the inverse con- ditional probability P(Domain Expert) in Figure 7 These plots illustrate the domain composition of tokens processed by each expert. The results indicate that our method, particularly the 49-class divergence variant, significantly enhances expert specialization. The token distribution for each
20

---

Published as a conference paper at ICLR 2026
(a) Layer
(b) Layer 4
(c) Layer 14
Figure 7: Inverse expert activation heatmaps of different domains for representative layers (0, 4, 14) of the 15B-A1.50 models. Each row shows how often an expert is selected for a particular domain. Darker colors indicate higher frequency.
expert becomes highly concentrated on specific domains rather than being uniformly distributed, confirming that experts have evolved into specialized modules rather than remaining generalists.
To further provide a holistic view the global specialization landscape, we project the P(Domain Expert) distributions onto ternary simplex, forming a triangle plot in Figure g In this visual- ization, points near the vertices represent "pure" specialists that exclusively process inputs from single domain, while points near the centroid represent generalists processing all domains equally. This global visualization reveals striking contrast: while baseline experts cluster near the center (indicating generalist behavior), method effectively pushes experts toward the distinct domain corners. This trend is consistent across layers, visually confirming that our experts have transformed into functionally specialized units.
I STABILITY AND GRANULARITY ANALYSIS
To demonstrate the statistical significance of improvements and potential fluctuations in evalua- tion results, we conducted a rigorous stability analysis. We evaluated our largest model (15B-A1,5B) across four distinct late-stage training checkpoints (94B, 96B, 98B, and 100B tokens).
As detailed in Table 6, both the 3-class and 49-class Expert Divergence Learning methods consis- tently outperform the standard MoE baseline every single checkpoint. We observe the following trends.
The standard deviation across these checkpoints is low (a = 0.13 - 0.21), indicating stable convergence.
Low Variance:
Significant Gap (Ours vs. baseline): The average performance improvement of our 49-class model over the baseline is 1.10 points (36.69 35.599, which significantly exceeds the standard deviation (1.10 3d)) This satisfies the 30 criterion for statistical significance, confirming that the observed gains are robust and not due to random training noise.
Impact of Finer Granularity: While the 3-class scheme is competitive, the 49-class setting achieves higher checkpoint-average score (36.69 vs. 36.36) and wins the majority of
21

---

Published as a conference paper at ICLR 2026
Expert Distribution (Temary) Layer_0
Math
Nath
Chinese
English
Chinese
English
Chinese
English
Basellie
3-class
49-class
(a) Layer 0
Expert Distribution (Temary) Layer_
Baselline
3-class
49-class
(b) Layer 4
Expert Distribution (Temmayy) Layer_14
Chinese
English
Chinese
Baseline
3-caas
49-class
(c) Layer 14
Figure 8: Ternary simplex plots showing the expert distribution across domains (English, Chinese, Math) for representative layers. Points near corners represent specialized experts, while points near the center represent generalists.
benchmarks (6 out of 7). Furthermore, the 49-class model achieves the lowest final training loss across all model scales (Figure ( suggesting provides a better optimization landscape. To investigate the upper limit of granularity, further perform ablation treating each docu- ment as a unique domain. This yields training loss trajectory slightly worse than the 49-class setting. This indicates trade-off: extreme granularity maximizes repulsion but penalizes rout- ing consistency between semantically similar documents. Thus, semantic grouping (as in our 49-class scheme) strikes better balance.
J COMPATIBILITY WITH ADVANCES IN MoE
We have validated our proposed Expert Divergence Learning within the standard and widely adopted MoE architecture and training pipeline. It serves flexible training objective that orthogonal to to-and can be effectively combined with-other recent advancements in MoE architectures and training strategies.
1. Shared-Expert Architectures. Recent architectures often employ "shared experts" that are always activated to capture common knowledge, alongside "routed experts" for specialized tasks (DeepSeek-Al 2024). Our method is highly compatible with this design. While shared ex- perts handle universal features. LED can be explicitly applied to the routed experts to maximize their functional distinctiveness. enforcing divergence among the routed experts, method fur-
22

---

Published as a conference paper at ICLR 2026
Table 6: Stability Analysis of 15B-A1.5B Models. Performance comparison across late- train- ing checkpoints (94B to 100B tokens). The results demonstrate consistent improvements with low variance, confirming the statistical significance of our method.
Model
Checkpolnt
C-Rval
MMLUU
CMMLU
ARC-c
RACE-m
RACE-h
Aug.
948
12.61
33.12
56.35
52.65
35,50
57.10
35.78
Baseline Mol
98B
32.78
33.28
57.45
32.05
28.55
35.68
1000
32,80
33.24
34,64
36.61
29.83
33.36
28.64
35.59
Mean (j+ + +)
32.74 + 0.08
33,25
0.10
34.57 + 0.08
56.88
01.49
29.90 + 0.27
33.13
0.40
28.65 + 0.29
35.59
0.21
94B
32.35
32.88
34,95
58, 33.40
53 20
32.95
28.40
36, 16
96R
32.48
33,02
35.25
59.25
53.65
33:15
28.65
36 46.49
3-class Div.
98B
32,50
32.96
35.15
58.65
33.15
33.65
26.90
36,43
1008
32.98
32.94
36.54
(a 1. d)
32,96
33.17
34.22
96B
33.34
33.22
35.15
59:23
33.49
34.45
28.83
36.81
+ 499-llass DNv.
988
35.58
32.95
35.45
58.79
33.55
34.58
29.07
36.84
100R
33.45
33.21
35.17
$7.85
53.56
34.54
28.76
36.65
Mean (j+ f o)
33.37 + 0.18
33.16 + 0.14
35.27 + 0.14
58.55 0.60
33.43 + 0.19
34.45 0.14
28,00
+ 0.21
36.69 0.13
ther enhances the architectural goal of separating general capabilities (shared) from specific ones (routed), potentially leading to cleaner modularization.
Luxiliary-Loss-Frre Balancing. Some recent works propose replacing the auxiliary load- balancing loss (CI.) with bias-based update mechanism (Wang et al.] 2024b; DeepSeek-AT] 2024). We clarify that this mechanism targets Load Balancing (enforcing uniformity), whereas our LED targets Expert Specialization (enforcing divergence). These objectives both important and non-conflicting. LED can be seamlessly integrated as an auxiliary loss alongside bias-based bal- ancing to guide semantic specialization while the bias terms independently manage computational load.
3. Unsupervised Router Regularization Methods like ERNIE 4.5 (Baidu-ERNIE- Team 2025
introduce regularizers to enforce orthogonality on the router weight matrices. In contrast, our method enforces divergence on the dynamic routing probability distributions. These approaches operate at different levels: ERNIE enhances the independence of the router's feature representation space in an unsupervised way, while LED utilizes data-driven supervision ensure that semantic differences translate into structural differences in expert usage. Combining them would likely yield additive benefits, improving both representation quality and semantic specialization.
K COMPARISON WITH ERNIE 4.5 ROUTING ORTHOGONAL LOSS
We provide empirical evidence to confirm the compatibility of our approach with the MoE routing orthogonal loss introduced in the contemporaneous work, ERNIE 4.5 Baidu-ERRNI Team/ 2025), as discussed in [ppendix [ As illustrated in the training loss curves in Figure 9 and the perplexity on validation sets in Table 7 combining both objectives on the 8B-A0.8B model consistently exhibits
additional gains compared to using the ERNIE 4.5 loss alone. These additive gains demonstrate that the two methods are not only compatible but mutually beneficial.
Table 7: Comparison of Validation Loss
Models
zhlval. 11000
en.val. 1000
math val_1000
Only ERNIE4.5 Loss
2.7974 2.7793
3.1905 3.1905
1.7556
Combine with Ours
1.7533
L GRANULARITY ABLATION: DOCUMENT-BASED vs. DOMAIN-BASED DIVERGENCE
To investigate the impact of granularity and validate our design, we conduct an ablation study by treating every document distinct class. This contrasts with proposed Expert Divergence
23

---

Published as a conference paper at ICLR 2026
2,650
Mot 49-class Cop FAznL
Banguage-Hoon Modelng Loss Cur
2,600
2.575
2.550
Language
2.525
2.500
2,450
7500
10000
12500
15000
17500
Triiiing Step Scale)
Figure 9:
Loss Comparison of Using ERNIE 4.5 Loss and Adding the Expert Divergence Loss. Adding our loss shows additional loss gains.
method, which relies on source-based (3-class) and topic-based (49-class) domain label schemes. We trained an 8B-A0.8B model from scratch for 100B tokens under this configuration.
We report the mean and standard deviation across the final three checkpoints (96B, 98B, and 100B) in the table below. The ablation configuration yielded suboptimal results (34.88), even underper- forming the Baseline MoE (35.12). This performance drop confirms the critical importance of se- mantically meaningful domain labels used. Without semantic grounding, the divergence loss indiscriminately separates similar sequences in the expert distribution space, disrupting the intra- domain consistency required for stable expert specialization. This outcome reinforces our arguments in two key aspects:
1. It validates the necessity of domain classifiers. Since both coarse data sources (3-class) and granular document IDs (this ablation) underperform the 49-class setting, the investment in domain classifier is proven essential.
2. It serves as negative control verify the validity of our performance gains. This ablation isolates the impact of the labeling strategy by keeping all other settings constant. The stark wohtrast-where our proposed method consistently improves performance while the document- level ablation consistently degrades ct-fonfimms that gains are significant and stem directly from the effective application of Expert Divergence Learning, rather than from random bench- mark fluctuations.
Table 8: Ablation Study of Domain Labels
Avg.
Baseline Mol
32.78 + 0.22
32.18 0.11
34.11 = 0.34
54.31 + 0.41
31.78 = 0.32
$2.31
28.36
0.37
35.12 0.17
One-class- per-doc Div.
32.67 + 0.33
31.94 = 0.09
33.92 = 0.39
$4.79 +: 0.52
30.51 = 0.22
32.43 0.3
27.92 0.21
34.88
M EXTENDED TRAINING TO 150B TOKENS
We scaled the training of the 15B-A1.5B models to 150B tokens (a 50% increase) to verify per- formance stability. By averaging statistics from the final three checkpoints, we observed key trends: First, while extended training improved convergence for all models, our method consis- tently retained its lead. Second, strict performance rank of "49-class 3-class ? Baseline" was evident on every dataset. These results validate that Expert Divergence Learning laiiinggppticuullllyy with fine-grained 49-class labels delivers robust and comprehensive performance gains.
Table Performance Comparison with Different Class Divisions
Model
CEVAL
MMLU
CMMLU
ARC-e
ARC-
RACE-m
RACE-h
Baseline MoB
34.54 + 0.25
33.65 0.31
35.89 + 0.49
$9.32
0.56
34.35 0.54
33.50
0.30
28.34 = 0.5
37.08 + 0.35
+3-class Dis:
35.43 1.24
34.60
0.25
36.52
0.33
60.73
0.54
35.14 0.41
33,89
1.08
28.69 0.37
37.86 4 0.27
+49-class
36.11 0.55
34.68
36.58
0.33
60.85
0.36
35.25 0.22
34.60
0.32
28.73 0.32
38.12 0.21
24


Reviews:
[{"review_id": "WA57s8oMfC", "reviewer": "Reviewer_Jprx", "summary": "This paper proposed a pre-training strategy to increase expert divergence in MoEs, hence, specialization through MoE experts. They use domain labels from the subsets of pretraining data and maximize the pairwise Jensen-Shannon divergence between expert routing distributions of data domains. They show that their method leads to better language modeling loss and also higher downstream performance.", "questions": "1. How sensitive is the Expert Divergence Learning to differences in data volume in the domain dataset? It is common that some domains might be much smaller than others in an actual pretraining run.\n\n2. How to ensure balancing across data labels in a single batch? Is the proposed method sensitive to such balance?\n\n3. It is very hard to understand how the specialization is realized from the current experiments (For example, 3-class div still has>60 experts to specialize). Would it not be good to decrease the expert number to show specialization by removing the specialized expert on one domain in the test time (such as Chinese)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Specialization in MoEs is often underexplored, but it is highly important for modularity and effectiveness. \n\n2. The idea is quite novel and very intuitive. \n\n3. A clear trend in language modeling loss suggests that the method proposed carries a strong potential.", "weaknesses": "1. I think the main weakness of the paper is the results, where there is no clear significant improvement in the majority of downstream tasks. More concretely, there are some items leading me to suspect the results: \na. Although the model size increases significantly (both total and active params), there is no monotonic increase in evals with large model size (except ARC_e). \nb. Improvement claimed by the proposed method is quite uneven, mainly concentrated on ARC_e\nc. Results are too close to draw a conclusion. There needs to be a significance test or multiple runs with different seeds to validate the overall improvements. \n\n2. The routing is token-wise; however, the proposed auxiliary loss is sequence-wise. It is not clear whether this could lead a suboptimal routing for MoE, which needs to be examined and showcased.  \n\n3. For the additional question, I refer to the \"Questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oY1YTA7Rst", "reviewer": "Reviewer_PgDX", "summary": "The paper proposes an additional loss for the training of transformer MoE models that promotes expert specialization per domain. In more detail, the loss maximizes the pairwise JS divergence between the distributions over experts (averaged per token for each domain). The paper shows that training language models using this loss in addition to the language modeling loss and the expert balancing loss results in better perplexity and better downstream performance. Finally, the authors analyze the behavior of the routers in such models and show some evidence that there is more per-domain specialization.", "questions": "I have mostly laid out my questions in the weaknesses section but maybe a small recap:\n\n- What is the per-checkpoint variance in these evaluations? How can we increase our confidence that the improvement is real and consistent given that the scores are so close?\n- Can we see the final loss and the evolution from 65B tokens and afterwards?\n- Is ΔPPL is the correct metric? How does it measure per-domain specialization?\n- Can we see different visualizations that clearly show that the experts are specialized per domain? ie some experts should be way more probable for math than for English and this should be clearer or stronger when using the loss.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The most important (albeit possibly weak as discussed in weaknesses) strength of the method proposed in this paper is the experimental results of section 4.2, namely the fact that adding this loss does indeed improve consistently over training the same model without using this loss.\n\nIn addition,\n\n- The paper is very well written and presented.\n- The idea is both simple and intuitive. It can probably be implemented in a few lines of code in any framework.\n- The authors have covered all possible aspects of the impact of their loss to the model and its training. In particular, I appreciate the analysis on training and inference throughput.", "weaknesses": "Given that the idea is simple (which I consider a major strength) the weight falls to the experimental results to convey consistent improvement and at the very least that the method has the intended impact on the model.\n\nHowever, the downstream evaluations have several relatively worrying qualities. Firstly, the average scores are compared and the conclusion that 49 domains is better and bigger models are more amenable to per-domain specialization. Looking closely at the scores though, there are many cases where the 3 domains are better and where the baseline is better than one of the two contenders. Including the fact that the differences are relatively small for evaluations that have quite high-variance from checkpoint to checkpoint it makes it very hard to draw the conclusion that the loss leads to consistent improvement.\n\nA further minor point for the downstream evaluations is that the scores do seem a bit low. For instance 100B tokens for a 15B/1.5B MoE should probably achieve more than 56.6 on Arc easy. Having said that, the dataset could be the reason for this discrepancy so I do not consider this a major issue. Another minor concern is the fact that the largest model performs worse on 2 out of 7 benchmarks compared to the model with half the parameters.\n\nThe plot of the log-likelihood also raises a few concerns. The differences are relatively small and even though the loss seems to improve the log-likelihood, we only see from ~35B to ~65B tokens. What are the final losses at 100B and how does the training compare in the other stages of training (that would only be the middle).\n\nRegarding section 4.3 that analyses the effects of the loss on the model. Permuting the distribution of the router and measuring ΔPPL measures how sensitive the model is to the particular expert choices but it is not clear how it measures per-domain specialization. Simply put a model that had collapsed to always select a single expert would get the worst score on that test and its experts would be anything but specialized.\n\nThe heat-maps more directly measure the intended effect of the loss to the model. However, I would propose to the authors to find both a better way to visually show whether 49-class Div vs the Baseline are per-domain specialized. To more clearly point to the problem, specialization can be seen for layer-0 and layer 14 but not for layer 4 which according to the previous test is the most-specialized. I suggest that the authors try to measure the portion of experts that are mostly selected on each domain. For instance if there are experts specialized for a particular domain it should be clearly visible in the inverse distribution eg how often is an expert selected for a token from a particular domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ruryJiWcph", "reviewer": "Reviewer_76XP", "summary": "When training mixture-of-experts, the router learns correlated experts, i.e. groups of experts that are often active at the same time. This routing configuration is not optimal, i.e. lower training loss can be achieved with routing that has less correlated experts. This paper \npropose to promote the JS divergence between routing patterns across different domains. Positive impact on the training loss and on end-task performance is shown with the proposed strategy.", "questions": "1. Are you working to compare your JSD regularizer with the orthogonalization method of Baidu ERNIE 4.5?\n2. Are classes necessary? See weaknesses 2.\n3. Does the JSD regularizer have a negative impact on the balancing regularizer L_LB? When training with JSD, are consecutive routing decisions for the same document more likely to overlap (promoting inter-example JSD might negatively impact intra-example JSD)?\n4. How did you pick the L_LB regularizer parameter alpha? \n5. Could JSD be useful only at the beginning of learning? I.e. once a good initial configuration is found, training can be pursued without JSD regularization. It seems the gap between standard and JSD training is not growing (Fig 2) what happens if JSD is turned off after ~20k steps?\n6. Since the largest number of classes give the best results, could better results be achieved with more domains? If the batch size is small compared to the number of classes, rarely the training algorithm will see two documents of the same class in the same batch. The method is equivalent to one class per document. Have you tried one class per document? Does your method work with smaller batch sizes?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper gives a good overview of related work in routing strategies. \n2. The experiments are performed over a variety of model sizes and multiple end-tasks are considered for evaluation. Performance is reported both in terms of loss/perplexity and end task performance.\n3. The method is simple and clearly presented.", "weaknesses": "1. The authors do not show whether better results could be achieved with a different routing strategy. So far JSD is only tried on top of z-loss with alpha=1e-3. Is it the best alpha? Would JSD be impactful on top of a different alpha? Is JSD impactful when the routing model has also always-active FFNs? Is JSD impactful when one uses another way to promote uniform expert assignments, e.g. DeepSeek bias based balancing https://arxiv.org/pdf/2408.15664? \n\n\n2. It is not clear whether the domain classifier is necessary. The best results are with 49 classes. From your batch size it seems that a batch contains only a couple documents per domain. Would it be simpler to define each document as its own domain and just push the JSD across all pairs of examples.\n\n\n3. It would be good to quantify how expert assignment changes with the proposed regularizer. When one takes a pair of documents, what is the typical number of experts per document? What is the number of experts shared by the two documents? Are these numbers the same across depths?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xlv72TegMt", "reviewer": "Reviewer_m6YE", "summary": "The paper tackles expert homogenization by maximizing routing divergence between domains using their inherent labels as supervision. Results show improvements over baseline MoE across different scales, with larger models and finer domain granularity yielding better gains.", "questions": "Table 3 shows minimal throughput reduction, but this only captures the direct computational cost. The paper doesn't account for the infrastructure needed to maintain domain labels, the cost of training and running domain classifiers at scale, or the additional memory overhead of tracking domain-specific statistics during training.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The core idea is well-motivated. Using domain labels already present in pretraining data to explicitly guide expert specialization addresses a real problem in MoE training. The theoretical framing through divergence decomposition is clean, and the experiments are thorough across multiple model scales with consistent improvements.", "weaknesses": "(1)  While the authors demonstrate improvements across three model sizes, the largest model evaluated is only 15B parameters. Given that many SOTA MoE models operate at larger scales and authors have the enough GPU resource (64*80GB), it remains unclear whether the observed benefits would translate to larger MoE LLMs. \n\n(2) The method fundamentally relies on having meaningful domain labels for the training data. However, the paper provides limited analysis on how sensitive the approach is to labeling noise or granularity mismatches. For example, what happens when documents contain mixed topics, or when the domain classifier makes systematic errors? The Chinese classifier was trained on only 500K examples from DeepSeek-r1, which may introduce biases that propagate through training. \n\n(3) The paper positions itself against standard MoE training but doesn't compare with other recently proposed solutions to expert homogenization. Shared expert architectures (referenced in the related work) represent a different paradigm for handling commonalities across domains. Similarly, the concurrent ERNIE 4.5 work on router weight orthogonality is mentioned but not empirically compared.\n\n(4) The improvement from 3-class to 49-class domain schemes is relatively modest (36.34 to 36.65 average score for 15B). This raises questions about the cost-benefit tradeoff of investing in high-quality, fine-grained topic classifiers. The paper would benefit from an analysis of the relationship between domain granularity and performance gains, perhaps including intermediate granularities (e.g., 10-class, 20-class) to identify diminishing returns.\n\n(5) All evaluation benchmarks fall roughly within the scope of the three training domains (English, Chinese, Math). The paper doesn't test how the model performs on domains significantly different from the training distribution. For example, code generation or scientific reasoning. If specialized routing patterns become too rigid, there's a risk of reduced flexibility when encountering truly novel inputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)